arXiv:2605.12610v1 [cs.SE] 12 May 2026
Fine-Tuning Models for Automated Code Review Feedback Smitha Kumar∗∗
Michael Adam Lones
[email protected] [email protected] Heriot-Watt University Dubai, United Arab Emirates
Heriot-Watt University Edinburgh, United Kingdom [email protected]
Manuel Maarek
Hind Zantout
Heriot-Watt University Edinburgh, United Kingdom [email protected]
[email protected] Heriot-Watt University Dubai, United Arab Emirates
Abstract Large Language Models (LLMs) have introduced new possibilities for programming education through personalized support, content creation, and automated feedback. While recent studies have demonstrated the potential for feedback generation, many techniques rely on proprietary models, raising concerns about cost, computational demands, and the ethical implications of sharing student code. Open LLMs provide an alternative approach, but they do not currently have the capabilities of proprietary models. To address this problem, we investigate whether parameter-efficient fine-tuning (PEFT) and prompt engineering—both of which distil knowledge from a dataset derived from a large, more capable model can be used to adapt and enhance the quality of feedback generated by the open LLM Code Llama. Feedback quality on buggy Java code was assessed using a combination of student evaluation, manual annotation and the automated metrics BLEU, ROUGE, and BERTScore. Our findings indicate that PEFT leads to notable improvements in feedback quality, and significantly outperforms prompt engineering, providing an avenue for developing freely-deployable feedback tools that can be effectively used to guide student learning. Student evaluation indicates that learners value the PEFT model’s feedback, and see it as being equally effective as the proprietary ChatGPT model. Participants suggested that incorporating additional explanation for technical terms in the PEFT model’s feedback could be more beneficial. This study demonstrates that fine-tuned models can effectively support critical thinking and guide the design of scalable pedagogical systems.
CCS Concepts • Applied computing → Computer-assisted instruction. ∗ all authors contributed equally to this research.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference acronym ’XX, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
Keywords Large Language Model, Programming Feedback, Fine-tuning ACM Reference Format: Smitha Kumar, Michael Adam Lones, Manuel Maarek, and Hind Zantout. 2018. Fine-Tuning Models for Automated Code Review Feedback. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX). ACM, New York, NY, USA, 8 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
Generative AI tools have significantly reshaped programming education in recent years. Prior to the emergence of Large Language Models (LLMs), automated feedback generation techniques were constrained by several technical challenges that hindered their broader adoption in educational settings [18]. Recent studies indicate that LLMs are capable of outperforming traditional feedback generation approaches [37], which has led to a significant increase in leveraging them for more precise, personalized, and effective feedback. According to Narciss [26], “feedback refers to all postresponse information which informs the learner on his/her actual state of learning or performance in order to regulate the further process of learning”. Feedback types include knowledge of performance (KP), knowledge of result or response (KR), and knowledge of the correct response (KCR). More elaborate forms of feedback, such as knowledge about task constraints (KTC), knowledge about concepts (KC), knowledge about mistakes (KM), knowledge about how to proceed (KH), and knowledge about meta-cognition (KMC) offer deeper guidance to learners. Students often rely on proprietary LLMs to receive prompt responses during programming education [34]. However, such models are often costly [16], lack transparency, and offer limited customization options for specific downstream tasks [22]. Despite their benefits, LLMs sometimes generate hallucinations, which can confuse learners [31]. Moreover, they may provide complete correct solutions without guiding learners through the problem-solving process. This can hinder the development of critical thinking and problem-solving skills, which are some of the key competencies in programming education [29]. Proprietary LLMs are black-box models that do not release the model weights, whereas open LLMs provides access to their model weights and can be freely deployed on local computer systems [38]. Open LLMs allow fine-tuning to suit diverse needs, while also providing greater transparency, privacy, and control [22]. Therefore,
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
harnessing the potential of open LLMs to suit pedagogical goals is an important step towards their widespread acceptance in educational settings. Recent studies on automated feedback generation techniques report that the majority of tools generate KM type feedback, with limited support for KH type feedback, which offers more effective guidance to learners [25]. This study aims to develop a novel framework based on PEFT to create high-quality actionable feedback for Java programming assignments. We also investigate how this approach compares against prompt engineering, an approach that can be used to generate KMtype and KH-type responses without additional training. A key challenge in programming education research is the lack of publicly available datasets. Most existing studies do not release their datasets, which limits the validation of their findings. To overcome this limitation, researchers have begun leveraging LLMs to generate diverse datasets [20]. As LLMs are largely trained on datasets written by humans, this approach offers a viable alternative for dataset generation. For the purposes of this research, we created and shared publicly a new dataset containing code, KM-feedback, and KH-feedback. This research study is guided by the following research question: How can large language models be optimized to generate high-quality targeted and actionable feedback? Our contributions include: • A novel PEFT framework to generate pedagogically structured feedback. • Empirical evidence demonstrating that PEFT leads to better feedback than baseline prompt engineering strategies and performs comparably to proprietary models. Student evaluations indicate a preference for this feedback over ChatGPT in a formative lab setting. • A new dataset1 of annotated Java programs with KM-KH responses covering diverse error types in Java programming. This article is organized as follows. In Section 2, we discuss the use of LLMs in programming education and review key optimization strategies. Section 3 outlines the research methodology. Section 4 presents the findings of this study including its limitations. Finally, conclusions and recommendations for future work are covered in Section 5.
2 Background 2.1 LLMs in Programming Education The programming education landscape has been profoundly impacted by LLMs that generate code. Existing literature reveals that LLMs are leveraged in programming education for tasks such as code generation, debugging, clarifying complex concepts, and providing personalized feedback. Existing research also shows that students generally hold positive attitudes towards the use of LLMs [2, 28]. LLMs help students learn independently at their pace, receiving instant feedback and explanations for coding challenges. This real-time assistance across multiple programming languages enhances the learning experience [6]. The proprietary models used in most of these studies are associated with financial constraints, making open LLMs a viable alternative [15]. However, these models have yet to match the 1 https://anonymous.4open.science/r/JCODE_KM_KH-4BEC
Trovato et al.
performance of proprietary models [17]. While previous studies have demonstrated the potential and challenges associated with LLMs, their effectiveness depends greatly on how they are optimized for specific use cases. Prompt engineering and fine-tuning have emerged as some of the most promising techniques to enhance the performance of the models on downstream tasks [30]. The following section provides an overview of prompt engineering and parameter-efficient fine-tuning (PEFT).
2.2
Prompt Engineering
Prompt engineering involves guiding a pre-trained LLM to generate a desired output using natural language queries. Zero-shot prompting and in-context learning are the most commonly used prompting strategies. In zero-shot prompting, the model is provided with a direct natural language instruction to generate a response for a task, and relies only on its trained knowledge. In-context learning extends this approach by including additional examples along with the query. Importantly,these approaches do not update the model’s internal parameters [30].
2.3
Parameter-Efficient Fine-Tuning
Fine-tuning has emerged as a practical approach for adapting large pre-trained code LLMs to various downstream tasks. It can be broadly classified into two types depending on the number of parameters updated: full fine-tuning and parameter-efficient fine-tuning (PEFT). Full fine-tuning potentially updates all model parameters and can require large amounts of training data and computational resources. In contrast, PEFT can be done using much more limited computational resources as it updates only a subset of the model parameters. This also helps to minimize the risk of catastrophic forgetting. PEFT can be implemented using various approaches, including Low-Rank Adaptation (LoRA) [12] and Quantized LoRA (QLoRA) [8]. LoRA adds trainable low-rank matrices to the LLM’s transformer layers while keeping the base model weights frozen. This approach significantly reduces the number of trainable parameters and computational overhead without compromising the model’s accuracy. Quantized LoRA (QLoRA), which we use in this research, builds on LoRA by incorporating quantization that further reduces the memory footprint without degrading performance. It uses a 4-bit floating-point value to store the model weights, applies double quantization, and uses paged optimizers to dynamically manage memory during training to achieve optimal performance.
3
Methodology
In this section we explain the dataset creation process, model architecture and configurations, and the evaluation metrics. An overview of the methodology is shown in Figure 1.
3.1
Dataset Creation
Due to the scarcity of openly shared data in the programming education domain, researchers have investigated the ability of LLMs to generate synthetic data and found that the generated samples are not significantly different from actual student submissions [20]. To ensure that the synthetic dataset accurately reflects student errors, we analysed the following research papers published in prominent
Fine-Tuning Models for Automated Code Review Feedback
A. Bug Type Dataset Creation
CSE Paper 1 • • • CSE Paper 13
List bug types identified by the literature
Bug Type 1 • • • Bug Type 85
For each bug type, use DeepSeek LLM to create 5 bug sample triplets (buggy code, KM, KH)
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Prompt 1 Bug Type
Bug Sample 1 • • • Bug Sample 425 If unsuitable bug sample triplet, discard and generate new triplet
B. Evaluation of Feedback Generation Strategies
PE-Fewshot Bug Sample 1 • • • Bug Sample 425
Prompt 3 Bug Sample
Fine tune Llama PEFT using 5- fold crossvalidation of bug samples Baseline
Manual evaluation metrics ● KM Accuracy ● KH Helpfulness ● Misleading Suggestions ● Prompt Adherence Automated evaluation metrics ● KM-BLEU ● KM-ROUGE ● KM-BertScore Student Evaluation
Prompt 2 Bug Sample
Figure 1: Overview of methodology, showing (A) the bug type dataset creation presented in Section 3.1, and (B) the different feedback strategies presented in Sections 3.2 and 3.3 and the evaluation metrics presented in Section 3.4
Prompt 1: Prompt template for triplets (Code, KM, KH) SYSTEM: You are an expert programming tutor with strong knowledge of common student mistakes from programming education literature. USER: Your task is to generate five buggy Java code snippets that illustrate the common mistakes made by students in introductory undergraduate programming courses (CS1 and CS2 levels) aligned with the bug type specified below. Bug type: <BUG_TYPE> For each code provide: ** KM (Knowledge about mistakes) ** : Provide feedback that helps students understand their mistakes. If there are multiple mistakes, list them individually. ** KH (Knowledge about how to proceed) ** : Provide guidance to the students on how to fix the mistakes. Do not provide the corrected code. Instead, please provide short, concise, and easily understandable explanations. Here are a few examples: <JAVA CODE> <KM: KM response KH: KH response>
Response 1: Sample bug triplet
computer science venues between 2005 to 2024, aiming to identify import java.util.*; common bug types frequently observed in student submissions [1, class Vehicle {} 3, 5, 7, 9–11, 13, 14, 19, 23, 24, 32]. We observed that certain errors class Car extends Vehicle { were repeated across multiple papers. In some instances, errors with void startEngine() { unclear or incomplete details were excluded. A total of 85 distinct System.out.println("Vroom!"); } } bug types were identified, each type assigned a unique identifier public class Main { (BugID) and recorded only once, regardless of its occurrence across public static void main(String[] args) { multiple publications. The full list of bug types is available at 2 . List<Vehicle> vehicles = The bug types served as the basis for generating the synthetic new ArrayList<>(); data in this study. These include syntax, semantic, logical errors vehicles.add(new Car()); and conceptual misunderstandings commonly found in student for (Vehicle v : vehicles) { submissions, and cover both compilation-time errors and runtime v.startEngine(); errors. To provide deeper insights into the model’s capabilities, the } } } bug types are classified into two categories: imperative (I), and Knowledge about mistakes (KM) : Compile-time type of object-oriented (O). ‘v‘ is ‘Vehicle‘, which lacks the ‘startEngine()‘ method. PolyWe employed prompt engineering with Deepseek-R13 to genermorphism doesn’t automatically expose subclass-specific ate labelled triplets in the form of <Code, KM-feedback, KH-feedback> methods without explicit casting. to train our model. Deepseek-R1 was chosen for this study because Knowledge about how to proceed (KH) : Cast ‘v‘ to ‘Car‘ its license permits the model’s output to be used in training other inside the loop before calling ‘startEngine()‘, ensuring ‘v‘ is models. Each bug type is illustrated with a corresponding code snipa ‘Car‘ instance. pet that demonstrates a distinct code scenario. In each iteration, the placeholder <BUG_TYPE> in Prompt 1 was replaced with a specific 3.2 PEFT-Driven Framework for Pedagogical bug type (BugID) from 4 . For each bug type, the model generated Code Feedback five distinct triplets, which were subsequently validated by the first Our framework builds on Code Llama-7b-Instruct-hf, a LLaMA author in order to verify the correctness of the feedback. A total of causal language model based on the transformer architecture with 425 triplets were created for this study, containing 5 samples for 32 layers and support for very long context lengths. The frameeach BugID in order to reflect natural variation in the occurrence work’s technical components include the base model, PEFT adapter of each bug. The generated feedback serves as the ground truth module, input encoding and the training dataset. Code Llama-7Bfor this research. This dataset has potential to support the developInstruct is based on the foundation model Llama 2 [36] and is trained ment of pedagogical models, and facilitate future studies. A sample on 500B tokens and designed to follow human instructions. Addicode snippet and the associated KM and KH response is shown in tionally, it extends the context length from 4,096 tokens used in Response 1. Llama 2 to 16,384 tokens. A longer context window enables the model to understand longer code samples, effectively process de2 https://anonymous.4open.science/r/JCODE_KM_KH-4BEC 3 https://huggingface.co/deepseek-ai/DeepSeek-R1 tailed prompts, and avoid truncations [33]. While larger models 4 https://anonymous.4open.science/r/JCODE_KM_KH-4BEC offer significant performance benefits, the 7B model is favored for
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
its efficient resource usage, ease of deployment in typical educational environments, and faster inference time. The dataset was restructured to conform to the format requirements of Code Llama training [33]: "bug_id": "ID", "prompt": "[INST] Generate detailed feedback in the format KM (Knowledge about Mistakes) and KH (Knowledge about how to proceed) for this Java code: <JAVA CODE> [/INST]", "response": "KM: <KM response> ‘."KH: <KH response>‘" The data was converted into JSON objects. To avoid overfitting and provide a robust picture of generalizability, we used 5-fold cross-validation, with each fine-tuned model instance trained on a subset of bug categories and then independently evaluated on the others. Each token is converted into a vector of size 4096. We employed QLoRA-style 4-bit quantization using nf4 format and double quantization on CodeLlama for this task-specific adaptation. During training the original model weights are frozen and the newly added weights are updated. This significantly reduces the number of training parameters from 6,743,789,568 to 5,242,880. The adapters were configured with a rank of 10, scaling factor 16 and a dropout rate of 0.08%. LoRA adapters were applied to the q_proj and v_proj projection layers. This fine-tuning strategy adapts the base model for downstream tasks with lower computational resource requirements. The prompt format is given in Prompt 2. Prompt 2: PEFT Llama and baseline prompt [INST]Generate detailed feedback in the format Knowledge about Mistakes (KM) and Knowledge about how to proceed (KH) for this Java code: <JAVA CODE>[/INST]
3.3
Prompt Engineering
We investigated the effectiveness of two prompting strategies, a baseline zero-shot prompt and in-context prompting, for feedback generation using CodeLlama-7b-Instruct-hf. For in-context prompting, we augmented the baseline prompt with three examples selected from the training dataset; see Prompt 3. A temperature value Prompt 3: In-context Llama prompt [INST]Generate detailed feedback in the format Knowledge about Mistakes (KM) and Knowledge about how to proceed (KH) for this Java code: <JAVA CODE>. Do NOT create additional examples. Example 1 <JAVA CODE> <Feedback : KM, KH> Example 2 <JAVA CODE> <Feedback : KM, KH> Example 3 <JAVA CODE> <Feedback : KM, KH> [/INST]
of 0.7, a maximum token limit of 8192, and a top-p (nucleus sampling) value of 0.7 were used. The feedback responses for the test dataset were collected using both prompting strategies.
Trovato et al.
3.4
Evaluation
Our evaluation process has three components. The first is done from the instructor perspective, and involves rubric-based manual annotations conducted by the first author, an instructor of CS1 courses. To provide a more objective perspective, this is supported by the second component, which involves automated metric analysis. The third component, to evaluate the student perspective, is a small focus group discussion involving CS1 students. For the instructor evaluation, we developed a rubric consisting of four binary criteria (see Table 1) where a score of 1 for a component indicates that the feedback perfectly aligns with the specified characteristic. In cases with multiple KM/KH responses, each response is examined for misleading information and evaluated accordingly. Cross-annotation was performed by a second researcher to ensure consistency and reliability of the labelling of the KHH, KMA, MS and PA categories. This process prompted meaningful discussions and refined the labelling criteria. For the objective evaluation, the automated evaluation metrics compute the similarity between the feedback produced by the models and the ground truth. BLEU [27] evaluates how many exact word/n-grams in the output match those in the reference text. ROUGE [21], on the other hand, measures how many n-grams from the reference text are present in the generated output. BLEU emphasizes precision, whereas ROUGE focuses on recall. Among the ROUGE variants, we report the ROUGE-L as it captures the longest common subsequence between the generated and reference feedback. These two metrics rely on exact word/phrase overlap, and they may miss feedback that is semantically similar but lexically different. Therefore, BERTScore [39], which captures the semantic similarity between the model response and the ground truth, was also employed. Specficially, we used BERTScore-F1, which balances precision and recall. Since feedback on the same idea can be expressed in multiple ways, evaluating it using multiple metrics enhances the robustness of the evaluation. The evaluation metrics were computed for all the samples in the test dataset. For the student evaluation, we employed a mixed-methods evaluation following a small focus group discussion to investigate the student’s perception of different types of feedback for a CS1 Java programming assignments. The study was conducted following the institution’s ethical approval. All undergraduate students in years 1 and 2 were invited to participate in this study. Eleven students signed the consent form, and seven students (P1 to P7) attended the meeting. Participants were provided with four buggy CS1-level Java code snippets and three different types of feedback to help them identify and fix the error in the code. The first feedback type was the compiler error message (E), the second was feedback generated by ChatGPT (C), and the third feedback was produced by the fine-tuned model (F) used in this study. Participants rated each type of feedback from 1 to 5 across three factors: usefulness, clarity and structure. In the second phase of the focus group discussion, the students were asked to reflect on their rating and discussed the rationale for it. The session was audio-recorded. We conducted a thematic analysis [4] of the transcripts to examine the students’ perception of the feedback provided.
Fine-Tuning Models for Automated Code Review Feedback
Metric KMA KHH MS PA
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Description KM Accuracy KH Helpfulness Misleading Suggestions Prompt Adherence
Addresses the core issue in the code Provides meaningful hint to the learner Wrong fix or irrelevant advice Follows the specified structure (KM, KH)
Table 1: Descriptions of feedback evaluation metrics.
4
Results, Discussion, and Limitations
The following section outlines the findings from our experiments involving PEFT and prompt engineering. We applied manual and automated evaluation metrics along with student evaluations to assess the quality of the generated responses.
4.1
Instructor Evaluation
Figure 2 summarizes the rubric-based comparative analysis conducted by the first author (CS1 course instructor). Our results show that the fine-tuned model (CL7B-LoRA-JavaFB) outperformed the other techniques on all of the four rubric criteria (see Table 1). Examples of feedback produced by each model, along with the dataset, are provided in the footnote 5 . Baseline
PE-FewShot
PEFT
100
95 86
83
80
Score (%)
61
60
60
60
54
54 47
46
40 26 20
20
e er en c
tio ns
pt Pr om
ng M
isl
ea di
Ad h
es Su gg
pf el H KH
KM
Ac c
ur
ul n
ac
es
s
y
0
Figure 2: Comparison of KM accuracy, KH helpfulness, number of misleading suggestions (lower is better), and prompt adherence across model configurations. Baseline refers to the standard Llama response. PE-FewShot refers to prompt engineering with few-shot examples. PEFT refers to the finetuned model. The fine-tuned model demonstrated superior KM accuracy, achieving 61% compared to 20% for the un-tuned baseline model. In terms of KH helpfulness, it achieved a score of 60% compared to 26%. We also observed that the rate of misleading suggestions dropped from 83% to 47%, and prompt adherence increased from 54% to 95%. These results demonstrate the potential of PEFT in generating pedagogically meaningful feedback. For the prompt engineering strategies, we see that in-context learning achieved better performance than the baseline model in 5 https://anonymous.4open.science/r/JCODE_KM_KH-4BEC
KM accuracy scores with 54% over 20%. Moreover, KH helpfulness increased from 26% to 46% and the misleading suggestions dropped to 60%. This shows that prompt engineering with taskspecific examples can also improve feedback quality. Both strategies consistently achieved high PA scores. Overall, PEFT demonstrated the strongest performance and produced the highest-quality feedback compared to other approaches. Additionally, the evaluation revealed that the PEFT model introduced minimal redundancy and produced the least number of misleading statements. This is an essential consideration in a pedagogical environment, as these elements directly affect student performance. The evaluation also assessed the performance of the PEFT model within the two error categories, imperative and objectoriented. It achieved an accuracy of 49% for imperative errors and 52% for object-oriented errors, suggesting that performance across error categories is relatively consistent.
4.2
Objective Evaluation
To complement the human annotated evaluations, we conducted an automated analysis of feedback using BLEU, ROUGE, and BERTScore. Figure 3 summarize the results. Generally, we see the same pattern as in the manual analysis, with the fine-tuned model performing best on these metrics, followed by in-context learning, and then the baseline model. Although the models demonstrated low BLEU and ROUGE scores, the high BERTScore indicates that they generated meaningful feedback semantically similar to the ground truth. The fine-tuned model achieved particularly high BERTScore values indicating strong semantic alignment. While these metrics provide valuable insights, they do not capture aspects such as clarity, format adherence or the pedagogical soundness of the feedback. Our observations also indicate that high BERTScores do not guarantee pedagogically soundness, cautioning against the use of automated metrics alone to evaluate model performance. For instance, while the baseline model achieved a BERTScore of 0.84, manual checking of the responses reported an accuracy rate of only 20-26%. Although the rate of misleading suggestions generated by the PEFT-model is significantly lower than the other techniques, this remains a concern and underscores the necessity of verifying results before deploying it in an educational setting. We observed that the generated feedback tend to be prescriptive in nature, offering direct instructions. This feedback can be enriched by integrating reasoning elements, scaffolding questions or partial hints to enhance its pedagogical effectiveness. Currently the PEFT model is trained with <Code, KM, KH> triplets. Incorporating task description into the training data is expected to improve the performance.
4.3
Student Evaluation
Based on the positive evaluation of the PEFT model, we carried out a student evaluation, where participants assessed the PEFT model responses alongside those from a proprietary model. The findings indicate that the feedback generated by ChatGPT (C) and the model (F) received positive responses across all three parameters, as shown in Figure 4. Participants unanimously agreed that compiler error messages (E) were ineffective. The thematic analysis of the transcripts identified five key themes: clarity of explanation, level of
0.8
0.6
0.6
(a) KM Feedback
Baseline
KH
4 2 E
C
F
(b) Clarity
4 2 E
-B LE KH
F
Score
0 UG KH E -B ER TS co re
0
U
0.2
C
(a) Usefulness
0.4
0.2
2 E
-R O
0.4
4
Score
0.8
Trovato et al.
Score
1
Score
1
KM -B LE U KM -R OU KM GE -B ER TS co re
Score
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
C
F
(c) Structure
Figure 4: Distributions of student-assessed scores for different feedback types: compiler error message (E), ChatGPT (C), and fine-tuned model (F).
(b) KH Feedback
PE-FewShot
PEFT
Figure 3: Comparison of BLEU, ROUGE, and BERTScore metrics for KM and KH feedback across different model configurations. detail, usefulness, actionability of feedback, and the presence or absence of code. 4.3.1 Theme - Clarity of explanation, Level of detail. Most participants noted that, while Feedback C was helpful, it was often overly verbose, which may be overwhelming for beginners. E.g., "detailed feedback but too many options" (P1), "too much information" (P4), "bit too wordy" (P5), "overexplain the concepts" (P6). However, P4 stated, "I didn’t feel like it was a lot of information. When I’m learning, I feel that I want to know why there’s a problem, and what other options and additional information are available.". Results indicate that learners’ preferred level of feedback detail may affect their learning experience. Another observation was that "Feeback F would be more appropriate for learners with basic knowledge of Java" (P1), "and is best suited for formative assessments in a lab setting, whereas Feedback C might be more appropriate for summative assessment feedback" (P5). 4.3.2 Theme - Usefulness, Actionability of feedback. Aligned with these observed preferences, all participants agreed that actionable feedback promotes engagement and motivates learners to tackle complex challenges. P7 stated, "Feedback C clearly identified the error and explained why the error was there, and how to fix it." and "Feedback F was precise, straight to the point explanation, encouraging student to think of solution, and directs them to the correct path." There was also a suggestion to update Feedback F with additional explanations for technical terms. P4 commented, "Some phrases such as ‘incompatible’ types could be explained more. I felt I could work with it and get the code fixed." 4.3.3 Theme - Presence or absence of code. Participants observed that Feedback C often provides the correct solution, which could lead to over-reliance and limited independent learning. For instance, P3 noted that, although "Feedback C was good, but the correct solution might result in some students skipping the feedback and copying the
code". Participants agreed that early access to the correct code can negatively affect critical thinking. Overall, the findings indicate that participants valued the feedback generated by the fine-tuned model, and its ratings were comparable to those of the proprietary model. According to the participants, Feedback F is well-suited for a lab setting, as it encourages students to engage in critical thinking and problem-solving. Our findings are consistent with those reported in [35].
4.4
Limitations
Our dataset was generated using Deepseek-R1; therefore, the model performance is influenced and limited by the original model. Additionally, the dataset used in this study is limited in size, which may affect the generalizability of the results. Furthermore, this analysis is confined to Java and a specific set of bugs identified in the literature. The student evaluation was based on a small focus group; additional research with larger samples would be useful to verify that the findings generalize.
5
Conclusions
LLMs offer potential in providing guidance to student learners, but the financial constraints of proprietary models and the computational demands of large-parameter architectures pose significant challenges in educational contexts. Open models, which are significantly smaller and freely deployable, present a viable alternative, but are generally less capable than proprietary models. To address this, in our work we have shown how the capabilities of a more capable model can be distilled into an open model by the use of synthetic data generation and parameter-efficient fine-tuning. These fine-tuned models offer significantly better guidance to students, in terms of providing actionable feedback on buggy Java programs, than the baseline model. Notably, the accuracy and helpfulness of the feedback approximately doubles after fine-tuning, and there is also considerable improvement in terms of reducing hallucinations. We also show that fine-tuning is more effective in this domain than prompt engineering, although the latter approach also offers improvement over the baseline model. The presence of misleading information in PEFT-based feedback cannot be ignored, highlighting the need for human evaluation in education settings. As part of future work, we plan to extend the dataset by including more
Fine-Tuning Models for Automated Code Review Feedback
diverse feedback covering various Java programming concepts and to optimise the model performance and effectiveness.
References [1] Marzieh Ahmadzadeh, Dave Elliman, and Colin Higgins. 2005. An analysis of patterns of debugging among novice computer science students. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (Caparica, Portugal) (ITiCSE ’05). Association for Computing Machinery, New York, NY, USA, 84–88. doi:10.1145/1067445.1067472 [2] Zishan Ahmed, Shakib Sadat Shanto, and Akinul Islam Jony. 2024. Potentiality of generative AI tools in higher education: Evaluating ChatGPT’s viability as a teaching assistant for introductory programming courses. STEM Education 4, 3 (2024), 165–182. doi:10.3934/steme.2024011 [3] Amjad Altadmri and Neil C.C. Brown. 2015. 37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (Kansas City, Missouri, USA) (SIGCSE ’15). Association for Computing Machinery, New York, NY, USA, 522–527. doi:10.1145/2676723.2677258 [4] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. arXiv:https://doi.org/10.1191/1478088706qp063oa doi:10.1191/ 1478088706qp063oa [5] Neil C.C. Brown and Amjad Altadmri. 2014. Investigating novice programming mistakes: educator beliefs vs. student data. In Proceedings of the Tenth Annual Conference on International Computing Education Research (Glasgow, Scotland, United Kingdom) (ICER ’14). Association for Computing Machinery, New York, NY, USA, 43–50. doi:10.1145/2632320.2632343 [6] Eason Chen, Ray Huang, Han-Shin Chen, Yuen-Hsien Tseng, and Liang-Yi Li. 2023. GPTutor: a ChatGPT-powered programming tool for code explanation. arXiv:2305.01863 [cs.HC] https://arxiv.org/abs/2305.01863 [7] Paul Denny, Andrew Luxton-Reilly, and Ewan Tempero. 2012. All syntax errors are not equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (Haifa, Israel) (ITiCSE ’12). Association for Computing Machinery, New York, NY, USA, 75–80. doi:10.1145/2325296. 2325318 [8] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314 [cs.LG] https: //arxiv.org/abs/2305.14314 [9] Andrew Ettles, Andrew Luxton-Reilly, and Paul Denny. 2018. Common logic errors made by novice programmers. In Proceedings of the 20th Australasian Computing Education Conference (Brisbane, Queensland, Australia) (ACE ’18). Association for Computing Machinery, New York, NY, USA, 83–89. doi:10.1145/ 3160489.3160493 [10] Ian Finlayson and Stephen Davies. 2024. Jguardrail: A Framework for Identifying Possible Errors in Student Java Code. J. Comput. Sci. Coll. 40, 3 (Oct. 2024), 322–333. [11] Maria Hristova, Ananya Misra, Megan Rutter, and Rebecca Mercuri. 2003. Identifying and correcting Java programming errors for introductory computer science students. In Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education (Reno, Navada, USA) (SIGCSE ’03). Association for Computing Machinery, New York, NY, USA, 153–156. doi:10.1145/611892.611956 [12] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 [cs.CL] https://arxiv.org/abs/2106.09685 [13] J. Jackson, M. Cobb, and C. Carver. 2005. Identifying Top Java Errors for Novice Programmers. In Proceedings Frontiers in Education 35th Annual Conference. T4C– T4C. doi:10.1109/FIE.2005.1611967 [14] Nadja Just, Janet Siegmund, and Belinda Schantong. 2025. From Bugs to Breakthroughs: Novice Errors in CS2. arXiv:2502.14438 [cs.SE] https://arxiv.org/abs/ 2502.14438 [15] Charles Koutcheme. 2022. Towards Open Natural Language Feedback Generation for Novice Programmers using Large Language Models. In Proceedings of the 22nd Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’22). Association for Computing Machinery, New York, NY, USA, Article 29, 2 pages. doi:10.1145/3564721.3565955 [16] Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, and Paul Denny. 2024. Open Source Language Models Can Provide Feedback: Evaluating LLMs’ Ability to Help Students Using GPT-4-As-A-Judge. arXiv:2405.05253 [cs.CL] https://arxiv.org/abs/2405.05253 [17] Charles Koutcheme and Arto Hellas. 2024. Propagating Large Language Models Programming Feedback. In Proceedings of the Eleventh ACM Conference on Learning @ Scale (Atlanta, GA, USA) (L@S ’24). Association for Computing Machinery, New York, NY, USA, 366–370. doi:10.1145/3657604.3664665 [18] Smitha S Kumar, Michael Lones, Manuel Maarek, and Hind Zantout. 2025. Navigating the landscape of automated feedback generation techniques for programming exercises. ACM Trans. Comput. Educ. (Sept. 2025). doi:10.1145/3764593 Just
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Accepted. [19] Chris Langhout and Maurício Aniche. 2021. Atoms of Confusion in Java. In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). 25–35. doi:10.1109/ICPC52881.2021.00012 [20] Juho Leinonen, Paul Denny, Olli Kiljunen, Stephen MacNeil, Sami Sarsa, and Arto Hellas. 2024. LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education. arXiv:2411.10455 [cs.CY] https://arxiv.org/abs/2411.10455 [21] Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013/ [22] Jorge Machado. 2025. Toward a Public and Secure Generative AI: A Comparative Analysis of Open and Closed LLMs. arXiv:2505.10603 [cs.CY] https://arxiv.org/ abs/2505.10603 [23] Davin McCall and Michael Kölling. 2019. A New Look at Novice Programmer Errors. ACM Trans. Comput. Educ. 19, 4, Article 38 (July 2019), 30 pages. doi:10. 1145/3335814 [24] Davin McCall and Michael Kölling. 2014. Meaningful categorisation of novice programmer errors. In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings. 1–8. doi:10.1109/FIE.2014.7044420 [25] Marcus Messer, Neil C. C. Brown, Michael Kölling, and Miaojing Shi. 2024. Automated Grading and Feedback Tools for Programming Education: A Systematic Review. ACM Trans. Comput. Educ. 24, 1, Article 10 (Feb. 2024), 43 pages. doi:10.1145/3636515 [26] Susanne Narciss. 2008. Feedback strategies for interactive learning tasks. In Handbook of research on educational communications and technology. Routledge, 125–143. [27] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (Philadelphia, Pennsylvania) (ACL ’02). Association for Computational Linguistics, USA, 311–318. doi:10.3115/1073083.1073135 [28] Olga Petrovska, Lee Clift, Faron Moller, and Rebecca Pearsall. 2024. Incorporating Generative AI into Software Development Education. In Proceedings of the 8th Conference on Computing Education Practice (Durham, United Kingdom) (CEP ’24). Association for Computing Machinery, New York, NY, USA, 37–40. doi:10. 1145/3633053.3633057 [29] Farman Ali Pirzado, Awais Ahmed, Román Alejandro Mendoza-Urdiales, and Hugo Terashima-Marin. 2024. Navigating the Pitfalls: Analyzing the Behavior of LLMs as a Coding Assistant for Computer Science Students—A Systematic Review of the Literature. IEEE Access 12 (2024), 112605–112625. doi:10.1109/ ACCESS.2024.3443621 [30] Chanathip Pornprasit and Chakkrit Tantithamthavorn. 2024. Fine-tuning and prompt engineering for large language models-based code review automation. Information and Software Technology 175 (2024), 107523. doi:10.1016/j.infsof.2024. 107523 [31] James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew LuxtonReilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent N. Reeves, and Jaromir Savelka. 2023. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE 2023). ACM, 108–159. doi:10.1145/3623762.3633499 [32] Yizhou Qian and James Lehman. 2017. Students’ Misconceptions and Other Difficulties in Introductory Programming: A Literature Review. ACM Trans. Comput. Educ. 18, 1, Article 1 (Oct. 2017), 24 pages. doi:10.1145/3077618 [33] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024. Code Llama: Open Foundation Models for Code. arXiv:2308.12950 [cs.CL] https://arxiv.org/abs/2308.12950 [34] Carlos Alexandre Gouvea da Silva, Felipe Negrelle Ramos, Rafael Veiga de Moraes, and Edson Leonardo dos Santos. 2024. ChatGPT: Challenges and Benefits in Software Programming for Higher Education. Sustainability 16, 3 (2024). doi:10. 3390/su16031245 [35] Lorenzo Lee Solano, Charles Koutcheme, Juho Leinonen, Alexandra Vassar, and Jake Renzella. 2025. Narrowing the Gap: Supervised Fine-Tuning of OpenSource LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools. arXiv:2507.05305 [cs.CY] https://arxiv.org/abs/2507.05305 [36] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [cs.CL] https://arxiv.org/abs/2307.09288 [37] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated Program Repair in the Era of Large Pre-Trained Language Models. In Proceedings of the 45th International Conference on Software Engineering (Melbourne, Victoria, Australia) (ICSE ’23). IEEE Press, 1482–1494. doi:10.1109/ICSE48619.2023.00129
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
[38] Jiawei Xu, Ying Ding, and Yi Bu. 2025. Position: Open and Closed Large Language Models in Healthcare. arXiv:2501.09906 [cs.CY] https://arxiv.org/abs/2501.09906
Trovato et al.
[39] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. arXiv:1904.09675 [cs.CL] https://arxiv.org/abs/1904.09675