PyMETA: A Benchmark Dataset for Hierarchical Student Code Error Classification with Python-Interpreter-Based Labels Chuyue Li* , Ziqi Tang* , Jingyi Wang, Yu Wu, Kazuma Hashimoto, Lingyu Gao CircleCat {cyli, ztang, jasmine, wuyu, kazumah, lygao}@circlecat.org
Abstract
arXiv:2606.30610v1 [cs.SE] 29 Jun 2026
With the advancement of Large Language Models (LLMs), code error detection has extended beyond traditional IDE diagnostics to contextsensitive debugging in educational scenarios. However, existing approaches lack large-scale datasets, multi-error analysis, and unified error taxonomies. To address this, we introduce P Y META, a large-scale Python code error classification dataset of 48,646 student submissions, with single-error labels for all samples and a diagnostic subset of 97 expert-annotated multi-error samples. The dataset uses a threelevel hierarchical taxonomy, from a binary error/no-error split down to 14 fine-grained error types grounded in Python’s official exception hierarchy. We evaluate multi-level classification tasks on two finetuned models and four LLMs with prompting, comparing their classification performance and runtime cost. For multi-error prompting, the best model, Gemini 2.5 Pro, achieves 81.8% macro F1 under the contains criterion. We observe that: 1) prompted LLMs still underperform finetuned smaller models; 2) models exhibit significant disparities across error types; 3) most LLMs over-classify code as Logic Error, with GPT3.5 showing the highest Logic Error Overprediction Rate and Gemini 2.5 Pro the lowest. Our work establishes a data foundation and provides insights for LLM-based code error research.
1
Introduction
As Large Language Models (LLMs) grow more capable, code-related applications span programming education (Shirafuji et al., 2023; Zhao et al., 2025; Amin et al., 2025), IDE/compiler integration (Chan et al., 2024; Lee et al., 2024; Wang et al., 2025b), and engineering (Dubniczky et al., 2025). These settings motivate a range of code-diagnosis tasks, including multi-error classification (Shirafuji et al., * Equal contribution.
2023; Amin et al., 2025), error localization (Han et al., 2023), and error correction (Han et al., 2023). However, this area still faces several challenges. First, public code error datasets lack broad coverage of problem types and error categories, largescale samples, and multi-error analysis (Han et al., 2023; Amin et al., 2025). Second, existing error categorizations are often fragmented or ad hoc, lacking a unified taxonomy bridging human understanding and IDE diagnostics (Shirafuji et al., 2023; Han et al., 2023). Third, standardized benchmarks for evaluating LLMs’ code error detection are scarce, particularly under training-free or lowresource settings (Sun et al., 2025). To address these, this paper makes the following contributions: 1) We introduce P Y META, a largescale Python code error dataset of 48,646 student submissions across 155 problems1 from 579 users. All samples are assigned single-error labels derived from IDE execution signals, while a 97-sample diagnostic subset is further annotated by experts for multiple concurrent errors. This subset enables us to examine whether apparent single-error failures involve additional underlying errors. 2 2) We establish a multi-level benchmark evaluating representative LLMs under both finetuning and prompting, revealing model-specific biases and performance gaps for future LLM-based code diagnosis research. 3) Our analyses reveal: (i) prompted LLMs substantially underperform finetuned smaller models; (ii) models exhibit a strong bias toward over-predicting Logic Error (single-error LERover 17.6–92.8%; multi-error 55.6–75.6%); (iii) multi-error prompting remains challenging even for strong models (best macro F1 81.8% under the contains criterion); and (iv) entropy and confusion analyses expose 1 See Section 5.1 for the finetuning split. A few problems are excluded during pre-processing. 2 The dataset, splits, annotation guidelines, and evaluation scripts are available at https://github.com/Circle-Cat/ pymeta.
model uncertainty in complex multi-error cases. We evaluate two finetuned models (CodeBERT, CodeLlama-7B) and four prompted LLMs (DeepSeek-V3, Gemini 2.5 Pro, GPT-3.5, GPT-4o). Our main finding is that finetuning a small model still works better than prompting much larger ones on this task: CodeLlama-7B reaches 80.6% macro F1 on single-error classification, while the best prompted model, Gemini 2.5 Pro, reaches 71.9%. Prompted models also share a common weakness: they over-predict Logic Error and do poorly on rare error types, showing where future work on LLM-based code diagnosis is most needed. We also report inference cost, which differs by more than ten times across models.
2
Related Work
2.1
Dataset for Code Error Classification
Code error detection, classification, localization, and analysis have long been studied in programming education, and the growing use of data-driven and LLM-based methods has increased the demand for high-quality large-scale datasets (Lan et al., 2026). Some studies (Lee et al., 2024; Amin et al., 2024, 2025) rely on existing open-source datasets like the AOJ ITP1 dataset.3 Others (Han et al., 2023; Xiang et al., 2024) introduce course based C language datasets such as COJ2022 and CPE28. Taking AOJ ITP1 as an example, it is based on 44 problems and thus does not capture diverse problems and error cases. These course-based datasets often exhibit limited problem diversity, homogeneous user populations, and small sample sizes, and typically lack fine-grained multi-error annotations. To tackle these limitations and to contribute an open dataset for code error classification, we introduce P Y META: a Python code error classification dataset. 2.2
Code Error Detection with Large Models
Recent studies aim to improve code error classification by training strategies and model architectures, including combining a CodeT5 encoder with MLKNN (Amin et al., 2024), finetuning two BERT variants for multi-label classification (Amin et al., 2025), and introducing a model that integrates BiLSTM and TextCNN (Xiang et al., 2024). Recent work also suggests that Code LLMs struggle to https://onlinejudge.u-aizu.ac.jp/courses/ lesson/2/ITP1 (accessed on 2025-12-30). 3
feature User ID Name Question ID Question Answer Attempt ID StudentAnswer TestOutcome AttemptStepID
definition unique identifier of the student problem type unique identifier of the problem provides the problem description sample of the correct code solution attempt number contains the student’s submitted code the Online Judge error messages outputs denotes the step identifier for the attempt
Table 1: Dataset Feature Descriptions. Column names and definitions for the 9 features included in each PYMETA sample.
reason about runtime behavior (Wang et al., 2025a; Gehring et al., 2025). However, we hypothesize that current LLMs may already possess inherent code error classification capabilities, yet systematic analyses remain limited. We thus compare common LLMs on single- and multi-error classification using basic finetuning and training-free approaches, analyzing potential biases. All P Y META labels derive from OJ execution output, including Logic Error for code that executes but fails test cases. This provides an execution-grounded standard, different from current models frequently misalign (Appendix A.11.4).4
3
Dataset Construction
In this section, we will introduce our proposed dataset. The data sources and data processing are detailed in Appendix A.4 and A.5. 3.1
Platform and Data Source
3.2
Dataset Overview
We constructed a Python code error dataset P Y META from student submission logs collected through the Circle Cat online learning platform (see Appendix A.1). The dataset spans 22 problem types across 155 distinct problems, with 48,646 real code submissions from 579 users and an expertannotated multi-error diagnostic subset of 97 samples. Each sample contains a problem description, the student’s Python code submission, its compilation and execution results from an online judge platform, Moodle CodeRunner,5 and the correct 4 Two patterns emerge: predicting Logic Error for parsetime failures, and Syntax Error for executable code. 5 CodeRunner: a Moodle question type for programming courses. http://coderunner.org.nz (accessed on 2025-1230)
Figure 1: Error Type Distribution in the Multi-Error Diagnostic Subset. Frequency of each error type across the 97 expert-annotated samples.
Task A
Error
Task B
Task C No Error Logic Error Syntax Error Name Error Type Error Indentation Error Unbound Local Error Key Error Explicit Error Index Error EOF Error Recursion Error Value Error Tab Error Other Errors
Count 23,207 11,387 5,618 2,565 2,074 1,355 482 417 400 277 334 190 162 178
Table 2: PyMETA Dataset Statistics. Sample counts for each error type under the three-level hierarchical taxonomy (Task A: binary, Task B: three-class, Task C: multi-class). Counts reflect the full 48,646-sample dataset.
code solution. Each sample is described by 9 features (Table 1), and submissions are categorized into 18 error types. The distributions of problem IDs and user IDs are shown in Figure 6 and Figure 7, respectively. In particular, the distribution of problem names and IDs is shown in Figure 6, and the distribution of user IDs is presented in Figure 7. In addition, we provide an expert-annotated subset in which each code sample is labeled with all of its error types, rather than just the first one. This supports research on tasks that require multi-error information, such as diagnosing whether a singleerror misclassification is caused by multiple cooccurring errors. The subset contains 97 samples covering 13 error types, with an average of 1.91 error types per sample (Figure 1).
4
Taxonomy Design
4.1
Hierarchical Error Taxonomy
Existing taxonomies follow two patterns, and each has its gap. First, diff-based schemes (Han et al., 2023; Shirafuji et al., 2023) compare student code to a reference solution, but often mark style or efficiency differences as errors even when the code runs correctly. Second, execution-outcome labels only check whether the code runs, and ignore the step-by-step analysis that teaching requires (Appendix A.3). Our taxonomy addresses both gaps: it is grounded in Python’s official exception hierarchy, so it matches standard IDE definitions and also supports the step-by-step analysis that education needs. Our taxonomy consists of three hierarchical task levels: Task A Binary, Task B Three-Class, and Task C Multi-class Classification. It is designed around two core distinctions not jointly addressed by prior work. First, it draws a clear boundary between Explicit Error (interpreter- or runtimedetectable failures) and Logic Error (semantically incorrect yet executable code), reflecting different diagnostic requirements in educational and IDE settings. Second, it incorporates expert-annotated multi-error labels on a diagnostic subset, enabling analysis of coexisting error types. It is critical for pedagogical feedback and IDE integration, yet largely absent from existing resources. Single-error gold labels align with Moodle CodeRunner outputs for IDE realism, and the progressive granularity of our three-level taxonomy supports systematic LLM evaluation. The multi-error diagnostic subset further bridges process-oriented educational analysis with result-oriented performance evaluation. Task A: Binary Classification The coarsest level of our taxonomy is the binary classification, which categorizes code submissions as either “No Error” or “Error.” “No Error” corresponds to a code that is executed without any exceptions, passing all test cases; “Error” means that a code execution is not completed or fails to pass at least one test. Task B: Three-Class Classification The second level divides code submissions into three categories: “No Error,” “Explicit Error,” and “Logic Error.” “Explicit Error” corresponds to a code whose execution is not completed, while “Logic Error” refers to a code that runs but fails to pass at least one test case.
Task C: Multi-class Classification The third level, multi-class classification, represents the finest-grained categorization of error types. In P Y META, we follow standard Python exceptions6 and define 18 error types. After merging 7 lowfrequency types, we conduct experiments on 14 classes in total, including No Error.
5
Experiment setup
Based on P Y META, we evaluate LLMs on singleerror classification with finetuning and prompting, as well as multi-error prediction under prompting. 5.1
Single-Error Finetuning Classification
In this experiment, we conduct single-error finetuning classification experiments on two pretrained models, CodeBERT (Feng et al., 2020) and CodeLlama-7B (Rozière et al., 2024), using the three hierarchical tasks described in Section 4.1. For single-error finetuning, we split P Y META at the level of QuestionID: rather than splitting individual submissions, we assign each coding problem (together with all its submissions) entirely to one of the train, dev, or test sets. This yields 134 problems / 38,919 samples for the training set, 12 problems / 4,864 for the dev set, and 9 problems / 4,865 for the test set, with no QuestionID shared across splits. This problem-level split prevents models from exploiting the same exercise statement or a fixed per-problem error template seen during training, which a submission-level random split would not guard against. Evaluation metrics include accuracy, macro/weighted F1, precision and recall. The finetuning results and in-depth analysis are presented in Section 6. 5.2
Multi-Error Diagnostic Subset Construction
We build a diagnostic subset of expert-annotated multi-error samples to test whether single-error misclassifications are caused by multiple errors in the code. Single-error gold labels record only the first error during execution, while models see the full code and may flag several errors at once, so comparing the two tells us whether a misclassification reflects genuine multi-error code or a model limitation. Rather than sampling at random, which would mostly yield clean single-error cases, we 6 https://docs.python.org/3/library/exceptions. html and https://docs.python.org/3/tutorial/ errors.html (accessed on 2025-11-06).
target likely multi-error samples using two signals: model misclassification (confusion-matrix-based) and model uncertainty (entropy-based), described below. Sampling strategies. We select candidate multierror samples from the CodeBERT Task C singleerror predictions, using two complementary strategies: (1) Confusion-Matrix-Based Extraction. We randomly sample 40 misclassified instances from the off-diagonal cells of the confusion matrix, excluding cells whose gold label is No Error or Logic Error (which implicitly exclude multi-error cases). Misclassifications here may stem from samples containing multiple error types. (2) Entropy-Based Extraction. We extract the top 60 samples with the highest prediction entropy, under the hypothesis that high entropy reflects model uncertainty caused by co-occurring errors. The two splits share 3 overlapping samples, yielding 97 unique samples. Each is annotated by 15 expert annotators over 3 rounds of mutual verification. The resulting subset covers 13 error types, with an average of 1.91 error types per sample (Figure 1). Evaluation metrics and results based on this subset are reported in Section 6.3. 5.3
Single-Error Prompting Classification
We evaluate several current LLMs, including Gemini 2.5-Pro (Pichai and Hassabis, 2023), DeepSeekV3 (DeepSeek-AI et al., 2024), GPT-3.5 (OpenAI, 2023), and GPT-4o (OpenAI, 2024) on the task of single-error prompting classification using a sampled set of 4,865 student code submissions. Following the taxonomy in Section 4.1, models are assessed by per–error-type performance and their characteristic misclassification patterns. To ensure stable comparison across error types with uneven sample sizes, we apply a row-normalized weighting strategy that equalizes class contribution and mitigates noise from low-frequency categories. This strategy aims at reporting comparative trends rather than procedural details. 5.4
Multi-Error Prompting Classification
Applying the same models from Section 5.3, we transition from single-error detection to a multierror task, requiring models to identify all concurrent errors within a single submission. We employ a Chain-of-Thought prompting strategy (see Appendix A.8) that uses an internal diagnostic trace,
Class No Error Error Macro avg Weighted avg
Precision (%) 64.2 95.4 79.8 85.7
Recall (%) 91.7 77.0 84.4 81.6
Class No Error Explicit Error Logic Error Macro avg Weighted avg
F1-score (%) 75.5 85.2 80.4 82.2
Table 3: CodeBERT Task A (Binary Classification) results on the test set under problem-level split by QuestionID. Following tables follows this QuestionID split.
Class 0 No Error 1 Logic Error 2 Syntax Error 3 Name Error 4 Type Error 5 Indentation Error 6 Unbound Local Error 7 Key Error 8 Index Error 9 EOF Error 10 Recursion Error 11 Value Error 12 Tab Error 13 Other Errors Macro avg Weighted avg
Results and Analysis
In this section, we present the results and data analysis of the experiments described above and offer actionable insights, including the main benchmark results (Section 6.1), single-error model behavior diagnostic analysis (Section 6.2), and multi-error diagnostic evaluation (Section 6.3). 6.1
Main Benchmark Results
We begin by presenting the main benchmark results under both finetuning and prompting settings. 6.1.1
Single-Error Finetuning Classification
We finetune CodeBERT and CodeLlama-7B on the three hierarchical tasks (Section 4.1) and report test-set metrics under the QuestionID split (Section 5.1) According to Table 3 to 8, we observe that foundation models such as CodeBERT and CodeLlama7B have achieved good performance on the current Python multi-level error classification finetuning tasks. Specifically, from Task A to Task B to Task C, as the classification granularity becomes finer and the task difficulty increases, the performance of both models shows a slight decline. When comparing the two models, CodeLlama7B consistently outperforms CodeBERT across all three tasks. As shown in Table 8, CodeLlama achieves the best performance on Task C, with 80.6% macro-average and 91.6% weighted-average F1. These strong finetuning results motivate us to examine whether prompting-based large language models can achieve comparable performance on the same tasks.
Recall (%) 91.1 51.7 78.4 73.7 73.9
F1-score (%) 78.0 64.2 76.5 72.9 73.1
Table 4: CodeBERT Task B (Three-class Classification) results on the test set.
evaluating syntax, scope, and logic, before final label output. We quantify classification performance using standard Precision, Recall, F1, and Exact Match metrics. The full 14-category taxonomy and prompt templates are detailed in Appendix A.8.
6
Precision (%) 68.2 84.7 74.6 75.8 75.8
Precision (%) 67 79 92 72 41 89 47 77 30 75 0 68 56 0 57.0 72.9
Recall (%) 95 79 71 46 19 25 18 59 21 40 0 29 77 0 41.0 73.4
F1-score (%) 78 79 80 56 26 39 26 67 25 52 0 41 65 0 45.2 71.0
Table 5: CodeBERT Task C (Fine-grained Multiclass Classification) results on the test set. Class indices follow the taxonomy in Section 4.1 and match the row/column order of the confusion matrices and the label IDs used in the prompts (Appendix A.8). Perclass precision/recall/F1 at integer precision; Macro and Weighted averages at one decimal place, same for Table 8.
6.1.2
Single-Error Prompting Classification
Building on the strong performance of finetuned small models, we evaluate whether larger LLMs can match this level of performance under the prompting paradigm. Results in Table 9 show that, except for Gemini 2.5 Pro (71.9 macro F1), all large models perform poorly. Despite competitive finetuning performance, prompting-based LLMs still have considerable room for improvement on this task. Performance analysis can be found in Table 9 and Table 20. We further note a consistent tendency across models to over-predict Logic Error, which is quantitatively characterized in Section 6.2.1; detailed per-model prompting capability is provided in Appendix A.11. 6.2
Single-Error Model Behavior Diagnostic Analysis
We now conduct diagnostic analyses to better understand model behavior in the single-error prompting experiments. 6.2.1
Error-Type Bias Analysis
We first investigate systematic error-type biases exhibited by the models.
Class No Error Error Macro avg Weighted avg
Precision (%) 92.1 99.2 95.7 97.0
Recall (%) 98.2 96.2 97.2 96.8
F1-score (%) 95.0 97.7 96.3 96.8
Table 6: CodeLlama Task A (Binary Classification) results on the test set. Class No Error Explicit Error Logic Error Macro avg Weighted avg
Precision (%) 92.5 92.2 95.3 93.3 93.5
Recall (%) 97.8 93.9 89.3 93.7 93.4
F1-score (%) 95.1 93.1 92.2 93.5 93.4
Table 7: CodeLlama Task B (Three-class Classification) results on the test set.
Across all evaluated models in Table 11, predictions show a recurring tendency to classify nonLogic-Error samples to the Logic Error category. We define the Logic Error Overprediction Rate (LERover ) as the proportion of instances whose ground-truth label is not Logic Error but are predicted as Logic Error. Formally, for a given model, LERover is computed as LERover =
1
X
|C¬LE |
P (ŷ = LE | y = c),
c∈C¬LE
(1) where y denotes the ground-truth error label, ŷ denotes the model-predicted error label, LE refers to the Logic Error class, C¬LE denotes the set of all error classes excluding Logic Error, and c indexes a specific non-Logic Error class. The conditional probability P (ŷ = LE | y = c) is estimated from the row-normalized confusion matrix as the proportion of instances with ground-truth label c that are predicted as Logic Error. GPT-3.5 exhibits the most severe Logic Error bias, with the highest LERover of 92.8%. The rownormalized (each row sums to 100%) confusion Class 0 No Error 1 Logic Error 2 Syntax Error 3 Name Error 4 Type Error 5 Indentation Error 6 Unbound Local Error 7 Key Error 8 Index Error 9 EOF Error 10 Recursion Error 11 Value Error 12 Tab Error 13 Other Errors Macro avg Weighted avg
Precision (%) 90 96 95 81 85 97 67 88 45 96 100 88 85 82 85.0 91.9
Recall (%) 99 89 97 93 82 91 72 87 36 96 60 64 85 40 78.0 91.7
F1-score (%) 94 93 96 86 84 94 69 88 40 96 75 74 85 54 80.6 91.6
Table 8: CodeLlama Task C (Fine-grained Multiclass Classification) results on the test set.
Model GPT-3.5 GPT-4o Gemini DeepSeek-V3
Accuracy (%) 40.3 71.5 85.9 73.6
Macro Precision (%) 14.2 36.2 84.4 45.5
Macro Recall (%) 10.5 26.4 69.1 27.5
Macro F1 (%) 8.8 28.0 71.9 29.1
Table 9: Single-Error Prompting Classification overall results on test set. Note. “Gemini” is used as a shorthand for Gemini 2.5 Pro throughout the tables and figures.
Error Type No Error Logic Error Syntax Error Name Error Type Error Indentation Error Unbound Local Error Key Error Index Error EOF Error Recursion Error Value Error Tab Error Other Errors
GPT-3.5 34.7 92.3 16.1 3.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Accuracy (%) GPT-4o Gemini DeepSeek-V3 85.6 83.2 81.9 76.2 89.6 85.2 78.0 96.4 85.9 36.2 96.3 73.5 18.3 91.0 26.2 36.8 84.1 46.8 0.0 60.9 6.1 11.1 92.3 20.5 4.2 62.5 30.3 0.0 4.3 0.0 0.0 71.4 4.8 0.0 71.4 0.0 0.0 35.3 0.0 0.0 0.0 0.0
Table 10: Single-Error Prompting Classification accuracy on test set
matrix of GPT-3.5 is presented in Figure 2. Gemini 2.5 Pro achieves the best performance on LERover , with only 17.6% LERover , with row-normalized confusion matrix presented in Figure 2. Nevertheless, non-zero Logic Error overprediction is observed for all evaluated models (see Figures 14 and 15, and Table 11). A detailed qualitative analysis of representative misclassification cases, in which instances with non-Logic Error ground-truth labels are predicted as Logic Error, is provided in Appendix A.11.1. 6.3
Multi-error Diagnostic Evaluation
To diagnose whether single-error classification failures stem from the presence of multiple concurrent errors, we construct and analyze a diagnostic subset of 97 expert-annotated multi-error samples. 6.3.1
Multi-Error Subset Evaluation Metric
In this section, we introduce the multi-error evaluation metric Coverage Rate used in subsequent experiments. Our gold labels record only the first sequential error and do not capture concurrent errors in a submission. To evaluate model behavior in multi-error settings, we conduct a human-annotation analysis using two diagnostic subset sampling strategies derived from single-error classification results: 1). Confusion-Matrix-Based sampling, 2). Entropy-
Figure 2: Row-Normalized Confusion Matrices for GPT3.5 and Gemini 2.5 Pro. Each row is normalized by the number of ground-truth instances of that class. Model GPT-3.5 GPT-4o Gemini DeepSeek-V3
Model GPT-3.5 GPT-4o Gemini DeepSeek-V3
LERover 92.8 57.2 17.6 51.5
Table 11: Logic Error Overprediction Rate (LERover , %) on the single-error prompting test set. LERover is the average proportion of non-Logic Error instances predicted as Logic Error, computed from row-normalized confusion matrices on the test set (4,865 samples).
Based sampling, using the dataset extracted from Sec. 5.2. More details on dataset construction, annotator background can be found in Sec. 5.2 and Appendix A.5. Let Gi denote the set of error labels annotated by human experts for sample i, Pi denote the first sequential error label predicted by the model (treated as a singleton set), and N denote the total number of samples. The Coverage indicator and Coverage Rate are defined as: ( 1, if Pi ∩ Gi ̸= ∅ Coveragei = (2) 0, otherwise N
1 X Coverage Rate = Coveragei N
(3)
i=1
A higher Coverage Rate score indicates better model performance. 6.3.2
Figure 3: Multi-Error Coverage Rate by predicted error type of confusion-matrix-based subset.
Multi-Error Prompting Classification Results Applying the diagnostic subset, we first report the coverage of model predictions over the humanannotated multi-error labels. We further evaluate models on a more challenging Multi-Error Prompting task, in which a predefined system prompt (see Section A.8) instructs the model to identify all concurrent error types present in a single code submission, rather than predicting only a single dominant error.
Precision (%) 71.4 71.4 85.7 71.4
Recall (%) 43.2 57.1 78.7 53.3
F1 (%) 51.5 62.7 81.8 59.1
Table 12: Multi-Error Prompting Classification Results on the test set. Model GPT-3.5 GPT-4o Gemini DeepSeek-V3
LERover (%) 34/45 = 75.6 25/45 = 55.6 25/45 = 55.6 34/45 = 75.6
95% Bootstrap CI [62.2%, 86.7%] [40.0%, 68.9%] [40.0%, 68.9%] [62.2%, 86.7%]
Difference vs. DeepSeek-V3 0.0 pp -20.0 pp -20.0 pp –
Table 13: Multi-error LERover with 95% bootstrap CIs. Rates are computed over the 45 samples without any Logic Error in the multi-error diagnostic subset (Section 6.3); CIs use M =1000 resamples.
Results for the multi-label setting are reported in Table 12. Consistent with our single-error findings, Gemini 2.5 Pro achieves the strongest overall performance with an F1 of 81.8%. We observe that models sometimes predict Logic Error for samples that do not contain a ground-truth logic error. An illustrative example is provided in Listing 5, where the model assigns Logic Error to non-standard or suboptimal but executable code, even though no execution-level error is present. Under our annotation protocol, such predictions are treated as overpredictions. Logic Error overprediction Analysis. We analyze model Overprediction behavior under multierror prompting, defining overpredictions as predictions of Logic Error absent from the humanannotated ground truth. Results on 97 expertannotated samples are summarized in Table 13. DeepSeek-V3 shows an approximately 20 percentage point higher LERover than Gemini 2.5 Pro and GPT-4o, a trend consistent across resampling runs despite partially overlapping bootstrap confidence intervals. However, DeepSeek-V3 and
Figure 4: Multi-Error Coverage Rate by predicted error type on the entropy-based 60-sample subset. Same metric as Figure 3.
GPT-3.5 exhibit comparable LERover , while differ sharply in the types of overpredictions they produce. We distinguish top-1 overpredictions, where the overpredicted label appears first in the model’s sequential output, from non-top-1 overpredictions. Under this definition, DeepSeek-V3 exceeds GPT3.5, Gemini 2.5 Pro, and GPT-4o in non-top-1 logic-related overpredictions by ≈ 32, 2, and 14 percentage points, respectively. Taken together, these results indicate that the evaluated models differ not only in how frequently they overpredict, but also in the structural form that overpredictions take. 6.3.3
Confusion Patterns
Beyond aggregate coverage rates, we further examine per-error-type confusion patterns on the diagnostic subset. Confusion-Matrix-Based Multi-Error Classification Analysis. For Confusion-Matrix-Based multi-error samples, only 50% of the model predictions fell within the label space defined by human expert annotations. Performance varies substantially by error type: Name Error achieves full coverage (100%), Logic Error reaches 71%, while rare classes are never captured. This indicates that while some categories (e.g., Name Error and Logic Error) align well with human judgments, others remain under-represented, reflecting model bias and uncertainty in multi-error contexts. Similar patterns are also observed in the entropy-based analysis presented below. Entropy-Based Multi-Error Classification Analysis. We distinguish between matched and unmatched multi-error predictions, where a prediction is matched if the model’s first predicted error label appears in the human-annotated gold set. For entropy-based multi-error cases, 52% of pre-
Figure 5: Prediction entropy for matched vs. unmatched predictions on the multi-error diagnostic subset. Entropy from CodeBERT Task C single-error finetuning; a prediction is matched if the top-1 label is in the humanannotated set.
dictions are matched. As shown in Figure 4, Name Error and Logic Error achieve full coverage, while several other error types are never captured, consistent with our confusion-matrix-based multierror analysis. Figure 5 shows that although both matched and unmatched samples appear at low entropy levels, unmatched samples dominate the high-entropy region. Matched predictions exhibit substantially lower average entropy (0.25) than unmatched predictions (0.84), indicating that higher predictive entropy is associated with increased uncertainty and a greater likelihood of falling outside the human-annotated label set.
7
Conclusion
In this work, we introduce P Y META, a large-scale Python code error dataset with high user diversity, rich problem types, and a fine-grained multi-error diagnostic subset. Based on this dataset, we design a multi-level classification taxonomy and associated tasks. CodeLlama-7B and Gemini 2.5 Pro achieve the best performance under their respective settings among our evaluated models. We observe that prompting-based LLMs underperform finetuned smaller models. We further uncover significant performance disparities across error types and a systematic bias toward over-predicting Logic Error, indicating the limitations of current models in balanced and fine-grained error detection. Our work provides a robust data foundation and baselines for future research on code error understanding, programming education, and LLM-based IDE systems, and opens up new directions for studying multi-error reasoning in code models. Future work includes extending the dataset to additional programming languages, such as Java
and C++, broadening its coverage of natural languages, and enriching the annotations with more fine-grained metadata, such as difficulty levels and grading criteria. We further plan to design a finergrained taxonomy of logic-error subtypes, building on the reasoning traces produced by our prompting experiments, to enable more detailed exploration of semantic error patterns in student code.
Aaron Chan, Michele Tufano, Jinu Jang, Neel Sundaresan, Anisha Agarwal, Roshanak Zilouchian Moghaddam, Shubham Chandel, Yevhen Mohylevskyy, and Shaun Miller. 2024. Copilot evaluation harness: Evaluating llm-guided software programming. Preprint, arXiv:2402.14261. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, and ... 2024. Deepseek-v3 technical report. Technical Report arXiv:2412.19437, arXiv.
Limitations Our proposed dataset P Y META has the following limitations: (1) Our dataset focuses exclusively on Python code errors and does not cover other programming language scenarios such as Java, C, and C++; we note that this single-language focus is shared by most existing code error datasets (see Appendix A.7), though broader language coverage remains an important direction for future work. (2) Our current multi-error annotations cover only a subset of the dataset; we have not performed complete expert annotation of multi-errors across the entire 48k-instance dataset. This limitation is inherent to the data: multi-error occurrences are not prevalent across all student submissions, and our targeted sampling strategies, confusionmatrix-based and entropy-based extraction, efficiently identify the most likely multi-error candidates without exhaustive full-dataset annotation. (3) In line with our privacy protection policy, the dataset does not collect or record student personal information or background demographics; as a result, it is not possible to analyze or report performance differences across demographic groups (see Appendix A.5).
Acknowledgments We are grateful to the volunteer expert annotators whose careful work made the multi-error diagnostic subset possible. We also thank the anonymous reviewers for their constructive feedback.
References Md. Faizul Ibne Amin, Atsushi Shirafuji, Md. Mostafizer Rahman, and Yutaka Watanobe. 2024. Multi-label code error classification using codet5 and ml-knn. IEEE Access, 12:100805–100820. Md Faizul Ibne Amin, Yutaka Watanobe, Md Mostafizer Rahman, and Atsushi Shirafuji. 2025. Source code error understanding using bert for multi-label classification. IEEE Access, 13:3802–3822.
Richard A. Dubniczky, Krisztofer Zoltán Horvát, Tamás Bisztray, Mohamed Amine Ferrag, Lucas C. Cordeiro, and Norbert Tihanyi. 2025. Castle: Benchmarking dataset for static code analyzers and llms towards cwe detection. Preprint, arXiv:2503.09433. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Codebert: A pre-trained model for programming and natural languages. Preprint, arXiv:2002.08155. Jonas Gehring, Kunhao Zheng, Gabriel Synnaeve, and 1 others. 2025. RLEF: Grounding code LLMs in execution feedback with reinforcement learning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of PMLR, pages 19034–19055. Siqi Han, Yu Wang, and Xuesong Lu. 2023. Errorclr: Semantic error classification, localization and repair for introductory programming assignments. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 1345–1354, New York, NY, USA. Association for Computing Machinery. Yunshi Lan, Xinyuan Li, Hanyue Du, Xuesong Lu, Ming Gao, Weining Qian, and Aoying Zhou. 2026. Survey of natural language processing for education: Taxonomy, systematic review, and future trends. IEEE Transactions on Knowledge and Data Engineering, 38(1):659–678. Yanggyu Lee, Suchae Jeong, and Jihie Kim. 2024. Improving llm classification of logical errors by integrating error relationship into prompts. Preprint, arXiv:2404.19336. OpenAI. 2023. Introducing GPT-3.5 turbo and whisper apis. https://openai.com/index/ introducing-gpt-35-turbo-and-whisper-apis/. Accessed: 2024-07-23. OpenAI. 2024. GPT-4o system card. arXiv preprint arXiv:2410.21276. Sundar Pichai and Demis Hassabis. 2023. Introducing gemini: Our largest and most capable AI model. https://blog.google/technology/ ai/google-gemini-ai/. Google Blog. Accessed: 2024-07-23.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, and 7 others. 2024. Code llama: Open foundation models for code. Preprint, arXiv:2308.12950. Atsushi Shirafuji, Taku Matsumoto, Md Faizul Ibne Amin, and Yutaka Watanobe. 2023. Rule-based error classification for analyzing differences in frequent errors. In 2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), page 1–7. IEEE. Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, Qipeng Guo, Xipeng Qiu, Pengcheng Yin, Xiaoli Li, Fei Yuan, Lingpeng Kong, Xiang Li, and Zhiyong Wu. 2025. A survey of neural code intelligence: Paradigms, advances and beyond. Preprint, arXiv:2403.14734. Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, and Yi Li. 2025a. Do code semantics help? A comprehensive study on execution trace-based information for code large language models. In Findings of EMNLP. Yuchen Wang, Shangxin Guo, and Chee Wei Tan. 2025b. From code generation to software testing: AI copilot with context-based retrieval-augmented generation. IEEE Software, 42(4):34–42. Chengguan Xiang, Maoqiu Yu, and Peipei Zhi. 2024. Programming error classification method for novices based on bilstm-textcnn model. In 2024 14th International Conference on Information Technology in Medicine and Education (ITME), pages 1055–1059. Yanni Zhao, Xufeng Ling, Huaizhong Zhu, Feng Zhou, and Zhiruo Deng. 2025. Exploration of computer programming teaching reform based on large language models. In 2025 11th International Conference on Computing and Artificial Intelligence (ICCAI), pages 359–363.
A
Appendix
A.1
Introduction to Circle Cat
The student submissions in P Y M ETA were collected from the online learning platform operated by Circle Cat Inc., a 501(c)(3) non-profit organization that provides free, accessible technical education and career-development guidance to Chinesespeaking women. Circle Cat’s programs span three areas: structured software-engineering coursework that takes students from foundational concepts to advanced skills; a residency program in which students work on real-world open-source, non-profit, and industry projects under one-on-one mentorship; and career support from mentors with industry experience. The coursework is delivered through a selfhosted Moodle instance with an integrated Online Judge, where learners submit Python solutions that are automatically compiled, executed, and graded with immediate feedback. The historical submission logs from this platform form the raw source of our dataset. Because these submissions are produced by learners of varying proficiency during ordinary coursework, they provide a realistic and diverse distribution of student code errors. A.2
Task C Multi-class Classification
Logic Error: Code that compiles successfully but fails one or more test cases due to incorrect logic or implementation. Syntax Error: Raised when the parser encounters a syntax error in the code. Name Error: Raised when a local or global name is not found. Type Error: Raised when an operation or function is applied to an object of inappropriate type. Indentation Error: Raised when there is incorrect indentation in the code. Unbound Local Error: Raised when a local variable is referenced before it has been assigned. Key Error: Raised when a dictionary key is not found. Index Error: Raised when a sequence subscript is out of range.
EOF Error: Raised when the input() function hits an end-of-file condition (EOF) without reading any data. Recursion Error: Raised when the maximum recursion depth is exceeded. Value Error: Raised when a function receives an argument of the correct type but an inappropriate value. Tab Error: Raised when indentation consists of inconsistent use of tabs and spaces. Attribute Error: Raised when an attribute reference or assignment fails. Runtime Error: Raised when an error is detected that does not fall in any of the other categories. Syntax Warning: Raised for dubious syntactic features but not necessarily invalid syntax. Zero Division Error: Raised when division or modulo by zero takes place for all numeric types. Memory Error: Raised when an operation runs out of memory. Module Not Found Error: Raised when a module could not be found. No Error: Code that compiles and passes all test cases successfully. A.3
Motivation for Our Taxonomy Design
This section explains the considerations that motivated the design of our taxonomy. At present, there is no consensus on taxonomies for single-label or multi-label code error classification. For example, the AOJ dataset aligns error categories with online judge outcomes, such as Runtime Error and Wrong Answer. COJ2022 (Han et al., 2023) computes line level diffs between student code submissions with potential errors and their corrected versions using difflib, and labels errors such as Function, Declaration, etc. The taxonomy in Dubniczky et al. (2025) includes 25 types of CWE vulnerabilities, such as buffer overflow, SQL injection, cross-site scripting, and others. Other work (Shirafuji et al., 2023) applies AST based analysis and reference solutions to the AOJ dataset for line level error judgments, yielding 21 coarse grained classes and 55 fine grained categories, including output, input, variable conversion, etc.
Current error taxonomies in code mostly focus on algorithmic optimization or are framed purely from a human-centered perspective. Many linewise mismatched code categories are essentially code optimizations rather than actual errors. Such categories would potentially cause confusion for both human learners and automated systems, and they may not always be helpful for detecting real errors. Meanwhile, we observe an increasing number of studies that integrate LLMs into IDEs to assist with code generation and error analysis. Some work examines the debugging challenges of LLMpowered IDE tools in code understanding, generation, and automated repair, and presents an automated testing system that enables error detection in lockstep with codebase updates (Lee et al., 2024; Wang et al., 2025b). Others (Chan et al., 2024) explore Copilot’s iterative error-checking mechanisms following code edits. These developments highlight the need to design an error taxonomy that targets real errors and is equally interpretable by both IDEs and human developers. In response, we propose a three-level, processbased code error taxonomy that uses Python’s official error types as the foundational layer to locate genuine error categories, serving both human coding education and IDE functionalities. A.4
submissions were cleaned and post-processed before expert annotation to construct a fine-grained multi-error code error dataset based on real learnerwritten code. Because the data were obtained from pre-existing educational records and involve no human subject intervention, ethics review board approval was not required under institutional guidelines. Details regarding data anonymization, privacy considerations, and annotation procedures are provided in Section A.5.
Figure 6: Question ID Distribution
Source Data and Platform
We describe the sources, collection process, and ethical considerations of the raw data used in this section. Our data originate from an existing online programming education platform developed and maintained by our engineering team. The platform provides Python and Java programming exercises, allowing learners to submit code solutions that are automatically compiled, executed, and evaluated through an online judge system, with immediate feedback returned to learners. The dataset was collected from historical platform logs and does not involve direct recruitment, paid participation, or experimental intervention. All code submissions were generated organically during normal platform usage. Data collection and usage follow the platform’s terms of use, which inform users that anonymized data may be used for educational and research purposes. From the code runner backend, we extracted 48,646 complete and valid code submission samples from 155 distinct questions and 579 users with different background as the initial dataset. The Question and User distribution are shown in Figure 6 and Figure 7. These code
Figure 7: User Distribution
A.5
Dataset Preprocess and Annotation
The dataset used in this study is derived from the Moodle-based educational platform developed by our team, which provides anonymized user interaction logs (e.g., attempt IDs, attempt step IDs) for research purposes. All identifiers have been irreversibly anonymized prior to analysis, ensuring that no personally identifiable information (PII) is present in the dataset. Therefore, the dataset poses no privacy risks and does not contain offensive content. Annotation for the code error taxonomy was performed by a team of 15 volunteer researchers familiar with Python programming and educational assessment. The 15 annotators consisted of soft-
ware engineers and researchers with backgrounds in computer science and educational assessment. All expert annotators had 3–5 years of engineering experience in Python language and were familiar with common student programming errors at the introductory level. The annotation guidelines and examples are provided below. A.5.1 Annotation Guidelines Annotators were provided with a detailed error taxonomy comprising 18 error types plus a “no error” category (ID 0). Each error type is defined by its associated runtime or compile-time behavior: Logic Error (ID 1) applies when code executes without explicit exceptions but fails one or more test cases; all other error types (IDs 2–18) are identified by matching the execution output against a specific error string (e.g., Syntax Error, Name Error, Type Error). For each student code sample identified by a (Question ID, Attempt ID) pair, annotators followed the procedure below: 1. Locate the specific programming problem on the online judge using the Question ID. 2. Paste the student’s answer into the judge, observe the first error, and increment the corresponding error-type count in the annotation spreadsheet. 3. Fix only that single error (referring to the expected answer as a reference), and verify that re-execution no longer produces the same error at the same location. 4. Repeat steps 2–3 until no explicit runtime or compile-time errors remain. 5. If the problem includes test cases, run them: failure to pass all test cases indicates a Logic Error; passing all test cases marks the sample as complete. A.6
A.8
Model Performance Evaluation Metrics and Prompt Templates
We evaluate model performance using precision, recall, and F1-score under two modes: Contains and Top-1. Precision (Contains). For each class c that appears in the gold labels: gold
and c ∈ yi
FPc = |{i | c ∈ / yi
gold
and c ∈ yi
Precisionc =
TPc TPc + FPc
TPc = |{i | c ∈ yi
Precisionfinal =
Related Datasets
Existing related work primarily focuses on a single programming language. Works that propose
pred
}|,
(4)
pred
}|
(5)
1 X Precisionc |C| c∈C
Recall (Contains). gold
FNc = |{i | c ∈ yi Recallc = Recallfinal =
pred
and c ∈ / yi
}|
TPc TPc + FNc 1 X Recallc |C| c∈C
F1 (Contains). F1c =
Use of AI Assistants
AI assistants were used solely for translation and grammar checking during the preparation of this manuscript. No AI-generated content was included in the scientific contributions, methodology, or experimental results reported in this paper. A.7
new datasets, such as COJ2022 and CPE28 proposed by (Han et al., 2023; Xiang et al., 2024), are single-language (C language) course-based datasets. Other works that utilize existing datasets also target only a single programming language; for example, (Shirafuji et al., 2023) and (Amin et al., 2024) are both Python-based works built on the AOJ ITP1 44 introductory problems.
2 × Precisionc × Recallc Precisionc + Recallc
F1final =
1 X F1c |C| c∈C
Precision / Recall / F1 (Top-1). The same formulas apply, except that the prediction set pred
yi
= {first predicted label}
If no prediction is made, the set is treated as empty.
Prompt Templates. We list the full prompt temindentation. Python relies on indentation to determine code blocks. plates used in our prompting classification experi26 6: UnboundLocalError - Raised when a local ments. Listing 1 shows the prompt for single-error variable is referenced before assignment ( typically within a function). This is a classification, and Listing 2 shows the prompt for subclass of NameError. multi-error classification. 27
Listing 1: Single-error prompt template used in the 28 prompting classification experiments. 1 2 3
# Define prompt template 29 prompt_template = You are an expert in Python code error classification, functioning as an automated 30 assessment assistant for a large-scale programming education platform. Your task is to analyze student code submissions, which 31 may contain errors, along with the corresponding coding questions and correct answers. You need to accurately identify whether the student code contains any errors 32 , as well as the specific type of error in 33 each case.
4 5 6 7 8
For each submission, you will receive structured information, including: - The question description - The expected answer - The student's code
9 10
**Important**: Many student submissions may be completely correct with no errors.
11 12 13
14
15
## Analysis Steps: 1. First, see the student's code to check if the code has explicit errors (Syntax Error, Name Error, Type Error, Indentation Error, UnboundLocal Error, Key Error, Index Error, EOF Error, Value Error, Tab Error) 2. If there is no explicit errors, see the question description, the student's code, and compare with the expected answer to check the logic of code 3. If the logic is wrong, there is an logic error; Else, there is no error.
Listing 2: Multi-error prompt template used in the prompting classification experiments. 1 2
16 17
Your goal is to classify the error into a predefined taxonomy of Python error types. Please output only the label number and label name, separated by a space, e.g., "0 No error" or "3 NameError".
18 19 20 21 22
23
24
25
7: KeyError - Raised when attempting to access a dictionary key that doesn't exist. 8: IndexError - Raised when attempting to access an index position in a sequence (like lists , tuples, or strings) that is out of range. 9: EOFError - Raised when the input() function hits an end-of-file condition without reading any data. 10: RecursionError - Raised when the maximum recursion depth is exceeded, typically caused by infinite recursion. 11: ValueError - Raised when a function receives an argument of the correct type but an inappropriate value. 12: TabError - Raised when indentation contains inconsistent use of tabs and spaces. 13: Other errors - Includes AttributeError ( Raised when attempting to access an attribute or method that doesn't exist on an object), RuntimeError (A generic error raised when an error is detected that doesn' t fall into any other category), SyntaxWarning (Issued when the syntax is questionable but not invalid enough to be a SyntaxError), ZeroDivisionError (Raised when division or modulo operation is performed with zero as the divisor), MemoryError ( Raised when an operation runs out of memory) , ModuleNotFoundError (Raised when a module could not be found. This is a subclass of ImportError), etc.
## Definition of Error Type Categories (Label number and names) 3 0: No error - The code runs successfully with no 4 explicit or logic errors. 1: LogicError - The code has no explicit error 5 but has a logic flaw. 6 2: SyntaxError - The code contains invalid 7 Python syntax and cannot be parsed. Examples 8 include missing colons, unmatched 9 parentheses, etc. 3: NameError - Raised when attempting to access 10 a variable, function, or module name that is 11 not defined. 12 4: TypeError - Raised when an operation or function is applied to an object of 13 inappropriate type or when types are incompatible. 5: IndentationError - The code has incorrect
prompt template = You are an expert in Python code error classification, functioning as an automated assessment assistant for a large-scale programming education platform. Your task is to analyze student code submissions, which may contain errors, along with the corresponding coding questions and correct answers. You need to accurately identify whether the student code contains any errors , as well as the specific type of error in each case. For each submission, you will receive structured information, including: - The question description - The expected answer - The student's code **Important**: Many student code submissions may be completely correct with no errors. --#Analysis Process (internal reasoning, not directly output): 1. Examine the student's code for explicit errors (Syntax Error, Name Error, Type Error , Indentation Error, Unbound Local Error, Key Error, Index Error, EOF Error, Value
14
15 16
Error, Tab Error). 52 2. If no explicit error is found, compare the 53 student's code with the expected answer to 54 check the logic. 55 3. If the logic is incorrect, classify as Logic Error. Otherwise, classify as No Error. Use this reasoning process internally to justify the final prediction.
17 18 19 20 21 22 23
24
25
26
27 28 29
30
31 32 33
--#Definition of Error Type Categories (Label numbers and names) 0: No Error: The code runs successfully with no 56 explicit or Logic Error. 1: Logic Error: The code has no explicit error 57 but has a logic flaw. 58 2: Syntax Error: The code contains invalid 59 Python syntax and cannot be parsed. 60 3: Name Error: Raised when attempting to access a variable, function, or module name that is not defined. 4: Type Error: Raised when an operation or function is applied to an object of inappropriate type or when types are incompatible. 5: Indentation Error: Incorrect indentation. Python relies on indentation to determine code blocks. 6: Unbound Local Error: A local variable is referenced before assignment (typically within a function). Subclass of Name Error. 7: Key Error: Raised when accessing a dictionary key that doesn't exist. 8: Index Error: Raised when accessing an index position in a sequence that is out of range. 9: EOF Error: Raised when the input() function hits an end-of-file condition without reading data. 10: Recursion Error: Raised when the maximum recursion depth is exceeded (infinite recursion). 11: Value Error: Raised when an argument has the correct type but an inappropriate value. 12: Tab Error: Raised when indentation inconsistently mixes tabs and spaces. 13: Other Errors: Includes AttributeError, RuntimeError, SyntaxWarning, ZeroDivisionError, MemoryError, ModuleNotFoundError, etc.
return x + y Output: Reasoning: Firstly, we examine the code for explicit errors in order. The function header is missing a colon (Syntax Error). The variable y is not defined (Name Error). While there are explicit errors, we continue to examine the logic. Even if y were defined, the logic is incorrect since it should return x * x instead of x + y (Logic Error). Predicted Label: Syntax Error (#2), Name Error (#3), Logic Error (#1) Test Example (for model to fill) Reasoning: Predicted Label:
A.9
Experiment Setup Details
We conduct finetuning experiments using CodeLlama-7B as the base model. The task is formulated as a multi-class classification problem with 14 error categories. The key hyperparameters required for reproducibility are summarized below. • Base model: CodeLlama-7B • Number of labels: 14 • Maximum sequence length: 512 • Training epochs: 5 • Effective batch size: 12 (batch size 4 with gradient accumulation steps 3) • Learning rate: 2 × 10−4 • Weight decay: 0.01 • Random seed: 88
34 35 36 37 38 39
--## Output Requirements Please provide output in the following format ( consistent and unified): Reasoning: brief explanation of detected errors Predicted Label: ErrorName (#n), ErrorName (#m), ...
40 41 42 43 44
--## Example ### Input: Question description: "Write a function that returns the square of a number."
45 46 47 48
Expected Answer: def square(x): return x * x
49 50 51
Student Code: def square(x)
We apply LoRA-based parameter-efficient finetuning with the following configuration: LoRA rank r = 16, LoRA alpha = 32, and dropout = 0.05. To improve memory efficiency, we employ 4-bit quantization using the NF4 scheme with bfloat16 computation. Training is performed using the AdamW optimizer with learning rate warmup and early stopping based on validation loss. Gradient checkpointing is enabled to reduce memory usage. Other training and implementation details follow standard practice. For API-based large language models, including GPT-3.5-Turbo, GPT-4o, DeepSeek-V3, and Gemini 2.5 Pro, we evaluate model performance using prompt-based inference without parameter updates.
We set the temperature to 0 to encourage deterministic outputs and constrain the maximum output length. Random seed control is not applicable for API-based models and is therefore not enforced. All sample extraction procedures are implemented using deterministic scripts with fixed random seeds to ensure reproducibility. Prior to evaluation, we apply rule-based sampling and filtering to control the distribution of extracted prediction samples. A.10
Single-Error Finetuning Classification
A.10.1
Dev Set Results
The following tables report dev set performance for all six finetuning experiments (CodeBERT and CodeLlama on Tasks A, B, and C) under the problem-level QuestionID split. These complement the test set results reported in the main text. For Task C, class 7 (Key Error) is absent from the dev set due to the random QuestionID split, but present in the test set. Class No Error Error Macro avg Weighted avg
Precision (%) 73.1 95.9 84.5 87.1
Recall (%) 94.6 78.1 86.4 84.5
F1-score (%) 82.5 86.1 84.3 84.7
Table 14: CodeBERT Task A (Binary Classification) results on the dev set under problem-level split by QuestionID. Per-class precision/recall/F1 are reported at one decimal place; Macro and Weighted averages follow the stored full-precision computation.
Class No Error Explicit Error Logic Error Macro avg Weighted avg
Precision (%) 81.1 80.3 67.1 76.2 76.6
Recall (%) 82.0 71.8 73.2 75.7 76.2
F1-score (%) 81.6 75.8 70.1 75.8 76.3
Table 15: CodeBERT Task B (Three-class Classification) results on the dev set under problem-level split by QuestionID. Per-class precision/recall/F1 are reported at one decimal place; Macro and Weighted averages follow the stored full-precision computation.
A.10.2
Entropy and Confidence Analysis
We further quantify model uncertainty on the diagnostic subset using entropy and confidence distributions. A.10.3
(Entropy–F1 Relationship analysis Experiment Setup
To examine how model uncertainty relates to human judgments in multi-error settings, we align
Class 0 No Error 1 Logic Error 2 Syntax Error 3 Name Error 4 Type Error 5 Indentation Error 6 Unbound Local Error 7 Key Error 8 Index Error 9 EOF Error 10 Recursion Error 11 Value Error 12 Tab Error 13 Other Errors Macro avg Weighted avg
Precision (%) 77 78 93 78 69 80 67 — 100 6 0 0 33 0 52 79
Recall (%) 96 72 83 56 36 60 19 — 50 10 0 0 40 0 40 79
F1-score (%) 85 75 87 65 47 69 30 — 67 7 0 0 36 0 44 78
Table 16: CodeBERT Task C (Fine-grained Multiclass Classification) results on the dev set under problem-level split by QuestionID. Class 7 (Key Error) shows “—” because the randomQuestionIDbased split yields no Key Error samples in the dev set; test set results (where Key Error is present, support = 113) are reported in Table 5. Per-class precision/recall/F1 are reported at decimal precision; Macro and Weighted averages follow the stored full-precision computation (one decimal place). Error types are indexed according to the taxonomy described in Section 4.1.
Class No Error Error Macro avg Weighted avg
Precision (%) 96.5 98.3 97.4 97.6
Recall (%) 97.2 97.8 97.5 97.6
F1-score (%) 96.8 98.0 97.4 97.6
Table 17: CodeLlama Task A (Binary Classification) results on the dev set under problem-level split by QuestionID. Per-class precision/recall/F1 are reported at one decimal place; Macro and Weighted averages follow the stored full-precision computation.
Class No Error Explicit Error Logic Error Macro avg Weighted avg
Precision (%) 96.9 95.6 94.9 95.8 95.9
Recall (%) 97.2 97.7 92.4 95.8 95.9
F1-score (%) 97.1 96.6 93.6 95.8 95.9
Table 18: CodeLlama Task B (Three-class Classification) results on the dev set under problem-level split by QuestionID. Per-class precision/recall/F1 are reported at one decimal place; Macro and Weighted averages follow the stored full-precision computation.
Class 0 No Error 1 Logic Error 2 Syntax Error 3 Name Error 4 Type Error 5 Indentation Error 6 Unbound Local Error 7 Key Error 8 Index Error 9 EOF Error 10 Recursion Error 11 Value Error 12 Tab Error 13 Other Errors Macro avg Weighted avg
Precision (%) 96 95 99 90 90 100 38 — 100 100 93 100 71 100 90 95
Recall (%) 98 93 99 98 86 99 14 — 50 100 81 14 100 43 75 95
F1-score (%) 97 94 99 93 88 99 21 — 67 100 87 25 83 60 78 95
Table 19: CodeLlama Task C (Fine-grained Multiclass Classification) results on the dev set under problem-level split by QuestionID. Class 7 (Key Error) shows “—” because the QuestionID-based split yields no Key Error samples in the dev set; test set results (where Key Error is present, support = 113) are reported in Table 8. Per-class precision/recall/F1 are reported at decimal precision; Macro and Weighted averages follow the stored full-precision computation (one decimal place). Error types are indexed according to the taxonomy described in Section 4.1.
Figure 9: CodeBERT Task C Multi-class Classification Confusion Matrix. Single-error finetuning results on the held-out dev set under problem-level QuestionID split.
each of the 97 expert annotated multi-error sample with its prediction entropy and top-k model output. Therefore, each sample contains both the humanidentified error set and the ranked error-type predictions of the model. Also, the entropy is calculated from the probability distribution of the model in error types and samples are grouped into entropy buckets to observe performance trends. For each bucket, we compare human expert annotations with the model’s top-k using Macro-F1. This comparative analysis highlights how increasing uncertainty (higher entropy) corresponds to reduced agreement between model predictions and human-annotated multi-error labels. A.10.4
Figure 8: CodeBERT Task A Binary Classification Confusion Matrix. Single-error finetuning results on the held-out dev set under problem-level QuestionID split.
Entropy–F1 Relationship analysis Experiment Results
Figure 10 illustrates the Macro-F1 scores across entropy buckets for Top1–Top4 predictions (15- and 20-bucket settings). A clear trend is observed: as entropy increases, Macro-F1 decreases, confirming our hypothesis that higher uncertainty correlates with lower prediction reliability. This downward trend is most pronounced for Top1 predictions, less distinct for Top2, and nearly disappears for Top3–Top4. The results indicate that including additional top-k candidates smooths the performance degradation across entropy, suggesting that while the model’s most confident predictions degrade sharply with uncertainty, its broader candidate set remains more stable. The standard deviation of Macro-F1 within
each bucket also increases with entropy, implying greater variability in model performance on uncertain samples.
Figure 11: CodeLlama-7b TaskA Binary Classification Confusion Matrix. Single-error finetuning results on the held-out test set under problem-level QuestionID split.
Figure 10: Entropy-topk-F1 Trend Multi-Error Diagnostic Analysis. Macro-F1 of top-k k model predictions (Top1–Top4) across entropy buckets (15- and 20-bucket settings) on the 97-sample multi-error diagnostic subset. Results derived from single-error finetuning (CodeBERT Task C); entropy is computed from the model’s predicted class probability distribution. Figure 12: CodeLlama-7b Task B Three-Class Classification Confusion Matrix. Single-error finetuning results on the held-out test set under problem-level QuestionID split.
A.11
Single-Error Prompting Classification
Table 21 summarizes the overall performance of four representative models. In addition to accuracy, the models exhibit substantial differences in inference runtime. As shown in Table 21, GPT-4o infers significantly faster,
Figure 13: CodeLlama-7b Task C Multi-Class Classification Confusion Matrix. Single-error finetuning results on the held-out test set under problem-level QuestionID split.
Model GPT-3.5 GPT-4o Gemini DeepSeek-V3
No Error (%) 34.7 85.6 83.2 81.9
Indentation Error (%) 0.0 36.8 84.1 46.8
Logic Error (%) 92.3 76.2 89.6 85.2
Name Error (%) 3.3 36.2 96.3 73.5
Type Error (%) 0.0 18.3 91.0 26.2
Table 20: Single-Error Prompting Classification: Perclass accuracy (in %) for selected error types across models.
Model GPT-3.5 GPT-4o Gemini DeepSeek-V3
Runtime 45 mins 45 mins 24 hours 5–6 hours
Accuracy (%) 40.3% 71.5% 85.9% 73.6%
Table 21: Single-Error Prompting Classification of Runtime and Accuracy. Overall accuracy and approximate inference runtime for four LLMs evaluated on the full test set (4,865 samples).
Figure 14: DeepSeek-V3 Row-Normalized Confusion Matrix of Single-Error Prompting Classification.
while Gemini 2.5 Pro models obtain higher accuracy at a higher computational cost. From the results, we observe the following trends. Gemini 2.5 Pro obtains comparable performance (85.0%) but requires significantly longer inference time (24 hours). GPT-4o reaches 71.5% and DeepSeek-V3 73.6%, though GPT-4o is much faster: 45 minutes vs. 5–6 hours. GPT-3.5 lags far behind at 40.3% despite similar runtime to GPT-4o. A notable pattern across all models is the tendency to over-predict Logic Error, leading to bias in error distribution. While Gemini 2.5 Pro demonstrate strong performance overall, the error bias remains an open challenge; per-class accuracy for representative error types and full evaluation metrics are provided in Table 20 and Table 9, respectively. In terms of category-specific performance, Gemini 2.5 Pro demonstrates the broadest proficiency, excelling across Indentation Error, Logic Error, Name Error, and Type Error. Similarly, DeepSeek-V3 shows strong capabilities, particularly in Logic Error and Name Error, with no significant weaknesses identified. In contrast, GPT4o exhibits no standout strengths in this comparison and notably struggles with both Logic Error and Name Error. A.11.1
Qualitative Analysis of SE→LE Misclassification To complement the quantitative results reported in Section 6.1.2, we inspect representative cases in
Two representative examples are shown below in Section A.11.3. In both cases, the code contains a syntactic error that would prevent execution, but the surrounding code makes the intended computation easy to read off, and the model returns a Logic Error label instead. A.11.2
Reasoning and Overprediction Case Studies To provide concrete illustrations of the observed biases and uncertainty, we conduct qualitative case studies on reasoning failures and Logic Error Overpredictions. A.11.3
Representative Examples
Listing 3: Example 1. A submission containing invalid assignment expressions (Syntax Error) misclassified as Logic Error.
Figure 15: GPT-4o Row-Normalized Confusion Matrix 1 input_str = input() of Single-Error Prompting Classification. 2 3
which Syntax Error instances are predicted as 4 Logic Error (SE→LE). 5 We find that such cases often involve code with 6 7 syntactic errors whose surrounding structure still 8 suggests what the student was trying to do. In 9 these cases, models tend to pick up on the intended 10 11 logic rather than flag the syntactic problem. This is 12 consistent with the intent-first bias discussed in the 13 14 main text. 15 We note that single-error gold labels follow the 16 IDE convention of recording the first execution 17 error. When a submission contains multiple cooccurring issues, the gold label is therefore fixed to the first execution error, even if a model could also reasonably identify another error in the code. The diagnostic multi-error subset (Section 6.3) is de- 1 2 signed precisely to examine model behavior under 3 such conditions. 4 A possible contributing factor is the co- 56 occurrence of explicit and logic errors, particularly in cases where logic-related anomalies appear ear- 7 8 lier in the code. Although the model is explicitly 9 instructed to consider Logic Error only in the 10 absence of other error types, these examples sug-11 gest that the model does not always strictly adhere to this constraint. We note that this behavior may stem from multiple mechanisms, including imperfect adherence to prompt instructions, incomplete recognition of co-occurring errors, or an implicit 1 2 bias toward logic errors; disentangling these mech- 3 anisms requires further investigation. 4
position = {} str(res) = ” # Error: Assignment to function call int(pos) = len(input str) # Error: Assignment to function call def get_first_duplicate(input_str): str(input_str) for i in input_str: if i not in position: position[i] = input_str.index(i) else: if input_str.index(i) < pos: res = i pos = input_str.index(i) return res print(get_first_duplicate(input_str))
Listing 4: Example 2. A submission with malformed expressions (Syntax Error) in a near-correct computation. age = int(input()) weight = int(input()) heart_rate = int(input()) time = int(int(input())) calories_burned = (age + weight + heart_rate + time - 75.4991) \ * time / (8.368 / age * 0.2757) \ / (8.368 / weight * 0.03295) \ / (8.368 / heart_rate * 1.0781) print(calories_burned)
Listing 5: Example 3. A submission with logical redundancy where a nested elif condition replicates its parent if statement. num_spiders = int(input()) num_cats = int(input()) if num_spiders > 10: if num_cats > num_spiders:
5 6 7 8 9 10
print('cat␣will␣eat␣spiders') elif num_spiders > 10: # Logical Redundancy (Dead Code) if num_cats <= num_spiders: print('cat␣will␣not␣eat␣all␣spiders') else: print('few␣spiders')
A.11.4
Behavioral Alignment Between Model Explanations and Actual Execution Behavior
Scope of this analysis The primary goal of our prompting experiments is to evaluate classification performance and identify systematic model biases, not to analyse internal execution-reasoning mechanisms. Nevertheless, our prompting setup instructs models to output a brief Reasoning string alongside each predicted label (see Appendix A.8). We use these model-generated explanations as a behavioural signal: by comparing what a model says about a submission against what the Python interpreter actually does when it processes that 1 code, we can assess whether the model’s decision is aligned with real execution behaviour. Concretely, Python errors follow a strict execution order: parse-time failures (e.g. Syntax Error) occur before any code runs; runtime exceptions (e.g. NameError, TypeError) occur during execution; and logic failures occur only after the code runs to completion but produces wrong output. A prediction is execution-aligned if the model’s explanation is consistent with the stage at which the error actually occurs. Observed patterns Inspecting the model explanations for misclassified samples, we identify two systematic misalignment patterns. Pattern 1: Intent-first reasoning (SE to LE). When a submission contains a parse-level error, the interpreter halts immediately and never evaluates any logic. Yet models frequently skip the parse stage and reason directly about algorithmic correctness, producing a Logic Error prediction for code 1 that cannot run at all. 2 3
Listing 6: Leap-year check containing a full-width colon ： (line 2). The parser fails at this character; no logic is 4 5 ever evaluated. 1 2 3 4 5 6 7
def is_leap_year(year): if year % 400 == 0： # full-width colon -parse fails here return "leap␣year" elif year % 100 == 0: return "not␣a␣leap␣year" elif year % 4 == 0: return "leap␣year"
True label: Syntax Error (parse-time failure on the full-width colon). Model prediction: Logic Error. Model explanation: “the leap year conditions are not correctly structured . . . the order of the conditions is incorrect.” The explanation evaluates the correctness of the conditional logic , a question that is only meaningful if the code can be parsed and executed. Because the parser fails on line 2, the logical structure is never reached. The model’s reasoning is grounded in the code’s intended behaviour, not in what the interpreter encounters. A second case shows the same pattern more starkly: Listing 7: Print statement with no quotation marks around the argument. The interpreter raises a Syntax Error/NameError immediately; the statement cannot be parsed as written. print(I am Python!)
True label: Syntax Error. Model prediction: Logic Error. Model explanation: “the issue lies in the logic of the code . . . the incorrect logic of not enclosing the string within quotation marks.” The model reframes a structural parse failure as a reasoning mistake by the student, describing the problem in terms of intent rather than interpreter state. Pattern 2: Surface-feature substitution (LE to SE). In the reverse direction, models sometimes predict Syntax Error for code that is syntactically valid and executes without error. The explanations cite visible surface features , unconventional spacing, unfamiliar constructs, or naming inconsistencies , as evidence of a syntax violation. Listing 8: Discount function that runs without error but returns strings instead of integers, giving wrong output (Logic Error). def get_discount(param): if param < 140: return '0' # executes fine; wrong type else: return'100' # missing space, but valid Python syntax
True label: Logic Error (returns ‘0’/‘100’ instead of 0/100; code runs without exception). Model prediction: Syntax Error. Model explanation: “return‘100’ lacks a space, violating PEP 8 . . . looks broken to automated
Figure 16: Single-Error Prompting Classification Accuracy by True Error Type. Per-class accuracy for all evaluated LLMs on the test set, under zero-shot prompting without finetuning.
tools; the parameter name param does not match the expected signature height , a static mismatch.” Neither observation prevents execution: Python does not require whitespace between return and its argument, and parameter naming does not affect runtime behaviour. The model’s prediction is driven by static features that resemble syntax problems rather than by whether the code actually executes. General trend Across both patterns, model explanations consistently focus on apparent structure and intent rather than on the execution stage at which the error occurs. In SE→LE cases, models reason about logic that is never reached. In LE→SE cases, models flag surface features that do not prevent execution. This behavioural misalignment is consistent with the quantitative bias reported in Section 6.2.1: models over-predict Logic Error because they tend to reason about code intent first, and under-predict fine-grained explicit errors because surface-level syntactic cues are ambiguous. We emphasise that the analysis above is based solely on model-generated explanations, not on controlled execution traces or step-by-step interpreter simulation. A more rigorous executiongrounded evaluation, for example, prompting models to explicitly simulate interpreter state at each line before producing a label, similar to the approach of Wang et al. (2025a) , would provide stronger evidence about whether models genuinely track execution behaviour. We leave this as a direc-
tion for future work, and will include this analysis in the next version of the paper. Detailed case studies and additional examples are provided in Appendix A.12.
Dataset Model Precision (%) Total (215 + 97 samples) DeepSeek-V3 85.7 Gemini 85.7 GPT-3.5 71.4 GPT-4o 64.3 Multi-Error Subset (97 samples) DeepSeek-V3 84.6 Gemini 84.6 GPT-3.5 69.2 GPT-4o 61.5 No Error Subset(134 samples) DeepSeek-V3 100.0 Gemini 100.0 GPT-3.5 100.0 GPT-4o 100.0 Logic Error Subset(81 samples) DeepSeek-V3 100.0 Gemini 100.0 GPT-3.5 100.0 GPT-4o 100.0
Recall (%)
F1 (%)
49.0 62.0 30.1 39.9
60.1 70.2 39.9 46.6
47.2 59.9 30.1 36.6
58.2 68.4 39.4 43.2
75.0 88.2 31.6 86.0
85.7 93.8 48.0 92.5
90.2 93.9 78.0 85.4
94.9 96.9 87.7 92.1
Table 22: Multi-Error Prompting Classification of Comprehensive Results in Contains Mode. Precision, Recall, and F1-score across four dataset partitions: The top block shows combined results over 215 + 97 samples; subsequent sections show subsets for multierror (97 samples), No Error Subset(134 samples), and Logic Error Subset(81 samples). All models evaluated under Chain-of-Thought prompting without finetuning.
A.11.5 Classification Reasoning A.12 Pedagogical Alignment: System vs. Human Labels While the main text establishes the necessity of manual annotation, this appendix provides qualitative evidence of the divergence between systemgenerated and human-refined labels. Systemgenerated labels are inherently execution-centric, capturing only the terminal symptom (e.g., the first runtime exception). in contrast, our humanannotated labels are pedagogical, identifying the root cause that obscures the student’s logical intent. We present representative cases below to illustrate how relying solely on mechanical execution logs introduces “diagnostic noise” that misguides pedagogical intervention. A.12.1 Case Studies of Label Divergence Case 3: This submission contains an illegal assignment (e.g., str(res) = ’ ’). • System Prediction: Logic Error (based on heuristic mapping). • Human Annotation: Syntax Error. Analysis: Although the logic is flawed, annotators prioritize the structural violation because code structure must be rectified before logical correctness can be meaningfully evaluated. The system fails to recognize this pedagogical dependency. Case 4: A multi-line formula contains misplaced operators that break execution at the line boundary. • System Prediction: Syntax Error (due to unexpected tokens). • Human Annotation: Logic Error. Analysis: Humans recognize the student’s attempt to implement a specific formula. The syntax error is merely a side effect of a conceptual misunderstanding of operator precedence, not a typo. Example: Static Intent vs. Runtime Execution. Consider the snippet: print(’int(input() 岁’). While the interpreter may treat the unbalanced parenthesis as a Logic Error (executing it as a string literal), human annotators identify it as a Syntax Error. The system blindly executes; the human diagnoses the structural breakage.