Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors Alexandru-Radu Moraru

Shreyan Biswas

Ujwal Gadiraju

Delft University of Technology

Delft University of Technology

Delft University of Technology

Delft, The Netherlands

Delft, The Netherlands

Delft, The Netherlands

alexradumoraru@gmail.com

s.biswas@tudelft.org

u.k.gadiraju@tudelft.nl

arXiv:2608.20896v1 [cs.SE] 21 Aug 2026

Abstract

Prior work has repeatedly shown that debugging is time-consuming, frustrating, and cognitively demanding across both educational and professional settings [1, 15, 16]. Poorly designed error messages can slow progress, increase mental effort, and undermine learners’ confidence [7, 26]. A survey by Becker et al. [4] identifies several recurring weaknesses in programming error messages, including excessive jargon, insufficient context, poor localization, and limited actionable guidance. These limitations have motivated a long line of work on improving compiler and interpreter diagnostics through clearer wording, better examples, enhanced localization, and pedagogically informed feedback [12, 22, 24].

Programming error messages are critical for software development, yet they remain difficult for novice programmers to interpret. While Large Language Models (LLMs) can rewrite these errors into clearer explanations, it remains unclear whether increased readability improves objective debugging performance or how explanation styles should align with programmer skill. We present a multi-stage crowdsourced study (𝑁 = 103) evaluating skill-targeted, LLM-generated Python error messages. Using a custom proficiency assessment, we categorized participants by skill level and tested standard interpreter messages against two LLM-generated styles: pragmatic (action-oriented) and contingent (scaffolded explanations). We measured both objective debugging metrics (fix rate, attempts, time-tofix) and subjective perceptions (readability, cognitive load, tone). Our results show that while LLM-rewritten messages significantly improved subjective evaluations, with pragmatic messages rated as clearer and less cognitively demanding, these perceived gains did not translate into statistically significant improvements in objective debugging performance. This highlights a critical human-AI complementarity gap: explanations that feel better to users do not necessarily make them more effective debuggers. We discuss design implications for adaptive AI feedback systems, arguing that future tools should pivot from static skill-targeted rewriting toward dynamic adjustments based on a user’s real-time repair trajectory.

Large language models (LLMs) introduce a new opportunity for improving programming error messages. Rather than relying only on fixed diagnostic templates, LLMs can rewrite interpreter output using the surrounding code, traceback, and user-facing design goals. In principle, such rewrites could make error messages more readable, less intimidating, and more actionable. However, prior work on scaffolding, cognitive load, and expertise-sensitive instruction suggests that feedback is rarely universally effective: support that helps less experienced learners can become redundant, distracting, or inefficient for more experienced users [10, 20, 25]. This creates a human computation problem for LLM-generated programming feedback: how should such messages be designed, calibrated, and evaluated with human users? A rewritten explanation that sounds helpful may still fail to improve debugging performance. Similarly, an explanation that is useful for a more experienced programmer may be too terse for a less experienced programmer, while a scaffolded explanation that supports a less experienced programmer may feel verbose or distracting to someone with stronger debugging fluency. Thus, the design of LLM-generated programming feedback requires not only model capability, but also systematic human evaluation across user skill levels and debugging contexts.

Keywords Human-AI interaction, debugging, programming error messages, large language models, adaptive explanations

1

Introduction

Programming error messages are one of the most frequent forms of feedback that programmers receive from software systems. When a program fails, the compiler or interpreter becomes an immediate explanatory interface: it identifies that something went wrong, points to a location, and offers diagnostic information that the programmer must interpret and act upon. In principle, such messages should support learning and repair. In practice, they often remain difficult to understand, especially for less experienced programmers who may lack the conceptual vocabulary and debugging experience needed to map a diagnostic message to an actionable fix. Even experienced programmers can spend substantial effort interpreting error messages, locating the underlying fault, and deciding how to proceed.

In this paper, we study skill-targeted LLM rewrites of Python programming error messages through a multi-stage crowdsourced evaluation. We use the term skill-targeted to describe explanation styles designed for different assessed proficiency levels; we do not claim that the system dynamically adapts during interaction. We focus on two explanation styles. Pragmatic messages provide concise, action-oriented guidance intended to help users move quickly from diagnosis to repair. Contingent messages provide more scaffolded explanatory support, offering additional context about the likely cause of the error and how the programmer might reason about it. Rather than treating LLM-generated explanations as universally 1

Moraru et al.

beneficial, we ask whether different explanation styles influence debugging performance and user perceptions for programmers with different assessed skill levels.

across objective debugging outcomes and subjective user perceptions. Third, we identify a gap between perceived explanation quality and debugging effectiveness, offering design implications for future human-AI programming support systems.

To support this evaluation, we first develop a short Python proficiency assessment focused on debugging-relevant competencies, including error comprehension, code understanding, fault localization, and error resolution. We then conduct a formative study to curate a set of buggy Python snippets and corresponding interpreter messages with appropriate difficulty for a crowdsourced debugging task. Finally, in a Prolific experiment with 103 participants, we compare standard Python interpreter messages against LLM-generated pragmatic and contingent rewrites. Participants attempt to repair a buggy Python snippet and then evaluate the message they received in terms of readability, cognitive load, and perceived tone.

2

Background and Related Work

Programming error messages are a central feedback channel between software systems and programmers. They are produced automatically by compilers, interpreters, and runtime environments, but their usefulness depends on how users interpret and translate them into repair actions. This makes them a useful setting for studying human-AI complementarity: automated systems can produce diagnostic information, but human programmers must still understand, trust, and act on that information. We review work on programming error messages, LLM-based diagnostic explanations, skill-targeted feedback, and human evaluation of AI-generated programming support.

Our study is guided by the following research questions: RQ1: How do skill-targeted LLM-generated Python error messages influence code-correction performance compared to standard interpreter messages across lower and higher assessed proficiency groups?

2.1

Programming Error Messages as Human-Facing Diagnostics

Programming error messages include both compile-time diagnostics, which are reported before a program runs, and runtime exceptions, which are reported during execution. Although these categories are often treated separately, the boundary is not strict across languages and execution environments. Many ecosystems combine ahead-of-time compilation, bytecode generation, just-in-time compilation, static analysis, and runtime diagnostics. We therefore use the broader term programming error messages to refer to both compiler and interpreter feedback.

RQ2: How do programmers across lower and higher assessed proficiency groups perceive the readability, cognitive load, and tone of skill-targeted LLM-generated Python error messages compared to standard interpreter messages? Our results show a divergence between subjective experience and objective debugging performance. LLM-rewritten messages were rated more favorably than standard interpreter messages on several perceptual measures, with pragmatic messages receiving the strongest subjective evaluations. Participants perceived pragmatic messages as clearer, more helpful, and less cognitively demanding. However, these subjective benefits did not reliably translate into statistically significant improvements in objective debugging outcomes such as fix rate, time-to-fix, or number of attempts. Contingent messages also showed favorable subjective patterns, but their effects were less consistent than those of pragmatic messages. This divergence reveals a human-AI complementarity gap in programming support: explanations that users experience as clearer and more helpful may not necessarily lead to better debugging performance. For HCI and human computation research, this finding highlights the need to evaluate AI-generated feedback not only by fluency or perceived helpfulness, but also by its effect on situated human problem solving. For programming-support systems, our results suggest that static skill-targeted rewriting may be insufficient on its own. Future systems may need to adapt explanations to the user’s interaction state, repair trajectory, and specific error context, rather than relying only on prior skill classification.

Prior work has repeatedly shown that programming error messages can be difficult for users to understand and act upon. Messages may be ambiguous, overly technical, poorly localized, or insufficiently explanatory, limiting their usefulness as repair guidance. Unclear or verbose messages can increase cognitive load, cause frustration, reduce productivity, and lead users toward ineffective repair attempts [5]. These problems are especially salient in educational contexts, where less experienced programmers may lack the conceptual vocabulary needed to map a diagnostic message to an underlying fault. However, more experienced programmers are also affected when messages are incomplete, misleading, or require substantial effort to interpret. The form and difficulty of programming error messages also depend on language design and execution model. Functional languages1 often produce complex type-related diagnostics, whereas object-oriented languages may surface errors related to inheritance, polymorphism, object state, or lifecycle management. Dynamically typed languages such as Python often produce runtime exceptions with limited local context. Because many modern languages are multi-paradigm2 , no single message style is likely to serve all contexts. Effective feedback must therefore account for both the error context and the user’s programming expertise.

This paper makes three contributions. First, we present a crowdsourced pipeline for designing and evaluating LLM-generated programming error explanations, including a debugging-oriented proficiency assessment and formative snippet-selection process. Second, we provide empirical evidence comparing standard Python interpreter messages with two styles of LLM-generated rewrites

1 Functional programming (e.g., Haskell, F#, Lisp) treats computation as the evaluation

of mathematical functions, emphasizing immutability, declarative expressions, and higher-order functions [8, 9]. 2 Python [6], for example, supports procedural, functional, and object-oriented constructs. 2

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

2.2

2.4

Improving Error Messages for Human Understanding

Research on compiler and interpreter diagnostics has long argued that error messages should do more than report failure. They should help users identify the fault, understand its cause, and determine a plausible repair. Becker et al. [4]’s survey of compiler error messages identifies recurring challenges, including excessive jargon, poor localization, insufficient contextual information, and lack of actionable guidance. Related work in computing education and programming languages has examined how learners interact with compiler messages, how diagnostics can be designed and evaluated, and why message clarity does not always translate into successful repair.

Programmers with different levels of expertise may need different forms of feedback. Less experienced programmers may benefit from explanations that unpack the likely cause of an error, connect it to relevant programming concepts, and provide scaffolded guidance. More experienced programmers may prefer concise messages that preserve workflow and quickly identify the likely repair direction. This dis tinction is consistent with foundational work on scaffolding and cognitive load, where effective support depends on what the learner already knows and how much additional guidance the task requires [20, 25]. It is also consistent with the expertise reversal effect: instructional guidance that benefits less experienced learners can become redundant, inefficient, or even counterproductive for more knowledgeable users [10].

This literature motivates several design goals for improved programming error messages. Messages should be readable, using clear wording and avoiding unnecessary jargon; actionable, connecting the observed failure to a likely cause and useful next step; and appropriately scoped, providing enough information to support repair without disrupting the user’s workflow. These goals align with broader HCI and learning-sciences work on feedback, scaffolding, and cognitive load. In programming contexts, the challenge is to support understanding without overwhelming users or interrupting the repair process.

2.3

Skill-Targeted Feedback and Explanation Style

In this paper, we examine two skill-targeted explanation styles. Pragmatic messages are concise and task-oriented: they name the error, localize the likely cause, and provide a short hint toward repair. Contingent messages provide more scaffolded guidance: they include additional context about the likely intent, possible misconception, and reasoning path. These styles are informed by work on actionable feedback, Bloom’s taxonomy [2, 11], and argumentation models such as claim-evidence-reasoning [14] and Toulmin’s model [3]. We use these frameworks to structure two explanation styles that may be differently suited to users with different assessed programming proficiency.

LLMs for Rephrasing and Explaining Diagnostics

We do not evaluate a fully dynamic tutor that updates its feedback in response to user behavior over time. Instead, we evaluate skill-targeted rewriting: the use of explanation styles designed for different proficiency needs. This provides an empirical basis for understanding whether static style differences are sufficient to improve debugging outcomes, or whether future systems need more interaction-aware adaptation.

LLMs create new opportunities for improving programming error messages. Unlike fixed diagnostic templates, they can combine the original traceback, surrounding code, and task-specific instructions to produce explanations that are more readable, contextual, or action-oriented. Recent work has explored LLMs for code explanation, repair, and diagnostic support, including rephrasing compiler or interpreter messages into more novice-friendly forms [13, 17, 19, 21]. These studies show promise but also important limitations: generated explanations can be conceptually accurate yet inconsistent, novice-friendly yet incomplete, or clearer without reliably improving objective debugging outcomes [13, 18, 21]. Prompt structure and context inclusion also matter, with concise prompts, source-code context, and controlled examples improving clarity and fix specificity in some settings [17, 19]. A large randomized study further suggests that GPT-generated messages can reduce repeated runs and additional attempts in authentic Python settings [23].

Evaluating LLM-generated programming feedback requires more than testing whether a model can produce a plausible explanation. It requires human-centered evaluation of how users with different skill levels interpret, trust, and act on that feedback. Crowdsourcing offers a practical way to conduct such evaluations at scale, but it also requires careful calibration of both participants and tasks. In our study, crowdsourced input is used not only for final evaluation, but also to develop a debugging-oriented proficiency assessment and select code-repair tasks of appropriate difficulty.

3

Study Design and Main Experiment

We used a multi-stage crowdsourced design to calibrate participant skill and debugging-task difficulty before evaluating LLMgenerated programming error messages. This design reflects a core challenge in studying AI-generated feedback: explanation effects are difficult to interpret if participants’ programming proficiency or task difficulty is poorly calibrated. Our pipeline consisted of three stages: (1) a debugging-oriented Python proficiency assessment, (2) a formative task-selection study, and (3) a main Prolific experiment comparing standard Python interpreter messages with two LLM-generated rewrite styles.

Together, this work suggests that LLM-generated diagnostics should not be treated as automatically beneficial. Model capability alone does not establish whether generated feedback improves human debugging: an explanation may be fluent, readable, and preferred by users while still failing to improve repair behavior. This makes LLM-generated diagnostics a human-centered evaluation problem rather than only a code-generation problem. Our work contributes to this line of research by evaluating both behavioral outcomes and subjective perceptions of LLM-rewritten Python error messages. 3

Moraru et al.

each candidate triggered the intended error and avoided errors that could lead to multiple simultaneous faults. We then conducted a formative task-selection study in which participants rated each candidate snippet and standard interpreter message on code difficulty, fix difficulty, error-message mental demand, and error-message usefulness. Based on these ratings, we selected four snippets that provided variation in error type while remaining within a moderate difficulty range for the target participant population. The full selected snippets, standard interpreter messages, and task-selection survey items are provided in Appendix B.

3.2

The main experiment compared three error-message styles. The standard condition showed the unmodified Python interpreter message. The pragmatic condition used an LLM-generated rewrite designed to provide concise, action-oriented guidance. The contingent condition used an LLM-generated rewrite designed to provide more scaffolded explanatory support, including additional context about the likely cause of the error and how the programmer might reason about it.

Figure 1: Overview of the multi-stage study design. We first calibrated participant skill, then calibrated the debugging tasks, and finally evaluated standard and LLM-generated error messages in a controlled Prolific experiment.

3.1

Crowdsourced Calibration

3.1.1 Participant Skill Calibration. Because our research questions compare explanation effects across programming skill levels, we developed a short Python proficiency assessment focused on debuggingrelevant competencies. Self-reported experience alone may not reliably capture the skills required to interpret error messages and repair code. We therefore created a pool of 56 multiple-choice items spanning code comprehension, error-message interpretation, fault localization, and repair selection. In a pilot study, each participant answered 16 items sampled across these themes. We then selected eight items for the main experiment using item difficulty and pointbiserial discrimination.

All participant-facing rewrites were generated using llama-3.1-8BInstruct. The prompts included the code snippet, the standard interpreter message, and line-numbered context. Generation used zero-shot prompting with temperature set to 0 to improve consistency and reproducibility. Full prompt templates and generation details are provided in Appendix C. Figure 2 shows an example of the three message styles.

Participants in the main study were assigned to lower or higher assessed proficiency using both their score on the eight-item assessment and self-reported years of Python experience. We use the terms lower and higher assessed proficiency to avoid treating the categories as complete measures of programming expertise. Full details of the item pool, pilot procedure, pilot correlations, and final item statistics are provided in Appendix A. MCQ score

Python experience

Skill group

≥6 ≤3 [4, 5] [4, 5]

Any Any ≥5 <5

Expert Novice Expert Novice

Message Conditions

(a) Standard inter- (b) preter message. rewrite.

Pragmatic

(c) Contingent rewrite.

Figure 2: Example comparison of the standard Python interpreter message, pragmatic LLM rewrite, and contingent LLM rewrite.

3.3

Main Prolific Experiment

The main study was deployed through a custom web application integrated with Prolific for recruitment and compensation. Participants were required to be at least 18 years old, fluent in English, have prior Python experience, have a Prolific approval rate above 90%, and use a desktop or laptop computer. Participants from earlier pilot studies were excluded to avoid carryover effects.

Table 1: Logic for assigning participants to skill level categories based on MCQ score and years of Python experience (YoE).

After consent and instructions, participants completed the Python skill assessment and were assigned to one of the four calibrated debugging tasks and one of the three message styles. Assignment used constrained randomization to maintain balance across message styles and snippets. Each participant saw exactly one code snippet and one error-message style.

3.1.2 Debugging Task Calibration. The main experiment required debugging tasks that were realistic enough to support meaningful repair behavior, but constrained enough for a crowdsourced study. We generated eight candidate Python snippets targeting common error families, including NameError, TypeError, and SyntaxError. Each snippet was self-contained, used only the Python standard library, and contained one primary fault. We manually checked that

Participants attempted to repair the code in an embedded editor. A submission was counted as correct if it removed the targeted error and passed a hidden semantic test suite. Participants were 4

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

allowed up to three attempts. After an incorrect attempt, the editor reset to the original code to avoid cumulative code drift. After the debugging task, participants rated the message they received in terms of readability, cognitive load, and perceived tone.

rewrite. We report available age, gender, and nationality metadata descriptively in Appendix D, but do not conduct demographic subgroup analyses because they are outside the scope of our research questions.

The final dataset included, N = 103 valid participants after quality control. Allocation was balanced across message style and assessed proficiency: 35 participants were assigned to the standard condition, 35 to the pragmatic condition, and 33 to the contingent condition. The sample included 38 participants in the lower assessed proficiency group and 65 in the higher assessed proficiency group. Detailed recruitment information, platform details, interface screenshots, quality-control procedures, survey items, and allocation tables are provided in Appendix D.

We report results from the main Prolific experiment with 103 valid participants. Each participant attempted one calibrated Python debugging task and received one of three message styles: Standard, Pragmatic, or Contingent. We first report objective debugging outcomes addressing RQ1, followed by subjective message evaluations addressing RQ2.

3.4

4

4.1

Measures

Objective and Subjective Outcomes

The participants of our main study (N = 103) attempted to fix one of four buggy Python snippets (balanced across conditions) and were randomly assigned to one of three message styles: Standard (baseline), Pragmatic, or Contingent. We first report objective performance, such as fix rates, Fix@{1,2,3}, and time-to-fix in Section 4.1.1. We then report subjective perceptions, namely readability, perceived cognitive load, and perceived authoritativeness in Section 4.1.2. Finally, we also perform exploratory analyses that investigate additional potentially unexplored patterns in Section 4.1.3, in addition to the main results.

We measured both objective debugging outcomes and subjective message evaluations. Objective outcomes addressed RQ1 and included fix rate, number of attempts, Fix@k, and time-to-fix. Fix rate captured whether a participant successfully repaired the snippet within three attempts. Fix@k captured whether the participant first succeeded on attempt 1, 2, or 3. Time-to-fix measured elapsed time from task start until the first correct submission for participants who successfully repaired the snippet. Subjective outcomes addressed RQ2 and included perceived readability, cognitive load, and tone. Readability was measured with items covering length, jargon, sentence structure, and vocabulary. Cognitive load was measured through intrinsic, extraneous, and germane load items. Tone was measured with a single item assessing whether the message felt respectful and reader-centered. Full item wording is provided in Appendix D.

3.5

Results

As a reminder for interpreting the results in the following sections, participants were classified according to our assignment logic, which combined their performance on the skill-assessment survey with their self-reported Python YoE. Out of the 103 participants, 65 were categorized as experts and 38 as novices. The distribution of participants across skill levels and snippets is provided in Table 11. 4.1.1 Fix Rate, Fix@k, and Time-to-Fix. For measuring participants’ objective performance on the code fix task, across different subgroups we record the following metrics:

Participants

We recruited participants through Prolific. Eligibility criteria required participants to be at least 18 years old, fluent in English, have prior Python experience according to Prolific screening, have a Prolific approval rate above 90%, and use a desktop or laptop computer. Participants from earlier pilot studies were excluded to avoid carryover effects. The study was initially estimated to take 15–20 minutes; after a soft launch with 10 participants, the expected duration was updated to approximately 24 minutes. Final completion times averaged 26 minutes, with a median of 23 minutes. Participants were compensated at £9/hour.

• Fix rate: proportion of participants whose final submission is fully correct. • Fix@1/2/3: correctness achieved within the first, second, or third submission attempt. • Time-to-fix: total seconds from code fix task start until the first correct submission. Fix Rate: Table 2 reports overall fix rates by snippet across all participants, and Table 3 breaks them down by message style and skill level. Fix rate, and therefore difficulty, varies significantly across snippets, with snippet A showing the lowest success rate. This outcome is expected because the standard interpreter output for snippet A fails to reveal the true root cause, and actually points to the non-offending line3 .

A total of 103 participants met all quality-control criteria and were included in the final analysis. Quality control relied on logged browser focus events, copy/paste blocking with browser alerts, task completion times, and submission patterns. Triggered cases were manually reviewed, and participants showing random typing, repeated unchanged submissions, or clear low-effort behavior were excluded.

Looking more in-depth, Tables 4 and 5 report fix rates by message style and snippet ID for novices and experts. For snippet A, LLMenhanced PEMs, which do correctly identify the offending line and

The final sample included 38 participants in the lower assessed proficiency group and 65 in the higher assessed proficiency group. Participants were distributed across message-style conditions as follows: 35 received the Standard interpreter message, 35 received the Pragmatic LLM rewrite, and 33 received the Contingent LLM

3 Interested readers can refer to Listing 9 and Figure 14 in Appendix F, which present

the code and the corresponding standard error message for the least solved buggy code snippet in our study (when being shown the standard interpreter output). 5

Moraru et al.

root cause, generally increase fix rates, with the pragmatic style among novices as the main exception. However, across snippets, style effects vary widely, so we refrain from making any conclusions.

Fix Rate

A

B

C

D

53% (𝑛 = 26)

84% (𝑛 = 26)

64% (𝑛 = 25)

69% (𝑛 = 26)

general, than the baseline, since they outperform standard at 𝑘 = 1. For contingent messages in particular, outcomes often cluster at either “fixed on the first attempt” or “not at all within three attempts”. Therefore, interpreting low Fix@2 and Fix@3 counts requires care, since they can result from either strong first-attempt performance or a large tail of unresolved cases. Future studies should relax the bound on 𝑘 (within reason) to better visualize the average number of attempts per error-message style until successful code repair.

Table 2: Fix rate (%) by snippet ID, across all participants; 𝑛 = the total number of participants assigned the given snippet ID.

Standard

Pragmatic

Contingent

46% 68%

54% 82%

42% 90%

Novice (𝑛 = 38) Expert (𝑛 = 65)

Table 3: Fix rate (%) by message style and skill group (all snippets pooled).

Figure 3: Fix@k, for 𝑘 ∈ 1, 2, 3, when varying error message style across all snippets and skill levels.

Time-to-Fix: As for the time-to-fix metric, Figure 4 shows that, averaged across skill levels, participants fixed errors fastest when A B C D shown pragmatic messages. Contingent messages also reduced Standard 33% (𝑛 = 3) 67% (𝑛 = 3) 33% (𝑛 = 3) 50% (𝑛 = 4) time-to-fix relative to the baseline, but less than pragmatic, which Pragmatic 0% (𝑛 = 3) 100% (𝑛 = 4) 67% (𝑛 = 3) 33% (𝑛 = 3) is unsurprising given their greater verbosity and the extra reading Contingent 67% (𝑛 = 3) 67% (𝑛 = 3) 0% (𝑛 = 3) 33% (𝑛 = 3) time required. However, when split by skill level (see Figure 5), both LLM-enhanced PEMs tend to speed fixes versus standard. With few Table 4: Fix rate (%) by message style and snippet ID for novice novice successes (standard 𝑛=6, pragmatic 𝑛=7, contingent 𝑛=5), participants. The number 𝑛, in parentheses, represents the statistical tests found no significant effects, though the current total number of participants assigned the given combination pattern seems to favor LLM explanations. As for experts, results of snippet ID and error message style. remain inconclusive.

Standard Pragmatic Contingent

A

B

C

D

33% (𝑛 = 6) 83% (𝑛 = 6) 80% (𝑛 = 5)

80% (𝑛 = 5) 100% (𝑛 = 5) 83% (𝑛 = 6)

67% (𝑛 = 6) 80% (𝑛 = 5) 100% (𝑛 = 5)

100% (𝑛 = 5) 67% (𝑛 = 6) 100% (𝑛 = 5)

Table 5: Fix rate (%) by message style and snippet ID for expert participants. The number 𝑛, in parentheses, represents the total number of participants assigned the given combination of snippet ID and error message style.

Figure 4: Box plot for time-to-fix (seconds) by message style. Time-to-fix sums all attempts until the first success. Standard (𝑀 = 415.69, 𝑆𝐷 = 377.75); Pragmatic (𝑀 = 324.56, 𝑆𝐷 = 333.12); Contingent (𝑀 = 423.85, 𝑆𝐷 = 501.47).

Fix@k: We initially introduced the Fix@k metric to quantify how many attempts participants needed to repair each buggy snippet. Our aim was to test whether error message style influenced repeated submissions across the full sample and within novice and expert subgroups. In practice, results were situated at the extremes. Most participants either fixed the error on the first try, or did not succeed at all within the three allowed attempts. Because Fix@k showed limited variation (Figure 3), we could not conduct meaningful further analysis, so Fix@k remains largely unexplored in this study. A visual reading of the figure nevertheless suggests that pragmatic and contingent messages may yield higher fix rates, in

4.1.2 Subjective Evaluations. Figures 6, 7, 8, show mean ratings by error message style and skill level. For novices, Holm-adjusted Dunn tests indicate lower extraneous load for Pragmatic vs. Standard (𝑝 = 0.039) and also for Contingent vs. Standard (𝑝 = 0.054). The results also report lower authoritativeness for Pragmatic vs. Standard (𝑝 = 0.011), while vocabulary also shows a trend favoring Pragmatic vs. Standard (𝑝 = 0.087), meaning Pragmatic messages are generally seen as using simpler vocabulary. For experts, Pragmatic vs. Standard is associated with lower intrinsic (𝑝 = 0.0049) and higher germane load (𝑝 = 0.0043), and 6

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

(a) Time-to-fix box plot for (b) Time-to-fix box plot for novices, by message style. experts, by message style. Standard (𝑀 = 471.58, 𝑆𝐷 = Standard (𝑀 = 393.34, 𝑆𝐷 = 375.0); Pragmatic (𝑀 = 389.57); Pragmatic (𝑀 = 253.71, 𝑆𝐷 = 178.52); Contin- 352.13, 𝑆𝐷 = 377.56); Contingent (𝑀 = 396.41, 𝑆𝐷 = 437.06). gent (𝑀 = 431.08, 𝑆𝐷 = 527.85).

(a) Cognitive load metrics (Novices).

(b) Cognitive load metrics (Experts).

Figure 5: Box plots for time-to-fix (seconds) by message style and skill level; left plot for novices, and right plot for experts.

Figure 7: Mean ratings for cognitive load by message style and skill group. Brackets with asterisks indicate pairwise differences from Dunn’s post hoc tests (Holm-adjusted) following Kruskal-Wallis: * (𝑝 < 0.1), ** (𝑝 < 0.05), and *** (𝑝 < 0.01).

both Pragmatic and Contingent are rated less authoritative than the Standard ones (𝑝 = 0.0097 in both contrasts). Experts also perceive messages as more verbose for Contingent vs. Standard (𝑝 = 0.035), with a trend toward simpler vocabulary for Pragmatic vs. Standard (𝑝 = 0.088). Overall, adaptive style, especially Pragmatic, are consistently rated by experts as lighter in cognitive load, less authoritative, and more useful than the Standard baseline. Contingent error messages are also, on average, rated better by experts across almost all metrics, yet their increased verbosity seems to impact their overall usefulness.

(a) Authoritativeness metric (Novices).

(b) Authoritativeness metric (Experts).

Figure 8: Mean ratings for authoritativeness by message style and skill group. Brackets with asterisks indicate pairwise differences from Dunn’s post hoc tests (Holm-adjusted) following Kruskal-Wallis: * (𝑝 < 0.1), ** (𝑝 < 0.05), and *** (𝑝 < 0.01).

(a) Readability (Novices).

metrics

vs. Standard (𝑝 = 0.0064). We also reported differences in germane load, both for Pragmatic vs. Standard (𝑝 = 0.0013) and Contingent vs. Standard (𝑝 = 0.0488).

(b) Readability metrics (Experts).

Finally, in terms of authoritativeness, both Pragmatic and Contingent differ from the Standard error message style (𝑝 = 0.0001 and 𝑝 = 0.0032, respectively). Overall, the most frequent contrasts involve Pragmatic (and, in several cases, Contingent) relative to the Standard, consistent with adaptive styles differing from the baseline interpreter output on readability, perceived load, and tone.

Figure 6: Mean ratings for readability by message style and skill group. Brackets with asterisks indicate pairwise differences from Dunn’s post hoc tests (Holm-adjusted) following Kruskal-Wallis: * (𝑝 < 0.1), ** (𝑝 < 0.05), and *** (𝑝 < 0.01). Figure 9 reports mean ratings for the full sample, aggregated only by message style. Similarly, we also perform Kruskal-Wallis tests followed by Holm-adjusted Dunn comparisons where necessary, and the results indicate several pairwise differences.

For the previously mentioned statistical analysis, we use the KruskalWallis test with Dunn’s pairwise comparisons (Holm-adjusted) because our outcomes are Likert ratings (i.e., ordinal, bounded), and after performing validation checks we saw that the data was not normally distributed. Kruskal-Wallis is an alternative to ANOVA that handles such data well and tolerates unequal group sizes. When the omnibus test indicates a difference, Dunn’s test tells us which styles differ. Holm’s adjustment controls false positives across the three pairwise checks per metric while being less conservative (and more powerful) than an alternative like Bonferroni.

In terms of readability, differences appear for sentence structure between Pragmatic and Standard (𝑝 = 0.0218), for vocabulary between Pragmatic and Standard (𝑝 = 0.0063) and Pragmatic vs. Contingent (𝑝 = 0.0423), and a trend for length between Standard and Contingent (𝑝 = 0.0949). For cognitive load, we observe differences in intrinsic load for Pragmatic vs. Standard (𝑝 = 0.0017) and Contingent vs. Standard (𝑝 = 0.0431). In extraneous load we notice differences only for Pragmatic

4.1.3 Exploratory Analysis. In addition to the previous results, we also performed additional exploratory analysis to see whether we 7

Moraru et al.

Figure 9: Mean ratings for readability, cognitive load, and authoritativeness by message style. Brackets with asterisks indicate pairwise differences from Dunn’s post hoc tests (Holmadjusted) following Kruskal-Wallis: * (𝑝 < 0.1), ** (𝑝 < 0.05), and *** (𝑝 < 0.01).

Figure 11: Prolific participants’ self-reported Python YoE distribution.

could find uncovered patterns. For instance, Figure 10 depicts statistics about the multiple-choice questions that compose the Python skill assessment in the first part of the Prolific study, where we are reporting mean completion time and question correctness percentage. On average, we noticed that correct answers tend to require more time to select, indicating that questions were mostly answered thoughtfully and legitimately, rather than at random.

5.1

the role of skill-targeted explanation styles, and the limitations of static rewriting as a form of programming support.

The clearest finding from our study is that participants experienced LLM-rewritten messages as better than standard interpreter messages, but this did not produce reliable objective performance gains. Pragmatic messages were perceived as clearer, less cognitively demanding, and less authoritative than standard messages. Contingent messages also improved several subjective ratings, although less consistently and with signs of increased verbosity. In contrast, objective outcomes such as fix rate, time-to-fix, and Fix@k were more variable and did not provide strong evidence that rewritten messages reliably improved repair performance.

Another aspect we wanted to verify was whether the piloted Python skill assessment survey reliably reflects true proficiency and predicts debugging performance in the main Prolific study. To that end, we tested whether performance on the survey, measured as the number of MCQs answered correctly, relates to fixing the assigned code snippet. A point-biserial correlation showed a moderate positive association, 𝑟 𝑝𝑏 = 0.368 with 𝑝 = 0.0001, indicating that higher MCQ scores correspond to a greater likelihood of producing a correct fix.

(a) Mean completion/answering time in seconds for each MCQ in the Prolific study.

This gap is important for the design and evaluation of AI-generated programming support. A message that feels clearer may still fail to help the user locate the fault, identify the correct repair, or implement that repair successfully. Debugging requires more than interpreting the diagnostic message: users must connect the message to the program state, reason about the intended behavior, and modify the code correctly. LLM-generated explanations may reduce the friction of reading an error message without resolving these downstream reasoning and repair challenges. For human-AI systems, this suggests that perceived helpfulness should not be treated as a proxy for task effectiveness. Subjective evaluations remain important because they capture whether users find the system understandable, respectful, and worth engaging with. However, they need to be interpreted alongside behavioral measures. Our results, therefore, support a more cautious evaluation standard for LLM-generated explanations: fluency, readability, and user preference are necessary but insufficient indicators of effective support.

(b) Percentage of correct and incorrect answers for each MCQ in the Prolific study.

Figure 10: Statistics, such as mean completion time and correctness percentage pertaining to the MCQs in the Prolific study.

5

Perceived Explanation Quality Does Not Guarantee Better Debugging

Discussion

5.2

Our study examined whether LLM-rewritten Python error messages can support code repair and improve users’ perceptions of programming diagnostics. Overall, the results show a consistent divergence between subjective and objective outcomes. Participants rated LLM-generated messages, especially Pragmatic messages, more favorably than standard Python interpreter messages on readability, cognitive load, and tone. However, these subjective advantages did not reliably translate into improved debugging performance. This section discusses the implications of that divergence,

Skill-Targeted Rewriting Was Useful, But Not Sufficiently Adaptive

Our results also complicate the assumption that more scaffolded explanations are necessarily better for less experienced programmers. Contingent messages were designed to provide additional context and reasoning support, but they did not consistently benefit the lower assessed proficiency group. One explanation is that scaffolding can introduce additional reading and interpretation costs. For users still struggling to understand the code, a longer explanation 8

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

may increase cognitive effort even if it contains useful information. This interpretation is consistent with the broader concern that instructional support must be matched not only to a user’s general skill level, but also to their immediate task state.

such assessments against richer indicators, including course performance, instructor ratings, prior programming tasks, or longitudinal debugging behavior. Second, our debugging tasks were constrained by the requirements of a crowdsourced study. The four Python snippets were short, synthetic, and designed to contain a single primary fault. This made the study feasible and controlled, but it limits ecological validity. Real-world debugging often involves larger files, multiple interacting faults, project-specific dependencies, build systems, and incomplete contextual knowledge. Effects observed in short webbased tasks may therefore differ from effects in authentic IDE or classroom settings.

Pragmatic messages produced the most consistent subjective benefits. Their concise and action-oriented form may have helped participants extract a likely repair direction without requiring them to process a longer explanation. This suggests that, for short debugging tasks, reducing friction may sometimes be more valuable than adding pedagogical detail. However, pragmatic messages also did not produce reliable objective gains, which indicates that concise explanations alone are not enough to ensure successful repair.

Third, our system used static message styles generated from fixed prompt templates. We used llama-3.1-8B-Instruct with temperature 0 to support reproducibility and stable generation, but did not systematically vary model size, decoding settings, prompt structure, or error-specific templates. Different models or prompts may produce different tradeoffs between correctness, verbosity, and usefulness.

These findings clarify the limits of our current approach. We evaluate skill-targeted rewriting: explanation styles designed for different assessed proficiency groups. We do not evaluate a fully dynamic tutor that adapts to the user’s behavior during debugging. A participant’s assessed Python proficiency may not reflect their local understanding of a specific bug, their familiarity with the underlying concept, or their moment-to-moment repair strategy. Future systems may therefore need to adapt explanations based on interaction signals such as repeated failed attempts, time spent before editing, edit location, error recurrence, or requests for additional detail. Static skill categories are a useful starting point, but they are likely too coarse to support robust personalization.

5.3

Future work should test skill-targeted programming explanations in more ecologically valid settings. One direction is classroom deployment, where true novice programmers use rewritten error messages during programming assignments, and learning outcomes can be measured over time. Another direction is IDE-based evaluation, where explanations can be shown alongside standard diagnostics during authentic debugging workflows. Such studies could measure not only the fix rate and time-to-fix, but also edit trajectories, repeated error recurrence, hint requests, and delayed learning outcomes. Future systems should move beyond static skill-targeted rewriting toward interaction-aware support. Rather than assigning users to a fixed explanation style, systems could progressively disclose more detail when users struggle, offer concise summaries when users move quickly, or adapt explanations based on repeated failures and repair behavior. Such systems should also be evaluated for risks, including over-reliance, false confidence, and the possibility that fluent but incorrect explanations reinforce misconceptions. Our findings suggest that LLM-generated error messages are promising, but their value depends on careful human-centered evaluation and adaptive design.

Implications for HCOMP and AI Feedback Evaluation

Our study also illustrates the value of crowdsourced calibration for evaluating AI-generated programming feedback. Our evaluation did not begin with the final Prolific experiment alone. Human input was used to pilot a debugging-oriented proficiency assessment, calibrate candidate code snippets, and evaluate message perceptions after repair attempts. This multi-stage process helped reduce two common threats in crowdsourced programming studies: poorly measured participant skill and poorly calibrated task difficulty. At the same time, the results show that calibration cannot fully remove heterogeneity. Snippet difficulty varied substantially, and some message-style effects appeared sensitive to the specific error context. This suggests that future evaluations of AI-generated programming support should avoid treating “the error message” as a single uniform intervention. The same explanation style may help in one error context and not another. Evaluations should therefore report effects by error type, task difficulty, and user proficiency where possible, while being cautious about overinterpreting small subgroups.

5.4

6

Conclusion

This paper examined whether LLM-generated rewrites of Python interpreter error messages can improve debugging support for programmers with different assessed proficiency levels. We developed a multi-stage crowdsourced pipeline that combined a debuggingoriented Python proficiency assessment, calibrated buggy code snippets, and a controlled Prolific experiment comparing standard interpreter messages with two LLM-generated styles: Pragmatic, a concise and action-oriented rewrite, and Contingent, a more scaffolded explanation intended to provide additional reasoning support. Our findings show a clear divergence between subjective and objective outcomes. Participants rated LLM-rewritten messages, especially Pragmatic messages, as more readable, less cognitively demanding, and less authoritative than standard interpreter output. However, these perceptual benefits did not reliably translate

Limitations and Future Work

Our study has several limitations. First, our proficiency assessment was designed and piloted for this study, but it was not validated at large scale against external measures of Python expertise. The lower and higher assessed proficiency groups should therefore be interpreted as task-aligned operational categories rather than complete measures of programming expertise. Future work should validate 9

Moraru et al.

into statistically significant improvements in objective debugging outcomes such as fix rate, time-to-fix, or attempts. Contingent messages provided additional explanatory context, but their greater verbosity may have reduced their usefulness in short repair tasks, particularly for lower assessed proficiency participants.

[11] David R Krathwohl. 2002. A revision of Bloom’s taxonomy: An overview. Theory into practice 41, 4 (2002), 212–218. [12] John Lee, Fengkai Liu, and Tianyuan Cai. 2024. Code Debugging with LLMGenerated Explanations of Programming Error Messages. 1–5. https://doi.org/ 10.1109/ICEED62316.2024.10923833 [13] Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using Large Language Models to Enhance Programming Error Messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 563–569. https://doi.org/10.1145/3545945.3569770 [14] Katherine L McNeill and Dean M Martin. 2011. Claims, evidence, and reasoning. Science and Children 48, 8 (2011), 52. [15] Ike Obi, Jenna Butler, Sankeerti Haniyur, Brian Hassan, Margaret-Anne Storey, and Brendan Murphy. 2024. Identifying Factors Contributing to Bad Days for Software Developers: A Mixed Methods Study. arXiv preprint arXiv:2410.18379 (2024). [16] Michael Perscheid, Benjamin Siegmund, Marcel Taeumel, and Robert Hirschfeld. 2017. Studying the advancement in debugging practice of professional software developers. Software Quality Journal 25, 1 (March 2017), 83–110. https://doi. org/10.1007/s11219-015-9294-2 [17] Audrey Salmon, Katie Hammer, Eddie Antonio Santos, and Brett A. Becker. 2025. Debugging Without Error Messages: How LLM Prompting Strategy Affects Programming Error Explanation Effectiveness. arXiv:2501.05706 [cs.SE] https: //arxiv.org/abs/2501.05706 [18] Eddie Antonio Santos and Brett A. Becker. 2024. Not the Silver Bullet: LLMenhanced Programming Error Messages are Ineffective in Practice. In Proceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research (Manchester, United Kingdom) (UKICER ’24). Association for Computing Machinery, New York, NY, USA, Article 5, 7 pages. https://doi.org/10.1145/ 3689535.3689554 [19] Eddie Antonio Santos, Prajish Prasad, and Brett A. Becker. 2023. Always Provide Context: The Effects of Code Context on Programming Error Message Enhancement. In Proceedings of the ACM Conference on Global Computing Education Vol 1 (Hyderabad, India) (CompEd 2023). Association for Computing Machinery, New York, NY, USA, 147–153. https://doi.org/10.1145/3576882.3617909 [20] John Sweller. 2011. Cognitive load theory. In Psychology of learning and motivation. Vol. 55. Elsevier, 37–76. [21] Andrew Taylor, Alexandra Vassar, Jake Renzella, and Hammond Pearce. 2024. dcc –help: Transforming the Role of the Compiler by Generating Context-Aware Error Explanations with Large Language Models. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Portland, OR, USA) (SIGCSE 2024). Association for Computing Machinery, New York, NY, USA, 1314–1320. https://doi.org/10.1145/3626252.3630822 [22] Emillie Thiselton and Christoph Treude. 2019. Enhancing Python Compiler Error Messages via Stack Overflow. arXiv:1906.11456 [cs.SE] https://arxiv.org/abs/ 1906.11456 [23] Sierra Wang, John Mitchell, and Chris Piech. 2024. A Large Scale RCT on Effective Error Messages in CS1. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Portland, OR, USA) (SIGCSE 2024). Association for Computing Machinery, New York, NY, USA, 1395–1401. https://doi.org/10. 1145/3626252.3630764 [24] Patricia Widjojo and Christoph Treude. 2023. Addressing Compiler Errors: Stack Overflow or Large Language Models? arXiv:2307.10793 [cs.SE] https: //arxiv.org/abs/2307.10793 [25] David Wood, Jerome S Bruner, and Gail Ross. 1976. The role of tutoring in problem solving. Journal of child psychology and psychiatry 17, 2 (1976), 89–100. [26] Stephanie Yang, Miles Baird, Eleanor O’Rourke, Karen Brennan, and Bertrand Schneider. 2024. Decoding Debugging Instruction: A Systematic Literature Review of Debugging Interventions. ACM Trans. Comput. Educ. 24, 4, Article 45 (Nov. 2024), 44 pages. https://doi.org/10.1145/3690652

These results suggest that LLM-generated programming explanations should not be evaluated only by fluency, readability, or user preference. Explanations that feel clearer may still fail to improve repair behavior. For human-AI programming support, this points to the need for evaluation methods that combine subjective perception, behavioral outcomes, task calibration, and user proficiency assessment. More broadly, our study suggests that static skill-targeted rewriting is a useful starting point but may be insufficient for robust debugging support. Future systems may need to adapt explanations dynamically to the user’s interaction state, repair trajectory, and error context.

Acknowledgements The authors of this paper used large language models to rephrase certain sentences and to assist in drafting some parts of the programming work, such as components of the experimental web application interface, data analysis utilities, and scripts for generating figures. All system outputs were carefully reviewed and validated by the authors before usage.

References [1] Abdulaziz Alaboudi and Thomas D. LaToza. 2023. What constitutes debugging? An exploratory study of debugging episodes. Empirical Softw. Engg. 28, 5 (Sept. 2023), 34 pages. https://doi.org/10.1007/s10664-023-10352-5 [2] Patricia Armstrong. 2010. Bloom’s taxonomy. Vanderbilt University Center for Teaching 12, 05 (2010), 2023. [3] Titus Barik, Denae Ford, Emerson Murphy-Hill, and Chris Parnin. 2018. How should compilers explain problems to developers?. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Lake Buena Vista, FL, USA) (ESEC/FSE 2018). Association for Computing Machinery, New York, NY, USA, 633–643. https://doi.org/10.1145/3236024.3236040 [4] Brett A. Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis J. Bouvier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, Peter-Michael Osera, Janice L. Pearce, and James Prather. 2019. Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research. In Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education (Aberdeen, Scotland Uk) (ITiCSEWGR ’19). Association for Computing Machinery, New York, NY, USA, 177–210. https://doi.org/10.1145/3344429.3372508 [5] Tessa Charles and Carl Gwilliam. 2023. The Effect of Automated Error Message Feedback on Undergraduate Physics Students Learning Python: Reducing Anxiety and Building Confidence. Journal for STEM Education Research 6, 2 (Aug. 2023), 326–357. https://doi.org/10.1007/s41979-022-00084-4 [6] Robert Dyer and Jigyasa Chauhan. 2022. An exploratory study on the predominant programming paradigms in Python code. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore) (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 684–695. https: //doi.org/10.1145/3540250.3549158 [7] Anja Hawlitschek, Veit Köppen, André Dietrich, and Sebastian Zug. 2019. Dropout in programming courses – prediction and prevention. Journal of Applied Research in Higher Education ahead-of-print (07 2019). https://doi.org/10.1108/ JARHE-02-2019-0035 [8] Paul Hudak. 1989. Conception, evolution, and application of functional programming languages. ACM Comput. Surv. 21, 3 (Sept. 1989), 359–411. https: //doi.org/10.1145/72551.72554 [9] John Hughes. 1989. Why functional programming matters. The computer journal 32, 2 (1989), 98–107. [10] Slava Kalyuga. 2007. Expertise reversal effect and its implications for learnertailored instruction. Educational psychology review 19, 4 (2007), 509–539. 10

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

A

Python Proficiency Assessment

self-reported experience, self-assessed skill, self-estimated accuracy, and actual assessment performance.

This appendix provides additional details on the Python proficiency assessment used to assign participants to lower and higher assessed proficiency groups in the main study. The assessment was designed to capture debugging-relevant competencies rather than general programming identity or self-confidence.

A.1

Assessment Themes

We created an initial pool of 56 multiple-choice items, with seven items across eight themes. These themes were selected to cover competencies involved in interpreting error messages, understanding code, locating faults, and selecting repairs.

• Python-Specific Error Understanding: Items focused on Python syntax, semantics, indentation, keywords, and language-specific constructs. • Code Reading and Understanding: Items assessing participants’ ability to interpret Python code and reason about its behavior.

Corr.

𝑝 -value

General programming YoE Dreyfus level Self-estimated accuracy Actual accuracy Self-estimated accuracy Actual accuracy

0.55P 0.52S 0.24S 0.28P 0.24S 0.06P

0.00001 0.00003 0.00655 0.01527 0.00655 0.62229

Self-estimated accuracy

0.03S

0.82473

Self-estimated accuracy

0.35S

0.00671

A.4

Final Item Selection

To select items for the main study, we calculated item difficulty and point-biserial discrimination. Difficulty is the proportion of participants who answered an item correctly; values closer to 1 indicate easier items. Point-biserial discrimination captures how well an item distinguishes higher-scoring from lower-scoring participants.

• Error Message Comprehension: Items assessing the ability to interpret exception types, traceback information, and offending lines. • Error Identification: Items assessing the ability to locate faults in Python code.

We used Classical Test Theory rather than Item Response Theory because the pilot sample was too small for stable item-response estimates. The final eight selected items are shown in Table 7.

• Error Resolution: Items assessing the ability to choose an appropriate repair after a fault and cause are known. • Natural Language Scenarios: Items requiring participants to infer likely causes of defects from natural-language problem descriptions.

Question Q4.6 Q5.2 Q6.5 Q7.1 Q7.2 Q8.2 Q9.5 Q10.1

• Miscellaneous: More complex items combining multiple categories or approximating compact real-world debugging scenarios. In the pilot study, each participant answered 16 items: two randomly sampled items from each theme. Theme order, item order, and answer-option order were randomized.

Difficulty

Discrimination (𝑟 𝑝𝑏 )

𝑝 -value

0.82 0.83 0.84 0.88 0.56 0.78 0.76 0.87

0.50 0.65 0.60 0.81 0.51 0.40 0.30 0.45

0.04224 0.00351 0.00631 0.00008 0.04598 0.10440 0.24669 0.08909

Table 7: Difficulty, discriminative power, and 𝑝-value for the final selected items of the Python proficiency assessment.

Pilot Procedure Although Q8.2, Q9.5, and Q10.1 did not reach conventional significance thresholds, they were retained because they represented otherwise under-covered themes and had acceptable difficulty and positive discriminative values.

The proficiency-assessment pilot was implemented in Qualtrics and distributed through convenience sampling. Participants reported their self-assessed Python proficiency, years of experience with Python, years of general programming experience, and their estimate of how many assessment items they had answered correctly.

A.5

A total of 78 participants started the survey and 60 completed it, yielding a completion rate of 76.92

A.3

Variable B

Python YoE Python YoE Python YoE Python YoE Dreyfus level General programming YoE General programming YoE Actual accuracy

Table 6: Correlations examined in the Python proficiencyassessment pilot. P denotes Pearson correlation and S denotes Spearman correlation. Python YoE = self-reported years of experience with Python; General Programming YoE = selfreported years of general programming experience; Dreyfus level = ordinal self-rating of Python experience; accuracy = number of questions answered correctly on a 0–16 scale.

• General Programming Error Understanding: Items assessing general knowledge about programming errors independent of any specific language.

A.2

Variable A

Skill Grouping Logic

In the main study, participants were assigned to lower or higher assessed proficiency using both MCQ score and self-reported Python years of experience. We use the terms lower and higher assessed proficiency rather than novice and expert because the grouping is task-aligned and operational, not a complete measure of programming expertise.

Pilot Correlations

Table 6 reports the correlations examined in the pilot. We use these correlations descriptively to understand the relationship among 11

Moraru et al.

MCQ score

Python experience

Skill group

≥6 ≤3 [4, 5] [4, 5]

Any Any ≥5 <5

Expert Novice Expert Novice

Construct assessed Code difficulty Fix difficulty

“This code snippet is difficult to understand.” “I would find it challenging to resolve the issue in this code snippet.” Error mental de- “Reading this error message feels mentally mand demanding.” Error usefulness “This error message is useful for identifying the problem.”

Table 8: Skill grouping logic used in the main study.

B

Debugging Task Selection

Table 9: Mapping between the pilot study’s survey dimensions and their corresponding items.

This appendix provides additional detail on how the debugging tasks were generated, screened, piloted, and selected for the main study.

B.1

Survey item

Candidate Generation

• variation in error type and fault structure;

We generated eight candidate buggy Python snippets targeting common error families, including NameError, TypeError, and SyntaxError. Each candidate was required to satisfy the following constraints:

• interpretable standard interpreter message;

• Self-contained: The program could be run as a standalone Python script.

• compatibility with hidden semantic tests.

B.4

• Standard library only: The snippet did not require external packages.

B.4.1 Snippet A. % Insert Snippet A code here .

• Moderate structure: The snippet included several functions or class methods.

1

Listing 1: Selected debugging task: Snippet A.

• Single primary fault: The snippet was designed to trigger one primary error.

% Insert standard interpreter error message for Snippet A here .

• Hidden-test compatible: The intended repair could be evaluated using a semantic test suite.

Listing 2: Standard Python interpreter message for Snippet A. B.4.2 Snippet B. import random

Each candidate was manually checked to ensure that it triggered the intended error and did not contain multiple simultaneous faults. When candidates violated these constraints, they were corrected or regenerated.

class UserData : " " " Represents user data with a name and a list of scores . " " " ``` def __init__ ( self , name , scores ) : self . name = name self . scores = scores

Task-Selection Pilot

Participants in the task-selection pilot rated each candidate snippet and its standard Python interpreter message on four dimensions: code difficulty, fix difficulty, error-message mental demand, and error-message usefulness. Table 9 lists the exact survey items.

def top_score ( self ) : " " " Returns the highest score . " " " return maximum ( self . scores ) if self . scores else 0 def add_score ( self , score ) : self . scores . append ( score )

The task-selection pilot included 20 valid responses after excluding one participant who failed an attention check.

B.3

Selected Snippets and Standard Error Messages

The selected snippets are provided below. Each snippet was shown to participants together with one assigned error-message style in the main study.

• Moderate length: The snippet was limited to at most 60 lines.

B.2

• absence of multiple primary faults;

```

def summarize_scores ( users ) : return { u . name : u . top_score () for u in users }

Selection Criteria

if ** name ** == ' ** main ** ': " " " Main routine to generate user data , summarize scores , and print the results . " " " users = [ UserData ( f " user_ { i + 1} " , [ random . randint (0 , 100) for _ in range ( random . randint (2 , 5) ) ]) for i in range (4) ] summary = summarize_scores ( users ) for name , score in summary . items () : print ( f " { name }: { score :.2 f } " )

The final four snippets were selected to provide coverage across error types while remaining appropriate for the target participant population. We prioritized candidates that met the following criteria: • moderate perceived code difficulty;

Listing 3: Selected debugging task: Snippet B.

• moderate perceived fix difficulty; 12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

25 26 27

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

C.3

% Insert standard interpreter error message for Snippet B here .

Pragmatic prompt template

Listing 4: Standard Python interpreter message for Snippet B. B.4.3 Snippet C. % Insert Snippet C code here .

INSTRUCTION: You are an assistant helping a Python programmer by explaining the error, in a clear and concise way.

1

Listing 5: Selected debugging task: Snippet C.

CONTEXT: The programmer's code and error message are below: --------------------------------------------------

% Insert standard interpreter error message for Snippet C here .

CODE:

Listing 6: Standard Python interpreter message for Snippet C. B.4.4 Snippet D. % Insert Snippet D code here .

```python {code} ``` ERROR MESSAGE:

1

``` {error} ```

Listing 7: Selected debugging task: Snippet D. % Insert standard interpreter error message for Snippet D here .

--TASK:

Listing 8: Standard Python interpreter message for Snippet D.

C

1. Identify the cause of the error and the relevant line number from the code snippet. DO NOT FETCH THE LINE NUMBER FROM THE ERROR MESSAGE. 2. Write exactly one paragraph (around 20-25 words or less) that:

LLM Prompt Templates and Generation Settings

* Begins with "**<ExceptionType>** at **line <line>**:" * Briefly states the cause and hints at a fix. * Focuses on providing helpful actionable insights, without directly giving the corrected code.

This appendix provides the full prompt templates and generation settings used to create the LLM-generated programming error messages.

C.1

FORMAT: You may use markdown for emphasis, but do NOT include lists or code fences.

Generation Model

All participant-facing rewritten messages were generated using llama-3.1-8B-Instruct. The model was selected because it could be self-hosted, followed instructions reliably in pilot testing, supported sufficient context for the selected snippets and tracebacks, and allowed deterministic generation. The candidate buggy snippets used during task selection were generated separately using GPT-4o-mini-high; however, all messages shown to participants in the main study were generated using llama-3.1-8B-Instruct.

C.2

Pragmatic Prompt Template

NOW WRITE YOUR RESPONSE:

C.4

Contingent Prompt Template

Contingent prompt template INSTRUCTION: Guide the Python programmer to resolve their error in 3-5 supportive sentences. Focus on actionable steps and encouragement, while avoiding an authoritative tone.

Generation Settings

We used zero-shot prompting and set temperature to 0. This setting was chosen to improve consistency and reproducibility across generated messages. In pilot testing, one-shot and few-shot prompts biased outputs toward the examples and reduced generality.

CONTEXT: The programmer's code and error message are below: -------------------------------------------------CODE:

Each prompt included:

```python {code} ```

• the full code snippet; • the standard Python interpreter error message;

ERROR MESSAGE:

• line-numbered code context;

``` {error} ```

• instructions to identify the cause and relevant line from the code, rather than copying the traceback line number alone. 13

Moraru et al.

(3) Debugging task: Participants received one calibrated buggy Python snippet and one assigned error-message style.

---

(4) Repair attempts: Participants had up to three attempts to fix the code. After each failed attempt, the editor reset to the original code.

TASK: 1. Identify the cause of the error and the relevant line number from the code snippet. DO NOT FETCH THE LINE NUMBER FROM THE ERROR MESSAGE. 2. Then write 3-5 sentences that:

(5) Post-task survey: Participants rated the error message they received on readability, cognitive load, and tone.

* Begin your response with "**<ExceptionType>** at **line <line>**:" on the first line. * Then confirm the likely intent or goal (e.g. "Did you mean to ...?") - this is the claim, showing you understand what the programmer was trying to do. * Only if useful, mention whether this error is common or if there are exceptions. * Only if relevant, note whether there are situations where the error might occur differently.

D.4

Participants were assigned to one of three message styles: standard, pragmatic, or contingent. Assignment was constrained to maintain balance across message styles and snippets. Each participant saw exactly one snippet and one message style.

D.5

NOW WRITE YOUR RESPONSE:

Main Study Details

This appendix provides additional details on recruitment, study workflow, interface design, quality control, and survey items for the main Prolific experiment.

D.1

D.6

Recruitment and Eligibility

Objective outcomes. Objective outcomes included: • Fix rate: whether the participant successfully repaired the code within three attempts. • Fix@k: whether the participant first succeeded on attempt 1, 2, or 3.

Platform

• Time-to-fix: elapsed time from task start until the first correct submission, for participants who successfully repaired the snippet.

The study was deployed through a custom web application. Prolific was used for recruitment, eligibility screening, and compensation management; the experimental tasks were administered on our platform. Participants arrived from Prolific with a unique Prolific ID in the URL, allowing us to enforce one completion per participant and align task records.

• Number of attempts: the number of submissions made before success or task termination. Subjective outcomes. After the debugging task, participants rated the message they received using 5-point Likert items covering readability, cognitive load, and perceived tone. The full items are shown in Table 10.

The application used a Next.js/React frontend, a FastAPI backend, and PostgreSQL for data storage. The debugging task was presented in an embedded code editor with Python syntax highlighting and non-AI rule-based inline completion. The system executed submissions against hidden semantic tests to determine whether the targeted bug had been repaired.

D.3

Outcome Measures

The main study measured both objective debugging outcomes and subjective message evaluations.

Participants were recruited through Prolific. Eligibility criteria required participants to be at least 18 years old, fluent in English, have prior Python experience according to the Prolific screener, have an approval rate above 90

D.2

Quality Control

The application logged integrity and quality-control signals, including copy, paste, cut, browser focus changes, tab switching, completion time, and repeated submissions. Participants showing clear evidence of low-effort behavior, random typing, repeated unchanged submissions, or likely external assistance were manually reviewed and excluded according to the study criteria disclosed to participants.

FORMAT: You may use markdown for emphasis, but do NOT include lists or code fences.

D

Assignment and Balancing

D.7

Participant Allocation

The final dataset included 103 valid participants after quality control. Table 11 shows allocation by message style and assessed proficiency group. Table 12 shows allocation by snippet and message style.

Participant Workflow

After providing consent, participants completed the following stages: (1) Instructions and sandbox: Participants reviewed task instructions and interacted with a sandbox version of the interface. (2) Skill assessment: Participants reported years of Python experience and answered the eight selected MCQ items. 14

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

Metric

Question

Scale (1-5)

“Is the message expressed using more words than needed?” “Does the message contain jargon and technical terms?” “How clear is the sentence structure of the message?” “How complex is the vocabulary used?”

Succinct Verbose

“This error message was inherently difficult to understand.” “The wording or formatting of this error message wasted mental effort.” “This error message helped me recognize the underlying error concept.”

Disagree - Agree

Readability Length

Jargon Sentence Structure Vocabulary

Less - More Clear - Unclear

(a) Participants’ nationality dis- (b) Participants’ gender distributribution. tion.

Simple - Complex

Cognitive load Intrinsic Load

Extraneous Load Germane Load

Disagree - Agree (c) Participants’ age distribution.

Figure 12: Descriptive statistics for Prolific participants’ demographics.

Disagree - Agree

E

Authoritativeness Tone (respectfulness)

Disrespectful Respectful

“How respectful (i.e., reader-centered) is the tone of the error message?”

This appendix collects material that was used for, or is directly relevant to, the Python Skill Level Assessment that informed our pilot study, which we launched on Qualtrics4 . It brings together the elements that shaped the assessment and the specific items that we later relied on in the main study5 .

Table 10: Survey items for readability, cognitive load, and authoritativeness, with response scales.

Standard

Pragmatic

Contingent

Total

Novice Expert

13 22

13 22

12 21

38 63

Total

35

35

33

103

Table 13 presents the self rating question for Python experience that we used in the pilot study. The question uses six levels that are informed by the Dreyfus model of skill acquisition. The table shows the exact wording of the question and the response options so that the reader can see how participants reported their level of experience. Figure 13 shows the eight multiple choice questions that we selected from an initial pool of 56 items6 . These eight items were identified as the most capable of distinguishing novices from experts in the context of debugging and understanding error messages. As explained in Section 3.1.1, this set of eight items was then carried forward and used in the main Prolific study.

Table 11: Participant allocation by message style and skill level.

A

B

C

D

Total

Standard Pragmatic Contingent

9 9 8

8 9 9

9 8 8

9 9 8

35 35 33

Total

26

26

25

26

103

Python Skill Level Assessment Qualtrics Questionnaire and Details

4 https://www.qualtrics.com/

Table 12: Participant allocation by snippet (A-D) within each message style.

5 Readers ought to remember that within the main contents of this paper, Section 3.1.1

describes the motivation for creating the assessment and the approach we followed during the pilot phase. It explains why a dedicated instrument was needed and how we constructed and refined it. 6 For brevity, we do not include all 56 pilot questions here. The full set, along with additional plots, and the analysis scripts and notebooks used to select the final eight, is available in our supplementary material. 15

Moraru et al.

Numerical Level 1 2

3

4

5

6

Experience level statement I have no previous experience in Python. Focusing on syntax & basics, relying on tutorials, lacking real-world project experience. Practical experience, small projects or assignments. Grasping basic concepts, and troubleshooting independently. Experience with projects. Able to independently plan and execute tasks, proficient with Python’s libraries. Experience in complex projects. Deep understanding, ability to consciously resolve difficult problems Significant experience in larger complex programs. Solves complex problems effortlessly and subconsciously

(a) Question ID Q4.6 of (b) Question ID Q5.2 of the Python Qualtrics Skill- the Python Qualtrics SkillAssessment Survey. Assessment Survey.

(c) Question ID Q6.5 of (d) Question ID Q7.1 of the Python Qualtrics Skill- the Python Qualtrics SkillAssessment Survey. Assessment Survey.

Table 13: Six-level self-rating of Python experience used in the study, informed by the Dreyfus skill-acquisition model.

F

(e) Question ID Q7.2 of (f) Question ID Q8.2 of the Python Qualtrics Skill- the Python Qualtrics SkillAssessment Survey. Assessment Survey.

Fantastic Four Code Snippets and Error Messages

This appendix presents the four buggy Python snippets used in the Prolific study, with their standard interpreter errors. Sections F.1, F.2, F.3, F.4 show each code snippet followed by its error message.

F.1

Snippet A: SyntaxError: unterminated string literal

class BookShelf : def __init__ ( self , log ) : self . log = log def preview ( self ) : """ Shows first two book titles , if any . """ if not os . path . exists ( self . log ) : return " No log found . " with open ( self . log ) as f : lines = f . readlines () preview = " " . join ( lines [:2]) return f" " " Preview :\ n { preview } " " def summary ( self ) : """ Gives a summary of the log . """ total = count_books ( self . log ) return f" Books logged : { total } " def add_book ( log , title ) : """ Adds a book entry with timestamp . """ with open ( log , 'a ') as f : f. write (f " { datetime . now () . isoformat () } - { title }\ n" ) def count_books ( log ) : """ Counts books in the log . """ if not os . path . exists ( log ) : return 0 with open ( log ) as f : return sum (1 for _ in f )

(g) Question ID Q9.5 of (h) Question ID Q10.1 of the Python Qualtrics Skill- the Python Qualtrics SkillAssessment Survey. Assessment Survey. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

16

Figure 13: Final set of survey items that most effectively discriminated between novices and experts in the Python Skill-Assessment Survey. The images reproduce the question options and interface, as presented in the Prolific web application during the main study.

Listing 9: One of the chosen four buggy Python code snippet, which when executed as a standalone file, will trigger a SyntaxError, due to the unclosed triple-quoted string literal on line 17.

Figure 14: Standard Python interpreter message displayed when running the code in Listing 9.

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

F.2

Snippet B: NameError: name not defined

import random class UserData : """ Represents user data with a name and a list of scores ." " " def __init__ ( self , name , scores ) : self . name = name self . scores = scores def top_score ( self ) : """ Returns the highest score . " " " return maximum ( self . scores ) if self . scores else 0 def add_score ( self , score ) : self . scores . append ( score ) def summarize_scores ( users ) : return {u. name : u . top_score () for u in users } if __name__ == ' __main__ ': """ Main routine to generate user data , summarize scores , and print the results . " " " users = [ UserData ( f " user_ { i + 1} " , [ random . randint (0 , 100) for _ in range ( random . randint (2 , 5) ) ]) for i in range (4) ] summary = summarize_scores ( users ) for name , score in summary . items () : print (f"{ name }: { score :.2 f } " )

F.3 1 2 3 4 5

Snippet C: TypeError: unsupported operand types

import random

6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

26 27 28

Listing 10: One of the chosen four buggy Python code snippet, which when executed as a standalone file, will trigger a NameError on line 13 of the code listing, due to the undefined variable maximum.

def generate_scores ( n ) : " " " Generate n random test scores . " " " return [ random . randint (0 , 100) for _ in range (n)] def average ( scores ) : " " " Compute the average score . " " " if not scores : return 0 return sum ( scores ) / len ( scores ) def filter_passing ( scores , threshold =60) : " " " Return scores that are passing . " " " return [ s for s in scores if s >= threshold ] class ScoreReport : def __init__ ( self , scores ) : self . scores = scores @classmethod @staticmethod def describe () : " " " Describe the scoring system . " " " return " Scores range from 0 to 100. " def passing_percentage ( self ) : " " " Return the percentage of passing scores .""" passing = filter_passing ( self . scores ) return 100 * len ( passing ) / len ( self . scores ) if self . scores else 0 def report ( self ) : " " " Return a formatted report . " " " avg = average ( self . scores ) pct = self . passing_percentage () desc = self . describe () return f " { desc }\ nAverage : { avg :.1 f }\ nPassing : { pct :.1 f }% " def main () : scores = generate_scores (12) report = ScoreReport ( scores ) print ( report . report () ) if __name__ == " __main__ " : main ()

Listing 11: One of the chosen four buggy Python code snippet, which when executed as a standalone file, will trigger a TypeError on line 21 of the code listing, due to adding both @classmethod and @staticmethod on the same line.

Figure 15: Standard Python interpreter message displayed when running the code in Listing 10.

Figure 16: Standard Python interpreter message displayed when running the code in Listing 11.

17

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

Moraru et al.

F.4

Snippet D: TypeError: object is not subscriptable

import math def normalize ( vec ) : norm = math . sqrt ( sum ( x ** 2 for x in vec ) ) return [x / norm for x in vec ] if norm else vec def dot (a , b): return sum (x * y for x , y in zip (a , b ) ) def cosine (a , b): if len (a) != len ( b ) : raise ValueError ( " Vectors must be of the same length " ) return dot ( normalize ( a ) , normalize ( b ) ) def fixed_vectors () : """ Returns a fixed set of vectors for testing purposes . " " " return [ [1.0 , 2.0 , 3.0] , [2.0 , 0.0 , 1.0] , [ -1.0 , 1.0 , 0.0] , [0.5 , -2.0 , 2.0] ] def most_similar_pair ( vectors ) : """ Finds the most similar pair of vectors based on cosine similarity . " " " max_sim = -2 pair = (0 , 1) for i in range ( len ( vectors ) ) : for j in range ( i + 1 , len ( vectors ) ) : sim = cosine ( vectors [ i ] , vectors [ j ]) if sim > max_sim : max_sim = sim pair = (i , j ) return pair def main () : vs = fixed_vectors () print (" Most similar pair : " , most_similar_pair ( vs ) ) for i in range ( len ( vs ) ) : print (" Vector " , i , " : " , vs . __getitem__ [ i ]) if __name__ == " __main__ " : main ()

G 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

Listing 12: One of the chosen four buggy Python code snippet, which when executed as a standalone file, will trigger a TypeError on line 39 of the code listing, due to using the wrong syntax for accessing a list’s elements.

Prolific Full-Stack Web Application Design Details

This appendix describes the full-stack platform we deployed on Prolific for the main study and explains the key design decisions behind it. We coordinated services with Docker Compose7 to keep components modular and easy to start, stop, and update. Containerization also allowed us to deploy everything on a Linux virtual machine with only the docker binary installed. This reduced configuration drift over time as we developed the application, and improved reproducibility, which was an important requirement for our study.

G.1

System Architecture

The platform consists of a web frontend, a Python backend, a relational database, a local LLM inference service, and a reverse proxy. Figure 18 shows the high-level architecture and how these components interact within a containerized environment.

Figure 18: System architecture of the Prolific study full-stack application: containerized frontend, backend, database, local LLM inference, and reverse proxy orchestrated with Docker Compose. The frontend was implemented with Next.js, React, Typescript, and TailwindCSS. We chose this stack mostly due to familiarity and because it was straightforward to build reusable components, such as forms, multi-step submission flows, and interactive elements for task execution and feedback collection. The backend was built with FastAPI8 in Python. FastAPI offers strong type support, automatic API documentation, and efficient asynchronous I/O. These features suited our needs for handling user-provided code execution and returning LLM-rephrased error messages. Keeping the backend in Python also simplified integration with our evaluation logic and the LLM client used for inference. We stored participant data, submissions, feedback, and event logs in PostgreSQL. PostgreSQL provides mature tooling, transactional guarantees, and expressive queries. A relational schema with ACID properties helped maintain integrity across multi-step tasks, for example when linking code submissions to feedback and timing events. Given our relatively modest scale, any other relational or non-relational database would have been equally suitable for our requirements, yet PostgreSQL aligned best with our planned analysis workflow.

Figure 17: Standard Python interpreter message displayed when running the code in Listing 12.

Local LLM inference was provided by Ollama. Running inference on a self-hosted service gave us control over data handling and latency while not incurring additional costs by using an external API, such 7 https://docs.docker.com/compose/ 8 https://fastapi.tiangolo.com/

18

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

as OpenAI’s API. The service loaded the model used for errormessage rephrasing. Our application can technically support any other open-source models available on Ollama within the available memory and latency budget. However, as described in Section 3.2, we selected llama-3.1-8B-Instruct.

secure while allowing internal services to restart or scale independently. In practice, load was very low and no scaling actions were even required.

Finally, Nginx9 served as the reverse proxy. It routed requests to the frontend and backend and cached static assets to improve responsiveness during data collection. It also restricted public access to a single entry point, which simplified networking, while reducing exposure and risks for seeing malicious activity.

Runtime configuration was provided through environment variables. Some of the settings included database credentials and the database URL for the backend, the model identifier for the LLM service, and the frontend host URL used for API requests. For our research deployment, however, we used a set of conventional defaults, as we did not need to deploy on multiple environments with different settings.

G.2

G.3

Deployment and Networking

All services ran as containers managed by Docker Compose. Only Nginx was exposed to the outside network on port 80. The backend, database, and LLM service communicated over an internal Docker network, which reduced the attack surface and allowed services to address each other by name. The frontend interacted with the backend through proxied API routes, and static assets benefited from Nginx caching. This layout kept the public interface relatively

Monitoring and Inspection

During development and data collection we used a small Python utility, to query selected tables and verify that submissions, feedback, and events were recorded as expected. The script also supported troubleshooting ingestion issues and exporting snapshots of collected data. These checkpoints helped us monitor progress, assess data quality, and decide whether to recruit additional participants when attrition or exclusion, due to malicious behavior, affected our stratified sampling procedure. 9 https://nginx.org/

19

Related documents

• Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors#662088
• Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors#661038
• Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision#2703
• If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs#997036
• CoDe-R: Refining Decompiler Output with LLMs via Rationale Guidance and Adaptive Inference#13009