When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

softwarearchitecture softwareengineering testing

software engineering, software architecture, testing

When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

arXiv:2607.05068v1 [cs.SE] 6 Jul 2026

Victor-Alexandru Pădurean

Kaitlin Riegel

MPI-SWS Saarbrücken, Germany [email protected]

University of Auckland Auckland, New Zealand [email protected]

Jyotika Mahapatra

Ahana Ghosh

MPI-SWS Saarbrücken, Germany [email protected]

Juho Leinonen

Aalto University Espoo, Finland [email protected]

Alkis Gotovos

MPI-SWS Saarbrücken, Germany [email protected]

Paul Denny

MPI-SWS Saarbrücken, Germany [email protected]

University of Auckland Auckland, New Zealand [email protected]

James Prather

Adish Singla

Abilene Christian University Abilene, TX, USA [email protected]

MPI-SWS Saarbrücken, Germany [email protected]

Abstract

CCS Concepts

As Generative AI (GenAI) becomes increasingly central to software development, CS education is integrating prompt-centered workflows where students describe intended program behavior in natural language to elicit code. However, professional practice requires careful review and verification of GenAI-generated code that may appear correct while containing subtle faults. This creates a challenge for CS1-level activities, where current models often solve tasks correctly and reduce students’ incentive to closely inspect generated outputs. We investigate how prompt-centered programming activities can be adapted to better foster these practices. Specifically, we explore an approach where realistic, runnable bugs are injected into otherwise correct solutions, thus requiring students to read and repair generated outputs. We analyzed 2,636 sessions from 917 students, and examined behavior across instances of naturally occurring prompt-related failures and deliberately injected bugs within each session. Our findings show that students responded differently across bug sources. Deliberately injected bugs more often led to direct code edits and higher next-attempt success, suggesting localized repair of near-miss solutions. Prompt-related failures instead more often led students to refine prompts by clarifying constraints, updating function signatures, adding edge cases, or reframing the task. Student reflections reinforce the emphasis on review and repair, describing useful practice in code understanding, code review, and debugging, as well as a more careful verification mindset and greater awareness of GenAI limitations. Ultimately, prompt-related failures and injected bugs together support a pedagogically useful GenAI workflow, where students practice both specification refinement through prompts and debugging through code editing.

• Social and professional topics → Computing education; • Computing methodologies → Artificial intelligence.

ICER 2026 Vol. 1, Uppsala, Sweden © 2026 Copyright held by the owner/author(s). This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the ACM Conference on International Computing Education Research Vol.1 (ICER 2026 Vol. 1), August 11–14, 2026, Uppsala, Sweden, https://doi.org/10.1145/3765964.3811667.

Keywords natural language programming, code-generating AI, bug injection, debugging ACM Reference Format: Victor-Alexandru Pădurean, Kaitlin Riegel, Alkis Gotovos, Jyotika Mahapatra, Ahana Ghosh, Paul Denny, Juho Leinonen, James Prather, and Adish Singla. 2026. When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code. In Proceedings of the ACM Conference on International Computing Education Research Vol.1 (ICER 2026 Vol. 1), August 11–14, 2026, Uppsala, Sweden. ACM, New York, NY, USA, 17 pages. https://doi.org/10. 1145/3765964.3811667

Introduction

Widespread adoption of Generative AI (GenAI) coding assistants is transforming professional software development. Most developers now integrate AI tools into their workflows [45, 80], and recent work has begun to explore how these tools redefine traditional development roles and practices [2, 62]. However, the ability to generate code quickly does not guarantee correctness, especially at higher levels of complexity in large code bases. Empirical evaluations of copilot-style suggestions show substantial rates of incorrect solutions even on well-scoped programming tasks [39]. Studies of GenAI-generated code report recurring bug patterns [68] and frequent security vulnerabilities [69]. Output variability also means developers often need to inspect and compare multiple alternatives rather than relying on a single suggestion [43]. Consequently, effective use of GenAI requires skills in clear specification as well as critical code review, systematic testing, and debugging. These needs are reflected in emerging workflows such as GenAI-supported code review that explicitly leverage prompt engineering and refinement [51]. Computing education has mirrored these developments, integrating GenAI-based workflows into the curriculum. Educators

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

Victor-Alexandru Pădurean et al.

Figure 1: The interface of Prompt Programming [46], exemplified on Array IV. The interaction proceeds as follows: (1) the student sends an initial prompt that mis-specifies the task (reversing the array segment from start to end); (2) the GenAI assistant responds with a reversing implementation, which is incorrect for the intended sorting task; (3) the student reprompts, correcting the request to sort the segment in ascending order; (4) the GenAI assistant generates a correct sorting solution, but our bug injection pipeline intercepts it and shows a near-miss version where the inner loop starts j at start instead of i+1; and (5) the student edits the code manually in the editor. have called for curricular adaptation [12] and the community is actively debating implications for learning goals and assessment [36]. Early CS1 course designs integrating LLMs emphasize fundamentals such as decomposition, explanation, and testing when producing software with GenAI [70]. Broader studies further argue for treating GenAI use as a first-class curricular concern, framing risks and opportunities while outlining concrete teaching responses [13, 52]. A growing line of work has explored how to incorporate GenAI into computing education through activities centered on naturallanguage specification. Prompt Problems, prompt-based programming, and related instructional designs treat prompting as a central learning goal, examining how students formulate and refine

prompts to communicate intent [15, 28, 46]. However, this emphasis can shift attention away from inspecting generated code. Empirical studies report that students frequently submit AI-generated solutions with minimal modification [24], and that novices often rely on models to produce and iteratively fix solutions without critically reasoning about the code [56]. More broadly, recent work shows that students may struggle to comprehend GenAI code while simultaneously developing an illusion of competence, believing they understand solutions they have not critically evaluated [55, 79]. These patterns are reinforced by the fact that, for many CS1-level tasks, modern models often produce correct solutions. If the generated code always seems to work, what incentive do students have

When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

to inspect it closely? This stands in contrast to professional practice, where code that appears correct may contain subtle faults and must be carefully reviewed, tested, and debugged. Within GenAImediated workflows at the introductory level, students may have few structured opportunities to practice verification and repair. To address this gap, we explore whether deliberately introducing small, realistic faults into otherwise correct GenAI-generated code can create valuable code review opportunities for students. To do this, we integrate a bug injection pipeline into Prompt Programming [46], our publicly available prompt-based programming platform. Students receive a task and write a natural language prompt which a GenAI assistant uses to generate code which is then run against hidden tests. If unsuccessful, they can iteratively respond to failures by engaging with the assistant or the code. We distinguish two scenarios. In the first case, the student-written prompt generates code that does not pass all the tests (we call this a ‘natural bug’) and we show the test case failures along with the code. In the second case, the student-written prompt generates code that would pass the tests. In this latter case, our pipeline replaces that code with a realistic, runnable near-miss variant containing a subtle fault (we call this an ‘injected bug’). In either case, the student can try to resolve the bug either by fixing the code directly or by modifying the prompt they send to the GenAI model (which will subsequently result in another natural or injected bug). We analyze student behaviors with this pipeline at scale in a classroom deployment, covering 2,636 student sessions (a complete sequence of prompts and code edits for a given task that ends when the student completes or abandons the task) and 6,071 bug-fixing turns (individual student follow-ups after a GenAI response with buggy code, natural or injected). Figure 1 illustrates the platform interface and a representative session in which a natural bug is followed by an injected bug, highlighting a workflow that requires both precise specification through prompting and careful code review to resolve. Our goal with this work is not to undermine students’ trust in GenAI or their own abilities, but to make code review an explicit part of using GenAI. In our design, injected bugs are small, realistic, and accompanied by feedback that mirrors authentic development workflows in which externally authored code, including AI-generated code, must be reviewed, tested, and repaired. This design creates structured opportunities for students to inspect generated code, verify its behavior, and make localized repairs, instead of relying only on passive acceptance of generated solutions. We leverage large-scale classroom data to evaluate how students interact with this integrated environment, focusing on behavioral patterns and qualitative feedback. Specifically, our analysis is guided by the following research questions: RQ1: How do different bug sources (natural vs. injected) influence students’ immediate response choices and immediate outcomes when interacting with buggy GenAI code? RQ2: How do students’ repair strategies differ across bug sources after encountering buggy GenAI code? RQ3: How do students reflect on their experience, understanding of AI-generated code, and perceived learning benefits of prompting tasks with deliberately buggy GenAI code? The rest of this paper is organized as follows. Section 2 reviews related work, Section 3 describes the study methodology, Section 4

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

presents the results, Section 5 discusses implications and limitations, while Section 6 concludes our paper.

Related Work

In this section, we discuss several areas of related work that motivate our study and help position our contributions.

2.1

Generative AI and Agents in Computing Education

GenAI, especially code-generating large language models (LLMs), has rapidly become central in computing education. Position papers and broad reviews argue that on-demand code and explanations change what it means to learn programming, and push the community to revisit pedagogy, assessment, and learning goals in light of widespread access to these tools [12, 13, 52, 53]. Recent work also reflects the expanding capabilities of these systems, including multimodal interactions in programming support and more agentic workflows in which LLMs take on multiple roles, such as generation and validation [21, 42, 44, 50]. Building on the potential of GenAI, CS education research has been studying concrete applications that help students during programming. Some examples include improved error messages, targeted feedback for syntax errors, providing code repairs, and semantic hint generation [30, 33, 49, 50, 73, 78]. Other studies examine GenAI as an interactive programming partner, including copilot-style support and chat-based assistants for multi-turn problem solving [46, 54, 63, 72]. In parallel, instructorfacing work studies how to integrate GenAI into classrooms while preserving pedagogical control, for example through assistants that avoid giving away full solutions and emphasize transparency and guardrails [26, 52, 53]. It also includes systems that generate programming learning materials, such as exercises, through automatic generation, personalization, and task synthesis [34, 42, 61]. A recurring concern in this rapid incorporation of GenAI across computing education is overreliance. GenAI can accelerate students who already have a plan, but it can also amplify metacognitive difficulties, create illusions of competence, and reduce engagement with code reasoning and verification [26, 55]. As a result, several CS education papers emphasize designing interventions that keep students actively involved (e.g., requiring checking and justification) and explicitly preparing learners to be critical, informed GenAI users [6, 52]. This sets up the need to study not just whether GenAI helps, but how learners respond when GenAI outputs fail, and which structured failure experiences support debugging and code understanding without outsourcing them to the model.

2.2

Prompt Problems and Prompt-Based Programming

Recent work increasingly treats prompts designed for GenAI to produce code as a first-class programming artifact rather than a simple request for help. Prompt-centered exercises make natural-language specification and refinement central to the task. Specifically, Prompt Problems operationalize these skills by assessing students on their ability to prompt an AI model to generate code that meets a target specification [15]. Recent work has shown how prompting-for-code can be integrated into introductory programming through structured labs that emphasize communicating intent and evaluating

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

whether generated code meets task requirements [28]. Beyond oneshot exercises, prompt-based programming is increasingly framed as an interactive workflow where students engage in multi-turn dialogue, run generated code, interpret failures, and revise their requests [46, 77]. This further aligns with broader arguments that natural-language prompting is becoming a significant abstraction layer in computing education [57]. Together, this work shifts attention from one-shot generation to the prompt–test–revise loop students follow when outputs are plausible but incorrect. In this study, we build on these foundations to investigate how learners respond to targeted failures, specifically exploring how we can encourage attentive code review and teach students to navigate the inherent dangers of overreliance on GenAI [25, 54, 55]. Using natural and injected bugs to create distinct failure contexts, we examine how near-miss failures can serve as a teaching tool to promote code inspection, verification, and debugging rather than passive acceptance of AI-generated code.

2.3

Debugging, Bug Injection, and Controlled Failures

Debugging is consistently reported as one of the hardest parts of learning to program, largely because novices struggle with the steps between seeing a failure and producing a concrete, testable explanation. Prior work shows beginners often follow an edit-and-test approach, making superficial code changes without a clear hypothesis, instead of systematically localizing the fault and validating hypotheses with evidence [18, 37]. Such patterns show up across multi-institution studies of novice debuggers and qualitative analyses of their tactics (e.g., tracing, print debugging, commenting out code, or small experimental edits) [23, 40]. These findings motivate explicit, structured debugging instruction, since debugging skill does not automatically follow from code-writing skill [38]. The need persists in GenAI-supported workflows, as even with an AI debugging tutor, novices vary in when they seek help and how they engage with suggestions [76]. Interviews suggest they do not treat the AI tutor as a primary source for learning debugging strategies. A recent systematic review also shows wide variation in how and how much debugging is taught across curricula [75]. A common instructional response is to use controlled failures, including seeded bugs and injected faults, so students can practice connecting symptoms to causes [18, 37, 38]. GenAI systems also make controlled-failure practice easier to scale, with tools like BugSpotter generating validated buggy programs for debugging practice [47], while HypoCompass uses a learn-by-teaching setup that pushes students to articulate and test hypotheses during debugging [35]. Related debugging-tool research also emphasizes explanation, for example by supporting explicit “why” and “why not” questions about program behavior [29]. However, this line of work largely treats debugging practice as separate from end-to-end GenAI programming workflows, and does not study deliberately injecting realistic faults into otherwise plausible model-generated solutions during prompt-based development.

Methodology

In this section, we describe the study setting and analysis approach. We first introduce the Prompt Programming platform, then detail

Victor-Alexandru Pădurean et al.

our bug injection pipeline, and summarize classroom deployment, including the five lab tasks. Next, we define key concepts, data collection, and describe the analysis procedures.

3.1

The Prompt Programming Platform

Several tools have been reported in the literature for supporting prompt-based programming activities, sharing several similar features [14, 46, 77]. To conduct our study, we used Prompt Programming [46], our publicly available web platform that supports prompt-based programming through dialogue with a GenAI assistant, hidden-test execution, and direct code editing (see Figure 1). For this study, we extended the platform with a bug injection pipeline that can replace otherwise correct generated code with validated, runnable near-miss variants. In Prompt Programming, for each task students receive a problem specification and are required to craft a natural-language prompt to generate a C implementation that matches it. Once a response is received, students can execute the code against hidden unit tests to receive immediate feedback on correctness (including any failed test case) via console output. When a run fails, students can either continue the dialogue by sending another prompt or switch to the built-in editor to make code changes and re-run the edited code. Our GenAI assistant was backed by GPT-4o-mini, and the “prompt-and-test” workflow allows for an iterative approach to problem-solving where students alternate between specification and repair. Our platform also includes a pipeline for injecting bugs into code generated by this model, as we detail in the following section.

3.2

Our Bug Injection Pipeline

To investigate student debugging behaviors, we incorporate our bug injection pipeline into the prompt-based programming workflow. This system creates deliberate, runnable failures, requiring students to engage in code verification and targeted repair, rather than relying only on prompt iteration to refine the specification. Students also encounter incorrect GenAI outputs during normal prompting. We refer to these naturally occurring prompt-related failures as natural bugs. These natural bugs often stem from specification mismatches or logic errors in the generated code. In contrast, authentic debugging scenarios also involve “near-miss” solutions, i.e., code that seems correct and close to a working solution, but contains a localized bug. We use a GenAI model for injection because it can propose context-aware edits that match the structure and style of the originally generated solution. We validate buggy candidates against the hidden tests to ensure the injected version remains executable and actually fails. Thus, our pipeline allows the existence of both natural and injected bugs during the same interaction, without disrupting the normal prompt-based programming workflow. The bug injection pipeline, illustrated in Figure 2, consists of multiple steps. It acts as a middleware layer between the GenAI assistant’s code generation and the code shown to students. First, the student sends a prompt, and the GenAI generates candidate solution code. The pipeline intercepts the generated code and executes it against hidden tests. If the code fails, it is sent to the student unchanged and is considered to contain a natural bug (‘n’). If the code passes, it is sent back to the same GenAI model (GPT-4o-mini) under a different system prompt, where it acts as a “bug injection

When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden NATURAL BUG (n)

INJECTED BUG (i)

FAIL PROMPT

CODE

FAIL

TEST CODE

PASS TEST CODE CANDIDATE BUGGY CODE

PASS

SLIP THROUGH

Figure 2: Bug injection pipeline in our study. After a student submits a prompt, the GenAI assistant generates code that is checked against hidden tests. Failing code is served unchanged as a natural bug (also referred to as ‘n’); passing code is routed through a bug injection step that proposes and validates candidate buggy code variants, and a validated failing variant is served as an injected bug (also referred to as ‘i’). If no candidate can be validated as a runnable failing variant, the original passing code is served unchanged, i.e., a slip-through. expert” and injects a realistic bug (examples shown in Table 2). The bug injection expert produces candidate buggy variants (we set the number of generated variants to 5), which are then validated against the hidden test cases. We use the first variant that compiles, but fails the tests, to replace the original code shown to the student, which is then treated as containing an injected bug (‘i’). If all generated variants do not compile or pass the tests, bug injection is considered unsuccessful and the student is shown the original correct code. We refer to these cases as slip-throughs. We limit bug injection to this set of candidates, skipping additional injection rounds and permitting some slip-through cases, since repeated attempts would slow responses and interfere with real-time interaction workflow. Importantly, because this middleware is applied to every assistantgenerated code response, students generally cannot solve the task through prompting alone (except in slip-through cases) and must verify and repair code directly to complete the task.

3.3

Classroom Deployment

Data was collected from an introductory C programming course (CS1) at the University of Auckland during the second semester of 2025, under the approval of the university human ethics committee. The overall data collection workflow is shown in Figure 3a. The study was conducted as part of a graded take-home lab assignment which could be completed within a one-week window. Students worked on five prompt-based programming tasks, namely Array II, Array III, Array IV, Matrix, and Binary (task details are presented in Figure 3b). Tasks covered core CS1 topics such as arrays, matrices, and binary operations. At the start of the activity, students were explicitly warned that the code generation model was likely to make mistakes, then instructed to carefully read generated code and manually edit it where needed, framing the lab as verification and repair practice alongside prompting. During the activity, the interface did not indicate whether failing code was due to a natural or injected

bug. Students received credit for solving at least two of the five problems. Data was collected in two stages. First, the platform recorded interaction logs during student attempts, including prompts, generated code, edits, and code executions. Second, immediately after completing the tasks, students completed a reflection questionnaire designed to capture their experiences with code understanding and debugging (reflection questions are shown in Figure 3c). Of 1037 enrolled students, 917 engaged with at least one task and are included in this analysis; 897 of these 917 students (97.82%) solved at least two problems and thus met the credit threshold.

3.4

A Student Session in the Platform

Our analysis is based on the platform’s event logs, which record student–GenAI message exchanges, code edits, execution history, and a bug injection audit trail. The audit log allows us to verify whether each GenAI solution was successfully modified with an injected bug or left unchanged. To reduce noise typical of web-based systems (e.g., timeouts or interrupted sessions), we apply conservative filtering and exclude sessions that do not follow strict alternation between student and assistant messages (i.e., prompts and GenAI replies must alternate in timestamp order, with each prompt followed by a reply; we did not apply any time-gap threshold) or for which no executable code artifact can be recovered. This removed 177 (6.3%) of 2,813 logged sessions, leaving 2,636 for our analysis. In our analysis, we define a ‘session’ as one student’s sequential interaction with the GenAI assistant while attempting to solve one problem, ending when the student either succeeds or disengages from the task. There is at most one session for a student-problem pair. A ‘turn’ is defined as a student’s follow-up action in response to a specific GenAI message, conditioned on the state of the immediately preceding GenAI message. Specifically, we categorize the preceding GenAI message as: (1) no-code (the message contains no code); (2) natural bug ‘n’ (the generated code contains a natural bug); (3) injected bug ‘i’ (the generated code contains an injected

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden LAB TASKS INSTRUCTIONS

Victor-Alexandru Pădurean et al. POST-TASKS REFLECTION

Array II

Task

Description

Array II

Count negative values in an array

Array III

[5-point Likert scale] Q1

Array IV

[5-point Likert scale] Q2

Array IV Partially sorting an array within a specified index range

[Free text response] Q3

Matrix

Propagate 1’s across the corresponding rows and columns in a 2D array (matrix)

Binary

Compute fixed-size binary addition

Matrix

Array III Index of the last zero in an array

Binary (a) Data collection procedure

(b) List of programming tasks

Question Description Q1 Q2 Q3

[5-point Likert scale] I found it easy to locate and fix bugs in the code [5-point Likert scale] I understand code I have written myself better than code generated by an AI model [Free-text response] What benefits do you believe this version of the prompt-based programming task (i.e., where the model deliberately makes mistakes) offers for your learning? (c) Post-tasks reflection questionnaire

Figure 3: Classroom deployment. (a) presents the two stage data collection procedure in the study. In the first stage, data is collected from platform logs. In the second stage, we collect student reflections after completing the tasks. (b) details the five programming tasks used for the study. (c) presents details of the post-tasks reflection questions. Details of data collection presented in Section 3.3.

STUDENT SESSION 1

STUDENT SESSION 2

. . . .

STUDENT SESSION K

INITIAL PROMPT

n:any, n:first

i:any

n:any, n:first

n:any

i:any, i:first i

Figure 4: Examples of student sessions, along with the terminology used throughout analysis. A session consists of a sequence of student–GenAI interactions, and each turn is defined as a student response to a preceding GenAI message. The GenAI response in a turn may contain a natural bug (n) or an injected bug (i). We label turns containing natural bugs as n:any and turns containing injected bugs as i:any. We additionally label the first turn in a session, when it contains buggy code, as n:first or i:first, respectively. Session 1 corresponds to the scenario in Figure 1 and contains two turns: the first turn is labeled n:any and n:first, and the second turn is labeled i:any. Session 2 shows multiple consecutive natural-bug turns, with only the first labeled n:first. Session K consists of a single injected bug turn, labeled both i:any and i:first. All behavioral analyses are aggregated at the turn level. Additional details are in Section 3. bug); and (4) slip-through (generated code is correct and passes all test cases). No-code messages typically occur when the assistant responds with clarification questions, explanations, or other noncode guidance rather than an identifiable program. For turn-level analyses, we use two complementary labels. The labels n:any and i:any denote any turn in which the student is responding to a natural or injected bug, respectively. We also mark the first turn in each session, when it contains buggy code, as n:first or i:first, which lets us analyze students’ initial response to failure before later turns in

the same session introduce repeated exposure or strategy carryover. Figure 4 illustrates three example sessions and these labels. For behavior and outcome analyses, we focus on turns where the student is responding to buggy code (natural or injected bug). Within each turn, we operationalize the student’s next-step response using the first logged follow-up action, i.e., prompt if the student submits a new prompt before making any manual edits, and edit if the student edits code before sending a follow-up prompt. Turns where neither occurs are labeled as noop.

When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

QUANTITATIVE ANALYSIS

RQ1 LOG DATA FROM TASKS

n:any, n:first, i:any, i:first

Quantitative Metrics

edit, prompt, noop

MIXED METHODS

RQ2

Quantitative Metrics, Strategy Themes from Qualitative Coding n:any, n:first, i:any, i:first

edit, prompt, noop

MIXED METHODS

STUDENT REFLECTION DATA

RQ3

[5-point Likert scale] Q1

Quantitative Metrics, Perception Themes from Qualitative Coding

[5-point Likert scale] Q2 [Free text response] Q3

Figure 5: Data analysis procedures. Overview of the data analysis procedures across research questions. (top) For RQ1 and RQ2, we analyze log data collected from students’ interactions with the programming tasks. Analyses are conducted at the turn level (n:any, n:first, i:any, i:first) and examine subsequent student actions (any, noop, edit, prompt). We perform quantitative analyses to measure behavioral patterns and outcomes, and qualitative coding to identify student strategies for responding to natural and injected bugs. (bottom) For RQ3, we analyze student reflection data collected at the end of the lab, including Likert-scale responses (e.g., perceived ease of bug fixing and code comprehension) and free-text responses. Quantitative analyses summarize Likert responses, while thematic coding extracts themes on perceived learning benefits from free-text reflections. Further details on data analysis procedure are presented in Section 3.5.

3.5

Data Analysis Procedures

Our analysis procedure is summarized in Figure 5, which shows how different data sources support answering RQs. In particular, RQ1 and RQ2 analyze student log data from the platform, while RQ3 analyzes the post-tasks reflection data. For RQ1 and RQ2, we analyze student behavior at the turn level, focusing on responses to natural and injected bugs. Within each buggy turn (i.e., excluding slip-throughs), we classify actions (prompt, edit, or noop) and measure immediate success conditional on that action. For turns where students choose to prompt, immediate success is defined counterfactually using the injection audit, and a turn is marked successful if the subsequent assistant output would have passed tests prior to bug injection. For turns where students edit, immediate success reflects whether the student obtains working code through only edit-and-run attempts without submitting another prompt. Because injected bugs are introduced only after an initially correct solution, comparisons between natural and injected bugs are interpreted as differences in response behavior rather than causal effects of bug source. For statistical testing, we restrict attention to the first buggy instance per session (n:first and i:first) to better satisfy independence assumptions. Each session therefore contributes at most one observation per problem. Testing results are reported at the per-problem level, and pooled summaries are descriptive. As tests are exploratory, we report uncorrected 𝑝-values. We use Pearson’s 𝜒 2 test for categorical outcomes and Mann–Whitney tests for continuous measures.

To analyze students’ prompting strategies during bug-fixing turns, we use a single-label coding scheme that assigns each sampled prompt one primary strategy (i.e., student’s approach to design a follow-up prompt to resolve the existing bug), following standard content analysis practice [41, 66]. We sampled prompts uniformly from bug-fixing turns where students chose to prompt, making sure that each student is represented only once. Two annotators followed a two-phase process. In the open coding phase, they independently coded 60 prompts to identify strategy categories and construct an inductive codebook. In the closed coding phase, they first calibrated on the open-coded examples, then independently coded 144 additional prompts. We estimated inter-rater reliability (IRR) using Krippendorff’s 𝛼, treating values above 0.8 as reliable [31, 32]. After computing IRR (𝛼 = 0.87), the annotators reconciled disagreements, and we use the reconciled labels for reporting. No new strategy categories were introduced during closed coding and reconciliation, suggesting the codebook had stabilized for the sampled prompts. In total, 204 prompts were coded. For RQ3, we analyze post-tasks reflections using both quantitative summaries of Likert-scale responses and multi-label thematic analysis of responses to the free-text question [5, 9]. For thematic analysis, we sampled reflections uniformly, as each student contributes at most one reflection about the activity. Thematic coding followed open and closed phases, with annotators coding 54 responses in open coding and an additional 150 responses in closed coding. Reliability was assessed using Krippendorff’s 𝛼 per label,

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

Victor-Alexandru Pădurean et al.

Table 1: Descriptive statistics by problem. The table reports participation, session-level outcomes, and turn-level coverage conditioned on the preceding GenAI message, for all turns (any turn) and first turns only (first turn). Turn-level categories are no-code (the GenAI message did not contain code), natural bug, injected bug, and slip-through. The total turns (6,706) include no-code and slip-through turns, whereas the bug-fixing turns (6,071) include only natural and injected bug turns. Session-level

Turn-level: any turn

Turn-level: first turn

Problem

Nb. students

Array II Array III Array IV Matrix Binary

900 866 386 270 214

900 99.00% 866 97.00% 386 76.94% 270 82.96% 214 50.93%

1848 3.03% 2348 1.32% 1040 2.12% 607 1.32% 863 3.36%

31.17% 37.61% 58.46% 42.17% 89.34%

50.70% 59.16% 29.42% 47.12% 6.84%

15.10% 1.92% 10.00% 9.39% 0.46%

900 2.44% 866 1.39% 386 1.81% 270 1.11% 214 3.27%

33.33% 42.73% 68.13% 45.56% 92.99%

49.78% 55.08% 21.76% 45.93% 3.74%

14.44% 0.81% 8.29% 7.41% 0.00%

All Probl.

917

2636 89.57%

6706 2.18%

46.14%

44.39%

7.29%

2636 1.93%

47.61%

43.29%

7.17%

Nb. Perc. Nb. No Natural Injected Slip Nb. No Natural Injected Slip sessions succ. turns code bug bug through turns code bug bug through

Table 2: Examples of injected bugs by problem. We also report the number of distinct injected bugs for each problem after normalizing away comments, formatting, and variable namings, while keeping logic-changing differences such as operators and numbers. We show two representative examples chosen from the five most frequent injected bugs for that problem. Each example is presented as a compact changed-line diff, where the first line shows the original code (highlighted in light green ) and the second line shows the buggy version (highlighted in light red ), followed by a short explanation of what was changed. Problem

Array II

Array III

Array IV

Matrix

Binary

Nb. bugs

104

112

Example bug diff

Explanation

int count = 0; int count;

The counter variable is no longer initialized to 0, so it starts with an indeterminate value.

count++; count += 2;

Each negative value is counted twice.

for (int i = size - 1; i >= 0; i--) for (int i = size; i >= 0; i--)

The loop starts one position past the valid range.

return i; return i + 1;

The index returned after finding the last zero is shifted by one.

for (int j = i + 1; j <= end; j++) for (int j = start; j <= end; j++)

The inner loop of the sorting logic wrongly starts from the beginning of the target subarray (also in Figure 1).

if (array[i] > array[j]) if (array[i] < array[j])

The comparison direction for sorting is flipped, pushing elements in the wrong order.

for (int i = 0; i < rows; i++) for (int i = 0; i <= rows; i++)

The loop runs one step past the valid range when iterating over matrix rows.

rowFlags[i] = 1; colFlags[j] = 1; rowFlags[j] = 1; colFlags[i] = 1;

The row and column indices are swapped when marking flags for propagation.

int carry = 0; int carry = 1;

The carry variable is initialized to 1 instead of 0, introducing a spurious initial carry.

carry = sum / 2; carry = sum % 2;

Carry propagation for the binary addition uses the remainder instead of overflow.

computed by binarizing label presence/absence per response (mean 𝛼 = 0.82, range = 0.59-0.96) [10, 31, 32], and MASI-based agreement (𝛼 = 0.80) [11, 48]. No new themes emerged during closed coding and reconciliation. We report results using 204 reconciled responses.

Results

In this section, we first show descriptive statistics of the collected data, and then present results of our study centered around the three research questions formulated in Section 1.

When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

4.1

Descriptive Statistics

Table 1 presents an overview of participation, session-level, allturns-level, and first-turn-level statistics. The “Natural bug” and “Injected bug” columns under “Turn-level: any turn” correspond to n:any and i:any, respectively, while the “Natural bug” and “Injected bug” columns under “Turn-level: first turn” correspond to n:first and i:first. Across all problems, the dataset includes 917 participating students and 2,636 sessions, with an overall session success rate of 89.57%. Session success and participation were highest on Array II and Array III and lowest on Binary, with Matrix and Array IV in between. Across all problems, natural and injected bug sources were broadly balanced in both the ‘any turn’ and ‘first turn’ levels. However, their distribution varied widely by problem. Table 2 shows examples of injected bugs observed in practice, collected from our bug audit logs. For each problem, we report the number of distinct bugs after grouping together superficial variants of the same change. We do this by comparing only the changed lines between the original and injected code, ignoring comments and formatting, and merging simple identifier renamings, while keeping logic-changing edits such as operator and numeric differences separate. Overall, the examples suggest that injected bugs are typically small, local mutations, the majority of which resemble plausible near-miss faults that students may realistically encounter in practice when reading, testing, and repairing code.

4.2

RQ1: Follow-Up Action Choices and Immediate Outcomes After Bugs

To address our first RQ, we analyze students’ actions and immediate success rates after buggy GenAI responses, conditioned on bug source. Figure 6 presents an overview of the results. Follow-up action choice. Specifically, Figure 6a reports student actions after a buggy GenAI response, conditioned on bug source (i.e., n:first, i:first, n:any, and i:any). We show results both pooled across problems and broken down per problem. Pooled across all problems, the ‘first turn’ summaries show a clear shift by bug source. Students are more likely to choose prompting in n:first than in i:first (51.71% vs. 31.46%), while edit actions are more common in i:first than in n:first (67.57% vs. 45.42%). The same trend appears in the ‘any turn’ summaries, with n:any having a higher share of prompt actions than i:any (60.57% vs. 48.17%), while i:any has a higher share of edit actions than n:any (49.11% vs. 34.94%). Noop responses, where students neither reprompt nor edit, are rare. The perproblem breakdown follows the same overall trend, with the clear contrasts in Array II and Array III, but smaller gaps in Array IV and Matrix. Binary has sparser coverage, making comparison noisier1 . The results suggest that injected bugs are often encountered as locally repairable near-misses. Relative to natural bugs, they are associated with less prompting and more editing. This pattern is consistent with injected bugs being introduced only after the assistant first produces correct code, necessitating from students a smaller and more localized repair and creating more direct opportunities for verification and repair. 1 Per problem, we compare n:first vs. i:first action distributions using 𝜒 2 . The direc-

tional difference was strongly supported in Array II ( 𝜒 2 (2) = 49.40, 𝑝 < .001) and Array III ( 𝜒 2 (2) = 51.57, 𝑝 < .001), but not in Array IV ( 𝜒 2 (2) = 1.20, 𝑝 = .550) or Matrix ( 𝜒 2 (2) = 3.25, 𝑝 = .197). Binary too sparse.

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

Immediate success rates. Figure 6b reports immediate success rates after a buggy GenAI response, conditioned on the student’s first action and bug source. We focus on the overall trends and therefore show pooled rates across problems. Immediate success rates following student responses to injected bugs are higher than those after natural bugs. In the ‘first turn’ summaries, success for any action is 89.92% for i:first vs. 51.16% for n:first; the same contrast holds within prompt actions (87.74% vs. 43.30%) and edit actions (92.22% vs. 63.33%). In the ‘any turn’ summaries, success is 84.95% for i:any vs. 38.30% for n:any, with the same trends for prompt (86.54% vs. 31.80%) and edit actions (88.10% vs. 54.49%). Under the n:first vs. i:first conditions, immediate success differs by bug source for any action in Array II, Array III, Array IV, and Matrix (𝜒 2 , df = 1, all 𝑝 < .001); Binary interpreted descriptively due to sparse data. The same trends hold within prompt and edit actions, with 𝑝 < .001 in most problems (except for prompt action in Array IV 𝑝 = .003, and edit action in Array II 𝑝 = .038).2 The immediate success pattern aligns with intervention design. In particular, injected bugs are introduced only after the assistant first produces a correct solution, leaving students to repair a localized fault. Natural bugs, in contrast, more often reflect specification or logic mismatches, making immediate resolution less likely even after a reasonable action. RQ1 summary. Overall, our results highlight differences by bug source in both follow-up actions and immediate success. We observe that injected bugs more often come with editing rather than reprompting, and are more likely to be resolved immediately. We do not interpret this as a causal difficulty comparison, because injected bugs are introduced only after the GenAI first produces a correct solution. Instead, these patterns suggest that bug source is associated with different response contexts and likely next actions. This comes in line with the intended design, i.e., natural bugs more often coincide with specification refinement, while injected bugs more often coincide with verification and localized repair.

4.3

RQ2: How Prompting and Editing Strategies Differ According to Bug Sources

To address our second RQ, we focus on how students repair buggy GenAI responses and how those behaviors differ by bug source (n:first, n:any, i:first, and i:any). Figure 7 presents an overview of the results. Editing behavior. For edit actions, Figures 7a and 7b measure repair effort using mean absolute Levenshtein distance from the initial buggy output to the student’s final edited version, and the mean number of edit-and-run submissions within the turn. They show clear differences in editing effort. Following injected bugs, students make smaller code changes and fewer edit-and-runs than after natural bugs. In the ‘first turn’ summaries, edit distance is lower for i:first than n:first (median 4 vs. 24, mean 21.96 vs. 93.92), and the same trend holds for ‘any turn’ (median 4 vs. 35, mean 25.18 vs. 121.02 for i:any vs. n:any). Edit-and-run counts follow the same trend in means (first turn: 1.51 vs. 2.15; any turn: 1.55 vs. 2.36), with medians typically 1 and small differences in some 2 Corresponding 𝜒 2 (1) statistics (Array II, Array III, Array IV, Matrix) are: any action (66.50, 88.96, 34.24, 44.43); prompt (46.39, 45.35, 9.08, 30.23); edit (4.32, 31.05, 25.55, 11.54). Binary is too sparse for reliable testing.

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

Victor-Alexandru Pădurean et al.

Fraction for action:prompt

Fraction for action:edit

n:first

n:any

i:first

i:any

1.0 0.5 0.0 1.0

0.5

Array III

Array IV

Matrix

N=59

N=8

N=771

N=199

N=286

N=124

N=256

N=123

N=306

N=84

N=608

N=263

N=477

N=883

N=370

N=937

N=448

Array II

N=1389

All Problems

N=576

N=300

N=2977

N=1141

0.0

N=3094

0.5 N=1255

Fraction for action:noop

0.0 1.0

Binary

n:first

1.0

n:any

i:first

i:any

action:prompt

N=1434

N=359

N=649

N=1874

action:edit

N=1462

N=771

N=570

N=81

N=11

N=36

N=139

action:noop

N=1081

action:any

N=2977

N=1141

0.0

N=3094

0.5 N=1255

Success rate

(a) Fraction of turns where students chose to edit first (top), prompted (middle), or did not follow up with an action (bottom).

(b) Fraction of turns with immediate success, conditioned on students’ actions.

Figure 6: RQ1 quantitative results. (a) shows the first-action distributions after a buggy GenAI response (shown for four source-based groups n:first, n:any, i:first, and i:any); fractions across the three action categories sum to 1 within source-based group. (b) shows the immediate success rates for the same groupings, conditioned by action. For cases where students prompted, immediate success is defined using the pre-injection audit outcome of the next generated code; for cases where the student edited, it is defined by reaching a passing result through edit-and-run attempts before the next prompt; ‘noop’ turns naturally have 0 immediate success. problems (e.g., 1 vs. 2 in Array IV/Binary for i:first vs. n:first, and in Matrix/Binary for i:any vs. n:any). Under the n:first vs. i:first conditions, per-problem Mann-Whitney tests indicate lower edit distance for injected than natural bugs in Array II, Array III, and Matrix (all 𝑝 < .001), but not in Array IV (𝑝 = .226); Binary is interpreted descriptively due to limited data. For run-edit counts under the same conditions, per-problem tests yield 𝑝 = .030 (Array

III), 𝑝 = .049 (Array IV), and 𝑝 < .001 (Matrix), with 𝑝 = .061 in Array II; Binary sparse as before.3 These results suggest that students treat injected bugs as localized repair tasks, possibly due to their confined nature. Students typically patch the small faults in near-miss solutions. In contrast, natural bugs often require broader code revisions or reworking, producing larger edits and potentially longer repair cycles. 3 Corresponding Mann-Whitney 𝑈 statistics (Array II, Array III, Array IV, Matrix) are:

edit distance (34990, 50256.5, 2218, 2921.5); run-edit counts (22316.5, 35118, 2351, 2626.5). Binary too sparse.

When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

22.1

3.3

3.0

Reframe the specification, reinterpret the goal of the task.

Update signature

30.6

15.4

3.3

1.0

Update signature Change the function signature or arguments (add, remove, rename).

Extend task

27.8

13.5

26.7

11.0

Restate

8.3

8.7

23.3

29.0

Provide guidance

8.3

14.4

26.7

33.0

Provide output

5.6

11.5

6.7

10.0

Provide code

2.8

9.6

6.7

7.0

Meta

Comment on the interaction (e.g., frustration) rather than the task. (d) Prompting strategy codebook.

40%

30%

20%

10%

Percentage within group

13.9

Extend task

N=1434

N=359

Task reframe

(N=100) i:any

(N=30) i:first

(b) Edit-and-run count

N=1874

N=649

Nb. characters N=1462

100

(N=104) n:any

Task reframe

N=771

200

(N=36) n:first

Primary intent

i:any

(a) Edit distance

Strategy

i:first

N=1081

N=1462

N=771

N=1081

n:any

N=570

Nb. run-edits

100

N=570

Edit distance

n:first

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

(e) Prompting strategy distribution by bug source.

Figure 7: RQ2 mixed-method results. (a) shows the mean absolute Levenshtein distance between the initial buggy GenAI code and the student’s last edited code, computed for cases where students chose an edit action. (b) shows the mean number of edit-and-run attempts within the same cases. (c) shows the mean prompt length in number of characters for cases where students chose a prompt action. (d) summarizes the prompting strategy codebook obtained during annotation. (e) shows the distribution of prompting strategies from the qualitatively coded prompt sample; cells are column-normalized fractions. We report both ‘first turn’ (n:first, i:first) and ‘any turn’ (n:any, i:any) bug-fixing turns; n:first is a subset of n:any and i:first is a subset of i:any. Prompting behavior. Figure 7c shows prompt length. It does not show systematic differences between bug source. In the pooled ‘first turn’ summaries, prompt length is similar for n:first and i:first (median 138 vs. 128, mean 173.29 vs. 177.25), and the pooled ‘any turn’ summaries are also close (median 149 vs. 148, mean 200.99 vs. 192.47 for n:any vs. i:any). Under the n:first vs. i:first conditions, per-problem Mann-Whitney tests also show mixed evidence: Array II (𝑝 = .063, medians 74 vs. 87), Array III (𝑝 = .035, medians 144 vs. 102), Array IV (𝑝 = .422, medians 180 vs. 169), and Matrix (𝑝 = .034, medians 154 vs. 231), with Binary too sparse4 . We therefore focus on prompting strategy rather than prompt length. 4 Corresponding Mann-Whitney 𝑈 statistics (Array II, Array III, Array IV, Matrix) are:

prompt length (9870, 13542, 2885.5, 1105) . Binary too sparse.

Figure 7d summarizes the reconciled prompting strategy codebook obtained during qualitative coding (Krippendorff’s 𝛼 = 0.87), while Figure 7e shows the distribution of prompting strategies by bug source (n:first, n:any, i:first, i:any). Natural bug states are dominated by specification-oriented strategies, while injected bug states show more guidance and restatement-oriented strategies. In n:first, prompting strategy is led by Update signature (30.6%) and Extend task (27.8%). In n:any, the distribution shifts toward tasklevel clarification, with the majority being Task reframe (22.1%). In i:first, prompting strategies are more mixed, with Provide guidance, Extend task, and Restate having similar frequencies. In i:any, Provide guidance becomes the dominant strategy (33.0%), while task reframing and signature updates are rare.

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

Victor-Alexandru Pădurean et al.

Strongly disagree

Disagree

Neutral

Strongly agree

I understand code I have written myself better than code generated by an AI model.

I found it easy to locate and fix bugs in the code.

Agree

100

(a) Likert responses for Q1 and Q2.

100

Solution exploration Considering alternative approaches for solving the problem. Concept learning

Learning or strengthening programming concepts and precision.

Bug types

Exposure to common or new bug patterns.

Other

Vague or uncategorizable comments. (b) Qualitative coding scheme used for Q3.

19 13 5

Solution exploration

Limitations of AI-generated code and the need for critical evaluation.

Bug types

AI limitations

Concept learning

Understanding requirements and communicating intent clearly.

Other

Specification

AI limitations

AI workflow strategy How to collaborate with the AI and when to switch to editing or writing code directly.

Specification

Locating bugs and deciding what to try next after a failure.

AI workflow strategy

Debugging

Code understanding Reading, tracing, and reviewing code to make sense of how it works.

Code understanding

Description

Count of coded labels

Qualitative code

Figure 8: RQ3 reflections on learning with buggy GenAI code. (a) shows perceived ease of locating and fixing bugs and comparative understanding of self-written vs. AI-generated code (Q1–Q2). (b) summarizes the reflection codebook. (c) reports perceived learning benefits from open-ended reflections (Q3) as code frequencies, with green for bug injection and purple for general prompt-based programming. This contrast suggests differences in how students respond to the two contexts. Natural bugs appear to signal a misunderstanding of the task, possibly encouraging students to revise or clarify their task description. Injected bugs seem to signal a near-miss solution, possibly encouraging students towards more guidance-oriented requests to the GenAI assistant. RQ2 summary. Overall, our results highlight differences between bug sources in how students proceed after a failure, beyond final outcome. We more often see localized repair and guidanceoriented prompting after injected bugs, whereas natural bugs more often come with specification-focused revisions and task-level clarification prompting strategies.

4.4

RQ3: Student Reflections on Debugging, Code Understanding, and Learning Benefits

In our third RQ, we complement the analysis of students’ behavior with their perceptions. We analyze student reflections collected

after task completion, Figure 8 presenting the results. Specifically, Figure 8a shows their responses to the first two Likert-scale questions, Q1 and Q2. For Q1, many students selected neutral (∼ 34%), but responses still leaned slightly positive overall. Just over half of students agreed or strongly agreed (∼ 52%) that they found it easy to locate and fix bugs, suggesting that this type of bug-fixing exercise was generally manageable for students. For Q2, agreement was stronger. Roughly two thirds agreed or strongly agreed (∼ 64%) that they understand self-written code better than AI-generated code, with very little disagreement (< 6%). This perceived gap between self-written and AI-generated code is also emphasized in many of the open-ended reflections, which we discuss next. To analyze responses to the free-text question, Q3, we used thematic coding. We first conducted open coding to identify recurring themes and then consolidated these into a closed coding scheme.

When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

Figure 8b summarizes the reconciled theme set, and Figure 8c reports label frequencies after disagreement reconciliation. Agreement was strong overall (mean Krippendorff’s 𝛼 = 0.82, MASIbased 𝛼 = 0.80), but was lower for Concept learning (𝛼 = 0.61) and Bug types (𝛼 = 0.59), which were less frequent and harder to distinguish consistently, so we interpret those categories more cautiously. Debugging and code comprehension. Many responses emphasized Code understanding and Debugging as primary learning benefits. Students often framed the activity as learning by diagnosing and recovering from failure as opposed to simply producing a solution. One student described it as “[...] learn[ing] by failing [...] you get a close enough result but you learn the process of debugging which is a vital skill to have, especially with team projects”. Another highlighted the value of careful tracing and inspection, noting that it “helps with explicitly ‘visualising’ code (i.e. executing it in my head) and helps with error-spotting, in the same sense that training chess puzzles is beneficial practice for finding tactics in real chess games [...]”. Similarly, a student reflected, “I think it forces you to actually really understand the nuances in the syntax because the tiniest errors will be what throw you off. Being able to see these errors quickly and get good at finding them is very useful. It also allows you to improve your debugging skills with more generic bugs rather than just the ones you are always making yourself. That way you learn how more bugs arise, and can become a more efficient coder who could also read other people’s code and fix it”. Overall, students reported practicing reading, tracing, and repairing plausible programs, suggesting the activity supported verification-centered skills alongside prompt refinement. AI workflow and its drawbacks. Students also reflected on AI workflow strategy and AI limitations, often emphasizing that effective use of the assistant requires active oversight and selective adoption of suggestions. One student described settling on a cautious workflow, writing: “I realised that the best way to write code is likely to try it yourself for the first time, then if you hit a roadblock consult the AI but only implement code that you understand, because if you do not then it is very easy to lose track of the code and mess it up. I found it is harder to interpret finished code then a code you wrote yourself of course”. Others reflected on unreliability and the need for human oversight. For example, one student (after being reminded that bugs are injected on purpose) wrote: “Really... it’s a model designed to intentionally make mistakes... I had my suspicions. I think that’s a bit of an issue a lot of different LLM’s are facing currently. The root of the issue I believe is that because of the fact that now LLM’s are being trained by each other, they often pass on bad habits such as the fact that if an output is deliberately wrong, it incentivises the user to submit another prompt [...] (more user engagement). [...] Because of the fact that this is such a prevalent issue, it is good to learn how to work around it”. Another student summarized this dependency succinctly: “That AI requires human assistance to make good code”. Overall, students viewed AI code as requiring human oversight, and the activity helped them decide when to reprompt, inspect, or repair locally. Specification and broader learning. Beyond the above themes, several reflections emphasized Specification, Concept learning, and exposure to Bug types. Some students noted that the activity pushed them to be more explicit about intended behavior and its implementation, for example: “It helps me understanding specifically on what I’m writing down and what my code should look like”. Others

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

described practicing core programming skills and building a better grasp of how things work, noting: “It was better because I felt like I had to use my own skills more, and it gave me more practice at figuring out how certain things worked”. Finally, students also valued encountering a diversified range of mistakes, as one student put it: “Gives me opportunity to fix coding errors that i would maybe not encounter coding by myself”. Overall, the activity reinforced links between specification, implementation, and failure analysis, beyond debugging alone. RQ3 summary. Our results show that students experienced the activity as practice in code understanding, debugging, and AI workflow strategy, and not just as a harder way to complete tasks. In the Likert responses and open-ended reflections, students emphasized that AI-generated code was harder to understand than self-written code and still required review and debugging. Dominant themes from the open-ended reflections support the instructional value of deliberate, runnable bugs in prompt-based programming activities.

Discussion

Our original goal with this research was to explore whether deliberately injecting bugs into a GenAI workflow would encourage students to engage in code review. Our results indicate that injected bugs are more likely to be resolved immediately than natural bugs, with immediate success rates much higher following injected bugs (e.g., 90% vs. 51% on first turns). This is consistent with the fact that injected bugs were introduced only after the assistant first produced correct code, so they were usually closer to a working solution and left a smaller repair to the student. In that sense, the observed behavioral difference is understandable. Students seem more willing to edit when the code is already near correct, whereas natural bugs more often coincide with specification mismatches that are more naturally addressed through reprompting. At the same time, this is precisely the design space we aimed to create, one in which students must inspect, verify, and locally repair generated code instead of relying only on repeated prompting. We now interpret students’ follow-up actions through two designrelevant lenses grounded in their behavior and reflections. Promptbased programming lets students iterate on natural-language specifications and patch code artifacts in the same workflow [15, 46, 57, 70]. This flexibility, although powerful, changes the meaning of ‘debug’, because students must decide whether to refine a specification, localize and repair a defect, or verify a plausible-looking solution before trusting it [12, 13]. Across RQ1–RQ2, bug source is associated with where students direct repair effort and how they follow up via prompts and edits. From RQ3, injected near-misses create opportunities for verification, which students often describe as useful practice in code understanding and debugging. We use these patterns to motivate two implications.

5.1

Diagnosis Before Action: Bug Source Shapes the Scope of Repair

Our RQ1 findings show that students do not respond to buggy GenAI code with a single default action. Instead, follow-up actions shift with bug source, suggesting that students may use the assistant’s response as one cue for the scope of repair. RQ2 strengthens this interpretation by showing differences in kind, not just

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

amount. Some prompts clarify specifications or constraints, while others focus on diagnosing failures or guiding the assistant toward a fix. Code edits show a similar split, ranging from small, localized patches to larger revisions and longer repair loops. One interpretation is that students use buggy outputs as signals about what needs repair – the task description or the code artifact – and they take their next action accordingly. In GenAI programming, where both options remain available, better diagnosis may help students avoid unproductive loops that apply the wrong kind of repair. A useful way to frame this behavior is as a metacognitive control decision under uncertainty. Learners monitor evidence such as test results, console output, and the alignment between the code and the specification, and they decide whether a failure reflects a specification mismatch or a local implementation defect. According to selfregulated learning theory, such monitoring and interpretation of evidence supports decisions about how to proceed when outcomes diverge from expectations [74, 81]. Metacognitive experience models similarly emphasize that felt difficulty and confidence can influence action selection under uncertainty, rather than implying one fixed response pattern [17]. This distinction aligns with prompt-versusedit choices in GenAI-assisted programming: prompting revises the specification the model uses to generate code, while editing targets a concrete artifact that can be inspected and repaired locally [29, 37]. From this perspective, the pedagogical target is not prompting or editing as isolated skills, but diagnosis that justifies the next step. Our findings suggest that students interpret bug sources and proximity to a correct solution as signals about how to proceed, which has implications for interface design. One practical scaffold is to ask students to commit to an intent, i.e., specification repair or code repair, and provide a checkable artifact that matches the intent before the system enables the next action. Help design research shows that learners often need support to seek the right kind of help at the right time, and that interface-level nudges can shift help use from unproductive dependency toward useful progress [1, 16, 19]. In GenAI workflows, lightweight feedback that connects observed failures to explicit hypotheses may encourage deliberate inspection and reduce blind trial-and-error [29, 77]. Debugging intervention work similarly suggests that teaching a systematic process can improve debugging performance and reduce unproductive struggle, which makes diagnose-before-action a natural approach to teach alongside prompting skills [38, 58, 75]. A bug-source-aware design could suggest default repairs while allowing these to be overriden, maintaining autonomy while nudging learners toward likely productive moves [74, 77]. More broadly, prompt-centered assignments should assess and scaffold diagnostic reasoning, not only prompt wording, because recovery success depends on acting on the right object, at the right time, with the right evidence [15, 37, 76].

5.2

Designed Near-Misses as Verification Scaffolds in GenAI Workflows

Our RQ1–RQ2 findings show that injected, runnable bugs do more than introduce errors, they change how students respond. Nearmiss outputs are close enough to a correct solution that they invite inspection and targeted repair rather than full regeneration via prompting. In our data, injected bugs are linked to different action choices and response patterns than natural bugs, and edits after

Victor-Alexandru Pădurean et al.

injected bugs are typically smaller and more localized. A plausible explanation is that near-misses make local repair both feasible and visible, signaling that the productive move is to verify and patch the code rather than continue prompting. RQ3 reflections reinforce this interpretation, as students describe the exercise as practice in reading code, tracing behavior, and debugging under realistic constraints. These results support a design claim: near-miss injection turns a generative workflow into a continuous practice environment for code verification, embedding diagnostic repair into the core interaction rather than treating it as a separate task. This is important because AI-generated code can look plausible while still being wrong, and verification remains a persistent challenge [43, 60, 71]. Large-scale studies of GenAI-generated code also report non-trivial correctness and bug prevalence, making verification a core skill rather than an optional add-on [39, 68, 69]. One interpretation of these patterns is that near-miss solutions create a form of manageable challenge, as they require students to critically read the code while still keeping the repair space manageable. Research on error management and productive failure suggests that there are benefits to learning when learners engage constructively with bounded difficulty [22, 27, 65]. Prior work on desirable difficulties similarly argues that learning tasks should be challenging enough to trigger deeper processing, but not so challenging that they lead to unproductive struggle [3, 4]. In programming education, effective debugging involves forming hypotheses, testing them, and revising based on evidence rather than making superficial edits [23, 37]. Formative feedback research adds that effective feedback should be specific and actionable [20, 64]. In our setting, runnable near-misses paired with immediate test feedback can support this bounded verification loop, where learners can form a local hypothesis, test it, and iterate within the same workflow, aligning the activity with evidence-driven debugging rather than trial-and-error [16, 29]. Design implications follow from treating bug injection as a verification scaffold that should be clear, low-friction, and fadeable [58]. One practical design is to make verification mode explicit when a near-miss appears, and require a minimal piece of evidence before enabling full regeneration. For example, the system could ask for one checkable artifact, such as a failing test case, an expected property, or a short hypothesis about a suspect location, before allowing another GenAI solution. This keeps help-seeking instrumental and ties assistance to evidence, which aligns with help-seeking theory and help-design guidance [1, 19]. Recent classroom systems similarly suggest that structured interactivity can help learners work productively with LLMs by encouraging testing, inspection, and response to concrete signals instead of treating the model as an oracle [59, 77]. At the same time, reflection should be used carefully. Reflection can improve monitoring and evaluation, but it can also add friction and reduce satisfaction [8]. Cognitive load theory suggests that overloaded learners may fall back on fluency-based heuristics, including trusting outputs that only look correct [67]. Self-explanation cues can counter shallow processing by requiring learners to articulate why code should work, but these cues need to stay lightweight to avoid becoming another source of friction [7]. A practical balance is to pair near-miss injection with lightweight verification cues that are easy to complete and fade, while making deeper reflection optional or triggered by repeated difficulty [64].

When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

This preserves an authentic workflow while turning “verify before you trust” into a practiced habit [13, 43, 71].

that adding runnable near-misses can make verification and repair a more consistent part of prompt-based programming practice.

5.3

Acknowledgments

Limitations

We acknowledge our findings should be interpreted in light of several limitations and give ideas for clear next steps. First, the study is tied to a specific setting, an introductory C course and a set of five platform problems, out of which only two were required to be solved for credit. Future work should aim to replicate the findings across languages, institutions, and more open-ended tasks to test generalizability. Second, follow-up actions are shaped by the platform design and the kind of feedback students get in a prompt-and-test workflow with hidden tests. Future work should vary the feedback and tools available. For example, it could explain a small counterexample, provide more detailed run-time feedback, let students write their own tests, or add lightweight debugging support, and then check whether the same bug source patterns still hold. Third, the current study did not disclose the source of bugs to students. Future work should examine how different levels of transparency regarding bug injection affect student trust and engagement. Finally, because this was an in-class deployment rather than a controlled comparison between conditions, differences we observe may also reflect course context and platform choices, not only bug source. A natural next step is an A/B study that changes one factor at a time, such as injected bugs on versus off, different injection rates, or editing enabled versus disabled, and evaluates both in-session behavior and later outcomes such as performance on new problems or delayed follow-up tasks. Another important direction is to test whether these workflows lead to measurable learning benefits, for example through controlled pre-test/post-test designs that assess code comprehension, debugging performance, and transfer to new tasks.

Conclusion

In this paper, we studied how introductory programming students respond to bugs while solving prompt-based programming tasks with a GenAI assistant. Across 2,636 analyzed sessions and 6,071 turns where a bug needed to be fixed, we found that students’ next actions are strongly shaped by the type of bug they encounter. Natural bugs are more often associated with specification-focused followup work, while injected near-miss bugs are more often associated with inspecting and repairing code. This difference is also visible in prompt content, which shifts between clarifying task intent and constraints versus providing more detailed guidance for a targeted fix. Students’ reflections further suggest that combining prompting with structured opportunities for review and repair can support verification of GenAI outputs and awareness of GenAI limitations. More broadly, our results suggest that mixing natural failures with realistic runnable near-misses can support both specification refinement and careful code review and debugging within the same workflow. Looking ahead, these findings motivate prompt-based programming activities and platforms that scaffold diagnosis and verification alongside prompting. One direction is to add lightweight cues that help learners clarify the task after repeated mismatch signals. Another is to trigger debugging-oriented checks when localized defects are more likely, for example asking for a brief hypothesis, a suspect location, or a small validating test. Overall, our findings suggest

This work was supported by Research Council of Finland grant #356114. Funded/Cofunded by the European Union (ERC, TOPS, 101039090). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

References [1] Vincent Aleven, Elmar Stahl, Silke Schworm, Frank Fischer, and Raven Wallace. 2003. Help Seeking and Help Design in Interactive Learning Environments. Review of Educational Research (2003). [2] Leonardo Banh, Florian Holldack, and Gero Strobel. 2025. Copiloting the Future: How Generative AI Transforms Software Engineering. Information and Software Technology (2025). [3] Elizabeth L Bjork and Robert A Bjork. 2011. Making Things Hard on Yourself, But in a Good Way: Creating Desirable Difficulties to Enhance Learning. Psychology and the Real World: Essays Illustrating Fundamental Contributions to Society (2011). [4] Robert A. Bjork. 1994. Memory and Metamemory Considerations in the Training of Human Beings. In Metacognition: Knowing about Knowing. MIT Press. [5] Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative Research in Psychology (2006). [6] Nikitha Donekal Chandrashekar, Sehrish Basir Nizamani, Margaret Ellis, and Naren Ramakrishnan. 2026. Demystify, Use, Reflect: Preparing Students To Be Informed LLM-users. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [7] Michelene TH Chi, Nicholas De Leeuw, Mei-Hung Chiu, and Christian LaVancher. 1994. Eliciting Self-explanations Improves Understanding. Cognitive science (1994). [8] Heeryung Choi, Tung Phung, Mengyan Wu, Adish Singla, and Christopher Brooks. 2025. Reflection-Satisfaction Tradeoff: Investigating Impact of Reflection on Student Engagement with AI-Generated Programming Hints. CoRR abs/2512.04630 (2025). [9] Victoria Clarke and Virginia Braun. 2017. Thematic Analysis. The Journal of Positive Psychology (2017). [10] Gergely Márk Csányi, István Üveges, Dorina Lakatos, Dóra Ripszám, Kornélia Kozák, Dániel Nagy, and János Pál Vadász. 2025. Sentence-Level Rhetorical Role Labeling in Judicial Decisions. Big Data and Cognitive Computing (2025). [11] Mehmet Arif Demirtas, Max Fowler, Nicole Hu, and Kathryn I. Cunningham. 2024. Validating, Refining, and Identifying Programming Plans Using Learning Curve Analysis on Code Writing Data. In Proceedings of the Conference on International Computing Education Research (ICER). [12] Paul Denny et al. 2024. Computing Education in the Era of Generative AI. Commun. ACM (2024). [13] Paul Denny, Sumit Gulwani, Neil T. Heffernan, Tanja Käser, Steven Moore, Anna N. Rafferty, and Adish Singla. 2024. Generative AI for Education (GAIED): Advances, Opportunities, and Challenges. CoRR abs/2402.01580 (2024). [14] Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2023. Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators. CoRR abs/2307.16364 (2023). [15] Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2024. Prompt Problems: A New Programming Exercise for the Generative AI Era. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [16] Stephen H. Edwards. 2004. Using Software Testing to Move Students from Trialand-Error to Reflection-in-Action. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [17] Anastasia Efklides. 2011. Interactions of Metacognition With Motivation and Affect in Self-Regulated Learning: The MASRL Model. Educational Psychologist (2011). [18] Sue Fitzgerald, Gary Lewandowski, Renée McCauley, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: Finding, Fixing and Flailing, a Multi-institutional Study of Novice Debuggers. Computer Science Education 18 (2008). [19] Sharon Nelson-Le Gall. 1981. Help-seeking: An Understudied Problem-solving Skill in Children. Developmental Review (1981). [20] John Hattie and Helen Timperley. 2007. The Power of Feedback. Review of Educational Research (2007).

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

[21] Xinying Hou, Ruiwei Xiao, Runlong Ye, Michael Liut, and John C. Stamper. 2026. Exploring Student Choice and the Use of Multimodal Generative AI in Programming Learning. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [22] Manu Kapur. 2008. Productive Failure. Cognition and Instruction (2008). [23] Irvin R. Katz and John R. Anderson. 1987. Debugging: An Analysis of BugLocation Strategies. Human-Computer Interaction (1987). [24] Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the Effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). [25] Majeed Kazemitabaar, Xinying Hou, Austin Z. Henley, Barbara Jane Ericson, David Weintrop, and Tovi Grossman. 2023. How Novices Use LLM-based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment. In Proceedings of the Koli Calling International Conference on Computing Education Research (Koli Calling). [26] Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Zachary Henley, Paul Denny, Michelle Craig, and Tovi Grossman. 2024. CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). [27] Nina Keith and Michael Frese. 2008. Effectiveness of Error Management Training: A Meta-analysis. Journal of Applied Psychology (2008). [28] Chris Kerslake, Paul Denny, David H. Smith, James Prather, Juho Leinonen, Andrew Luxton-Reilly, and Stephen MacNeil. 2024. Integrating Natural Language Prompting Tasks in Introductory Programming Courses. In Proceedings of the Virtual Global Computing Education Conference (SIGCSE Virtual). [29] Amy J. Ko and Brad A. Myers. 2008. Debugging Reinvented: Asking and Answering Why and Why Not Questions About Program Behavior. In Proceedings of the International Conference on Software Engineering (ICSE). [30] Nachiket Kotalwar, Alkis Gotovos, and Adish Singla. 2024. Hints-in-browser: Benchmarking Language Models for Programming Feedback Generation. In Annual Conference on Neural Information Processing Systems (NeurIPS). [31] Klaus Krippendorff. 2011. Computing Krippendorff’s Alpha-Reliability. [32] Klaus Krippendorff. 2018. Content Analysis: An Introduction to Its Methodology. SAGE Publications. [33] Juho Leinonen, Arto Hellas, Sami Sarsa, Brent N. Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using Large Language Models to Enhance Programming Error Messages. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [34] Evanfiya Logacheva, Arto Hellas, James Prather, Sami Sarsa, and Juho Leinonen. 2024. Evaluating Contextually Personalized Programming Exercises Created with Generative AI. In Proceedings of the Conference on International Computing Education Research (ICER). [35] Qianou Ma, Hua Shen, Kenneth R. Koedinger, and Sherry Tongshuang Wu. 2024. How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging. In Proceedings of the Artificial Intelligence in Education (AIED). [36] Stephen MacNeil, Juho Leinonen, Paul Denny, Natalie Kiesler, Arto Hellas, James Prather, Brett A. Becker, Michel Wermelinger, and Karen Reid. 2024. Discussing the Changing Landscape of Generative AI in Computing Education. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [37] Renée McCauley, Sue Fitzgerald, Gary Lewandowski, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: A Review of the Literature from an Educational Perspective. Computer Science Education (2008). [38] Tilman Michaeli and Ralf Romeike. 2019. Improving Debugging Skills in the Classroom: The Effects of Teaching a Systematic Debugging Process. In Proceedings of the Workshop in Primary and Secondary Computing Education (WiPSCE). [39] Ran Mo, Dongyu Wang, Wenjing Zhan, Yingjie Jiang, Yepeng Wang, Yuqi Zhao, Zengyang Li, and Yutao Ma. 2025. Assessing and Analyzing the Correctness of GitHub Copilot’s Code Suggestions. ACM Transactions on Software Engineering and Methodology (2025). [40] Laurie Murphy, Gary Lewandowski, Renée McCauley, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: the Good, the Bad, and the Quirky – A Qualitative Analysis of Novices’ Strategies. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [41] Kimberly A. Neuendorf. 2002. The Content Analysis Guidebook. SAGE Publications. [42] Manh Hung Nguyen, Victor-Alexandru Padurean, Alkis Gotovos, Sebastian Tschiatschek, and Adish Singla. 2025. Synthesizing High-Quality Programming Tasks with LLM-Based Expert and Student Agents. In Proceedings of the Artificial Intelligence in Education (AIED). [43] Julian Oertel, Jil Klünder, and Regina Hebig. 2025. Don’t Settle for the First! How Many GitHub Copilot Solutions Should You Check? Information and Software Technology (2025). [44] Eng Lieh Ouh, Kar Way Tan, Siaw Ling Lo, and Benjamin Kok Siew Gan. 2025. Evaluating ChatGPT to Answer Multi-Modal Exercises in Computer Science Education. In Proceedings of the Innovation and Technology in Computer Science Education Conference (ITiCSE).

Victor-Alexandru Pădurean et al.

[45] Stack Overflow. 2024. Stack Overflow Developer Survey: AI section. https: //survey.stackoverflow.co/2024/ai. [46] Victor-Alexandru Padurean, Paul Denny, Alkis Gotovos, and Adish Singla. 2025. Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Models. In Proceedings of the Innovation and Technology in Computer Science Education Conference (ITiCSE). [47] Victor-Alexandru Padurean, Paul Denny, and Adish Singla. 2025. BugSpotter: Automated Generation of Code Debugging Exercises. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [48] Rebecca J. Passonneau. 2006. Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation. In Proceedings of the Conference on Language Resources and Evaluation (LREC). [49] Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models. In Proceedings of the International Conference on Educational Data Mining (EDM). [50] Tung Phung, Victor-Alexandru Padurean, Anjali Singh, Christopher Brooks, José Cambronero, Sumit Gulwani, Adish Singla, and Gustavo Soares. 2024. Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation. In Proceedings of the International Learning Analytics and Knowledge Conference (LAK). [51] Chanathip Pornprasit and Chakkrit Tantithamthavorn. 2024. Fine-Tuning and Prompt Engineering for Large Language Models-Based Code Review Automation. Information and Software Technology (2024). [52] James Prather et al. 2023. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education. In Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR). [53] James Prather et al. 2024. Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools. In Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR). [54] James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett B. Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2024. "It’s Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers. ACM Transactions on Computer-Human Interaction (2024). [55] James Prather, Brent N. Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S. Randrianasolo, Brett A. Becker, Bailey Kimmel, Jared Wright, and Ben Briggs. 2024. The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. In Proceedings of the Conference on International Computing Education Research (ICER). [56] Christian Rahe and Walid Maalej. 2025. How Do Programming Students Use Generative AI? Proceedings of the ACM on Software Engineering (2025). [57] Brent N. Reeves, James Prather, Paul Denny, Juho Leinonen, Stephen MacNeil, Andrew Luxton-Reilly, Sebastian Mateos Nicolajsen, and Claus Brabrand. 2025. Prompts First, Precision Later: Reviving the Vision of Natural Language Programming for Computing Education. In Proceedings of the Koli Calling International Conference on Computing Education Research (Koli Calling). [58] Brian J. Reiser. 2004. Scaffolding Complex Learning: The Mechanisms of Structuring and Problematizing Student Work. Journal of the Learning Sciences (2004). [59] Jake Renzella, Alexandra Vassar, Lorenzo Lee Solano, and Andrew Taylor. 2025. Compiler-Integrated, Conversational AI for Debugging CS1 Programs. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [60] Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence?. In Proceedings of the Conference of the Psychology of Programming Interest Group (PPIG). [61] Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the Conference on International Computing Education Research (ICER). [62] Jaakko J. Sauvola, Sasu Tarkoma, Mika Klemettinen, Jukka Riekki, and David S. Doermann. 2024. Future of Software Development with Generative AI. Automated Software Engineering (2024). [63] Andreas Scholl and Natalie Kiesler. 2025. SCRIPT - Supportive Chatbot for Resolving Introductory Programming Tasks. In Proceedings of the Innovation and Technology in Computer Science Education Conference (ITiCSE). [64] Valerie J Shute. 2008. Focus on Formative Feedback. Review of Educational Research (2008). [65] Debra Steele-Johnson and Zachary T Kalinoski. 2014. Error Framing Effects on Performance: Cognitive, Motivational, and Affective Pathways. The Journal of Psychology (2014). [66] Steve Stemler. 2000. An Overview of Content Analysis. Practical Assessment, Research, and Evaluation (2000). [67] John Sweller. 1988. Cognitive Load During Problem Solving: Effects on Learning. Cognitive science (1988). [68] Florian Tambon, Arghavan Moradi-Dakhel, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Giuliano Antoniol. 2025. Bugs in Large Language

When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code

Models Generated Code: An Empirical Study. Empirical Software Engineering (2025). [69] Norbert Tihanyi, Tamás Bisztray, Mohamed Amine Ferrag, Ridhi Jain, and Lucas C. Cordeiro. 2025. How Secure Is AI-Generated Code? A Large-Scale Comparison of Large Language Models. Empirical Software Engineering (2025). [70] Annapurna Vadaparty, Daniel Zingaro, David H. Smith IV, Mounika Padala, Christine Alvarado, Jamie Gorson Benario, and Leo Porter. 2024. CS1-LLM: Integrating LLMs into CS1 Instruction. In Proceedings of the Innovation and Technology in Computer Science Education Conference (ITiCSE). [71] Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). [72] Olga Viberg, Jacqueline Wong, Yael Feldman-Maggor, Nora Dunder, and Carrie Demmans Epp. 2025. Chatting with Code: Exploring LLMs as Learning Partners in Programming Education. In Proceedings of the Artificial Intelligence in Education (AIED). [73] Sierra Wang, John C. Mitchell, and Chris Piech. 2024. A Large Scale RCT on Effective Error Messages in CS1. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [74] Philip H. Winne and Allyson F. Hadwin. 1998. Studying as Self-Regulated Learning. In Metacognition in Educational Theory and Practice. Lawrence Erlbaum Associates, 277–304.

ICER 2026 Vol. 1, August 11–14, 2026, Uppsala, Sweden

[75] Stephanie Yang, Miles Baird, Eleanor O’Rourke, Karen Brennan, and Bertrand Schneider. 2024. Decoding Debugging Instruction: A Systematic Literature Review of Debugging Interventions. ACM Transactions on Computing Education (2024). [76] Stephanie Yang, Hanzhang Zhao, Yudian Xu, Karen Brennan, and Bertrand Schneider. 2024. Debugging with an AI Tutor: Investigating Novice Help-seeking Behaviors and Perceived Learning. In Proceedings of the Conference on International Computing Education Research (ICER). [77] Thomas Y. Yeh, Karena Tran, Ge Gao, Tyler Yu, Wai On Fong, and Tzu-Yi Chen. 2025. Bridging Novice Programmers and LLMs with Interactivity. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE). [78] Jialu Zhang, José Pablo Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Verbruggen. 2024. PyDex: Repairing Bugs in Introductory Python Assignments using LLMs. Proceedings of the ACM on Programming Languages (2024). [79] Yangtian Zi, Luisa Li, Arjun Guha, Carolyn Jane Anderson, and Molly Q. Feldman. 2025. "I Would Have Written My Code Differently’: Beginners Struggle to Understand LLM-Generated Code. In Proceedings of the Conference on the Foundations of Software Engineering (FSE). [80] Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2024. Measuring GitHub Copilot’s Impact on Productivity. Commun. ACM (2024). [81] Barry J. Zimmerman. 2002. Becoming a Self-Regulated Learner: An Overview. Theory Into Practice (2002).

Record · ID 343534 · SHA-256 3f34431f610e5cf9
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.