DebugTracker: Lightweight Process Evidence for Classroom Debugging

DebugTracker: Lightweight Process Evidence for Classroom Debugging Jiatong Liu

Monash University Melbourne, Australia [email protected]

Zehua Zhang

Monash University Melbourne, Australia [email protected]

arXiv:2607.05871v1 [cs.SE] 7 Jul 2026

Abstract Debugging exercises are often assessed from final code and test outcomes, yet these artifacts hide how students reproduced failures, formed hypotheses, inspected evidence, edited code, and verified fixes. We present DebugTracker, a Visual Studio Code extension that records lightweight debugging-process evidence for classroom tasks. DebugTracker separates uncoached Evaluation Mode traces from coached Training Mode traces, stores append-only JSONL events, and exports timeline and Markdown reports for human review. The prototype records test commands, editor and debugger metadata, student checkpoints, source snapshots, optional image evidence, human labels, and optional AI-assisted practice feedback. DebugTracker is largely language-agnostic: it captures process evidence through standard VS Code mechanisms rather than language-specific tooling, although debugger evidence depends on the relevant VS Code language extension. We validate the prototype with debugging tasks in Python, TypeScript, and Java, 16 automated checks, and an 11-case manual trial matrix spanning packaged VSIX installation and three operating systems.

CCS Concepts • Software and its engineering → Software testing and debugging; • Social and professional topics → Software engineering education.

Keywords debugging, software testing education, VS Code extension, tool demonstration

1

Introduction

Debugging is a process of observing failures, asking questions, inspecting state, testing hypotheses, and validating repairs [9, 12]. It is also one of the hardest skills for students to learn because the reasoning that separates a systematic investigation from lucky guessing happens between the failing test and the final patch. Classroom assessment, however, commonly sees only that final patch, the final test result, or a short written note. The intermediate steps— reproducing the failure, narrowing it to a suspect location, forming a hypothesis, and checking that the repair fixes the observed symptom—leave no durable record once the session ends.

Xue Yao

Monash University Melbourne, Australia [email protected]

Yongqiang Tian

Monash University Melbourne, Australia [email protected] This creates a process-visibility gap in assessment. Viewed from the student’s side, a debugging exercise unfolds as the rich, multistep process just described. Viewed from the instructor’s or examiner’s side, however, the available artifact is often only the submitted code plus a pass/fail test result. Consider two students who both submit a passing solution: one may have followed an evidencebased debugging process, while the other may have edited and tested repeatedly until the suite turned green. The final code can look identical even when the underlying debugging competence differs. When the goal of the exercise is to teach a process, feedback that ignores the process misses what the exercise was meant to assess. Richer alternatives exist but fit the classroom poorly. Screen recording and live observation capture the process faithfully, yet they do not scale to large cohorts, are intrusive, and collect far more data—faces, unrelated windows, keystrokes—than a review task requires. Commit history is too coarse and is easily rewritten, so it rarely reflects the moment-to-moment investigation. Automated debugging and fault-localization tools target the act of finding bugs, not the assessment of how a learner did so, and prior work cautions that such automation does not straightforwardly help programmers in practice [7]. Classroom assessment therefore needs a lightweight, structured record of the debugging process: one that is cheap to produce, cheap to read, and bounded to the task at hand. DebugTracker targets this gap with a task-scoped IDE tool. Its goal is not to replace debuggers, automated fault localization, or interactive explanation systems such as the Whyline [4]; nor does it claim that automated debugging alone can assess student practice. Instead, it makes the evidence surrounding ordinary tests, breakpoints, edits, checkpoints, and verification inspectable after a session ends, so an instructor can review the debugging narrative rather than only its outcome. DebugTracker is designed for three users: students who need a lightweight way to record their debugging narrative, instructors and teaching assistants who need reviewable evidence, and researchers who need structured traces without full screen capture. The paper contributes (1) a VS Code extension implementing this workflow, (2) a privacy-conscious event and report model for debugging-process evidence, and (3) a validation package with cross-language sample tasks, automated tests, a demo script, and a packaged VSIX build. A demonstration video is available at https://youtu.be/-hzR-Whu2s4.

ISSTA ’26, October 3–9, 2026, Oakland, CA, USA

Jiatong Liu, Xue Yao, Zehua Zhang, and Yongqiang Tian

Training Mode coached practice

Evaluation Mode uncoached assessment

Student debugging

Run tests

Observe failure

Hypothesize

Inspect code

Verify

Instructor review

Evidence package

Event log

Edit fix

Timeline

Snapshots/ images

Report

Checklist labels

Figure 1: DebugTracker workflow from session mode to reviewable evidence. A session begins in one of two mutually exclusive modes (top). Evaluation Mode (left, blue) records an uncoached assessment trace: hints, solution prompts, and after-session feedback are disabled. Training Mode (right, orange) records coached practice: process prompts and optional AI feedback are enabled, but direct patches are still disallowed. Both modes drive the same debugging path (middle): run tests, observe a failure, inspect code, state a hypothesis, edit the fix, and verify the result. DebugTracker captures this path as an event log, timeline, report, snapshots or images, and human labels (bottom), giving instructors a process-oriented basis for feedback while keeping assessment traces and coached practice on separate, clearly labeled tracks.

2

DebugTracker

Figure 1 summarizes the workflow that DebugTracker makes visible. The first choice is the session mode: Evaluation Mode produces an uncoached assessment trace, whereas Training Mode records coached practice. Within either mode the student carries out the same ordinary debugging routine inside VS Code—the path shown across the middle of Figure 1, from running tests to verifying the fix (we walk a concrete instance in Section 3). From the instructor’s perspective, those same actions become a structured evidence package: an event log, timeline, report, snapshots or images, and optional human labels. The command surface is deliberately small: start an Evaluation session, start a Training session, add a checkpoint—a timestamped failure observation, hypothesis, or verification note—and finish. The two modes differ only in coaching. Evaluation Mode is uncoached: it disables process prompts, solution hints, and after-session feedback, and marks its reports as assessment traces. Training Mode is coached: it prompts the student through the debugging process— for example, reminding them to reproduce the failure, record a hypothesis before editing, and verify the fix afterward—and can add optional, OpenAI-compatible feedback on how the student is debugging. This coaching deliberately targets the process, not the answer: it never reveals the fix or patches the code for the student. The explicit mode boundary keeps these coached practice traces separate from assessment evidence.

At finish time, DebugTracker exports a Markdown report and a timeline that together summarize the session: its mode and privacy boundary, the detected tests and failures, the student’s hypotheses and edits, and the post-edit verification. Instructors can add human labels afterward, and those labels are stored as separate events rather than overwriting the raw trace.

3

Demonstration

This section describes the live demonstration: a single scripted debugging session that an audience can follow end to end, and the assessment evidence it leaves behind. The demonstration centers on the included Python checkoutpricing task; TypeScript and Java variants are also available. The task starts from a failing test whose bug is small enough to fix on stage but still demands a genuine debugging chain: the shippingeligibility check uses the original subtotal instead of the discounted taxable subtotal, so a discounted order is incorrectly granted free shipping. Table 1 lists the script and, for each step, the assessment question that the step’s recorded evidence lets an instructor answer. The presenter installs the VSIX (or launches the extension from source), opens the sample task, selects the Evaluation branch of Figure 1, and runs python -m unittest. Once the failing test appears, the presenter follows the student path from the figure: record the observed failure, inspect src/pricing.py, state the hypothesis

DebugTracker: Lightweight Process Evidence for Classroom Debugging

# DebugTracker Student Summary (Evaluation) Recorded: test yes; checkpoint yes; edit yes; snapshots yes. Timeline: 02:34 unittest -> 02:35 failure -> hypothesis -> 02:36 edit src/pricing.py -> 02:37 verification. {"type":"failure.observed","source":"student","mode":"evaluation", "summary":"discounted␣order␣wrongly␣gets␣free␣shipping", "payload":{"checkpointKind":"failure"}} {"type":"student.hypothesis","source":"student","mode":"evaluation", "summary":"free-shipping␣threshold␣uses␣original,␣not␣discounted,␣ subtotal", "payload":{"checkpointKind":"hypothesis"}} {"type":"verification.recorded","source":"test","mode":"evaluation", "summary":"unittest␣passes␣after␣fix","payload":{"checkpointKind":" verification"}}

Figure 2: Exported Markdown report excerpt and syntaxhighlighted JSONL event excerpts from the Python checkout-pricing evaluation task. that the free-shipping threshold uses the wrong subtotal, edit the shipping-basis assignment, rerun the test, and record verification. A final step opens the timeline and Markdown report; Figure 2 shows an excerpt, including the underlying JSONL events whose schema is defined in Section 4. The payoff is the trace, not the fix. Reading only the final patch, the two students contrasted in the introduction are indistinguishable; reading the exported trace, an instructor can answer every question in the last column of Table 1—whether the failure was reproduced, whether the suspect code was inspected, whether the hypothesis preceded the edit, whether the edit was targeted, whether verification followed it, and whether any coaching was involved— without having watched the session. Table 1: Demonstration script and the instructor checks enabled by each step. Demo step

Captured evidence

Start session

Mode, purpose, privacy boundary, Coaching used? and session id.

Run failing test

Test command and failure context.

Failure reproduced?

Inspect code

Editor navigation and debugger activity.

Suspect code inspected?

Record hypothesis

Pre-edit failure note and hypothesis.

Hypothesis before edit?

Edit fix

Source snapshot and edit metadata.

Targeted repair?

Verify and export

Passing test event, timeline, report, and optional labels.

Verification after edit?

ISSTA ’26, October 3–9, 2026, Oakland, CA, USA

Because the questions in Table 1 map onto common debugging rubrics, they can attach rubric labels through the human-label mechanism; these are stored as later events, leaving the student’s original trace intact.

4

Design and Implementation

DebugTracker is implemented in TypeScript as a VS Code extension targeting VS Code 1.88 or newer, using the VS Code extension API [6]. It contributes a DebugTracker activity-bar container, a session dashboard, a timeline tree/webview, command-palette actions, and workspace settings for capture and storage policy. All primary artifacts are workspace-local: JSONL session logs, Markdown reports, source snapshots, and optional image attachments. The event schema in src/eventTypes.ts records a session id, timestamp, mode, source, event type, optional file location, a short summary, and a structured payload. Table 2 shows the implemented fields and event families. The log is append-only, so raw behavior, human labels, and AI feedback remain distinguishable during review. Table 2: Implemented DebugTracker event schema. Schema item

Implemented values or purpose

Required fields

id, timestamp, sessionId, type, source, mode, summary, payload. filePath, line, and column when an event is tied to source code. evaluation and training. session, editor, debugger, terminal, test, student, human, ai. Session start/stop, file open/edit, debugger lifecycle (start, stop, step, breakpoint, continue), terminal commands, test runs, source snapshots, failure evidence, image evidence, student checkpoint/hypothesis/note, verification, human labels, AI analysis, and AI coaching. Source-specific structured metadata, such as command lines, checkpoint text, snapshot paths, image attachment paths, and label details.

Optional location Modes Sources Event families

Instructor check

The same artifacts support a lightweight classroom workflow beyond the stage. An instructor publishes a debugging task with its source, tests, and any custom test-command patterns; students open it in VS Code, run an Evaluation session, debug normally, and submit the exported report and trace alongside their final solution. An instructor or teaching assistant then reviews the process evidence directly, without screen recordings or commit-history archaeology.

Payload

Internally, capture is organized as a thin set of listeners over existing VS Code events feeding a single append-only writer. When a session starts, the extension subscribes to editor, debugger, and terminal events, normalizes each one into the schema of Table 2, and appends it to the session’s JSONL log; the timeline tree/webview and the Markdown report are derived views over that same log rather than independent stores. Test runs are detected from shell-integrated terminal commands against built-in and instructorconfigured patterns, debugger activity is observed through the Debug Adapter Protocol, and student checkpoints and human labels enter through explicit command-palette actions. Because every artifact is reconstructed from the ordered event stream, the trace remains the single source of truth: re-rendering a report or timeline never mutates the underlying evidence, and a later instructor sees the same raw sequence the student produced. DebugTracker is language-agnostic by design. Because it captures evidence through these standard VS Code mechanisms rather

ISSTA ’26, October 3–9, 2026, Oakland, CA, USA

than a language-specific toolchain, the same event model applies across languages. Built-in patterns recognize common JavaScript, Python, Java, and C/C++ test commands when shell integration is available, and instructors can extend coverage through the debugTracker.testCommandPatterns setting. The languages used in this paper are therefore a validation set, not a constraint on the tool. DebugTracker treats privacy as a capture-policy problem: it records task-relevant metadata rather than a full interaction replay. By default, it does not store complete terminal output, screen video, mouse movement, or per-keystroke history. For instructor-authored tasks, it saves baseline and final source snapshots so instructors can compare the starting bug with the submitted fix. Reports also separate captured evidence from instructor-only interpretation. Student-facing reports describe what was captured and, in Training Mode, provide practice feedback. Instructor actions come later and never overwrite the trace: human labels are appended as separate events, and instructor-side reports are derived from the log. This keeps the student’s raw assessment trace distinct from the instructor’s interpretation of it.

5

Implementation Validation

This section reports engineering evidence that the implemented workflow installs, runs, and produces the expected evidence across configurations. It validates the artifact rather than the pedagogy: it shows that DebugTracker reliably does what Sections 3 to 4 describe, not that the resulting evidence improves grading. We return to that distinction at the end of the section. The automated test entry point dist/test/runTests.js runs 16 checks that exercise the mechanisms behind those claims: session identifiers, terminal test-command detection, mode policy, report generation, AI-coach prompt construction, image evidence, source-snapshot reporting, timeline rendering, session-summary rendering, human-label entry, and Training Mode feedback. All 16 pass in the current workspace. A manual practice matrix of 11 documented cases covers what unit tests cannot: packaged VSIX installation; Windows, macOS, and Linux task wrappers; Evaluation Mode on the Python, TypeScript, and Java tasks; Training Mode feedback; AI-coach guidance; image evidence; instructor-side report generation; and the missingdebugger warning. The three language tasks deliberately share a single intended bug—free-shipping eligibility computed from the original rather than the discounted taxable subtotal—so that running them confirms DebugTracker records the debugging process through one shared event model rather than a flow specialized to a particular language, exercising the language-agnostic design of Section 4.

6

Related Work

DebugTracker sits at the intersection of debugging education, programming-process analytics, and automated debugging, and it is positioned as a complement to, rather than a replacement for, work in each area. Debugging as a learned skill. Debugging is repeatedly identified as one of the harder competencies for students to acquire. McCauley et al. [5] survey the educational literature and treat debugging as

Jiatong Liu, Xue Yao, Zehua Zhang, and Yongqiang Tian

a distinct skill rather than a by-product of writing code, while the multi-institutional study of Fitzgerald et al. [2] reports that novices oscillate between systematic investigation and ad hoc “flailing.” These accounts align with classic characterizations of debugging as a question-driven, hypothesis-testing activity [9, 12]. Such work motivates assessing the process rather than only the final patch; DebugTracker operationalizes this by recording the reproduce– observe–hypothesize –edit–verify chain as reviewable evidence. Programming-process data. A substantial body of computingeducation research captures fine-grained student activity. Blackbox [1] collects snapshots and compilation events from BlueJ at large scale, Marmoset [10] captures incremental snapshots through a submission and testing system, and ProgSnap2 [8] standardizes a portable format for programming-process data; surveys of educational data mining catalog how such traces are analyzed [3]. These systems are optimized for large-scale, often institution-wide, research collection. DebugTracker differs in granularity and intent: it produces a task-scoped, human-readable narrative for individual review, records task-relevant metadata rather than full keystroke replay, and keeps coached and uncoached traces separate. Automated debugging and explanation. Interactive systems such as the Whyline [4] let developers ask why and why-not questions about program behavior, and automated fault localization has been surveyed extensively [11]. Parnin and Orso [7], however, caution that such automation does not straightforwardly help programmers in practice. DebugTracker is complementary to this line of work: it does not localize faults or explain behavior, but instead makes the surrounding human investigation inspectable so that an instructor can judge how a student debugged.

7

Tool Availability

DebugTracker is distributed as a self-contained artifact. It contains the VS Code extension source, a prebuilt VSIX installer, crosslanguage sample tasks, a demo script, student and instructor guides, reproducibility notes, and a practice test report. The extension targets VS Code 1.88 or newer and installs from the VSIX without additional services; the optional AI coach is off by default and requires only an OpenAI-compatible endpoint when explicitly enabled. The source repository is available at https://github.com/t3research/DebugTracker, a demonstration video is available at https: //youtu.be/-hzR-Whu2s4, and an archived snapshot of the artifact is available at https://doi.org/10.5281/zenodo.20955037. The artifact is released under CC BY-NC-SA 4.0.

8

Conclusion

DebugTracker demonstrates a practical middle ground between final-code-only grading and intrusive screen capture. It records enough structured evidence for instructors to review how a student debugged while keeping coached practice, human labels, and raw assessment traces separate. By turning ordinary tests, breakpoints, edits, checkpoints, and verification into a reviewable, privacy-conscious trace, it makes the debugging process—not just its outcome—something an instructor can see, discuss, and assess. Future work will study DebugTracker in real classrooms, measuring whether process traces improve review efficiency, inter-instructor agreement, and the specificity of debugging feedback.

DebugTracker: Lightweight Process Evidence for Classroom Debugging

References [1] Neil Christopher Charles Brown, Michael Kölling, Davin McCall, and Ian Utting. 2014. Blackbox: A Large Scale Repository of Novice Programmers’ Activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE ’14). ACM, New York, NY, USA, 223–228. doi:10.1145/2538862.2538924 [2] Sue Fitzgerald, Gary Lewandowski, Renee McCauley, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: Finding, Fixing and Flailing, a Multi-Institutional Study of Novice Debuggers. Computer Science Education 18, 2 (2008), 93–116. doi:10.1080/08993400802114508 [3] Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, Jürgen Börstler, Stephen H. Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, Miguel Ángel Rubio, Judy Sheard, Bronius Skupas, Jaime Spacco, Claudia Szabo, and Daniel Toll. 2015. Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies. In Proceedings of the 2015 ITiCSE on Working Group Reports (ITiCSE-WGR ’15). ACM, New York, NY, USA, 41–63. doi:10.1145/2858796.2858798 [4] Andrew J. Ko and Brad A. Myers. 2004. Designing the Whyline: A Debugging Interface for Asking Questions about Program Behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’04). ACM, New York, NY, USA, 151–158. doi:10.1145/985692.985712 [5] Renee McCauley, Sue Fitzgerald, Gary Lewandowski, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: A Review of the Literature from an Educational Perspective. Computer Science Education 18, 2 (2008), 67–92. doi:10.1080/08993400802114581 [6] Microsoft. 2026. Visual Studio Code Extension API. https://code.visualstudio. com/api. Accessed 2026-06-24.

ISSTA ’26, October 3–9, 2026, Oakland, CA, USA

[7] Chris Parnin and Alessandro Orso. 2011. Are Automated Debugging Techniques Actually Helping Programmers?. In Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA ’11). ACM, New York, NY, USA, 199–209. doi:10.1145/2001420.2001445 [8] Thomas W. Price, David Hovemeyer, Kelly Rivers, Ge Gao, Austin Cory Bart, Ayaan M. Kazerouni, Brett A. Becker, Andrew Petersen, Luke Gusukuma, Stephen H. Edwards, and David Babcock. 2020. ProgSnap2: A Flexible Format for Programming Process Data. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ’20). ACM, New York, NY, USA, 356–362. doi:10.1145/3341525.3387373 [9] Jonathan Sillito, Gail C. Murphy, and Kris De Volder. 2006. Questions Programmers Ask during Software Evolution Tasks. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT FSE ’06). ACM, New York, NY, USA, 23–34. doi:10.1145/1181775.1181779 [10] Jaime Spacco, David Hovemeyer, William Pugh, Fawzi Emad, Jeffrey K. Hollingsworth, and Nelson Padua-Perez. 2006. Experiences with Marmoset: Designing and Using an Advanced Submission and Testing System for Programming Courses. In Proceedings of the 11th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE ’06). ACM, New York, NY, USA, 13–17. doi:10.1145/1140124.1140131 [11] W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A Survey on Software Fault Localization. IEEE Transactions on Software Engineering 42, 8 (2016), 707–740. doi:10.1109/TSE.2016.2521368 [12] Andreas Zeller. 2009. Why Programs Fail: A Guide to Systematic Debugging (2 ed.). Morgan Kaufmann, Burlington, MA, USA. https://www.sciencedirect.com/ book/9780123745156/why-programs-fail