Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$

Pdb or Not to Be: Comparing Debugging Interfaces With PassX @k

arXiv:2611.08666v1 [cs.AI] 5 Nov 2026

Ryan Magnuson Code Metal

Ellie Kitanidis Code Metal

Ariel Lee Code Metal

{ryan.magnuson, ellie, ariel.lee}@codemetal.ai

Abstract Interactive debuggers are indispensable to human software development, yet coding agents overwhelmingly rely on edit-execute-observe loops, using common bash tools, test scripts, and print statements to debug via repeated shell commands. Stepping through a debugger provides live access to runtime state, which may enable different problem-solving capabilities. However, prior work typically introduces the debugger as an additional tool or through a new subagent, making its individual contribution difficult to isolate from the associated scaffold changes. To address this, we compare a debugger-only agent against a bash-only agent on all 500 SWE-bench Verified instances, holding the rest of the scaffold fixed. We evaluate both architectures on two models, GPT-OSS-120B and Nemotron-3-Ultra-550B, with three independent runs per architecture × model configuration. Crucially, to distinguish interface specialization from expanded coverage under repeated sampling, we introduce passX @k, a cross-agent pass@k that provides an unbiased estimator of the probability that at least one of k agents is successful. Although the bash and debugger agents solve partially disjoint subsets of the benchmark in any single pair of runs, our passX @2 estimator reveals that this is largely due to run-to-run variance rather than tool complementarity. Across both models, we find that two bash rollouts outperform the combination of one bash and one debugger rollout. Our findings highlight the importance of accounting for stochastic variance in evaluating agent and tool diversity, and we hope that passX @k will serve as a useful framework for the community.

1

Introduction

Most coding agents, including production-grade assistants [1, 15, 6], debug programs primarily through a shell: inserting print statements, executing code, and parsing outputs or exception stack traces. Although several works have demonstrated benefits from execution state inspection for agentic program repair [27, 20, 18], live debuggers remain underexplored as an observation and executioncontrol interface, despite offering a direct path to runtime state information—live stack frames, bound values, and controlled stepping—that shell-only workflows typically must reconstruct through instrumentation and run artifacts. A central question is whether this interface enables qualitatively different debugging capabilities, or merely provides an alternative way to access information that a shell-only agent can already recover. Existing debugger-enabled agents typically attach the debugger to an existing scaffold by incorporating it as a first-class tool [24, 10, 9] or as part of a multi-agent system [19, 5, 7]. Because the debugger is additive in these studies, it is difficult to disentangle the value of the debugger’s runtime evidence from the complex impact that expanding the tool-call action space or multi-agent framework can have on the policy as a whole [21]. Comparative studies are further limited by the fact that agent performance is commonly reported for a single rollout [26, 19, 5], Workshop on Who Verifies the Agents? Toward Reliable Agent Development, NeurIPS 2026.

despite the fact that membership in the solved task set can shift dramatically under repeated sampling of the same architecture [11, 2]. To distinguish potential specialization between tools from stochastic variance between runs, we define passX @k, a cross-agent equivalent of pass@k that combines n runs of each of k agents into a single unbiased estimator of their combined solve rate. With this metric in hand, we ask: given a live debugger instead of a shell, how is the agent’s performance impacted, and is there evidence that the two interfaces provide complementary capabilities? To answer this, we compare an agent equipped with an interactive Python debugger (pdb-agent) against one equipped with a general bash tool for debugging (bash-agent) on SWE-bench Verified [8, 13], with the model, prompts, budgets, sandbox, and all other tools held fixed. A single run of each agent misleadingly suggests that pdb-agent solves problems bash-agent misses. However, as Figure 4 shows, the debugger-only agent’s exclusive solves shrink in a repeat analysis with three runs per agent, indicating that much of its apparent specialization is sampling variance. Figure 3 illustrates that two bash-agent rollouts outperform the combination of one bash-agent plus one pdb-agent rollout at every relay budget and under both models evaluated, while an ablation demonstrating that simultaneous access to both tools provides no apparent benefit to bash-agent is deferred to Appendix D. Both the size of the performance gap and the token efficiency of the architecture are model-dependent, as shown in Figure 2, with GPT-OSS paying roughly twice as much per solved problem under pdb. That said, the shell’s generality cuts both ways; an interface that can execute anything is also difficult to contain, and in Section 4.3 we report an early run in which shell access exposed a benchmark leak. Taken together, our results provide a controlled, budgetmatched comparison showing that, for these models, a live debugger neither matches a shell nor adds complementary coverage to one. We also contribute the passX @k estimator, which lets any such cross-agent comparison be made with an unbiased, variance-aware statistic.

2

Related work

Live debuggers for code repair agents. Prior work has demonstrated that exposing internal execution state to an agent, either through captured evidence [27, 18] or agent-queryable interfaces [20], can improve its performance at program repair. Recently, several studies have explored integrating conventional suspended-process debuggers into agent scaffolds. Debug-gym [24] compares an agent with ordinary code execution capabilities against the same agent with additional pdb capabilities, finding that even the strongest models struggle to fully leverage pdb commands. A concurrent work to ours, SWE-PDB [9], also reports low utilization of runtime inspection commands in base models with pdb access, while training significantly improves both effective tool use and performance. Debug2Fix [5] retains a shell-capable repair agent and delegates pdb or JDB interaction to a specialized subagent equipped with a fixed set of debugger tools. Its ablations show that exposing the debugger tools directly to the main agent is ineffective, whereas delegation improves repair performance. SWEDoctor [7] uses a dedicated debugging agent to produce structured runtime diagnoses that guide a separate patch-generation agent. InspectCoder [19] similarly separates runtime diagnosis from patch generation, assigning pdb inspection and runtime perturbation to a specialized inspector agent. The authors report 33 benchmark instances solved by InspectCoder but by none of four baselines lacking live debugger access, though this comparison is based on a single trajectory per system, making it unclear if the exclusivity of these solves is robust to stochastic variation across independent runs. Tool and agent diversity. Several recent works have proposed frameworks to take advantage of tool diversity through heterogeneous tool-use strategies [4] or multi-tool aggregation [23]. However, the effectiveness of an added tool depends not only on the new capabilities it provides, but also on how each tool is exposed and presented to the agent. [21] examines six different tool architectures with similar underlying capabilities and finds substantial differences in agent behavior, efficiency, and consistency between architectures. Our work provides an orthogonal analysis: we compare two tools with potentially different capabilities in an otherwise shared agent harness. Since each tool defines a distinct agent configuration in this setting, our analysis also relates to studies of agent diversity. [26] compares the joint performance of k different coding agents against k runs of the same coding agent on SWE-bench Lite, and argues that inter-agent diversity leads to more uniquely solved instances than intra-agent sampling. However, the collective inter-agent performance of the k agents relies on a single public leaderboard submission per agent rather than combining repeated independent runs per agent into an unbiased estimator. 2

Using pass@k for agent trajectories. Pass@k [3], originally defined to evaluate single-shot code generation, estimates the probability that at least one of k sampled candidates is correct. In an agentic setting, each candidate is a complete interaction trajectory rather than a single completion. Repeated sampling can therefore be used to characterize performance in long-horizon software-engineering agents [22, 17, 2, 25]. [25] furthermore introduces pass@(k, T ), which generalizes agentic pass@k to condition also on interaction depth T . We define a similar depth-conditioned estimator in Section 3.3 and extend the pass@k formalism to describe the concept of a cross-agent passX @k.

3

Methods and experiments

3.1

Experimental setting

We evaluate on all 500 instances of SWE-bench Verified, the human-validated subset of SWE-bench [8]. Each instance specifies a repository at a base commit, an issue description, and two test sets: FAIL_TO_PASS (F2P), which the repair must fix, and PASS_TO_PASS (P2P), which must not regress. Test files are restored to their benchmark state before grading to prevent solutions through test modification. We retain known problematic instances and count them as failures rather than filtering them [16]. We select GPT-OSS-120B [14] and Nemotron-3-Ultra-550B [12] as open-weight models we can deploy in secure environments. Each model is evaluated through vLLM using its default sampling configurations and medium reasoning effort. Each rollout executes in an isolated Podman container initialized from the same SWE-bench base repository state with git metadata sanitized to prevent access to the reference patch or commit history. Full provisioning, isolation, and infrastructure details are provided in Appendix C. 3.2

Agent architectures and rollout protocol

We construct two matched agent architectures, bash-agent and pdb-agent, on a common harness, illustrated in Figure 1. Across each architecture × model arm, we hold fixed the sampling configuration, budget, prompt scaffold, repository snapshot, sandboxes, a shared substrate of tools for navigation, testing, and editing, and a precomputed codebase map generated once per instance. The bash-agent executes arbitrary shell commands through run_bash. The prompt requests observation before editing, but the harness does not enforce this constraint. The pdb-agent has no arbitrary shell access. It retains restricted Python execution for setup and interacts with a persistent debugger. Following the tool-limiting strategy of Debug2Fix [5], which disables file editing until the debugger has been called at least once, the pdb-agent’s replace_text is unavailable until at least one debugger command succeeds in the current relay. We adopted the gate after early pdb runs showed a recurring failure mode: the agent edited files without ever inspecting program state, then thrashed between the failing tests and successive blind edits until the relay budget was exhausted. Requiring one successful debugger observation before the first edit of each relay reduced this behavior in our pilot runs. We preferred the tool-level gate over stronger prompt instructions because it leaves the shared prompt scaffold lean and identical across arms. The bash-agent has no enforced equivalent beyond prompting. Each rollout consists of at most 10 relays, each containing at most 50 agent turns. At a relay boundary, a handoff step summarizes the current progress and initializes a fresh model context. As with standard compaction, workspace edits persist, but message history does not. We reinstate the edit gate at each relay in an effort to help prevent the flailing blind edits in the pdb-agent. 3.3

Evaluating and comparing agents

For agent (architecture × model) m, task j, and run i ∈ {1, . . . , n}, we let Cijm = 1{run i of agent m is correct on task j}

(1)

and define the usual pass@k estimator, ◊ =1− pass@k

n n−cjm
X k
, cjm = Cijm , n k i=1

3

(2)

Figure 1: Controlled execution interface comparison. The two upper bands are identical across arms: models, sampling, prompts, relay and turn budgets, sandbox, repository snapshot, and a shared substrate of navigation, testing, and editing tools. Only the execution interface differs. The bash-agent runs arbitrary shell commands. The pdb-agent trades the shell for a debugger session plus a scoped script runner for setup, and its edit tools stay locked until one debugger command succeeds in the current relay. The gate is the one enforced constraint in the study, added after pilot runs showed the pdb arm editing blindly without ever inspecting state. which estimates the probability that at least one out of k runs of agent m will succeed on task j (estimated by sampling n ≥ k runs of the same architecture). To evaluate the benefits of a diverse agent portfolio of k agents against a baseline where the same compute is allocated to k runs of a single agent, we define the cross-pass-at-k estimator, passX @k:

Ÿ pass X @k = 1 −

k Y m=1

n−cjm 1
n 1

=1−

k 1 Y (n − cjm ), nk m=1

(3)

where the second term is the probability that all k agents fail on task j. Appendix E establishes unbiasedness under iid sampling within, and mutual independence between, each agent’s set of runs. In our work, we compare two agents and sample three runs per agent, so n = 31 and k = 2, and the intra- and inter-agent pass@2 are, respectively, 3−cjm 2

◊ pass@2 mm = 1 −

3 (3 − cjm )(3 − cjm′ ) Ÿ pass , X @2mm′ = 1 − 9 where the latter captures the probability that a pair of (bash, pdb) runs will solve task j. 1 Our definition explicitly assumes each agent is sampled an equal number of times.

4

(4) (5)

We can also threshold these statistics by the number of relays r, letting Cijm → Sijm (r): Sijm (r) = 1{run i of agent m is correct on task j and Rijm ≤ r} n X sjm (r) = Sijm (r)

(6) (7)

i=1

where Rijm is the number of relays used by this observed run [25].

4

Results

4.1

Overall performance by model and architecture

The final scores of each model × architecture arm is given in Table 1 and plotted as a function of relay budget in Figure 3. With three independent runs for each of the four arms, we report pass@k and its complement (passˆk, the probability that all k draws succeed) for k = 2 and k = 3. For pass@1, which is equivalent to the mean over the three runs, we also report the standard deviation. Table 1: Correctness rates by model and architecture, averaged over the entire dataset and with a maximum budget of 10 relays. For pass@1 (i.e. the mean over three runs), we also report st. dev. Model

Arch

pass@1

pass@2

pass@3

passˆ2

passˆ3

Nemotron-3-Ultra (n=500)

bash pdb cross

56.7 ± 1.2 49.6 ± 1.4 -

62.7 55.9 60.3

65.4 59.2 -

50.7 43.3 -

47.4 40.4 -

GPT-OSS-120B (n=500)

bash pdb cross

56.7 ± 0.6 40.3 ± 0.8 -

64.7 47.8 60.0

68.8 51.2 -

48.7 32.7 -

44.8 28.6 -

Figure 3 shows substantial performance gains from scaling sampling breadth k, an effect that remains consistent across observed relay depths and all architecture × model arms. Furthermore, bash-agent demonstrates a substantial lead over pdb-agent under both models, which perform comparably under this architecture. However, when only pdb tools are available, Nemotron-3-Ultra substantially outperforms GPT-OSS, and this margin is consistent across sampling breadth and relay depth.

Figure 2: Share of runs solved at or under a given per-run spend threshold for each of the four arms. Analyzing performance as a function of token spend in Figure 2, we see highly model-dependent efficiency curves, which may also inform the observed asymmetry in the pdb vs. bash gap between models. Nemotron’s token spend is greater than GPT-OSS’s (its median solved run costs several times more tokens) and scales similarly for either agent architecture, whereas the debugger agent roughly doubles the token cost per solved problem for GPT-OSS, which is able to exploit the bash tool relatively cheaply but quickly plateaus in performance as the token budget is increased. 5

4.2

Cross-architectural performance by model

We also ask whether the two interfaces are complementary: for a given model, are there classes of problems where a debugger or a shell is the higher-leverage tool? In Figure 3, we see that the combination of a bash run and a pdb run (green triangles) consistently performs worse than two bash runs (red circles), a result that holds for both models and across all relay budgets.

Figure 3: Cumulative solve rate as a function of relay budget. Dotted curves are pass@1, with the shaded envelope spanning the three seeds. Solid curves are pass@2. Two bash rollouts (solid red) dominate the passX @2 of one bash + one pdb rollout (green, dashed) at every relay threshold. The oracle reranker (gray, dotted) unions all six runs at 3× the budget and is plotted as a ceiling. Figure 4 highlights how the subset of instances that are seemingly solved by one agent but not the other shrinks substantially once all three seeds per arm are pooled. We conclude that any apparent specialization is mostly sampling variance between runs rather than a difference in what the two interfaces can solve. As an additional ablation, we test whether pass@1 is affected when adding the debugger alongside the shell, finding it fails to improve performance (see Appendix D). 4.3

Additional observations

Complexity strata. We stratify instances by gold-patch size (files touched and lines changed) as a rough proxy for bug complexity. As discussed in Appendix A, pass rates fall under both arms as patches grow, but the bash–pdb gap shows no consistent trend. Most gold patches are small, with only 22 of the 500 instances in the 3+ file bucket and eight in the >100-line bucket, which limits our ability to conclusively determine whether debuggers help solve more complex bugs. 6

Figure 4: Who solved it. Top row: what a single-seed comparison sees (mean over all nine pairings). Bottom row: comparing the unions of the full three runs per architecture.

Shell access leak. In an early run, the bash agent (Nemotron) used git to traverse the repository’s commit history until it found the gold patch and then applied it. We subsequently sanitized every workspace (remotes removed, all refs but HEAD deleted, reflog expired) and added a tool-level guard on git invocations (see Appendix C). The episode illustrates a structural property of the comparison: an unstructured shell exposes the entire environment, including artifacts the benchmark assumes hidden, so containment must be added piecemeal. Tool-name aliasing. A recurring tool-use failure mode was the model calling a tool by a plausible name it does not have e.g. reaching for search_replace when the registered tool is replace_text. Registering aliases that map the names models actually reach for onto the intended tool converted these failed calls into successful ones at no cost to the interface, eliminating a class of wasted rounds.

5

Conclusions, limitations, and future work

We investigate the effect of replacing, rather than adding, a bug-fixing agent’s shell-based debugging channel with a live debugger, while holding all else equal. Despite the initial appearance of interface complementarity under a single-seed, Debug2Fix-style pilot study (described in more detail in Appendix B), we find that allocating a rollout to each architecture performs worse than allocating two rollouts to the bash-agent when evaluated under a budget-matched passX @2. However, our study is limited to two open-weight reasoning models and a single debugger; different models and debugging interfaces may yield different performance curves. An important confounding factor is the unknown extent to which each underlying model has been trained on particular tools, and on the benchmark itself: the Nemotron-3-Ultra technical report lists SWE-bench Verified among the public seed sets for its agentic post-training data, without specifying the split [12], so some of its bash familiarity may be instance-specific rather than general. Additionally, SWE-bench Verified is a Python-only benchmark whose instances may under-represent the long-horizon, state-dependent failures that often necessitate execution-state inspection. SWE-bench Verified’s complexity strata are too sparse at the hard end (n=22 for 3+ files, n=8 for >100 lines) to draw robust conclusions about relative performance on this sub-class of problems. In future work, we plan to expand our evaluation to more diverse tasks featuring stateful runtime failures, and to explore how our results change after fine-tuning the models on general pdb tool use. 7

References [1] Anthropic. Claude code, 2024. URL https://www.anthropic.com/claude-code. Accessed: 2025-07-14. [2] Bjarni Haukur Bjarnason, André Silva, and Martin Monperrus. On randomness in agentic evals. ArXiv, abs/2602.07150, 2026. URL https://api.semanticscholar.org/ CorpusID:285453946. [3] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. URL https://arxiv.org/ abs/2107.03374. [4] Yongchao Chen, Jiefeng Chen, Rui Meng, Ji Yin, Na Li, Chuchu Fan, Chi Wang, Tomas Pfister, and Jinsung Yoon. Tumix: Multi-agent test-time scaling with tool-use mixture, 2025. URL https://arxiv.org/abs/2510.01279. [5] Spandan Garg and Yufan Huang. Debug2Fix: Can interactive debugging help coding agents fix more bugs?, 2026. URL https://arxiv.org/abs/2602.18571. [6] GitHub, Inc. GitHub Copilot: Your AI pair programmer, 2024. URL https://github. com/features/copilot. Accessed: 2025-07-14. [7] Yaoqi Guo, Yang Liu, Jie M. Zhang, Yun Ma, Yiling Lou, and Zhenpeng Chen. Swe-doctor: Guiding software engineering agents with runtime diagnosis from multi-faceted bug reproduction tests, 2026. URL https://arxiv.org/abs/2607.00990. [8] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=VTF8yNQM66. [9] Jiaxing Liu, Xing Hu, and Xin Xia. SWE-PDB: Teaching LLMs to leverage debugging tools via agentic training. Proceedings of the ACM on Software Engineering, 3(ISSTA), 2026. doi: 10.1145/3832117. [10] Zhengyao Liu, Yunlong Ma, Jing-Xuan Xu, Junchen Ai, Xiang Gao, Hai-Long Sun, and Abhik Roychoudhury. Agent that debugs: Dynamic state-guided vulnerability repair. ArXiv, abs/2504.07634, 2025. URL https://api.semanticscholar.org/CorpusID: 277667352. [11] Zairah Mustahsan, Abel Lim, Megna Anand, Saahil Jain, and Bryan McCann. Stochasticity in agentic evaluations: Quantifying inconsistency with intraclass correlation, 2025. URL https://arxiv.org/abs/2512.06710. [12] NVIDIA. Nemotron 3 Ultra: Open, efficient mixture-of-experts hybrid Mamba-transformer model for agentic reasoning. arXiv preprint arXiv:2606.15007, 2026. URL https://arxiv. org/abs/2606.15007. [13] OpenAI. Introducing SWE-bench Verified. OpenAI Research Blog, August 2024. URL https: //openai.com/index/introducing-swe-bench-verified/. Accessed: 202608-29. [14] OpenAI. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. URL https://arxiv.org/abs/2508.10925. [15] OpenAI. OpenAI Codex, 2025. URL https://openai.com/codex/. Accessed: 202507-14. [16] OpenAI. Why SWE-bench Verified no longer measures frontier coding capabilities. OpenAI Research Blog, February 2026. URL https://openai.com/index/ why-we-no-longer-evaluate-swe-bench-verified/. Accessed: 2026-08-28. 8

[17] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. In ICML, 2025. URL https://arxiv.org/abs/2412.21139. [18] Nicolas Schuler, Vincenzo Scotti, and Raffaela Mirandola. Tracegate: A framework for policycontrolled failure evidence in llm-assisted program repair. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering, ASE ’26, page 3369–3372, New York, NY, USA, 2026. Association for Computing Machinery. ISBN 9798400728822. doi: 10.1145/3832783.3834596. URL https://doi.org/10.1145/3832783.3834596. [19] Yun-Kun Wang, Yue Zhang, Guo-Chang Li, Chen Zhi, Bin-Hua Li, Fei Huang, Yong-Bin Li, and Shui-Guang Deng. Inspectcoder: Dynamic analysis-enabled self repair through interactive llm-debugger collaboration. ArXiv, abs/2510.18327, 2025. URL https://api. semanticscholar.org/CorpusID:282246183. [20] Jiahong Xiang, Xiaoyang Xu, Xiaopan Chu, Hongliang Tian, and Yuqun Zhang. Empowering autonomous debugging agents with efficient dynamic analysis. Proceedings of the ACM on Software Engineering, 3(FSE), 2026. doi: 10.1145/3797126. URL https://arxiv.org/ abs/2604.24212. [21] Xiangzhe Xu, Hamidreza Saghir, Qianhui Wu, Marc-Alexandre Côté, Tong Wang, Kiran Lakkaraju, Kexin Pei, and Xiangyu Zhang. The devil is in the interface: Evaluating how tool architecture shapes coding agent behavior, 2026. URL https://arxiv.org/abs/2608. 11386. [22] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering, 2024. URL https://arxiv.org/abs/2405.15793. NeurIPS 2024. [23] Bohan Yao and Vikas Yadav. Diverse multi-tool aggregation with large language models for enhanced math reasoning. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, pages 25264–25282, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.1377. URL https://aclanthology.org/2025.findings-emnlp.1377/. [24] Xingdi Yuan, Morgane Moss, Charbel El Feghali, Chinmay Singh, Darya Moldavskaya, Drew MacPhee, Lucas Caccia, Matheus Pereira, Minseon Kim, Alessandro Sordoni, and MarcAlexandre Côté. debug-gym: A text-based environment for interactive debugging, 2025. URL https://arxiv.org/abs/2503.21557. [25] Zhiyuan Zhai, Wenjing Yan, Xiaodan Shao, and Xin Wang. Does rl expand the capability boundary of llm agents? a pass@(k,t) analysis, 2026. URL https://arxiv.org/abs/ 2604.14877. [26] Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, Bo Pang, Yingbo Zhou, Shelby Heinecke, Silvio Savarese, Huan Wang, and Caiming Xiong. Diversity empowers intelligence: Integrating expertise of software engineering agents, 2024. URL https://arxiv.org/abs/2408.07060. [27] Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step-by-step, 2024. URL https://arxiv.org/ abs/2402.16906.

9

A

Complexity strata

Figure 5: Pass rate by gold-patch complexity (files touched). Both interfaces decline in parallel; the debugger’s deficit is roughly constant across buckets rather than shrinking on harder instances. The changed-lines stratification gives the same picture (Figure 6).

Figure 6: Pass rate by gold-patch complexity (changed lines). As with the files-touched stratification (Figure 5), both interfaces decline in parallel as patches grow. The pdb–bash deficit is roughly stable through 100 lines and vanishes in the >100-line bucket under both models, but that bucket is too small to draw general conclusions, n=8.

10

B

Preliminary single-seed campaign

Before the main study, we ran a single-seed campaign over four architectures: bash-agent, pdb-agent, and two Debug2Fix-style [5] compositions: an oracle agent that replaces direct debugger access with a dedicated debug subagent, and an orchestrator agent that additionally delegates file editing to an edit-specialized subagent. The results pointed in several directions at once (Table 2): the architecture ranking differed between models, with Nemotron’s pdb arm outscoring its bash arm and the reverse for GPT-OSS, and every architecture owned a small set of exclusive solves (Figure 7). Rather than ablate this design space until differences surfaced, we narrowed the study to the single controlled axis of the execution interface and measured it with seed-replicated runs. The seed-variance results of Section 4.2 explain much of what this campaign appeared to show. Table 2: Resolve rates from the preliminary single-seed campaign: one run per (model, architecture, instance), four architectures. Rates are the fraction of instances resolved; with n=1 seed no variance estimate is available, and single-seed gaps of this size can flip between draws (§4.2). Model

Arch

Solved

Rate (%)

Nemotron (n=500)

bash oracle orchestrator pdb

278/500 262/500 276/500 285/500

55.6 52.4 55.2 57.0

GPT-OSS-120B (n=500)

bash oracle orchestrator pdb

292/500 269/500 265/500 261/500

58.4 53.8 53.0 52.2

Figure 7: Which subset of the four architectures solved each instance in the preliminary single-seed campaign (Table 2). Both models are dominated by the two extremes—solved by all four arms (169 GPT-OSS, 204 Nemotron) or by none (145, 158)—while every exclusive pattern is small and no single architecture owns a large private set of solves. With one seed per arm, these small exclusive counts are consistent with the sampling variance quantified in Figure 4 rather than with per-interface specialization.

11

C

Evaluation infrastructure and provisioning

We execute sweeps using a queue-driven EC2 fleet. SWE-bench instances are enqueued in Amazon SQS and consumed by workers launched from a common AMI containing the evaluation harness, Podman environment, and benchmark datasets. The scheduling unit is a (SWE-bench instance, model endpoint) pair: after claiming a message, a worker evaluates both agent architectures and all three repetitions for that instance against that endpoint, writes trajectory-level artifacts to S3, and claims the next available message. Workers terminate once the queue is exhausted, while interrupted jobs are returned to the queue for re-execution. Model inference is served independently through vLLM endpoints, making serving throughput rather than compute the primary concurrency constraint. We therefore choose parallelism empirically by increasing concurrent trajectories until aggregate token throughput saturates the available served models. Model inference was served on 12 NVIDIA B300s and 8 RTX 6000s over roughly 3 days of continuous operation, approximately 1,700 GPU-hours in total; cumulative agent-trajectory wall-clock across all runs was approximately 21,900 hours (912 days). All in, we processed 54.76 billion tokens through the runs—54.38B prompt + 386.3M completion. Each instance is provisioned by cloning the repository, initializing at the base commit, applying the test_patch from the dataset, and scoping the new tests. Applying the test patch at provisioning means the agent can read and run the failing tests during its attempt; this is deliberate, and departs from leaderboard-style evaluation. The per-run workspace’s git history is sanitized after harness setup—the remote is removed, all refs except HEAD are deleted, and the reflog is expired—making the gold patch and all post-base-commit history unreachable from inside its run environment. A tool-level guard also exists triggering on any tool call containing git. Agents then operate on isolated per-run copies of the workspace in independent containers, run through Podman. Per-project test commands were incorporated into the run_tests tool to ensure agents could run tests through the same interface regardless of the framework.

D

Additive debugger arm: shell plus pdb

Debug2Fix [5] reports that exposing debugger commands directly to a shell-capable agent is ineffective: in its single-run Java ablation, solve rate was flat for two of three models and dropped for the third, and the agent rarely invoked the tools. To check whether our substitution result is consistent with that additive finding, we ran a third architecture, pdb_shell, that keeps the full run_bash tool of the bash-agent and adds the persistent pdb session of the pdb-agent. The edit gate is not enforced in this arm. All other settings (models, sampling configuration, prompt scaffold, 10 relays × 50 turns, sandbox, codebase map) are identical to the two main arms, and we again perform three independently seeded runs over all 500 instances. Figure 9 shows the result. Adding the debugger on top of the shell leaves pass@1 essentially unchanged: 58.1% for Nemotron-3-Ultra and 57.9% for GPT-OSS-120B against 56.7% for bashagent under both models, within about 1.5 points. This replicates the Debug2Fix observation that direct debugger access is roughly neutral, or destructive, for a shell-equipped agent, and it localizes the bash–pdb gap of Table 1 to the removal of the shell rather than to the debugger itself. Under the budget-matched pass@2, the picture also differs from the substitution setting: the cross-architecture pairing of one bash run with one pdb_shell run reaches 65.6% (Nemotron) and 66.3% (GPT-OSS120B), marginally above two pdb_shell runs (64.7% and 65.3%) and above two bash runs (62.7% and 64.7%). The margin over the stronger single arm is about one point under both models, and the mixed pair trails two bash runs at the smallest relay budgets (r ≤ 2), so we read this as two near-equivalent arms whose exclusive solves are symmetric (Figure 8) rather than as evidence of complementary abilities. When the debugger is additive, a diverse pair is at least as good an allocation as resampling either arm; when it replaces the shell, it is not.

12

Figure 8: Who solved it, bash versus pdb_shell, same construction as Figure 4. Top row: mean over the nine bash×pdb_shell seed pairings. Bottom row: union of all three runs per architecture. Unlike the substitution comparison, the exclusive-solve bars are symmetric between arms (union: 4.2% bash-only vs. 6.4% pdb_shell-only for Nemotron; 5.8% vs. 5.2% for GPT-OSS-120B) and both shrink between rows, consistent with two arms of equal strength sampled independently.

Figure 9: Cumulative solve rate as a function of relay budget r for the additive pdb_shell arm against bash-agent, same construction as Figure 3. Dotted curves are pass@1 with the 3-seed envelope; solid red and blue are intra-architecture pass@2 (Eq. (4)); green is the cross-architecture pass@2 of one bash run and one pdb_shell run (Eq. (5)). With the shell retained, pdb_shell pass@1 matches bash pass@1 within about 1.5 points, and the mixed pair edges out both intra-architecture pairs at the full budget by roughly one point while trailing two bash runs at r ≤ 2.

E

Unbiasedness of cross-pass-at-k

We establish unbiasedness of the cross-pass-at-k estimator in Equation 3 using the notation of Section 3.3. For a given task j and fixed portfolio of k agents, we assume that C1jm , . . . , Cnjm are independent and identically distributed (iid) for each agent m, and that the run batches are disjoint and mutually independent across agents. Ÿ Proposition. Under these assumptions, pass X @k in Equation 3 is an unbiased estimator of ÄSk ä Pr {C = 1} , the probability that at least one of k independently sampled runs, one ijm m=1 per agent, succeeds. 13

Proof.

By the definition of cjm and identical distribution of runs within each agent’s batch, E[n − cjm ] =

n X

Pr(Cijm = 0) = n Pr(C1jm = 0).

(8)

i=1

Using Equation 3, 1 Ÿ E[pass X @k] = 1 − k E n

" k Y

# (n − cjm )

m=1

k 1 Y =1− k E[n − cjm ] n m=1

=1−

k Y

(9)

Pr(Cijm = 0)

m=1

= Pr

k [

! {Cijm = 1} .

m=1

Thus, Ÿ Ÿ Bias(pass X @k) = E[passX @k] − Pr

k [

! {Cijm = 1}

= 0.

(10)

m=1

Averaging over a fixed benchmark preserves unbiasedness by linearity of expectation, and independence across tasks is not required. For any fixed relay threshold r, the same argument applies upon replacing Cijm by Sijm (r) and cjm by sjm (r) from Equations (6)–(7), provided the sampling assumptions hold for the complete run records. This yields pointwise unbiasedness for the probability that at least one run succeeds within the per-run relay threshold.

14

Related documents

• CVE-2026-58814#1538683
• SEDCoT: Enhancing LLM-Based COBOL Code Translation via Symbolic Execution and Delta Debugging#343564
• SEDCoT: Enhancing LLM-Based COBOL Code Translation via Symbolic Execution and Delta Debugging#821463
• SCRIBE: Practical Static Binary Patching via Binary-Aware Recompilation of Decompiled Code#155396
• Soliduzz: Finding Miscompilations in Smart-Contract Compilers with Assertion-Driven Equivalence Oracles#1343682