SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle
Hao Guan1∗,⋄ , Lingyue Fu1∗ , Shao Zhang1 , Yaoming Zhu2 , Kangning Zhang1 , Lin Qiu2 , Xunliang Cai2 , Xuezhi Cao2 , Weiwen Liu1 , Weinan Zhang1 , Yong Yu1 1 Shanghai Jiao Tong University, 2 Meituan,
arXiv:2605.13139v1 [cs.SE] 13 May 2026
∗
Equal contribution
⋄
Work done while interning at Meituan
Abstract As autonomous code agents move toward end-to-end software development, evaluating their practical autonomy becomes critical. Current benchmarks hide friction by testing agents in pre-configured environments, and their static evaluation pipelines frequently fail when parsing fully autonomous trajectories. We address these limitations with SWE-Cycle, a benchmark of 489 rigorously filtered instances. SWE-Cycle evaluates agents across three isolated tasks, including environment reconstruction, code implementation, and verification test generation, as well as an end-to-end FullCycle task that integrates all three. The FullCycle task requires agents to work autonomously in a bare repository without human scaffolding. To reliably assess these complex execution paths, we developed SWE-Judge. By combining static code review with dynamic testing, this execution-capable evaluation agent accurately verifies functional correctness and eliminates the systematic measurement errors of traditional static parsers. We evaluate code agents powered by six state-of-the-art LLMs across these four tasks. The results reveal a sharp drop in solve rates when transitioning from isolated tasks to FullCycle execution, exposing critical bottlenecks in handling cross-phase dependencies and maintaining code quality. Together, SWE-Cycle and SWE-Judge provide a comprehensive framework for accurately measuring the end-to-end capabilities of autonomous software agents.
1
Introduction
The continuous enhancement of large language models (LLMs) in tool invocation and complex reasoning [1, 11], alongside the rapid evolution of agentic frameworks [25, 2, 22], has driven significant progress in autonomous code agents. Modern code agents have transitioned from completing isolated algorithmic functions to maintaining entire engineering-level projects. By navigating large legacy codebases and synthesizing cross-file edits, they can now execute continuous development and integrate into real-world software pipelines [20]. As these code agents evolve into autonomous developers, their expanding capabilities require robust evaluation paradigms that accurately capture their practical autonomy. Evaluation frameworks quantify this progress across distinct paradigms. Some benchmarks [15, 5, 7, 35] focus on issue resolution within existing codebases, requiring code agents to generate verified source code patches across diverse programming languages and complexities. Beyond patching, another paradigm assesses the full software development lifecycle through greenfield project creation, where agents execute system design, module implementation, and testing [12, 36]. Additionally, a third category targets specific development stages, evaluating code agents on configuring execution environments [9] and generating automated test cases [28]. Across these categories, execution-based metrics verify the functional correctness and runtime stability of the generated outputs. While recent Preprint.
evaluations cover a broader range of software engineering tasks, current benchmarks still fail to capture autonomous execution across the complete issue resolution lifecycle and rely on brittle, static evaluation pipelines. Existing frameworks structurally fragment the development pipeline and bypass the complexity of legacy codebases [36, 12, 8], thereby failing to reflect an agent’s true autonomy in real-world maintenance. Successfully resolving a real-world issue requires a unified progression: reconstructing the execution environment, implementing the fix, and generating verification tests. When these interrelated phases are artificially isolated [9, 28], evaluations shield code agents from cascading errors and the friction of full-process integration, such as dependency resolution and version control navigation [4]. Consequently, current benchmarks project an illusion of capability: high performance on pre-configured tasks severely overestimates true autonomy, masking the reality that agents frequently fail completely when forced to onboard codebases, implement fixes, and verify their work without intervention. Moreover, static execution pipelines based on predefined unit tests and rigid parsers introduce systematic measurement errors and collapse in fully autonomous workflows. These deterministic systems are highly brittle. At the validation level, predefined static tests often suffer from flawed ground truths, such as misaligned checks or underspecified assertions [5, 40]. At the extraction level, strict parsers frequently misclassify functionally correct code due to trivial formatting deviations. Consequently, these rigid scripts frequently penalize valid alternative implementations and erroneously overlook deep logical flaws. More critically, this static paradigm is structurally incompatible with end-to-end scenarios. Because predefined scoring scripts cannot adapt to a code agent’s dynamic and autonomous behaviors, traditional execution evaluation is fundamentally inadequate for assessing the complete issue resolution cycle. To address these limitations, we present SWE-Cycle, a benchmark evaluating code agents across the complete issue resolution lifecycle. We construct this benchmark from SWE-bench Verified, Pro, and Multilingual, rigorously filtering them to retain 489 high-quality instances. Mapping directly to realworld software engineering, each instance integrates three essential tasks: environment reconstruction, code implementation, and verification test generation. SWE-Cycle supports two evaluation settings. In the Isolated Task setting, each phase is evaluated independently with the remaining stages fully provided, enabling controlled comparisons with existing benchmarks. In the FullCycle setting, the agent receives only a bare repository and an issue description, requiring it to autonomously complete all three stages without human scaffolding. To overcome the structural failures of rigid execution protocols, we further propose SWE-Judge, a hybrid evaluation agent integrating static code review with dynamic execution. Moving beyond the binary pass/fail metrics of predefined scripts, SWE-Judge employs task-specific validation protocols to accommodate diverse valid implementations, uncover structural defects, and capture fine-grained partial correctness. We systematically evaluate code agents powered by six state-of-the-art LLMs, reporting comprehensive results across both the isolated tasks and the FullCycle task. Our empirical analysis reveals that even within Isolated Tasks, traditional deterministic script evaluation produces severe misjudgments and false signals. In contrast, SWE-Judge provides a reliable assessment protocol across all four SWE-Cycle tasks and is strongly validated against human annotations. Ultimately, SWE-Cycle and SWE-Judge establish a unified benchmark and evaluation framework for assessing code agents across the complete issue resolution lifecycle. In summary, our key contributions are as follows: • We introduce SWE-Cycle, the first benchmark evaluating agents across the complete issue resolution lifecycle. Curated via a rigorous filtering pipeline to retain 489 high-quality instances, it supports both independent Isolated Task evaluation and a FullCycle setting that requires agents to operate autonomously from a bare repository. • We propose SWE-Judge, the only evaluation paradigm capable of assessing the complete issue resolution tasks. Integrating static code review with dynamic execution, SWE-Judge overcomes the limitations of rigid scripts to accommodate diverse valid implementations, uncover structural defects, and capture fine-grained partial correctness. • We systematically evaluate six LLMs across both the isolated tasks and the FullCycle task. This extensive evaluation establishes a comprehensive capability profile, explicitly measuring model 2
Table 1: Comparison between SWE-Cycle and current software engineering benchmarks. Benchmark
Task
Scenario
End-to-End
Env.
Impl.
TestGen.
Evaluation Protocol Judger
Partial Scoring
SWE-bench & Variants [15, 5, 7, 35] EnvBench [9] TestEval [28] DevBench [12] PRDBench [10] NL2Repo [8]
Issue Env Setup Existing Code Greenfield Greenfield Greenfield
✗ ✓ ✗ ✗ ✗ ✗
✓ ✗ ✗ ✓ ✓ ✗
✗ ✗ ✓ ✓ ✗ ✗
✗ ✗ ✗ ✗ ✓ ✓
UnitTest UnitTest UnitTest UnitTest+LLM Agent UnitTest
✗ ✗ ✗ ✗ ✓ ✗
SWE-Cycle (Ours)
Issue
✓
✓
✓
✓
Agent
✓
proficiency in environment reconstruction, code implementation, and verification test generation to provide a holistic view of their true autonomous potential.
2
Related Work
Code Agent Benchmarks. SWE-bench [15] has established the standard for evaluating code agents on real-world GitHub issues. This benchmark has spurred the rapid development of specialized agent architectures [31, 29]. Subsequent variants refine the evaluation along multiple dimensions: human-verified instance quality [5], longer-horizon tasks [7], multilingual coverage [35], continuous updates [13], heterogeneous comprehensive tasks [30], and large-scale training data synthesis [32]. Despite these diverse advances, all SWE-bench variants inherit the same structural limitation: they supply pre-built Docker environments and evaluate agents against fixed gold test suites. This design entirely excludes environment reconstruction and test generation from the evaluation scope. Alternative benchmarks address orthogonal aspects of software engineering, evaluating agents on greenfield project creation [36], feature development [17, 39], or automated code review [37]. However, greenfield development differs fundamentally from maintaining legacy systems; real-world engineering predominantly involves navigating complex existing codebases rather than starting from empty directories. Meanwhile, EnvBench [9] isolates environment reconstruction as a standalone capability, completely disconnecting it from downstream code implementation and verification test generation. Consequently, no existing benchmark captures the complete issue resolution lifecycle within a unified progression. Automated Evaluation Approaches. Existing code agent benchmarks predominantly rely on unit test execution, which suffers from well-documented limitations: flawed gold tests, binary pass/fail scoring that discards partial correctness, and inapplicability when agents must generate their own verification code [5]. LLM-as-a-judge offers an alternative approach that assesses output quality on continuous scales without requiring gold references [38, 14]. However, systematic studies reveal significant reliability concerns, including position bias that causes agreement fluctuations of up to 14% [27] and the fundamental limitations of execution-free judges when verifying runtime behavior [19]. To address these shortcomings, Agent-as-a-Judge represents an emerging paradigm that augments LLM judges with agentic capabilities, including planning, tool use, and iterative verification [33]. Recent works demonstrate that such agentic evaluators, especially when fine-tuned, substantially outperform pure LLM-as-a-judge methods in both human alignment and cost efficiency [41, 10]. SWE-Cycle adopts this paradigm through SWE-Judge, an execution-capable evaluation agent that combines static code review with dynamic test execution to overcome the limitations of execution-free assessment. Comparison with Existing Benchmarks. Table 1 summarizes the structural distinctions between current benchmarks. SWE-Cycle uniquely enforces the full issue resolution cycle while evaluating outputs through an execution-aware, fine-grained judge.
3
Methodology
To evaluate code agents across the software development lifecycle, we construct SWE-Cycle, a benchmark targeting the full issue resolution process. As detailed in Figure 1, the framework consists of three core components: dataset distillation to eliminate contamination, task formulation separating isolated capabilities from end-to-end execution, and hybrid evaluation via SWE-Judge. By combining 3
SWE-Bench Verified SWE-Bench Pro SWE-Bench Multilingual
Input Docker Image
Environment Reconstruction (Env)
Contamination Detection Lifecycle Complexity
Code Implementation (Impl)
Isolated Tasks
Test Reliability
SWE-Cycle
Verification Test Generation (TestGen)
FullCycle Task
SWE-Judge
Issue Description
Env
Impl
TestGen
Static Judge
Dynamic Judge
Artifacts Config
Step-wise Execution
Logic Requirements
Raw Log Analysis
Assertion & Coverage
Mutation Resilience
Intervention Mechanism Agent TestGen Quality？
✅ ❌
Eval with Agent Tests
⚠ Refine with Gold
🏆
0-2 Score
Figure 1: Overview of the SWE-Cycle Framework. Left: High-quality instances are curated through a rigorous filtering pipeline. Center: Agents execute environment reconstruction, code implementation, and test generation in either isolated tasks or the FullCycle task. Right: SWE-Judge evaluates outputs via hybrid static-dynamic analysis and a test intervention mechanism to yield a robust 0-2 score.
static analysis with dynamic execution, this architecture overcomes the brittleness of predefined scripts, ensuring robust evaluation of autonomous workflows.
3.1
Dataset Curation
To evaluate code agents on issue resolution tasks, we source 1,531 initial instances from three established datasets: SWE-bench Verified [5], SWE-bench Pro [7], and SWE-bench Multilingual [35]. However, the raw instances across these benchmarks suffer from data contamination, trivial task complexity, and invalid tests. To ensure the high quality required for assessing the complete issue resolution lifecycle, we design a three-stage filtering pipeline that resolves these flaws, distilling the initial pool into 489 rigorous instances. Contamination Detection. Recent evidence confirms severe memorization in existing benchmarks: models achieve 35% exact 5-gram match rates on reference patches and locate buggy files with 76% accuracy without accessing the repository structure [18]. To eliminate training data leakage, we introduce a zero-context probing mechanism. Specifically, we prompt a recent LLM to generate a patch without providing any issue description. If the model successfully generates the correct patch without context, we assume the instance was seen during training. This step removes 128 contaminated instances. Lifecycle Complexity Filtering. A benchmark targeting the complete development lifecycle requires tasks that reflect realistic engineering effort rather than instantaneous, isolated edits. To exclude trivial code tweaks, we filter instances using two criteria: (1) the pull request must contain at least one code review comment, and (2) the resolution cycle must span at least one day. These criteria remove same-day quick fixes and unreviewed submissions, ensuring the remaining instances represent complex maintenance tasks. After this step, the data pool is reduced to 523 instances. Test Reliability Filtering. Recent audits reveal severe defects in existing benchmark test suites: nearly 60% of SWE-bench Verified tests are flawed (often enforcing excessively narrow implementation details) [24], and insufficient test coverage causes up to 28.4% of incorrect patches to be falsely accepted [34]. To prevent these unreliable tests and environment dependency rot from compromising our evaluation, we execute the gold tests for each instance to verify strict state transitions: FAIL_TO_PASS tests must fail and PASS_TO_PASS tests must pass in the buggy state, and both must pass in the fixed state. This verification protocol removes 34 instances with corrupted test behavior. Ultimately, SWE-Cycle yields 489 instances: 225 from SWE-bench Verified, 203 from SWE-bench Pro, and 61 from SWE-bench Multilingual. The detailed results of each filtering stage are summarized in Appendix A. 4
3.2
Task Formulation
In SWE-Cycle, we decompose the issue resolution lifecycle into three isolated tasks: • Environment Reconstruction (Env) task evaluates the environment configuration capabilities of code agents. Given only the source code, code agents must independently build the execution environment and resolve all dependencies within a Docker container. • Code Implementation (Impl) task assesses issue-driven development capabilities. Given an issue description and a preconfigured codebase, agents must modify the repository to implement the requested changes. • Verification Test Generation (TestGen) task measures the capability to design effective unit tests. Given an issue description and a patched codebase, agents must generate tests targeting the specified issue. We also integrate these phases into an end-to-end task, FullCycle, to simulate a realistic developer workflow. Given an issue description and a raw codebase, the agent must handle environment setup, code implementation, and test generation within a single autonomous session. This dual design allows us to evaluate specific engineering skills in isolation, while still testing the model’s ability to manage the complete lifecycle without step-by-step guidance. 3.3
SWE-Judge
We introduce SWE-Judge to score the aforementioned tasks by combining static code review with dynamic execution. Unlike traditional script-driven methods, SWE-Judge applies task-specific criteria to yield independent static and dynamic scores (ranging from 0 to 2) for each task. The evaluation pipeline for each task is outlined below. Detailed scoring rubrics are available in Appendix B. Env Task Evaluation. SWE-Judge first statically examines configuration artifacts for missing dependencies and version conflicts. It then executes a three-stage dynamic validation: the activation stage verifies toolchain setup, the import stage checks core package compilation, and the collection stage ensures the test runner successfully gathers cases without dependency failures. By pairing static artifact inspection with this step-by-step execution, SWE-Judge accurately captures the configuration status and isolates the exact step where a setup failure occurs. Impl Task Evaluation. Traditional script evaluations for issue code implementation struggle to detect logical errors outside the predefined test coverage. SWE-Judge first extracts the core requirements, including the root cause, expected behavior, and critical edge cases, from the issue description and the official patch. It then statically reviews the submitted code against these requirements to verify its logical correctness. During dynamic execution, instead of relying solely on the rigid parsing of test outcomes, SWE-Judge analyzes the raw execution logs to accurately diagnose the actual runtime behavior. Consequently, SWE-Judge uncovers deep logical flaws that superficially pass rigid script evaluations, ultimately providing a more comprehensive and in-depth assessment. TestGen Task Evaluation. Conventional script-driven evaluations merely require tests to fail on the buggy repository and pass on the patched one. This condition is easily exploited and ignores test quality metrics like assertion accuracy and scenario coverage. To address this, SWE-Judge supplements dynamic execution with static analysis: it evaluates assertion quality against the official test suite and maps scenario coverage to verify critical execution paths. This dual verification ensures the generated tests genuinely isolate the target bug rather than exploit evaluation loopholes. FullCycle Task Evaluation. Traditional script evaluations rely entirely on predefined test suites, which is incompatible with the open-ended nature of the FullCycle task. As outlined in Algorithm 1, SWE-Judge addresses this by integrating the evaluation protocols from the isolated tasks into a sequential, fault-tolerant pipeline. SWE-Judge first conducts the Env evaluation, assigning zero to subsequent dynamic scores if the setup fails. Next, it evaluates the agent-generated unit tests. Since poor-quality generated tests cannot reliably verify the agent’s subsequent code implementation, SWE-Judge intervenes: if the generated tests fail the evaluation, it refines the agent-generated tests based on the official gold tests to ensure accurate verification. Finally, SWE-Judge evaluates the code implementation by statically reviewing the submission and dynamically executing it against the finalized unit tests (the originally submitted tests or their refined versions). Ultimately, this 5
Algorithm 1 SWE-Judge Evaluation Pipeline for the FullCycle Task Require: Agent submissions Aenv , Atest , Aimpl ; Official tests Tgold Ensure: Scores for each task (Senv , Stest , Simpl ), including dynamic and static scores stat stat stat 1: Senv , Stest , Simpl ← E VAL S TATIC(Aenv , Atest , Aimpl ) ▷ Static scores are always preserved dyn
2: Senv ← E VAL E NV DYNAMIC (Aenv ) dyn
stat 3: Senv ← (Senv , Senv )
dyn 4: if Senv = 0 then
▷ Halt: Upstream failure blocks execution
stat stat return Senv , (Stest , 0), (Simpl , 0) 6: end if dyn 7: Stest ← E VALT EST DYNAMIC (Atest ) dyn stat 8: Stest ← (Stest , Stest ) 9: if I S P OOR Q UALITY(Stest ) then 10: Texec ← R EFINE T ESTS(Atest , Tgold ) 11: else 12: Texec ← Atest 13: end if dyn 14: Simpl ← E VAL I MPL DYNAMIC (Aimpl , Texec ) 5:
▷ Intervention: Refine agent tests
dyn
stat 15: Simpl ← (Simpl , Simpl )
16: return (Senv , Stest , Simpl )
step-wise, fault-tolerant approach ensures that upstream failures do not silently invalidate downstream assessments, enabling a robust evaluation across the complete issue resolution cycle.
4
Experiments
In this section, we evaluate code agents powered by six state-of-the-art LLMs on SWE-Cycle to quantify their capabilities across the full issue-resolution lifecycle and validate the reliability of SWE-Judge. Specifically, our experiments, including an ablation study of SWE-Judge, are structured around four core research questions: • RQ1: How reliable is the evaluation produced by SWE-Judge? • RQ2: How do evaluated code agents perform across the four SWE-Cycle tasks? • RQ3: How does SWE-Judge overcome the limitations of traditional script-based evaluations? • RQ4: How do agent behaviors differ when resolving an issue end-to-end versus step-by-step? 4.1
Experimental Setup
We evaluate code agents powered by six state-of-the-art LLMs spanning both proprietary and openweight families: GPT-5.4 [23], Claude-Sonnet-4.6 [1], Qwen-3.5 [26], GLM-5.1 [11], Kimi-K2.5 [16], and MiniMax-M2.7 [21]. To guarantee reproducibility and support custom configurations, we adopt the open-source OpenCode1 framework across the evaluation pipeline. Each task executes within an isolated Docker container. We allocate 90 minutes per instance for the isolated tasks (Env, Impl, TestGen) and 3 hours for the FullCycle task; we evaluate all generated artifacts even if a timeout occurs. To balance assessment depth with computational cost, we use Claude-Opus-4.5 [3] as the SWE-Judge backbone (see Appendix C for robustness validation of backbone evaluation model). The evaluation agent runs directly in the same container to inherit the solving agent’s environment and artifacts. Metrics. We measure agent performance in SWE-Cycle using four metrics. Static assesses structural correctness or static analysis results without execution. Dynamic (Dyn.) measures functional correctness through actual execution. Both are initially scored as 0, 1, or 2, then normalized to a 0–1 scale. Score averages the Static and Dynamic results; for the FullCycle setting, this is macro-averaged 1 https://github.com/anomalyco/opencode/releases/tag/v1.4.6
6
across all three phases. Finally, Solve denotes the perfect resolution rate, representing the fraction of instances where Score equals 1. All metrics are reported as percentages. 4.2
Reliability of SWE-Judge (RQ1)
To validate the reliability of SWE-Judge, we manually annotate samples across all four tasks. For each task, we randomly select over 100 submissions generated by different agents across various issues. For the FullCycle task, we annotate one submission per issue. As shown in Table 2, SWE-Judge aligns with human judgment in over 95% of cases across all tasks. This confirms that SWE-Judge can reliably evaluate both isolated tasks and open-ended FullCycle resolutions without script guidance. Appendix D details the annotation protocol and reports per-task alignment rates. 4.3
Table 2: Alignment between human annotations and SWE-Judge across the four tasks. N denotes the number of sampled instances. Task
N
Alignment%
Env Impl TestGen FullCycle
143 113 201 489
99.3 95.6 99.5 96.9
Main Results (RQ2)
Isolated Tasks. Table 3 reports performance on the three isolated tasks, where gold inputs prevent cross-phase error propagation. Among these tasks, Env proves the most straightforward with solve rates reaching 78.1%, whereas Impl remains the primary bottleneck for all models. Additionally, the noticeable drop from average scores to strict solve rates indicates that static and dynamic evaluations capture distinct errors, exposing multiple categories of LLM failures. End-to-End Task. Table 4 presents results for the end-to-end FullCycle task, revealing three key findings. (1) Compared to the isolated tasks, phase-specific scores show noticeable improvement. This indicates that engaging in interconnected phases provides agents with valuable context. For instance, the process of writing test cases directly enhances performance on the core code implementation. (2) Static scores are consistently lower than dynamic scores. This discrepancy occurs because agents often generate submissions that hack the corresponding tests to pass runtime execution, but these flawed implementations are caught by static analysis. (3) No model achieves a strict overall solve rate above 14%, underscoring that completing the entire issue resolution lifecycle autonomously remains highly challenging for current code agents. Appendix E decomposes performance by dataset (Verified, Multi, Pro), and Appendix F reports efficiency metrics including median token consumption and execution time per model. To further analyze the performance of code agents 283 120 36 468 MiniMax-M2.7 in the FullCycle task, we categorize the failures of 263 112 25 24 37 477 Kimi-K2.5 unsuccessful instances in Figure 2. The distribu108 239 38 51 456 Qwen-3.5 tion reveals that compound errors dominate across 95 180 62 68 436 GPT-5.4 all evaluated models. Specifically, simultaneous 64 198 46 78 423 GLM-5.1 failures in both implementation and test generation 103 183 30 40 60 429 Claude-Sonnet-4.6 (Impl+TestGen), or across all three phases, consti0 100 200 300 400 500 tute the vast majority of unsuccessful instances. In Number of Instances contrast, isolated single-phase failures account for All Three Env+Impl Only Impl Impl+TestGen Only TestGen Only Env a significantly smaller fraction of the total. This pattern indicates a strong cascading effect within the issue resolution lifecycle, where a breakdown Figure 2: Distribution of failure categories across in one component is highly correlated with failures models in the FullCycle task. in the interconnected phases. 4.4
Effectiveness of SWE-Judge (RQ3)
SWE-Judge vs. Script Evaluation. To evaluate the effectiveness of SWE-Judge, we sampled 371 instances across all three phases where it diverged from traditional script metrics. As shown in Table 5, human adjudication confirms that SWE-Judge is correct in 98.6% of these disagreements, achieving 100% accuracy in the TestGen and Env phases, whereas scripts were correct in only 0.5% of cases. Analyzing these script failures reveals three primary structural flaws: excessive strictness 7
Table 3: Leaderboard on isolated tasks. Best results are in bold, and second best are underlined. Env
Model
Impl
TestGen
Static Dyn. Score Solve Static Dyn. Score Solve Static Dyn. Score Solve Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
90.59 87.73 87.83 81.09 56.13 82.62
84.97 76.48 74.44 68.80 60.94 75.05
87.78 82.11 81.13 74.95 58.54 78.83
78.12 73.01 71.78 61.34 45.40 66.46
68.00 65.95 69.12 64.38 52.86 56.24
57.16 53.68 51.84 51.06 43.46 43.35
62.58 59.82 60.48 57.72 48.16 49.80
40.08 37.83 39.67 34.46 30.88 28.83
91.00 85.89 74.44 80.79 52.66 72.90
85.79 82.82 76.38 82.44 55.11 76.89
88.39 84.36 75.41 81.61 53.89 74.90
67.28 60.53 42.13 52.07 33.33 46.01
Table 4: Leaderboard on FullCycle task. Best results are in bold, and second best are underlined. Env
Model Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
Impl
TestGen
Static
Dyn.
Static
Dyn.
Static
Dyn.
84.56 87.93 87.22 65.88 46.01 85.07
97.55 96.73 93.15 89.81 55.83 94.89
61.35 61.76 62.88 58.91 31.70 56.24
89.67 89.37 88.96 80.90 43.35 80.06
68.61 70.35 65.95 56.22 26.38 56.34
81.39 82.82 81.49 71.35 35.79 74.03
Score
Solve
80.52 81.49 79.94 70.51 39.84 74.44
12.27 13.50 10.84 2.15 4.29 6.75
(36.0%) that penalizes functionally equivalent alternatives, evaluation breakdown (32.8%) caused by brittle static pipelines, and excessive leniency (27.0%) that allows superficial or overfitted fixes to bypass basic execution checks. By semantically interpreting test outputs and dynamically adapting to the execution context, SWE-Judge resolves these systemic biases. Detailed category results and case studies are presented in Appendix G. Adaptive Evaluation Workflow. To understand Table 5: Human adjudication of 371 disagreehow SWE-Judge operates without gold tests, we ments between SWE-Judge and script-based metanalyzed its trajectories across all valid FullCycle rics. Percentages denote how often humans instances. Table 6 reports the average invocations agreed with each method. per trajectory (Mean) and the percentage of evaluations using each action (Coverage). Instead of Task N SWE-Judge Script Neither relying on static heuristics, SWE-Judge executes Impl 93 94.6% 2.2% 3.2% a systematic pipeline. It universally anchors on TestGen 173 100.0% 0.0% 0.0% code diffs and reference comparisons, then selecEnv 105 100.0% 0.0% 0.0% tively runs tests and build verifications to avoid Overall 371 98.6% 0.5% 0.8% false positives from broken environments. Crucially, it compensates for missing or flawed agent artifacts by actively writing custom evaluation scripts (34.6%) and using fault injection to verify test robustness (4.8%). These dynamic behaviors prove SWE-Judge’s capability for end-to-end evaluation. Appendix H presents three representative workflows. Appendix I validates our design choice to anchor on reference patches: removing them causes a severe 18.4 percentage point inflation in static scores, further highlighting the necessity of SWE-Judge’s comprehensive dynamic checks to ensure accurate evaluation. 4.5
End-to-End vs. Isolated Evaluation (RQ4)
To quantify how end-to-end integration reshapes model behavior, we evaluate 489 instances paired across FullCycle and isolated tasks. Figure 3a visualizes the performance shift by plotting the absolute difference (∆ = FullCycle − Isolated) in average score and overall solve rate across the three tasks. The results show a straightforward trend: integration improves upstream performance but degrades downstream metrics. Upstream Env benefits for most models because subsequent runtime execution exposes configuration defects that models then return to fix. Midstream Impl gains average dynamic functionality through this iterative write-run-fix loop, but the continuous patching compromises static structural quality, dropping the solve rate. Downstream TestGen degrades across all metrics. This occurs because models often hack the verification step by writing trivial tests that simply pass their 8
5
Score (%) Solve (%)
10
+7.4
0
0
5
-9.9
10 15
Env
Impl
5.0
20
Env
10
-5.8 -18.8
TestGen
-8.0%
20
+0.2%
2.5 0.0
Impl
-5.8%
+1.2%
TestGen
-7.0%
-11.0%
Env
Impl
2.5 5.0
Score (%)
Score (%)
10
+12.2
+10.8
Solve Phase (%) Removed
15
7.5 10.0
TestGen
(b)
(a)
Figure 3: End-to-end integration effects. (a) Per-dimension score and solve rate change (∆ = FullCycle − Isolated) across three tasks. (b) Score degradation of remaining phases when one phase is removed from the FullCycle task.
own implementations, aiming to terminate the task as quickly as possible rather than rigorously testing the code. To measure the inter-dependency of these tasks, Figure 3b maps a phase-ablation experiment. We Table 6: SWE-Judge action distribution across systematically remove a single phase from the FullCycle evaluations. FullCycle setup and record the resulting score Action # Avg Coverage degradations across the remaining active phases. Code Review (git diff) 9.7 100% These results confirm that the phases are tightly Reference Comparison (Read) 7.9 100% interlocked. Removing any single phase causes Env & File Inspection 16.5 99.9% scores in the others to drop. Notably, removing Test Execution 1.6 45.7% the downstream test phase severely penalizes upBuild verification 0.4 36.1% Adaptive eval scripting — 34.6% stream environment and code performance. This Fault injection — 4.8% highlights that a code agent’s verification capability is a critical driver of overall success. End-toTotal steps 39.5 — end evaluation measures an orchestration capability that isolated tasks structurally fail to capture (a detailed analysis is provided in Appendix J).
5
Conclusions & Limitations
In this paper, we propose SWE-Cycle, an innovative full-lifecycle issue resolution benchmark specifically designed to address critical challenges in evaluating the end-to-end autonomy of code agents. Alongside this benchmark, we introduce SWE-Judge, a robust agentic evaluator that uniquely integrates static and dynamic analysis, thereby providing comprehensive and reliable assessments. Our rigorous human validation and case studies demonstrate that SWE-Judge effectively overcomes the brittleness inherent in traditional evaluation scripts, enabling accurate and scalable evaluation across complex software engineering tasks. Additionally, leveraging this comprehensive framework, we uncover a critical blind spot in current LLM evaluation: optimizing localized accuracy in isolated tasks does not yield full-lifecycle autonomy. Results confirm that during end-to-end integration, cross-phase feedback paradoxically boosts dynamic correctness while accumulating structural debt, which ultimately degrades downstream test generation. A current limitation of SWE-Cycle lies in its evaluation mechanism. To ensure rigorous evaluation accuracy, our current evaluator relies on the Claude API, which inevitably makes the framework susceptible to external API fluctuations and version updates. To address this issue, future work will focus on enhancing open-weight models to serve as robust verifiers, thereby providing a stable, reliable, and independent evaluation backend. Overall, SWE-Cycle and SWE-Judge collectively represent a significant advancement in the evaluation of code agents. They necessitate a paradigm shift from isolated code generation to global planning and long-term maintainability, paving the way for the next evolution of autonomous software engineering by providing the rigorous standards required to guide future agent development. 9
References [1] Anthropic. Claude 4.6 sonnet system card. Technical report, Anthropic, 2025. URL https://assets.anthropic.com/m/785e231869ea8b3b/original/ Claude-4-6-Sonnet-System-Card.pdf. [2] Anthropic. Claude code, 2025. URL https://github.com/anthropics/claude-code. [3] Anthropic. Introducing claude opus 4.5. Anthropic Blog, 2025. URL https://www. anthropic.com/news/claude-opus-4-5. [4] Mert Cemri, Melissa Z. Pan, Shuyi Yang, et al. Why do multi-agent LLM systems fail?, 2025. URL https://arxiv.org/abs/2503.13657. [5] Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. Introducing SWE-bench verified. OpenAI Blog, 2024. URL https://openai.com/index/introducing-swe-bench-verified/. [6] DeepSeek-AI and Others. Deepseek-v3.2: Pushing the frontier of open large language models, 2025. URL https://arxiv.org/abs/2512.02556. [7] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. SWE-bench pro: Can AI agents solve long-horizon software engineering tasks?, 2025. URL https://arxiv.org/abs/2509.16941. [8] Jingzhe Ding and Others. Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents, 2026. URL https://arxiv.org/abs/2512.12730. [9] Aleksandra Eliseeva, Alexander Kovrigin, Ilia Kholkin, Egor Bogomolov, and Yaroslav Zharov. EnvBench: A benchmark for automated environment setup, 2025. URL https://arxiv.org/ abs/2503.14443. [10] Lingyue Fu, Bolun Zhang, Hao Guan, Yaoming Zhu, Lin Qiu, Weiwen Liu, Xuezhi Cao, Xunliang Cai, Weinan Zhang, and Yong Yu. Automatically benchmarking llm code agents through agent-driven annotation and evaluation, 2025. [11] GLM-5 Team. Glm-5: From vibe coding to agentic engineering. arXiv:2602.15763, 2026.
arXiv preprint
[12] Pareesa Ameneh Golnari, Adarsh Kumarappan, Wen Wen, Xiaoyu Liu, Gabriel Ryan, Yuting Sun, Shengyu Fu, and Elsie Nallipogu. Devbench: A realistic, developer-informed benchmark for code generation models, 2026. URL https://arxiv.org/abs/2601.11895. [13] Alex Gu, Naman Jain Liu, Nikhil Thakur, Wen-Ding Shi, Dídac Suris, Sanjay Jain, Naomi Saphra, Celine Lee Xia, Graham Neubig, and Aditi Raghunathan. SWE-bench goes live!, 2025. URL https://arxiv.org/abs/2505.23419. NeurIPS 2025 Datasets and Benchmarks Track. [14] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A survey on LLM-as-a-judge, 2024. URL https://arxiv.org/abs/2411.15594. [15] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2310.06770. [16] Kimi. Kimi k2.5: Scaling reinforcement learning with llms. Kimi Blog, 2025. URL https: //www.kimi.com/blog/kimi-k2-5. 10
[17] Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. Fea-bench: A benchmark for evaluating repository-level code generation for feature implementation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 17160–17176, 2025. [18] Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. The swe-bench illusion: When state-of-the-art llms remember instead of reason, 2025. URL https://arxiv.org/ abs/2506.12286. [19] Xichang Liu et al. CodeJudgeBench: Benchmarking LLM-as-a-judge for coding tasks, 2025. URL https://arxiv.org/abs/2507.10535. [20] Zhengyu Liu et al. AI-augmented CI/CD pipelines: From code commit to production with autonomous decisions, 2025. URL https://arxiv.org/abs/2508.11867. [21] MiniMax. Minimax 2.7. MiniMax Blog, 2026. URL https://www.minimaxi.com/models/ text/m27. [22] OpenAI. Introducing introducing-codex/.
codex,
2025.
URL
https://openai.com/index/
[23] OpenAI. Introducing gpt-5.4. OpenAI Blog, 2026. URL https://openai.com/index/ introducing-gpt-5-4/. [24] OpenAI. Why swe-bench verified no longer measures frontier coding capabilities. https:// openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/, February 2026. [25] OpenCode Contributors. Opencode, 2026. URL https://github.com/opencode-ai/ opencode. [26] Qwen. Qwen3.5: Towards native multimodal agents. Qwen Blog, 2026. URL https: //qwen.ai/blog?id=qwen3.5. [27] Vibhu Raina et al. Judging the judges: A systematic study of position bias in LLM-as-a-judge. In Proceedings of the International Joint Conference on Natural Language Processing (IJCNLP), 2025. URL https://arxiv.org/abs/2406.07791. [28] Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. Testeval: Benchmarking large language models for test case generation, 2025. URL https://arxiv.org/abs/2406.04531. [29] Xingyao Wang, Boxuan Chen, Parker Adler, Zijian Cheng, Kexun Hu, Jieyu Li, Yuqi Li, Ziniu Liu, Yufan Lu, Jiasheng Ning, et al. OpenHands: An open platform for AI software developers as generalist agents, 2024. URL https://arxiv.org/abs/2407.16741. [30] Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Huaixi Tang, Haoyang Huang, Zhiyi Lai, Zizheng Zhan, Yanan Wu, Chenchen Zhang, Kepeng Lei, Yifan Yao, Xinping Lei, Wenqiang Zhu, Zongxian Feng, Han Li, Junqi Xiong, Dailin Li, Zuchen Gao, Kun Wu, Wen Xiang, Ziqi Zhan, Yuanxing Zhang, Wuxuan Gong, Ziyuan Gao, Guanxiang Wang, Yirong Xue, Mengtong Li, Mengfei Xie, Xiaojiang Zhang, Jinghui Wang, Wenhao Zhuang, Zheng Lin, Huiming Wang, Zhaoxiang Zhang, Yuqun Zhang, Haotian Zhang, Bin Chen, and Jiaheng Liu. Swe-compass: Towards unified evaluation of agentic coding abilities for large language models, 2025. URL https://arxiv.org/abs/2511.05459. [31] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Liber, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-agent: Agent-computer interfaces enable automated software engineering, 2024. URL https://arxiv.org/abs/2405.15793. [32] John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling data for software engineering agents, 2025. URL https://arxiv.org/abs/2504.21798. NeurIPS 2025 Datasets and Benchmarks Track Spotlight. 11
[33] Runyang You, Hongru Cai, Caiqi Zhang, et al. A survey on agent-as-a-judge, 2026. URL https://arxiv.org/abs/2601.05111. [34] Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. Utboost: Rigorous evaluation of coding agents on swe-bench. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025. URL https://arxiv.org/abs/2506.09289. [35] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-SWE-bench: A multilingual benchmark for issue resolving, 2025. URL https://arxiv.org/abs/2504.02605. [36] Zhengran Zeng, Yixin Li, Rui Xie, Wei Ye, and Shikun Zhang. Benchmarking and studying the LLM-based agent system in end-to-end software development, 2025. URL https://arxiv. org/abs/2511.04064. [37] Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen, and Abhik Roychoudhury. Code review agent benchmark. arXiv preprint arXiv:2603.23448, 2026. [38] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena, 2024. URL https://arxiv.org/abs/2306.05685. [39] Q Zhou, J Zhang, H Wang, R Hao, J Wang, M Han, Y Yang, S Wu, F Pan, L Fan, D Tu, and Z Zhang. Featurebench: Benchmarking agentic coding for complex feature development. arXiv preprint arXiv:2602.10975, 2026. [40] Yuxuan Zhu, Tian Jin, Yada Pruksachatkun, Aston Zhang, Shayne Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Richard Weiss, et al. Establishing best practices for building rigorous agentic benchmarks. arXiv preprint arXiv:2507.02825, 2025. [41] Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. Agent-as-a-judge: Evaluate agents with agents. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. URL https://arxiv.org/abs/2410.10934.
12
A
Dataset Construction
A.1
Detailed Filtering Statistics
Table 7 reports the number of retained instances at each filtering stage. Below, we detail the concrete procedure for each stage. We run DeepSeek-V3.2 [6] for Contamination Detection. Table 7: The SWE-Cycle filtering pipeline and the number of retained instances at each step. Targeted Problem Verified Pro Multi Total Retained
A.2
Initial Source Pool
500
731
300
1,531
Contamination Detection
499
607
297
1,403
Lifecycle Complexity Filtering
239
209
75
523
Test Reliability Filtering
225
203
61
489
Language Distribution and Environment Setup
The final 489 instances span 9 programming languages: Python (68.9%), Go (18.0%), C (3.3%), Ruby (2.7%), JavaScript (2.5%), Rust (1.4%), TypeScript (1.2%), Java (1.2%), and PHP (0.8%). For Impl and TestGen tasks, each instance includes a pre-built Docker image with pinned runtimes and dependencies to ensure reproducible evaluation. In contrast, the Env task starts from a minimal base image containing only the OS and language runtime; the agent must independently resolve all project-level dependencies, build configurations, and toolchain setup.
B
Scoring Rubric
This appendix details the scoring criteria used by SWE-Judge for each task type in SWE-Cycle. All task types except FullCycle use a two-dimensional rubric (Static Analysis + Dynamic Execution), each scored 0–2, yielding a maximum of 4 points. FullCycle uses three dimensions (Environment, Code, Test), each with a static and dynamic sub-dimension (0–2 each), yielding a maximum of 12 points. The final metric is the score ratio = total score / maximum achievable score. Detailed eval prompts are in GitHub repo. B.1
Env Task
This task evaluates whether the agent correctly configured the required project dependencies and test runtime environment. Table 8: Scoring rubric for the Environment task (max 4 points). Dimension Static Analysis
Score 0 1 2
Dynamic Execution
0 1 2
Criteria No environment configured: core dependencies completely absent; setup.sh missing or fundamentally broken. Environment created, core dependencies basically installed, but obvious shortcomings: missing dependencies, version incompatibility risk, setup.sh not reproducible. Environment complete: all necessary dependencies correctly installed, setup.sh clear and reproducible, no compatibility issues. Cannot activate environment; or all tests fail after activation. Can activate, but tests partially fail (some FAIL_TO_PASS not passing or PASS_TO_PASS regressions). Can activate, all FAIL_TO_PASS pass, PASS_TO_PASS has no new failures.
13
B.2
Impl Task
This task evaluates whether the agent’s code patch correctly and completely resolves the reported bug. Table 9: Scoring rubric for the Development task (max 4 points). Dimension
Score 0
Static Code Analysis
1 2 0 1
Dynamic Execution
2
B.3
Criteria Patch is empty or unrelated to the problem; or contains a fundamental logic error. Fix direction is correct but has obvious shortcomings: incomplete logic, or change scope exceeds what is necessary. Fix logic is correct and complete, change scope is reasonable, targets the root cause with no significant side effects. Tests cannot run; or FAIL_TO_PASS tests have failures. All FAIL_TO_PASS tests pass, but PASS_TO_PASS has new failures (regression). All FAIL_TO_PASS tests pass, PASS_TO_PASS has no new failures.
TestGen Task
This task evaluates the quality, coverage, and effectiveness of the test cases written by the agent. The dynamic dimension uses a two-phase protocol: Phase 1 reverts the code fix (expect tests to fail); Phase 2 restores the fix (expect tests to pass). Table 10: Scoring rubric for the TestCase task (max 4 points). Dimension Static Analysis
Score 0 1 2
Two-Phase Dynamic
0 1 2
B.4
Criteria eval.sh missing; contains git operations; all meaningless assertions (assert True); content unrelated to the bug. Direction correct but quality insufficient: imprecise assertions, single scenario, low alignment with gold test scenarios; or tests merely mirror the code diff without additional insight. Good design: targets bug core behavior, meaningful assertions, covers main scenarios with good alignment to gold test coverage; demonstrates understanding beyond the diff (e.g., edge cases, boundary conditions). Phase 1 passes on buggy code (cannot detect bug); or eval.sh cannot execute. Phase 1 fails but with imprecise cause (import error / setup failure / permanently-false assertion); or Phase 2 still fails. Phase 1 fails precisely (points to the bug), Phase 2 passes.
FullCycle Task
This task evaluates the agent’s end-to-end performance. The scoring criteria aggregate the individual rubrics from the Environment, TestCase, and Development tasks, with execution dependencies following the pipeline pseudocode detailed in the main text. CODE_DYNAMIC uses the agent’s tests directly if I S P OOR Q UALITY(Stest ) returns false (Algorithm 1); otherwise the evaluator refines the agent-generated tests based on the official gold tests to ensure accurate verification. Detailed eval prompts are shown in GitHub repo.
C
Eval Model Robustness
We test whether SWE-Judge’s scores depend on the choice of judge model by having Claude-Opus4.5 and GPT-5.4 independently evaluate identical agent outputs. We randomly assign 414 instances across 6 coding models for all four task types. Every trial is scored by both eval models, using solve rate as the primary metric. We report solve rate as the primary metric. 14
Table 11: Scoring rubric for the FullCycle task (max 12 points). Dim.
Sub-dim.
Score
ENV (0–4)
0 Static 1 2
Dynamic
0 1
TEST (0–4)
2 0 Static 1 2 0 Dynamic 1
CODE (0–4)
2 0 Static 1 2
Dynamic
0 1 2
Criteria No environment created; core dependencies absent; setup.sh has fundamental errors. Environment created, basically installed, but obvious gaps (missing deps / version issues / no editable install). Complete and excellent: all deps correctly installed, setup.sh clear and reproducible. Layer 1 fails: cannot activate environment / toolchain missing. Layer 1 OK, but Layer 2 or Layer 3 fails (import/build/collection errors). Layers 1–3 all pass. No test code; only meaningless assertions; eval.sh contains git operations. Direction correct, quality insufficient: covers only one scenario, imprecise assertions, low alignment with gold scenarios. Good: targets bug core behavior, meaningful assertions, covers main scenarios, good alignment with gold test coverage. Phase 1 passes on buggy code (cannot detect bug); or no test code exists. Phase 1 fails but with imprecise cause (ImportError/setup failure); or Phase 2 still fails. Phase 1 fails precisely (points to bug), Phase 2 passes. No functional code changes; fundamental logic error; completely unrelated to the issue. Direction correct but incomplete: misses key edge cases, overly broad scope, or partial fix. Complete and precise: targets root cause, changes minimal, edge cases handled, highly aligned with gold direction. Environment failure prevents running; or all functional tests fail. Some functional tests pass (core logic OK, missing edge/secondary scenarios). All (or almost all) tests pass.
Table 12: Solve rate comparison between two eval models across task types. Diff = Claude-Opus-4.5 − GPT-5.4. Task Type Claude-Opus-4.5 GPT-5.4 Diff Impl TestGen Env FullCycle
0.4192 0.3985 0.6032 0.0630
0.4034 0.3718 0.5929 0.0419
+0.016 +0.027 +0.010 +0.021
The solve rate difference between Claude-Opus-4.5 and GPT-5.4 is within 3% across all four task types (largest: +2.7% on TestGen). The relative ranking of coding models is preserved under both eval models. Same-family bias check. Our default eval model (Claude-Opus-4.5) shares a model family with one coding model (Claude Sonnet 4.6). We compare the mean score boost from Claude-Opus-4.5 (per-trial Claude-Opus-4.5 − GPT-5.4) for Claude Sonnet against the average boost for the other five coding models. Across all four task types, Claude Sonnet’s boost is equal to or lower than the average of other models (Impl: −0.001 vs. +0.006; Env: +0.003 vs. +0.008; FullCycle: +0.143 vs. +0.194), except TestGen where the difference is negligible (+0.025 vs. +0.024). Claude-Opus-4.5 does not exhibit same-family scoring bias. 15
D
Human Annotation for SWE-Judge Reliability
We annotate 946 instances across all four tasks to validate SWE-Judge’s reliability. Scope. For isolated tasks (Impl, TestGen, Env), we annotate 457 cases. For FullCycle, we annotate all 489 instances (100% coverage). Annotation Criteria. Annotators adjudicate the factual correctness of SWE-Judge’s verdict by crossreferencing multiple evidence sources: the issue description, gold reference patch, agent’s submitted patch, and execution logs. For isolated tasks, annotators determine whether SWE-Judge or the script evaluator is correct. For FullCycle, annotators independently assign scores on 6 dimensions (each 0–2) using the same rubric as SWE-Judge and flag cases of overrating, underrating, or hallucination. Quality Control. A second researcher conducted a blind spot-check on 48 instances. The audit found 1 discrepancy (2.1%), which was corrected. D.1
Human Annotators
Our annotation team consisted of three full-time professionals, each holding at least a bachelor’s degree in computer-related disciplines (e.g., information security and software engineering) and possessing over two years of Python development experience. They maintained an average annotation rate of one task per hour, with daily working hours capped at eight. All annotators were compensated in strict compliance with local labor regulations. D.2
Isolated Tasks Detailed Results
Table 13 reports the per-task alignment between human annotations and SWE-Judge for isolated tasks. Table 13: Per-task alignment on isolated tasks. Alignment = annotator confirms SWE-Judge is correct. Task N Alignment% I MPL T EST G EN E NV
113 201 143
95.6% 99.5% 99.3%
Overall
457
98.5%
For isolated tasks, our annotation covers all cases where SWE-Judge and the script evaluator disagree (371 instances), plus a 10% random sample of agreement cases (86 instances). Among the agreement sample, 97.8% (85/86) were confirmed correct by human annotators, indicating that evaluator consensus reliably reflects true correctness with negligible risk of systematic co-failure. D.3
F ULL C YCLE Bias Analysis
Table 14 reports the direction of scoring disagreements: how often SWE-Judge scores higher vs. lower than human annotators. SWE-Judge slightly favors overrating (16 cases) over underrating (10 cases), but both are below 1%. TEST dynamic is the only dimension where underrating exceeds overrating (5 vs. 1). Note: per-dimension N varies slightly below 489 because a small number of evaluation runs failed to produce a parseable score for the corresponding dimension.
E
Per-Dataset Results
Tables 15–17 decompose the aggregate Isolated results (Table 3) by benchmark source. Static, Dynamic, and Score are percentages normalized to [0, 100]; Solve is the fraction of instances achieving a perfect score. 16
Table 14: Bias direction per dimension: frequency of SWE-Judge scoring higher (overrate) or lower (underrate) than human. Dimension N Judge > Human Exact Judge < Human ENV static ENV dynamic CODE static CODE dynamic TEST static TEST dynamic
486 483 485 482 485 482
2 (0.4%) 3 (0.6%) 6 (1.2%) 0 (0.0%) 4 (0.8%) 1 (0.2%)
484 (99.6%) 479 (99.2%) 478 (98.6%) 482 (100.0%) 478 (98.6%) 476 (98.8%)
0 (0.0%) 1 (0.2%) 1 (0.2%) 0 (0.0%) 3 (0.6%) 5 (1.0%)
Total
2,903
16 (0.6%)
2,877 (99.1%)
10 (0.3%)
Table 15: Per-dataset CodeImpl performance. Best results are in bold, and second best are underlined. Verified
Model
Multi.
Pro
Static Dyn. Score Solve Static Dyn. Score Solve Static Dyn. Score Solve Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
74.00 71.50 75.50 70.81 68.50 55.50
70.00 66.00 64.50 65.16 61.50 49.00
72.00 68.75 70.00 67.99 65.00 52.25
53.3 48.0 52.0 48.0 47.1 36.4
60.50 60.50 58.00 56.03 43.50 61.50
55.50 51.50 50.00 49.14 37.00 51.50
58.00 56.00 54.00 52.59 40.25 56.50
34.4 29.5 31.1 24.1 24.6 32.8
63.50 61.50 65.50 59.54 38.00 55.50
43.50 40.50 38.00 35.57 25.50 34.50
53.50 51.00 51.75 47.55 31.75 45.00
28.1 30.5 29.1 22.2 16.3 19.2
Table 16: Per-dataset TestGen performance. Best results are in bold, and second best are underlined. Verified
Model
Multi.
Pro
Static Dyn. Score Solve Static Dyn. Score Solve Static Dyn. Score Solve Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
92.50 88.00 69.00 82.00 71.00 73.50
96.50 93.50 78.50 92.89 79.50 86.50
94.50 90.75 73.75 87.44 75.25 80.00
85.8 77.3 40.9 63.6 51.6 56.9
84.50 83.50 73.00 74.56 31.00 69.00
86.00 90.00 81.00 81.58 32.00 81.00
85.25 86.75 77.00 78.07 31.50 75.00
70.5 68.9 50.8 54.4 21.3 42.6
91.00 84.50 81.00 81.19 38.50 73.50
73.50 68.50 72.50 71.04 35.00 65.00
82.25 76.50 76.75 76.11 36.75 69.25
45.8 39.4 40.9 38.6 16.7 35.0
Table 17: Per-dataset Env performance. Best results are in bold, and second best are underlined. Verified
Model
Multi.
Pro
Static Dyn. Score Solve Static Dyn. Score Solve Static Dyn. Score Solve Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
96.00 93.00 94.50 77.56 77.00 94.00
89.00 86.00 87.00 68.67 68.50 87.50
92.50 89.50 90.75 73.11 72.75 90.75
85.3 83.6 84.4 59.1 63.1 84.4
95.00 95.00 96.50 96.49 72.00 87.50
17
98.50 98.50 87.50 95.61 90.00 94.50
96.75 96.75 92.00 96.05 81.00 91.00
95.1 95.1 88.5 91.2 72.1 83.6
83.50 79.50 77.50 80.67 28.00 68.50
76.50 59.50 56.50 61.08 43.50 55.50
80.00 69.50 67.00 70.88 35.75 62.00
66.0 55.2 56.2 55.2 18.2 43.3
Performance degrades consistently from Verified to Pro across all models. The Multilingual subset has an uneven difficulty profile: Env scores are often higher than Verified (smaller repositories, simpler dependency chains), while CodeImpl scores drop sharply (unfamiliar language semantics and toolchains). TestGen on Multilingual tracks Verified for top-tier models but diverges for weaker ones, due to the difficulty of generating discriminative tests in non-Python ecosystems. Confidence Intervals. We report 95% confidence intervals for all core metrics. For binary metrics (Solve), we use Wilson score intervals. For continuous scores (Score), we use bootstrap percentile intervals with 10,000 resamples. Table 18: Per-dataset CodeImpl performance with 95% CI. Score uses bootstrap CI; Solve uses Wilson CI. Model Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
Verified
Multi.
Pro
Score
Solve
Score
Solve
Score
Solve
72.00±4.50 69.00±4.50 70.00±4.50 67.99±4.64 65.00±5.00 52.25±5.50
53.3±6.5 48.0±6.5 52.0±6.5 48.0±6.6 47.1±6.5 36.4±6.2
58.25±9.75 56.25±9.00 54.00±9.75 52.59±9.27 40.25±10.25 56.50±9.50
34.4±11.6 29.5±11.2 31.1±11.3 24.1±11.2 24.6±10.6 32.8±11.5
53.25±4.75 50.75±5.00 51.75±4.75 47.55±4.70 32.00±5.00 45.00±4.50
28.1±6.1 30.5±6.3 29.1±6.2 22.2±5.9 16.3±5.1 19.2±5.4
Table 19: Per-dataset TestGen performance with 95% CI. Score uses bootstrap CI; Solve uses Wilson CI. Model Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
Verified
Multi.
Pro
Score
Solve
Score
Solve
Score
Solve
94.75±2.00 90.75±2.75 73.75±3.75 87.44±2.61 75.25±4.25 80.00±4.00
85.8±4.6 77.3±5.4 40.9±6.4 63.6±6.2 51.6±6.5 56.9±6.4
85.25±7.00 87.00±6.25 77.00±7.50 78.07±7.68 31.50±10.75 75.00±7.75
70.5±11.2 68.9±11.3 50.8±12.2 54.4±12.3 21.3±10.1 42.6±12.0
82.50±2.75 76.50±3.50 76.75±3.25 76.11±3.22 36.75±5.50 69.25±4.25
45.8±6.8 39.4±6.7 40.9±6.7 38.6±6.7 16.7±5.1 35.0±6.5
Table 20: Per-dataset Env performance with 95% CI. Score uses bootstrap CI; Solve uses Wilson CI. Verified
Model Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
F
Multi.
Pro
Score
Solve
Score
Solve
Score
Solve
92.25±2.75 89.50±3.50 91.00±3.00 73.11±4.89 73.00±5.25 90.75±3.00
85.3±4.6 83.6±4.8 84.4±4.7 59.1±6.2 63.1±6.3 84.4±4.7
96.75±3.75 96.75±3.75 92.25±5.25 96.05±3.95 81.25±8.25 91.00±5.00
95.1±5.9 95.1±5.9 88.5±8.1 91.2±7.0 72.1±11.0 83.6±9.2
80.00±4.25 69.50±5.00 67.00±5.00 70.88±4.90 35.75±5.00 62.00±5.25
66.0±6.5 55.2±6.8 56.2±6.8 55.2±7.2 18.2±5.3 43.3±6.8
Efficiency Analysis
Table 22 reports median output tokens, agent solve time, and evaluation time across all task types. FullCycle requires substantially more tokens and time than isolated tasks. Claude-Sonnet-4.6 achieves the best performance (Table 3) with moderate token consumption and fast execution. Kimi-K2.5 shows moderate solve times (10–30 minutes) with higher token consumption than most models, reflecting a thorough exploration strategy. Evaluation time is stable across models (2–9 minutes for isolated tasks, 6–16 minutes for FullCycle). 18
Table 21: Per-dataset FullCycle performance with 95% CI. Score uses bootstrap CI; Solve uses Wilson CI. Verified
Model Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
Multi.
Pro
Score
Solve
Score
Solve
Score
Solve
83.50±2.00 85.50±1.67 83.92±1.67 68.42±2.47 49.00±5.17 77.83±2.58
20.4±5.2 22.2±5.4 15.1±4.7 0.4±0.7 8.4±3.7 12.4±4.3
82.50±3.17 82.75±3.42 79.00±4.75 76.13±5.30 43.00±8.75 74.42±5.50
8.2±7.1 8.2±7.1 9.8±7.6 10.2±7.6 1.6±4.2 3.3±5.1
76.58±1.92 76.67±3.08 75.83±2.42 71.24±2.40 28.58±4.83 70.67±2.42
4.4±2.9 5.4±3.2 6.4±3.4 1.6±1.9 0.5±1.3 1.5±1.9
Table 22: Efficiency metrics across task types. OutTok = median output tokens (K); Solve = median agent execution time (min); Eval = median verifier execution time (min). All values are medians computed over all instances across three datasets. Impl
Model
TestGen
Env
FullCycle
OutTok Solve Eval OutTok Solve Eval OutTok Solve Eval OutTok Solve Eval Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Qwen-3.5 Kimi-K2.5 MiniMax-M2.7
5.4K 3.0K 2.8K 8.9K 8.5K 4.2K
5.7 11.6 4.2 23.5 10.8 12.3
3.6 3.1 3.2 3.1 4.0 3.2
7.3K 3.0K 2.6K 6.7K 7.5K 5.9K
8.4 12.1 4.9 18.8 10.4 16.1
G
Script Evaluation Failure Analysis
G.1
Disagreement Annotation Protocol
3.9 4.0 4.0 3.9 5.0 3.8
3.1K 1.5K 4.4K 2.6K 4.0K 2.0K
8.2 12.7 20.1 21.9 29.5 14.1
2.9 2.6 2.6 2.7 3.5 2.8
11.0K 5.9K 6.0K 12.4K 10.6K 1.9K
11.7 8.0 13.8 7.7 10.9 8.6 29.5 7.6 27.8 16.0 5.7 6.0
To categorize Script–SWE-Judge disagreements, two graduate researchers with software engineering experience independently label each case. For each disagreement instance, annotators receive a review package containing: the issue description, the agent’s submitted patch, the gold reference patch, SWE-Judge’s scoring with reasoning, the script evaluator’s binary verdict with execution logs, and LLM-generated auxiliary analysis highlighting potential discrepancies. Each annotation follows a structured protocol: 1. Read the issue description to understand the problem context. 2. Examine the gold reference patch to establish the correct solution approach. 3. Review the agent’s submission to understand what the agent implemented. 4. Read SWE-Judge’s scoring and reasoning. 5. Cross-reference with execution logs and LLM auxiliary analysis when static review is insufficient. 6. Assign a failure category from the predefined taxonomy and record which evaluator is correct. Human Verification Results. To validate the LLM-assisted categorization and rule out selection bias, we conduct human deep annotation on all 371 disagreement instances plus 86 agreement instances (a 10% random sample of cases where SWE-Judge and the script concur). For disagreement cases, human annotators confirm SWE-Judge as correct in 98.6% (366/371), the script as correct in 0.5% (2/371), and neither in 0.8% (3/371). For agreement cases, 97.8% (85/86) are confirmed correct by human review, indicating that evaluator consensus reliably reflects ground truth with negligible risk of systematic co-failure. G.2
Disagreement Categories
Table 23 summarizes the categorization results. 19
Table 23: Script Evaluation Errors from 3,267 Script–SWE-Judge Disagreements. % denotes the proportion of each category. Category N % Excessive strictness Evaluation breakdown Excessive leniency Others
G.3
1176 1072 882 137
36.0 32.8 27.0 4.2
Representative Case Studies
We present representative cases organized by the three failure categories. G.4
Excessive Strictness
Scripts demand exact alignment with the golden patch and reject functionally equivalent alternatives. Two manifestations appear: alternative implementations receiving zero credit, and partial fixes losing all information through binary scoring. Case 1: Alternative Implementation in Valkey. In valkey-io/valkey#1499, the golden patch modifies the command table to fix a permission checking issue. The agent instead uses executing_client->cmd to check the actual command being executed, with null-safety handling. All FAIL _ TO _ PASS tests pass and no PASS _ TO _ PASS regressions occur. The script assigns 0 because its tests are coupled to the specific implementation path of the golden patch. SWE-Judge performs static analysis, confirms the semantic equivalence of both approaches, and assigns 1.0. Case 2: Constant Exporting in Teleport. In a Teleport issue requiring namespace configuration constants, the golden patch inlines string literals across multiple files. The agent exports NamespaceEnv and ReleaseNameEnv as package-level constants and updates all references. This is a cleaner refactoring that produces identical behavior. All 13 FAIL _ TO _ PASS tests pass, but the script assigns 0 because the modified file set differs from the expected set. SWE-Judge recognizes the functional equivalence and awards full marks. Case 3: Partial Fix in PHPSpreadsheet. In a PHPSpreadsheet task, the agent correctly adds a __toString() method to the StructuredReference class, fixing the immediate string conversion error. However, it misses additional changes for cross-worksheet table and structured reference handling that the golden patch includes. The script assigns 0, indistinguishable from a completely wrong submission. SWE-Judge assigns 0.25, recognizing that the core direction is correct but coverage is incomplete. This proportional credit separates near-miss attempts from zero-effort submissions. Case 4: Qutebrowser Path Resolution. The agent creates FilePathCategory with proper path resolution for file://, tilde, and absolute paths, integrates it into the URL model, and updates documentation. The implementation handles all major scenarios but misses minor edge cases in the golden patch. SWE-Judge assigns 0.75, reflecting a nearly complete solution. The script’s binary 0 fails to capture this meaningful progress. G.5
Evaluation Breakdown
Nearly a third of disagreements occur because the evaluation pipeline itself fails, independent of solution quality. Parser incompatibilities and infrastructure rot are the primary causes. Case 5: Gradle Output Parsing in Apache Lucene. Multiple Apache Lucene environment tasks exhibit this pattern. The agent correctly configures JDK 21, the Gradle wrapper, and all build dependencies. SWE-Judge independently confirms via JUnit XML that all 108 tests pass with 0 failures. However, the SWE-bench evaluation parser expects Maven-style output format and cannot 20
parse Gradle’s BUILD SUCCESSFUL format, reporting a zero score. This is a failure of the evaluation tool, not the agent. Case 6: Maven Daemon Timeout in Google Gson. In a Google Gson environment task, the agent’s setup correctly installs Java and Maven. All 10 tests pass when executed with standard Maven. However, the evaluation script uses mvnd (Maven Daemon), which times out during cold start. SWE-Judge identifies this as an infrastructure artifact: the original script evaluation failures stem from mvnd daemon issues (timeout/crashes), not actual test failures. The agent receives full marks from SWE-Judge. Case 7: Node.js Workspace Corruption. After yarn install, the node_modules state file is missing or corrupted in approximately 178 cases, causing all subsequent commands to fail with “Couldn’t find the node_modules state file.” The agent’s code is never evaluated because the test framework collapses before reaching any relevant assertion. SWE-Judge evaluates the agent’s configuration through static analysis and awards credit based on the quality of the submitted patch, independent of whether the test infrastructure executed successfully. These cases illustrate a structural limitation: script evaluation conflates “the framework crashed” with “the solution is wrong.” As dependencies deprecate and runtime versions drift, this conflation worsens over time. G.6
Excessive Leniency
Script evaluation can conflate superficial execution success with semantic correctness. Case 8: Trivial State Transition in TestGen. In an Ansible test task, the agent writes tests that import set_multipart_encoding at module level. This function exists only after the fix. On buggy code, the test fails with ImportError before any test logic executes. The script’s dynamic state transition protocol checks only whether Phase 1 (buggy code) produces a non-zero exit code and Phase 2 (fixed code) passes. Both conditions are met, so the script awards full marks. SWEJudge recognizes the failure mechanism: “When code_patch is reverted, module import fails with AttributeError before any tests can run to detect actual bug behavior.” The test provides zero discriminative power because any pre-fix version would fail regardless of the specific bug. Case 9: Regression Escape in Django. In django/django#13590, the agent correctly fixes namedtuple support in Values() by unpacking values. However, the patch unconditionally unpacks all types, breaking regular list/tuple construction. The FAIL _ TO _ PASS test passes, but 5 PASS _ TO _ PASS tests fail with TypeError. The script monitors only the target test scope and awards full marks. SWE-Judge runs the complete test suite and identifies the regression: the golden patch uses hasattr(type_, ’_make’) to detect namedtuples and only unpacks for those types. SWE-Judge assigns 0.25. Case 10: Incomplete Test Coverage in coreutils. In uutils/coreutils#6575 (TestGen), the agent tests non-UTF-8 filename handling but covers only CRC mode, missing the SHA256 mode test present in the golden patch. The script’s pass/fail check accepts this single-scenario test as fully correct. SWE-Judge evaluates coverage depth against the golden patch and assigns 0.25, recognizing that the test provides insufficient coverage to serve as a reliable regression test. Case 11: Testing Unchanged Code in Vuls. In a Vuls test task, the agent modifies an existing test to call convertToModel() (which is unchanged between buggy and fixed states) instead of testing the actual bug in config/os.go. The test passes in both states, providing zero discriminative power. The script awards full marks based on the exit code. SWE-Judge compares the test logic against the issue description and golden patch, identifying that the tested function is irrelevant to the reported bug. Score: 0.25.
H
SWE-Judge Workflow Case Studies
We select three FullCycle evaluation workflows from Table 6 that each illustrate a distinct capability: adaptive eval scripting, fault injection, and build verification with multi-dimensional scoring. 21
H.1
Case 1: Adaptive Eval Scripting
Instance. NodeBB/NodeBB#8168c6c4 (FullCycle, Claude-Sonnet-4.6). The issue requires implementing profile image cleanup: when users remove cover photos or avatars, the corresponding files on disk must be deleted. Workflow Summary.
SWE-Judge executes 6 steps across 47 tool calls:
1. Instruction and gold patch review. SWE-Judge reads the issue description and golden patch to establish the expected behavior: file deletion via rimraf with glob patterns for accumulated profile images. 2. Agent patch macro-review. SWE-Judge reviews the agent’s diff across 5 modified files (src/groups/cover.js, src/socket.io/user/picture.js, src/user/delete.js, src/user/picture.js). It identifies a critical divergence: the agent uses getLocalCoverPath/getLocalAvatarPath to delete only the current file, while the golden patch uses glob patterns to delete all accumulated files. 3. ENV evaluation. Static: setup.sh runs npm install correctly. Dynamic: Node.js v18.20.8 available, packages import successfully, 359 tests collected. Score: 4/4. 4. TEST evaluation. Static: Agent covers 3 of 4 key scenarios (missing account deletion cleanup test). Dynamic: Phase 1 fails with TypeError: User.getLocalCoverPath is not a function (imprecise failure). Score: 2/4. 5. Adaptive eval scripting (triggered by TEST_STATIC < 2). The agent’s test suite lacks coverage for account deletion cleanup. SWE-Judge writes eval_improved.sh and test/improved-image-cleanup.js, a custom test that creates dummy profile images, calls the account deletion function, and verifies that 0 files remain afterward. The first execution discovers 3 orphaned files. After debugging a path configuration issue and re-executing, the test confirms the agent’s implementation leaves orphaned files during account deletion. 6. CODE evaluation using custom test results. The custom test output directly informs CODE_DYNAMIC: 3/4 tests pass (group cover, user cover, user avatar succeed; account deletion cleanup fails). Score: 2/4. Final Scores.
ENV: 4, TEST: 2, CODE: 2. Total: 8/12 (0.667).
This case shows that SWE-Judge writes its own verification scripts when existing coverage is insufficient (34.6% of FullCycle evaluations), exposing gaps that the agent’s own tests miss. H.2
Case 2: Fault Injection
Instance. internetarchive/openlibrary (FullCycle, Qwen-3.5). The issue requires adding a Solr boolean clause limit configuration (-Dsolr.max.booleanClauses=30000) to docker-compose.yml and a corresponding FILTER_BOOK_LIMIT constant in bookshelves.py. Workflow Summary. 1. Initial review. SWE-Judge reads the instruction, golden patch, and agent patch. The agent correctly implements both required changes: adding -Dsolr.max.booleanClauses=30000 to SOLR_OPTS and defining FILTER_BOOK_LIMIT = 30_000. 2. Agent test execution (Phase 2). SWE-Judge runs the agent’s test suite (eval.sh) on the fixed code. Both tests pass: test_filter_book_limit_constant_exists and test_solr_opts_has_boolean_clauses_limit. 3. Fault injection (Phase 1). SWE-Judge reverts the agent’s changes to simulate the buggy state: git show base_commit:docker-compose.yml > /tmp/docker-compose-buggy.yml cp /tmp/docker-compose-buggy.yml docker-compose.yml git show base_commit:openlibrary/core/bookshelves.py > \ /tmp/bookshelves-buggy.py cp /tmp/bookshelves-buggy.py openlibrary/core/bookshelves.py 22
SWE-Judge then re-runs the agent’s tests against this reverted codebase. Both tests now fail: FILTER_BOOK_LIMIT is not found in bookshelves.py, and -Dsolr.max.booleanClauses is absent from SOLR_OPTS. 4. Verdict. The tests correctly discriminate between buggy and fixed states. SWE-Judge confirms the agent’s tests are not trivial or overfitted: they verify specific code content rather than relying on indirect signals. TEST_DYNAMIC: 2/2. 5. ENV evaluation. Static: Agent uses venv instead of the requested conda environment, deviating from the instruction. Dynamic: Python 3.11.1 available, but core package import fails (ModuleNotFoundError: web). Score: 2/4. Final Scores.
ENV: 2, TEST: 3, CODE: 4. Total: 9/12 (0.75).
Fault injection verifies that the agent’s tests genuinely detect the bug rather than passing for spurious reasons. SWE-Judge uses this technique in 4.8% of evaluations, typically when the tests appear suspiciously simple or when configuration changes could easily produce false positives. H.3
Case 3: Build Verification and Multi-Dimensional Scoring
Instance. flipt-io/flipt#292fdac (FullCycle, Claude-Sonnet-4.6). The issue requires implementing an optional configuration versioning feature for the Flipt feature flag server (Go). Workflow Summary. 1. Code review via git diff. SWE-Judge examines the agent’s changes: adding a Version field to the configuration struct, implementing validation logic, updating the schema, and creating test data files. 2. Reference comparison. SWE-Judge reads the golden patch and performs a structural comparison. The agent’s implementation aligns closely with the golden patch, using cleaner error handling patterns in some cases. 3. Build verification. SWE-Judge runs go build ./... to confirm compilation succeeds, then uses a non-matching test pattern to verify test collection without execution. 4. Test execution with fault injection. SWE-Judge reverts the code to the buggy state and runs the agent’s tests. Tests fail with cfg.Version undefined (compilation error). SWE-Judge notes this is a weaker detection mechanism (compile-time rather than assertion-based) but still validates that the tests cannot pass without the fix. 5. Multi-dimensional scoring. • ENV: Static 2/2 (complete setup), Dynamic 2/2 (Go toolchain available, packages import, tests collect). Score: 4/4. • TEST: Static 2/2 (comprehensive test coverage aligned with golden patch), Dynamic 1/2 (Phase 1 failure is imprecise: compilation error rather than assertion failure). Score: 3/4. • CODE: Static 2/2 (correct implementation matching golden patch), Dynamic 2/2 (all target tests pass on fixed code). Score: 4/4. Final Scores.
ENV: 4, TEST: 3, CODE: 4. Total: 11/12 (0.917).
Build verification (used in 36.1% of FullCycle evaluations, mostly compiled languages) serves as a gate: a failed build immediately invalidates dynamic scores. The multi-dimensional scoring here separates a correct implementation (CODE: 4/4) from an imprecise test design (TEST: 3/4), a distinction that binary pass/fail cannot express.
I
Ablation: Reference-Guided vs. Blind Evaluation
Does access to the official patch cause the evaluator to penalize valid alternative implementations? We compare Gold eval (evaluator receives the reference solution) against Blind eval (evaluator judges solely from the problem description, repository state, and submitted patch). 23
I.1
Score Comparison
Table 24 reports the dimension-level comparison across 8,678 paired trials. Table 24: Gold vs. Blind evaluation comparison across all task categories. Isolated: n=5,771; FullCycle: n=2,907. All scores normalized to percentages. Diff = Blind − Gold (percentage points). Setting Category Gold (%) Blind (%) Diff (pp) Isolated
Impl TestGen
64.0 64.1
68.4 66.4
+4.4 +2.3
FullCycle
ENV CODE TEST
83.2 64.3 62.9
82.5 82.7 68.1
−0.7 +18.4 +5.3
In Isolated evaluation, Blind eval inflates Impl scores by +4.4 pp and TestGen by +2.3 pp. In FullCycle, the inflation concentrates in CODE (+18.4 pp), where assessing correctness without a reference is hardest. ENV scores remain stable (−0.7 pp) because environment correctness is largely verifiable through execution. The inflation is driven almost entirely by static sub-scores (∆S = + 0.17) while dynamic sub-scores remain unchanged (∆D = + 0.01): execution-based verification is objective regardless of reference availability. I.2
False Negative Analysis
If reference access biased against correct alternatives, Gold eval would show an elevated false negative (FN) rate. Table 25 measures this using the script evaluator as ground truth (threshold 0.5). Table 25: Error rates by category (binary threshold = 0.5). FP = false positive (incorrect submission scored as pass). FN = false negative (correct submission scored as fail). Category Gold FP Blind FP Gold FN Blind FN Impl TestGen
8.7% 35.2%
14.3% 34.9%
0.3% 0.2%
0.2% 0.2%
Gold eval’s FN rate is ≤0.3% across both categories, virtually identical to Blind eval. Providing the reference does not cause the evaluator to reject valid submissions. The measurable difference is in false positives: Blind eval’s FP rate on Impl is 1.6× that of Gold (14.3% vs. 8.7%). The reference improves precision without increasing rigidity. I.3
Illustrative Case: Gold Eval Favors a Correct Alternative
We present a case where Gold eval is more lenient than Blind eval toward an alternative implementation. django__django-16877 (Impl, Claude 4.6) — Script=0, Gold=4/4, Blind=1/4 Gold (static=2): “Implementation is functionally identical to gold.patch—correctly implements escapeseq filter with equivalent logic.” Blind (static=1): “Fix direction correct, but agent left unresolved merge conflicts in test file.” Gold eval confirms semantic equivalence with the reference and correctly identifies the merge conflict markers as irrelevant to functional correctness. Blind eval, lacking this anchor, is misled by the cosmetic issue and penalizes a correct submission. Reference access here protects the alternative implementation by providing a semantic equivalence check. I.4
Difficulty Stratification
Table 26 stratifies trials by Gold score to examine where the Gold–Blind gap concentrates. 24
Table 26: Difficulty stratification: Blind–Gold score difference by Gold score bin (n=5,771 Isolated trials). Gold Score Bin n Blind Mean Diff = 0 (clearly wrong) (0, 0.25] (0.25, 0.5] (0.5, 0.75] (partial) (0.75, 1.0] (correct)
1,017 1,033 388 1,033 2,300
0.024 0.289 0.599 0.886 0.978
+0.024 +0.039 +0.099 +0.136 −0.022
The pattern forms an inverted-U: inflation peaks at the (0.5, 0.75] bin (+0.136) and reverses for near-perfect submissions (−0.022). If Gold eval penalized correct alternatives, the highest-scoring bin would show Gold > Blind. Instead, the slight negative difference confirms Gold eval does not under-score correct submissions. The only divergence direction is Blind eval over-scoring partial fixes in the ambiguous middle range.
J
End-to-End vs. Isolated Details
Per-model breakdowns supporting Section 4.5. All bonus counts exclude timeout-driven flips. J.1
Instance-Level Flip Counts
Table 27: Per-model instance-level flips between FullCycle and Isolated (N =489 instances per model). Degrad. = instances solved in Isolated but imperfect in FullCycle. Bonus = instances unsolved in Isolated but perfect in FullCycle (timeout-driven cases excluded).
Env Model
CodeImpl
TestGen
Degrad.
Bonus
Degrad.
Bonus
Degrad.
Bonus
Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
73 51 55 169 112 42
31 75 59 31 67 64
92 83 101 78 95 81
23 33 46 26 9 28
193 158 111 177 134 149
27 40 56 26 17 26
Total
356
301
519
152
815
177
The pipeline gradient holds across all models: degradation counts increase monotonically from Env to TestGen, confirming downstream dimensions bear a heavier integration tax. MiniMax-M2.7 shows the most severe net Env degradation. GLM-5.1 achieves a net Env bonus (75 vs. 51), indicating effective use of downstream signals to repair environment defects. For CodeImpl, GPT-5.4 has the highest bonus count (46) but also the highest degradation (101). Even the best models (Claude-Sonnet-4.6: 193 degradation vs. 27 bonus) suffer a roughly 7:1 degradation-to-bonus ratio in TestGen. J.2
Degradation Root Cause Analysis
Table 28 shows a capability divide. For CodeImpl, SOTA models (Claude-Sonnet-4.6, GLM-5.1, GPT-5.4) degrade almost exclusively through static-only loss (85–98%): their code runs correctly but sacrifices structural quality during iterative patching. MiniMax-M2.7 shows the opposite, with 64% joint collapse in CodeImpl and 77% in Env, meaning weaker models cannot maintain functional correctness under integration pressure. The same pattern holds for TestGen: SOTA models show 59–74% static-only loss (tests execute but coverage drops), while MiniMax-M2.7 suffers 67% joint failure. Kimi-K2.5 follows the SOTA pattern in Env (76% static-only) and CodeImpl (72% static25
Table 28: Degradation root cause distribution per model and dimension. S-only = static score drops while dynamic remains perfect. Both = both static and dynamic degrade. Percentages are computed over score-related cases only (excluding timeout). Env Model Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
CodeImpl
TestGen
S-only
Both
Total
S-only
Both
Total
S-only
Both
Total
62 (89%) 35 (70%) 30 (58%) 117 (76%) 22 (22%) 25 (61%)
7 (10%) 14 (28%) 19 (37%) 37 (24%) 77 (77%) 15 (37%)
70 50 52 154 100 41
83 (92%) 79 (98%) 86 (85%) 60 (72%) 32 (36%) 63 (79%)
7 (8%) 1 (1%) 14 (14%) 23 (28%) 56 (64%) 17 (21%)
90 81 101 83 88 80
138 (74%) 116 (74%) 65 (59%) 120 (54%) 43 (33%) 91 (61%)
37 (20%) 32 (20%) 38 (35%) 101 (46%) 88 (67%) 55 (37%)
187 157 110 221 132 148
only), but shows a higher joint collapse rate in TestGen (46%), suggesting that its strong isolated TestGen performance degrades more under integration pressure. J.3
Static vs. Dynamic Score Comparison
Table 29: Static and Dynamic sub-scores (0–1 scale) in Isolated (ISO) vs. FullCycle (FC). The CodeImpl dimension exhibits a stark reversal: Static declines while Dynamic surges, driven by the write-run-fix loop. Env
CodeImpl
TestGen
Model
SISO
SFC
DISO
DFC
SISO
SFC
DISO
DFC
SISO
SFC
DISO
DFC
Claude-Sonnet-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 MiniMax-M2.7 Qwen-3.5
0.91 0.88 0.88 0.81 0.56 0.83
0.85 0.88 0.87 0.66 0.46 0.85
0.85 0.76 0.74 0.69 0.61 0.75
0.98 0.97 0.93 0.90 0.56 0.95
0.68 0.66 0.69 0.64 0.53 0.56
0.61 0.62 0.63 0.59 0.32 0.56
0.57 0.54 0.52 0.51 0.43 0.43
0.90 0.89 0.89 0.81 0.43 0.80
0.91 0.86 0.74 0.81 0.53 0.73
0.69 0.70 0.66 0.56 0.26 0.56
0.86 0.83 0.76 0.82 0.55 0.77
0.81 0.83 0.81 0.71 0.36 0.74
Table 29 quantifies the Static-Dynamic reversal from Section 4.5: Env. For SOTA models, Static scores remain stable or decline slightly (Claude: 0.91 → 0.85) while Dynamic scores rise (Claude: 0.85 → 0.98), because cross-phase runtime feedback catches configuration defects that isolated evaluation misses. Kimi-K2.5 follows the typical pattern with a Static decline (0.81 → 0.66) and a strong Dynamic gain (0.69 → 0.90), consistent with the runtime feedback mechanism. MiniMax-M2.7 is the only model where both Env sub-scores decline. CodeImpl. The reversal is sharpest here. In Isolated mode, SOTA models show Static > Dynamic (Claude: 0.68 > 0.57), producing well-structured code that fails at runtime. In FullCycle, this inverts to Dynamic ≫ Static (Claude: 0.90 ≫ 0.61). The average Dynamic gain across SOTA models is +0.35, while Static declines by only −0.06: the write-run-fix loop improves runtime correctness but no equivalent signal guards structural quality. MiniMax-M2.7 shows no Dynamic improvement (0.43 → 0.43) alongside severe Static collapse (0.53 → 0.32). TestGen. Static scores drop for all models (Claude: 0.91 → 0.69, Qwen: 0.73 → 0.56), consistent with attention depletion at the pipeline tail. Dynamic scores are mixed: SOTA models maintain or slightly lose ground (Claude: 0.86 → 0.81), while GPT-5.4 improves (0.76 → 0.81) through the self-implementation knowledge mechanism. MiniMax-M2.7 collapses on both sub-scores.
26