A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

SWE-RPG: A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents Xin Zhou1 , Chun Yong Chong2 , Kisub Kim3 , Yun Peng4 , Rui Shu5 , Zihan Wu6 , Xu Han 7 , Guowen Yuan8 , Zeyang Zhuang4 , Jounghoon Kim7 , Jeongjin Ju3 , Seongmin Ju3 , Taein Yoon3 , David Lo1 1

arXiv:2608.09072v1 [cs.SE] 10 Aug 2026

Singapore Management University, Singapore 2 Monash University Malaysia, Malaysia 3 Daegu Gyeongbuk Institute of Science and Technology (DGIST), Republic of Korea 4 The Chinese University of Hong Kong, Hong Kong SAR, China 5 North Carolina State University, USA 6 City University of Hong Kong, Hong Kong SAR, China 7 The Hong Kong University of Science and Technology, Hong Kong SAR, China 8 The University of Hong Kong, Hong Kong SAR, China Abstract Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs. Yet existing repositorylevel benchmarks typically evaluate only whether the final patch passes tests. Satisfying a user request requires a long chain of interdependent reasoning and decisions: an agent must recover explicit and implicit requirements, formulate a repository-grounded implementation plan, and translate it into correct code. A pass/fail outcome cannot characterize how an unsuccessful trajectory diverges from the requirements and implementation process needed for a correct patch. To address this gap, we introduce SWE-RPG, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for (1) Requirement Clarification and (2) Implementation Planning. These intermediate GTs support retrospective, GT-aligned diagnosis of complete coding-agent trajectories across clarification, planning, code generation, and artifact submission. SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 LLM backends, including Claude-Sonnet-5 and GPT-5.6-Terra. Results show that the evaluated popular coding agents still struggle to implement user requests in existing repositories, achieving an average resolved rate of only 31.5% on SWE-RPG. Intermediate-GT diagnosis further identifies implicit requirement recovery as the main bottleneck, accounting for 24.5%– 46.0% of agent runs. This result suggests implicit-requirement recovery as a key candidate direction for improving coding agents. The benchmark data and evaluation code are available at https://github.com/Xin-Zhou-smu/SWE-RPG-Bench.

1

Introduction

Large language model-powered coding agents are reshaping software development by automating repository-level maintenance and evolution, from bug fixing (Yang et al. 2024) to feature implementation (Li et al. 2025). Reliably translating users’ raw requirements into correct changes in existing codebases is therefore central to their practical deploy-

Figure 1: Comparison of SWE-RPG with popular benchmarks. Unlike them, SWE-RPG jointly evaluates implicit requirement clarification, planning, and code generation.

ment. To assess this capability, repository-level issue resolution benchmarks have progressed from SWE-bench (Jimenez et al. 2024) to recent efforts such as RACE-bench (Liu et al. 2026) and Dialogue SWE-Bench (King and Flanigan 2026). These benchmarks provide valuable end-to-end insights by evaluating generated patches with executable tests. However, producing a correct patch requires a long chain of decisions spanning requirement clarification, implementation planning, and code generation. A final patch verdict cannot identify the actual bottleneck or reveal clear directions for improving coding agents. Limitations of Existing Benchmarks. We argue that faithfully diagnosing coding agents requires observing the complete issue-resolution1 process. Accordingly, we pose the following question: When a coding agent fails, does the primary bottleneck lie in 1 requirement clarification, 2 implementation planning, or 3 code generation? However, prior works cannot answer this question in three respects, as summarized in Table 1: (1) Requirement Clarification. 1 In repository-level issue-resolution task, we call the original input request either the issue description or the user requirement.

Benchmark

Time (yymm)

SWE-bench (Jimenez et al. 2024) SWE-bench Verified (OpenAI 2024) SWE-bench Multimodal (Yang et al. 2025) Multi-SWE-bench (Zan et al. 2025) SWE-PolyBench (Rashid et al. 2025) SWE-rebench (Badertdinov et al. 2025) SWE-bench-Live (Zhang et al. 2025) SWE-bench Pro (Deng et al. 2026b) SWE-Compass (Xu et al. 2026) SWE-Evo (Thai et al. 2025) EvoClaw (Deng et al. 2026a) RACE-bench (Liu et al. 2026) SWE-Explore (Zhang et al. 2026) Dialogue SWE-Bench (King and Flanigan 2026) SWE-RPG (Ours)

(1) Req. Clarification Implicit Req.

Categorized Ref.

(2) Planning Step Ref.

Reprod. Impl.

(3) Evaluation Exec. Patch

Stage Attr.

2310 2408 2410 2504 2504 2505 2505 2509 2511 2512 2603 2603 2606 2606 –

Table 1: Comparison of SWE-RPG with previous benchmarks. SWE-RPG combines implicit requirements, implementation-sufficient planning, and stage-aware evaluation. (1) Req. Clarification: Implicit Req. recovers implicit requirements; Categorized Ref. provides gold references under a practitioner-informed taxonomy. (2) Planning: Step Ref. provides actionable gold steps; Reprod. Impl. verifies the plans can generate equivalent code. (3) Evaluation: Exec. Patch evaluates patches with executable tests; Stage Attr. assigns main failure reasons to specific stages. / / denote no/partial/direct support.

In practice, initial requirement descriptions are often incomplete or ambiguous, requiring developers to recover implicit constraints from task and repository context (Knauss et al. 2015; Franch et al. 2023). Recent benchmarks expand evaluation to long-horizon and continuous-evolution tasks (Deng et al. 2026b; Thai et al. 2025; Deng et al. 2026a), assess high-level issue understanding (Liu et al. 2026), or study clarification in simulated dialogue (King and Flanigan 2026). However, none provides validated references for systematically evaluating agents’ recovery of implementation-critical requirements left implicit in original requests. (2) Planning. Implementation plans translate intended behavior into actionable repository changes (Bairi et al. 2024) and should be sufficient to generate functionally equivalent implementations. Most benchmarks provide no implementation-plan references. RACE-bench comes closest with structured task steps, but does not test their plan-to-code reproducibility. (3) Evaluation. Existing benchmarks primarily determine whether submitted patches pass executable tests, revealing whether agents fail but not where. Some provide moduleor failure-level analyses, but none consistently attributes unsuccessful trials to aligned requirement, planning, or coding stages. Our Solution. We propose SWE-RPG, a new stageaware benchmark for evaluating coding agents’ capabilities in requirement clarification, planning, and coding during repository-level issue resolution. SWE-RPG addresses the aforementioned limitations through three main features: (1) Requirement Clarification. SWE-RPG evaluates agents’ ability to recover implicit requirements using gold clarification references across six practitioner-informed dimensions (e.g., functional intent and technical context). (2) Planning. Each task provides implementation-sufficient gold plan steps, which we use to assess the quality of implementation plans generated by coding agents. (3) Evaluation. SWE-RPG evaluates final patches with executable tests, analyzes trajectories to assess clarification and planning, and

attributes failures to requirement, planning, or coding stages. Evaluation. We evaluate 3 coding-agent frameworks paired with 6 LLM backends. Agents achieve an average resolved rate of only 31.5%, with 46.7% of agent runs failing primarily during requirement clarification or implementation planning. Claude Code, Codex, and OpenCode also expose distinct stage-aware failure profiles across requirement, planning, implementation, and artifact stages. These findings highlight distinct capability bottlenecks and show how stage-aware evaluation can suggest candidate areas for improvement. In summary, our contributions are as follows: • We propose SWE-RPG, a stage-aware benchmark comprising 163 tasks from 31 Python/Java repositories. To our knowledge, it is the first repository-level benchmark to jointly evaluate agents’ capabilities in requirement clarification, implementation planning, and code generation. • We evaluate three popular coding-agent frameworks paired with six recent LLMs in a 3 × 6 matrix on SWERPG. Agents achieve an average resolved rate of only 31.5%, while 46.7% of all agent runs fail during requirement clarification or implementation planning. Our stageaware analyses reveal framework-specific bottlenecks and suggest candidate areas for improving coding agents.

2

Related Work

Repository-level Issue-resolution Benchmarks. SWEbench established an executable benchmark for repositorylevel issue resolution: agents modify code repositories to complete requirements, and their patches are evaluated with tests (Jimenez et al. 2024). Later benchmarks improve reliability and freshness (OpenAI 2024; Badertdinov et al. 2025; Zhang et al. 2025) or broaden programming-language and task coverage (Yang et al. 2025; Zan et al. 2025; Rashid et al. 2025; Deng et al. 2026b; Xu et al. 2026; Thai et al. 2025; Deng et al. 2026a). However, as Table 1 shows, most benchmarks primarily evaluate final patches and lack validated requirement and planning references for diagnosing unsuccessful trajectories. Close Works. RACE-bench (Liu et al. 2026) and Dialogue SWE-Bench (King and Flanigan 2026) are the two closest works. RACE-bench. RACE-bench is a concurrent work that shares our goal of exposing intermediate reasoning (Liu et al. 2026). However, the following two features distinguish SWERPG from RACE-bench. First, RACE-bench does not provide references for implicit requirements, a central challenge in requirement clarification; it summarizes only the highlevel intent already stated in the original requirement. In contrast, SWE-RPG provides evidence-supported references for implementation-critical information not fully specified in the initial request. Second, RACE-bench provides plan steps and validates their semantic necessity, but does not test whether code generated from them can reproduce a functionally equivalent code. In contrast, SWE-RPG directly checks plan-to-code reproducibility through code generation.

A GTs Construction for Requirment Clarification ① Practitioner-Guided Clarification Taxonomy & Examples

10 SE Engineers Interviews

Identify req. clarification categories and provide typical examples

② Synthesis

C1: Functional Intent (2 examples) C2: Business Semantics (12 examples) C3: Technical Context (12 examples) C4: Interface/Protocol Specification (13 examples) C5: Code Structure and Conventions (10 examples) C6: Data-Structure Semantics (8 examples)

Examples (QAs)

1. Analyze PR and Repo

Code Repos

2. Identify Missing Req.

Original Req. (Issue)

3. Clarify Missing Req.

Pull Request

Synthesis Agent

Inputs

③ Validation Clarified Req. Points (QA)

1. Code Repo 2. GT Patch 3. GT Plans

(Reconstrcucted Initial GT)

Validation Agent

Evidence Check

Clarified Req. Points (QA format) C1: QA Pair1 , ... C2: QA Pair1, ... C3: QA Pair1, ... Clarified Req. (Reconstructed Initial GT)

Req. (C1) Req. (C2) ....

Manual Review

B

GTs Construction for Planning

① Synthesis

Pull Request

3. Synthesize Plans

Inputs

Initial Planning

Plan Steps (Reconstructed Initial GT)

Synthesis Agent

② Validation Step-by-Step Validation

Step 1: ... Step 2: ... ...

2. Understand Raw Req.

Original Req. (Issue)

Plan (Step 1)

Initial Planning

1. Analyze PR and Repo

Code Repos

Apply Step 1’s Plan

Revise Plan

Code Generation Agent

Generate Patch

Patch Gap Analysis & Resolution Agent

Generated Patch (Step 1)

NO

Semantic Equivelent?

Gold Patch (Step 1)

Validation Agent

YES

Verfied GTs for Req. Clarification (Evidence Checked; Manually Reviewed )

Plan (Step 1)

Manual Review

Plan (Step 2)

....

Verfied GTs for Plannings (Can Replicate GT Patches ; Manually Reviewed )

Figure 2: Ground-truths Construction for Requirement Clarification and Planning. Using the original requirement, repository, merged PR, and tests, we synthesize Clarification GT and an actionable implementation plan. Clarification GT is validated for evidential support, and cross-stage consistency. Planning GT is validated step by step by checking whether each step can guide an implementation functionally equivalent to its gold subpatch. Finally, two authors independently review every resulting reference for evidential support and cross-stage consistency. Dialogue SWE-Bench. Dialogue SWE-Bench reformulates SWE-bench Verified (OpenAI 2024) tasks as interactive sessions. For each task, it constructs a shortened initial query by deliberately omitting critical details, then studies whether agents can recover them through simulated dialogue. The resulting clarification setting is therefore synthetic and do not provide references mined from naturally implicit requirements. In contrast, SWE-RPG retains the original request and mines implementation-critical implicit requirements from real development evidence. We also provide reproducible planning references, which Dialogue SWE-Bench lacks.

3

SWE-RPG Benchmark

SWE-RPG is a repository-level issue-resolution benchmark with executable patch oracles and validated intermediate GTs. In this section, we first formulate the benchmark task and its GT-aligned diagnostic protocol, then describe task and annotation construction and summarize the resulting dataset. Figure 2 summarizes how we reconstruct and validate the ground-truth references for Req. Clarification and Planning.

3.1

Task Formulation and Example

Input and Output. The input to each task consists of (i) a repository checked out at a specified base commit and (ii) a user requirement in natural language. Given these inputs, the coding agent autonomously modifies the repository and outputs a code patch. Example Instance. Figure 3 illustrates a SWE-RPG instance, in which TSV import incorrectly removes boundary whitespace when trimStrings is disabled. The instance provides stage-specific ground-truth artifacts for diagnosis: Clarification GT makes implicit requirements explicit, including the TSV-only scope and unchanged CSV behavior; Plan GT provides the implementation steps. The Gold Patch, Functional Tests, and Docker environment support

Example: OpenRefine-6609. Raw requirement. TSV import incorrectly removes boundary whitespace when trimStrings is disabled. Clarification GT. Functional Intent (C1) Q: Regarding the fixing scope, should the fix change whitespace handling for all separator-based imports? A: No. It must affect only the TSV path; CSV behavior must remain unchanged . . . Plan GT. Step 1: Locate the TSV branch in SeparatorBasedImporter.parseOneFile. Step 2: Disable Univocity’s automatic removal of leading and trailing whitespace in TsvParserSettings . . . Patch GT. TsvParserSettings settings = ...; + settings.setIgnoreLeadingWhitespaces(false); + settings.setIgnoreTrailingWhitespaces(false); ... Tests. readDoesNotTrimLeadingTrailingWhitespace ...

Environment: in a reproducible docker container

Figure 3: An illustrative SWE-RPG instance.

executable, reproducible patch evaluation. Together, these artifacts support stage-aware analysis and evaluation.

3.2

Benchmark Construction

Repository and Task Selection. SWE-RPG collects merged issue–PR pairs from mature, actively maintained Python and Java repositories. Each pair must include reproducible tests, involve a recent and traceable change, and not overlap with existing issue-resolution benchmarks. This screening yields 2000+ issue–PR instances with task-relevant tests from 100+ repositories. We further retain only bugfixing and new-feature tasks whose tests directly exercise the target behavior, resulting in 1000+ candidate instances.

Repo and Task Collection

Task Filtering

Environment Build

Test Validation

Intermediate GT Validation

2,000+ Eligible PR-Issue Pairs

1,000+ Relevant tasks

400+ With Runnable Environments

200+ With Stable Tests

163 With Verified Intermediate GTs

Table 2: Benchmark Candidate reduction during benchmark construction.

Tasks Java / Python tasks Bug fixes

SWE-RPG Dataset Composition 163 Repositories 85 / 78 Java / Python repos. 113 (69.3%) Feature implementations Task Statistics

Statistic Requirement Words Codebase LOC Gold-patch LOC Test cases Statistic # of Implicit Clarification Points C1 Functional intent C2 Business semantics C3 Technical context

31 17 / 14 50 (30.7%)

Plan GT Avg. / Max. 301.10 / 2,195 272.51K / 1.59M 56.74 / 639 2,248.56 / 15,119

Statistic Plan steps File locations Rule constraints

Clarification GT Avg. / Max. Statistic 4.53 / 8 1.21 / 4 C4 Interface/protocol 0.60 / 3 C5 Structure/conventions 0.72 / 2 C6 Data-structure semantics

Avg. / Max. 2.06 / 10 2.47 / 10 11.80 / 50

Avg. / Max. 0.99 / 2 0.47 / 1 0.54 / 2

Table 3: Statistics of SWE-RPG and its intermediate GTs. Building Environments. We reuse the automated agentic build pipeline of SWE-bench-Live (Zhang et al. 2025) to construct an isolated Docker image for each candidate at its base commit. We extend this pipeline with expert-guided recovery: when an automated build fails, the first author inspects its logs, identifies missing dependencies or configuration, and feeds these corrections into the next build attempt. We validate each image by comparing test outcomes before and after applying the gold patch. We repeat tests to remove flaky instances and manually verify that fail-to-pass tests directly target the intended behavior. This process yields 400+ runnable images and 200+ qualified instances. Overview of Ground-Truth Construction. We construct ground truth differently for the three stages. Original requirements often omit implementation-critical details (Bo et al. 2024; Knauss et al. 2015; Franch et al. 2023), while existing benchmarks generally provide no explicit implementationplan references (Table 1). We therefore construct the Requirement Clarification and Planning references through synthesis and validation. During synthesis, we derive the references from the base repository, issue and PR discussion, developer patch, and selected tests. During validation, we check them against the development evidence and implemented behavior. We instantiate the agentic components of this construction pipeline with GPT-5.4 (OpenAI 2026c), the latest available model at the time of benchmark construction, to maximize the quality of the synthesized references. For Code Generation, no synthesis is needed: the merged PR provides the developer patch, and its task-relevant tests provide the executable correctness oracle. We detail the synthesis and validation procedure for each stage below. Ground-Truth Construction for Req. Clarification. Practitioner-Guided Clarification Taxonomy. Requirement clarification is difficult to define because it targets implementation-critical requirements that remain implicit in

the original request. Which implicit decisions matter to developers, and which aspects they repeatedly clarify in real software development, are unknown to some extent. We therefore conducted seminar-style interviews with ten experienced software engineers from Fortune Global 500 technology companies. We asked what information developers need before implementation, what missing information they recover during clarification, and which omissions most often cause incorrect implementations. Their responses served two purposes: deriving a practitioner-grounded taxonomy and collecting representative QA examples for GT synthesis. The resulting taxonomy uses standardized QA pairs and comprises six categories: C1: Functional Intent, C2: Business Semantics, C3: Technical Context, C4: Interface and Protocol Specifications, C5: Code Structure and Naming Conventions, and C6: Detailed Data-Structure Semantics. We curated 57 seed question-answer pairs. Synthesis and Validation. As illustrated in Figure 2, the taxonomy guides a synthesis-and-validation pipeline. Using the corresponding seed QA examples as few-shot demonstrations, synthesis agents examine the issue, repository, and merged PR to identify implementation-critical information that is underspecified or implicit in the issue and formulate candidate questions. Next, a validation agent checks each QA pair against the repository, developer patch, and validated plan for evidential support, cross-stage consistency, and absence of implementation detail leakage. Finally, two authors independently review every resulting clarification reference for evidential support and cross-stage consistency. Ground-Truth Construction for Planning. Planningground-truth construction has two phases: synthesis and validation. During synthesis, a single synthesis agent examines the original issue, base repository, and merged PR, recovers the implementation intent, explains how the developer patch realizes it, and organizes these findings into an ordered sequence of modular steps. Each step specifies its goal, relevant files, intended changes, and repository constraints. These fields make the step actionable and auditable: they tell a coding agent what to achieve, where and how to modify the repository, and which constraints to preserve, while enabling step-level alignment with the gold patch. During validation, an alignment agent associates each plan step with the corresponding subset of the complete developer patch. We call this subset the step’s gold subpatch. We validate the plan sequentially, one step at a time. For each step, a coding agent follows the step to generate a patch, and an evaluator judges whether it is semantically equivalent to the corresponding gold subpatch. If the check fails, a gap-analysis and resolution agent revises the step plan. Steps that pass are retained. This process establishes functional reproducibility: an ac-

Average Time: 8.5 min

Average Cost: $1.59 OC-KimiK3

50

CC-KimiK3

45

45

Codex-KimiK3

40

Codex-DSV4

OC-GLM5.2

CC-DSV4

OC-DSV4

35

Average Resolve Rate: 31.5% 30

OC-Claude5 Codex-GLM5.2

25

20

CC-Claude5 CC-M3 Codex-Claude5

OC-GPT5.6 CC-GPT5.6

Codex-GPT5.6 OC-M3

Codex-M3

CC-GLM5.2

OC

15 10−1

Resolve Rate (%)

Codex-KimiK3 Resolve Rate (%)

OC-KimiK3

50

CC-KimiK3

40 35 30

OC-DSV4

Average Resolve Rate: 31.5% CC-Claude5 OC-Claude5

OC-GPT5.6 25CC-GPT5.6

Codex

100

15

Codex-GLM5.2

CC-GLM5.2 OC-M3

Codex-M3 OC

2

4

6

8

10

Average Time per Task (min)

Average Cost per Task (USD, Log Scale)

(a) Cost versus resolve rate

CC-M3 Codex-Claude5

Codex-GPT5.6

20 CC

Codex-DSV4

OC-GLM5.2

CC-DSV4

12

CC

14

Codex

16

(b) Time versus resolve rate

Figure 4: Cost and execution time versus resolve rate.

36.2%

36.8%

25.2%

20.2%

49.7%

CC

31.3%

39.3%

25.8%

22.7%

28.8%

49.1%

Codex

26.4%

40.5%

23.3%

22.1%

17.8%

41.1%

Claude5

DSV4

GLM5.2

GPT5.6

M3

KimiK3

Agent

30.1%

80 60 40 20

Resolve Rate (%)

100

OC

0

LLM

Figure 5: Resolve rates across agent–LLM combinations.

lines of code (LOC) on average, with the largest reaching 1.59M LOC; and each task contains 2,248.56 functional test cases on average. For each task, Clarification GT recovers 4.53 implicit and implementation-critical requirement points on average, while Plan GT provides an average of 2.06 actionable implementation steps with 11.80 implementation constraints. Our benchmark covers across 7 engineering domains, including static analysis, compilers, and schema systems, algorithms, mathematics, and scientific computing.

4 cepted plan must provide sufficient guidance for a coding agent to produce a functionally equivalent implementation. Finally, two authors independently review every resulting plan for evidential support and cross-stage consistency. Ground-Truth Construction for Code Generation. Unlike Req. Clarification and Planning, Code Generation requires no annotation reconstruction. Each merged PR directly provides the developer implementation patch, while its taskrelevant tests serve as the executable correctness oracle. Quality Assurance. As described above, we enforce quality control through the following stage-specific validation and expert-review measures. At the task level, we retain tasks with buildable environments and stable tests. For Planning, every step must support an implementation functionally equivalent to its gold subpatch. For Requirement Clarification, every point (QA pair) must be supported by repository evidence, consistent with the validated plan and final implementation. Lastly, human experts review every retained task and its GTs across all three stages.

3.3

Benchmark Statistics and Characteristics

Table 2 demonstrates the rigor and selectivity of our qualityoriented construction: successive checks of task relevance, environment reproducibility, test stability, and GT validity reduce more than 2,000 real-world PR–issue candidates to 163 quality-controlled instances. Table 3 further reflects substantial repository-scale complexity: codebases contain 272.51K

Experimental Setup

Evaluated Agents and Models. We evaluate three popular coding-agent frameworks (Claude Code, Codex, and OpenCode) paired with six recent LLMs: Claude-Sonnet5 (Anthropic 2026), DeepSeek-V4-Pro (DeepSeek-AI 2026), GLM-5.2 (Z.AI 2026), GPT-5.6-Terra (OpenAI 2026b), MiniMax-M3 (MiniMax 2026), and MoonshotAI-KimiK3 (Moonshot AI 2026). We run each agent-LLM configuration twice on every task and report run-level aggregates over the two runs. Computing Infrastructure. All experiments were conducted on Ubuntu 26.04 LTS using an Intel Xeon 6982P-C CPU (48 cores and 96 threads) with 182 GiB of RAM. We accessed all LLMs through their official API providers. Evaluation Metric. We use Resolve Rate as the primary evaluation metric, following the SWE-bench execution protocol (Jimenez et al. 2024). A task is considered resolved only if its patch applies successfully, all fail-to-pass tests pass, and no pass-to-pass test regresses. Failure Attribution to Stages. For each unresolved run, we use GPT-5.6-Sol (OpenAI 2026a) as an LLM judge. Its dedicated prompt jointly examines the task, intermediatestage GTs, and the agent trajectory. To make long trajectories tractable without discarding stage-relevant evidence, we remove only verbose tool-call payloads and outputs unrelated to requirement understanding, planning, implementation, or verification, while retaining relevant repository observations, decisions, edits, and validation actions. The judge assigns

Resolved

100 16.0%

Percentage of Agent Runs (%)

80

60

6.7%

25.8%

23.9% 12.3%

5.5% 24.5%

Planning Failure

7.4%

9.8% 18.4%

Requirement Failure

12.9%

17.2%

25.2%

6.1% 9.2%

18.4%

15.3%

9.8%

14.7%

8.0% 40.5%

35.6%

Code Generation Failure

26.4%

39.3%

17.8% 10.4%

19.6%

Others (Including Timeout, Patch Format, etc.)

14.1%

11.0%

0

24.5%

37.4%

33.1%

11.0%

10.4% 30.7%

40.5%

39.3%

40.5%

37.4%

38.7%

42.9%

8.6% 46.0%

39.9%

35.6%

29.4% 49.7%

OC KimiK3

49.1%

CC KimiK3

41.1%

Codex KimiK3

40.5%

Codex DSV4

39.3%

CC DSV4

36.8%

OC GLM5.2

36.2%

OC DSV4

31.3%

CC Claude5

30.1%

OC Claude5

28.8%

CC M3

36.2%

17.8%

6.1%

40

20

14.1%

22.7% 17.8%

14.7%

6.1%

33.7%

26.4%

25.8%

25.2%

23.3%

22.7%

22.1%

20.2%

17.8%

Codex Claude5

CC GLM5.2

OC GPT5.6

Codex GLM5.2

CC GPT5.6

Codex GPT5.6

OC M3

Codex M3

Figure 6: Failure diagnoses across agent–LLM configurations. Green denotes resolved runs; the remaining segments show the proportions attributed to requirement, planning, code-generation, and other failures (e.g., timeouts).

each failure to the earliest deviating stage: requirement understanding, planning, implementation, or others. On a stratified sample of 50 unresolved runs, its labels exactly match human consensus in 46 cases (92%). Pre-code Stage Evaluation. Our Clarification and Plan GTs decompose each task into labeled information points. Clarification spans six categories (C1–C6), whereas Planning covers the goal, target location, implementation approach, constraints, and validation strategy. For each run, the LLM judge (GPT-5.6-Sol) determines whether the agent’s trajectory covers each GT information point. Because trajectories can be long, we remove verbose tool-call payloads and outputs that carry no evidence about requirement understanding or planning, while retaining relevant repository observations, decisions, edits, and validation actions. We compute category- and dimension-level coverage rates to evaluate the Clarification and Planning stages from full-trajectory evidence rather than from the final patch outcome alone. To accommodate multiple valid plans, the prompt evaluates semantic implementation responsibilities rather than exact plan matching: it accepts alternative files, symbols, architectures, and step orderings when they realize the same responsibility and satisfy repository constraints, without requiring textual or structural similarity to the reference plan. On a manually annotated sample of 50 information-point assessments, the judge’s binary coverage decisions agree with the human consensus in 96% of cases.

5 5.1

Results and Analysis

Patch Correctness

Overall Patch Correctness Results. On average, coding agents achieve a resolved rate of only 31.5%, indicating the difficulty of the SWE-RPG tasks. At the configuration level, OpenCode with MoonshotAI-Kimi-K3 achieves the highest resolved rate (49.7%). Thus, even the best configuration leaves roughly half of the tasks unresolved, indicating sub-

stantial room for improvement in repository-level issue resolution. Effectiveness of Agents, LLMs, and Pairings. Figure 5 compares performance at the agent, LLM, and agent– LLM pairing levels. Among agents, OpenCode achieves the highest average resolved rate (33.0%), followed closely by Claude Code (32.8%) and Codex (28.5%). Among LLMs, MoonshotAI-Kimi-K3 performs best, averaging 46.6% across the three agents, followed by DeepSeekV4-Pro at 38.7%; the remaining backends average 22.3– 29.2%. At the pairing level, OpenCode–MoonshotAI-KimiK3 is strongest (49.7%), closely followed by Claude Code– MoonshotAI-Kimi-K3 (49.1%), whereas Codex–MiniMaxM3 is weakest (17.8%). The resulting spread shows substantial descriptive variation across backend–scaffold pairings. Cost Analysis. Figure 4(a) relates resolve rate to cost, with dashed lines marking the average cost ($1.59 per task) and average resolve rate (31.5%). OpenCode–MoonshotAI-KimiK3 achieves the highest resolve rate at below-average cost, whereas Claude Code with the same backend attains similar performance at above-average cost. DeepSeek-V4-Pro pairings likewise perform above average but vary substantially in cost, showing that cost-effectiveness depends on both the backend and agent scaffold. Time Analysis. Figure 4(b) shows a partial performance– time trade-off. Many above-average configurations, particularly those using MoonshotAI-Kimi-K3, exceed the average execution time of 8.5 minutes. However, Claude Code– DeepSeek-V4-Pro performs above average in substantially less time, while several slower configurations remain below average. Thus, longer execution does not consistently produce better patches, and pairings exhibit distinct time– performance trade-offs.

5.2

Failure Diagnosis

Patch correctness reveals whether a run succeeds, but not why it fails. We therefore align agent trajectories with vali-

Planning

100

OC

54.8%

37.7%

49.5%

33.6%

29.3%

26.1%

41.2%

43.8%

31.1%

24.8%

CC

86.6%

74.2%

74.2%

64.7%

54.2%

63.9%

74.8%

79.7%

64.5%

41.6%

Codex

75.1%

56.4%

63.8%

47.3%

42.0%

41.9%

60.8%

65.6%

49.0%

37.1%

C1 Intent & Scope

C2 Business

C3 Technical

C4 Interface

C5 Structure

C6 Data Semantics

Goal

Location

Approach

Constraints & Validation

80 60 40 20

Quality Score (%)

Capability Analysis by Agent

Requirement Clarification

0

Figure 7: Clarification-category and planning-dimension coverage by agent, averaged across six LLMs. Boxes mark the lowest scores.

dated Clarification and Plan GTs to diagnose stage-specific failures. Figure 6 decomposes results into resolved cases, and unresolved cases are attributed to main failure stages. Requirement failure is the largest component for most configurations (24.5–46.0%), followed by codegeneration (7.4–37.4%) and planning (5.5–17.8%) failures. This ordering identifies implicit requirement recovery as the most common bottleneck, while planning errors are also non-negligible. Code-generation failure rates show substantial descriptive variation across LLM–scaffold pairings, particularly for several MiniMax-M3 pairings. Similar resolve rates mask different bottleneck profiles. Codex– and Claude Code–DeepSeek-V4-Pro achieve similar resolve rates (40.5% versus 39.3%) but markedly different profiles: their requirement-failure rates are 26.4% and 40.5%, whereas their implementation/verification-failure rates are 23.9% and 7.4%. Similar end-to-end performance can therefore accompany distinct stage-attribution profiles. SWE-RPG suggests candidate areas for agent improvement. Grounded in validated intermediate GTs, SWE-RPG provides evidence of where failures arise and identifies candidate modification areas beyond what previous outcomeonly evaluation can support. These diagnoses suggest stagespecific directions: requirement failures motivate investigating stronger implicit-constraint recovery, while planning failures motivate investigating better implementation plans.

5.3

Req. Clarification and Planning of Agents

We compare each agent’s trajectory with the validated Clarification and Plan GT information points to measure how completely it captures the expected pre-code information. Figure 7 reports coverage rates by clarification category and planning dimension, averaged across 6 LLMs for each agent. Lower clarification coverage is concentrated in interfaces, structure, and data semantics. All three agents cover intent and scope most reliably, but weaken on the implementation-facing categories. Claude Code’s lowest clarification score is code structure (54.2%); Codex falls to 42.0% on structure and 41.9% on data semantics; These results suggest candidate requirement-clarification targets, including interface contracts, repository conventions, and data invariants beyond the high-level intent.

Lower planning coverage is concentrated in approach, constraints, and validation. For every agent, planning coverage decreases from target location to implementation approach and then to constraints and validation: 79.7% to 64.5% to 41.6% for Claude Code, 65.6% to 49.0% to 37.1% for Codex, and 43.8% to 31.1% to 24.8% for OpenCode. This consistent cascade shows that agents often identify where to edit without fully specifying how to implement and verify the change. These lower-coverage dimensions suggest evaluating plans that more explicitly capture the implementation mechanism, boundary conditions, compatibility requirements, and validation obligations.

6

Conclusion

SWE-RPG combines 163 real PR-derived tasks, executable patch oracles, and validated Clarification and Plan GTs. Beyond measuring final patch correctness, it supports fulltrajectory, GT-aligned retrospective diagnosis of requirement, planning, implementation, and artifact gaps. These diagnostic profiles make otherwise identical unresolved outcomes more interpretable and suggest stage-specific hypotheses for future improvement. The Code and Data Supplement contains the code needed to run the benchmark and reproduce the reported analyses.

Limitations SWE-RPG contains 163 Python and Java tasks from 31 repositories, so its findings may not generalize to other languages or ecosystems. Because each agent runs autonomously end to end, the benchmark cannot assess the effect of human feedback or interactive refinement. Its intermediate GTs and retrospective alignment labels depend on LLM-assisted construction and judging, although all GTs are manually validated. Future work should expand task and model coverage and strengthen annotation audits, including inter-annotator agreement. Generative AI tools assisted with manuscript drafting and language editing; the authors reviewed and verified all content, claims, references, and final wording.

References Anthropic. 2026.

Introducing Claude Sonnet 5.

https:

//www.anthropic.com/news/claude-sonnet-5. Model announcement; accessed July 29, 2026. Badertdinov, I.; Golubev, A.; Nekrashevich, M.; Shevtsov, A.; Karasik, S.; Andriushchenko, A.; Trofimova, M.; Litvintseva, D.; and Yangel, B. 2025. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. In Advances in Neural Information Processing Systems, volume 38. Bairi, R.; Sonwane, A.; Kanade, A.; D. C., V.; Iyer, A.; Parthasarathy, S.; Rajamani, S.; Ashok, B.; and Shet, S. 2024. CodePlan: Repository-Level Coding Using LLMs and Planning. Proceedings of the ACM on Software Engineering, 1(FSE): 675–698. Bo, L.; Ji, W.; Sun, X.; Zhang, T.; Wu, X.; and Wei, Y. 2024. ChatBR: Automated Assessment and Improvement of Bug Report Quality Using ChatGPT. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 1472–1483. ACM. DeepSeek-AI. 2026. DeepSeek-V4-Pro API Documentation. https://api-docs.deepseek.com/. Model documentation; accessed July 29, 2026. Deng, G.; Chen, Z.; Yu, Z.; Fan, H.; Liu, Y.; Yang, Y.; Parikh, D.; Kannan, R.; Cong, L.; Wang, M.; Zhang, Q.; Prasanna, V. K.; Tang, X.; and Wang, X. 2026a. EvoClaw: Evaluating AI Agents on Continuous Software Evolution. In Proceedings of the 43rd International Conference on Machine Learning. Deng, X.; Da, J.; Pan, E.; He, Y. Y.; Ide, C.; Garg, K.; Lauffer, N.; Park, A.; Pasari, N.; Rane, C.; Sampath, K.; Krishnan, M.; Kundurthy, S.; Hendryx, S. M.; Wang, Z.; Zhang, C. B. C.; Jacobson, N.; Liu, B.; and Kenstler, B. 2026b. SWEBench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? In Proceedings of the 43rd International Conference on Machine Learning. Franch, X.; Palomares, C.; Quer, C.; Chatzipetrou, P.; and Gorschek, T. 2023. The State-of-Practice in Requirements Specification: An Extended Interview Study at 12 Companies. Requirements Engineering, 28(3): 377–409. Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. R. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In International Conference on Learning Representations. King, B.; and Flanigan, J. 2026. Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents. arXiv preprint arXiv:2606.13995. Knauss, E.; Damian, D.; Cleland-Huang, J.; and Helms, R. 2015. Patterns of Continuous Requirements Clarification. Requirements Engineering, 20(4): 383–403. Li, W.; Zhang, X.; Guo, Z.; Mao, S.; Luo, W.; Peng, G.; Huang, Y.; Wang, H.; and Li, S. 2025. FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 17160–17176. Liu, S.; Zhao, Z.; Hu, X.; Liu, K.; Yang, X.; and Xia, X. 2026. RACE-bench: A Benchmark for Evaluating Repository-

Level Code Agents with Intermediate Reasoning on Feature Addition Task. arXiv preprint arXiv:2603.26337. MiniMax. 2026. MiniMax M3: Frontier Coding, 1M Context, Native Multimodality. https://www.minimax.io/blog/ minimax-m3. Model announcement; accessed July 29, 2026. Moonshot AI. 2026. Kimi K3: Our Most Capable Model. https://forum.moonshot.ai/t/kimi-k3-is-here-ourmost-capable-model/480. Official model announcement; accessed July 29, 2026. OpenAI. 2024. Introducing SWE-bench Verified. https: //openai.com/index/introducing-swe-bench-verified/. Technical blog post; accessed July 27, 2026. OpenAI. 2026a. GPT-5.6 Sol Model. https://developers. openai.com/api/docs/models/gpt-5.6-sol. Model documentation; accessed July 29, 2026. OpenAI. 2026b. GPT-5.6 Terra Model. https://developers. openai.com/api/docs/models/gpt-5.6-terra. Model documentation; accessed July 29, 2026. OpenAI. 2026c. Introducing GPT-5.4. https://openai.com/ index/introducing-gpt-5-4/. Technical blog post; accessed July 29, 2026. Rashid, M. S.; Bock, C.; Zhuang, Y.; Buchholz, A.; Esler, T.; Valentin, S.; Franceschi, L.; Wistuba, M.; Sivaprasad, P. T.; Kim, W. J.; Deoras, A.; Zappella, G.; and Callot, L. 2025. SWE-PolyBench: A Multi-Language Benchmark for Repository Level Evaluation of Coding Agents. arXiv:2504.08703. Thai, M. V. T.; Le, T.; Manh, D. N.; Nhat, H. P.; and Bui, N. D. Q. 2025. SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios. arXiv preprint arXiv:2512.18470. Xu, J.; Deng, K.; Li, W.; Yu, S.; Tang, H.; Huang, H.; Lai, Z.; Zhan, Z.; Wu, Y.; Zhang, C.; et al. 2026. SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models. In Proceedings of the 43rd International Conference on Machine Learning. Yang, J.; Jimenez, C. E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K.; and Press, O. 2024. SWE-agent: Agentcomputer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, volume 37. Yang, J.; Jimenez, C. E.; Zhang, A. L.; Lieret, K.; Yang, J.; Wu, X.; Press, O.; Muennighoff, N.; Synnaeve, G.; Narasimhan, K. R.; Yang, D.; Wang, S. I.; and Press, O. 2025. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? In The Thirteenth International Conference on Learning Representations. Z.AI. 2026. GLM-5.2: Built for Long-Horizon Tasks. https: //z.ai/blog/glm-5.2. Model announcement; accessed July 29, 2026. Zan, D.; Huang, Z.; Liu, W.; Chen, H.; Zhang, L.; Xin, S.; Chen, L.; Liu, Q.; Zhong, X.; Li, A.; et al. 2025. Multi-SWEbench: A Multilingual Benchmark for Issue Resolving. In Advances in Neural Information Processing Systems, volume 38. Zhang, L.; He, S.; Zhang, C.; Kang, Y.; Li, B.; Xie, C.; Wang, J.; Wang, M.; Huang, Y.; Fu, S.; Nallipogu, E.; Lin, Q.; Dang,

Y.; Rajmohan, S.; and Zhang, D. 2025. SWE-bench Goes Live! In Advances in Neural Information Processing Systems, volume 38. Zhang, S.; Wang, Y.; Liang, J.; Shi, Y.; Zeng, W.; Wang, M.; He, S.; Xu, N.; Ye, S.; Cai, K.; and Gu, X. 2026. SWEExplore: Benchmarking How Coding Agents Explore Repositories. arXiv preprint arXiv:2606.07297.

Related documents

• Constraint Decay: The Fragility of LLM Agents in Backend Code Generation#1035486
• Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks#1128610
• SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents#927033
• AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering#1116993
• Code2Math: Can Your Code Agent Evolve Math Problems Through Exploration?#1128563