SWE-C HAIN: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
arXiv:2605.14415v1 [cs.SE] 14 May 2026
Man Ho Lam1
Chaozheng Wang1∗ Hange Liu2 Jingyu Xiao1 Haau-sing Li3,4 Jen-tse Huang5 Terry Yue Zhuo6 Michael R. Lyu1 1 2 3 The Chinese University of Hong Kong Independent ELLIS 4 5 6 Technical University of Darmstadt Johns Hopkins University Monash University [email protected] [email protected] GitHub Code
HuggingFace Dataset
Abstract Coding agents powered by large language models are increasingly expected to perform realistic software maintenance tasks beyond isolated issue resolution. Existing benchmarks have shifted toward realistic software evolution, but they rarely capture continuous maintenance at the granularity of package releases, where changes are bundled, shipped, and inherited by subsequent versions. We present SWE-C HAIN, a benchmark for evaluating agents on chained release-level package upgrades, where each transition builds on the agent’s prior codebase. To produce upgrade specifications, we design a divide-and-conquer synthesis pipeline that aligns release notes with code diffs for each version transition, ensuring the requirements are grounded in actual code changes, informative to agents, and feasible to implement. SWE-C HAIN contains 12 upgrade chains across 9 real Python packages, with 155 version transitions and 1,660 grounded upgrade requirements. Across nine frontier agent-model configurations, agents achieve an average of 44.8% resolving, 65.4% precision, and 50.2% F1 under the Build+Fix regime, with Claude-Opus-4.7 (Claude Code) leading at 60.8% resolving, 80.6% precision, and 68.5% F1. These results show that SWE-C HAIN is both feasible and discriminative, and reveal that current agents still struggle to make correct upgrades across chained package releases without breaking existing functionality.
1
Introduction
Powered by advanced Large Language Models (LLMs), coding agents such as Codex [OpenAI, 2025] and Claude Code [Anthropic, 2025] have evolved into software engineering assistants that can navigate repositories, execute commands, inspect errors, and edit code through tool-use interfaces [Yang et al., 2024a, Wang et al., 2025a, Xie et al., 2025, Xia et al., 2025]. Accordingly, benchmarks have evolved from task-level problems [Liu et al., 2023, Gu et al., 2024, Jain et al., 2025, Zhuo et al., 2025, Lam et al., 2025] to repository-level issue resolution in real-world projects [Jimenez et al., 2024, Liu et al., 2024, Li et al., 2024, Deng et al., 2025]. Complementary efforts further broaden software-engineering evaluation to multimodal settings involving visual artifacts [Yang et al., 2024b, Wan et al., 2024, Xiao et al., 2025b,a]. More recently, benchmarks like τ -Bench [Yao et al., 2025], Terminal-Bench [Merrill et al., 2026], and AgencyBench [Li et al., 2026] have shifted towards realistic software development scenarios with tool use, terminal interaction, and multi-turn decision-making. Consequently, growing attention has shifted towards long-horizon software engineering that goes beyond isolated issue resolution, raising a fundamental design question: how should long software ∗ Corresponding author.
Preprint.
trajectories be segmented and evaluated? SWE-CI [Chen et al., 2026] defines tasks by benchmarkselected base-to-oracle commit pairs, SlopCodeBench [Orlanski et al., 2026] uses synthetic checkpoints, and EvoClaw [Deng et al., 2026] reconstructs mid-release milestones. These scenarios provide important insights into modeling software engineering lifecycles, exposing failures in iterative code degradation and in continuous integration (CI)-style maintainability. However, none of these segmentation boundaries corresponds to a moment when an upstream maintainer actually ships a software version. Specifically, they capture merely part of the real-world package maintenance dynamics, where changes are bundled at release boundaries, and earlier mistakes can carry over to later releases. SWE-EVO [Thai et al., 2026] takes the first step in this direction by treating release versions as evaluation units. However, its specifications are constructed by concatenating release notes with raw content from GitHub issues and pull requests, leaving agents to process noisy artifacts such as terminal dumps, reproduction snippets, and image references rather than unified and coherent upgrade specifications. Furthermore, each task is an isolated end-to-end release transition, which does not evaluate whether agents can continuously maintain a codebase built on their previous modifications. This motivates our central research question: Can agents maintain a real package through a chain of releases while carrying their own changes forward without breaking existing functionality? We therefore introduce SWE-C HAIN, a benchmark for evaluating coding agents on chains of consecutive release upgrades. To address the noisy-specification issue of prior release-level evaluation, we introduce DecompSynth, a divide-and-conquer synthesis pipeline that synthesizes specifications from maintainer-authored release notes and actual behavioral changes in the gold code diff, ensuring that the requirements in the specification are grounded in actual code changes, informative to the agent, and feasible to implement. The version-aligned specifications allow us to compose individual version transitions into upgrade chains, in which the agent-produced codebase at one release becomes the starting point for the next, enabling us to model the release-level maintenance cycle. Our contributions include: 1. SWE-C HAIN: We introduce the first benchmark for chained release-level maintenance, comprising 12 upgrade chains across 9 packages, 155 version transitions, and 1,660 grounded tasks. 2. DecompSynth: We implement and open-source a lightweight yet scalable pipeline that leverages release notes and code diffs to construct high-quality upgrade specifications, replacing manual authoring and noisy one-shot synthesis. 3. Comprehensive evaluation: We evaluate 9 frontier model-agent configurations, achieving an average resolving rate of 44.8%, precision of 65.4%, and F1 of 50.2% under the Build+Fix regime, which allows one fix attempt for execution-level errors. These results show current agents can partially solve chained upgrades but still struggle to preserve functionality across releases.
2
SWE-C HAIN
This section outlines the construction of SWE-C HAIN, including data collection, environment setup, cross-version validation, specification synthesis, and evaluation protocol, as illustrated in Figure 1. 2.1
Benchmark Construction
We collect released package versions from PyPI or GitHub and retrieve the corresponding release notes, as well as relevant GitHub issues and pull requests (PRs). Table 1 summarizes the selected upgrade chains and their scale, with more details in Appendix B. We then implement lightweight, package-specific fetchers and decomposition functions to automatically parse release notes into raw tasks, followed by a manual quality review to normalize the formatting and exclude unsuitable releases or chains. To ground each version upgrade in implementation evidence, we extract the Python code and test diffs between consecutive versions using git diff, which we then use to synthesize the specification in §2.2. Docker environment setup. To provide a consistent chain-level Docker environment, we manually design a Dockerfile for each chain. Each Dockerfile pins the Python interpreter and cross-version dependencies, while allowing other packages to resolve to their release-specific versions. This enables 2
ver. 2.0
Step 1: Benchmark Construction Diff (2.0.1-2.0.2) diff --git ... --- a/src/flask/scaffold.py +++ b/src/flask/scaffold.py @@ -21,7 +21,7 @@
ver. 2.0.1 Release Note 2.0.2 Task 1: Fix teardown_* [#4093] ... Task 5: Enhance typing [#4095] ...
Docker Environment app
test_options_work test_request_dispatching
docs
from .templating import + ...
src/flask Task 1: Fix teardown_* ... #4093: Wrong typing for teardown_request decorator According to ...
Batch 1 (7 files)
diff --git a/.../typing.py @@ -33,14 +33,24 @@ ...
diff --git a/.../scaffold.py @@ -21,7 +21,7 @@
Specification
...
Task 5: Enhance typing ...
diff --git a/.../typing.py
#4095: Improve typing for errorhandler decorator Today the errorhandler ...
Resolved 3/11
cli.py
Description Requirement Constraints Expectations Behaviors Acceptance Criteria
Regressed 18
tests
Preserved 442
specs_2.0.1_2.0.2.md
error_report.txt
Write the error report
Agent Workspace
Task 5
Step 2: Divide-and-Conquer Specification Synthesis
Build
app.py
Task 1: Correct typing..
@@ -29,6 +29,7 @@
@@ -33,14 +33,24 @@
test_nested_callback_order test_dotenv_optional test_find_best_app test_flaskgroup_debug[True] test_flaskgroup_debug[False]
code
Ensuring no error & < 0.25% fail
ver. 2.0.2
Fix
isolation
Hidden test
Step 3: Version-Upgrade Protocal
ver. 2.3.3
Figure 1: Overview of the SWE-C HAIN pipeline. For each version upgrade, we collect release notes, extract diffs, and prepare environments. An agent matches hunks to tasks and synthesizes specifications, which are used to evaluate agents sequentially, with an extra fixing step for errors. Table 1: Scale of the chains in SWE-C HAIN. LOCD is the total Python code diff lines across all tests upgrade steps, and LOCno counts lines in the starting codebase excluding the test suite. Struckstart through versions denote skipped releases (e.g., yanked versions or negligible upgrades). Package Chain attrs conan Flask Jinja2 poetry PyJWT pytest urllib3 xarray
21.3.0 → 26.1.0 2.12.0 → 2.20.1 2.23.0 → 2.28.1 2.0.0 → 2.3.3 2.8 → 2.10.3 1.5.0 → 1.8.5 (1.6.1) 2.0.0 → 2.12.1 7.0.0 → 7.4.4 8.0.0 → 8.3.5 (8.1.0, 8.3.1) 2.0.7 → 2.6.3 2022.11.0 → 2023.7.0 2025.6.0 → 2026.2.0
tests #Upgrades #Tasks (+Extra) LOCD LOCno start
13 16 11 17 12 10 15 16 12 12 10 11
92(+6) 243(+18) 167(+6) 107(+4) 67(+7) 196(+3) 145(+5) 134(+6) 114(+7) 72(+7) 87(+7) 141(+19)
7, 834 13, 901 7, 284 8, 013 4, 730 8, 977 3, 369 7, 225 10, 328 8, 141 13, 082 14, 120
13k 67k 76k 29k 23k 21k 3.4k 79k 89k 17k 119k 152k
a single image to run tests for all versions while preserving each release’s dependency landscape. The sanity check before cross-version validation ensures that every version executes without setup errors and yields fewer than 0.25% failed test cases on its corresponding test suite. Cross-version validation. For a version upgrade Vi from vi−1 to vi , we identify upgrade-related test cases by running the original test suite Qvi on both the old and new gold codebases, cvi−1 and cvi , respectively. Following SWE-Bench [Jimenez et al., 2024], we include both the FAIL_TO_PASS and ERROR_TO_PASS cases as upgrade-related tests. Formally, let r(q, c) ∈ {pass, fail, error} denote the execution result of test case q on codebase c. We define the upgrade-related test set as Ui = {q ∈ Qvi | r(q, cvi−1 ) ∈ {fail, error} ∧ r(q, cvi ) = pass}. 2.2
(1)
DecompSynth
We synthesize the specification for each consecutive version pair primarily from two sources: (i) Release note R with corresponding fetched GitHub issues or pull requests, which is represented as a list of tasks [T (1) , . . . , T (n) ]; and (ii) Diff context D, consisting of source-code diffs as the primary grounding evidence and test-suite diffs for calibrating behavioral intent. It is organized as a list of file-level diffs [Df1 , . . . , Dfm ], where each file-level diff Dfℓ consists of a sequence of hunks hfℓ ,∗ . 3
Divide the diffs. To synthesize the specification, we first conduct batch-level multi-label hunk matching (see Algorithm 1). Specifically, the agent πθmap receives the task-structured release note R and a batch of file-level diffs Bj ⊆ D, and assigns each hunk to one or more task labels. The label space is defined as L = {1, . . . , n} ∪ {doc, others}, where labels 1, . . . , n correspond to tasks T (1) , . . . , T (n) , doc denotes documentation-only changes, and others denotes residual changes that cannot be aligned to any task. Let H(Bj ) denote the hunks in batch Bj . The batch-level matching result is Φj = πθmap (R, Bj ), Φj : H(Bj ) → 2L . (2) After processing all batches, we merge the batch-level mappings into a global assignment Φ and reconstruct the matched hunk set M(k) for task T (k) by M(k) = { h ∈ H(D) : k ∈ Φ(h) }. (3) Synthesize the specifications. Given the matched hunk set M(k) , the agent πθsyn then generates a specification S (k) that describes the upgrade requirement for task T (k) (see Algorithm 2): S (k) = πθsyn (T (k) , M(k) ). (4) For hunks assigned to others, the agent πθsyn determines whether they collectively imply an additional upgrade task that is not explicitly covered by the release note. To improve reliability and accommodate diverse levels of granularity, we adopt an oracle-like synthesis procedure inspired by Zhuo et al. [2025] in which the agent is requested to generate the specification containing (1) problem statement, (2) grounded/conceptual expectations and constraints, (3) behaviors, and (4) acceptance criteria (see Prompt 3 for details). We study their granularity effect in §4.3. Synthesis scale. In total, we use GPT-5.4 [OpenAI, 2026b] (Codex [OpenAI, 2025]) with xhigh reasoning effort as the agent to match the hunks and synthesize the specifications. For the divide stage, we split file-level diffs into batches of at most 20 files, 150 hunks, and 3,000 diff lines. Using Prompts 1, 2, 3, and 4 in Appendix F.1, we process 155 version transitions across 12 upgrade chains and synthesize 1,660 grounded upgrade tasks. See Appendix J.1 for examples of synthesis artifacts. Stability check. Since the divide stage is the most susceptible to stochastic variation in our synthesis pipeline, we assess the stability on the largest diff (Pytest ver. 8.2.2 → 8.3.0), which contains 13k lines, 102 files, and 986 hunks. Across 10 repeated hunk-matching runs, the assignments achieve 93.66% exact match, 94.54% Jaccard similarity, and 95.38% core agreement, suggesting that the agent remains stable even under the most challenging transition. See Appendix C for more details. 2.3
Version-Upgrade Protocol
Each chain C is an ordered list of version-upgrade steps [V1 , . . . , VN ]. Starting from the initial codebase cv0 at version v0 (ĉv0 = cv0 ), the agent πθ sequentially performs each upgrade step (k) i conditioned on the synthesized specification Si = {Si }K k=1 (see Algorithm 3): ĉvi = πθ (Si , ĉvi−1 ), i = 1, . . . , N. (5) Build+Fix regularization. To avoid unfairly penalizing agents for non-behavioral execution incompatibilities, such as import, collection, or other setup-related failures, we design an additional regularization step. This design is inspired by the multi-attempt setting (Pass Rate 2) used in Aider benchmark [Aider, 2024]. Specifically, we allow the agent to fix the codebase once to make it compatible with the execution setup when execution-level errors occur.
3
Experimental Setup
3.1
Environment and Configuration
Container environment and workspace. For each chain, we build a container from the startingversion environment and place the workspace under /app. We then install the agent CLI inside the container, where it runs as a non-root user in /app and persists throughout the entire chain, rather than rebuilding a fresh environment at each step. After each completed step, the codebase and specification are archived in a root-owned directory, keeping the active workspace clean and preventing the agent from being distracted by previous artifacts. 4
Anti-cheating controls. We leverage multiple anti-cheating controls to prevent shortcutting and information leakage. At the agent level, we explicitly instruct them not to access or copy the released source code and enforce a shared set of tool-use restrictions for all agents, such as disabling web search and package installation commands (see Appendix D). At the sandbox level, we blacklist some code-hosting and package-registry domains, such as githubusercontent.com and pypi.org. 3.2
Model and Agent Configuration
We evaluate eight frontier models from four providers: OpenAI (GPT-5.3-Codex [OpenAI, 2026a], GPT-5.4 [OpenAI, 2026b], and GPT-5.5 [OpenAI, 2026c]), Anthropic (Claude-Sonnet-4.6 [Anthropic, 2026c], Claude-Opus-4.6 [Anthropic, 2026a], and Claude-Opus-4.7 [Anthropic, 2026b]), MiniMax (MiniMax-M2.7-HS, the high-speed variant [MiniMax, 2026]), and Z.AI (GLM-5.1 [Z.AI, 2026]) on three agent CLIs: Codex [OpenAI, 2025], Claude Code [Anthropic, 2025], and OpenCode [OpenCode, 2025]. We use the xhigh reasoning effort for the two strongest models, Claude-Opus-4.7 and GPT-5.5, while all other models use high reasoning effort when available. We conduct all experiments between April 17 and May 5, 2026, using the following agent CLI versions: Claude Code (2.1.113 - 2.1.126), Codex CLI (0.121.0 - 0.128.0), and OpenCode (1.14.18 - 1.14.32). Agent invocation. Following Terminal-Bench [Merrill et al., 2026], we invoke all agents in headless mode through a unified harness, which is useful for benchmarking raw model capabilities [Orlanski et al., 2026]. For each step, we place the specification and the agent-produced codebase under /app, then start a new agent session to avoid distractions from history compactions and better reflect realistic long-horizon maintenance, where developers return to a project after time away rather than continuing from an unbounded interactive context [Orlanski et al., 2026]. To keep prompts concise, we use the prompts that specify only the paths to the documents and the source tree, requiring the agent to read the relevant information directly from the workspace rather than embedding it in the prompt (see Prompts 5 and 6 in Appendix F.2 and examples in Appendix J). 3.3
Evaluation
Evaluation configuration and protocol. We use the same evaluation protocol for cross-version validation in §2.1, hidden check, and post hoc evaluation, differing only in whether the codebase is extracted from the agent container or reconstructed by replaying code changes from the starting version. Both are conducted in an isolated container built from the target package version, preventing contamination or leakage from the agent workspace. To improve the robustness of pytest to the partial breakage typical of mid-upgrade code, we design collection and import plugins to ensure test coverage. Concretely, unavailable APIs imported by test files are stubbed and logged, and phase-level reports are continuously written to disk so that collection failures and abnormal session termination still produce usable reports. Additionally, we override the target version’s pytest configuration to remove incompatible options, preventing shortcuts via test configuration edits for evaluation. Evaluation metrics. For each upgrade step Vi , we evaluate all tests in Qvi on the agent’s previous and current codebases, ĉvi−1 and ĉvi . Using the pre-defined upgrade-related set Ui in Eq. 1, each test is classified by whether q ∈ Ui and by its status transition from the previous to the current codebase. Table 2 summarizes the resulting categories. We compute three metrics for each chain and report their averages across chains in §4.1: • Resolving: measures the agent’s ability to implement the required upgrade behavior; • Precision: measures maintenance safety by penalizing regressions on previously passing; and • F1-score: balances upgrade resolving and behavior preservation. Since upgrade steps vary in scope, we micro-average test outcomes across steps before computing each chain-level metric: Pn Pn TPi i=1 TPi P P Resolving = n , Precision = n i=1 , (TP + FN ) (TP i i i + FPi ) i=1 i=1 Pn 2 × i=1 TPi F1 = Pn . i=1 (2TPi + FPi + FNi ) 5
Table 2: Test classification at upgrade step Vi over all tests in Qvi . Upgrade-related tests follow Ui in Eq. 1. Here, fail includes assertion failures and execution errors, and “·” denotes a wildcard. Category
Upgrade-related
Results (ĉvi−1 , ĉvi )
Symbol
resolved unresolved
Yes Yes
(·, pass) (·, fail)
TPi FNi
preserved regressed
No No
(pass, pass) (pass, fail)
TNi FPi
recovered unrecovered
No No
(fail, pass) (fail, fail)
RECi RECi
Table 3: Average performance across 12 chains. OpenCode; Agent
,
Provider
,
, and
Model
,
, and
denote Claude Code, Codex, and
denote Anthropic, OpenAI, Z.AI, and MiniMax. Resolving
Precision
F1-score
Final Passing
Build
Build + Fix
Claude-Opus-4.7
58.3%
Claude-Opus-4.6
38.8%
Claude-Sonnet-4.6
34.4%
39.8%
53.4%
66.5%
45.9%
84.5%
GPT-5.5
52.2%
57.5%
72.0%
80.1%
64.8%
92.2%
GPT-5.4
45.3%
47.5%
59.2%
72.2%
54.9%
90.4%
GPT-5.3-Codex
47.1%
51.8%
70.5%
76.3%
59.2%
90.9%
GPT-5.4
39.3%
43.4%
54.7%
63.8%
46.5%
89.7%
GLM-5.1
27.5%
38.1%
31.8%
49.9%
40.1%
84.1%
MiniMax-M2.7-HS
14.5%
20.2%
25.8%
34.2%
21.2%
70.3%
39.7%
44.8%
54.0%
65.4%
50.2%
86.6%
Average
4
Results
4.1
Overall Performance on SWE-C HAIN
Build
Build + Fix
60.8%
67.4%
80.6%
68.5%
91.9%
44.3%
51.3%
64.9%
50.8%
85.8%
Table 3 presents the overall results across 9 agent-model configurations under two regimes. Across all agents and chains, the average Build+Fix resolving rate is only 44.8%, with 65.4% precision and 50.2% F1-score. The higher precision suggests that successful changes are relatively targeted and non-regressive, while the resolving rate indicates that agents still leave more than half of the upgrade-related test behaviors unresolved, highlighting the challenge of continuous version-upgrade maintenance for current agents. Nevertheless, Claude-Opus-4.7 (Claude Code) achieves the best performance, with 60.8% resolving, 80.6% precision, and 68.5% F1-score, followed by GPT-5.5 (Codex). We also report Final Passing as an auxiliary measure of final codebase health over the chain, which broadly follows the F1-score trend. The substantially stronger performance of frontier agents shows that SWE-C HAIN is a feasible and discriminative benchmark. We also observe three clear patterns: (1) frontier closed-source models outperform the other models in their respective families, and closed-source models generally outperform GLM-5.1 and MiniMaxM2.7-HS; (2) GPT-5.3-Codex slightly outperforms GPT-5.4 under the Codex CLI, suggesting that Codex-native agentic variants can remain competitive with newer general-purpose models in continuous maintenance; and (3) GPT-5.4 performs better with Codex than OpenCode, suggesting better alignment between the native agent CLI and its underlying model. These patterns suggest that SWE-C HAIN can distinguish differences in model capability, coding specialization, and agent CLIs. Across agents, Build+Fix improves average precision from 54.0% to 65.4%, while average resolving increases more modestly from 39.7% to 44.8%, indicating that the additional fixing mainly addresses brittle failures and regressions rather than directly solving upgrade requirements. Since reasonable 6
1.0
Claude-Opus-4.7 (ClaudeCode)
76.9%
60.5%
50.3%
64.0%
66.6%
81.1%
56.0%
66.1%
27.5%
47.1%
62.7%
70.9%
60.8%
Claude-Opus-4.6 (ClaudeCode)
71.7%
69.2%
28.5%
62.2%
41.5%
61.7%
26.8%
41.7%
22.8%
41.8%
38.0%
25.4%
44.3%
Claude-Sonnet-4.6 (ClaudeCode)
70.5%
67.0%
21.2%
60.7%
35.0%
64.0%
30.2%
14.9%
22.8%
41.6%
20.9%
28.5%
39.8%
GPT-5.5 (Codex)
77.5%
81.1%
61.6%
62.0%
53.3%
77.0%
45.4%
53.4%
35.6%
50.1%
21.7%
71.6%
57.5%
GPT-5.4 (Codex)
71.1%
73.5%
62.3%
62.0%
45.5%
38.7%
39.1%
48.6%
28.3%
36.9%
19.5%
44.4%
47.5%
GPT-5.3-Codex (Codex)
74.6%
82.7%
64.2%
46.1%
45.5%
68.5%
46.1%
47.1%
24.7%
27.6%
21.4%
73.4%
51.8%
GPT-5.4 (OpenCode)
72.8%
32.4%
58.3%
60.4%
34.1%
50.0%
35.5%
42.2%
23.3%
46.1%
20.8%
45.3%
43.4%
GLM-5.1 (OpenCode)
59.5%
58.4%
53.6%
55.2%
30.7%
60.8%
13.0%
23.9%
16.7%
39.3%
20.5%
25.9%
38.1%
MiniMax-M2.7-HS (OpenCode)
42.2%
50.3%
7.3%
38.4%
17.6%
23.0%
10.4%
9.5%
8.1%
9.9%
10.5%
15.0%
20.2%
68.5%
63.9%
45.3%
56.8%
41.1%
58.3%
33.6%
38.6%
23.3%
37.8%
26.2%
44.5%
PyJWT (2.0.0)
Jinja2 (2.8)
Flask (2.0.0)
attrs (21.3.0)
pytest (7.0.0)
urllib3 (2.0.7)
poetry (1.5.0)
pytest (8.0.0)
conan (2.23.0)
conan (2.12.0)
0.8
0.6
0.4
0.2
Average
xarray xarray (2022.11.0) (2025.6.0)
44.8%
Average
0.0
Figure 2: Build+Fix resolving rate by agent and chain, order left-to-right by increasing difficulty.
implementations in black-box evaluation may still trigger broad failures due to setup incompatibilities (e.g., import errors or dependency issues), our Build+Fix regime mitigates this over-penalization by allowing a single correction based on error feedback, such as collection or import errors. However, this does not make the benchmark easy, as the average resolving rate remains at only 44.8%, and weaker models like MiniMax-M2.7-HS still fall far below the initial Build performance of frontier models. Notably, GPT-5.5 and GPT-5.3-Codex already achieve high Build precision and remain stable after Build+Fix, suggesting that they tend to produce safer first-pass patches, whereas ClaudeOpus-4.7 achieves a Build resolving rate higher than the Build+Fix resolving rate of all other agents, showing stronger first-pass upgrade coverage. See Appendix G for Build and Build+Fix trajectories. Key Finding: Overall Performance • SWE-Chain is challenging and highly discriminative: agents achieve only 44.8% resolving, 65.4% precision, and 50.2% F1-score on average under the Build+Fix regime. • Claude-Opus-4.7 achieves the best performance in SWE-C HAIN, followed by GPT-5.5. • Build+Fix is necessary to prevent over-penalization due to incompatible setups, improving precision and leaving capability gaps intact. 4.2
Cross-Chain Difficulty
Figure 2 illustrates the heatmap of Build+Fix resolving rate by chain and shows that performance varies sharply across chains. Even on easier chains (e.g., PyJWT and Jinja2), agents resolve only below 70% of upgrade-specific behaviors on average, while harder chains (e.g., conan and xarray) can fall below 30%. The codebase scale shown in Table 1 partially explains this trend, but it is insufficient. For example, poetry is smaller than pytest, but agents achieve lower resolving rates on poetry because its upgrades have a larger average code-diff scale. The same pattern appears between Flask and Jinja2, suggesting that chain difficulty depends jointly on package size, upgrade scale, and density. We provide the detailed difficulty ranking in Appendix B. Furthermore, no agent dominates across all chains: Claude-Opus-4.7 is stronger on chains like xarray (2022.11.0) and pytest, whereas GPT-based agents lead on chains such as Flask, Jinja2, and conan, indicating modelspecific strengths across package structures. Moreover, open-weight models remain competitive on easier chains but degrade sharply on harder chains, supporting that SWE-C HAIN is discriminative across different maintenance scenarios. We also provide the corresponding F1 and precision heatmaps in Appendix H, and the per-step resolving trajectories in Appendix I. Key Finding: Difficulty Is Chain-Specific SWE-C HAIN captures diverse difficulty across chains: the chain-level average resolving rate ranges from 23.3% to 68.5%, which discriminates agents across different upgrade scenarios.
7
Table 4: Effect of specification granularity on the pytest 8.0 chain under the Build+Fix regime.
4.3
Model (Agent)
Metric
L1
L2
L3
L4
L5
Claude-Opus-4.7
Resolving Precision F1
44.5% 8.9% 14.8%
62.6% 7.8% 13.9%
66.1% 88.1% 75.5%
66.7% 89.2% 76.3%
73.9% 93.5% 82.5%
GPT-5.5
Resolving Precision F1
47.4% 9.9% 16.3%
44.5% 9.5% 15.7%
53.4% 92.1% 67.6%
68.1% 93.3% 78.7%
67.8% 89.7% 77.3%
Effect of Specification Granularity
To study how the specification granularity affects agent performance, we construct five specification variants on the pytest (ver. 8.0) chain, with examples shown in Appendix J.2: • L1: Release notes with GitHub issues and PRs, simulating SWE-EVO setting [Thai et al., 2026]. • L2: Problem statements only. • L3 (default setting in §4.1): Problem statements with conceptual expectations and constraints. • L4: Problem statements with grounded (concrete API/behavior details) expectations and constraints. • L5: Problem statements, grounded expectations and constraints, behaviors, and acceptance criteria. As shown in Table 4, the largest gap appears from L2 to L3, where expectations and constraints improve precision much more than resolving rate for both agents, as they clarify the intended behavior, implementation boundary, and preservation constraints, making the specification more informative and feasible for agents to conduct the version transition. While L1 uses raw changelog entries with linked GitHub issues and PRs, which contain the issues-driven requirements and useful evidence, it is noisy, unevenly structured, and may expose implementation details. Meanwhile, higher-granularity specifications provide more grounded information and generally improve performance, but they are closer to oracle-style guidance, making them better suited for assessing agents’ implementation capability than realistic upgrade workflows. However, they remain useful beyond upper-bound probes, especially for reliability evaluations where the objective is to sustain agent progress over many steps without excessive early collapse. Key Finding: Granularity Control Raw issue/PR artifacts are poor specifications for release-level upgrades, while expectations and constraints are essential to avoid regressions. Our default L3 setting balances guidance and leakage control without relying on the more oracle-like information in L4 and L5. 4.4
Efficiency and Resource Usage
Table 5 reports the efficiency, cost, token usage, and tool-call activity of each agent averaged per chain. The strongest configurations with xhigh reasoning effort are also among the most expensive: Claude-Opus-4.7 and GPT-5.5 lead in performance but cost $150.39 and $131.34 per chain on average. However, higher resource usage does not inevitably imply better results. For example, Claude-Sonnet-4.6 has the longest runtime and high token usage, and MiniMax-M2.7-HS uses more tokens and tool calls than OpenCode GPT-5.4, but both perform substantially worse than top models. Furthermore, tool-call success rates are consistently high across models, the performance gaps are unlikely to be driven by tool failures. Overall, SWE-C HAIN measures maintenance capability rather than merely interaction volume or execution reliability.
5
Related Works
AI4SE benchmarks. The evaluation of coding agents has advanced rapidly through a series of increasingly challenging benchmarks. EvalPlus [Liu et al., 2023], BigCodeBench [Zhuo et al., 2025], 8
Table 5: Average efficiency, cost, token usage and tool-use statistics per chain. Runtime measures the agent execution time for the chain (excluding hidden-test execution). Maximum values are in red bold; minimum values are in blue underlined. Agent
Provider
Model
Runtime (h)
Cost
Build
Build+Fix
Claude-Opus-4.7
3.13
3.43
$150.39
Claude-Opus-4.6
2.68
3.25
$71.78
Claude-Sonnet-4.6
4.13
4.84
GPT-5.5
3.23
GPT-5.4
2.12
GPT-5.3-Codex
2.58
GPT-5.4
2.17
GLM-5.1 MiniMax-M2.7-HS
Tools
Tokens Calls
Success
350.7M
1, 748
98.3%
179.4M
2, 008
98.4%
$58.68
203.9M
1, 866
98.6%
3.48
$131.34
184.7M
1, 736
95.6%
2.28
$39.84
100.9M
1, 165
94.6%
2.77
$41.59
155.2M
1, 318
93.8%
2.44
$22.98
61.1M
1, 098
95.4%
4.09
4.44
$20.09
67.5M
1, 057
97.4%
2.19
2.68
$9.51
115.0M
1, 432
95.4%
and LiveCodeBench [Jain et al., 2025] evaluate functional correctness, library-level API usage, and contamination-free evaluation across diverse programming tasks. SWE-Bench [Jimenez et al., 2024] pioneers agent evaluation on real-world GitHub issues, requiring models to localize faults and implement fixes across open-source repositories. Its variants [OpenAI, 2024, Yang et al., 2024b, Deng et al., 2025] further improve evaluation reliability, extend to visual artifacts, and increase task difficulty. τ -Bench [Yao et al., 2025] evaluates agents on tool-use and decision-making in multi-turn interactive environments, where agents must complete user-specified goals by invoking APIs and reasoning over structured databases under realistic constraints. Terminal-Bench [Merrill et al., 2026] pushes further into agentic territory by assessing agents on long-horizon command-line tasks inside persistent shell sessions, exposing limitations in environment state tracking, tool chaining, and error recovery across interaction trajectories. Long-horizon maintenance benchmarks. Recent benchmarks evaluate coding agents beyond isolated tasks. NL2RepoBench [Ding et al., 2025], MaintainCoder [Wang et al., 2025b], VibeCodeBench [Tran et al., 2026], and CodeFlowBench [Wang et al., 2026] study long-horizon generation, maintainability under changing requirements, and multi-step coding workflows, but they primarily focus on generation-oriented or requirement-driven development rather than continuous evolution. SWE-EVO [Thai et al., 2026] approaches repository evolution by evaluating agents on real software upgrades, but its tasks are still end-to-end, and their specifications remain coarse-grained when derived directly from GitHub issues and pull requests. SlopCodeBench [Orlanski et al., 2026] studies degradation when agents iteratively extend their own generated code, which is closer to repository construction like NL2RepoBench than to maintenance of mature human-developed codebases. SWE-CI [Chen et al., 2026] evaluates maintenance using CI traces, and EvoClaw [Deng et al., 2026] reconstructs milestone-level evolution from commit histories. While these works model important aspects of software engineering lifecycles, their task boundaries do not align with official package releases, where maintainers bundle, document, test, and ship version-level changes constantly. Unlike prior benchmarks, SWE-C HAIN segments evolution at maintainer-defined release nodes and uses fine-grained synthetic specifications to address the coarse granularity of release-level upgrades.
6
Conclusion
We introduce SWE-C HAIN, a benchmark for chained release-level package upgrades, comprising 12 upgrade chains across diverse domains, scales, and difficulty levels to evaluate coding agents’ codebase maintenance capabilities. We design DecompSynth that aligns maintainer release notes with gold code diffs to produce grounded, reliable, informative, and feasible upgrade specifications at scale. Across 9 frontier configurations, agents achieve only 44.8% resolving and 50.2% F1 on average under Build+Fix. These results show that current agents still struggle to carry their own changes forward across chained releases without breaking existing functionality, highlighting the need for agents that can maintain coherent, reliable, and evolvable codebases across long chains.
9
References Aider. Aider’s polyglot benchmark, 2024. URL https://aider.chat/2024/12/21/polyglot. html#the-polyglot-benchmark. Anthropic. Claude code overview, 2025. URL https://code.claude.com/docs/en/overview. Anthropic. Introducing claude opus 4.6, 2026a. URL https://www.anthropic.com/news/ claude-opus-4-6. Anthropic. Introducing claude opus 4.7, 2026b. URL https://www.anthropic.com/news/ claude-opus-4-7. Anthropic. Introducing claude sonnet 4.6, 2026c. URL https://www.anthropic.com/news/ claude-sonnet-4-6. Jialong Chen, Xander Xu, Hu Wei, Chuan Chen, and Bing Zhao. Swe-ci: Evaluating agent capabilities in maintaining codebases via continuous integration. arXiv preprint arXiv:2603.03823, 2026. Gangda Deng, Zhaoling Chen, Zhongming Yu, Haoyang Fan, Yuhong Liu, Yuxin Yang, Dhruv Parikh, Rajgopal Kannan, Le Cong, Mengdi Wang, Qian Zhang, Viktor Prasanna, Xiangru Tang, and Xingyao Wang. Evoclaw: Evaluating ai agents on continuous software evolution. arXiv preprint arXiv:2603.13428, 2026. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025. Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Qizhi Chen, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Weihao Xie, Zihan Wang, Yishuo Yuan, Jiayu Zhang, Enduo Zhao, Yunfei Zhao, He Zhu, Liya Zhu, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyang Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, Wenhao Huang, and Ge Zhang. Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents. arXiv preprint arXiv:2512.12730, 2025. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: a benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. Man Ho Lam, Chaozheng Wang, Jen tse Huang, and Michael Lyu. Codecrash: Exposing LLM fragility to misleading natural language in code reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. Jia Li, Ge Li, Xuanming Zhang, Yunfei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. Evocodebench: An evolving code generation benchmark with domain-specific evaluations. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. 10
Keyu Li, Junhao Shi, Yang Xiao, Mohan Jiang, Jie Sun, Yunze Wu, Dayuan Fu, Shijie Xia, Xiaojie Cai, Tianze Xu, Weiye Si, Wenjie Li, Dequan Wang, and Pengfei Liu. Agencybench: Benchmarking the frontiers of autonomous agents in 1m-token real-world contexts. arXiv preprint arXiv:2601.11044, 2026. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. In International Conference on Learning Representations (ICLR), 2024. Mike A Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Kumar Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Kwesi Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Jenia Jitsev, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In The Fourteenth International Conference on Learning Representations, 2026. MiniMax. Minimax m2.7: Early echoes of self-evolution, 2026. URL https://www.minimax.io/ news/minimax-m27-en. OpenAI. Introducing SWE-bench verified, 2024. introducing-swe-bench-verified/.
URL https://openai.com/index/
OpenAI. Introducing codex, 2025. URL https://openai.com/index/introducing-codex/. OpenAI. Introducing gpt-5.3-codex, introducing-gpt-5-3-codex/.
2026a.
URL https://openai.com/index/
OpenAI. Introducing gpt-5.4, 2026b. URL https://openai.com/index/introducing-gpt-5-4/. OpenAI. Introducing gpt-5.5, 2026c. URL https://openai.com/index/introducing-gpt-5-5/. OpenCode. Opencode: The open source ai coding agent, 2025. URL https://opencode.ai. Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Frederic Sala, and Aws Albarghouthi. Slopcodebench: Benchmarking how coding agents degrade over long-horizon iterative tasks. arXiv preprint arXiv:2603.24755, 2026. Minh V. T. Thai, Tue Le, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. Sweevo: Benchmarking coding agents in long-horizon software evolution scenarios. arXiv preprint arXiv:2512.18470, 2026. Hung Tran, Langston Nashold, Rayan Krishnan, Antoine Bigeard, and Alex Gu. Vibe code bench: Evaluating ai models on end-to-end web application development. arXiv preprint arXiv:2603.04601, 2026. Yuxuan Wan, Yi Dong, Jingyu Xiao, Yintong Huo, Wenxuan Wang, and Michael R Lyu. Mrweb: An exploration of generating multi-page resource-aware web code from ui designs. arXiv preprint arXiv:2412.15310, 2024. 11
Sizhe Wang, Zhengren Wang, Dongsheng Ma, Yongan Yu, Rui Ling, Zhiyu Li, Feiyu Xiong, and Wentao Zhang. Codeflowbench: A multi-turn, iterative benchmark for complex code generation. arXiv preprint arXiv:2504.21751, 2026. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for AI software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2025a. Zhengren Wang, Rui Ling, Chufan Wang, Yongan Yu, Sizhe Wang, Zhiyu Li, Feiyu Xiong, and Wentao Zhang. Maintaincoder: Maintainable code generation under dynamic requirements. arXiv preprint arXiv:2503.24260, 2025b. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying LLM-based software engineering agents. Proceedings of the ACM on Software Engineering, 2(FSE), 2025. doi: 10.1145/3715754. Jingyu Xiao, Man Ho Lam, Ming Wang, Yuxuan Wan, Junliang Liu, Yintong Huo, and Michael R Lyu. Designbench: A comprehensive benchmark for mllm-based front-end code generation. arXiv preprint arXiv:2506.06251, 2025a. Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zixin Wang, Xinyi Xu, Wenxuan Wang, Zhiyao Xu, Yuhang Wang, and Michael R Lyu. Interaction2code: Benchmarking mllm-based interactive webpage code generation from interactive prototyping. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 241–253. IEEE, 2025b. Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. SWE-fixer: Training open-source LLMs for effective and efficient GitHub issue resolution. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1123–1139, 2025. doi: 10.18653/v1/2025.findings-acl.62. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.03859, 2024b. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. {$\tau$}-bench: A benchmark for \underline{T}ool-\underline{A}gent-\underline{U}ser interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, 2025. Z.AI. Glm-5.1, 2026. URL https://docs.z.ai/guides/llm/glm-5.1. Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, 2025.
12
Contents A Notation
14
B Benchmark Scale and Difficulty Ranking
15
C Stability of the Divide Step
16
D Forbidden Tools
17
E Pseudocode for SWE-C HAIN
18
F Prompts
20
F.1
Prompts for DecompSynth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
F.2
Prompts for Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
G Build+Fix Trajectories
25
H Cross-Chain Precision and F1
26
I
Per-Step Resolving Trajectories
26
J
Examples
28
J.1
DecompSynth Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
28
J.2
Specification Granularity Examples . . . . . . . . . . . . . . . . . . . . . . . . .
33
K Limitations
37
L Broader Impacts
37
M Declaration of LLM usage
37
13
A
Notation Table 6: Notation used in SWE-C HAIN. Symbol
Meaning Version and Codebase Notation
i vi Vi cvi ĉvi ĉv0 C Qvi q r(q, c)
Index of a version-upgrade step in an upgrade chain. Version identifier at step i. Version-upgrade step from version vi−1 to vi . Gold codebase of version vi . Agent-produced codebase after upgrading to version vi . Initial agent codebase, initialized as the grounded codebase cv0 . An upgrade chain consisting of sequential version-upgrade steps [V1 , . . . , VN ]. Test suite of version vi . A test case in a version test suite. Execution result of test case q on codebase c, taking values in {pass, fail, error}. Release Note and Diff Notation
R T (k) D Dfℓ hfℓ ,∗ H(D)
Release note or changelog represented as a list of tasks [T (1) , . . . , T (n) ]. The k-th task extracted from the release note. Diff context between two consecutive versions, consisting of code and test diffs. File-level diff for file fℓ . Sequence of hunks contained in file-level diff Dfℓ . Set of all hunks in the diff context D. Specification Synthesis Notation
Bj H(Bj ) L Φj Φ M(k) S (k) πθmap πθsyn
A batch of file-level diffs used for batch-level hunk matching. Set of hunks contained in diff batch Bj . Label space for hunk matching, defined as {1, . . . , n} ∪ {doc, others}. Batch-level mapping from hunks in H(Bj ) to one or more labels in L. Global hunk-to-label assignment obtained by merging all batch-level mappings. Matched hunk set for release-note task T (k) . Synthesized specification for task T (k) . Agent used for batch-level multi-label hunk matching. Agent used for specification synthesis. Version-Upgrade Evaluation Notation
Ui Ki (k) Si Si πθ
Upgrade-related test set for version-upgrade step Vi . Number of synthesized task specifications for version-upgrade step Vi . Synthesized specification for the k-th task at step Vi . (k) i Set of synthesized specifications for step Vi , defined as {Si }K k=1 . Evaluated coding agent that performs sequential version upgrades. Evaluation Outcome Categories
TPi FNi TNi FPi RECi RECi
Resolved upgrade-related tests at step Vi (passing on ĉvi ). Unresolved upgrade-related tests at step Vi (failing on ĉvi ). Preserved non-upgrade tests at step Vi (passing on both ĉvi−1 and ĉvi ). Regressed non-upgrade tests at step Vi (passing on ĉvi−1 , failing on ĉvi ). Recovered non-upgrade tests at step Vi (failing on ĉvi−1 , passing on ĉvi ). Unrecovered non-upgrade tests at step Vi (failing on both ĉvi−1 and ĉvi ).
14
B
Benchmark Scale and Difficulty Ranking
Table 7: Source-tree size of the starting and ending package versions for each chain. The “no tests” columns drop the package test folder, with sub-columns “Python” (.py only) and “All” (all file types); the rightmost LOCstart and LOCend count every file with no exclusions. Package
attrs
Repo
License
python-attrs/attrs
MIT
tests LOCno start
Chain
tests LOCno end
LOCstart
LOCend
Python
All
Python
All
21.3.0 → 26.1.0
5.8k
13k
7.4k
23k
25k
38k
2.12.0 → 2.20.1
52k
67k
57k
73k
172k
188k 203k
conan
conan-io/conan
MIT
2.23.0 → 2.28.1
59k
76k
62k
78k
193k
Flask
pallets/flask
BSD-3-Clause
2.0.0 → 2.3.3
9.0k
29k
9.6k
29k
36k
37k
Jinja2
pallets/jinja2
BSD-3-Clause
2.8 → 2.10.3
13k
23k
15k
24k
27k
30k
poetry
python-poetry/poetry
MIT
1.5.0 → 1.8.5
21k
21k
22k
23k
81k
87k
PyJWT
jpadilla/pyjwt
MIT
2.0.0 → 2.12.1
1.6k
3.4k
3.1k
5.9k
5.9k
11k
pytest
pytest-dev/pytest
MIT
7.0.0 → 7.4.4
31k
79k
34k
86k
128k
140k
8.0.0 → 8.3.5
35k
89k
36k
93k
145k
152k
2.0.7 → 2.6.3
13k
17k
13k
21k
32k
40k
2022.11.0 → 2023.7.0
76k
119k
81k
131k
179k
192k
2025.6.0 → 2026.2.0
104k
152k
108k
159k
230k
244k
urllib3 xarray
urllib3/urllib3
MIT
pydata/xarray
Apache-2.0
Table 7 reports the codebase size of every chain at its starting and ending package versions, with separate counts that drop the package test folder, which is removed in evaluation. Difficulty ranking. We rank the 12 upgrade chains using two scale-based factors: (1) the Pythonno tests only starting codebase size excluding tests, measured by LOCpy, ; and (2) the average codestart change burden per upgrade step, measured by LOCD/#Upgrades. We min–max normalize both factors across the 12 chains and average them: LOCD 1 no tests Difficulty = norm LOCpy, + norm . start 2 #Upgrades Table 8 shows the resulting difficulty ranking, where larger Python codebases and denser per-upgrade code changes generally lead to higher difficulty. Table 8: Difficulty ranking in SWE-C HAIN, which combines normalized starting codebase size and LOCD/#Upgrades. Rank 1 is the hardest. no tests LOCpy, start
LOCD/#Upgrades
Difficulty
2025.6.0 → 2026.2.0
104k
1283.6
0.989
2022.11.0 → 2023.7.0
76k
1308.2
0.866
conan
2.12.0 → 2.20.1
52k
868.8
0.544
4
conan
2.23.0 → 2.28.1
59k
662.2
0.484
5
pytest
8.0.0 → 8.3.5
35k
860.7
0.456
6
poetry
1.5.0 → 1.8.5
21k
897.7
0.404
7
urllib3
2.0.7 → 2.6.3
13k
678.4
0.263
8
pytest
7.0.0 → 7.4.4
31k
451.6
0.246
9
attrs
21.3.0 → 26.1.0
5.8k
602.6
0.195
10
Flask
2.0.0 → 2.3.3
9.0k
471.4
0.150
11
Jinja2
2.8 → 2.10.3
13k
394.2
0.135
12
PyJWT
2.0.0 → 2.12.1
1.6k
224.6
0.000
Rank
Package
Chain
1
xarray
2
xarray
3
15
C
Stability of the Divide Step
As mentioned in §2.2, the divide step is the most susceptible to stochastic variation in DecompSynth. We run the matching agent πθmap 10 times on the largest version transition, pytest 8.2.2 → 8.3.0, which contains 13k diff lines, 102 files, and 986 hunks, to model the most stressful situation. By Eq. 2, each run r produces a hunk-to-label mapping Φ(r) : H → 2L , where H is the set of hunks and Φ(r) (h) is the set of task labels assigned to hunk h in run r. We compare every pair of runs (r, s) over the common hunk set H using three hunk-level similarity metrics. Exact match.
Exact match requires the assigned label sets to be identical: i 1 X h (r) Exact(r, s) = 1 Φ (h) = Φ(s) (h) . |H| h∈H
Jaccard similarity.
Jaccard similarity measures partial overlap between assigned label sets: Jaccard(r, s) =
1 X |Φ(r) (h) ∩ Φ(s) (h)| , |H| |Φ(r) (h) ∪ Φ(s) (h)| h∈H
where two empty label sets are represented by a similarity of 1. Core agreement. Since providing additional hunks is less harmful than missing core task-relevant hunks, we define the core agreement as a superset-tolerant metric beyond Jaccard similarity. Core agreement counts two runs as agreeing on a hunk if their assigned label sets overlap, or if both are empty. i 1 X h (r) Core(r, s) = 1 Φ (h) ∩ Φ(s) (h) ̸= ∅ ∨ Φ(r) (h) = Φ(s) (h) = ∅ . |H| h∈H
We report the average over all 10 2 = 45 run pairs. The results are 93.66% exact match, 94.54% Jaccard similarity, and 95.38% core agreement, showing that the divide step remains stable even in the most challenging situation.
16
D
Forbidden Tools
As described in §3.1, we restrict the tools and commands available to agents during task execution in Artifact 1 to prevent external dependency installation and remote repository access. Agents must complete each upgrade using only the files, packages, and tools already available in the local environment. Forbidden Tools List Web access and remote fetching: - WebFetch - WebSearch - curl / wget Git remote operations: - git fetch - git pull - git clone - git push - git ls-remote - git remote - git submodule - git archive –remote Remote shell and file transfer: - ssh - scp - sftp - rsync - nc / netcat - telnet - openssl s_client Package installation and dependency downloading: - pip install / pip download / pip wheel - python -m pip install / download / wheel - python3 -m pip install / download / wheel - uv pip install / download / compile - poetry add / poetry install - npm / yarn / pnpm - apt / apt-get - conda install / mamba install / micromamba install
Artifact 1: Forbidden tools and commands during agent execution.
17
E
Pseudocode for SWE-C HAIN
This appendix gives the pseudocode for the three procedures of SWE-C HAIN: the divide step (Algorithm 1) and conquer step (Algorithm 2) of DecompSynth in §2.2, and the version-upgrade evaluation protocol (Algorithm 3) in §2.3. Algorithm 1 DecompSynth (Divide): Hunk-to-task matching. Require: Release note R; file-level diffs D; matching agent πθmap ; retry budget q Ensure: Per-task hunk lists, plus doc and others buckets 1: D ← SortByLineCount(D) ▷ descending 2: {Bj }J ▷ all limits must hold j=1 ← GreedyPack(D; 20 files, 150 hunks, 3000 diff lines) 3: Φ ← ∅ 4: for each batch Bj do ▷ batch-level hunk matching 5: Hj ← ExtractHunks(Bj ) 6: for t = 1 to q do ▷ default: q = 3 7: Φ̂j ← πθmap (R, Bj ) 8: (Hmiss , Hspur ) ← CheckCoverage(Φ̂j , Hj ) ▷ extract missing or invented hunks 9: if Hmiss = ∅ and Hspur = ∅ then 10: Φ ← Φ ∪ Φ̂j 11: break 12: end if 13: Bj ← AppendFeedback(Bj , Hmiss , Hspur ) 14: end for 15: end for 16: M ← GroupByLabel(Φ) ▷ tasks, doc, and others 17: return M
18
Algorithm 2 DecompSynth (Conquer): Per-task specification synthesis. Require: Release note R; matched hunk buckets M; synthesis agent πθsyn ; retry budget q Ensure: Structured task specifications S 1: S ← ∅ 2: for each release-note task T (k) ∈ R in parallel do ▷ task-level synthesis 3: M(k) ← GetMatchedHunks(M, k) 4: I (k) ← (T (k) , M(k) ) 5: for t = 1 to q do ▷ default: q = 3 6: Ŝ (k) ← πθsyn (I (k) ) 7: if ValidateSchema(Ŝ (k) ) then 8: S ← S ∪ {Ŝ (k) } 9: break 10: end if 11: I (k) ← AppendSchemaFeedback(I (k) ) 12: end for 13: end for 14: if M(others) ̸= ∅ then ▷ handle others labeled hunks 15: I (new) ← (S, M(others) ) 16: for t = 1 to q do 17: Ŝ (new) ← πθsyn (I (new) ) 18: if ValidateSchema(Ŝ (new) ) then 19: S ← S ∪ Ŝ (new) 20: break 21: end if 22: I (new) ← AppendSchemaFeedback(I (new) ) 23: end for 24: end if 25: return S
Algorithm 3 Version-upgrade protocol with Build+Fix regularization. Require: Chain v0 → v1 → · · · → vN ; starting codebase cv0 ; specifications {Si }N i=1 ; agent πθ Ensure: Agent-produced codebases {ĉvi } for all reached steps 1: ĉv0 ← cv0 2: for i = 1 to N do 3: ĉvi ← πθ (Si , ĉvi−1 ) ▷ Build phase 4: if BuildFailed(ĉvi ) then 5: break ▷ terminate the entire chain 6: end if 7: Ei ← RunHiddenTests(ĉvi , vi ) 8: if HasExecutionErrors(Ei ) then ▷ Build+Fix: one controlled repair 9: ρi ← WriteErrorReport(Ei ) ▷ unique execution-error blocks 10: ĉvi ← πθ (Si , ρi , ĉvi ) ▷ Fix phase 11: Ei ← RunHiddenTests(ĉvi , vi ) 12: end if 13: end for 14: return {ĉvi }
19
F
Prompts
F.1
Prompts for DecompSynth
This section shows the prompts used for DecompSynth in §2.2 where blue-highlighted text denotes the input parameters: • Prompt 1 requests the agent to assign each diff hunk in a batch to one or more release-note tasks, doc, or others. • Prompt 2 requests the agent to synthesize an upgrade requirement for the target task using its matched code diffs, with the matched test diffs as supplementary reference only. Prompt 3 further requires the agent to output it in structured JSON format that includes four core parts: (1) requirements, (2) conceptual and grounded constraints and exceptions, (3) behaviors, and (4) acceptance criteria. • Prompt 4 requests the agent to review hunks assigned to others and synthesize additional tasks when they indicate meaningful behavioral changes not covered by the release note. Diff Hunk-to-Task Matching Prompt The current directory contains: - {release_note_file} (Release Note — each task has a task_id and GitHub context) - {code_diff_file} (Code Diff) - output.json (pre-filled with all task_id keys as empty lists, plus "doc" and "others") Analyze the library update between two versions of a codebase, based on the Release Note and the CodeDiff. Map each diff hunk to the task_id(s) it supports. Write the result to output.json. Rules: - A hunk is identified by its @@ header up to and including the closing @@. Do NOT include trailing context (e.g., "@@ -185,6 +185,15 @@" not "@@ -185,6 +185,15 @@ class Foo:"). - A hunk can match multiple tasks (multi-label). - "doc": purely documentation/comments/type-only changes with no runtime effect. - "others": does not match any task. - Every hunk must be assigned. Do NOT leave any unmatched. Output format — each task_id maps to a list of: {"header": "diff –git a/... b/...", "hunks": ["@@ -a,b +c,d @@", ...]}
Prompt 1: Prompt used to assign each code-diff hunk to one or more release-note tasks, doc, or others.
20
Task Specification Synthesis Prompt The current directory contains: - {release_note_file} (Release Note content for task {task_id}) - {code_diff_file} (Code Diff relevant to this task — your primary focus) - {test_diff_file} (Test Diff for reference only) - output.json (write your result here) Analyze the library update from {old_version} to {new_version} based on the Release Note and Code Diff. Synthesize a user-facing task requirement and write the result as JSON to output.json. ### Important guidelines - The Test Diff is provided only as supplementary context to help you understand expected behavior. Do NOT derive requirements from test changes. - The synthesized requirement MUST NOT include any test case modifications, test file changes, or testing instructions. - The requirement must be feasible to understand and implement solely from the release note and code diff. ### Objectives 1. Understand what changed between versions from a user and product perspective 2. Produce a detailed breakdown of meaningful changes as a single upgrading task and requirement 3. Capture both the user requirement (from release note) and behavioral semantics (from code diff) ### Requirement fields (grounded vs conceptual) 1. Grounded Specification: - Grounded in release note and code diff evidence - May include precise public API names, modules, functions, classes, error types, and edge-case triggers - DO NOT include exact code snippets, messages, or implementation details - MUST NOT include patches, step-by-step implementation instructions, or exact code edits 2. Conceptual Specification: - High-level user-facing, pure natural language specification - Focus on observable behavior contracts and acceptance criteria only - MUST NOT include localization cues (file paths, internal identifiers, diff-derived details) - DO NOT prescribe implementation strategies ### Field definitions - `problem_statement`: User-visible problem/feature. Checklist-style for fixes ("Ensure that ..."). No file paths or implementation steps. - `expectation.grounded`: May mention public API names and observable signals (return/raise/warn/state). No code snippets, patches, file paths, exact messages. -
`expectation.conceptual`:
High-level
behavior
description.
May
mention
public
API
names,
return/raise/warn types. No file paths, internal identifiers, patches. - `constraints.grounded`: Non-regression / safety boundaries in precise terms. No "must use X / place at Y". - `constraints.conceptual`: Same boundaries in high-level terms. - `behavior`: List precise behavioral changes and observable signals. Derived from code diff, not test diff. - `acceptance_criteria`: Checklist items as observable checks (Given/When/Then or "When..., it should..."). Must describe production behavior, not test assertions.
Prompt 2: Prompt used to synthesize a user-facing upgrade task specification from the release note and task-relevant code diff.
21
Task Specification Synthesis Prompt (cont.) ## Output Return ONLY valid JSON wrapped in a markdown code block (```json ... ```). Output Format: ```json { "task_id": "{task_id}", "title": "A short title of the task", "type": "select one from FEATURE | FIX | DEPENDENCY | BREAKING | TYPING | DOCUMENTATION | PERFORMANCE | REFACTOR | OTHER", "runtime_impact": true | false, "description": "Description of what changed and why without implementation details.", "synthesized_requirement": { "problem_statement": "...", "expectation": { "grounded": "...", "conceptual": "..." }, "constraints": { "grounded": "...", "conceptual": "..." }, "behavior": [ "Precise behavioral changes: ’When X, function Y raises/returns/emits Z’", "Observable signals: ’Add a module-level variable that indicates X’", "..." ], "acceptance_criteria": ["..."] }, "significance": "high | medium | low", "confidence": "high | medium | low", "difficulty": "hard | medium | easy" }
Prompt 3: Continuation of the task specification synthesis prompt.
22
Task Completing Prompt The current directory contains: - synthesized_tasks.json (Already synthesized specifications from the release note) - {code_diff_file} (Unmatched code diff hunks not assigned to any task) - {test_diff_file} (Unmatched test diff hunks; supporting evidence only, may be empty) - new_tasks.json (Write your result here) Review the unmatched diff hunks from a library upgrade ({old_version} to {new_version}). Determine if any represent meaningful behavioral changes not already covered by synthesized_tasks.json. Use {code_diff_file} as the primary source of truth. Any new task you synthesize must be grounded in the code diff. Use {test_diff_file} only as supporting evidence to clarify or validate behavior suggested by the code diff. Do not create a new task based solely on tests. If yes, write new task specifications to new_tasks.json using the same schema as synthesized_tasks.json, with IDs starting from "new_task_1". If no, leave new_tasks.json as ‘‘. Highlight the type of change for each new task from: "FEATURE | FIX | DEPENDENCY | BREAKING | TYPING | DOCUMENTATION | PERFORMANCE | REFACTOR | OTHER" Ignore version bumps, formatting/linting, import reordering, and internal refactors with no observable behavior change.
Prompt 4: Prompt used for task-wise specification synthesis. Blue text indicates input arguments.
23
F.2
Prompts for Evaluation
This section shows the prompts used for the evaluation protocol in §3.2, where blue-highlighted text denotes the input parameters: • Prompt 5 is used in the Build stage, where the agent implements the target version upgrade from the synthesized specification. • Prompt 6 is used in the Fix stage, where the agent receives an execution error report and performs one controlled repair attempt. Upgrade Implementation (Build) Prompt Now implement the upgrade of {package_name} from version {old_version} to {new_version}. The specification is at ‘./{specs_filename}‘. The source code is under ‘./code/‘. The original test suite has been explicitly removed. Implement all required changes according to the specification.
Ensure consistency across all files,
especially when renaming or removing symbols and APIs. You may also update ancillary version-bookkeeping files (e.g., ‘__version__‘ and CHANGELOG) when it matters for the upgrade. Do NOT fetch, download, or request any external resource. Do NOT use the internet, WebSearch, WebFetch, remote APIs, package registries, or remote git hosts.
Prompt 5: Prompt used to instruct the agent to implement a package version upgrade from the old version to the target version. Interface Repair (Fix) Prompt We ran the hidden test suite against your {package_name} upgrade ({old_version} to {new_version}) and found interface mismatches. Read ‘./error_report.md‘ for the list of issues and reconcile your public API names and signatures in ‘./code/‘ so that downstream callers resolve correctly. The upgrade specification is still at ‘./{specs_filename}‘. Do NOT fetch, download, or request any external resource. Do NOT use the internet, WebSearch, WebFetch, remote APIs, package registries, or remote git hosts.
Prompt 6: Prompt used in the Build+Fix regularization stage, where the agent receives an error report after hidden-test execution.
24
G
Build+Fix Trajectories
To complement the effects of the Build+Fix regime in §4.1 and Table 3, we visualize the trajectory from Build to Build+Fix in the resolving-precision plane in Figure 3. Most agents show a larger upward than rightward shift, indicating that Build+Fix improves precision more than resolving rate. The figure also shows that open-weight models remain below closed-source models even after Build+Fix, suggesting that the repair step does not erase the underlying capability gap. 1.0
F1=0.9
0.8
F1=0.8
0.6
F1=0.6
0.4
F1=0.4
0.2
0.0
0.0
Claude Code Claude-Opus-4.7 Claude-Opus-4.6 Claude-Sonnet-4.6
Codex GPT-5.5 GPT-5.4 GPT-5.3-Codex
0.2
0.4
OpenCode GPT-5.4 GLM-5.1 MiniMax-M2.7-HS 0.6
Build Fix Build
0.8
F1=0.2
Fix
1.0
Figure 3: Effect of Build+Fix on resolving rate and precision. Hollow and filled markers denote Build and Build+Fix, respectively. Gray dashed curves indicate F1-score iso-curves.
25
H
Cross-Chain Precision and F1
To complement the heatmap Figure 2 in §4.2, we report the corresponding Build+Fix precision and F1 heatmaps in Figures 4 and 5, respectively. Overall, precision is substantially higher than resolving rate across most chains, which raises the resulting F1 scores. This suggests that agents are often relatively targeted when they implement upgrade-specific behaviors. However, chains such as urllib3 and conan 2.23.0 also show lower precision, which further lowers their F1 scores. Hence, the F1 heatmap provides a balanced view of both chain difficulty and maintenance capability, while the resolving-rate heatmap more directly reflects the requirement implementation ability. 1.0
Claude-Opus-4.7 (ClaudeCode)
97.1%
81.2%
92.7%
97.7%
84.0%
59.4%
79.5%
88.1%
41.1%
83.0%
80.5%
83.5%
80.6%
Claude-Opus-4.6 (ClaudeCode)
96.9%
92.1%
15.4%
96.8%
70.5%
38.4%
52.9%
76.7%
49.4%
57.3%
97.8%
34.8%
64.9%
Claude-Sonnet-4.6 (ClaudeCode)
97.6%
62.3%
12.3%
97.6%
61.1%
36.1%
80.6%
72.2%
45.1%
54.3%
86.7%
91.7%
66.5%
GPT-5.5 (Codex)
96.4%
95.5%
93.9%
99.1%
94.5%
71.5%
76.7%
92.1%
36.0%
82.0%
80.8%
43.1%
80.1%
GPT-5.4 (Codex)
94.6%
89.5%
86.2%
98.0%
83.1%
4.5%
85.7%
78.2%
45.3%
75.5%
56.2%
69.5%
72.2%
GPT-5.3-Codex (Codex)
97.7%
92.2%
94.2%
99.1%
86.0%
67.6%
69.0%
86.8%
43.4%
61.0%
78.6%
40.0%
76.3%
GPT-5.4 (OpenCode)
97.7%
89.6%
93.6%
96.9%
11.2%
5.4%
63.9%
69.0%
48.3%
69.9%
91.8%
28.3%
63.8%
GLM-5.1 (OpenCode)
99.0%
60.0%
89.0%
89.9%
3.3%
49.6%
12.6%
14.2%
15.5%
32.9%
62.6%
69.5%
49.9%
MiniMax-M2.7-HS (OpenCode)
91.2%
78.2%
3.5%
93.6%
1.3%
2.7%
56.6%
1.0%
31.9%
2.6%
29.0%
19.1%
34.2%
96.5%
82.3%
64.5%
96.5%
55.0%
37.2%
64.2%
64.3%
39.5%
57.6%
73.8%
53.3%
PyJWT (2.0.0)
Jinja2 (2.8)
Flask (2.0.0)
attrs (21.3.0)
pytest (7.0.0)
urllib3 (2.0.7)
poetry (1.5.0)
pytest (8.0.0)
conan (2.23.0)
conan (2.12.0)
0.8
0.6
0.4
0.2
Average
xarray xarray (2022.11.0) (2025.6.0)
65.4%
Average
0.0
Figure 4: Build+Fix precision by agent and chain, order left-to-right by increasing difficulty. 1.0
Claude-Opus-4.7 (ClaudeCode)
85.8%
69.3%
65.2%
77.3%
74.3%
68.6%
65.7%
75.5%
32.9%
60.1%
70.5%
76.7%
68.5%
Claude-Opus-4.6 (ClaudeCode)
82.4%
79.0%
20.0%
75.7%
52.2%
47.3%
35.6%
54.0%
31.2%
48.3%
54.7%
29.4%
50.8%
Claude-Sonnet-4.6 (ClaudeCode)
81.9%
64.6%
15.6%
74.8%
44.5%
46.2%
43.9%
24.8%
30.3%
47.1%
33.7%
43.5%
45.9%
GPT-5.5 (Codex)
85.9%
87.7%
74.4%
76.3%
68.1%
74.2%
57.1%
67.6%
35.8%
62.2%
34.2%
53.8%
64.8%
GPT-5.4 (Codex)
81.2%
80.7%
72.3%
76.0%
58.8%
8.0%
53.7%
59.9%
34.9%
49.6%
29.0%
54.2%
54.9%
GPT-5.3-Codex (Codex)
84.6%
87.2%
76.4%
62.9%
59.5%
68.0%
55.3%
61.1%
31.5%
38.0%
33.7%
51.8%
59.2%
GPT-5.4 (OpenCode)
83.4%
47.6%
71.8%
74.4%
16.9%
9.8%
45.7%
52.4%
31.5%
55.5%
33.8%
34.8%
46.5%
GLM-5.1 (OpenCode)
74.4%
59.2%
66.9%
68.4%
6.0%
54.7%
12.8%
17.8%
16.1%
35.9%
30.9%
37.7%
40.1%
MiniMax-M2.7-HS (OpenCode)
57.7%
61.2%
4.7%
54.4%
2.5%
4.9%
17.6%
1.7%
12.9%
4.2%
15.5%
16.8%
21.2%
79.7%
70.7%
51.9%
71.1%
42.5%
42.4%
43.0%
46.1%
28.5%
44.5%
37.3%
44.3%
PyJWT (2.0.0)
Jinja2 (2.8)
Flask (2.0.0)
attrs (21.3.0)
pytest (7.0.0)
urllib3 (2.0.7)
poetry (1.5.0)
pytest (8.0.0)
conan (2.23.0)
conan (2.12.0)
0.8
0.6
0.4
0.2
Average
xarray xarray (2022.11.0) (2025.6.0)
50.2%
Average
0.0
Figure 5: Build+Fix F1 by agent and chain, order left-to-right by increasing difficulty.
I
Per-Step Resolving Trajectories
To visualize how agents progress along each upgrade chain, we plot the cumulative resolving progress at each upgrade for all chains in Figure 6. For a chain of length n, the value at step x is defined as: Px TPi Resolving≤x = Pn i=1 . (TP i + FNi ) i=1 The figure shows that agents usually progress smoothly across patch-level upgrades, where the number of upgrade-specific tests and the scope of changes is smaller. In contrast, larger separations between agents often emerge around minor or major upgrades, where the upgrade scope becomes broader and missed requirements accumulate more visibly. 26
6(a) attrs (21.3.0)
6(b) conan (2.12.0)
6(c) conan (2.23.0)
6(d) Flask (2.0.0)
6(e) Jinja2 (2.8)
6(f) poetry (1.5.0)
6(g) PyJWT (2.0.0)
6(h) pytest (7.0.0)
6(i) pytest (8.0.0)
6(j) urllib (2.0.7)
6(k) xarray (2022.11.0)
6(l) xarray (2025.6.0)
Figure 6: Per-step Build+Fix resolving-progress trajectories across all upgrade chains. Each curve point corresponds to one version-upgrade step. Note that the x-axis ticks mark only minor or major version milestones and are spaced proportionally by semantic-version distance.
27
J
Examples
This appendix uses the pytest 8.2.0 → 8.2.1 upgrade as an example to illustrate the SWE-C HAIN pipeline in §2 and the resulting upgrade specification used in §4. J.1
DecompSynth Examples
As described in §2.1, we first collect the raw release note, shown in Example 1, and decompose it into task-structured entries with the corresponding GitHub issues or pull requests, shown in Example 2. Then, as described in §2.2, the agent takes the task-structured release note and code diffs, performs hunk-to-task matching, and produces the intermediate mapping document in Example 3. Next, the agent synthesizes a structured task specification for each matched release-note task, as shown in Example 4. Finally, we select the required fields and integrate all task-level specifications into the final upgrade specification for pytest ver. 8.2.0 → 8.2.1, shown in Example 5. Raw Release Note (.md) ## pytest 8.2.1 (2024-05-19)[¶](#pytest-8-2-1-2024-05-19) ### Improvements[¶](#improvements) - [#12334](https://github.com/pytest-dev/pytest/issues/12334): Support for Python 3.13 (beta1 at the time of writing). ### Bug Fixes[¶](#id254) - [#12120](https://github.com/pytest-dev/pytest/issues/12120): Fix ‘PermissionError‘ crashes arising from directories which are not selected on the command-line. - [#12191](https://github.com/pytest-dev/pytest/issues/12191): Keyboard interrupts and system exits are now properly handled during the test collection. - [#12300](https://github.com/pytest-dev/pytest/issues/12300): Fixed handling of ‘Function not implemented’ error under squashfuse_ll, which is a different way to say that the mountpoint is read-only. - [#12308](https://github.com/pytest-dev/pytest/issues/12308): Fix a regression in pytest 8.2.0 where the permissions of automatically-created ‘.pytest_cache‘ directories became ‘rwx——‘ instead of the expected ‘rwxr-xr-x‘. ### Trivial/Internal Changes[¶](#trivial-internal-changes) - [#12333](https://github.com/pytest-dev/pytest/issues/12333): pytest releases are now attested using the [Artifact Attestation](https://github.blog/2024-05-02-introducing-artifact-attestations-now-in-public-beta/) support from GitHub, allowing users to verify the provenance of pytest’s sdist and wheel artifacts.
Example 1: Maintainer release note for the pytest 8.2.1 upgrade.
28
Task-Structured Release Note (.json) [{ "task_id": "task_1", "content": "[#12334] Support for Python 3.13 (beta1).", "github": [{ "link": "https://github.com/pytest-dev/pytest/pull/12334", "type": "pr", "content": "### GitHub Pull Request #12334 Add Python 3.13 support\nFix #12323" }]},{ "task_id": "task_2", "content": "[#12120] Fix PermissionError crashes from unselected directories.", "github": [{ "link": "https://github.com/pytest-dev/pytest/issues/12120", "type": "issue", "content": "### GitHub Issue #12120 Pytest crashes if a subdirectory has no read access\nPytest crashes if there is a subdirectory it does not [omitted...]" }]},{ "task_id": "task_3", "content": "[#12191] KeyboardInterrupt and SystemExit are handled during collection.", "github": [{ "link": "https://github.com/pytest-dev/pytest/pull/12191", "type": "pr", "content": "### GitHub Pull Request #12191 Consider KeyboardInterrupt/SystemExit at collection time\n![Screenshot from 2024-04-06 21-48-46] [omitted...]" }]},{ "task_id": "task_4", "content": "[#12300] Treat squashfuse_ll ENOSYS as a read-only mountpoint case.", "github": [{ "link": "https://github.com/pytest-dev/pytest/issues/12300", "type": "issue", "content": "### GitHub Issue #12300 Crashing under a squashfuse_ll read-only mount\npytest is crashing with OSError: [Errno 38] Function not implemented [omitted...]" }]},{ "task_id": "task_5", "content":
"[#12308]
Restore
expected
permissions
for
automatically-created
.pytest_cache
directories.", "github": [{ "link": "https://github.com/pytest-dev/pytest/issues/12308", "type": "issue", "content": "### GitHub Issue #12308 EACCES: permission denied, scandir .pytest_cache\nWhen upgrading to Pytest 8.2.0, I was getting an error [omitted...]" }]},{ "task_id": "task_6", "content": "[#12333] Attest pytest release artifacts with GitHub Artifact Attestations.", "github": [{ "link": "https://github.com/pytest-dev/pytest/pull/12333", "type": "pr", "content": "### GitHub Pull Request #12333 Attest package provenance\nThis uses the new build provenance support added in build-and-inspect-python-package [omitted...]" }]}]
Example 2: Task-structured release-note entries with linked GitHub issue and pull-request context.
29
Intermediate Hunk Mapping Result (.json) { "task_1": [{ "diff_header": "diff –git a/src/_pytest/pytester.py b/src/_pytest/pytester.py", "old_path": "a/src/_pytest/pytester.py", "new_path": "b/src/_pytest/pytester.py", "hunks": ["@@ -289,7 +289,8 @@", "@@ -760,6 +761,9 @@"] }, [omitted...]], "task_2": [{ "diff_header": "diff –git a/src/_pytest/python.py b/src/_pytest/python.py", "old_path": "a/src/_pytest/python.py", "new_path": "b/src/_pytest/python.py", "hunks": ["@@ -176,7 +176,12 @@"] }], "task_3": [{ "diff_header": "diff –git a/src/_pytest/runner.py b/src/_pytest/runner.py", "old_path": "a/src/_pytest/runner.py", "new_path": "b/src/_pytest/runner.py", "hunks": ["@@ -388,7 +388,9 @@"] }], "task_4": [{ "diff_header": "diff –git a/src/_pytest/assertion/rewrite.py b/src/_pytest/assertion/rewrite.py", "old_path": "a/src/_pytest/assertion/rewrite.py", "new_path": "b/src/_pytest/assertion/rewrite.py", "hunks": ["@@ -1171,7 +1171,10 @@"] }], "task_5": [{ "diff_header": "diff –git a/src/_pytest/cacheprovider.py b/src/_pytest/cacheprovider.py", "old_path": "a/src/_pytest/cacheprovider.py", "new_path": "b/src/_pytest/cacheprovider.py", "hunks": ["@@ -213,6 +213,13 @@"] }], "task_6": [], "doc": [{ "diff_header": "diff –git a/src/_pytest/outcomes.py b/src/_pytest/outcomes.py", "old_path": "a/src/_pytest/outcomes.py", "new_path": "b/src/_pytest/outcomes.py", "hunks": ["@@ -114,6 +114,9 @@", "@@ -142,6 +145,9 @@", "@@ -163,6 +169,9 @@", "@@ -188,6 +197,9 @@", "@@ -227,6 +239,9 @@"] }, [omitted...]], "others": [{ "diff_header": "diff –git a/src/_pytest/config/__init__.py b/src/_pytest/config/__init__.py", "old_path": "a/src/_pytest/config/__init__.py", "new_path": "b/src/_pytest/config/__init__.py", "hunks": ["@@ -574,8 +574,8 @@"] }, [omitted...]] }
Example 3: Intermediate mapping from release-note tasks to matched code-diff hunks.
30
Synthesized Task Specification Output (.json) { "task_2": { "task_id": "task_2", "title": "Avoid collection crashes from unreadable unselected directories", "type": "FIX", "runtime_impact": true, "description": "The update fixes a runtime collection failure in pytest when an unreadable directory exists in the project tree but is not part of the command-line selection. Instead of aborting collection with a PermissionError while checking whether that directory is a Python package, pytest should ignore that directory and continue collecting the explicitly requested tests.", "synthesized_requirement": { "problem_statement": "Ensure that running pytest against selected test paths does not crash when unrelated directories in the project tree are unreadable.", "expectation": { "grounded": "During directory collection, pytest should treat a PermissionError raised while checking whether a directory contains a package initializer as a non-collectable directory and continue processing the command-line-selected test targets.
If the unreadable directory is not selected for
collection, pytest should not abort the session because of that directory.", "conceptual": "Pytest should remain resilient to unreadable directories that are outside the user’s requested test scope. Running selected tests should continue normally instead of failing because pytest inspected an unrelated directory it cannot read." }, "constraints": { "grounded":
"The fix is limited to PermissionError encountered while determining whether a
directory should be collected as a Python package. Directories that can be inspected and contain an initializer should still be collected as packages, and directories without that condition should remain uncollected as before.", "conceptual": "Only the unreadable-directory crash should be removed. Existing package discovery behavior for accessible directories should remain unchanged." }, "behavior": [ "When pytest inspects a directory during collection and reading its package initializer status raises PermissionError, pytest skips collecting that directory instead of propagating the exception.", "An unreadable directory that was not selected on the command line no longer prevents pytest from collecting and running the explicitly requested tests.", "Accessible directories that qualify as Python packages continue to be collected as package nodes." ], "acceptance_criteria": [ "When a project contains an unreadable sibling directory outside the selected test path, invoking pytest on the selected test path should not fail with PermissionError caused by that sibling directory.", "When pytest encounters PermissionError while determining whether a directory is a package, it should treat that directory as not collectable and continue collection.", "When a directory is accessible and contains a valid package initializer, pytest should still collect it as a package." ] }, "significance": "medium", "confidence": "high", "difficulty": "easy" } }
Example 4: Structured synthesized requirement for one matched upgrade task.
31
Upgrade Specification (.md) # Upgrade Specification from 8.2.0 to 8.2.1 ## New Features ### Task 1 Add Python 3.13 compatibility and stricter pytester.makefile validation Ensure that the library works correctly on Python 3.13 for traceback and pytester workflows that depend on frame-local state, and ensure that invalid ‘Pytester.makefile()‘ calls with ‘ext=None‘ are rejected explicitly. - Expectation: The library should support Python 3.13 without breaking traceback analysis or ‘pytester‘ helper workflows because of changes in how Python exposes local variables. Invalid file-creation requests that omit an extension value entirely should fail clearly instead of being processed. - Constraints:
The compatibility fix should preserve prior observable behavior outside the Python
3.13-specific locals change. Normal file creation through the helper should continue to work, and only clearly invalid missing-extension usage should be blocked. ## Bug Fixes ### Task 2 Avoid collection crashes from unreadable unselected directories Ensure that running pytest against selected test paths does not crash when unrelated directories in the project tree are unreadable. - Expectation: Pytest should remain resilient to unreadable directories that are outside the user’s requested test scope. Running selected tests should continue normally instead of failing because pytest inspected an unrelated directory it cannot read. - Constraints: Only the unreadable-directory crash should be removed. Existing package discovery behavior for accessible directories should remain unchanged. ### Task 3 Preserve system-exit behavior during pytest collection Ensure that pytest handles ‘KeyboardInterrupt‘ and ‘SystemExit‘ correctly when they occur during test collection, so the run stops immediately instead of treating them like normal collection errors. - Expectation: If test discovery is interrupted by a user interrupt or an explicit process exit, pytest should stop discovery immediately and surface that interrupt or exit as such, rather than reporting it as a standard collection failure. - Constraints: Only true system-level interruptions during discovery should get special handling. Normal collection results and ordinary collection errors should keep their existing behavior. ### Task 4 Handle read-only squashfuse_ll mounts without crashing Ensure that running the library on a read-only squashfuse_ll mount does not crash when cache directories cannot be created, and that this condition is treated the same way as other read-only filesystem failures. - Expectation: When the runtime is on a read-only filesystem that reports the failure using ’Function not implemented’ instead of a standard read-only error, the library should skip cache directory creation and continue operating instead of crashing. - Constraints: Treat only recognized read-only mount failures as recoverable, and do not hide unrelated filesystem errors. ### Task 5 Restore default .pytest_cache directory permissions Ensure that automatically created .pytest_cache directories no longer use owner-only permissions after upgrading to pytest 8.2.1, and instead remain accessible according to normal system defaults so repository-scanning tools do not fail with permission errors. - Expectation: Pytest-created cache directories should behave like ordinary project directories: accessible to other tools according to the environment’s normal permission policy, instead of becoming unexpectedly private to the current user. - Constraints: The change should remove the regression without making cache directory permissions broader than the system would normally allow, and without changing how users interact with pytest’s cache feature. ## Additional Changes - Ensure that users upgrading to pytest 8.2.1 can verify the provenance of official pytest distribution artifacts, including both source distributions and wheels, without changing pytest’s runtime behavior.
Example 5: Core specification used in the main experiments in §4.1.
32
J.2
Specification Granularity Examples
In §4.3, we study how specification granularity affects agent performance. We present the five specification variants used in the granularity study: • L1: Raw release notes with GitHub issues and pull requests, shown in Example 7. • L2: Problem statements only, shown in Example 6. • L3: Problem statements with conceptual expectations and constraints, shown in Example 5. • L4: Problem statements with grounded expectations and constraints, shown in Example 8. • L5: Problem statements, grounded expectations and constraints, behaviors, and acceptance criteria, shown in Example 9. L2: Problem-Only Specification (.md) # Upgrade Specification from 8.2.0 to 8.2.1 ## New Features ### Task 1 Add Python 3.13 compatibility and stricter pytester.makefile validation Ensure that the library works correctly on Python 3.13 for traceback and pytester workflows that depend on frame-local state, and ensure that invalid ‘Pytester.makefile()‘ calls with ‘ext=None‘ are rejected explicitly. ## Bug Fixes ### Task 2 Avoid collection crashes from unreadable unselected directories Ensure that running pytest against selected test paths does not crash when unrelated directories in the project tree are unreadable. ### Task 3 Preserve system-exit behavior during pytest collection Ensure that pytest handles ‘KeyboardInterrupt‘ and ‘SystemExit‘ correctly when they occur during test collection, so the run stops immediately instead of treating them like normal collection errors. ### Task 4 Handle read-only squashfuse_ll mounts without crashing Ensure that running the library on a read-only squashfuse_ll mount does not crash when cache directories cannot be created, and that this condition is treated the same way as other read-only filesystem failures. ### Task 5 Restore default .pytest_cache directory permissions Ensure that automatically created .pytest_cache directories no longer use owner-only permissions after upgrading to pytest 8.2.1, and instead remain accessible according to normal system defaults so repository-scanning tools do not fail with permission errors. ## Additional Changes - Ensure that users upgrading to pytest 8.2.1 can verify the provenance of official pytest distribution artifacts, including both source distributions and wheels, without changing pytest’s runtime behavior.
Example 6: L2 specification: problem statements only.
33
Specification L1: Specification with Raw Release Notes and GitHub Context (.md) 1.
[#12334](https://github.com/pytest-dev/pytest/issues/12334): Support for Python 3.13 (beta1 at the
time of writing). 2.
[#12120](https://github.com/pytest-dev/pytest/issues/12120): Fix ‘PermissionError‘ crashes arising
from directories which are not selected on the command-line. 3. [#12191](https://github.com/pytest-dev/pytest/issues/12191): Keyboard interrupts and system exits are now properly handled during the test collection. 4.
[#12300](https://github.com/pytest-dev/pytest/issues/12300):
Fixed
handling
of
‘Function
not
implemented’ error under squashfuse_ll, which is a different way to say that the mountpoint is read-only. 5. [#12308](https://github.com/pytest-dev/pytest/issues/12308): Fix a regression in pytest 8.2.0 where automatically-created ‘.pytest_cache‘ directories became ‘rwx——‘ instead of ‘rwxr-xr-x‘. 6. [#12333](https://github.com/pytest-dev/pytest/issues/12333): pytest releases are now attested using GitHub Artifact Attestations, allowing users to verify sdist and wheel provenance. ### GitHub Pull Request #12334 Add Python 3.13 support Fix #12323 ### GitHub Issue #12120 Pytest crashes if a subdirectory has no read access Pytest crashes if there is a subdirectory it does not [omitted...] ### GitHub Pull Request #12191 Consider KeyboardInterrupt/SystemExit at collection time  <!– Thanks for submitting a PR, your contribution is really appreciated! Here is a quick checklist that should be present in PRs. - [ ] Include documentation when adding new features. - [ ] Include new tests or update existing tests when applicable. - [X] Allow maintainers to push and squash when merging my commits. Please uncheck this if you prefer to squash the commits yourself. If this change fixes an issue, please: -
[
]
commits
Add (where
text
like
“closes
“XYZW“
is
the
#XYZW“ issue
to
the
PR
number).
description See
the
and/or [github
docs](https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue# linking-a-pull-request-to-an-issue-using-a-keyword) for more information. Unless your change is trivial or a small documentation fix (e.g., a typo or reword of a small section) please: - [ ] Create a new changelog file in the ‘changelog‘ folder, with a name like ‘<ISSUE NUMBER>.<TYPE>.rst‘. See
[changelog/README.rst](https://github.com/pytest-dev/pytest/blob/main/changelog/README.rst)
for
details. Write sentences in the **past or present tense**, examples: * *Improved verbose diff output with sequences.* * *Terminal summary statistics now use multiple colors.* Also make sure to end the sentence with a ‘.’. - [ ] Add yourself to ‘AUTHORS‘ in alphabetical order. –> ### GitHub Issue #12300 Crashing under a squashfuse_ll read-only mount pytest is crashing with OSError: [Errno 38] Function not implemented [omitted...] ### GitHub Issue #12308 EACCES: permission denied, scandir .pytest_cache When upgrading to Pytest 8.2.0, I was getting an error [omitted...] ### GitHub Pull Request #12333 Attest package provenance This uses the new build provenance support added in build-and-inspect-python-package [omitted...]
Example 7: L1 specification: release notes with linked GitHub issues and pull requests.
34
L4: Grounded Expectation Specification (.md) # Upgrade Specification from 8.2.0 to 8.2.1 ## New Features ### Task 1 Add Python 3.13 compatibility and stricter pytester.makefile validation Ensure that the library works correctly on Python 3.13 for traceback and pytester workflows that depend on frame-local state, and ensure that invalid ‘Pytester.makefile()‘ calls with ‘ext=None‘ are rejected explicitly. - Expectation: On Python 3.13, behavior that reads frame locals in traceback processing and in ‘pytester‘ hook-recording checks must continue to work even when frame locals are exposed through a proxy-style mapping rather than a plain dict. ‘Pytester.makefile()‘ must raise ‘TypeError‘ when ‘ext‘ is ‘None‘ instead of accepting that input. - Constraints: Existing behavior for supported Python versions must remain intact. Recursion detection in tracebacks must still treat repeated frames with equivalent local-variable state as recursive. Valid ‘Pytester.makefile()‘ calls with a real extension string must continue to follow the existing extension rules, with only ‘None‘ being newly rejected. ## Bug Fixes ### Task 2 Avoid collection crashes from unreadable unselected directories Ensure that running pytest against selected test paths does not crash when unrelated directories in the project tree are unreadable. - Expectation: During collection, pytest should treat a ‘PermissionError‘ while checking package-initializer status as a non-collectable directory and continue selected test collection. [omitted...] - Constraints: The fix is limited to ‘PermissionError‘ during package-discovery checks; accessible package directories should keep existing collection behavior. [omitted...] ### Task 3 Preserve system-exit behavior during pytest collection Ensure that pytest handles ‘KeyboardInterrupt‘ and ‘SystemExit‘ correctly during test collection. - Expectation: Collection-time ‘KeyboardInterrupt‘ and ‘SystemExit‘ should be re-raised as system exceptions rather than converted into ordinary collection failures. [omitted...] - Constraints: Other collection outcomes and non-system exceptions should continue to follow existing collection reporting semantics. ### Task 4 Handle read-only squashfuse_ll mounts without crashing Ensure that running on a read-only squashfuse_ll mount does not crash when cache directories cannot be created. - Expectation: Assertion rewrite cache setup should treat ‘errno.ENOSYS‘ from squashfuse_ll as a recoverable read-only filesystem condition, like ‘errno.EROFS‘. [omitted...] - Constraints: Only recognized read-only-equivalent cache-directory failures should be suppressed; unrelated ‘OSError‘ values must still propagate. ### Task 5 Restore default .pytest_cache directory permissions Ensure that automatically created .pytest_cache directories no longer use owner-only permissions after upgrading to pytest 8.2.1. - Expectation: ‘.pytest_cache‘ should receive normal directory permissions derived from the current process umask instead of owner-only access bits. [omitted...] - Constraints: Permission handling must respect the active umask and avoid changing the public cache API shape. [omitted...] ## Additional Changes - Ensure that users can verify the provenance of official pytest 8.2.1 source and wheel artifacts without changing pytest runtime behavior.
Example 8: L4 specification: problem statements with grounded expectations and constraints.
35
L5: Full Synthesized Specification (.md) # Upgrade Specification from 8.2.0 to 8.2.1 ## New Features ### Task 1 Add Python 3.13 compatibility and stricter pytester.makefile validation The 8.2.1 update adds Python 3.13 support for runtime behaviors that depend on frame locals and makes invalid pytester file-creation calls fail explicitly when no extension is provided. Ensure that the library works correctly on Python 3.13 for traceback and pytester workflows that depend on frame-local state, and ensure that invalid ‘Pytester.makefile()‘ calls with ‘ext=None‘ are rejected explicitly. - Expectation: On Python 3.13, behavior that reads frame locals in traceback processing and in ‘pytester‘ hook-recording checks must continue to work even when frame locals are exposed through a proxy-style mapping rather than a plain dict.
‘Pytester.makefile()‘ must raise ‘TypeError‘ when ‘ext‘ is ‘None‘
instead of accepting that input. - Constraints: Existing behavior for supported Python versions must remain intact. Recursion detection in tracebacks must still treat repeated frames with equivalent local-variable state as recursive. Valid ‘Pytester.makefile()‘ calls with a real extension string must continue to follow the existing extension rules, with only ‘None‘ being newly rejected. - Acceptance Criteria: - When the library runs on Python 3.13 and traceback processing needs to compare local-variable state across repeated frames, it should still recognize recursion correctly. - When code uses ‘pytester‘ features that inspect recorded hook calls on Python 3.13, those checks should complete successfully without errors caused by non-dict frame locals. - When ‘Pytester.makefile()‘ is called with ‘ext=None‘, it should raise ‘TypeError‘. - When ‘Pytester.makefile()‘ is called with a valid extension string, it should continue to create files under the existing extension-validation rules. - Behavioral Description: - On Python 3.13, traceback recursion detection continues to work when frame locals are no longer provided as a plain dict. - On Python 3.13, ‘pytester‘ hook-recording checks that evaluate against caller locals continue to work without failing because of the locals representation. - ‘Pytester.makefile()‘ now raises ‘TypeError‘ immediately when called with ‘ext=None‘. ## Bug Fixes ### Task 2 Avoid collection crashes from unreadable unselected directories The update fixes a runtime collection failure in pytest when an unreadable directory exists in the project tree but is not part of the command-line selection. Instead of aborting collection with a PermissionError while checking whether that directory is a Python package, pytest should ignore that directory and continue collecting the explicitly requested tests. Ensure that running pytest against selected test paths does not crash when unrelated directories in the project tree are unreadable. - Expectation: During directory collection, [omitted...] - Constraints: The fix is limited to PermissionError [omitted...] - Acceptance Criteria: [omitted...] - Behavioral Description: [omitted...] ### Task 3 Preserve system-exit behavior during pytest collection [omitted...] ### Task 4 Handle read-only squashfuse_ll mounts without crashing [omitted...] ### Task 5 Restore default .pytest_cache directory permissions [omitted...] ## Additional Changes - Ensure that users upgrading to pytest 8.2.1 can verify the provenance of official pytest distribution artifacts, including both source distributions and wheels, without changing pytest’s runtime behavior.
Example 9: L5 specification: problem statements, grounded expectations and constraints, behaviors, and acceptance criteria. 36
K
Limitations
First, we do not conduct human quality assessment of the synthesized specifications. Human quality assessment is prohibitively expensive even for a sampled subset, because it requires package-specific understanding of the release note, all linked issues or PRs, whole code changes, and the synthesized behavioral requirements. It is too expensive at our scale of 155 version transitions and 1,660 grounded upgrade tasks, but we make the pipeline controllable, auditable, stable, and grounded through releasenote and code-diff alignment, and schema constraints. The stability check in Appendix C shows that the matching agent remains stable even under the largest and most challenging transition. Second, SWE-C HAIN currently consists of only 12 Python package chains, so its coverage of other languages, build systems, and ecosystems remains limited. Third, we evaluate eight models under three agent CLIs, with only two open-weight models, and future work should cover more agent frameworks and open-source systems. Finally, we do not perform sampling-based repeated runs because long-horizon evaluation is costly, with a single chain costing up to around $150 for strong models.
L
Broader Impacts
SWE-C HAIN supports research on reliable coding agents by evaluating continuous release-level software maintenance scenarios. Agents must update existing codebases across consecutive package upgrades while resolving upgrade-related behaviors and avoiding regressions. This can help identify reliability gaps before deploying coding agents in real development workflows. Besides, our specification-granularity study in §4.3 provides a flexible setting for future research, enabling studies under both realistic upgrade guidance and more oracle-like conditions for probing upper-bound implementation capability and long-horizon reliability. However, the main risks are overgeneralization and unsafe automation. SWE-C HAIN focuses on open-source Python packages with release metadata, code diffs, and executable test suites, so conclusions may not directly generalize to proprietary software, non-Python ecosystems, UI-heavy systems, or domains with poorly documented behavioral requirements. Stronger coding agents may also enable large-scale, unsafe, or poorly reviewed code changes. We mitigate these risks by framing SWE-C HAIN as a controlled evaluation benchmark. The dataset is built from public open-source resources, does not intentionally include private information, and evaluates agents in sandboxed Docker environments with restricted tool access.
M
Declaration of LLM usage
We used LLMs to revise paper drafts and assist with writing basic framework code, such as standard utility functions and implementation scaffolding. All generated or assisted code was manually reviewed line by line by the authors before and after integration.
37