arXiv CS open access

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

2026 · arxiv_cs

arXiv CS · Papers · 2026

software-architecture software-engineering testing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring Yuling Shi1∗ , Jinghan Xu1∗ , Kelin Fu2 , Wenhao Zeng1 , Shilin He4 Lei Zhang5 , Yue Liu6 , Zelin Zhao4 , Terry Yue Zhuo7 , Jialun Cao3 , Siyu Ye4 Tianyu Liu2 , Kai Cai4 , Shing-Chi Cheung3 , Xiaodong Gu1† 1

Shanghai Jiao Tong University 2 Peking University The Hong Kong University of Science and Technology 4 Douyin Group 5 University of Chinese Academy of Sciences 6 National University of Singapore 7 Monash University

arXiv:2608.09802v1 [cs.CL] 10 Aug 2026

Abstract As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests— either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements—and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Dataset: https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax Correspondence: xiaodong.gu@sjtu.edu.cn

Introduction

AI coding agents have advanced rapidly in recent years, with benchmarks playing a central role in measuring and driving progress. The field has moved from function-level evaluation, such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), to repository-level challenges like SWE-bench (Jimenez et al., 2024) and its successors (Deng et al., 2025; Zan et al., 2026; Merrill et al., 2026; Le et al., 2025; Huang et al., 2026). As models improve, however, a pressing question emerges: do rising scores on these benchmarks genuinely reflect stronger software engineering ability, or are they reaching the limits of what current benchmark designs can measure? ∗ Equal contribution.

† Corresponding author. Published as a conference paper at COLM 2026.

INTRODUCTION

Modified Files Breakdown

100

12% 30% 80

Modified LOC Breakdown

100

13%

2-3

24%

4-5

Percentage (%)

25%

20%

86% 17%

34%

19%

21%

20%

33%

19% 18%

9% 0

SWE-bench Verified

101-200

93%

15%

SWE Bench ProMax SWE-bench Pro

51-100 >200

21%

<=20 21-50

16%

6-10 >10

32%

SWE Bench ProMax SWE-bench Pro

SWE-bench Verified

Figure 1 Distribution of modified files (left) and lines of code (right) per instance across benchmarks. SWE-Bench ProMax instances are substantially larger: 30% modify more than 10 files and 32% change over 200 lines of code, while 86% of SWE-bench Verified instances modify only a single file.

Code refactoring, the disciplined process of restructuring existing code without changing its external behavior, is one of the most frequent activities in professional software development and a key mechanism for managing technical debt in evolving codebases (Nyirongo et al., 2025). Industry leaders have identified refactoring as a prototypical long-horizon task: OpenAI highlights “project-scale refactors” as a primary use case for sustained, multi-context-window agents (OpenAI, 2025a,c), and Cursor reports that real-world developer tasks increasingly span many files and tools (Jain, 2026). Yet as a benchmark domain, refactoring remains underexplored: existing benchmarks primarily target isolated bug fixes and feature implementations (Jimenez et al., 2024; Deng et al., 2025), while the few efforts focused on refactoring are limited in scale or restricted to a single language (Xu et al., 2026; Gautam et al., 2025). We argue that real-world refactoring tasks, which require understanding large codebases (Peng et al., 2025; Shi et al., 2026), coordinating changes across many files, and preserving subtle behavioral invariants, offer a substantially more demanding and realistic test of agent capability than the tasks that dominate current evaluations. Instance: nasa/fprime#3422

Language: C++

244 files

+591 / −514 lines

Modified files (excerpt): Autocoders/Python/src/.../component/cpp.tmpl Drv/BlockDriver/BlockDriverImpl.cpp Fw/FPrimeBasicTypes.hpp [NEW] Fw/FPrimeBasicTypes.h [NEW] Fw/Types/BasicTypes.h FppTest/component/active/ActiveTest.cpp Os/Baremetal/TaskRunner/TaskRunner.cpp Svc/ActiveLogger/ActiveLoggerImpl.cpp ... and 238 more files Figure 2 A representative SWE-Bench ProMax instance: refactoring NASA’s F’Prime flight software framework requires coordinated changes across 244 files spanning the entire codebase.

Building a reliable refactoring benchmark, however, exposes two weaknesses that pervade existing benchmarks more broadly. The first is difficulty. On SWE-bench Verified, frontier agents now exceed 75% resolve rate (Singh et al., 2025), and the gap among top systems continues to narrow—prompting both academic (Kwa et al., 2025) and industry (Jain, 2026) observers to question whether public benchmarks still meaningfully differentiate frontier capabilities. A key reason is scope: most existing benchmark instances involve modifications to a small number of files with limited lines of code. Real-world refactoring operates at a fundamentally different scale, often requiring coordinated changes across dozens of files and hundreds of lines of code, demanding sustained cross-file reasoning that current agents struggle to maintain (Feng et al., 2026; Desai et al., 2026). The second weakness is evaluation quality. Test suites in existing benchmarks suffer from two complementary defects: overly narrow tests that enforce specific implementation details and reject functionally correct solutions, and overly broad tests that check behavior not specified in the task description (Deng et al., 2025; Merrill et al., 2026). A recent audit of SWE-bench Verified found these defects in nearly 60% of unsolved instances—35.5% 2

RELATED WORK

2.1.0.0

had narrow tests and 18.8% had broad tests—leading OpenAI to deprecate the benchmark entirely (OpenAI, 2026). Compounding this, problem descriptions are often imprecise or ambiguous, and growing evidence of data contamination in benchmarks sourced from public repositories further erodes the correlation between benchmark scores and genuine capability (OpenAI, 2026; Shi et al., 2024b). Additionally, despite growing multilingual efforts (Zan et al., 2026; Rashid et al., 2025), most benchmarks remain Python-centric, leaving open how well agents generalize across language paradigms such as Rust’s ownership model, C’s manual memory management, or Java’s type hierarchies. We introduce SWE-Bench ProMax to address these gaps. The benchmark comprises 170 refactoring instances drawn from real commits in actively maintained GitHub repositories, spanning seven programming languages: Python, Java, TypeScript, Go, C, C++, and Rust. As Figure 1 illustrates, SWE-Bench ProMax operates at a fundamentally different scale than existing benchmarks: 30% of our instances modify more than 10 files and 32% require over 200 lines of code, compared to SWE-bench Verified where 86% of instances modify only a single file. To make this concrete, Figure 2 shows a representative task from SWE-Bench ProMax: refactoring NASA’s F’Prime flight software framework requires migrating 244 files from a monolithic header to a new unified entry point—touching autocoders, drivers, OS layers, services, and build configurations—while preserving identical runtime behavior1 . The full problem statement and additional examples across all seven languages are presented in Appendix C. Our contributions are as follows. First, we ensure benchmark quality through rigorous, multi-stage expert curation. Issue descriptions are rewritten from scratch to provide precise, unambiguous specifications that fully define the expected refactoring. Test suites are manually reviewed to eliminate overly narrow and overly broad tests, and each problem statement is verified to serve as both a necessary and sufficient condition for the gold patch. Second, the benchmark presents a genuine challenge to frontier models. During curation, tasks with insufficient complexity, limited cross-file scope, or too few lines of code are systematically filtered out. The remaining instances require large-scale, coordinated modifications averaging 11.4 files and 261.6 lines of code, and the best model in our evaluation achieves only 41.2% resolve rate. Notably, higher cost does not guarantee proportionally better performance: Claude Sonnet 4.6 averages $4.77 per instance yet resolves 38.8%, while GLM-5 attains a comparable 36.5% at only $0.24—suggesting that open-weight models can approach frontier performance at a fraction of the cost. Agent trajectory analysis further reveals that failed attempts consistently modify fewer files than the gold patch requires while consuming more interaction rounds, pointing to incomplete cross-file coordination as the dominant failure mode. Third, the benchmark spans seven diverse programming languages, enabling analysis of how language-specific features, including type systems, memory models, and build ecosystems, affect agent performance on code transformation tasks.

Related work

2.1

Coding benchmarks

As frontier code models (Zhu et al., 2024; Cao et al., 2026; Zeng et al., 2026a; Team et al., 2025; Singh et al., 2025) and autonomous agents (Yang et al., 2024; Wang et al., 2025; Xia et al., 2024; Zhao et al., 2026) have rapidly advanced, with complementary work on efficient code reasoning, repository-level context construction, collaborative inference, and long-horizon agent memory (Zeng et al., 2025; Wang et al., 2026; Hu et al., 2026; Shi et al., 2025; Zeng et al., 2026c; Li et al., 2025; Gao et al., 2026; Chen et al., 2025), benchmarks have become the critical bottleneck for measuring real progress. Table 1 compares SWE-Bench ProMax with existing benchmarks across six dimensions. From function-level to long-horizon evaluation. Early benchmarks such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and LiveCodeBench (Jain et al., 2024) evaluate isolated code generation. SWEbench (Jimenez et al., 2024) introduced repository-level evaluation from real GitHub issues, with subsequent work extending along multiple axes: Multi-SWE-bench (Zan et al., 2026) and SWE-PolyBench (Rashid et al., 2025) broaden language coverage; SWE-bench Pro (Deng et al., 2025) and SWE-EVO (Le et al., 2025) target harder, long-horizon tasks (SWE-EVO averages 21 modified files); Terminal-Bench (Merrill et al., 2026) curates 89 hard CLI challenges with human-written verification; and SWE-bench Live (Zhang et al., 2026), 1 https://github.com/nasa/fprime/pull/3422

SWE-BENCH PROMAX

3.2

SWE-rebench (Badertdinov et al., 2025), SWE-Factory (Guo et al., 2025) and Dockerless (Zeng et al., 2026b) address contamination, infrastructure and verification. Current benchmarks are rapidly saturating—frontier agents exceed 75% on SWE-bench Verified (Jimenez et al., 2024)—and recent studies show that agent capabilities on short tasks are doubling every few months (Kwa et al., 2025), while all state-of-the-art agents achieve less than 20% on long-horizon tasks (Feng et al., 2026). SWE-Bench ProMax targets this difficulty frontier: our instances require coordinated changes across 11.4 files and 261.6 lines of code on average, and the best evaluated model achieves only 41.2% resolve rate. Refactoring benchmarks. Despite the prevalence of refactoring in professional development, only two benchmarks specifically target it. RefactorBench (Gautam et al., 2025) provides 100 handcrafted multi-file tasks across 9 Python repositories but is limited to a single language, with instances averaging only 4.3 modified files. SWE-Refactor (Xu et al., 2026) offers 1,099 instances from 18 Java repositories with automated validation but covers only Java and lacks human verification of test quality. Neither combines multilingual coverage, large-scale complexity, and expert-curated test suites—the combination that SWE-Bench ProMax provides. Expert curation. The degree of human curation varies widely across benchmarks (Table 1). At the repository level, curation ranges from fully automated (SWE-bench (Jimenez et al., 2024)) to extensive: Multi-SWEbench (Zan et al., 2026) employs 68 annotators, Terminal-Bench (Merrill et al., 2026) invests approximately three reviewer-hours per task, and SWE-bench Pro (Deng et al., 2025) rewrites issue descriptions and reviews test scope. However, even well-curated benchmarks retain quality issues: an audit of SWE-bench Verified found that nearly 60% of unsolved instances had material defects in test design or problem descriptions (OpenAI, 2026). SWE-Bench ProMax applies expert curation systematically to every instance: issue descriptions are rewritten from scratch as precise specifications informed by the gold patch and test suite, and inappropriate tests (overly narrow or overly broad) are identified and removed, ensuring mutual alignment between specification and evaluation.

2.2

Code refactoring

Traditional refactoring tools such as RefactoringMiner (Tsantalis et al., 2020) focus on detecting refactorings from commit histories rather than generating them, and empirical studies reveal that 61 out of 100 identified refactoring types remain unsupported by existing engines (Nyirongo et al., 2025; Liu et al., 2025). Recent work has begun evaluating LLMs as refactoring agents, revealing consistent limitations: agents perform only low-level edits and fail to address high-level design issues (Horikawa et al., 2025), struggle with complex contextdependent refactoring (Cordeiro et al., 2024), and achieve only 7.7% alignment when autonomously discovering needed changes (Thillen et al., 2026). Crucially, each study uses different evaluation methodologies—code smell counts (Cordeiro et al., 2024), compilability (Tapader et al., 2025), alignment scores (Thillen et al., 2026)—making cross-study comparison impossible and motivating a standardized, execution-based benchmark with human-verified tests. SWE-Bench ProMax fills this role.

SWE-Bench ProMax

This section describes how the benchmark is constructed, verified, and composed.

3.1

Task formulation

Each instance in SWE-Bench ProMax consists of four components: (1) a pre-configured Docker environment containing the target repository at the commit immediately before the refactoring, with all dependencies installed; (2) an issue description specifying the intended refactoring in precise natural language; (3) a test suite that validates whether the refactoring has been correctly applied; and (4) a gold patch recording the original developer’s solution. Given the environment and the issue description, an agent must autonomously modify the repository so that all tests pass. An instance is considered resolved if and only if the agent’s modifications pass every test in the suite. This formulation is outcome-driven: we evaluate the final state of the repository rather than the specific commands or intermediate steps the agent takes.

SWE-BENCH PROMAX

3.2.0.0

Execution Based

Benchmark

Repo Level

HumanEval (Chen et al., 2021) MBPP (Austin et al., 2021) LiveCodeBench (Jain et al., 2024)

✓ ✓ ✓

SWE-bench (Jimenez et al., 2024) Multi-SWE-bench (Zan et al., 2026) SWE-PolyBench (Rashid et al., 2025) SWE-bench Pro (Deng et al., 2025) SWE-EVO (Le et al., 2025) Terminal-Bench (Merrill et al., 2026)

✓ ✓ ✓ ✓ ✓ ✓

RefactorBench (Gautam et al., 2025) SWE-Refactor (Xu et al., 2026)

✓ ✓

SWE-Bench ProMax (Ours)

✓

Multi Lingual

Avg. >5 Files

Refactoring

Expert Curated ✓ ✓

✓ ✓

✓ ✓ ✓ ✓ ✓ ✓

✓

Table 1 Comparison of SWE-Bench ProMax with existing benchmarks. Avg. >5 Files: the average gold patch modifies more than 5 files. Expert Curated: problem descriptions and/or test suites undergo manual expert review or authoring beyond automated collection. Stage 1: Data Collection

Stage 2: Environment Construction

GitHub API

Search Repositories

Refactoring Commits

≥500 Stars License Lang≥80%

Clone Repository

Build Docker Environment

Analyze Commit Changes

Pre-refatoring State

After 2025 "refactor" No "bug fix"

Apply Gold Patch

Test & Source Patch

Run Test Suite Discard Refactoring Commits

Refine Too Narrow or Too Wide Quality Filtering

Human

Install Dependencies

Candidate Repositories

Extract Commits

Stage 3: Filtering & Problem Rewriting

Validated Instances

Rewrite Problem Statement

Human LLM

Human Verification

SWE Bench ProMax

Figure 3 Data collection and curation pipeline for SWE-Bench ProMax.

3.2

Dataset construction and curation

Figure 3 illustrates our three-stage pipeline, which progresses from automated collection through environment validation to expert-driven curation. The quality of a benchmark is determined not by its size but by the reliability of its evaluation (Merrill et al., 2026; Le et al., 2025); accordingly, every instance in SWE-Bench ProMax undergoes multi-stage review before inclusion, and our final benchmark retains only 170 out of 29,782 initial candidates. Stage 1: Data collection. We use the GitHub API to identify candidate repositories that meet three criteria: at least 500 stars, an approved open-source license, and a primary language (comprising at least 80% of the codebase) among our seven target languages. From these repositories, we extract commits submitted after January 2025 whose messages contain the keyword “refactor” but not “bug fix,” and that modify both test and non-test files. This yields a large initial pool of refactoring-related commits across all seven languages. Stage 2: Environment construction. For each candidate commit, we construct an isolated Docker environment containing the repository at the pre-refactoring state with all build dependencies installed, leveraging automated 5

SWE-BENCH PROMAX

Issue Description Tokens

3.3.0.0

Mean

Max

685.3

2,092

TypeScript (28) Python (29)

Gold Patch (source) # Files Lines of code Tokens

11.4 261.6 8,179.5

182 4,503 72,623

Test Patch # Files Lines of code Tokens

4.5 185.5 3,980.8

66 1,959 52,031

Total # Files

15.9

244

16.5%

C++ (22)

17.1%

12.9% 12.9%

Rust (22)

11.8%

C (20)

13.5% 15.3%

Go (23) Java (26)

Table 2 Dataset statistics of SWE-Bench ProMax. Figure 4 Language distribution in SWE-Bench ProMax.

environment construction tools (Guo et al., 2025; Fu et al., 2026). The repository is cloned at the pre-refactoring commit, the gold patch (comprising both source and test changes) is applied, and the full test suite is executed. Instances where a working environment cannot be established or where the gold patch fails to pass the test suite are discarded, yielding a set of validated instances. Stage 3: Filtering and problem rewriting. Because our instances are mined from commits rather than curated issues, the raw data lacks the precise problem descriptions that a benchmark requires—commit messages are written for fellow developers, not as task specifications for AI agents. The final stage therefore involves human experts working with LLM assistance to transform validated instances into high-quality benchmark tasks through four steps. (1) Commit analysis. Experts analyze the commit diff, assisted by LLMs that summarize changes and identify affected components, to understand the scope, intent, and structural impact of each refactoring before any filtering or rewriting decisions are made. (2) Quality filtering. Instances with insufficient complexity are removed: we discard tasks confined to a single file, those with too few lines of code modified, or those involving overly simplistic patterns. We also review each test suite to identify and remove overly narrow tests that enforce specific implementation details rather than behavioral outcomes, and overly broad tests that check behavior beyond the scope of the refactoring—directly addressing the quality defects that have undermined prior benchmarks (OpenAI, 2026). (3) Problem statement rewriting. Original commit messages are typically terse (e.g., “refactor auth module”), ambiguous, or reference internal context unavailable to an agent. Experts therefore rewrite the issue description from scratch with LLM assistance, producing a precise, self-contained specification that states which components should change, what the expected transformation is, and what behavioral invariants must be preserved. Each description is verified to serve as both a necessary and sufficient condition for the gold patch: a correct solution should satisfy the description, and the description should not admit unintended solutions. (4) Human verification. A final round of expert review ensures consistency across the issue description, test suite, and gold patch. The description must fully specify the refactoring, the tests must pass if and only if the refactoring is correctly applied, and no unstated requirements remain.

3.3

Dataset composition

Language coverage. SWE-Bench ProMax spans seven programming languages that represent diverse paradigms:

RESULTS AND ANALYSIS

5.1

dynamically typed (Python), statically typed with garbage collection (Java, Go), gradually typed (TypeScript), systems languages with manual memory management (C, C++), and a language with an ownership-based memory model (Rust). Instances are drawn from 70 distinct repositories (Appendix A), ensuring broad coverage beyond a small number of projects. This diversity enables us to examine how language-specific features affect agent performance on refactoring tasks. Scale and complexity. A defining characteristic of SWE-Bench ProMax is the scale of required code changes. As shown in Table 2, gold patches average 11.4 source files and 261.6 lines of code (8,179.5 tokens), with the most complex instances modifying up to 182 source files. Test patches add a further 4.5 files and 185.5 lines on average, bringing the total to 15.9 files per instance. Issue descriptions average 685.3 tokens, providing detailed specifications of the required refactoring. As shown in Figure 1, our instances require substantially more extensive modifications than those in SWE-bench Verified or SWE-bench Pro, reflecting the true complexity of production-scale refactoring.

4 4.1

Experiments Experimental setup

Agent scaffolds. We evaluate all models under two agent scaffolds. The first is mini-swe-agent (Yang et al., 2024), a minimal reimplementation of the widely adopted SWE-agent scaffold that has been used as the default evaluation framework (Deng et al., 2025; Zan et al., 2026; Merrill et al., 2026); it provides file viewing, editing, searching, and bash execution in an iterative observe-think-act loop. The second is OpenHands (Wang et al., 2025), an open platform for generalist software agents that equips models with a richer runtime, including sandboxed command execution and structured file-editing tools. Given the large scale of refactoring patches in SWE-Bench ProMax, we set a step limit of 300 and a cost limit of $10 per instance under both scaffolds. Applying the same scaffolds and limits to all models ensures fair comparison. Execution environment. Each instance ships with a pre-built, isolated Docker container constructed during benchmark curation using SWE-Factory (Guo et al., 2025) and manually verified to be functional. The container contains the repository at the pre-refactoring commit with all dependencies installed, so evaluation requires no additional environment setup. Evaluation protocol. An instance is resolved if the agent’s modifications pass every test in the suite. Our primary metric is the resolve rate (Pass@1): the percentage of resolved instances. We evaluate on all 170 instances and report both overall and per-language resolve rates. We also analyze average cost ($) per instance across models.

4.2

Models

We evaluate six frontier models spanning both proprietary and open-weight families: Proprietary models. (1) Gemini-3-Pro (Google DeepMind, 2025), Google’s flagship model; (2) Claude Sonnet 4.6 (Anthropic, 2026), Anthropic’s latest Sonnet-class model; (3) GPT-5.2 (OpenAI, 2025b), OpenAI’s most recent reasoning model. Open-weight models. (4) GLM-5 (Zeng et al., 2026a), a mixture-of-experts model from Zhipu AI; (5) Kimi-

K2.5 (Team et al., 2026), Moonshot AI’s agentic intelligence model; (6) Qwen3.5 (Cao et al., 2026), Alibaba’s latest code-oriented model. This selection covers diverse architectures, scales, and training paradigms, enabling analysis of how these factors affect refactoring performance.

Results and analysis

RESULTS AND ANALYSIS

Overall

5.2

Avg. Steps

Avg. Cost

Per-Language Resolve Rate Py

Java

C++

Rust

Mini-SWE-Agent Proprietary Gemini-3-Pro Claude Sonnet 4.6 GPT-5.2

26.5 30.6 21.8

58.0 99.5 25.2

$0.60 $2.32 $0.19

17.2 13.8 17.2

15.4 19.2 15.4

14.3 32.1 21.4

26.1 13.0 13.0

50.0 65.0 45.0

45.5 40.9 31.8

27.3 40.9 13.6

Open-weight GLM-5 Kimi-K2.5 Qwen3.5

22.9 26.5 20.6

108.9 85.3 155.4

$0.10 $0.37 $0.93

13.8 17.2 17.2

7.7 23.1 7.7

25.0 21.4 10.7

21.7 17.4 13.0

50.0 60.0 45.0

22.7 31.8 27.3

27.3 22.7 31.8

OpenHands Proprietary Gemini-3-Pro Claude Sonnet 4.6 GPT-5.2

19.4 38.8 41.2

51.2 117.9 115.1

$1.49 $4.77 $3.60

13.8 17.2 48.3

19.2 30.8 19.2

0.0 53.6 35.7

8.7 26.1 26.1

45.0 50.0 75.0

36.4 36.4 36.4

22.7 63.6 54.5

Open-weight GLM-5 Kimi-K2.5 Qwen3.5

36.5 32.9 36.5

114.2 99.6 141.2

$0.24 $0.72 $0.78

20.7 24.1 37.9

34.6 30.8 26.9

28.6 10.7 17.9

34.8 43.5 39.1

65.0 70.0 65.0

45.5 45.5 54.5

36.4 18.2 22.7

Table 3 Resolve rates (%) of all evaluated models across scaffolds on SWE-Bench ProMax.

5.1

Main results

All models find SWE-Bench ProMax challenging. The best-performing model, GPT-5.2, achieves only

41.2% resolve rate—far below the 75%+ that frontier agents achieve on SWE-bench Verified (Singh et al., 2025). This confirms that multi-file refactoring, with its requirements for cross-file coordination and behavioral preservation, remains a substantial unsolved challenge. Open-weight models are competitive with proprietary ones at a fraction of the cost. Under OpenHands,

GLM-5 and Qwen3.5 (both 36.5%) and Kimi-K2.5 (32.9%) come within a few points of GPT-5.2 (41.2%) and Claude Sonnet 4.6 (38.8%), while spending only a fraction as much per instance ($0.24, $0.78, and $0.72 versus $3.60 and $4.77). The choice of scaffold also matters substantially: every model except Gemini-3-Pro improves markedly when moving from mini-swe-agent to OpenHands (e.g., GPT-5.2 from 21.8% to 41.2%), suggesting that richer runtime tooling is particularly beneficial for large-scale refactoring tasks. Performance varies substantially across languages. No single model dominates all languages. Claude

Sonnet 4.6 leads on TypeScript (53.6%) and Rust (63.6%), GLM-5 leads on Java (34.6%), while GPT-5.2 performs best on Python (48.3%) and C (75.0%). Kimi-K2.5 achieves its best result on Go (43.5%), while Qwen3.5 performs best on C++ (54.5%). This diversity suggests that different model architectures and training data compositions lead to complementary language-level strengths. TypeScript and Rust show surprising variance. Despite their reputation as complex languages, TypeScript

and Rust yield high resolve rates for some models (Claude Sonnet 4.6: 53.6% on TypeScript and 63.6% on Rust; GPT-5.2: 54.5% on Rust) while remaining difficult for others (Gemini-3-Pro: 0.0% on TypeScript; Kimi-K2.5: 18.2% on Rust). This variance may reflect differences in language-specific training data rather than inherent language difficulty.

5.2

Agent behavior analysis

Figure 5 reveals two complementary failure patterns across two representative models (Claude Sonnet 4.6 and Kimi-K2.5). The left panel compares the number of files modified by each agent against the gold patch. Both Claude Sonnet 4.6 and Kimi-K2.5 closely track the gold patch distribution for small changes (up to ∼5 files), but diverge sharply for larger patches: whereas the gold patch CDF reaches 90% only around 20 files, both agents reach

CONCLUSION

100

80 70

Gold Patch Kimi Claude

60 500

Cumulative %

100 Cumulative %

60 40

Kimi (pass) Kimi (fail) Claude (pass) Claude (fail)

20 00

10 20 30 40 50 60 70 80 # Modified Files

100

150

# Interaction Rounds

200

250

Figure 5 Left: Cumulative distribution of files modified by agents (Claude Sonnet 4.6, Kimi-K2.5) versus the gold patch. Agents modify fewer files than required overall, with the gap widening for larger patches. Right: Cumulative distribution of interaction rounds on resolved (solid) versus unresolved (dashed) instances. Failed attempts consume substantially more rounds than successful ones.

90% by approximately 10 files. This indicates that the dominant failure mode is incomplete refactoring—agents identify and modify some affected files but fail to propagate changes to all locations requiring coordinated updates. Critically, this is not a matter of failing to find the right files: agents often correctly locate and edit the core files involved in a refactoring, but stop short of applying the same transformation to peripheral call sites, documentation, configuration files, and test fixtures that also need updating. This partial coverage means that even when the central logic is correctly refactored, the test suite fails because downstream dependencies remain inconsistent. The right panel separates resolved (solid) from unresolved (dashed) instances. For both models, successful resolutions complete in markedly fewer interaction rounds: the pass curves rise steeply and plateau early, while the fail curves are shifted rightward and more gradual. This gap reveals that agents entering unproductive cycles—repeatedly reading files, attempting edits, encountering test failures, and reverting—consume many rounds without expanding the scope of their modifications. In contrast, successful agents exhibit focused behavior, efficiently identifying and modifying the full set of required files in fewer steps. Together with the file-count analysis, these patterns suggest that the key bottleneck is not reasoning ability per se, but rather the capacity to maintain a coherent plan across many files and persist through the cascading consequences of a large-scale structural change.

5.3

Cost and efficiency analysis

Table 3 reports the average number of agent steps and API cost per instance for each model. A striking pattern emerges: higher cost does not translate to proportionally higher resolve rate. Under OpenHands, Claude Sonnet 4.6 is the most expensive model ($4.77 per instance, 117.9 steps on average) yet trails GPT-5.2 (41.2% at $3.60), and Gemini-3-Pro spends $1.49 per instance for only 19.4%. Open-weight models are dramatically more cost-efficient: GLM-5 resolves 36.5% at just $0.24 per instance—roughly one-twentieth the cost of Claude Sonnet 4.6—and Kimi-K2.5 achieves 32.9% at $0.72. We also observe that a larger step budget does not guarantee progress. Manual inspection of trajectories shows that models facing large multi-file patches can enter repetitive edit–revert cycles—which we term unproductive exploration—consuming many steps without expanding the scope of their modifications. Qwen3.5 represents an extreme case: it takes the most steps under both scaffolds (155.4 and 141.2 on average) yet never leads overall, and under mini-swe-agent it achieves the lowest resolve rate (20.6%), suggesting that excessive exploration without effective cross-file coordination is counterproductive.

Conclusion

We presented SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark designed to address three critical gaps in current AI coding evaluation: the saturation of existing benchmarks, the 9

REFERENCES

lack of evaluation quality assurance, and the absence of refactoring as a benchmark domain. SWE-Bench ProMax comprises 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust) and 70 repositories, selected from 29,782 initial candidates through a rigorous three-stage pipeline. Every instance undergoes expert curation: issue descriptions are rewritten from scratch to eliminate ambiguity and answer leakage, and test suites are manually reviewed to remove overly narrow tests that reject valid solutions and overly broad tests that check unstated requirements. Evaluation of six frontier models reveals that SWE-Bench ProMax remains far from saturated: the best model (GPT-5.2) achieves only 41.2% resolve rate, and no model dominates across all languages. Our analysis uncovers two key findings. First, open-weight models are highly competitive with proprietary ones on large-scale refactoring: GLM-5 and Qwen3.5 come within five points of the best proprietary model at a fraction of the cost, and manual trajectory inspection shows that higher spending often reflects unproductive exploration cycles rather than genuine progress. Second, the dominant failure mode is incomplete refactoring: agents consistently modify fewer files than the gold patch requires, indicating that sustained cross-file coordination remains a fundamental bottleneck for current AI coding agents. Acknowledgments This research was supported by the National Key Research and Development Program of China (Grant No. 2023YFB4503802), the Natural Science Foundation of Shanghai (Grant No. 25ZR1401175), the Hong Kong Research Grant Council General Research Fund (Grant No. 16206524), and the Hong Kong Research Grant Council Theme-based Research Scheme (Grant No. T41-517_25-N). We thank Kexin Pei for valuable feedback and advice.

Ethics statement All repositories included in our benchmark are publicly available on GitHub and use approved open-source licenses (e.g., MIT, Apache 2.0, BSD), and we respect each project’s licensing terms in our distribution. No personal or sensitive data is collected; our dataset consists solely of code, commit metadata, and test cases derived from public repositories. The benchmark is intended exclusively for the evaluation and improvement of AI coding agents, and we encourage responsible use that aligns with the goals of advancing software engineering research.

LLM usage disclosure In accordance with the policy on LLM usage, we disclose the following uses of large language models in this work: (1) Human annotators collaboratively used LLMs as writing assistants to rewrite problem statements and review test suites during benchmark curation (Stage 3 of our pipeline); the LLM served as an interactive tool under human direction, not as an autonomous generator. (2) Claude Sonnet 4.6 was used for multi-label classification of task categories and required reasoning skills (Appendix B); these classifications are used for analysis only and do not affect the benchmark instances or evaluation results. (3) LLMs were used for minor assistance in drafting and editing portions of this paper.

References Anthropic. Claude sonnet 4.6. https://www.anthropic.com/news/claude-sonnet-4-6, 2026. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents. arXiv preprint arXiv:2505.20411, 2025. Ruisheng Cao, Mouxiang Chen, Jiawei Chen, Zeyu Cui, Yunlong Feng, Binyuan Hui, Yuheng Jing, Kaixin Li, Mingze Li, Junyang Lin, et al. Qwen3-coder-next technical report. arXiv preprint arXiv:2603.00729, 2026.

REFERENCES

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Silin Chen, Shaoxin Lin, Yuling Shi, Heng Lian, Xiaodong Gu, Longfei Yun, Dong Chen, Lin Cao, Jiyang Liu, Nu Xia, et al. Swe-exp: Experience-driven software issue resolution. arXiv preprint arXiv:2507.23361, 2025. Jonathan Cordeiro, Shayan Noei, and Ying Zou. An empirical study on the code refactoring capability of large language models. ACM Transactions on Software Engineering and Methodology, 2024. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025. Rishi Desai, Jesse Hu, Joan Cabezas, Neel Harsola, Pratyush Shukla, Roey Ben Chaim, Adnan El Assadi, Omkaar Mukund Kamath, Fenil Faldu, Prannay Hebbar, et al. Swe-marathon: Can agents autonomously complete ultra-long-horizon software work? arXiv preprint arXiv:2606.07682, 2026. Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, et al. Longcli-bench: A preliminary benchmark and study for long-horizon agentic programming in command-line interfaces. In Findings of the Association for Computational Linguistics: ACL 2026, pages 29952–29963, 2026. Dayuan Fu, Shenyu Wu, Yunze Wu, Zerui Peng, Yaxing Huang, Jie Sun, Ji Zeng, Mohan Jiang, Lin Zhang, Yukun Li, et al. davinci-env: Open swe environment synthesis at scale. arXiv preprint arXiv:2603.13023, 2026. Shuzheng Gao, Wenhao Zeng, Zhaojian Yu, Jianqiao Wangni, Chaozheng Wang, Kai Cai, Shilin He, and Michael R Lyu. Swe-mem: Learning adaptive memory management for long-horizon coding agents. arXiv preprint arXiv:2606.28434, 2026. Dhruv Gautam, Spandan Garg, Jinu Jang, Neel Sundaresan, and Roshanak Zilouchian Moghaddam. Refactorbench: Evaluating stateful reasoning in language agents through code. arXiv preprint arXiv:2503.07832, 2025. Google DeepMind. Gemini 3 pro. https://blog.google/products-and-platforms/products/gemini/gemini-3/, 2025. Lianghong Guo, Yanlin Wang, Caihua Li, Wei Tao, Pengyu Yang, Jiachi Chen, Haoyu Song, Duyu Tang, and Zibin Zheng. Swe-factory: Your automated factory for issue resolution training data and evaluation benchmarks. arXiv preprint arXiv:2506.10954, 2025. Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, and Ahmed E Hassan. Agentic refactoring: An empirical study of ai coding agents. arXiv preprint arXiv:2511.04824, 2025. Chao Hu, Wenhao Zeng, Yuling Shi, Beijun Shen, and Xiaodong Gu. In line with context: Repository-level code generation via context inlining. Proceedings of the ACM on Software Engineering, 3(FSE):1469–1491, 2026. Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. Deepswe: Measuring frontier coding agents on original, long-horizon engineering tasks. arXiv preprint arXiv:2607.07946, 2026. Naman Jain. How we compare model quality at Cursor. https://cursor.com/blog/cursorbench, 2026. Accessed: 2026-03-11. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024. Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, et al. Measuring ai ability to complete long tasks. arXiv preprint arXiv:2503.14499, 352, 2025. Tue Le, Minh VT Thai, Dung Nguyen Manh, Huy Phan Nhat, and Nghi DQ Bui. Swe-evo: Benchmarking coding agents in long-horizon software evolution scenarios. arXiv preprint arXiv:2512.18470, 2025. Han Li, Yuling Shi, Shaoxin Lin, Xiaodong Gu, Heng Lian, Xin Wang, Yantao Jia, Tao Huang, and Qianxiang Wang. Swe-debate: Competitive multi-agent debate for software issue resolution. arXiv preprint arXiv:2507.23348, 2025.

REFERENCES

Bo Liu, Hui Liu, Nan Niu, Yuxia Zhang, Guangjie Li, He Jiang, and Yanjie Jiang. An automated approach to discovering software refactorings by comparing successive versions. IEEE Transactions on Software Engineering, 51 (5):1358–1380, 2025. Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. Bridget Nyirongo, Yanjie Jiang, Nan Niu, and Hui Liu. An empirical study of software refactorings in real-world open-source java projects. IEEE Transactions on Software Engineering, 2025. OpenAI. GPT-5.1-Codex-Max. https://openai.com/index/gpt-5-1-codex-max/, 2025a. Accessed: 2025-11-19. OpenAI. GPT-5 system card update: GPT-5.2. https://openai.com/index/gpt-5-system-card-update-gpt-5-2/, 2025b. OpenAI. Introducing upgrades to Codex. https://openai.com/index/introducing-upgrades-to-codex/, 2025c. Accessed: 2025-09-15. OpenAI. Why SWE-bench Verified no longer measures frontier coding capabilities. https://openai.com/index/why -we-no-longer-evaluate-swe-bench-verified/, 2026. Accessed: 2026-02-23. Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen, and Xiaodong Gu. Swe-qa: Can language models answer repository-level code questions? arXiv preprint arXiv:2509.14635, 2025. Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents. arXiv preprint arXiv:2504.08703, 2025. Yuling Shi, Songsong Wang, Chengcheng Wan, Min Wang, and Xiaodong Gu. From code to correctness: Closing the last mile of code generation with hierarchical debugging. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE 2026), 2024a. Yuling Shi, Hongyu Zhang, Chengcheng Wan, and Xiaodong Gu. Between lines of code: Unraveling the distinct patterns of machine and human programmers. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE 2025), 2024b. Yuling Shi, Yichun Qian, Hongyu Zhang, Beijun Shen, and Xiaodong Gu. Longcodezip: Compress long context for code language models. In 2025 IEEE/ACM 40th International Conference on Automated Software Engineering (ASE), 2025. Yuling Shi, Chaoxiang Xie, Zhensu Sun, Yeheng Chen, Chenxu Zhang, Longfei Yun, Chengcheng Wan, Hongyu Zhang, David Lo, and Xiaodong Gu. Codeocr: On the effectiveness of vision language models in code understanding. arXiv preprint arXiv:2602.01785, 2026. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. Md Raihan Tapader, Md Mostafizer Rahman, Ariful Islam Shiplu, Md Faizul Ibne Amin, and Yutaka Watanobe. Code refactoring with llm: A comprehensive evaluation with few-shot settings. arXiv preprint arXiv:2511.21788, 2025. Kimi Team, Yifan Bai, Yiping Bao, Y Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. Alex Thillen, Niels Mündler, Veselin Raychev, and Martin Vechev. Codetaste: Can llms generate human-level code refactorings? In Forty-third International Conference on Machine Learning, 2026. Nikolaos Tsantalis, Ameya Ketkar, and Danny Dig. Refactoringminer 2.0. IEEE Transactions on Software Engineering, 48(3):930–950, 2020. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume 2025, pages 65882–65919, 2025.

REFERENCES

Yuhang Wang, Yuling Shi, Mo Yang, Rongrui Zhang, Shilin He, Heng Lian, Yuting Chen, Siyu Ye, Kai Cai, and Xiaodong Gu. Swe-pruner: Self-adaptive context pruning for coding agents. arXiv preprint arXiv:2601.16746, 2026. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489, 2024. Yisen Xu, Jinqiu Yang, et al. Swe-refactor: A repository-level benchmark for real-world llm-based code refactoring. arXiv preprint arXiv:2602.03712, 2026. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Li Aoyan, Lu Chen, Xiaojian Zhong, et al. Multi-swe-bench: A multilingual benchmark for issue resolving. Advances in Neural Information Processing Systems, 38, 2026. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026a. Wenhao Zeng, Yaoning Wang, Chao Hu, Yuling Shi, Chengcheng Wan, Hongyu Zhang, and Xiaodong Gu. Pruning the unsurprising: Efficient code reasoning via first-token surprisal. arXiv preprint arXiv:2508.05988, 2025. Wenhao Zeng, Yuling Shi, Xiaodong Gu, Chao Hu, Chaofan Wang, Yuhao Cui, Hongting Zhou, Mengnan Qi, Jianqiao Wangni, Zhaojian Yu, et al. Dockerless: Environment-free program verifier for coding agents. arXiv preprint arXiv:2606.28436, 2026b. Wenhao Zeng, Xuteng Zhang, Yuling Shi, Chao Hu, Yuting Chen, Beijun Shen, and Xiaodong Gu. Glimprouter: Efficient collaborative inference by glimpsing one token of thoughts. arXiv preprint arXiv:2601.05110, 2026c. Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. Swe-bench goes live! Advances in Neural Information Processing Systems, 38, 2026. Jiale Zhao, Guoxin Chen, Fanzhe Meng, Minghao Li, Jie Chen, Hui Xu, Yongshuai Sun, Wayne Xin Zhao, Ruihua Song, Yuan Zhang, et al. Immersion in the github universe: Scaling coding agents to mastery. arXiv preprint arXiv:2602.09892, 2026. Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024.

REFERENCES

Additional dataset details

Table 4 Repositories and licenses used in SWE-Bench ProMax, grouped by language. 70 repositories across 7 languages. Language

Repository

License

betaflight/betaflight aviggiano/redis-roaring davidesantangelo/krep radareorg/radare2 CESNET/libyang arkq/bluez-alsa aws/s2n-tls bitcoin-core/secp256k1 openssl/openssl

GPL-3.0 MIT BSD-2-Clause LGPL-3.0 BSD-3-Clause MIT Apache-2.0 MIT Apache-2.0

C++

deskflow/deskflow ETLCPP/etl nasa/fprime Icinga/icinga2 LMMS/lmms OpenOrienteering/mapper WasmEdge/WasmEdge bloomberg/blazingmq biojppm/rapidyaml

GPL-2.0 MIT Apache-2.0 GPL-3.0 GPL-2.0 GPL-3.0 Apache-2.0 Apache-2.0 MIT

cli/cli go-gitea/gitea TecharoHQ/anubis restic/restic OpenListTeam/OpenList caddyserver/caddy derailed/k9s gitleaks/gitleaks gohugoio/hugo grpc/grpc-go istio/istio jesseduffield/lazygit kubernetes/kubernetes rqlite/rqlite samber/lo trufflesecurity/trufflehog

MIT MIT MIT BSD-2-Clause AGPL-3.0 Apache-2.0 Apache-2.0 MIT Apache-2.0 Apache-2.0 Apache-2.0 MIT Apache-2.0 MIT MIT AGPL-3.0

Java

bazelbuild/bazel plantuml/plantuml hibernate/hibernate-orm apache/hbase apache/fesod apache/iceberg apache/maven apache/pinot google/gson alibaba/nacos swagger-api/swagger-core

Apache-2.0 GPL-3.0 Apache-2.0 Apache-2.0 Apache-2.0 Apache-2.0 Apache-2.0 Apache-2.0 Apache-2.0 Apache-2.0 Apache-2.0

Python

confident-ai/deepeval google/adk-python optuna/optuna stanfordnlp/dspy

Apache-2.0 Apache-2.0 MIT MIT Continued on next page

TASK CATEGORY ANALYSIS

Language

Repository

License

vibrantlabsai/ragas albumentations-team/albumentations huggingface/transformers langchain-ai/langchain verl-project/verl django/django google/langextract huggingface/lerobot hummingbot/hummingbot icloud-photos-downloader/icloud_photos_downloader mikf/gallery-dl pandas-dev/pandas pypa/pipenv roboflow/supervision

Apache-2.0 MIT Apache-2.0 MIT Apache-2.0 BSD-3-Clause Apache-2.0 Apache-2.0 Apache-2.0 MIT GPL-2.0 BSD-3-Clause MIT MIT

Rust

astral-sh/ruff rust-lang/cargo openai/codex qdrant/qdrant tracel-ai/burn

MIT MIT/Apache-2.0 Apache-2.0 Apache-2.0 MIT/Apache-2.0

TypeScript

angular/angular ant-design/ant-design

MIT MIT

The 70 repositories span a broad range of open-source licenses—predominantly Apache-2.0 and MIT, with representation from GPL, BSD, and AGPL—reflecting the diversity of the open-source ecosystem. Repository concentration varies by language: Go draws from 16 distinct repositories yielding 23 instances, while TypeScript’s 28 instances come from only 2 repositories, with Angular alone contributing 25. This skew is a natural consequence of selecting large, actively maintained projects with substantial refactoring activity.

A.1

Per-language summary Language

#Repos

#Inst.

Avg. #Files

Avg. LOC

Avg. #Non-test

C C++ Go Java Python Rust TypeScript

9 9 16 11 18 5 2

20 22 23 26 29 22 28

17.9 21.4 16.0 20.8 10.6 14.5 11.9

424.1 196.3 227.4 309.8 299.8 284.8 122.6

15.2 16.0 9.4 16.8 7.0 11.0 7.5

Overall

170

15.9

261.6

11.4

Table 5 Per-language statistics for SWE-Bench ProMax, including number of repositories, instances, and average patch complexity. Java and C++ exhibit markedly larger patches, averaging 20.8 and 21.4 modified files per instance respectively, with C having the highest average LOC (424.1)—reflecting the cross-cutting nature of refactoring in codebases with deep type hierarchies and extensive header dependencies. In contrast, higher-level languages such as Python, Rust, and TypeScript average 10.6–14.5 files per instance, yet still require non-trivial multi-file coordination that distinguishes SWE-Bench ProMax from single-file benchmarks.

Task category analysis

Real-world refactoring commits rarely involve a single type of change. To characterize the diversity of skills required by SWE-Bench ProMax, we use Claude Sonnet 4.6 to perform multi-label classification of each instance into ten predefined categories: API Interface Change, Refactoring Cleanup, Bug Fix, New Feature, Documentation, Error Handling, Performance Optimization, Dependency Integration, Test Improvement, and Security Patch.

TASK CATEGORY ANALYSIS

Refactoring Cleanup Api Interface Change 74 (43.5%) New Feature 70 (41.2%) Bug Fix 36 (21.2%) Documentation Nl 25 (14.7%) Error Handling 14 (8.2%) Dependency Integration 10 (5.9%) Performance Optimization Test Improvement 9 (5.3%) Security Patch 2 (1.2%) 0

113 (66.5%) 111 (65.3%)

% of instances (multi-label, sums > 100%)

Figure 6 Distribution of task categories across SWE-Bench ProMax instances (multi-label; percentages sum to more than 100%). The dominant categories—Refactoring Cleanup (66.5%) and API Interface Change (65.3%)—confirm the refactoring focus of the benchmark, while the substantial presence of New Feature (43.5%) and Bug Fix (41.2%) reflects the multi-faceted nature of real-world code restructuring. Figure 6 shows that while refactoring-related categories dominate—Refactoring Cleanup appears in 66.5% of instances and API Interface Change in 65.3%—a substantial fraction of instances simultaneously involve new features (43.5%), bug fixes (41.2%), or documentation updates (21.2%). As Figure 7 illustrates, 46.5% of instances span three or more categories, with no instance involving fewer than two. The co-occurrence matrix (Figure 8) reveals strong coupling between API changes and refactoring cleanup (79 co-occurrences), as well as between refactoring and bug fixing (48 co-occurrences)—patterns consistent with the empirical finding that restructuring code frequently exposes latent defects (Nyirongo et al., 2025). This multi-faceted nature distinguishes SWE-Bench ProMax from benchmarks that test isolated skills and better reflects the compound challenges that developers face in practice. The high prevalence of Bug Fix (41.2%) is particularly noteworthy. Empirical studies of software maintenance have long observed that refactoring and bug fixing are deeply intertwined: restructuring code frequently exposes latent defects that were masked by the original design, and developers routinely address these defects within the same commit rather than deferring them to a separate change (Nyirongo et al., 2025). This coupling means that an agent attempting to resolve a refactoring task in SWE-Bench ProMax must not only apply the intended structural transformation but also recognize and correctly fix any bugs that surface during the process (Shi et al., 2024a)—a compound challenge that synthetic benchmarks, which typically isolate refactoring from bug fixing, cannot reproduce. Similarly, the presence of New Feature (43.5%) reflects cases where refactoring serves as a prerequisite for introducing new capabilities: the structural improvement enables or unblocks a feature addition that is delivered in the same commit. These overlapping concerns make SWE-Bench ProMax tasks substantially harder than they would be if each category were tested in isolation. Required skills. Complementing the task-type analysis above, we also classify each instance by the reasoning skills required for successful resolution. Figure 9 shows the distribution. The near-universal prevalence of cross-file reasoning (99.4%) and API semantics (98.8%) validates the benchmark’s design goal of testing agents’ ability to coordinate changes across file boundaries. The high frequency of pattern matching (91.8%) reflects that refactoring tasks often require identifying and systematically transforming recurring code patterns throughout a codebase. Domain knowledge (79.4%) and type system reasoning (50.6%) are also common, indicating that many instances demand understanding of project-specific conventions or language-specific type constraints beyond generic code manipulation.

REPRESENTATIVE INSTANCES

C.1

% of instances

3 0

# categories per instance

Figure 7 Number of categories per instance. Every instance involves at least two categories, and nearly half (46.5%) involve three or more simultaneously, indicating that SWE-Bench ProMax tasks require holistic software engineering skills rather than isolated refactoring ability.

Lang.

Repository

C++ Java C Rust Go Python TS

nasa/fprime plantuml/plantuml betaflight/betaflight tracel-ai/burn OpenListTeam/OpenList google/langextract ant-design/ant-design

Files

LOC

Refactoring summary

244 94 62 49 47 30 27

559 1,629 846 1,084 608 1,960 97

Unify header includes across framework Add hour-level time resolution to Gantt engine Rename motor protocol configuration fields Unify scalar arguments across tensor operations Refactor upload stream buffering across drivers Centralize provider output-format handling Unify destroyOnHidden across components

Table 6 Summary of representative instances in SWE-Bench ProMax, one per language, sorted by number of modified files.

Representative instances

We present one representative instance per language to illustrate the scale and diversity of SWE-Bench ProMax. For each instance we show the repository, number of modified files and lines of code, the commit URL for full inspection, an abbreviated file tree, and a summary of the rewritten problem statement. Table 6 provides an overview.

C.1

C++: nasa/fprime (244 files, 559 LOC)

Commit: https://github.com/nasa/fprime/pull/3422 File tree (excerpt): nasa/fprime (244 files, +591/-514) |-- Autocoders/Python/src/.../component/cpp.tmpl |-- Autocoders/Python/src/.../impl/cpp.tmpl |-- config/FpConfig.fpp |-- config/FpConfig.h |-- Fw/FPrimeBasicTypes.h [NEW] |-- Fw/FPrimeBasicTypes.hpp [NEW] |-- cmake/platform/unix/Platform/... [NEW] |-- Fw/Types/BasicTypes.h |-- Drv/BlockDriver/BlockDriverImpl.cpp |-- Svc/ActiveLogger/ActiveLoggerImpl.cpp |-- ... (234 more files) Problem statement (first paragraph):

REPRESENTATIVE INSTANCES

C.2

Refactoring Cleanup 113 79 28 48 16 11

Api Interface Change 79 111 45 33 18

New Feature 28 45 74 17 23

Bug Fix 48 33 17 70

Documentation Nl 16 18 23 Error Handling 11

Dependency Integration 8

Performance Optimization 6

Test Improvement 5

Security Patch 1

100 80 60

2 40 20 0

fa c Ap tori i In ng ter Cle fa c a n e C up Ne ha n w ge Fe atu Do cu Bu re me g nt Fi De pe Err atio x P e nd o r n N rf o e nc Ha l rm y I ndl an nt ing ce eg O r Tes ptim ation t Im iz pro ation Se vem cu e rit nt yP atc h

Figure 8 Co-occurrence matrix of task categories. API Interface Change and Refactoring Cleanup co-occur in 79 instances, while Bug Fix frequently accompanies both (33 and 48 instances respectively), reflecting how refactoring often surfaces latent defects that must be addressed concurrently. During routine development and maintenance of the F’Prime framework, teams have observed growing friction around header organization and build dependencies. The monolithic FpConfig.hpp header has accumulated responsibilities spanning basic fixed-width types (like I32, U64), platform-specific type configurations, project-level aliases, framework constants, and build-time switches. This conflation of concerns creates tangible workflow disruptions: any modification—even a minor adjustment to a single alias or configuration value—triggers near-total recompilation across the entire framework and dependent projects, significantly extending build cycles during iterative development. [. . . ]

C.2

Java: plantuml/plantuml (94 files, 1,629 LOC)

Commit: https://github.com/plantuml/plantuml/commit/c910f8b File tree (excerpt): plantuml/plantuml (94 files, +1550/-2212) |-- src/.../chronology/ChronologyDiagram.java |-- src/.../chronology/ComplementHour.java |-- src/.../chronology/HourPattern.java |-- src/.../project/ConstantPlan.java |-- src/.../project/GanttDiagram.java |-- src/.../project/Load.java |-- src/.../project/OpenClose.java |-- src/.../project/time3/Day.java |-- ... (86 more files) Problem statement (first paragraph):

REPRESENTATIVE INSTANCES

C.3

cross_file_reasoning api_semantics interface_contract pattern_matching data_flow domain_knowledge control_flow type_system 15 (8.8%) protocol_understanding security_reasoning 10 (5.9%) memory_management 8 (4.7%) concurrency 8 (4.7%) error_handling 2 (1.2%) 0

93 (54.7%) 86 (50.6%)

% of instances

169 (99.4%) 168 (98.8%) 165 (97.1%) 156 (91.8%) 151 (88.8%) 135 (79.4%)

100

Figure 9 Distribution of required reasoning skills across SWE-Bench ProMax instances (multi-label). Nearly all instances require cross-file reasoning (99.4%) and API semantics understanding (98.8%), confirming that the benchmark systematically tests sustained multi-file comprehension. Pattern matching (91.8%) and interface contract reasoning (97.1%) are also near-universal, while data flow (88.8%), domain knowledge (79.4%) and type system reasoning (50.6%) appear in the majority of instances. The Gantt diagram engine in PlantUML has historically operated exclusively at day-level resolution, treating each calendar day as an indivisible atomic unit for all scheduling calculations. While sufficient for high-level project visualization, this architectural constraint increasingly limits the system’s ability to support evolving user requirements involving finer temporal precision. Users attempting to model scenarios such as tasks spanning partial days (e.g., “9 AM to 3 PM”), resources with intra-day availability patterns, or dependencies requiring hour-level alignment encounter fundamental limitations. The current implementation forces approximations—like splitting single-day efforts across multiple artificial days—which introduce inaccuracies in duration calculations, resource load reporting, and constraint validation. These approximations become especially problematic when integrating with external tools that export time data with sub-day precision or when users require precise effort tracking across non-standard work intervals. [. . . ]

C.3

C: betaflight/betaflight (62 files, 846 LOC)

REPRESENTATIVE INSTANCES

|-- src/platform/common/stm32/... |-- ... (52 more files)

C.5

[NEW]

Problem statement (first paragraph): The motor configuration subsystem in the motorDevConfig_t structure uses field names that inaccurately imply PWM-specific scope for settings that apply across all motor protocol families. Specifically, the field motorPwmProtocol stores the motor protocol type for all protocols—including digital ones like Dshot and ProShot—yet its name suggests it is limited to PWM. Similarly, motorPwmInversion controls signal inversion for any motor output regardless of protocol family, and useUnsyncedPwm governs whether motor updates run continuously or are synchronized to the PID loop, a concept that is not inherently PWM-specific. These misleading names create cognitive friction during development and code review, and they cause confusion when configuring digital protocols where “PWM” terminology does not apply. [. . . ]

C.4

Rust: tracel-ai/burn (49 files, 1,084 LOC)

Commit: https://github.com/tracel-ai/burn/pull/4337 File tree (excerpt): tracel-ai/burn (49 files, +1010/-945) |-- crates/burn-autodiff/src/ops/int_tensor.rs |-- crates/burn-autodiff/src/ops/tensor.rs |-- crates/burn-backend/src/backend/ops/tensor.rs |-- crates/burn-backend/src/backend/ops/int_tensor.rs |-- crates/burn-backend/src/element/mod.rs |-- crates/burn-backend/src/element/scalar.rs [NEW] |-- crates/burn-candle/src/ops/tensor.rs |-- crates/burn-fusion/src/ops/int_tensor.rs |-- crates/burn-router/src/ops/int_tensor.rs |-- ... (40 more files) Problem statement (first paragraph): In the Burn deep learning framework, scalar arguments to tensor operations such as add_scalar, clamp_min, equal_elem, mask_fill, powf_scalar, and similar functions currently use backend-specific element types (e.g., FloatElem<B>, IntElem<B>) or raw primitives (like f32, i32). This creates pervasive type-mismatch friction across the codebase: autodiff checkpointing must store and replay scalar values without a common runtime representation, fusion engines must serialize scalars into IR without knowing the originating tensor’s concrete element type, and cross-backend routers must forward scalar literals through layers where the backend type is erased. Every new scalar-accepting operation requires duplicating conversion boilerplate (.elem() calls, explicit casts) across autodiff, fusion, router, and backend implementation files. [. . . ]

C.5

Go: OpenListTeam/OpenList (47 files, 608 LOC)

REPRESENTATIVE INSTANCES

|-- pkg/buffer/bytes_test.go |-- ... (37 more files)

C.7

[NEW]

Problem statement (first paragraph): During file upload operations across multiple storage drivers—including 115 Cloud, 123 Cloud, Google Drive, OneDrive, and others—the application encounters critical reliability issues rooted in stream handling, caching behavior, and underlying buffer management. These problems affect several distinct but interrelated subsystems. [. . . ]

C.6

Python: google/langextract (30 files, 1,960 LOC)

Commit: https://github.com/google/langextract/pull/239 File tree (excerpt): google/langextract (30 files, +2455/-619) |-- examples/ollama/Dockerfile |-- examples/ollama/demo_ollama.py [NEW] |-- examples/ollama/quickstart.py |-- langextract/annotation.py |-- langextract/core/base_model.py |-- langextract/core/format_handler.py [NEW] |-- langextract/core/schema.py |-- langextract/extraction.py |-- langextract/resolver.py |-- tests/format_handler_test.py [NEW] |-- ... (20 more files) Problem statement (first paragraph): The library supports extraction workflows across multiple language model providers—such as Ollama, Gemini, and others—each with distinct expectations for output structure. These include variations in serialization format (JSON vs. YAML), the presence of Markdown fence markers (e.g., json blocks), wrapper keys like “extractions”, and attribute naming conventions. Currently, the logic governing these format decisions is distributed across prompt generation, model configuration, resolver parsing, and provider-specific implementations. This fragmentation creates subtle but persistent challenges during integration and maintenance. [. . . ]

C.7

TypeScript: ant-design/ant-design (27 files, 97 LOC)

Commit: https://github.com/ant-design/ant-design/pull/53739 File tree (excerpt): ant-design/ant-design (27 files, +132/-48) |-- .dumi/theme/builtins/ComponentTokenTable/index.tsx |-- .dumi/theme/common/ComponentChangelog/ComponentChangelog.tsx |-- components/avatar/AvatarGroup.tsx |-- components/collapse/Collapse.tsx |-- components/drawer/index.tsx |-- components/dropdown/dropdown.tsx |-- components/image/index.tsx |-- components/modal/Modal.tsx |-- components/tabs/index.tsx |-- components/tooltip/index.tsx |-- ... (17 more files) Problem statement (first paragraph):

REPRESENTATIVE INSTANCES

C.7

In the Ant Design component library, multiple components—including Modal, Drawer, Collapse, Tabs, Tooltip, Dropdown, and Image—provide functionality to unmount internal content when the component becomes hidden from view. This capability is critical for optimizing performance in complex applications, particularly when managing resource-intensive child elements or preserving clean component state between interactions. However, the current implementation suffers from significant API inconsistency across the component suite. [. . . ]

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

Analyze & explore related research in Conceptio

Related documents