arXiv:2606.25530v1 [cs.SE] 24 Jun 2026
Evaluating LLMs on Real-World Software Performance Optimization
Ezgi Sarıkayak Siemens AG Technical University of Munich Munich, Germany [email protected]
Wenchao Gu Technical University of Munich Heilbronn, Germany [email protected]
Hesham Ghonim Technical University of Munich Munich, Germany [email protected]
Chunyang Chen Technical University of Munich Heilbronn, Germany [email protected]
Abstract Software performance optimization is a notoriously complex and manual task. Despite the growing use of Large Language Models (LLMs) for code refinement, we still lack benchmarks that capture how optimization actually happens in real-world codebases. Existing frameworks often oversimplify the problem by focusing on isolated functions or a single performance metric, missing the critical trade-offs between execution time and memory footprint, the inherent noise of the measurement environment, and the variability introduced by different input data and execution conditions. We address this by introducing SWE-Pro, a repository-level benchmark derived from 102 expert-written optimizations from open-source projects. Unlike previous benchmarks, SWE-Pro pairs each task with parameterized tests to evaluate runtime, peak memory, and Time-Weighted Memory Usage (TWMU) across varying input data and execution conditions under noise-aware measurement conditions. Our evaluation shows that current LLMs struggle significantly: runtime gains are negligible, and memory optimizations are nearly non-existent. This stands in sharp contrast to expert implementations, which achieve an aggregate speedup of 15.5× and peak memory reduction of 171.3× over benchmark tasks. Expert-written improvements are observed in 91.2% of tasks for runtime and 65.7% for peak memory. Our findings expose a substantial gap between current LLM capabilities and the demands of expert-level engineering.
1
Introduction
In recent years, LLMs have fundamentally reshaped software engineering, demonstrating remarkable proficiency in domains such as code generation [13] and automated bug fixing [42]. As these models increasingly automate large-scale software production, the evaluative focus is pivoting from basic functional correctness toward long-term sustainability and execution efficiency. In this AI-driven development era, sub-optimal code can rapidly aggregate into "performance debt," significantly inflating computational overhead [15]. Consequently, this technological shift has catalyzed interest in LLM-based code performance optimization—the critical task of refining implementations to enhance runtime and memory efficiency without altering functional semantics. Despite its importance, performance optimization remains notoriously challenging [25]. Unlike binary correctness, which provides deterministic pass/fail signals, optimization is inherently multidiPreprint.
mensional and context-sensitive. A single optimization may involve conflicting trade-offs between runtime and memory usage, and its efficacy often fluctuates across different execution environments [33]. Moreover, performance is inextricably linked to workload characteristics; measurements are dynamic, context-dependent observations rather than static attributes [41]. Given the increasing deployment of LLMs for such tasks, a rigorous benchmark is essential to evaluate their ability to navigate the complexities of real-world software performance. Current evaluation paradigms have laid a foundational groundwork but face scalability and precision issues when transitioning from function-level to repository-level analysis. Early benchmarks like EffiBench [10] and COFFE [28] assess individual functions in isolation, often neglecting the complex dependencies inherent in real-world repositories. While recent repository-level benchmarks such as SWE-Perf [1], SWE-FFICIENCY [20], and GSO [35] operate at scale, they often lack sufficient evaluative rigor. Common pitfalls include using unit tests as a crude proxy for performance [1] or aggregating heterogeneous execution variants into coarse-grained scores [35], which can obscure input-specific performance fluctuations. Such approaches offer a restricted view of how optimizations generalize across varying workloads (e.g., different input sizes, data properties, functional options) [23] and typically lack the statistical rigor needed to distinguish genuine performance gains from environmental noise. These limitations underscore a fundamental gap: existing benchmarks focus on task success under fixed conditions, failing to capture the input-dependent, statistically grounded, and multi-metric nature of modern performance optimization. This deficit motivates our investigation into whether LLMs can truly achieve robust repository-level optimization when subjected to rigorous, multi-workload scrutiny. To address this, we present SWE-Pro, a comprehensive repository-level benchmark comprising 102 expert-written optimization cases curated from real-world open-source projects. SWE-Pro is designed to shift the evaluation paradigm toward robust, workload-aware analysis. Unlike prior benchmarks, SWE-Pro evaluates tasks across diverse workloads via parameterized performance tests that reflect realistic execution scenarios. By conducting systematic parameter sweeps over input scales, data characteristics and function properties, our framework exposes performance behaviors that remain invisible to single-input benchmarks. To ensure reliability, we introduce an adaptive measurement mechanism: rather than relying on fixed-count profiling, SWE-Pro dynamically determines the necessary iterations based on the statistical convergence of the 90% Relative Confidence Interval Width (RCIW). This effectively filters out system variance, enabling the reproduction of 87% of expert-level improvements. Furthermore, we employ a multi-metric framework—incorporating peak memory and TWMU alongside runtime—to uncover critical time–memory trade-offs, providing a nuanced perspective on LLM-generated optimizations in resource-constrained contexts. Through extensive experiments on state-of-the-art LLMs, we provide the first systematic study of repository-level code optimization that jointly evaluates runtime and memory performance across varying workloads under noise-aware conditions. Our findings reveal that while modern models excel at local logic refinements, they struggle significantly to deliver measurable efficiency gains; their runtime improvements remain negligible, and memory optimizations are nearly non-existent. This stands in sharp contrast to expert implementations within our benchmark, which achieve 15.48× runtime speedup, 171.31× peak memory reduction, and 619.22× TWMU reduction, with 91.2% of tasks showing reproducible runtime improvements, 65.7% showing reliable peak memory gains, and 52.0% showing TWMU improvements. Such a substantial performance gap suggests clear trajectories for future research in developing LLMs that can navigate the demands of expert-level engineering. To support the community, We open-sourced SWE-Pro and our automated evaluation pipeline at https://github.com/probench-swe/SWE-Pro.
2
2
PRO-Bench
Table 1: Comparison of repository-level performance optimization benchmarks. Performance Scope: how performance is evaluated per task. Structured Parameter Sweep: whether test cases are generated via structured sweeps over input dimensions. Runtime and Memory: performance metrics considered. Adaptive Iterations: whether measurement iterations are dynamically determined based on statistical convergence. Structured Parameter Sweep
Runtime
Memory
Adaptive Iterations
Multiple Scripts Multiple Unit Tests Single Workload
✗ ✗ ✗
✓ ✓ ✓
✗ ✗ ✗
✗ ✗ ✗
Multiple Workloads
✓
✓
✓
✓
Benchmark
Performance Scope
GSO SWE-Perf SWE-FFICIENCY PRO-Bench
2.1
Preliminaries
Formulation. Repository-level performance optimization benchmarks task an agent with producing a patch to a given codebase that improves runtime efficiency while preserving functional correctness. SWE-Perf [1] grounds tasks in performance-improving Pull Requests (PRs), pairing each instance with the full repository, a set of performance-related unit tests, and an expert-authored patch; evaluation reports patch applicability, functional correctness, and runtime improvement. Building on a similar construction, SWE-fficiency [20] provides each agent with a complete codebase and a single fixed workload, requiring the agent to localize bottlenecks, edit the repository, and pass existing correctness tests; performance is quantified via the speedup ratio against a gold expert run on that workload configuration. In contrast, GSO [35] constructs multiple performance tests per task from real commit histories, where each test is an experiment function that exercises the target code path across a range of input patterns and execution variants and is timed as a unit; performance is then evaluated via the OPT@K metric, measuring whether at least one of the top-K generated patches matches or exceeds the expert runtime. Limitations. Current benchmarks are predominantly limited to runtime evaluation, entirely neglecting memory usage in their performance formulations. Furthermore, assessment under fixed or coarsely aggregated conditions significantly constrains the depth of analysis: SWE-Efficiency [20] relies on a single workload per task, precluding any assessment of patch robustness across varying inputs; GSO [35] conflates heterogeneous input variants into a single aggregate score, which obscures the origin of speedups and masks potential regressions at specific scales; and SWE-Perf [1] employs pre-existing unit tests that were never designed to capture performance sensitivity. A systemic deficiency across these platforms is the absence of input sweeps and adaptive profiling, leaving results susceptible to measurement noise and preventing the identification of solutions that might fail under alternative configurations. To address these shortcomings, SWE-Pro evaluates tasks across multiple structured workloads while jointly tracking runtime and memory consumption. It ensures measurement reliability through adaptive profiling, where the number of iterations is dynamically determined based on RCIW convergence. Moreover, SWE-Pro only accepts performance effects that exceed a noise-aware threshold, derived from both within-container variation and cross-container variation across repeated runs. The comprehensive advantages of SWE-Pro relative to existing benchmarks are summarized in Table 1. 2.2
Collection
Repository selection. We focus on data-intensive Python libraries where performance optimizations are both frequent and measurable. To systematically identify optimization scenarios, we restrict our selection to repositories that maintain explicit performance-related labels on PRs, enabling reliable extraction of developer-identified optimization tasks. Based on these criteria, we select pandas, scikit-learn, and xarray as target repositories. These projects are widely used, actively maintained, and provide comprehensive unit test suites that enable 3
reliable correctness validation. Further dataset statistics and repository-specific details are provided in Appendix A. PR filtering. We collect merged PRs via the GitHub API and retain only those that (i) are explicitly labeled as performance-related, (ii) are merged into the main branch, (iii) involve modifications to Python source files, and (iv) include evidence of developer acknowledgment, such as entries in changelogs or whatsnew documentation. Requiring changelog or whatsnew entries acts as a proxy for significance, filtering out minor or incidental changes and retaining optimizations that are explicitly acknowledged by developers. Environment setup. For each PRs, we reconstruct the baseline and optimized codebase states and execute them within isolated Docker environments. Each environment is built using repositoryspecific dependencies corresponding to the target commit, including development requirements to ensure compatibility with the test suite. The setup process requires careful inspection of repository documentation and iterative refinement to ensure that the codebase can be built and executed reliably. Using Docker ensures reproducibility and consistent execution across workloads. Detailed Docker configurations and reproducibility settings are provided in Appendix E.6. Unit Test Selection and Parameterized Performance Test Construction. For each PRs, we identify unit tests associated with the target function and retain the full set to ensure comprehensive coverage of its functionality during evaluation. Based on PRs descriptions, code changes, and developer discussions, we construct parametrized performance test scenarios that capture the optimized operation. Each scenario defines an execution entry point and a parameterized space designed to trigger the performance behavior introduced by the PR. Detailed scenario design, parameter selection, and examples are provided in Appendix B. Validation. All instances are manually validated to remove inconsistent, non-reproducible, or unstable cases. We exclude PRs with mixed modifications where the repository cannot be reliably built or tested in the reconstructed environment, ensuring reproducible tasks.
2.3
Features
Targeted scenario construction for optimization impact. Tasks are derived from performanceimproving PRs, grounding each instance in developer-identified optimization problems. Task descriptions are derived from PR metadata, discussions, and code updates to define the optimization target and associated tests. They exclude hints and specify the target function and relevant input configurations. Parameterized performance tests. Each instance is evaluated under a parameterized setup defined by performance-relevant dimensions. We adopt a three-level abstraction: (i) parameters, which define performance-relevant dimensions, (ii) parameter options, which specify concrete values for each dimension, and (iii) execution configurations, which specify how the performance-improving scenario is exercised with each combination in the parameter grid. This structure separates the definition of performance dimensions from their execution, enabling systematic and reproducible evaluation across diverse inputs. It allows analysis of how optimizations generalize and whether improvements hold across workloads or degrade under specific inputs. 4
Phase III: Evaluation
Phase I: Calibration
Workload
Calibrated Parameters
Repository
Calibrated Parameters Repository
Phase II: LLM-based Optimization
Evaluation Results
LLM Task Description
Repository
LLM Patch
Human Patch
Patch
Figure 2: SWE-Pro performance measurement framework. Table 2: Median characteristics of SWE-Pro task instances. Category
Attribute
Median
Codebase
# Python lines (non-test) # Python files (non-test)
197K 383
Gold Patch
# Lines edited # Files edited
40 3
# Unit tests
10
# Parameters # Execution configs.
2 6
Correctness Tests Performance Tests
pandas 79 12
scikit-learn xarray
11 Figure 1: Distribution of SWEPro instances across repositories.
Dataset Distribution. The final dataset consists of 102 validated benchmark instances collected across three repositories. Figure 1 shows the distribution across pandas, scikit-learn, and xarray, with pandas comprising the majority of instances. This reflects the availability of performance-labeled PRs in open-source repositories rather than a deliberate selection bias. Table 2 summarizes the median characteristics of each benchmark instance, including the size of the codebase, the size of the gold patch, and the structure of both correctness and parameterized performance tests. To validate that each instance represents a genuine optimization opportunity, we evaluate gold solutions against their baselines using relative performance improvements in runtime, memory, and TWMU. We report only performance effects that pass Signal-to-Noise Ratio (SNR) threshold, ensuring that observed differences are statistically distinguishable from measurement noise. Gold solutions achieve average runtime improvement of 15.48× (93.1% of tasks), a peak memory improvement of 171.31× (70.6% of tasks), and a TWMU improvement of 619.22× (55.9% of tasks). 2.4
Reliable Performance Measurement Framework
Reliable performance measurement requires separating genuine optimization effects from pervasive environmental noise. We implement a hierarchical measurement framework to ensure that observed improvements in SWE-Pro are statistically significant and reproducible. For a more detailed discussion of our measurement methodology, see Appendix E. Hierarchical Noise Modeling. Performance measurements in containerized environments exhibit two independent sources of variance [16]: within-container noise, arising from scheduler jitter and interpreter behavior, and across-container noise, arising from CPU state and memory layout variations. SWE-Pro addresses this by executing workloads across multiple fresh container instances, each performing adaptive measurement iterations inside. 5
The Measurement Pipeline. As illustrated in Figure 2, our pipeline consists of four integrated stages: • Calibration: Performed once per workload to ensure comparability. For runtime, it determines the number of function invocations per iteration to overcome timer resolution. For memory, it sets the sampling interval required to capture allocation behavior with sufficient temporal resolution. • Warmup: To mitigate cold-start effects (e.g., cache initialization, lazy imports), we use a sliding-window convergence test based on the Coefficient of Variation (CV) of execution times. Measurement begins only after this criterion is met or a predefined limit is reached. • Adaptive Sampling: Unlike fixed-iteration benchmarks, we collect iterations until the RCIW for all metrics falls below a threshold (RCIW0.9 < ϵ). This ensures that the uncertainty of the mean estimate is statistically bounded. • Validation: We apply two criteria—stability (via RCIW) and detectability (via SNR with a threshold of 2). Only measurements satisfying both are admitted for final reporting. Multi-dimensional Efficiency Metrics. Beyond runtime, SWE-Pro tracks memory consumption through two lenses: Peak Memory Usage above baseline and TWMU. Let m(t) denote the sampled memory at time t, mbase the baseline footprint, and T the duration. We define TWMU as: Z 1 T m̄twmu = max(0, m(t) − mbase )dt (1) T 0 This quantity is approximated via trapezoidal integration. By normalizing for execution time, TWMU provides a stable estimate of sustained memory pressure, preventing transient spikes from misrepresenting overall resource efficiency.
3
Evaluating on SWE-Pro
3.1
Experimental Setup
Retrieval. We evaluate two context provision strategies: oracle-based context and BM25-based retrieval. The oracle strategy provides ground-truth relevant files extracted directly from the target patch, while BM25 approximates this context using lexical similarity over the repository state. This setup follows the methodology introduced in SWE-bench [14]. The prompt templates used for both strategies are provided in Appendix D. The oracle setting serves as an upper bound on retrieval quality, isolating the effect of context availability from model capability. Consequently, the performance gap between oracle and BM25-based retrieval reflects the limitations of retrieval. Implementation details for BM25 are provided in Appendix C. Models. We evaluate six large language models from multiple providers: GPT-5.2 [26] (OpenAI), Claude Sonnet 4.6 [2] (Anthropic), Kimi K2.5 [22] (Moonshot AI), Gemini 3.1 Flash-Lite [9] (Google), GLM-5.1 [40] (Z.ai), and MiniMax M2.7 [21] (MiniMax). Evaluation metrics. We evaluate model performance along three dimensions: (i) patch generation success, and (ii) functional correctness, and (iii) performance. Patch success. Patch success measures whether a generated patch can be applied without error. For a set of tasks T , it is defined as 1 X Patch@✓ = I[patch applied(t)]. |T | t∈T
Correctness. Correctness is evaluated using the associated test suite. A patch is considered correct if all relevant tests pass after application: 1 X Test@✓ = I[tests pass(t)]. |T | t∈T
Performance metrics. Performance is evaluated only on tasks for which the patch applies successfully and passes all correctness tests. We report three metrics: runtime, peak memory usage, and TWMU, 6
capturing execution speed, peak memory footprint, and total memory pressure over time respectively. Each metric quantifies a resource cost, where a reduction in the optimized version relative to the baseline constitutes an performance improvement. We express this relationship through the Improvement Factor (IF), defined identically for all three metrics. Only workloads passing the SNR criterion are retained, ensuring reported effects are statistically distinguishable from measurement noise. For each PR, workload-level IF values are aggregated using the harmonic mean [12] to obtain a single PR-level score. We then report the arithmetic mean of these PR-level scores across all PRs: 1 X baselinei IFLLM = H , LLMi i∈WPR |P| PR∈P
where WPR denotes the set of workloads associated with a given PR, and P denotes the set of PRs whose aggregated improvement satisfies both the stability criterion based on RCIW and the minimum SNR threshold. The operator H denotes the harmonic mean used to aggregate workloadlevel improvement factors into a single PR-level value. A value of IFLLM > 1 indicates an overall performance improvement across workloads, whereas IFLLM < 1 indicates that regressions outweigh improvements. Further details on SNR-based filtering and results aggregation are provided in Appendix E.4 and E.5.
4
Results
Producing reliable performance optimizations requires LLMs to satisfy three key conditions: generating patches that can be applied successfully, preserving functional correctness, and producing measurable performance improvements that survive SNR filtering. Figure 3 shows the proportion of generated patches that satisfy each condition for different models, highlighting where optimization attempts fail. Table 3 further reports two complementary measures under both Oracle and BM25 retrieval settings: the pass rate, representing the proportion of tasks with detectable performance effects, and the IF, measuring the magnitude of improvement on those tasks. Gold Solution MiniMax M2.7
55%
GLM-5.1
19% 52%
70%
Gemini 3.1 Flash
GPT-5.2
22%
41%
Kimi K2.5
Claude Sonnet 4.6
Patch Failed Correctness Failed No Signal Regressed Improved
91%
12%
43% 10%
37%
18%
15% 13%
71%
16%
75%
Figure 3: Task progression through the evaluation pipeline for each model under Oracle retrieval. Patch application and correctness failures are independent of performance evaluation. The remaining segments show runtime outcomes, the only metric where LLMs can achieve measurable effects. The Gold solution is included as reference. LLMs can generate syntactically correct optimized code but struggle to maintain functional consistency. Under Oracle retrieval, patch application rates range from 30.4% to 97.1% across models, while correctness rates are ranging from 18.6% to 79.4%. This gap shows that LLMs frequently produce syntactically applicable patches that nonetheless fail to preserve functional behavior. Under BM25 retrieval, both rates drop further, reflecting the sensitivity of patch quality to retrieval context. Full patch and correctness results under both retrieval settings are provided in Table 8 in the Appendix. Even LLM-generated patches can pass correctness checks, most of them fail to produce signal beyond noise. Even among tasks that pass both patch and correctness gates, the no-signal segment 7
Table 3: Pass rate and magnitude of detectable performance effects under Oracle and BM25 retrieval. Each cell reports the percentage of all tasks passing the SNR filter and the corresponding IF. — denotes that no PR exhibited an SNR effect above the detectability threshold. ∗ BM25 retrieval is not applicable to the Gold solution by design. Oracle Model
BM25
Runtime
Peak Mem.
TWMU
Runtime
Peak Mem.
TWMU
GPT-5.2
Pass Rate IF
3.9% 0.69×
— —
1.0% 0.85×
— —
— —
— —
Claude Sonnet 4.6
Pass Rate IF
2.0% 6.67×
1.0% 16.61×
2.0% 28.82×
12.7% 2.28×
1.0% 16.62×
2.0% 3556.03×
Gemini 3.1 Flash
Pass Rate IF
12.7% 1.27×
— —
— —
13.7% 1.19×
— —
— —
Kimi K2.5
Pass Rate IF
3.9% 1.18×
— —
— —
2.9% 1.20×
— —
— —
GLM-5.1
Pass Rate IF
3.9% 1.16×
— —
— —
5.9% 1.27×
— —
— —
MiniMax M2.7
Pass Rate IF
4.9% 1.14×
— —
1.0% 1.11×
2.0% 1.23×
— —
— —
Pass Rate IF
93.1% 15.48×
70.6% 171.31×
55.9% 619.22×
∗
∗
∗
∗
∗
∗
Gold solution
dominates across all models in Figure 3. The majority of patches that preserve correctness produce no performance change that passes the SNR filter, meaning the observed differences cannot be distinguished from measurement noise. This holds across all three metrics and both retrieval settings. The Gold solution, in contrast, shows that 93.1% of tasks admit measurable and reliable runtime improvements, confirming that the benchmark contains substantial optimizable structure that is not being captured by LLMs. Even LLM-generated patches can produce signal beyond noise, the gains are sparse and modest. Under Oracle retrieval, Gemini 3.1 Flash achieves the highest pass rate of tasks with detectable runtime effects (12.7%), while the corresponding IF is 1.27×, indicating modest improvements in magnitude. Claude Sonnet 4.6, in contrast, attains a lower runtime pass rate (2.0%) but achieves a larger IF of 6.67×, suggesting that successful optimizations are less frequent but occasionally more impactful. Under BM25, GPT-5.2 produces no reliable runtime improvements. Claude Sonnet 4.6 remains the strongest model under BM25 in terms of IF. In comparison, the Gold solution achieves a runtime IF of 15.48× across most tasks, substantially exceeding all LLM-based improvements in both scale and consistency. Sometimes LLM-generated patches even introduce performance regressions. GPT-5.2 achieves a runtime IF of 0.69× under Oracle retrieval, indicating that the generated patches more often degrade performance than improve it. A similar decline is observed for TWMU (0.85×). These results show that LLM-generated optimizations can introduce systematic regressions rather than measurable performance gains. 8
impr. factor
Runtime (s)
5.1×
5.1×
33.2×
32.0×
103×
99.9×
100k× 10k×
Peak memory (MiB) 16.5×
16.5×
16.7×
16.7×
16.6×
16.7×
TWMU (MiB) 3.0k×
2.7k×
19.0×
19.1×
460k×
469k×
5 5 5 10 10 10 str= 000, str= 00, str= 000, str= 00, str= 000, str= , 0 0 0 0 0 00 nt=1 nt=100 nt=1 nt=1 nt=100 nt=1
1k× 100× 10×
Figure 4: Impact of input configuration on optimization effectiveness for PR #7578 under Oracle retrieval. Per-workload IF varies with rolling size (nt) and stride (s), highlighting input-dependent performance behavior. Memory optimization remains largely beyond current LLM capabilities. Across models, peak memory and TWMU entries in Table 3 are almost entirely absent, indicating that consistent memory improvements are not achieved. The only notable exception is Claude Sonnet 4.6 achieving a peak memory IF of 16.61× and a TWMU IF of 28.82× under Oracle retrieval. Under BM25 retrieval, the same model attains an even higher TWMU IF of 3556.03×, surpassing the Oracle result. However, these improvements are entirely driven by this single instance and do not generalize across tasks. This suggests that while LLMs can achieve strong memory optimizations in isolated cases, such behavior is not yet reliable or systematic. The Gold solution, however, consistently achieves large improvements across all metrics, including peak memory (171.31×) and TWMU (619.22×), indicating that these gains are broadly attainable when guided by expert-level reasoning rather than model-generated heuristics. Figure 4 provides a detailed analysis of PR#7578, which is responsible for the large peak-memory and TWMU improvements observed in the results, by showing per-workload IF across varying input sizes for all three metrics. Runtime improvement increases substantially with input size, around 100× for large workloads. Peak-memory improvement remains relatively stable at approximately 16× across configurations, while TWMU exhibits the largest gains at scale, surpassing 460× for the largest workloads. This workload-dependent behavior highlights the importance of workload-relevant performance profiling, as optimization effects may only become visible under sufficiently large or stressed inputs. In contrast, the Gold solutions achieve improvements across most tasks, rather than relying on a single extreme case, resulting in broader and more consistent performance improvements overall. Detailed results for this PR and the corresponding patch are provided in Appendix E.7.1.
5
Related Work
LLMs for Code Optimization. Recent research has extensively explored LLMs for improving code efficiency through strategies such as performance-aware fine-tuning, iterative refinement, and searchbased optimization [36, 6, 27]. Despite these advancements, achieving robust generalization and, in particular, reliable performance evaluation remains an open challenge [8]. To assess optimization quality, existing evaluation frameworks have progressed across multiple levels of granularity, ranging from function-level benchmarks that abstract away system interactions (e.g., EffiBench, ENAMEL, ECCO, Mercury) [10, 31, 32, 38, 5], to program-level kernels that capture controlled computational patterns [28], and more recently to repository-level benchmarks such as GSO [35], SWE-Perf [1], and SWE-Efficiency [20], which incorporate real-world software systems. However, across all levels, evaluation is still predominantly based on execution time under fixed configurations, limiting the ability to capture realistic performance behavior. Performance Measurement and Multi-Metric Tradeoffs. This limitation is fundamentally rooted in the nature of performance itself. Performance optimization is inherently multidimensional: improvements in one metric, such as runtime, do not necessarily translate to gains in others, such 9
as memory usage or energy consumption, requiring careful navigation of tradeoffs in real-world systems [33, 37, 29, 11, 3]. Moreover, system behavior is tightly coupled with workload characteristics and execution context. Variations in input size, data distribution, and hardware conditions can shift bottlenecks and invalidate previously effective optimizations [23, 18, 39]. Reliable performance measurement further introduces challenges due to non-determinism from hardware effects, OS scheduling, and runtime compilation, making naive evaluation statistically unsound [24, 7, 16]. Even in controlled microbenchmark settings, optimization decisions must balance execution speed with result fidelity [17]. Consequently, performance cannot be adequately characterized by a single metric or a static configuration; instead, it must be evaluated across multiple resource dimensions and workload variations to reflect true system behavior [33, 23, 24, 3].
6
Limitations
SWE-Pro focuses on Python-level optimization in data-intensive libraries where workload behavior can be systematically varied through input parameters. The benchmark is restricted to repositories containing explicitly performance-labeled PRs, which ensures reproducibility but inherently limits coverage. As a result, findings may not generalize to other domains such as low-level systems software, I/O-bound applications, or non-Python execution environments. Extending SWE-Pro to support multi-language remains an important direction for future work.
7
Conclusion
We presented SWE-Pro, a repository-level benchmark for evaluating LLM capabilities in realworld performance optimization. SWE-Pro comprises 102 expert-authored optimizations with parameterized performance tests covering runtime, peak memory, and TWMU. Our results reveal a substantial gap between current LLMs and expert developers: reliable runtime improvements are rare, memory optimizations are largely absent, and the majority of generated patches fail to produce any measurable gain. The primary bottleneck lies not in the magnitude of improvements when they occur, but in reliably identifying optimization opportunities in the first place. Expert solutions, by contrast, achieve strong and consistent improvements across tasks, establishing a challenging upper bound for future systems.
References [1] Anonymous. SWE-perf: Can language models optimize code performance on real-world repositories? In Submitted to The Fourteenth International Conference on Learning Representations, 2025. under review. [2] Anthropic. Claude sonnet 4.6. https://www.anthropic.com/news/claude-sonnet-4-6, 2026. [3] Prasanna Balaprakash, Ananta Tiwari, and Stefan M. Wild. Multi objective optimization of hpc kernels for performance, power, and energy. In PMBS@SC, 2013. [4] Phillip Borman and David Elder. Q2(R1) Validation of Analytical Procedures, chapter 5, pages 127–166. John Wiley & Sons, Ltd, 2017. [5] Mingzhe Du, Luu Anh Tuan, Bin Ji, Qian Liu, and See-Kiong Ng. Mercury: a code efficiency benchmark for code large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA, 2024. Curran Associates Inc. [6] Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael R. Lyu. Search-Based LLMs for Code Optimization, page 578–590. IEEE Press, 2025. [7] Andy Georges, Dries Buytaert, and Lieven Eeckhout. Statistically rigorous java performance evaluation. In Proceedings of the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages and Applications, OOPSLA ’07, page 57–76, New York, NY, USA, 2007. Association for Computing Machinery. 10
[8] Jingzhi Gong, Vardan Voskanyan, Paul Brookes, Fan Wu, Wei Jie, Jie Xu, Rafail Giavrimis, Mike Basios, Leslie Kanthan, and Zheng Wang. Language models for code optimization: Survey, challenges and future directions, 2025. [9] Google DeepMind. gemini/, 2026.
Gemini 3.1 flash.
https://deepmind.google/technologies/
[10] Dong HUANG, Yuhao QING, Weiyi Shang, Heming Cui, and Jie Zhang. Effibench: Benchmarking the efficiency of automatically generated code. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [11] Md. Monzurul Amin Ifath and Israat Haque. Characterizing performance–energy trade-offs of large language models in multi-request workflows. Proc. ACM Meas. Anal. Comput. Syst., 10(1), March 2026. [12] Bruce Jacob and Trevor N. Mudge. Notes on calculating computer performance. Technical Report CSE-TR-231-95, University of Michigan, EECS Department, Advanced Computer Architecture Lab, 1995. Technical Report. [13] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ACM Trans. Softw. Eng. Methodol., 35(2), January 2026. [14] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. [15] Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. Understanding and detecting real-world performance bugs. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’12, page 77–88, New York, NY, USA, 2012. Association for Computing Machinery. [16] Tomas Kalibera and Richard Jones. Rigorous benchmarking in reasonable time. In Proceedings of the 2013 International Symposium on Memory Management, ISMM ’13, page 63–74, New York, NY, USA, 2013. Association for Computing Machinery. [17] Christoph Laaber, Stefan Würsten, Harald C. Gall, and Philipp Leitner. Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, page 989–1001, New York, NY, USA, 2020. Association for Computing Machinery. [18] Luc Lesoil, Mathieu Acher, Arnaud Blouin, and Jean-Marc Jézéquel. Input sensitivity on the performance of configurable systems an empirical study. Journal of Systems and Software, 201:111671, 2023. [19] Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: An easy-to-use python toolkit to support replicable IR research with sparse and dense representations. CoRR, abs/2102.10073, 2021. [20] Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, and Parthasarathy Ranganathan. Swe-fficiency: Can language models optimize real-world repositories on real workloads?, 2025. [21] MiniMax. MiniMax M27. https://www.minimax.io/news/minimax-m25, 2026. [22] Moonshot AI. Kimi K2.5. kimi-k2-5-quickstart, 2026.
https://platform.kimi.ai/docs/guide/
[23] Stefan Mühlbauer, Florian Sattler, Christian Kaltenecker, Johannes Dorn, Sven Apel, and Norbert Siegmund. Analyzing the impact of workloads on modeling the performance of configurable software systems. In Proceedings of the 45th International Conference on Software Engineering, ICSE ’23, page 2085–2097. IEEE Press, 2023. 11
[24] Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter F. Sweeney. Producing wrong data without doing anything obviously wrong! In Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XIV, page 265–276, New York, NY, USA, 2009. Association for Computing Machinery. [25] Adrian Nistor, Tian Jiang, and Lin Tan. Discovering, reporting, and fixing performance bugs. In 2013 10th Working Conference on Mining Software Repositories (MSR), pages 237–246, 2013. [26] OpenAI. Introducing GPT-5.2. https://openai.com/index/introducing-gpt-5-2/, 2025. [27] Yun Peng, Akhilesh Deepak Gotmare, Michael R. Lyu, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Perfcodegen: Improving performance of llm generated code with execution feedback. In 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge), pages 1–13, 2025. [28] Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. Coffe: A code efficiency benchmark for code generation, 2025. [29] Rui Pereira, Marco Couto, Francisco Ribeiro, Rui Rua, Jácome Cunha, João Paulo Fernandes, and João Saraiva. Energy efficiency across programming languages: how do energy, time, and memory relate? In Proceedings of the 10th ACM SIGPLAN International Conference on Software Language Engineering, SLE 2017, page 256–267, New York, NY, USA, 2017. Association for Computing Machinery. [30] Python Software Foundation. tracemalloc — Trace memory allocations. https://docs. python.org/3/library/tracemalloc.html, 2026. Python Standard Library, version 3.14.3 documentation. [31] Yuhao QING, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie M. Zhang, Heming Cui, Siu Ming Yiu, Dong HUANG, See-Kiong Ng, and Anh Tuan Luu. Effibench-x: A multi-language benchmark for measuring efficiency of LLM-generated code. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [32] Ruizhong Qiu, Weiliang Will Zeng, James Ezick, Christopher Lott, and Hanghang Tong. How efficient is llm-generated code? a rigorous & high-standard benchmark, 2025. [33] Rui Rua and João Saraiva. A large-scale empirical study on mobile performance: energy, run-time and memory. Empirical Softw. Engg., 29(1), December 2023. [34] Ripon K. Saha, Matthew Lease, Sarfraz Khurshid, and Dewayne E. Perry. Improving bug localization using structured information retrieval. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 345–355, 2013. [35] Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, and Ion Stoica. Gso: Challenging software optimization tasks for evaluating swe-agents, 2025. [36] Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improving code edits, 2024. [37] Nicolas van Kempen, Hyukje Kwon, Dung Trung Nguyen, and E. Berger. It’s not easy being green: On the energy efficiency of programming languages. 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1553–1565, 2024. [38] Siddhant Waghjale, Vishruth Veerendranath, Zora Zhiruo Wang, and Daniel Fried. Ecco: Can we improve model-generated code efficiency without sacrificing functional correctness?, 2024. [39] Jiyuan Wang, Jason Teoh, Muhammand Ali Gulza, Qian Zhang, and Miryung Kim. Perfgen: Automated performance benchmark generation for big data analytics, 2024. [40] Z.ai. GLM-51. https://z.ai/blog/glm-5.1, 2026. 12
[41] Yutong Zhao, Lu Xiao, Xiao Wang, Lei Sun, Bihuan Chen, Yang Liu, and Andre B. Bondi. How are performance issues caused and resolved?-an empirical study from a design perspective. In Proceedings of the ACM/SPEC International Conference on Performance Engineering, ICPE ’20, page 181–192, New York, NY, USA, 2020. Association for Computing Machinery. [42] Fida Zubair, Maryam Al-Hitmi, and Cagatay Catal. The use of large language models for program repair. Comput. Stand. Interfaces, 93(C), April 2025.
A
Dataset Construction and Statistics
This section provides implementation-level details of the dataset construction process, complementing Section 2. We focus on repository-specific configurations and the resulting dataset composition. A.1
Pull Request Collection
PRs are retrieved via the GitHub API. We filter PRs based on repository-specific performance-related labels and the presence of corresponding changelog updates, retaining only source-level changes to Python files. Merge commits are excluded to ensure that extracted changes correspond directly to the PR. While the collection procedure is consistent across repositories, configurations vary in label definitions, changelog locations. These repository-specific settings are summarized in Table 4 with repository license. Table 5 further shows that benchmark instances span a wide range of change sizes, from small edits to substantial modifications, indicating diverse optimization complexity.
Criterion
Table 4: Repository-specific details for PR collection. pandas scikit-learn xarray
Performance label Changelog path License
Performance doc/source/whatsnew BSD 3-Clause
Performance doc/whats_new BSD 3-Clause
topic-performance doc/whats-new Apache 2.0
Table 5: Summary statistics of code changes across benchmark instances. Metric Min Median Mean Max Lines changed Files changed
B
3.0 2.0
39.5 3.0
77.7 3.6
1,424.0 14.0
Construction and Parameterization of Performance Tests
Parameterized performance tests are constructed from PR descriptions, developer discussions, and associated code changes. These sources capture intended usage patterns and the conditions under which the optimization is effective. B.1
Workload Categories.
Parameters are grouped into four categories: • Input size: Defines the scale of the input data (e.g., number of elements, rows, or features). • Data distribution: Captures statistical and structural properties of the input, such as sparsity, duplication, or missingness, which influence execution behavior. • Data type: Specifies the representation of the input data (e.g., numeric, categorical, or mixed), affecting internal processing and memory layout. • Function-level configuration: Includes parameters and options that control the behavior of the target function independently of the input data. 13
B.2
Workload Selection and Options.
Parameters are selected to expose performance-relevant behavior while maintaining a tractable workload space. For each parameter, a set of parameter options is defined, representing concrete values used to instantiate workloads. The selection process is guided by the following criteria: • Activation of optimized behavior: Selecting values that trigger the optimized execution paths targeted by the PR. • Performance sensitivity: Including values that expose meaningful variation in runtime or memory behavior. • Coverage vs. complexity: Limiting the number of parameter options to avoid combinatorial explosion while preserving coverage of critical behaviors. • Resource feasibility: Ensuring that all workloads can be executed within runtime and memory constraints. To ensure practical feasibility, each scenario is limited to at most 24 workloads. Workloads are generated by combining parameter options via a Cartesian product; therefore, the number of options per parameter is chosen such that the total number of resulting combinations remains within this bound. Table 6 summarizes the resulting workload configurations across all scenarios. Table 6: Summary statistics of workload configuration space across 102 benchmark scenarios, reporting the distribution of parameter dimensions and workload configurations per scenario. Min Median Mean Max Parameter dimensions Workload configurations B.3
1 2
2.0 6.0
1.9 6.9
4 24
Parametrized Performance Test Implementation.
Each performance test consists of three components: • setup(): constructs inputs and configuration from the workload parameter set, • warmup() (optional): stabilizes cache-sensitive behavior, • run(): invokes only the target API. Isolating the target call in run() ensures that only the operation of interest is profiled. Each test is accompanied by a workloads field that defines the parameter space. Each parameter entry specifies (i) a name, (ii) a short identifier, (iii) a set of admissible values, and (iv) a brief description of its role. Inputs are generated deterministically using a fixed random seed to ensure reproducibility. The following examples illustrate one parameterized performance test from each of the three target repositories, including their corresponding workloads. pandas — PR#50778 PR#50778 optimizes Series.replace when using a large dictionary for substitution. The test varies the series length (N) and the number of values to replace (R) to expose how performance scales with dictionary size and series length. Performance Scenario — pandas PR#50778 import numpy as np import pandas as pd from probench.scenarios.scenario_base import PerfScenario class PR50778Scenario(PerfScenario):
14
""" PR #50778 - Benchmark Series.replace when using a large dict for to_replace. Parameters: - N: Series length - R: Number of values to replace """ def setup(self): N = int(self.params["N"]) R = int(self.params["R"]) arr = self.rng.standard_normal(N) arr1 = arr.copy() self.rng.shuffle(arr1) ser = pd.Series(arr) to_replace = self.rng.choice(arr, R) values = self.rng.choice(arr1, R) self.args = { "series": ser, "replace_dict": dict(zip(to_replace, values)) } def run(self): return self.args["series"].replace(self.args["replace_dict"])
Parameter Configuration — pandas PR#50778 "params": [ { "name": "Series Length", "short_indicator": "N", "values": ["100000", "500000", "1000000"], "explanation": "Number of elements in the Series" }, { "name": "Number to Replace", "short_indicator": "R", "values": ["1000", "5000"], "explanation": "Number of elements to replace" } ]
scikit-learn — PR#21808 PR#21808 improves LogisticRegression with the lbfgs solver for multiclass classification. The scenario varies the number of samples (n), features (p), and classes (c), as the optimization benefit increases with the number of classes. Performance Scenario — scikit-learn PR#21808 from sklearn.linear_model import LogisticRegression from probench.scenarios.scenario_base import PerfScenario class PR21808Scenario(PerfScenario): """ PR #21808 - Benchmark LogisticRegression.fit with lbfgs solver for multiclass classification.
15
Parameters: - n: Number of samples - p: Number of features - c: Number of classes """ def setup(self): n = int(self.params["n"]) p = int(self.params["p"]) c = int(self.params["c"]) X = self.rng.standard_normal((n, p)) y = self.rng.integers(0, c, size=n) self.args = { "model": LogisticRegression( solver="lbfgs", max_iter=100, multi_class="multinomial" ), "X": X, "y": y } def run(self): return self.args["model"].fit(self.args["X"], self.args["y"])
Parameter Configuration — scikit-learn PR#21808 "params": [ { "name": "n_samples", "short_indicator": "n", "values": ["50000", "100000"], "explanation": "Number of samples" }, { "name": "n_features", "short_indicator": "p", "values": ["50", "100"], "explanation": "Number of features" }, { "name": "n_classes", "short_indicator": "c", "values": ["5", "10", "20"], "explanation": "Number of classes (benefit increases with more classes)" } ]
xarray — PR#7578 PR#7578 optimizes DatasetRolling.construct with stride on datasets without indexes. The workloads varies the size of the rolling dimension (nt) and the stride (str) to capture how performance scales with both rolling window size and step size. Performance Scenario — xarray PR#7578 import numpy as np import xarray as xr
16
from probench.scenarios.scenario_base import PerfScenario class PR7578Scenario(PerfScenario): """ PR #7578 - Benchmark DatasetRolling.construct with stride on Datasets without indexes. Parameters: - nt: Size of rolling dimension - str: Stride for construct """ def setup(self): nt = int(self.params["nt"]) stride = int(self.params["str"]) data_vars = { "temperature": (["time", "space"], self.rng.standard_normal((nt, 100))), "pressure": (["time", "space"], self.rng.standard_normal((nt, 100))), } coords = { "time": np.arange(nt), "space": np.arange(100), "time_label": ("time", [f"t_{i}" for i in range(nt)]), "space_weight": ("space", self.rng.random(100)), } ds = xr.Dataset(data_vars=data_vars, coords=coords) self.args = { "func": ds.rolling(time=10), "stride": stride } def run(self): return self.args["func"].construct( "window_dim", stride=self.args["stride"] )
Parameter Configuration — xarray PR#7578 "params": [ { "name": "n_time", "short_indicator": "nt", "values": ["1000", "10000", "100000"], "explanation": "Size of rolling dimension" }, { "name": "stride", "short_indicator": "str", "values": ["5", "10"], "explanation": "Stride for construct" } ]
17
C
Retrieval Configuration
The two context provision strategies differ in how relevant code is identified and supplied to the model. Oracle Context. Oracle context is extracted offline during dataset construction. For each sample, the ground-truth files modified by the developer patch are identified and stored directly in the dataset. At inference time, no retrieval is performed, the relevant context is loaded directly from the precomputed dataset record. BM25 Retrieval. BM25 context is retrieved at runtime from the repository checked out at the baseline commit. The repository is indexed using Pyserini Lucene backend [19], and retrieval is performed using a query constructed from the task description. All experiments use a fixed BM25 configuration with k1 = 1.0 and b = 0.3, following prior work on code retrieval [34].
D
Prompt Construction: Template and Examples
We describe the prompt construction used in our experiments. Each prompt consists of three components: (i) a task description specifying the optimization objective and constraints, (ii) a code context providing the relevant source code, and (iii) a structured output specification that constrains the model to produce search-and-replace code edits. <task description> [start of src/module/example.py] ### Code context ‘‘‘python # complete file source ‘‘‘ [end of src/module/example.py] <response instruction> Task Description. The task description defines a fixed optimization objective, followed by explicit behavioral constraints and a task-specific goal derived from PR metadata. It captures both the target function and the input conditions under which performance issues arise (e.g., large inputs, sparsity, or missing values), reflecting the developer’s intended optimization. You are an expert Python performance engineer. Your objective is to optimise <entry_point> for speed and memory efficiency without altering its behaviour, public APIs, or test expectations. CONSTRAINTS: - Preserve exact semantics and outputs. - Preserve all public APIs and return types. - Do not modify tests or test expectations. OPTIMIZATION REQUEST: <task description> Optimization function: <entry_point> Response Instruction. The response instruction defines a structured output format that constrains the model to generate deterministic search-and-replace code edits. OUTPUT FORMAT — SEARCH/REPLACE BLOCKS: For each change emit exactly one block: path/to/file.py 18
[SEARCH] <code to find> [/SEARCH] [REPLACE] <replacement code> [/REPLACE] RULES: - Output ONLY the blocks. No prose, no explanations, no markdown fences. - Multiple blocks are allowed — one per logical change, in any order. - SEARCH must be copied VERBATIM from the file: * exact line breaks — never collapse a multiline signature to one line * exact indentation — do not add or remove leading whitespace * exact quotes — do not change ’ to " or vice versa * exact spacing — do not add or remove spaces around = or () - SEARCH must be the SMALLEST snippet that uniquely identifies the location. Never include whole functions or docstrings unless the whole thing changes. - REPLACE may be empty to delete the matched code. - Multiple blocks on the same file are applied top-to-bottom. ANCHORING GUIDANCE: - REPLACE: SEARCH = the lines to change, REPLACE = the new lines. - ADD: SEARCH = nearest unique surrounding line(s), REPLACE = those same lines plus the new code inserted before or after. - DELETE: SEARCH = the lines to remove, REPLACE = (empty). - If a function signature spans multiple lines in the file, your SEARCH must span those exact same lines — do not rewrite it as a single line. EXAMPLES: # 1. Replace a function, method, or class body mypackage/module.py [SEARCH] def compute(self, x): return x * 2 [/SEARCH] [REPLACE] def compute(self, x: int) -> int: return x * 3 [/REPLACE] # 2. Replace a multiline signature mypackage/module.py [SEARCH] def process( self, value: int, ) -> int: [/SEARCH] [REPLACE] def process( self, value: int, scale: float = 1.0, ) -> int: [/REPLACE] # 3. Add a function, method, or class after an existing one mypackage/module.py
19
[SEARCH] def existing_method(self): pass [/SEARCH] [REPLACE] def existing_method(self): pass def new_method(self): return True [/REPLACE] # 4. Delete a function, method, or class mypackage/module.py [SEARCH] def deprecated_method(self): do_old_thing() return result [/SEARCH] [REPLACE] [/REPLACE]
E
Measurement Pipeline Details
This appendix provides the full algorithmic specification and parameter settings of the measurement pipeline described in Section 3. Each subsection corresponds to a stage of the pipeline and presents the associated implementation details. E.1
Configuration Parameters
This subsection summarizes the configuration parameters of the measurement pipeline, grouped by the warmup, calibration, and measurement stages. For each parameter, we report its value and purpose. Table 7 lists all parameters using notation consistent with Algorithms 1–3, ensuring alignment between the specification and implementation. The parameter values are chosen to balance measurement stability and computational cost. We follow established principles of rigorous benchmarking, including warmup execution, repeated measurements, and variance-aware stopping criteria [16, 7, 17]. Parameters not prescribed by prior work are set empirically based on pilot experiments and kept fixed across all scenarios to ensure consistency and comparability. For runtime measurement, parameters control the number of invocations per iteration and the minimum sample duration, ensuring that measurements are not dominated by timer resolution or short-term system noise. For memory measurement, parameters define the sampling configuration used during tracing, with values chosen to provide sufficient temporal resolution while limiting instrumentation overhead. Finally, parameters related to convergence and validation are configured to ensure that collected measurements are both stable and detectable performance effects. The rationale behind these criteria is discussed separately in Appendix E.4 and Appendix E.5. E.2
Calibration
This subsection describes the calibration procedure used to derive workload-specific measurement parameters. The objective is to obtain runtime and memory measurements that are both statistically stable and representative, while minimizing instrumentation overhead. To this end, calibration determines (i) the number of function invocations per iteration and (ii) the memory sampling interval. Runtime measurements in microbenchmark settings are inherently noisy due to system-level variability such as scheduling, caching, and background processes. Single executions are therefore 20
Table 7: Measurement pipeline configuration and parameters used in Algorithms 1–3. Category
Parameter
Value
Purpose
nwarmup,min
5
nwarmup,max
12
w
4
τwarmup
0.08
Minimum number of function calls executed before checking for stability. Ensures early executions do not appear artificially stable. Maximum number of warmup calls. Stops warmup even if stability is not reached. Number of most recent runtimes used to compute the CV. Defines the window over which stability is evaluated. Threshold on the CV. Warmup stops once runtime variability falls below this value.
nprime
3
nprobe,min
3
nprobe,max
10
tprobe,min
1s
twindow
0.3 s
α
1.2
ninv,min
5
ninv,max
1 × 103
K
60
δmin
2 × 10−4 s
δmax
5 × 10−2 s
niter,min
5
niter,max
12
tbudget
30 s
τCI
0.1
Warmup Phase
Runtime calibration
Memory Calibration
Adaptive Convergence
Number of untimed invocations executed before the probe phase. Removes one-time overhead such as imports and cache initialization. Minimum number of timing samples collected during the probe phase, where execution time is estimated. Maximum number of timing samples collected during the probe phase. Limits calibration cost for longrunning functions. Minimum total duration of the probe phase used to estimate the representative execution time. Target duration used to derive the number of invocations per iteration, based on the estimated execution time. Safety factor applied during the computation of ninv to compensate for runtime variability and timer noise. Lower bound applied to the computed ninv before the measurement phase. Ensures sufficient work per iteration. Upper bound applied to the computed ninv before the measurement phase. Prevents excessive work for very fast functions. Target number of memory samples per function call. Determines how frequently memory usage is recorded. Minimum time between memory samples. Avoids excessive overhead for fast executions. Maximum time between memory samples. Ensures sufficient resolution for slower executions. Minimum number of measurement windows collected before checking convergence. Maximum number of measurement windows. Stops measurement if convergence is not reached. Maximum total wall-clock time allowed for measurement. Terminates measurement when this limit is exceeded. Threshold on the relative confidence interval width (RCIW). Measurement stops once results are sufficiently precise.
21
insufficient to provide reliable estimates. To address this, each measurement iteration aggregates multiple invocations of the target function, and the median runtime is used as a robust estimator. This reduces the influence of outliers and transient fluctuations. We distinguish between invocations and iterations. An invocation corresponds to a single execution of the target function f , whereas an iteration represents one statistical sample used in the evaluation. By grouping multiple invocations into a single iteration, we ensure that each sample has sufficient signal strength while maintaining comparability across iterations. In contrast, memory measurements obtained via tracemalloc [30] exhibit lower variance but introduce non-negligible overhead due to continuous allocation tracking. Increasing the number of invocations would amplify this overhead. Instead, we calibrate the sampling interval to balance coverage and cost, ensuring that relevant allocation events are captured while limiting instrumentation overhead. Calibration parameters are inherently system-dependent and therefore not transferable across environments. However, since SWE-Pro reports relative performance differences measured under identical conditions, locally calibrated parameters are both necessary and sufficient to ensure comparisons. Table 7 summarizes the calibration configuration, including timing targets, iteration bounds, and memory sampling parameters. Algorithm 1 Calibration Require: f , calibration parameters (Table 7) Ensure: ninv (runtime measurement invocations per iteration), δ (memory sampling interval) 1: for i = 1 to nprime do 2: R ESTORE; f () ▷ Warmup 3: end for 4: S ← [ ], tprobe ← 0 5: while |S| < nprobe,min or tprobe < tprobe,min do 6: if |S| ≥ nprobe,max then break 7: end if 8: R ESTORE; t ← time(f ()); S ← S ∪ {t}; tprobe += t 9: end while 10: t̃ ← median(S) ▷ Representative runtime estimates twindow · α 11: ninv ← clamp , ninv,min , ninv,max ▷ Compute invocations per iteration t̃ t̃ , δmin , δmax ▷ Compute memory sampling interval 12: δ ← clamp max(K, 1) 13: return ninv , δ The R ESTORE operation reconstructs input data and resets any modified state before each invocation, ensuring independent and identical execution conditions. It is not measured. Before timing begins, nprime untimed invocations warm up the Python interpreter and function-level caches, preventing first-call overhead such as bytecode compilation from affecting measurements. The probe loop then collects per-invocation timings until a minimum duration tprobe,min is reached or nprobe,max samples are gathered. From the resulting sample set S, two parameters are derived: the number of invocations per measurement iteration, twindow · α ninv = clamp , ninv,min , ninv,max , t̃ and the memory sampling interval, δ = clamp
t̃ , δmin , δmax . max(K, 1)
The median runtime t̃ of the probe sample set is used as a robust estimate of the representative execution time. Based on this estimate, the number of invocations ninv is chosen such that the total runtime of one measurement iteration approximates the target window duration twindow . The safety factor α compensates for measurement overhead and residual runtime variability. 22
The same representative runtime t̃ is further used to determine the memory sampling interval δ. Using the median instead of the mean ensures robustness to outliers and avoids overly coarse sampling intervals that could miss short-lived allocation patterns. Calibration is performed once per workload on a baseline implementation. The resulting parameters ninv and δ are then reused for all other implementations to ensure consistent and comparable measurements across versions. E.3
Warmup
The warmup phase repeatedly executes f and evaluates the CV, defined as stdev(W )/mean(W ) over a trailing window W of the most recent w invocations. As a scale-independent stability measure, it generalises across workloads with heterogeneous execution times [17]. Following [7], warmup targets two properties: an initialised state, where startup effects have subsided, and an independent state, where successive times are approximately i.i.d. The trailing CV serves as an automated proxy for both, discarding early high-variance observations while detecting local stability. Warmup terminates when CV < τwarmup , subject to a minimum of nwarmup,min invocations and a hard cap of nwarmup,max . Algorithm 2 summarises the procedure. Algorithm 2 Warmup Require: f , window w, threshold τwarmup , bounds nwarmup,min , nwarmup,max Ensure: convergence before measurement 1: T ← [ ] 2: for i ← 1 to nwarmup,max do 3: R ESTORE; T ← T ∪ {runtime(f ())} 4: if |T | ≥ max(nwarmup,min , w) then 5: CV ← stdev(T[−w:] )/mean(T[−w:] ) ▷ trailing window over last w samples 6: if CV < τwarmup then break 7: end if 8: end if 9: end for 10: return CV < τwarmup
E.4
Measurement
After warmup convergence, each workload is evaluated over multiple iterations, each consisting of two separated phases to avoid instrumentation interference. Measurement is performed per workload in an isolated container using repeated measurement iterations. Each iteration consists of a runtime phase and a memory phase. Runtime phase. Within each iteration, the target function is executed multiple times, as determined during calibration, and the median of these executions yields a single, stable runtime measurement. These per-iteration measurements are then used for convergence analysis. Memory phase. A single invocation is executed under tracemalloc [30] with a background sampler, where the sampling rate is calibrated based on the function runtime to ensure sufficient temporal resolution. From the sampled values, we compute the TWMU, while peak memory usage above baseline is obtained directly with tracemalloc. Adaptive Convergence Iterations are collected until one of the following stopping conditions is met: (1) the iteration count reaches a predefined maximum, (2) the total measurement time exceeds a predefined limit after at least a minimum number of samples has been collected, or (3) a minimum number of samples has been collected and all monitored metrics satisfy a target RCIW. For a sample vector x = (x1 , . . . , xn ), the RCIW is defined as RCIW(x) =
CI1−α (x)upper − CI1−α (x)lower . mean(x) 23
The (1 − α) confidence interval of the mean is given by stdev(x) stdev(x) CI1−α (x) = mean(x) − qα/2 · √ , mean(x) + qα/2 · √ , n n where qα/2 denotes the critical value of the standard normal distribution corresponding to the chosen confidence level. Let r = (r1 , . . . , rn ) denote the per-iteration runtime samples, where each ri is computed as the median runtime over multiple function invocations within iteration i. The number of function invocations per iteration is determined during a prior calibration phase to reduce measurement noise and ensure stable estimates for RCIW. Similarly, let mtwmu = (mtwmu , . . . , mtwmu ) denote the n 1 peak peak peak time-weighted memory usage samples, and m = (m1 , . . . , mn ) the peak memory increase samples. Adaptive sampling terminates once RCIW(r) ≤ τCI ,
RCIW(mtwmu ) ≤ τCI ,
RCIW(mpeak ) ≤ τCI .
where τCI denotes the convergence threshold. Condition (3) serves as the primary stopping criterion, ensuring that all reported metrics satisfy a predefined relative error bound at the chosen confidence level. The full measurement procedure is shown in Algorithm 3. E.5
Validation Aggregation
We aggregate validation results in two stages: first within each PR, and then across PRs. This separation reflects the distinction between workloads belonging to the same optimization context and independent optimization cases originating from different PRs. Within a PR, each workload configuration is evaluated independently. Every workload consists of multiple container runs, and each run produces a sequence of profiling samples for each performance metric. To reduce the influence of transient measurement artifacts, we apply Median Absolute Deviation (MAD)-based outlier trimming. For a set of observations x1 , . . . , xn , the MAD is defined as: MAD = median (|xi − median(x1 , . . . , xn )|) An observation xi is retained if its deviation from the median satisfies: |xi − median(x)| ≤ k · c · MAD where k is a configurable sensitivity parameter and c is the consistency factor used to align the estimator with the standard deviation under normality assumptions. The MAD trimming procedure is applied at two levels: within containers and between containers. Profiling samples inside each container run are first filtered to reduce transient effects, after across container-level summaries to reduce anomalous runs caused by external system variability. To ensure reliability, a workload is considered stable only when the RCIW of both the baseline and LLM-optimized measurements remains below the configured system-noise threshold. Stability is evaluated at two granularities: within-run variability and between-run variability. Stable workloads are subsequently filtered using a SNR:
SNR =
|∆%| max (RCIWwithin-baseline , RCIWwithin-llm , RCIWbetween-baseline , RCIWbetween-llm )
where ∆% denotes the relative performance change. A workload is retained only if the observed effect sufficiently exceeds the estimated measurement noise. Following prior recommendations, we use an SNR threshold of 2, corresponding to a conservative detectability boundary between signal and noise [4]. 24
Algorithm 3 Adaptive Measurement Require: target function f , runtime invocations ninv , sampling interval δ, convergence threshold τCI , bounds niter,min , niter,max , time budget tbudget Ensure: sample set I 1: WARMUP(f ) 2: I ← [ ] 3: tstart ← now() 4: loop 5: R ← [] 6: D ISABLE GC 7: for j = 1 to ninv do 8: R ESTORE 9: append R ← runtime(f ()) 10: end for 11: E NABLE GC 12: r ← median(R) 13: R ESTORE 14: S TART M EMORY T RACER 15: S TART BACKGROUND S AMPLER(δ) 16: f () 17: S TOP BACKGROUND S AMPLER 18: S TOP M EMORY T RACER 19: mpeak , mtwmu ← C OMPUTE M EMORY M ETRICS(·) 20: append I ← (r, mpeak , mtwmu ) 21: n ← |I| 22: telapsed ← now() − tstart 23: if n ≥ niter,max then 24: break 25: end if 26: if n ≥ niter,min then 27: R ← {ri } 28: Mtwmu ← {mtwmu } i 29: Mpeak ← {mpeak } i 30: if RCIW(R) ≤ τCI and RCIW(Mtwmu ) ≤ τCI and RCIW(Mpeak ) ≤ τCI then 31: break 32: end if 33: if telapsed ≥ tbudget then 34: break 35: end if 36: end if 37: end loop 38: return I The remaining workloads are classified as either improvements, regressions and conflicting. Since IFs represent multiplicative, rate-like changes, workload-level IFs within a PR are aggregated using the harmonic mean, following recommendations for summarizing normalized performance ratios [12]. Given workload ratios r1 , . . . , rn , the PR-level summary is computed as: n HPR = Pn
1 i=1 ri
Only workloads that satisfy the stability and SNR criteria contribute to this aggregation. Each metric for a given PR is then classified according to the following hierarchy: • Not measurable: The proportion of stable workloads falls below the minimum quality requirement. • No signal: No stable workload passes the SNR criterion. 25
• Improved: Improvements dominate and no regressions are observed. • Regressed: Regressions dominate and no improvements are observed. • Conflicting: Both improvements and regressions exceed the configured conflict tolerance. After computing PR-level summaries, results are aggregated across PRs. Each PR classified as improved or regressed is treated as one independent optimization case. The final cross-PR estimate is computed as the arithmetic mean of the corresponding PR-level harmonic means: m
1 X S= HPR,j m j=1 where m denotes the number of eligible PRs. This two-level aggregation strategy reflects the different semantic roles of workloads and PRs. The harmonic mean captures the joint effect of related workloads within a single optimization context, while the arithmetic mean summarizes independent optimization cases across PRs. By excluding unstable measurements prior to aggregation, the framework ensures that both workload-level and cross-PR estimates remain robust against measurement noise and transient execution artifacts. E.6
System Configuration and Execution Environment
To ensure reproducible performance evaluation, SWE-Pro relies on pre-built Docker images for both baseline and optimized versions of each scenario. These images are constructed once and stored with immutable identifiers (tags and digests), guaranteeing that all experiments are executed in identical environments. Docker setup Each benchmark is executed within a resource-constrained Docker container with fixed resource and runtime settings: • CPU pinning: Containers are restricted to a fixed CPU core (cpuset=2) and memory node (cpuset_mems=0) • Memory limit: A fixed memory budget (30GB) is enforced across runs • Thread control: Numerical backends are restricted to single-threaded execution using OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1, MKL_NUM_THREADS=1, NUMEXPR_MAX_THREADS=1, VECLIB_MAXIMUM_THREADS=1, and BLIS_NUM_THREADS=1, ensuring consistent execution behavior across runs Experimental Setup All experiments were conducted on a MacBook Pro equipped with an Apple M4 Max chip (14-core CPU with 10 performance and 4 efficiency cores) and 36 GB unified memory. Patch application and evaluation were executed in isolated Docker containers running on the same host. Image Versioning Rather than rebuilding environments during benchmarking, PRO-Bench uses pre-built Docker images for both baseline and optimized versions. These images are stored in a registry and referenced via immutable digests. E.7
Additional Results
Under this subsection, we present additional results that did not fit in the main section. Table 8 reports patch generation and correctness success rates under both Oracle and BM25 retrieval settings. 26
Table 8: Patch generation and correctness pass rate under Oracle and BM25 retrieval. Patch denotes the successful patch application rate (%); Test denotes the percentage of all evaluated samples that pass all relevant tests (%). Oracle
E.7.1
BM25
Model
Patch ✓
Test ✓
Patch ✓
Test ✓
GPT-5.2 Claude Sonnet 4.6 Gemini 3.1 Flash Kimi K2.5 GLM-5.1 MiniMax M2.7
95.1% 90.2% 93.1% 30.4% 97.1% 45.1%
79.4% 72.5% 50.0% 18.6% 55.9% 23.5%
86.3% 70.6% 78.4% 17.6% 63.7% 21.6%
68.6% 55.9% 57.8% 11.8% 39.2% 8.8%
Claude Sonnet 4.6 Optimization Analysis for PR #7578
Claude Sonnet 4.6 patch with BM25 retrieval for PR #7578 diff --git a/xarray/core/rolling.py b/xarray/core/rolling.py index d49cb6e1..c0749e2b 100644 --- a/xarray/core/rolling.py +++ b/xarray/core/rolling.py @@ -770,12 +770,11 @@ class DatasetRolling(Rolling["Dataset"]): dims = [d for d in self.dim if d in da.dims] if dims: wi = {d: window_dims[i] for i, d in enumerate(self.dim) if d in da.dims} st = {d: strides[i] for i, d in enumerate(self.dim) if d in da.dims} dataset[key] = self.rollings[key].construct( window_dim=wi, fill_value=fill_value, stride=st, + stride=1, keep_attrs=keep_attrs, ) else: @@ -787,9 +786,14 @@ class DatasetRolling(Rolling["Dataset"]): attrs = self.obj.attrs if keep_attrs else {} + + + + + + + +
return Dataset(dataset, coords=self.obj.coords, attrs=attrs).isel( {d: slice(None, None, s) for d, s in zip(self.dim, strides)} ) result = Dataset(dataset, coords=self.obj.coords, attrs=attrs) if any(s != 1 for s in strides): result = result.isel( {d: slice(None, None, s) for d, s in zip(self.dim, strides)} ) return result
Claude Sonnet 4.6 patch with Oracle retrieval for PR #7578 diff --git a/xarray/core/rolling.py b/xarray/core/rolling.py index d49cb6e1..f1393c57 100644 --- a/xarray/core/rolling.py
27
+++ b/xarray/core/rolling.py @@ -787,9 +787,21 @@ class DatasetRolling(Rolling["Dataset"]): attrs = self.obj.attrs if keep_attrs else {} + + + + + + + + + + + + + + +
return Dataset(dataset, coords=self.obj.coords, attrs=attrs).isel( {d: slice(None, None, s) for d, s in zip(self.dim, strides)} ) # Apply stride to coordinates along rolling dimensions. # Data variables already have stride applied inside each # DataArrayRolling.construct call, so we must NOT apply isel again # on the whole Dataset (that would double-stride data variables). stride_indexer = {d: slice(None, None, s) for d, s in zip(self.dim, strides)} new_coords = { name: coord.isel( {d: stride_indexer[d] for d in stride_indexer if d in coord. dims} ) if any(d in coord.dims for d in stride_indexer) else coord for name, coord in self.obj.coords.items() } return Dataset(dataset, coords=new_coords, attrs=attrs)
impr. factor
Runtime (s)
6.2×
6.3×
36.7×
36.7×
108×
106×
Peak memory (MiB)
16.5×
16.5×
16.7×
16.7×
16.6×
16.7×
TWMU (MiB) 2.8k×
2.4k×
39.1k×
39.0k×
486k×
487k×
100k× 10k×
10 10 10 r=5 r=5 r=5 , str= 1000, st 000, str= 0000, st 000, str= 0000, st 0 0 0 0 0 nt= nt=1 nt=1 nt=100 nt=1 nt=1
1k× 100× 10×
Figure 5: Impact of input configuration on optimization effectiveness for PR #7578 under BM25 retrieval. Per-workload IF varies with rolling size (nt) and stride (s), highlighting input-dependent performance behavior. Figure 5 presents the workload-level optimization behavior of the BM25-retrieved context with Claude Sonnet 4.6 for PR #7578, while the complementary Oracle-retrieved result is shown in Figure 4. The workload configurations and performance tests used for this PR in this analysis are described in Section B.3. Both patches target the same inefficiency in the rolling-window construction pipeline caused by repeated stride applications during dataset construction. The BM25-retrieved patch simplifies the execution flow by enforcing stride=1 during rolling construction and applying slicing only once afterward. In contrast, the Oracle-retrieved patch applies stride handling more selectively by operating only on coordinates and avoiding double-striding of data variables. As shown in Figure 5, the BM25-retrieved patch achieves a stronger optimization effect on TWMU for workload nt=10000, with runtime improvements exceeding 100× and TWMU reductions approaching 490×. The Oracle-retrieved patch exhibits similar optimization trends in Figure 4, suggesting that 28
the main performance gains originate from eliminating repeated stride operations rather than from coordinate-handling logic itself. The nt=10000 workload thus serves as the decisive differentiator in this evaluation. This comparison also illustrates how optimization behavior can vary considerably across different execution configurations with different solutions.
29