arXiv CSopen access

PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

arXiv:2607.19653v1 [cs.SE] 22 Jul 2026

Ryan Deng 1 , Yuanzhe Liu 2 , Bastian Lipka 3 , Yao Ma 2 , Xuhao Chen 4 , Tim Kaler 1† , and Jatin Ganhotra 5† 1 Massachusetts Institute of Technology, Cambridge, Massachusetts, USA 2 Rensselaer Polytechnic Institute, Troy, New York, USA 3 IBM, Ehningen, Germany 4 Michigan State University, Lansing, Michigan, USA 5 IBM, Thomas J. Watson Research Center, Yorktown Heights, New York, USA Abstract—Large language model (LLM) agents now perform well on correctness-oriented repository-level tasks, including SWE-Bench issue resolution and feature implementation in real codebases. However, they still struggle with repository-level code optimization, which requires preserving behavior while improving runtime performance. Passing tests is not enough in this setting; a patch must preserve behavior, implement code optimization, and approach expert speedups. Current agents often miss bottlenecks hidden behind abstraction layers and native extensions, stop after shallow speedups, or insufficiently test the code patches that thus may silently break edge cases. We present PerfAgent, a profiler-guided, verifier-in-the-loop workflow that gives an off-the-shelf coding agent the feedback needed to find real hotspots, improve beyond the first passing patch, and use profiler evidence rather than timing alone to decide what to optimize next. On two challenging optimization benchmarks, GSO and SWE-fficiency-Lite, PerfAgent more than doubles the rate of expert-matching patches over OpenHands with GPT-5.1, improving from 19.6% to 39.2% on GSO and from 26% to 74% on SWE-fficiency-Lite. It also surpasses an oracle best-of-five baseline at substantially lower cost, showing that the gains come from better feedback rather than additional test-time sampling. Index Terms—software performance engineering, repositorylevel code optimization, LLM agents, profiling, feedback loop

I. I NTRODUCTION LLM agents have shown remarkable capability in solving real-world software engineering tasks. Frontier LLMs have achieved strong results on benchmarks such as SWE-Bench, SWE-Bench Pro and Multi-SWE-Bench [1]– [3], which evaluate agents on their ability to solve issues and implement features on a wide range of complex realworld code repositories. However, these benchmarks only focus on correctness and ignore the performance impact of the changes produced by the agent. Recent benchmarks such as GSO [4], SWE-fficiency [5] and SWE-Perf [6] require agents to optimize code in realworld repositories. Unfortunately, LLM agents struggle to produce changes that match the performance of those † Co-senior authors.

made by human experts [4]. In general, optimizing code within a repository is more challenging than optimizing individual compute kernels, since changes required to get better performance are often complex and span multiple languages and abstraction boundaries in the repository. Specifically, by diving into the benchmarks, we identify three failure modes that prevent LLM agents from matching human experts on repository-level code optimization. First, agents do not use profiling tools to their full effect, and thus often miss the real bottlenecks, especially those hidden across abstraction boundaries or inside native extensions, in the repository. Second, agents often prematurely terminate optimizing after shallow speedups, leaving more performance improvement opportunities on the table. Third, agents tend to test the changes narrowly, leading to patches that are fast on target workloads but may silently break edge cases and downstream code paths elsewhere. We introduce PerfAgent, a workflow for repositorylevel code optimization to specifically address the above challenges. As shown in Fig. 1, PerfAgent supplies the agent with targeted information at three stages of the agentic loop. First, it runs a sampling profiler on the given workload and feeds the agent a curated summary of hotspots—their location, call context, and share of runtime—so the agent can localize where time is actually spent, including inside native extensions. Second, an objective-driven loop controller refuses to accept the agent’s first passing patch: after each submission it rebuilds, revalidates, and re-profiles the repository and asks the agent to keep optimizing, retaining the fastest correct patch across multiple attempts. Third, a selective test stage runs the curated test cases affected by the agent’s current changes, sufficiently catching any regressions with low cost and returning them as feedback. This guarantees that correctness is not forgotten during optimization. Together, these components turn a one-shot agent into a profiler-guided, verifier-in-the-loop optimizer. We

updated hotspot summary + measured speedup

(≤ θ iterations)

1 Curated Profiler Usage hotspots: location, call context, self/total time; incl. native extensions (py-spy)

3

2 Coding Agent summary

Selective Validation

off-the-shelf base (Mini-SWE-Agent) edits repo, submits a patch

patch

rebuild + pytest-testmon (only tests affected by agent’s changes)

pass

Measure & re-profile speedup on workload; updated hotspot summary

build / test failure returned as feedback

Best-patch selector fastest correct patch across ≤ θ iterations — not the last one submitted

Fig. 1: Overview of the PerfAgent workflow. Unlike a plain retry loop, PerfAgent injects targeted feedback at three points in each iteration: ① a curated profiler summary is fed to the agent before it begins, localizing hotspots across abstraction boundaries including native extensions; ② a selective validation (pytest-testmon) stage that checks each submitted patch against only the tests affected by the agent’s changes, catching correctness regressions cheaply without running the full suite; and after a passing patch, the ③ objective-driven controller re-profiles and returns the updated hotspot summary alongside the measured speedup, driving the next of up to θ iterations (θ=5 in Section IV). Additionally, a best-patch selector retains the fastest correct patch across all iterations rather than the last one submitted. Fig. 3 traces a concrete run of this loop on a GSO task. implement PerfAgent on top of Mini-SWE-Agent 0 as the base agent. Experiments across both frontier and open-source models show that our workflow yields large and consistent gains over strong general-purpose SWE agents [10], [11]. Specifically, with GPT-5.1, PerfAgent raises the fraction of expert-matching patches by 2× on GSO (39.2% vs. 19.6%) and 2.8× on SWE-fficiencyLite (74% vs. 26%). We also show that PerfAgent can improve results for open-source models like Kimi-K2 by 1.5× on both GSO and SWE-fficiency-Lite. Importantly, PerfAgent is sample-efficient: it surpasses an oracle bestof-five inference-scaling baseline at a fraction of the cost, evidence that the gains stem from better feedback, not merely more test-time compute. This paper makes the following contributions: • We

characterize three failure modes that prevent general-purpose LLM agents from matching human experts on repository-level code optimization: (1) missing real bottlenecks, (2) premature termination after the first passing patch, and (3) insufficient testing that leaves correctness regressions undetected. • We present PerfAgent, a profiler-guided, objectivedriven agentic workflow for repository-level code optimization. PerfAgent features three dedicated mechanisms, i.e., curated profiler summary, selective test, and objective-driven controller, which specifically address our identified three failure modes. • We implement PerfAgent and evaluation with various models shows that PerfAgent more than doubles the rate of expert-matching patches over strong baselines (2× on GSO, 2.8× on SWE-fficiency-Lite). PerfAgent 0 PerfAgent can be applied to any off-the-shelf coding agent such as SWE-Agent, Trae Agent or iSWE-Agent [7]–[9]

also beats an oracle best-of-five baseline while spending far less, isolating profiler-guided feedback—rather than additional compute—as the source of the improvement. Note that we view performance engineering as complementary to existing agentic workflows, but it imposes a stricter requirement: whereas issue-resolution benchmarks ask only that a patch be correct, repository-level optimization demands a patch that both be correct and can improve performance, judged against the optimization a human expert produced. II. B ENCHMARKS O VERVIEW We evaluate PerfAgent on GSO and SWE-fficiency [4], [5], two repository-level code optimization benchmarks. Table I provides an overview of the benchmarks. Note that in this paper, we evaluate on SWE-fficiency-Lite, a randomly sampled subset of 100 tasks from SWE-fficiency, which is used by [5] when presenting the benchmark. In code optimization benchmarks, the agent is given a code repository at a particular commit along with a script that shows an example workload involving the use of an API implemented in the repository. The agent is tasked with making changes to the repository to improve the performance of the script, and its changes are evaluated both in terms of correctness and performance. Correctness is validated on hidden tests that ensure the existing behavior of the API is preserved. Performance is measured against a human expert baseline by computing Speedup

a speedup ratio SR = Speedup Agent , where SpeedupAgent Human and SpeedupHuman are the speedups achieved over the base repository by the agent’s and the human expert’s changes, respectively. The agent’s final score on that task is a function of this ratio.

TABLE I: GSO and SWE-fficiency-Lite benchmark overview. It shows the median speedup achieved by the human expert baseline over the base repository, the median patch size (lines of code modified) of the human expert baseline, and the percentage of tasks in which the human expert baseline modified non-Python code such as C/C++ or Rust. *Python indicates Python-only changes in the human expert baseline patch; the remaining languages may be used in conjunction with Python. Benchmark GSO SWE-fficiency-Lite

# Repos

# Tasks

Speedup

Patch Size

% Non-Py

10 9

102 100

2.43× 3.57×

140 20

59 12

• GSO is composed of 102 tasks spanned across 10

popular python repositories such as numpy, pandas, and pydantic, and involves languages such as Python, Cython, C, C++ and Rust. In addition to the test script provided to the LLM agent, GSO also has other hidden tests that are used for evaluation. These hidden tests exercise the target API in diverse ways, and the benchmark uses them to assess both the correctness and performance of the agent’s changes. The speedup ratio computed is averaged over all hidden tests. • SWE-fficiency-Lite is composed of 100 tasks that span 9 popular python repositories such as pandas, scikit-learn and sympy and involves languages such as Python, Cython, C and C++. Similar to GSO, SWE-fficiency provides a script to the LLM agent to optimize. Unlike GSO, however, it has no hidden tests. The provided test script alone is used for performance evaluation, while correctness is validated against a curated subset of the repository’s existing test suite. III. P ERFA GENT ’ S W ORKFLOW FOR C ODE O PTIMIZATION General-purpose LLM agents fail on repository-level code optimization in three characteristic ways: • Missing the real bottleneck. Agents struggle to localize where time is actually spent: they often either skip profiling entirely or reach for Python’s builtin cProfile, which carries high overhead and sees only Python-layer frames. As a result they fall back on surface-level Python edits and overlook hotspots hidden inside native extensions or beneath several layers of abstraction, yielding only modest speedups relative to a human expert’s patch. On the NumPy string kernel of Fig. 3, for example, cProfile cannot observe that 80% of runtime is consumed by perelement C scalar allocation, so an unguided agent stops at the first Python-level fast path it finds at the original abstraction layer, whereas profiler feedback drives PerfAgent to a purpose-built native C kernel (5.87×); we trace this case end to end in Section IV-D2. • Premature termination. Once a patch passes the relevant tests and shows a noticeable speedup, the agent emits a stop signal, even when substantial headroom remains. Human experts, by contrast, have deep insight into how much improvement is still possible and when

Language Breakdown (Python*, C/C++, Cython, Rust) (41%, 45%, 10%, 4%) (88%, 1%, 11%, 0%)

to stop, so an agent that settles for the first modest speedup performs far worse than an expert in practice. • Insufficient testing of complex changes. The changes that yield the largest speedups are often complex and reach into core library code. Yet coding agents only test the changes narrowly by running a limited range of tests. This results in a patch that is fast on the target workload but can silently break edge cases and downstream code paths elsewhere. PerfAgent introduces one targeted component to address each failure mode (Sections III-B to III-D). A. PerfAgent’s Workflow Overview Fig. 1 shows PerfAgent’s overall workflow. Each iteration of the loop consists of two phases: an inner agent phase and an outer evaluation phase. In the inner phase, the agent is given the most recent profiler summary (see Section III-B) and runs inside our harness, until it submits a patch. The first iteration uses a profiler summary computed on the unmodified base repository and subsequent iterations use a summary computed on the repository with the agent’s submitted patch from the previous iteration. In the outer phase, the controller (detailed in Section III-C) takes the submitted patch and applies it to the repository, rebuilds, and runs the curated test suite described in Section III-D. If the build fails or any test fails, we exit the outer phase early and return the error output to the agent as feedback for the next iteration. Otherwise, we run the profiler on the resulting repository with the agent’s changes to produce a new summary that highlights the new hotspots, and pass it to the agent for its next attempt. The controller allows the LLM up to θ submissions before it terminates. Across all iterations, we record the speedup of each submitted patch on the provided workload, and report the patch with the highest speedup as the final output. PerfAgent is built on Mini-SWE-Agent [7] but is harness-agnostic: the same workflow can be layered onto any coding agent that emits a stop signal, such as OpenHands, Trae Agent, or iSWE-Agent [8]–[10]. B. Hitting the Bottlenecks via Curated Profiler Usage To address the first failure mode, missing the real bottleneck, PerfAgent runs a profiler on the workload to localize the code responsible for the runtime. When

RAW STACK FRAMES … libc · <module> · main

PARSED

POST-PROCESSING

workload (perf_script.py:47) add (defchararray.py:337) _vec_string_with_args

Filter drop unrelated frames

Parse aggregate stacks

workload

HUMAN-READABLE SUMMARY

1039 SAMPLES

99.8%

HAND OFF

└─ add └─ _vec_string_with_args

57.2%

└─ PyArray_Scalar

38.7%

LLM

summarize

PyArray_Scalar

99.8% of time is in np.char.add, churning a NumPy scalar per element. Avoid the per-element boxing.

(scalarapi.c:599) 133 ⋮ 1,039 samples total

Fig. 2: Overview of PerfAgent’s profiler pipeline. Raw stack frames are collected by running py-spy on the workload, setup-related frames are filtered out, and the remaining samples are parsed and aggregated into hotspots with self-time, total-time, call context, and a native-vs-library flag. A separate LLM call then produces a concise naturallanguage summary that is passed to the agent. In contrast, naive profiler usage — running cProfile and forwarding its raw output — yields noisy per-call counters covering only Python-layer frames, misses hotspots inside native extensions, and floods the agent’s context window with irrelevant detail. general-purpose agents profile at all, they reach for Python’s built-in cProfile, which as noted above, carries high overhead and sees only Python-layer frames. We instead use the py-spy sampling profiler, which has much lower overhead and profiles both Python code and native extensions [12], removing both limitations; it interrupts execution at a fixed interval (100 times a second) and snapshots the call stack. However, a low-overhead, native-aware profiler is necessary but not sufficient: its raw output is voluminous, and forwarded as-is it would flood the agent’s context with detail that buries the real bottleneck. PerfAgent therefore turns py-spy’s raw samples into a compact, actionable summary (Fig. 2) rather than forwarding them directly. We augment the provided workload script to run repeatedly for a fixed 10-second profiling window (the while loop at line 15) so that py-spy gathers enough samples, then discard samples related to setup such as reading or initializing data. We aggregate the rest into hotspots, each annotated with its location (filename, function name, line number), full call stack, share of samples, and an external-library flag that keeps the agent from editing code outside the repository. We report both self-time (excluding calls to other functions) and totaltime (including them) so the agent can choose between speeding up a function and calling it less often. Finally, instead of dumping the full output into the prompt, a separate LLM call condenses it into a natural-language summary, bounding what the agent must read. We quantify the payoff of this curated usage over naive profiling in Section IV-E3. Overall, we find that detailed profiling output helps agents pinpoint the parts of a repository responsible for performance overhead. This is especially valuable in large

modern codebases where the many layers of abstraction can make it difficult for the agent to identify these hotspots by simply navigating the repository. We quantify how often this matters in Section IV-D2: PerfAgent reaches low-level code (C, C++, Cython, or Rust) on 48% of GSO instances versus 31% for the OpenHands baseline, including 21 instances where it goes low-level and OpenHands modifies only Python. C. Objective-Driven Loop Controller To address the second failure mode, premature termination, we push the agent past its first passing patch with an objective-driven loop controller. Simply prompting the agent to keep going is not enough: without a concrete target to optimize toward, it tends to stop again after a marginal change, and without re-checking each new attempt, a faster but broken patch is accepted as progress. Our controller instead makes measured speedup a firstclass objective and re-validates every attempt, so the agent keeps improving without sacrificing correctness. PerfAgent’s loop controller is an instance of verifier-inthe-loop self-refinement, in the lineage of Reflexion and Self-Refine [13], [14], but specialized for performance: the feedback injected between attempts is concrete profiler and test output, and the objective being refined is measured speedup rather than correctness alone.1 In Mini-SWE-Agent, when the agent is finished with its task, it emits a STOP command that signals the harness to terminate the agentic loop. PerfAgent’s workflow intercepts this message and obtains the agent’s submitted 1 Iterating an agent toward an objective has recently become common in practice, e.g., the “Ralph” loop and scheduled agent re-invocation. Our contribution is not the loop itself but the optimization-specific feedback and objective that drive it.

Mini-SWE-Agent patch Turn 1 1.40× ✓ build ✓ tests generic dispatch Profiler: replace/_vec_string 67% total; PyArray_Scalar, Py_DECREF churn Agent: adds a C fast path for scalar args (_vec_string_scalar_args) re-profile Turn 2 2.36× ✓ build ✓ tests Python wrapper Profiler: _to_bytes_or_str_array 29% total; scalar-arg handling 18% self Agent: Python fast path in np.char.replace that skips the array conversion re-profile Turn 3 5.56× ✓ build ✓ tests native C kernel Profiler: per-element PyArray_Scalar/object overhead persists (29%+12% self) Agent: writes a native C kernel chararray_replace + _unicode_do_replace re-profile Turn 4 5.87× ⋆ ✓ build ✓ tests KEPT Profiler: _unicode_string_length 11% self (per-element length scan) Agent: removes the per-element length scan inside the kernel re-profile Turn 5 5.85× ✓ build ✓ tests Profiler: near memory-bound (libc ∼22% self) Agent: a further tweak that slightly regresses

Speedup (×) # failures

patch, but does not terminate the agentic loop. Instead, it applies the patch to the repository, builds the repository, runs tests and profiles the code, and the resulting feedback is returned to the agent who is then asked to continue optimizing based on the updated feedback on its latest attempt. We run this loop for up to θ iterations per task and report the patch with the highest measured speedup on the provided workload, rather than the last one submitted. This design pushes the agent past its natural stopping point without compromising correctness as every iteration re-validates the patch against the test suite, so a fast-but-broken submission is caught and surfaced as feedback rather than accepted. In addition, the controller helps the LLM agent refine its implementation, or even explore new directions, as the profiler gives immediate feedback on how each change performs. In Fig. 3, we trace how PerfAgent’s loop controller affects an agent’s changes over each turn. The agent’s submitted patch enters at the top; after each submission the controller returns the measured speedup, the outcome of rebuilding and running the selectively chosen tests, and a fresh profiler summary, and the agent then optimizes further. Updated profiler feedback drives the agent from Python-level fast paths (Turns 1–2) to a purpose-built native C kernel (Turn 3) and its refinement (Turn 4)—an optimization at a new abstraction layer. The controller

GSO SWE-fficiency-Lite

6 4

n =5

2

40 20 0

1

2

3

4

5

Controller turn

Fig. 4: Per-turn behavior of the loop controller on GSO (solid) and SWE-fficiency-Lite (dashed). Top: average (harmonic-mean) speedup of the patch submitted at each turn. Bottom: number of instances that hit a validation failure (build error or failing test) before submitting at that turn. The SWE-fficiency-Lite turn-5 average is computed over only n=5 instances, as most exhaust the per-task budget within one or two turns. retains the best patch (Turn 4, 5.87×), not the last (Turn 5, 5.85×, which regresses). In addition, we see how the loop controller affects the speedup and correctness of an agent’s changes over the course of 5 turns in Fig. 4. We report the average (harmonic mean) speedup of the agent’s submitted changes at each turn, along with the number of instances that encountered validation errors such as build errors and test failures before submission at that turn. Not all instances complete all 5 turns, as some instances run out of budget before the full 5 turns complete. Therefore, if an instance does not have a patch for a particular turn, we take the latest patch instead. On GSO, both quantities improve monotonically: the average speedup rises from 2.5× at turn 1 to 6.4× at turn 5, while the number of instances with validation failures falls from 53 to 5, showing that the controller extracts additional, correct speedup well past the agent’s initial stopping point. On SWE-fficiency-Lite this trend is confounded by budget: most instances exhaust the pertask cost limit within one or two turns, so the speedup plateaus on later turns. In particular, PerfAgent often spends many steps and tokens exploring the repository before writing any code. Therefore, with a $5 budget, it is difficult to study the full impact of the control loop on an agent’s changes in SWE-fficiency-Lite. We explore this in more detail in Section IV-E5.

discarded

Fig. 3: A concrete run of the PerfAgent loop (Fig. 1) on the GSO task numpy__numpy-1b861a2 (GPT-5.1) which optimizes the numpy.char.replace API.

D. Selective Validation against Correctness Regressions The third failure mode refers to the correctness regressions due to insufficient testing. An alternative solution is to blindly include all the tests, but running the full test suite is prohibitively expensive on large real-world repositories.

To sufficiently catch any correctness regressions without raw bash commands. When evaluating PerfAgent, we run causing significant overhead, PerfAgent’s controller runs the controller loop for θ = 5 iterations, set a maximum a curated test suite against the agent’s patch after each cost of 5 dollars per task and a step limit of 200 steps iteration, and returns any failures as feedback. Specifically, per task. The prompt used to evaluate PerfAgent is in we use pytest-testmon, a pytest plugin that uses code- Section A of the supplemental material. coverage information to precisely select only the tests We run all agents in a separate AWS ec2 c6i.8xlarge affected by the agent’s changes, minimizing testing cost instance which has 32 CPUs and 64GB of RAM. For without missing any correctness regressions. evaluation, we use an AWS m8i.16xlarge which has 64 Compared to running the repository’s full test suite, CPUs and 256GB of RAM, mirroring the evaluation our selective scheme reduces the number of tests run by setup described in GSO and SWE-fficiency [4], [5]. When 66%–99% on GSO and by 47%–98% on SWE-fficiency-Lite. running GPT-5.1, we use high reasoning effort on all Details about the cost savings are described in Section experiments, matching the experimental settings used in B of the supplemental material. More importantly, we GSO and SWE-fficiency [4], [5]. When running Kimi-K2, observe a clear increase in correctness of the final patch we use the default recommended settings [17], [18]. with our selective and curated test validation approach, 1) Evaluation Metrics: GSO and SWE-fficiency report in comparison to the ad-hoc testing done by the coding different evaluation metrics based on the speedup ratios Speedup agent. We provide additional details in Section IV, with SR = Speedup Agent achieved on each task. GSO uses Human results in Table II. Opt@1: the percentage of tasks that are correct and have a speedup ratio of at least 0.95. For SWE-fficiency, each IV. E VALUATION Our evaluation is designed to test both the end-to-end task is assigned a score based on the speedup ratio. If the not pass correctness, then the score effectiveness of PerfAgent and the feedback mechanisms agent’s changes do 1 assigned is SpeedupHuman . Otherwise, it is equal to the that explain its gains. speedup ratio. The overall score for an LLM on the SWEWe organize the main results around the following fficiency benchmark is the harmonic average of the scores three questions. RQ1: does PerfAgent improve repositoryfor each task. While this metric rewards LLM agents level optimization performance over strong agent basefor producing patches that surpass the performance lines? RQ2: which parts of the workflow (profiler feedof a human expert baseline, it can heavily skew the back, objective-driven iteration, and validation) account overall performance due to outliers in the benchmark. For for the improvement? RQ3: how does iterative profiler example, producing an incorrect patch on a task in which feedback change the agent’s optimization behavior? the human expert baseline achieves a large speedup over Further, we run analyses to rule out alternative explathe base repository is much more punishing compared nations and characterize limitations: an oracle best@5 to producing an incorrect patch on a task in which the comparison tests whether the gains are merely due to human expert baseline achieves a small speedup. more test-time sampling, open-source model results check Therefore, for this paper, we report all benchmark whether the workflow transfers beyond GPT-5.1, rewardresults using GSO’s Opt@1 score. In addition, we also hacking analysis shows why optimization loops require report Correctness, the percentage of tasks that pass the robust measurement and hack-adjusted scoring, and a benchmark’s correctness tests, and Speedup@1 numbers budget sensitivity analysis characterizes how the per-task (Sp@1), the percentage of tasks that achieve at least some cost cap constrains iteration on SWE-fficiency-Lite. speedup over the base repository. For Sp@1, we use A. Experimental Setup a threshold of 1.2× speedup, taken from GSO, as the We evaluate PerfAgent on GSO and SWE-fficiency-Lite threshold is large enough to remove patches that may using GPT 5.1, a frontier model, and Kimi-K2, an open- achieve speedup solely due to timing variance [4]. 2) Hack Detection: GSO and SWE-fficiency both employ source model [15], [16]. For Kimi, on GSO we use KimiK2-0711 and on SWE-fficiency-Lite, we use Kimi-K2-0905, different complementary methods to prevent LLM agents as those are the specific versions each benchmark eval- from cheating on code-optimization benchmarks. GSO uated on. Each benchmark provides patches produced uses an LLM-as-a-judge to detect reward hacking, and by an agent implemented with OpenHands, which we SWE-fficiency identifies when LLM agents use stackintrospection to change behavior when it is being exevaluate to obtain baseline results [10]. We implement PerfAgent on top of Mini-SWE-Agent ecuted in a timing loop versus a correctness test. [7], a lightweight single-agent harness with tool access. We found that SWE-fficiency’s detector, when used We use two variants depending on the model: for GPT- alone, missed common reward-hacking patterns. The 5.1 we use the default Mini-SWE-Agent with access to most common hack we observed was caching results bash; for Kimi-K2 we augment it with a structured file inside benchmark timing loops. SWE-fficiency’s harness editing tool from OpenHands [10], since open-source runs each evaluation in a separate Python process to models struggle to view and edit files reliably through prevent run-to-run caching of results, but within a single

TABLE II: Results on the GSO and SWE-fficiency-Lite benchmarks for GPT-5.1. Correctness, Sp@1, Opt@1 score, and the hack-adjusted score are reported for each benchmark. Method

Correctness

Sp@1

88.2 89.2 91.2 96.1 95.1 96.1

46.1 48.0 67.6 47.1 70.6 77.5

20.6 18.6 33.3 24.5 36.3 44.1

19.6 17.7 29.4 20.6 34.4 39.2

82 80 80 93 83 90

47 59 64 64 73 83

27 39 49 49 59 75

26 39 46 46 57 74

GSO OpenHands Codex A L (loop only) + Tests + Profiler PerfAgent SWE-fficiency-Lite OpenHands Codex A L (loop only) + Tests + Profiler PerfAgent

Opt@1 Hack-Adj.

TABLE III: Per-task win/loss record of PerfAgent against the OpenHands baseline. An agent wins a task when it matches or exceeds the human expert baseline (and passes the hack detector) while the other does not; the remaining tasks are ties that both or neither agent optimizes. Per-task outcome

GSO

SWE-fficiency-Lite

PerfAgent wins OpenHands wins

27 7

52 4

Tie, both optimize Tie, neither optimizes

13 55

22 22

Total tasks

102

100

Per-Task Win/Loss Analysis. Here we break our headline GPT-5.1 comparison down task by task, tabulating where PerfAgent and the OpenHands baseline each match or exceed the human expert baseline on GSO and SWErun timing is performed multiple times in a loop and fficiency-Lite (Table III). Because both agents run on is vulnerable to caching. Indeed, we observed many the same tasks, this is a paired comparison: tasks where instances where agents exploited this structure to reduce they agree—both optimize, or neither—say nothing about reported benchmark timings. GSO’s LLM-as-a-judge de- which agent is better, so significance comes entirely from tector is better able to detect this style of reward hacking the decisive tasks, where exactly one agent optimizes. that is missed by SWE-fficiency’s detector. However, SWE- We perform a more in-depth analysis of the instances in fficiency’s detector is still a useful addition to GSO’s which the OpenHands baseline performs better in Section LLM judge because it is able to identify a unique form of D of the supplemental material. reward hacking where an agent uses stack introspection C. RQ2: Which Components Drive the Improvement to run different versions of a program during benchmarks We now turn to RQ2: which parts of the workflow and correctness tests. account for the improvement? We isolate the contribution We implemented a hack detector that combines the of each component in PerfAgent’s loop controller by hack detection methods used by SWE-fficiency and GSO. running several agents with PerfAgent’s controller for 5 For both benchmarks, we report the results of running iterations. We evaluate on 3 separate agents which we the combined hack detector as the hack-adjusted score. describe below. B. RQ1: End-to-End Optimization Performance • A L : An LLM agent with a stripped-down version of PerfAgent’s loop controller, where only timing feedback We first address RQ1: does PerfAgent improve is given to the LLM at the end of each turn. repository-level optimization performance over strong • A L +Tests: An LLM agent with PerfAgent’s loop conagent baselines? We report end-to-end results against the troller that provides timing feedback and feedback from OpenHands and Codex baselines (Table II), then break running the curated test suite. the comparison down task by task (Table III). • A +Profiler: An LLM agent with PerfAgent’s loop L GSO and SWE-fficiency-Lite Results. On GSO and controller that provides timing and profiler feedback. SWE-fficiency-Lite, we compare PerfAgent against the We compare these 3 agents to PerfAgent which proOpenHands baseline provided by each benchmark, as well as running on Codex, a frontier agentic harness vides a combination of timing feedback, profiler feedback, developed by OpenAI [11], and the results are shown in and feedback from running the curated test suite. Table II. For each benchmark, we report the Opt@1 and Table II provides the results of our comparison of these hack-adjusted score. We see that on both benchmarks, agents with the OpenHands baseline and PerfAgent. We PerfAgent significantly outperforms both the OpenHands find that A L outperforms both the OpenHands and Codex and Codex baselines. The Opt@1 scores on GSO are baselines in terms of Opt@1 and hack-adjusted scores, much lower compared to SWE-fficiency-Lite as GSO has showing the value of test-time scaling with runtime hidden performance tests which evaluate the agent’s feedback. The addition of correctness-test feedback in changes on different workloads, while SWE-fficiency A L +Tests increases correctness on both benchmarks, but evaluates performance solely based on the provided fails to improve Opt@1 or hack-adjusted scores on GSO script. Therefore, in GSO, the agent’s changes may yield and SWE-fficiency. In fact, on GSO the addition of high speedups on the provided workload, but fail to correctness tests leads to markedly worse performance generalize to unseen inputs. on these metrics. The addition of profiling feedback in

TABLE IV: GSO results broken down by language: 42 Python-only tasks vs. 60 non-Python-only tasks. Values reported are the percentage of tasks that match (or exceed) the human expert baseline and pass the hack detector. Method OpenHands A L (loop only) + Tests + Profiler PerfAgent

Python only

Non-Python-only

31.0 45.2 35.7 47.6 50.0

11.7 18.3 10.0 25.0 31.7

A L +Profiler improves Opt@1 and hack-adjusted scores on both benchmarks relative to A L , but produces a correct patch less often than A L +Tests. The best Opt@1 and hack-adjusted scores are obtained by PerfAgent which receives both profiler and correctness test feedback. The combination of both correctness test and profiler feedback in PerfAgent achieves the best Opt@1 and hack-adjusted scores. PerfAgent is able to pursue ambitious optimizations that are informed by profiling feedback, and the correctness test feedback helps it avoid correctness issues that arise from complex code changes. D. RQ3: Effect of Profiler Feedback We next study RQ3: how does iterative profiler feedback change the agent’s optimization behavior? We first show in aggregate that profiler feedback pushes the agent across abstraction boundaries, then trace a single task through the loop as a case study. 1) Optimization Across Abstraction Boundaries: A key driver of these gains is that profiler feedback pushes the agent across abstraction boundaries and allows the agent to better understand the performance of native code. PerfAgent modifies low-level code (C, C++, Cython, or Rust) on 48% of instances, compared to 31% for the OpenHands baseline, which leads to better results on these lower-level tasks. To further demonstrate this, we also report the results of GSO, broken down by Pythononly tasks and the remaining non-Python-only tasks in Table IV. On GSO, 41.2% of tasks in GSO involve a human expert baseline that only modifies Python code files, and on the remaining tasks, the human expert baseline additionally modifies low-level code (C, C++, Cython, Rust) files. Here, we see that agents perform worse on these low-level tasks compared to the Python-only tasks. However, on the agents that use a profiler, performance on the non-Python-only tasks significantly improves.2 2) Case Study: Impact of PerfAgent’s Control Loop: We further expand on the example shown in Fig. 3 to demonstrate how PerfAgent enables an agent to iteratively improve its implementation. We look at task: numpy__numpy-1b861a2 from GSO which aims to optimize numpy.char.replace to demon2We do not perform the same breakdown on SWE-fficiency-Lite, because 89% of its tasks involve Python-only changes.

TABLE V: Results on the GSO and SWE-fficiency-Lite benchmarks for Kimi-K2. Correctness, Sp@1, Opt@1, and hack-adjusted scores are reported. Method

Correctness

Sp@1

64.7 89.2

23.5 31.4

9.8 10.8

3.9 5.9

74 87

39 67

26 46

22 32

GSO OpenHands PerfAgent SWE-fficiency-Lite OpenHands PerfAgent

Opt@1 Hack-Adj.

strate the impact of PerfAgent’s control loop on the agent’s implementation. Fig. 3 traces how the agent’s implementation evolves across the 5 controller iterations. The initial profiler report attributes most of the runtime to the replace call site and the generic _vec_string dispatch path, and the agent responds with a C-level fast path for scalar arguments (Turn 1, 1.40×) and then a Python-level fast path that avoids an array-conversion step (Turn 2, 2.36×). After each attempt, the updated profiler feedback continues to report large overheads from per-element scalar creation (PyArray_Scalar) and object reference counting that cannot be removed at the Python or genericdispatch layer. This pushes the agent to implement a dedicated native kernel, chararray_replace, in C (Turn 3), more than doubling the speedup to 5.56×. The feedback on the new kernel then localizes a per-element stringlength scan (_unicode_string_length), which the agent removes to reach its best patch (Turn 4, 5.87×). The final attempt explores a further change that slightly regresses (Turn 5, 5.85×), so the controller reports the Turn-4 patch rather than the last submission, illustrating why we retain the best measured patch instead of the final one. As in the aggregate trend (Fig. 4), the updated profiler feedback lets the agent discover an optimization at an entirely new abstraction layer, matching the approach taken by the human expert baseline. By contrast, the OpenHands baseline stops at the first fast path it finds at the original abstraction layer, as that already achieves speedup over the base repository. E. Additional Analyses Finally, we run additional analyses to rule out alternative explanations for PerfAgent’s gains and to characterize its limitations. 1) Test-Time Scaling: The improved performance of PerfAgent is not merely due to its use of additional test-time compute. To demonstrate this, we evaluated PerfAgent against a best-case test-time scaling approach that runs the OpenHands workflow 5 times and then uses an oracle to pick the best result. The use of an oracle to pick the best patch, in this setting, is the “best-case” for test-time scaling because it uses the results of hidden performance and correctness tests from the evaluation harness that are not made available to PerfAgent or other

TABLE VI: Comparison with test-time scaling running on OpenHands best@5 with an oracle judge. Correctness, Sp@1, Opt@1, Hack-adjusted score and cost in terms of average $ per task are reported for each benchmark.

across all profiling iterations. While this introduces some variance in the reported runtime, we find it to be small (< 3% of overall runtime on average). The motivation is to discourage reward hacking: an agent receiving average-run timing as feedback can cache the result of Method Correctness Sp@1 Opt@1 Hack-adj. Cost ($) an expensive operation on the first run and replay it on GSO subsequent runs, collapsing the average runtime without OpenHands best@5 99.0 67.6 26.5 26.5 11.01 performing any real optimization. The loop-only ablation PerfAgent 96.1 77.5 44.1 39.2 2.88 A L makes this concrete. SWE-fficiency-Lite On SWE-fficiency-Lite, the loop-only agent A L origOpenHands best@5 99 93 71 68 9.91 PerfAgent 90 83 75 74 4.25 inally achieved a 62% Opt@1 score but a 44% hackadjusted score as 18 of its high-performing patches are TABLE VII: Results comparing PerfAgent to an agent labeled as hacks by our hack detector. We find that 14 given instructions in the prompt on how to use a profiler. of the 18 hacks are related to maintaining some form of persistent state or caching results. In the provided Method Correctness Sp@1 Opt@1 Hack-Adj. test script, SWE-fficiency times the task by running the GSO provided workload many times, so an agent that caches OpenHands 88.2 46.1 20.6 19.6 an expensive result on the first run and replays it on subAP 96.1 67.6 17.6 16.7 sequent runs collapses the average runtime and appears PerfAgent 96.1 77.5 44.1 39.2 to achieve a large speedup without optimizing anything. SWE-fficiency-Lite Therefore, we remove this by separately running A L with OpenHands 82 47 27 26 a test script that only times the execution of the first run. AP 85 75 45 44 After making this change, we re-run A L with the updated PerfAgent 90 83 75 74 test script, and the results are shown in Table II. We see that the number of hacks decreases significantly, from baselines during inference. The results of this experiment 18 to 3, indicating that if the evaluation harness is not are provided in Table VI. On GSO, PerfAgent achieves constructed properly, then it can lead to reward hacking a better Opt@1 score than OpenHands best@5 (44.1% by the agent. We perform a more in-depth analysis of each vs 26.5%) and a better hack-adjusted score (39.2% vs reward hacking instance in Section C of the supplemental 26.5%) while costing over 3× less. On SWE-fficiency-Lite, material. 5) Ablation on SWE-fficiency-Lite Budget per Task: In the results are similar with PerfAgent achieving a better Opt@1 score (75% vs 71%) and hack-adjusted score (74% Section IV-D2, we discussed that many of the instances vs 68%) than OpenHands best@5 while costing 2× less. in SWE-fficiency-Lite ran out of budget before being able 2) Open-Source Models: We also evaluate two open- to submit a patch on later turns, which led to speedups source models: Kimi-K2-0711 on GSO, and Kimi-K2-0905 plateauing, as compared to GSO, the agent spends more on SWE-fficiency-Lite, as those are the model versions turns and tokens reading files and running tests, and we show this in Table VIII. Therefore, we separately used in the OpenHands baselines. The results are shown in Table V and show that evaluate SWE-fficiency-Lite with a $10 budget per task PerfAgent outperforms the OpenHands baseline on both and report the results below in Table IX. Here, many more benchmarks. Overall, we find that open-source models instances submitted attempts beyond turn 2, and we see generally struggle to identify the relevant information the speedups steadily increasing over time compared to from the profiling output and implement the optimiza- running with a $5 budget. We also observe improvements tions suggested from the profiler feedback. However, in performance on the SWE-fficiency-Lite compared to despite this, PerfAgent still outperforms the OpenHands running with a $5 budget, achieving an Opt@1 score of 79% compared to 75%, and a hack-adjusted score of 77% baselines on both benchmarks. 3) Profiler Instruction in Prompt: An alternative to compared to 74%. PerfAgent’s workflow that runs the profiler and provides TABLE VIII: Average number of file read commands and a curated summary to the LLM is to give the agent test commands per instance, as well as the average output, instructions in the prompt on how to use the profiler, in number of tokens, of the read and test commands per and have the agent decide when to use it. We separately instance in GSO and SWE-fficiency-Lite. evaluate an agent A P that is given instructions in the prompt detailing how PerfAgent uses a profiler, and the # File # Test Read Test Benchmark Reads Invoc. Tokens Tokens results are shown in Table VII. 4) Reward Hacking: A key design choice in PerfAgent GSO 35.4 7.4 31,399 2,990 SWE-fficiency-Lite 42.7 6.5 41,911 7,294 is that when benchmarking, PerfAgent only times the execution time of the first run rather than the average

TABLE IX: Average (harmonic mean) Speedup of an agent’s submitted change at each turn and # of instances that submitted a patch at that turn on SWE-fficiency-Lite. The remaining instances that did not submit a patch ran out of budget. $5 budget per task Turn

Avg. Speedup

1 2 3 4 5

1.697× 2.486× 2.638× 2.667× 2.672×

$10 budget per task

# Submitted

Avg. Speedup

# Submitted

89 69 40 17 6

2.727× 3.492× 3.888× 3.945× 4.023×

98 84 67 38 12

V. R ELATED W ORK A. LLM Coding Benchmarks Kernel-level benchmarks. HumanEval and MBPP target completion of simple Python programs [19], [20], while LiveCodeBench and BigCodeBench target competitive-programming problems [21]–[23]. EvalPerf, Effibench, and Enamel evaluate the efficiency of LLMgenerated Python programs [24]–[26], and ParEval evaluates correct and efficient parallel code [27]. For kernels, AlgoTune optimizes routines from numerical computations [28], KernelBench and MultiKernelBench write optimized CUDA kernels from a PyTorch specification across architectures [29], [30], and TritonBench generates Triton kernels [31]. Repository-level and Long-horizon benchmarks. SWEBench and its extensions evaluate an LLM’s ability to resolve issues on real-world repositories [1]–[3], [32]– [34]. ProgramBench [35] and FrontierSWE [36] evaluate LLMs on long-horizon software engineering tasks such as writing a SQL engine from scratch. SWE-Perf, SWEfficiency, and GSO instead target the performance of the agent’s changes, not only correctness [4]–[6]

designs have also been explored at finer granularity: PRAGMA routes hardware-profiling signals through a multi-agent loop for kernel generation [49], ProfilingAgent guides structured pruning and quantization of neural networks [50], and TritonForge optimizes Triton kernels [51]; PEACE draws on historical edits and external knowledge to optimize real-world repositories [52]. Our work differs on two axes. First, we target real-world repositories with native extensions (C, C++, Cython, Rust) and hold the agent to a human-expert bar, rather than synthetic programs, isolated kernels, or model-compression objectives. Second, we show sample-efficiency: our loop surpasses an oracle best-of-five baseline at a fraction of the cost, isolating feedback quality rather than added compute as the driver of the gains. VI. T HREATS TO VALIDITY

Construct and internal validity. PerfAgent and both benchmarks measure performance on the single workload given to the agent, so a patch that is fast there is not guaranteed to be a general optimization. GSO mitigates this with hidden performance tests, whereas SWE-fficiencyLite times only the provided script and is thus easier to game—the source of the reward hacking we observe. We report hack-adjusted scores from a combined detector (Section IV-A2) and manually reviewed every flagged instance, but an LLM-as-judge detector may still miss subtle exploits or over-flag legitimate fast paths. Because our controller validates with pytest-testmon, which tracks only Python coverage, a native-extension regression can slip between iterations; this deflates rather than inflates our score, since each benchmark’s own correctness suite runs at evaluation time. External and conclusion validity. Our findings span two benchmarks, roughly a dozen Python-centric repositories, two models (GPT-5.1 and Kimi-K2), and a single base harness (Mini-SWE-Agent), and may not transfer to B. LLMs for Code Optimization other languages, models, or harnesses; although PerfASBLLM proposes a search-based framework for itergent is harness-independent by construction, we have not ative optimization of competitive-programming probmeasured its gains on OpenHands or Codex. Results are lems [37], and PIE finetunes LLMs on curated competition also single-run, so small differences fall within noise—the data to improve code optimization [38]. LessonL is a Kimi-K2 GSO gap (Table V) is about two tasks and should multi-agent framework in which agents learn from each not be over-interpreted. We therefore base our headline other’s experience optimizing computational kernels [39]. GPT-5.1 claims on a task-by-task comparison that is Stark, Astra, Pike, and KernelFalcon take multi-agent apstatistically significant on both benchmarks (Table III). proaches to generating or optimizing CUDA kernels [40]– [43], while CudaForce uses hardware feedback [44]. PEAK VII. C ONCLUSION AND F UTURE W ORK optimizes GPU kernels across architectures via naturallanguage transformation specifications [45], and AccelOpt We presented PerfAgent, a profiler-guided, verifier-inand AutoComp target kernels for specialized accelerators the-loop workflow that turns an off-the-shelf coding agent [46], [47]. into a repository-level performance optimizer. PerfAgent Profiler-guided and iterative optimization. Closest attacks three failure modes of general-purpose agents on to our setting, POLO couples a profiler with an LLM optimization tasks: missing the real bottleneck, stopping agent loop, using an iterative weighting algorithm to at the first passing patch, and under-testing complex localize bottlenecks while separate generator and decision changes, with three matching components: a curated agents propose and select edits [48]. Profiler-in-the-loop profiler summary, an objective-driven loop controller that

retains the fastest correct patch, and selective re-validation after every iteration. PerfAgent can be run on top of more capable singleand multi-agent harnesses such as Codex or Claude Code, which we plan to evaluate. Other directions include extending selective testing to native-extension coverage , hardening the profiling and timing template against new classes of reward hacking, and broadening to additional languages, models. More broadly, we view performance engineering as a distinct and demanding axis of agentic software engineering; one that rewards agents able to reason about how code runs, not merely whether it is correct. VIII. A CKNOWLEDGMENTS This research was sponsored in-part by the MIT-IBM Watson AI Lab. This research was sponsored in-part by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. R EFERENCES [1] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=VTF8yNQM66 [2] X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, V. Bharadwaj, J. Holm, R. Aluri, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler, “Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?” 2025. [Online]. Available: https://arxiv.org/abs/2509.16941 [3] D. Zan, Z. Huang, W. Liu, H. Chen, L. Zhang, S. Xin, L. Chen, Q. Liu, X. Zhong, A. Li, S. Liu, Y. Xiao, L. Chen, Y. Zhang, J. Su, T. Liu, R. Long, K. Shen, and L. Xiang, “Multi-swe-bench: A multilingual benchmark for issue resolving,” 2025. [Online]. Available: https://arxiv.org/abs/2504.02605 [4] M. Shetty, N. Jain, J. Liu, V. Kethanaboyina, K. Sen, and I. Stoica, “Gso: Challenging software optimization tasks for evaluating swe-agents,” 2025. [Online]. Available: https: //arxiv.org/abs/2505.23671 [5] J. J. Ma, M. Hashemi, A. Yazdanbakhsh, K. Swersky, O. Press, E. Li, V. J. Reddi, and P. Ranganathan, “Swe-fficiency: Can language models optimize real-world repositories on real workloads?” 2025. [Online]. Available: https://arxiv.org/abs/2511.06090 [6] X. He, Q. Liu, M. Du, L. Yan, Z. Fan, Y. Huang, Z. Yuan, and Z. Ma, “Swe-perf: Can language models optimize code performance on real-world repositories?” 2025. [Online]. Available: https://arxiv.org/abs/2507.12415 [7] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://arxiv.org/abs/2405.15793

[8] T. R. Team, P. Gao, Z. Tian, X. Meng, X. Wang, R. Hu, Y. Xiao, Y. Liu, Z. Zhang, J. Chen, C. Gao, Y. Lin, Y. Xiong, C. Peng, and X. Liu, “Trae agent: An llm-based agent for software engineering with test-time scaling,” 2025. [Online]. Available: https://arxiv.org/abs/2507.23370 [9] J. Ganhotra, S. Serhan, A. A. Nassar, A. Shinnar, Z. Nevo, and M. Hirzel, “Resolving java code repository issues with iswe agent,” 2026. [Online]. Available: https://arxiv.org/abs/2603.11356 [10] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig, “OpenHands: An Open Platform for AI Software Developers as Generalist Agents.” [Online]. Available: https://arxiv.org/abs/2407.16741 [11] OpenAI, “Codex CLI: Lightweight coding agent that runs in your terminal,” https://github.com/openai/codex, 2025. [12] B. Frederickson, “Sampling profiler for Python programs,” 2026. [Online]. Available: https://github.com/benfred/py-spy [13] N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” 2023. [Online]. Available: https://arxiv.org/abs/2303. 11366 [14] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self-refine: Iterative refinement with self-feedback,” 2023. [Online]. Available: https://arxiv.org/abs/2303.17651 [15] OpenAI, “GPT-5.1: A smarter, more conversational ChatGPT,” https://openai.com/index/gpt-5-1/, Nov 2025. [16] K. Team, Y. Bai, Y. Bao, Y. Charles, C. Chen, G. Chen, H. Chen, H. Chen, J. Chen, N. Chen, R. Chen, Y. Chen, Y. Chen, Y. Chen, Z. Chen, J. Cui, H. Ding, M. Dong, A. Du, C. Du, D. Du, Y. Du, Y. Fan, Y. Feng, K. Fu, B. Gao, C. Gao, H. Gao, P. Gao, T. Gao, Y. Ge, S. Geng, Q. Gu, X. Gu, L. Guan, H. Guo, J. Guo, X. Hao, T. He, W. He, W. He, Y. He, C. Hong, H. Hu, Y. Hu, Z. Hu, W. Huang, Z. Huang, Z. Huang, T. Jiang, Z. Jiang, X. Jin, Y. Kang, G. Lai, C. Li, F. Li, H. Li, M. Li, W. Li, Y. Li, Y. Li, Y. Li, Z. Li, Z. Li, H. Lin, X. Lin, Z. Lin, C. Liu, C. Liu, H. Liu, J. Liu, J. Liu, L. Liu, S. Liu, T. Y. Liu, T. Liu, W. Liu, Y. Liu, Y. Liu, Y. Liu, Y. Liu, Z. Liu, E. Lu, H. Lu, L. Lu, Y. Luo, S. Ma, X. Ma, Y. Ma, S. Mao, J. Mei, X. Men, Y. Miao, S. Pan, Y. Peng, R. Qin, Z. Qin, B. Qu, Z. Shang, L. Shi, S. Shi, F. Song, J. Su, Z. Su, L. Sui, X. Sun, F. Sung, Y. Tai, H. Tang, J. Tao, Q. Teng, C. Tian, C. Wang, D. Wang, F. Wang, H. Wang, H. Wang, J. Wang, J. Wang, J. Wang, S. Wang, S. Wang, S. Wang, X. Wang, Y. Wang, Y. Wang, Y. Wang, Y. Wang, Y. Wang, Z. Wang, Z. Wang, Z. Wang, Z. Wang, C. Wei, Q. Wei, H. Wu, W. Wu, X. Wu, Y. Wu, C. Xiao, J. Xie, X. Xie, W. Xiong, B. Xu, J. Xu, L. H. Xu, L. Xu, S. Xu, W. Xu, X. Xu, Y. Xu, Z. Xu, J. Xu, J. Xu, J. Yan, Y. Yan, H. Yang, X. Yang, Y. Yang, Y. Yang, Z. Yang, Z. Yang, Z. Yang, H. Yao, X. Yao, W. Ye, Z. Ye, B. Yin, L. Yu, E. Yuan, H. Yuan, M. Yuan, S. Yuan, H. Zhan, D. Zhang, H. Zhang, W. Zhang, X. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Z. Zhang, H. Zhao, Y. Zhao, Z. Zhao, H. Zheng, S. Zheng, L. Zhong, J. Zhou, X. Zhou, Z. Zhou, J. Zhu, Z. Zhu, W. Zhuang, and X. Zu, “Kimi k2: Open agentic intelligence,” 2026. [Online]. Available: https://arxiv.org/abs/2507.20534 [17] Moonshot AI, “Kimi-K2-Instruct,” 2025. [Online]. Available: https://huggingface.co/moonshotai/Kimi-K2-Instruct [18] ——, “Kimi-K2-Instruct-0905,” 2025. [Online]. Available: https: //huggingface.co/moonshotai/Kimi-K2-Instruct-0905 [19] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107.03374

[20] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732 [21] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” 2024. [Online]. Available: https://arxiv.org/abs/2403.07974 [22] Z. Zheng, Z. Cheng, Z. Shen, S. Zhou, K. Liu, H. He, D. Li, S. Wei, H. Hao, J. Yao, P. Sheng, Z. Wang, W. Chai, A. Korolova, P. Henderson, S. Arora, P. Viswanath, J. Shang, and S. Xie, “Livecodebench pro: How do olympiad medalists judge llms in competitive programming?” 2025. [Online]. Available: https://arxiv.org/abs/2506.11928 [23] T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. Gong, T. Hoang, A. R. Zebaze, X. Hong, W.-D. Li, J. Kaddour, M. Xu, Z. Zhang, P. Yadav, N. Jain, A. Gu, Z. Cheng, J. Liu, Q. Liu, Z. Wang, B. Hui, N. Muennighoff, D. Lo, D. Fried, X. Du, H. de Vries, and L. V. Werra, “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” 2025. [Online]. Available: https://arxiv.org/abs/2406.15877 [24] J. Liu, S. Xie, J. Wang, Y. Wei, Y. Ding, and L. Zhang, “Evaluating language models for efficient code generation,” 2024. [Online]. Available: https://arxiv.org/abs/2408.06450 [25] D. Huang, Y. Qing, W. Shang, H. Cui, and J. M. Zhang, “Effibench: Benchmarking the efficiency of automatically generated code,” 2025. [Online]. Available: https://arxiv.org/abs/2402.02037 [26] R. Qiu, W. W. Zeng, J. Ezick, C. Lott, and H. Tong, “How efficient is llm-generated code? a rigorous & high-standard benchmark,” 2025. [Online]. Available: https://arxiv.org/abs/2406.06647 [27] D. Nichols, J. H. Davis, Z. Xie, A. Rajaram, and A. Bhatele, “Can large language models write parallel code?” in Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, ser. HPDC ’24. ACM, June 2024, p. 281–294. [Online]. Available: http://dx.doi.org/10.1145/3625549.3658689 [28] O. Press, B. Amos, H. Zhao, Y. Wu, S. K. Ainsworth, D. Krupke, P. Kidger, T. Sajed, B. Stellato, J. Park, N. Bosch, E. Meril, A. Steppi, A. Zharmagambetov, F. Zhang, D. Perez-Pineiro, A. Mercurio, N. Zhan, T. Abramovich, K. Lieret, H. Zhang, S. Huang, M. Bethge, and O. Press, “Algotune: Can language models speed up general-purpose numerical programs?” 2025. [Online]. Available: https://arxiv.org/abs/2507.15887 [29] A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Ré, and A. Mirhoseini, “Kernelbench: Can llms write efficient gpu kernels?” 2025. [Online]. Available: https://arxiv.org/abs/2502.10517 [30] Z. Wen, Y. Zhang, Z. Li, Z. Liu, L. Xie, and T. Zhang, “Multikernelbench: A multi-platform benchmark for kernel generation,” 2025. [Online]. Available: https://arxiv.org/abs/2507. 17773 [31] J. Li, S. Li, Z. Gao, Q. Shi, Y. Li, Z. Wang, J. Huang, H. Wang, J. Wang, X. Han, Z. Liu, and M. Sun, “Tritonbench: Benchmarking large language model capabilities for generating triton operators,” 2025. [Online]. Available: https://arxiv.org/abs/2502.14752 [32] L. Zhang, S. He, C. Zhang, Y. Kang, B. Li, C. Xie, J. Wang, M. Wang, Y. Huang, S. Fu, E. Nallipogu, Q. Lin, Y. Dang, S. Rajmohan, and D. Zhang, “Swe-bench goes live!” 2025. [Online]. Available: https://arxiv.org/abs/2505.23419 [33] W. Huang, C. Lee, L. Tng, and S. Ge, “Deepswe: Measuring frontier coding agents on original, long-horizon engineering tasks,” 2026. [Online]. Available: https://github.com/datacurve-ai/deep-swe [34] E. Lu et al., “Introducing FrontierCode,” https://cognition.ai/blog/ frontier-code, Jun. 2026, cognition AI blog, published 2026-06-08. [35] J. Yang, K. Lieret, J. Ma, P. Thakkar, D. Pedchenko, S. Sootla, E. McMilin, P. Yin, R. Hou, G. Synnaeve, D. Yang, and O. Press, “Programbench: Can language models rebuild programs from scratch?” 2026. [Online]. Available: https: //arxiv.org/abs/2605.03546 [36] E. Chu, R. Agarwal, A. Thangamuthu, B. Graham, J. Mattern, F. Jiang, P. Cento, S. Jain, M. Abbasi, M. H. Rezaei, G. Wang, A. Zhang, S. Guo, K. Nguyen, A. Bidgoli, A. Dalmia, A. Dankar, A. Vaddela, C. Chen, K. Kumar, K. Vaish, N. Pour, R. Kondra, S. Badiyani, S. Giri, S. Das, S. Gaikwad, S. Shah, V. Dilawari,

and V. Agarwal, “Frontierswe,” Proximal Blog, 2026. [Online]. Available: https://frontierswe.com/blog [37] S. Gao, C. Gao, W. Gu, and M. Lyu, “Search-based llms for code optimization,” 2024. [Online]. Available: https://arxiv.org/abs/ 2408.12159 [38] A. Shypula, A. Madaan, Y. Zeng, U. Alon, J. Gardner, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh, “Learning performance-improving code edits,” 2024. [Online]. Available: https://arxiv.org/abs/2302.07867 [39] Y. Liu, R. Deng, T. Kaler, X. Chen, C. E. Leiserson, Y. Ma, and J. Chen, “Lessons learned: A multi-agent framework for code llms to learn and improve,” 2025. [Online]. Available: https://arxiv.org/abs/2505.23946 [40] J. Dong, Y. Yang, T. Liu, Y. Wang, F. Qi, V. Tarokh, K. Rangadurai, and S. Yang, “Stark: Strategic team of agents for refining kernels,” 2025. [Online]. Available: https://arxiv.org/abs/2510.16996 [41] A. Wei, T. Sun, Y. Seenichamy, H. Song, A. Ouyang, A. Mirhoseini, K. Wang, and A. Aiken, “Astra: A multi-agent system for gpu kernel performance optimization,” 2025. [Online]. Available: https://arxiv.org/abs/2509.07506 [42] L. e. a. Wang, “KernelFalcon: Deep agent architecture for autonomous GPU kernel generation,” PyTorch Blog, Nov. 2025. [43] K. Nagaitsev, L. Grbcic, S. Williams, and C. Iancu, “Optimizing pytorch inference with llm-based multi-agent systems,” 2026. [Online]. Available: https://arxiv.org/abs/2511.16964 [44] Z. Zhang, R. Wang, S. Li, Y. Luo, M. Hong, and C. Ding, “Cudaforge: An agent framework with hardware feedback for cuda kernel optimization,” 2025. [Online]. Available: https://arxiv.org/abs/2511.01884 [45] M. U. Tariq, A. Jangda, A. Moreira, M. Musuvathi, and T. Sorensen, “Peak: A performance engineering ai-assistant for gpu kernels powered by natural language transformations,” 2025. [Online]. Available: https://arxiv.org/abs/2512.19018 [46] G. Zhang, S. Zhu, A. Wei, Z. Song, A. Nie, Z. Jia, N. Vijaykumar, Y. Wang, and K. Olukotun, “Accelopt: A self-improving llm agentic system for ai accelerator kernel optimization,” 2025. [Online]. Available: https://arxiv.org/abs/2511.15915 [47] C. Hong, S. Bhatia, A. Cheung, and Y. S. Shao, “Autocomp: A powerful and portable code optimizer for tensor accelerators,” 2025. [Online]. Available: https://arxiv.org/abs/2505.18574 [48] J. Bai, R. Xu, S. Wu, D. Yang, J. Zhao, and G. Chen, “POLO: An LLM-powered project-level code performance optimization framework,” in Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, 2025, pp. 7319–7328. [49] K. Lei, H. Yang, H. Zhang, X. You, K. Zhang, Z. Luan, Y. Liu, and D. Qian, “Pragma: A profiling-reasoned multi-agent framework for automatic kernel optimization,” 2025. [Online]. Available: https://arxiv.org/abs/2511.06345 [50] S. Jafari, A. Sarkar, M. Bilwal, and A. Jannesari, “Profilingagent: Profiling-guided agentic reasoning for adaptive model optimization,” 2025. [Online]. Available: https://arxiv.org/abs/2509.05584 [51] H. Li, K. Man, P. Kanuparthy, H. Chen, W. Sun, S. Tallam, C. Zhu, K. Zhu, and Z. Qian, “Tritonforge: Profiling-guided framework for automated triton kernel optimization,” 2025. [Online]. Available: https://arxiv.org/abs/2512.09196 [52] X. Ren, J. Wan, Y. Peng, Z. Liu, M. Liang, D. Chen, W. Jiang, and Y. Li, “Peace: Towards efficient project-level efficiency optimization via hybrid code editing,” 2025. [Online]. Available: https://arxiv.org/abs/2510.17142

A PPENDIX A B ASE P ERFA GENT P ROMPT Prompt for PerfAgent I’ve uploaded a python code repository in the directory /testbed. There is a python script /perf_script.py which shows an example usage of the repository.

Do not build the repository in other ways. Important: Do not run scripts from within /testbed or add /testbed to PYTHONPATH, as it contains source code, and you may run into issues with circular imports. Instead, change to a neutral directory before running: cd / && python /path/to/script.py Never use cd /testbed && python script.py

Can you help me implement the necessary changes to the repository so that the runtime of the test scripts is optimized?

For running scripts, run from the / directory. For example, to run /perf_script.py run cd / && python perf_script.py.

This will be a very difficult problem and may involve modifying multiple files across the repository. Do not be afraid to make large changes to the repository in order to implement your optimization.

To install a package, run uv pip install <package>. For example, to install pytest, run uv pip install pytest. You can execute bash commands and edit files to implement the necessary changes.

I have obtained an initial summary from a profiler showing the bottlenecks. The profiler report summarizes the output of /profile_prob_script.py, which profiles /perf_script.py. Please use the following report to guide your optimization. Report {{ perf_report }} To improve performance, either optimize the bottleneck itself or avoid calling the bottleneck as frequently.

Testing /run_tests.sh runs tests that can be used to verify the correctness of your implementation. Additional hidden tests may be used to evaluate your implementation.

Basic guidelines 1) Your task is to make changes to non-test files in the /testbed directory to improve the performance of the test scripts. Do not make changes to any other files as they will not count towards your submission. 2) Make changes while ensuring the repository is functionally equivalent to the original. 3) Do not overoptimize for just the specific inputs in the provided perf_script.py. Make general performance improvements for the usage scenario shown. 4) You may need to rebuild the repo for your changes to take effect before testing. Some rebuilds may take time to run, so be patient. Running /build.sh rebuilds the repository.

Workflow High-Level Problem Solving Strategy 1) Understand the problem deeply. Carefully read the issue and think critically about what is required. 2) Investigate the codebase. Explore relevant files, search for key functions, and gather context. 3) Develop a clear, step-by-step plan. Break down the fix into manageable, incremental steps. 4) Implement the optimization incrementally. Make small, testable code changes while ensuring the repository is functionally equivalent to the original. 5) Debug as needed. Use debugging techniques to isolate and resolve issues. 6) Test frequently. Run tests after each change to verify correctness. 7) Iterate until the performance has significantly improved and all tests pass. 8) Continue to debug performance. Find if there are any additional bottlenecks in your solution and resolve them. 9) Reflect and validate comprehensively. After tests pass, write additional tests to ensure correctness, and remember there are hidden tests that must also pass before the solution is truly complete. 1. Deeply Understand the Problem • Carefully read the test script and think hard

about a plan to optimize its performance

before coding. 2. Codebase Investigation • Explore relevant files and directories. • Search for key functions, classes, or variables

related to the issue. • Read and understand relevant code snippets. • Identify the root cause of the problem. • Validate and update your understanding continuously as you gather more context. 3. Develop a Detailed Plan • Outline a specific, simple, and verifiable se-

quence of steps to fix the problem. • Break down the fix into small, incremental changes. 4. Making Code Changes • Before editing, always read the relevant file

contents or section to ensure complete context. • If a patch is not applied correctly, attempt to reapply it. • Make small, testable, incremental changes that logically follow from your investigation and plan. 5. Debugging • Make code changes only if you have high

confidence they can solve the problem. • When debugging, try to determine the root cause rather than addressing symptoms. • Debug for as long as needed to identify the root cause and identify a fix. • Use print statements, logs, or temporary code to inspect program state, including descriptive statements or error messages. • To test hypotheses, you can also add test statements or functions. • Revisit your assumptions if unexpected behavior occurs. 6. Testing • Run

tests frequently by running the perf script cd / && python perf_script.py, or create your own tests and run those. • After each change, verify correctness by running relevant tests. • If tests fail, analyze failures and revise your patch. • Ensure all tests pass before finalizing.

7. Final Verification • Confirm the performance has improved. • Review your solution for logic correctness

and robustness. • Iterate until you are extremely confident the fix is complete and all test scripts pass. 8. Improving Performance • Continue to analyze the changed code to find

bottlenecks and resolve them. iterating on the test script (if needed) to test the performance of your changes on various inputs.

• Continue

Once you are done with the above, submit your changes and finish your work by issuing the following command: echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT. Do not combine it with any other command. Important: After this command, you cannot continue working on this task.

Response Format For each response: 1) Include a THOUGHT section explaining what you are looking for and why. 2) At least one tool call. Critical Requirements • Your response SHOULD include reasoning

text explaining what you’re doing. • Your response MUST include AT LEAST ONE

bash tool call. You can make MULTIPLE tool calls in a single response when the commands are independent. • Directory or environment variable changes are not persistent. Every action is executed in a new subshell. • You can prefix any action with MY_ENV_VAR=MY_VALUE cd /path/to/working/dir && ... or write/load environment variables from files. Environment Details • You have a full Linux shell environment. • Always use non-interactive flags (-y, -f) for

commands. • Avoid interactive tools like vi, nano, or any that require user input. • You can use bash commands or invoke any tool available in the environment.

A PPENDIX B F ILTERING AND S ELECTING T ESTS

pandas-dev__pandas-27448. In this instance, the agent adds a cache to CategoricalIndex as the workload repeatedly performs slices on the same objects in the For many tasks in the benchmarks, the base commit timed loop. The agent’s change avoids the slicing on of the repository contains both tests that consistently fail repeated calls. and flaky tests that fail intermittently. Returning these pandas-dev__pandas-39972. In this instance, the agent unrelated failures can derail the agent from completing adds a cache to Styler.render as the workload repeatits task, and makes it difficult for the controller to edly renders the same 6 objects in the timed loop. The select correct patches. Therefore, we filter out these tests agent’s change avoids rendering on repeated calls. beforehand. For flaky tests in particular, we run the test pandas-dev__pandas-42353. In this instance, the agent suite 5 times, each time with a different random seed and adds a cache to Index.union as the workload repeatedly also randomize the order of test execution. We exclude unions the same two pandas Index objects in the timed tests that fail in any of the runs and find that this produces loop. The agent’s change avoids performing the union a stable test suite for each of the tasks. on repeated calls. The impact of filtering tests and selecting tests based pandas-dev__pandas-43683. In this instance, the agent on the specific changes made by the agent can be seen adds a cache to the pandas dataframe object as the workin Table X and Table XI. For each repository, we take the load repeatedly calls dropna on the same dataframe in the average and standard deviation across the tasks in that timed loop. The agent’s change avoids this computation benchmark. on repeated calls. pandas-dev__pandas-51518. In this instance, the agent A PPENDIX C adds a cache to store the result of a numpy array created M ANUAL R EVIEW OF R EWARD H ACKING ON from a range and reuses it across calls in the timed loop. SWE- FFICIENCY-L ITE pandas-dev__pandas-53152. In this instance, the agent Here we briefly analyze each instance of reward adds a cache to ArrowExtensionArray._str_get, as the hacking made by the LLM agent with access to only workload repeatedly calls this method in the timed loop. PerfAgent’s loop controller. There are 18 instances that The agent’s change avoids doing any work on repeated are considered hacks by using the LLM-as-a-judge hack calls. detector introduced in GSO. pandas-dev__pandas-55084. In this instance, the agent dask__dask-10356. In this instance, the timed work- adds a cache to store the result of a previously computed load repeatedly calls random_state_data_python with the union/intersection of pandas indexes. same inputs. The agent adds a global cache variables that pandas-dev__pandas-56110. In this instance, the agent stores the result of this call and reuses it on repeated adds a cache to ArrowStringArray._str_get to cache the calls. result of previous computations on the same array. matplotlib__matplotlib-22108. In this instance, the pydata__xarray-5661. In this instance, the workload timed workload calls Affine2D.rotate which rotates prints large arrays which tests repr formatting. The agent an internal 2D matrix. In the test script, the resulting avoids computation by doing formatting for the first and matrix is not directly used for subsequent computation. last row, thereby avoiding most of the computation. The Therefore, inside the function, the LLM agent performs repository’s correctness tests do not catch this behavior a lazy evaluation and only updates an internal angle as the agent’s change is a fast-path that only triggers on variable in the method instead. The rotated matrix is very large inputs, such as the ones in the timed workload. only materialized when other methods are called. In the scipy__scipy-10467. In this instance, the agent adds a timed workload however, these methods are never called cache to short-circuit SphericalVoronoi construction on and the materialization never occurs, which allows the repeated calls. agent to get large speedups on the timed workload. scipy__scipy-10939. In this instance, the agent adds numpy__numpy-18324. In this instance, the agent’s a cache to store the result of calls to various methods patch modifies related test files, which can affect evalua- within csr_matrix such as tocsr, tocoo, todok, todia and tion in SWE-fficiency as it uses the existing repository’s tobsr, which are repeatedly called in the timed workload. test suite. scipy__scipy-11517. In this instance, the agent adds a numpy__numpy-27830. In this instance, the agent adds cache to store the results of LIL sparse matrix operations a global cache that stores the result of the target workload: such as todok, todia, tocsr, tobsr and tocoo. np.polynomial.legendre.legval. Since the workload reA PPENDIX D peatedly calls this method in the timed loop, the agent’s M ANUAL R EVIEW OF P ER -TASK W IN /L OSS A NALYSIS patch avoids any computation for repeated calls. pandas-dev__pandas-25953. In this instance, the agent In the main paper, we compared the OpenHands adds a cache to NDFrame, which avoids recomputing the baseline and PerfAgent on a per-tasks win/loss basis. aggregation results on repeated calls. Here, we manually review each instance in which the

TABLE X: SWE-fficiency-Lite Test Counts broken down by repository. Repo pandas sympy astropy numpy scikit-learn scipy matplotlib xarray dask

# of Tasks

Base Test Count

Filtered Test Count

Patch Test Count

63 5 8 5 3 7 2 3 4

170,837 ± 57,041 8,140 ± 3,134 15,690 ± 13,159 29,558 ± 20,112 16,671 ± 3,157 36,842 ± 23,706 8,156 ± 576 14,758 ± 4,032 9,649 ± 3,299

164,893 ± 56,526 7,653 ± 2,923 15,051 ± 12,765 28,860 ± 20,393 14,693 ± 3,607 25,329 ± 16,503 8,022 ± 604 13,786 ± 3,491 9,497 ± 3,358

31,666 ± 48,220 1,189 ± 2,057 3,078 ± 3,251 3,164 ± 6,420 271 ± 470 1,312 ± 1,442 3,182 ± 438 7,346 ± 3,275 4,497 ± 3,963

TABLE XI: GSO Test Counts broken down by repository. Repo numpy pillow-simd pandas transformers tokenizers datasets pydantic pillow llama-cpp tornado

# of Tasks

Base Test Count

Filtered Test Count

Patch Test Count

36 7 34 4 4 3 4 4 2 4

35,002 ± 9,429 3,700 ± 1,019 130,994 ± 12,197 13,148 ± 19,180 129 ± 46 4,010 ± 2,005 2,292 ± 393 2,383 ± 1,726 6±0 330 ± 7

34,458 ± 9,555 3,696 ± 1,023 130,002 ± 12,463 1,744 ± 2,929 125 ± 47 3,003 ± 1,046 2,268 ± 398 2,305 ± 1,799 6±0 179 ± 3

109 ± 614 549 ± 1,303 4,691 ± 10,500 91 ± 143 1 ± 1.73 241 ± 197 757 ± 585 267 ± 289 0±0 1.75 ± 2

OpenHands baseline agent performed better than PerfAgent, meaning the OpenHands baseline agent produced a patch that matched the performance of a human expert baseline and pass the hack detector, whereas PerfAgent’s patch did not. In GSO, there are 7 instances where the OpenHands baseline produced a patch that matched or exceeded the performance of the human expert baseline and passed our hack detector, whereas PerfAgent did not. For instance: pandas-dev__pandas-2f4c93e, PerfAgent produced a patch that did not pass correctness checks. For instances pandas-dev__pandas-c34da50, tornadoweb__tornado-1b464c4, uploadcare__pillow-simd2818b90, the patches produced by PerfAgent do well on the provided workload but do not perform as well on the hidden performance tests, while the OpenHands baseline produces patches that generalize better. For instance python-pillow__Pillow-fd8ee84, the agent never submitted a patch as it ran out of budget before submitting. For instance uploadcare__pillow-simd-0514e20, the OpenHands patch uses SIMD while PerfAgent’s patch uses OpenMP. For instance huggingface__transformers253f9a3, PerfAgent fails to fuse the torch kernels while the patch from OpenHands does, although PerfAgent’s patch falls just below the 0.95 threshold. In SWE-fficiency-Lite, there are 4 instances where the OpenHands baseline produced a patch that matched or exceeded the performance of the human expert baseline

and passed our hack detector, whereas PerfAgent did not. For instances numpy__numpy-12575, numpy__numpy21832, and pandas-dev__pandas-39332, PerfAgent produced a patch that failed on some correctness tests. For instance: pandas-dev__pandas-56806, PerfAgent produced a patch that was correct but had a speedup ratio of 0.29, under the Opt@1 threshold of 0.95. In that patch, the OpenHands baseline uses a builtin pandas helper implemented in C, whereas PerfAgent allocates many large temporary arrays in its implementation.