BOXBOX: Recursive Harness Self-Improvement for Long-Running Performance Optimization

B OX B OX : R ECURSIVE H ARNESS S ELF -I MPROVEMENT FOR L ONG -RUNNING P ERFORMANCE O PTIMIZATION Yuhao Tan1,† Hanyun Jiang2,† Fangkai Yang4,* Yuan Yao1 Taolue Chen3 Bo Qiao4 Yu Kang4 Lu Wang4 Pu Zhao4 Xiaoxing Ma1,* Qingwei Lin4 1

3

Nanjing University 2 Zhejiang University Birkbeck, University of London 4 Microsoft

yhtan@smail.nju.edu.cn,

fangkaiyang@microsoft.com *

Work done at Microsoft.

Corresponding authors.

Project page: http://aka.ms/boxbox

A BSTRACT Automated harness optimization improves agents by learning from trajectories and task feedback. Existing offline methods use reference answers to optimize harnesses for reuse on held-out tasks. However, many real-world tasks are openended, while long runs expose new difficulties that offline-optimized harnesses may not address. To study harness optimization in these settings, we introduce P ERF R ACE, a benchmark for long-running performance optimization of software projects, and investigate whether agents can use task-specific experience to recursively improve their harnesses during an ongoing task. To this end, we introduce B OX B OX, which enables agent-triggered harness revision and kernel-managed hot reloading while preserving execution context. We evaluate B OX B OX on P ER F R ACE under matched test-time budgets, comparing it with Frozen Harness, offline Harness Evolution, and Harness Scaling. B OX B OX achieves higher speedup with more efficient use of task time and model cost, reaching a Speedup@4h of 1.70×, compared with 1.54× for the strongest baseline. Frozen Harness Harness Evolution Harness Scaling BOXBOX

1.6

1.702×

$16.41

1.7 1.6

1.539×

1.5

Speedup (×)

1.7

Speedup (×)

arXiv:2611.09218v1 [cs.SE] 6 Nov 2026

†

1.442×

1.4 1.3

1.281×

$12.98

1.5

$23.21

1.4 1.3

$24.56

1.2

1.2 1.1

1.1

1.0

1.0

0

1

2

3

0

4

5

10

15

20

25

30

Model cost ($)

Task time (h)

(b) Cost efficiency

(a) Task performance

Figure 1: Performance of B OX B OX and baselines on 50 P ERF R ACE test tasks, with three runs per method and task. Curves show aggregate best-so-far correct speedup against task time (a) and mean cumulative model cost per run (b), sampled at matching task times, with shaded bands indicating 95% fixed-task run-resampling intervals. B OX B OX achieves higher speedup with greater time and cost efficiency than the baselines.

1

I NTRODUCTION

Large language model (LLM) agents interact with their environment through harness that determines both the available actions and the observations they receive (Yang et al., 2024; Merrill et al., 1

2026; Lin et al., 2026). Recent studies have shown that harness design can substantially affect agent performance and behavior, even when the underlying model is fixed (Lin et al., 2026; Wang et al., 2026b). These findings have motivated automatic optimization of agent harness, in which an optimizer analyzes agent trajectories and evaluation feedback, proposes modifications to the harness, and evaluates the resulting agent on benchmark tasks. Existing offline harness optimization methods (Park et al., 2026; Zhang et al., 2026b; Lin et al., 2026; Wang et al., 2026b) often use reference answers or task-completion feedback to diagnose failures, and revise harnesses to improve performance on held-out tasks. This setup aligns with existing agent benchmarks that evaluate whether an agent completes a specified task (Jimenez et al., 2024; Merrill et al., 2026; Froger et al., 2026). However, many real-world tasks are open-ended and long-running, making end-state evaluations alone insufficient to capture how agents progress throughout these runs. Moreover, this setting poses two challenges for offline harness optimization. First, open-ended tasks make it difficult to identify useful harness revisions, as they may lack clear criteria for diagnosing what needs improvement. Prior evaluations show that harness evolution does not reliably improve performance without unit-test feedback (Wang et al., 2026b). Second, continued execution exposes new task-specific difficulties that may limit the generalization of harness revisions learned from other tasks (Nie et al., 2026; Wang et al., 2026b). To study harness optimization in these settings, we introduce P ERF R ACE, a benchmark for optimizing performance of real-world software projects. Our experiments show that offline Harness Evolution (HE) underperforms Frozen Harness on P ERF R ACE (Figure 1), with no consistent improvement during training or validation (Appendix A.2), consistent with prior findings of unreliable gains and limited generalization (Wang et al., 2026b). Nevertheless, the current task’s trajectory can reveal difficulties encountered during execution and provide task-specific evidence for harness revision (Nie et al., 2026). We therefore investigate whether an agent can use this experience to improve its own harness during the ongoing task, making harness optimization an online, task-specific process. Existing methods allocate budget between task execution and harness optimization through an outer loop, with harness updates scheduled by predefined triggers or a separate meta-agent (Wang et al., 2026b; Liu et al., 2026; Zhang et al., 2026c). However, external control limits the task-solving agent’s ability to respond to emerging difficulties. First, despite observing task progress, the agent cannot decide whether to spend its remaining budget on task execution or harness revision. Second, updates occur between task attempts or execution segments, delaying their use in the ongoing task. To address these limitations, we introduce B OX B OX1 to support agent-triggered revision of the full executable harness through kernel-managed hot reloading. When requested, the kernel pauses task execution and launches an agent review session to diagnose difficulties from the trajectory and revise the harness. After validation, it resumes the task with preserved execution context under the revised harness, which governs both subsequent execution and future reviews, enabling recursive harness improvement. We evaluate B OX B OX on P ERF R ACE against Frozen Harness, offline Harness Evolution and externally controlled Harness Scaling under matched task budgets, comparing best-so-far correct speedup and model cost. B OX B OX achieves a Speedup@4h of 1.70×, compared with 1.44× for Frozen Harness, 1.28× for Harness Evolution, and 1.54× for Harness Scaling. Our main contributions are summarized as follows: • We introduce P ERF R ACE, a benchmark for evaluating agents on open-ended performance optimization tasks, tracking correctness-preserving improvements to real-world software throughout task execution. • We present B OX B OX, which combines agent-triggered, full-harness revision from task-specific experience with kernel-managed hot reloading of the runtime while preserving execution context. • Our experiments show that B OX B OX achieves higher best-so-far correct speedup with greater time and cost efficiency than the evaluated baselines, supporting harness optimization as part of an agent’s own capabilities.

1

Named after the F1 radio call “box, box,” requesting a pit stop for repairs and adjustments during the race.

2

2

R ELATED W ORK

Automated harness optimization. Prior work has extensively studied the automated optimization of prompts for LLM-based systems (Zhou et al., 2023; Pryzant et al., 2023; Yuksekgonul et al., 2025; Khattab et al., 2024; Agrawal et al., 2025). Subsequent work extends automated optimization to agents, using execution trajectories and task feedback to improve their memory, skills, and workflows (Zhao et al., 2024; Zhang et al., 2025; Ni et al., 2026; Hu et al., 2024; Zhang et al., 2024). Meta-Harness and AHE further extend automated optimization to executable harness code, using trajectories and evaluation feedback to guide iterative revisions (Lee et al., 2026; Lin et al., 2026). Building on this setting, offline methods revise harnesses using reference answers or task-completion feedback from training tasks, aiming to improve held-out performance (Park et al., 2026; Zhang et al., 2026b; Lin et al., 2026). However, an evaluation finds unreliable gains without unit-test feedback and limited transfer from reusable harnesses, but reports gains from Harness Scaling, which alternates task attempts and harness revisions at test time (Wang et al., 2026b). Following this task-specific direction, JIT-Agent synthesizes a tailored harness for each task before execution begins (Zhang et al., 2026a). TTHE further explores harness optimization without reference answers, using trajectories and execution-derived feedback to revise executable harnesses on unlabeled test batches (Nie et al., 2026). Continual Harness moves harness optimization into ongoing task execution through an outer loop that periodically invokes a refiner while preserving the task state (Karten et al., 2026). In B OX B OX, the task-solving agent decides when to allocate its remaining budget to full-harness revision, while a separate kernel manages runtime hot reloading and preserves execution context. Long-Horizon and Open-Ended Agent Evaluation. Existing agent benchmarks cover diverse capabilities but commonly evaluate tasks with explicit end states, reporting task success, pass rates, or final artifact quality (Mialon et al., 2023; Liu et al., 2024b; Jimenez et al., 2024; Xie et al., 2024; Merrill et al., 2026; Sun et al., 2026). Recent iterative optimization benchmarks evaluate solution quality but still use reference implementations, human performance, or reference outputs for task construction or scoring (Chan et al., 2025; Lyu et al., 2026; Imajuku et al., 2025; Xu et al., 2026; Mang et al., 2025; Zhu et al., 2026). Closer to our setting, Frontier-Eng and FrontierSWE support iterative optimization of engineering designs and software projects, with tasks curated from established problems and expert contributions (Chi et al., 2026; Chu et al., 2026). In contrast, P ERF R ACE constructs long-running performance optimization tasks from real-world software projects and workloads without requiring reference optimizations. Its construction pipeline provides a scalable way to continually source new tasks from public software projects. Benchmarks for Performance Optimization. Shypula et al. (2024) introduce PIE for evaluating LLMs’ ability to make performance-improving edits to single-file competitive programming solutions while preserving functional correctness. Subsequent benchmarks broaden the optimization domains to GPU kernels, general-purpose numerical programs, and algorithmic redesign, while retaining function- or file-level optimization tasks (Du et al., 2024; Huang et al., 2024; Liu et al., 2024a; Waghjale et al., 2024; Ouyang et al., 2025; Press et al., 2025). GSO and SWE-Perf extend evaluation to repository-level performance optimization, deriving tasks and human reference optimizations from GitHub commits and pull requests (Shetty et al., 2025; He et al., 2026). SWE-fficiency and FormulaCode further incorporate a targeted workload per task and multiple community-maintained workloads, respectively (Ma et al., 2026; Sehgal et al., 2026). These repository-level benchmarks construct tasks around a targeted performance test, whereas real performance engineering starts from an end-to-end workload and discovers bottlenecks through profiling. P ERF R ACE adopts this workload-first formulation, leaving agents to identify and implement correctness-preserving improvements without a known target patch.

3

P ERF R ACE

P ERF R ACE benchmarks real-world software performance optimization with three design principles: (1) Live: projects are continually sourced from public GitHub repositories; (2) Open-ended: tasks 3

(a) Construction Discover

Filter

Claude’s C Compiler

Pair Verification Agent

Human

Accessible repo Redistributable license

Workload design

Realism

Source review

Measurement setup

Reliability

Evaluable workload Optimization potential

Reward-hacking tests

Integrity

Repo precheck

Remove crate-wide and per-item clippy silences Co-Authored-By:

Claude Opus 4.6

PerfRace task

(b) Task Format

Docker image

Task metadata

(c) Evaluation Loop

workspace/

tests/

instruction.md

workload/

project/

heldout/

editable

workload/ read-only run.sh

hidden

workspace

Submit snapshot

Agent

correctness.sh

Judge Verdict

>_ Verdict + metric

Metric

Best-so-far speedup

performance.sh

Elapsed time

Figure 2: Overview of P ERF R ACE. (a) Construction discovers projects from GitHub coding-agent activity, applies a two-stage Filter comprising a deterministic precheck and source-based assessment, and uses Agent–Human Pair Verification to produce a P ERF R ACE task. (b) Task format separates the agent workspace from hidden evaluation files. (c) The evaluation loop repeatedly submits workspace snapshots to the Judge and tracks best-so-far correct speedup.

use project workloads without reference optimizations or known target patches; and (3) Long horizon: agents iteratively optimize code and choose when to submit for feedback within a fixed budget. 3.1

TASK F ORMULATION

Each P ERF R ACE task provides the agent with an editable project workspace, a fixed workload that the agent can run locally, and access to an external Judge. For official evaluation, the Judge uses a separate copy of this workload and additional held-out correctness cases. During the evaluation loop, the agent iteratively modifies the project and submits changes to the Judge for feedback. Let ti denote the evaluation start time of submission i and si its score, with si = 0 for incorrect submissions. Scores are attributed retrospectively to evaluation start times; verification time also counts toward the task budget. We include the unmodified project as a baseline submission with t0 = 0 and s0 = 1. We define the best-so-far correct speedup as S(t) = maxi:ti ≤t si , which the agent aims to maximize within a fixed task budget B. 3.2

C ONSTRUCTION

We construct P ERF R ACE by identifying real projects with optimization potential and building reliable tasks around realistic workloads (Figure 2(a)). Each P ERF R ACE task adopts the Harbor Task Format (Merrill et al., 2026), as illustrated in Figure 2(b). Discovery. To identify less-explored optimization opportunities beyond mature, heavily maintained repositories, we search GitHub coding-agent activity for candidate projects. Specifically, we identified 103,475 candidate projects from public GitHub pull requests and branch pushes made by known coding agents between May and August 2026 (Appendix B.3). Filter. We filter candidate projects in two stages. First, deterministic prechecks verify repository and source-revision accessibility, license redistributability, and repository size and popularity, retaining 7,257 candidates. We then use GPT-5.6-Luna to assess each candidate’s setup process, evaluable workload, runtime feasibility, and optimization potential from its source code. Of these candidates, 239 pass this assessment and proceed to Pair Verification (Appendix B.3). 4

Pair Verification. We use agent–human Pair Verification to construct runnable tasks and assess their quality against three criteria (Appendix B.4): • Realism. The workload and metric represent meaningful project use. • Reliability. Repeated equivalent runs support the same correctness and performance conclusions. • Integrity. The score reflects genuine task completion rather than an invalid or shortcut solution. Specifically, we first let the agent derive workload inputs and scale from real project use and propose correctness checks, measurement boundaries, and performance metrics for human review. Second, the agent configures warmup and runs repeated tests to detect flakiness and distinguish genuine speedups from noise; the reviewer uses these results to approve or revise the measurement protocol. Finally, the agent conducts red-team tests against reward hacking using invalid, incomplete, hardcoded, and work-skipping solutions; the reviewer requests verifier fixes as needed before approving the task for packaging and release. Eventually, we curated 100 tasks, each requiring an average of 9.1 hours of curation, 12 agent–human interactions, and $11.35 in agent cost; detailed statistics are provided in Appendix B.3. 3.3

B ENCHMARK C OMPOSITION

The resulting benchmark contains 100 tasks across seven domains (Figure 3): developer tools (30 tasks) compiles, builds and validates software; data systems (25 tasks) ingests, indexes and queries data; scientific computing (15 tasks) runs simulations and numerical solvers; media processing (9 tasks) renders graphics and audio; geometry (8 tasks) processes maps and 3D models; machine learning (7 tasks) trains models and runs inference; and security (6 tasks) encrypts data and analyzes attack logs. Each task pairs a real project with a fixed workload; for example, one task optimizes CCC while it compiles SQLite. Appendix B.2 provides a detailed characterization of the tasks.

PERF RACE Data Systems DuckDB / import taxi data

WiredTiger / load pageviews

Developer Tools ML 7 GEO 8 MEDIA 9

SEC 6

Claude C Compiler / compile SQLite

DATA 25

100

rsbuild / build a web app

Scientific Computing Chaste / heart simulation

SUNDIALS / ignition simulation

Media Processing Embree / render a chess scene

TASKS

FluidSynth / render MIDI audio

Geometry SCI 15

DEV 30

SolveSpace / export a 3D model

TileServer GL / render a map

Machine Learning Eole / generate GPT-2 text

Reactant / train an image model

Security Cosmian KMS / bulk AES-GCM encryption

Hayabusa / analyze attack logs

Figure 3: Composition of P ERF R ACE: seven workload domains (inner ring), 100 tasks (outer ring), and representative project–workload pairs (right).

3.4

E VALUATION

In the evaluation loop, the agent chooses when to submit snapshots for correctness and performance feedback from the Judge (Figure 2(c)). The score si is the speedup over the unmodified project for correct submissions and zero otherwise. Runs without a correctness-preserving improvement receive a score of 1×. During the final five minutes, the agent is prompted to end optimization and leave the project in a coherent state for the Judge’s final evaluation. Prior studies show that agents can inflate scores by tampering with evaluation code or accessing reference outputs (Wang et al., 2026a; Von Arx et al., 2025). We therefore use shadow verification, evaluating project snapshots in separate containers while keeping verifier code and held-out test data 5

out of the agent’s workspace. Time spent in shadow verification counts toward B. Appendix B.5 provides further details on shadow verification.

4

B OX B OX

We now present B OX B OX for agent-triggered harness revision during long-running tasks. The following subsections present the problem formulation, explain the kernel-managed harness update mechanism, and describe a simple self-review strategy for diagnosis and revision. 4.1

P ROBLEM F ORMULATION

Consider a fixed language model M and an executable harness that may change during a task. Let Hk , ek , and τk denote the active harness, environment state, and accumulated trajectory at interaction step k, respectively (Yao et al., 2023). Task execution follows ak ∼ pM,Hk (· | τk ), ak ∈ A(Hk ), (ek+1 , ok+1 ) ← ExecuteHk (ek , ak ), τk+1 = τk ⊕ (ak , ok+1 ),  ′ Hk , if a revision is activated, Hk+1 = Hk , otherwise.

(1)

Here, ak is a tool call from the action space A(Hk ) exposed by the active harness, and pM,Hk is its distribution induced by Hk constructing model inputs from τk and invoking M. ExecuteHk resolves and executes ak through the harness’s tool implementation, updates the environment state, and processes the resulting feedback into ok+1 ; ⊕ appends the action–observation pair to the history. When a revision is attempted, the agent revises Hk based on the interaction history τk+1 , producing a candidate Hk′ . If activated, it becomes the active harness for the next interaction; otherwise, execution continues under Hk . Subsequent action selection and execution therefore use Hk+1 . 4.2

K ERNEL -M ANAGED H ARNESS U PDATES

B OX B OX treats the harness as a task-specific implementation, allowing a revision to be tailored to the requirements of the ongoing task. To support such adaptations, B OX B OX exposes the full executable harness, allowing the agent to translate task-specific needs into direct changes to tools, control flow, and state management. Let H denote the set of harness implementations compatible with the execution interface. Methods with predefined editable components restrict candidate revisions to a subset Hbounded ⊆ H (Zhang et al., 2026b; Lin et al., 2026; Park et al., 2026), whereas B OX B OX allows revisions Hk′ ∈ H. In B OX B OX, the harness is the editable agent implementation, while the runtime is its running version that controls the agent’s execution and maintains its state. These revisions change the agent’s execution code, so a faulty update can interrupt the task and make it difficult for the agent to recover on its own. B OX B OX therefore introduces a separate kernel that manages runtime hot reloading and can safely retain or restore the previous runtime if an update fails validation or activation. Specifically, we register boxbox() as a tool that yields control to the kernel, with boxbox() ∈ A(Hk ). When the agent invokes boxbox(), the kernel suspends task execution and preserves the agent’s execution context, including the accumulated trajectory, terminal session, and workspace state. It then launches a review session for the agent to examine the trajectory, revise the harness as needed, and commit any changes. After the review session concludes, the kernel validates any new commits, loads the revised runtime on success, and otherwise retains the current runtime. During activation, the kernel transfers the preserved execution context to the new runtime and reattaches the existing terminal session. On successful activation, Hk+1 = Hk′ , and subsequent interactions follow Eqn. 1. Finally, the kernel returns control to the agent, allowing it to resume the ongoing task. Beyond agent-initiated updates, runtime faults also trigger kernel-managed recovery and review based on the recorded fault evidence. Appendix C provides further implementation details. 6

Current harness

Pit Stop

Review session Candidate commit

boxbox()

Read trajectory

Revise harness

Pit stop

Kernel Suspend task

Launch review

Validate & activate

Resume task

Updated harness

Figure 4: B OX B OX mechanism. The agent requests a review on demand, triggering a trap that transfers control to the kernel. The kernel then launches the review, validates revisions, and hotreloads the runtime with preserved execution context and rollback if activation fails.

4.3

AGENT-C ONTROLLED H ARNESS O PTIMIZATION

Based on this mechanism, B OX B OX lets the task-solving agent decide when to request a harness review, so it can adapt its harness in response to difficulties encountered during task execution. Unlike prior methods that use an outer loop (Lee et al., 2026; Karten et al., 2026; Liu et al., 2026), the agent itself allocates the shared budget B between project-code optimization and harness revision to maximize S(B). Within the review session, we provide a default skill (Appendix D.3) that guides the agent through a simple self-review strategy based on diagnosis and harness revision: • Diagnosis. The agent examines the trajectory for execution failures, repeated work, and successful patterns that could be automated or reused. It uses these observations to identify limitations of the current harness, such as missing capabilities or costly tool interactions. • Revision. The agent selects harness revisions based on their expected benefit within the remaining budget, such as adding guidance, improving tools, or changing execution logic. For example, repeated tool calls can be combined into a reusable tool. We track harness revisions as Git commits so the agent can inspect prior changes alongside execution records in subsequent reviews, assess their effects, and decide whether further revisions are needed.

5

E XPERIMENTS

5.1

E XPERIMENTAL S ETUP

Baselines. We compare B OX B OX with three baselines: Frozen Harness (F ROZEN) retains the initial harness throughout task execution. Harness Evolution (HE) optimizes the harness offline on training tasks and selects it by validation performance, then holds it fixed during test-task execution. Harness Scaling (HS) uses an external controller to revise the harness between attempts on the same task, using feedback from each completed attempt. Both HE and HS are adapted from Wang et al. (2026b). Following Meta-Harness (Lee et al., 2026), all methods start from the same initial harness H0 , Terminus-KIRA (KRAFTON AI & Ludo Robotics, 2026), a simple terminal-based harness that executes shell commands and returns their outputs to the model. Evaluation protocol. We split the 100 P ERF R ACE tasks into 40 training and 10 validation tasks for HE, and 50 held-out test tasks for evaluating all methods. Both task execution and harness optimization are performed using GPT-5.6 Luna with high reasoning effort. Each method is evaluated with three independent runs per test task, each sharing a four-hour budget (B = 4h) across task execution, harness optimization, and evaluation. Detailed configurations are provided in Appendix A. 7

Table 1: Results on 50 test tasks with three four-hour runs per task, reporting Speedup@4h, mean model cost per run, and mean task-level speedup per dollar. Method

Speedup@4h

Mean model cost

Speedup/$ ↑

F ROZEN HE HS B OX B OX

1.442 1.281 1.539 1.702

$23.22 $24.56 $12.98 $16.43

0.2142 0.1243 0.2872 0.4886

Metrics. To account for randomness, we summarize task i’s best-so-far correct speedup across Q 1/R (r) R R = 3 runs as S̄i = . Let Costi be its mean model cost in US dollars, r=1 Si (B) including harness optimization. Across N = 50 tasks, we use harmonic mean speedup to limit the influence of extreme values (Shetty et al., 2025) and arithmetic mean task-level speedup per dollar: Speedup@4h = PN

N

N

, −1

Speedup/$ =

i=1 S̄i

1 X S̄i . N i=1 Costi

(2)

Reported mean model cost averages Costi across tasks. Cost accounting and statistical details appear in Appendix A. 5.2

M AIN R ESULTS

We report performance and model cost in Table 1 and plot speedup against task time and cumulative model cost in Figure 1. Appendix A.5 provides optimization trajectories for all test tasks. Offline harness optimization does not reliably improve performance. HE reaches 1.281× at four hours, below F ROZEN (Table 1). HE struggles to identify consistently effective harness revisions during training, while the revisions selected through validation show limited generalization to longrunning test tasks. Appendix A.2 provides detailed results and failure analysis. HE costs an average of $24.56 per test run. Including offline training and validation costs, shared across the 150 test runs, brings this average to at least $33.73. This additional offline expenditure does not translate into better test performance or greater cost efficiency than F ROZEN. Task-specific harness optimization supports continued improvement. Both HS and B OX B OX improve over F ROZEN, reaching 1.539× and 1.702× at four hours, respectively, compared with 1.442× for F ROZEN (Table 1). Both also reduce mean model cost, to $12.98 and $16.43 per run, respectively, compared with $23.22 for F ROZEN. Figure 1 shows the corresponding speedup trajectories against elapsed task time and mean cumulative model cost. Over the second half of the four-hour budget, the F ROZEN curve largely levels off, increasing from 1.410× to 1.442×, while HS improves from 1.431× to 1.539× and B OX B OX from 1.530× to 1.702×. These trends show that task-specific harness optimization supports further progress as execution continues. Agent-controlled harness optimization improves performance and efficiency. Figure 1(a) shows that B OX B OX achieves higher speedups than HS at matched elapsed task time. This result compares HS’s attempt-level external control with B OX B OX’s finer-grained, agent-triggered revision during task execution. Whereas HS relies on an external controller to initiate revisions, B OX B OX lets the agent decide when to allocate budget to adapting its harness to the needs of the ongoing task. This allows the agent to address harness limitations as they arise and apply useful revisions directly to the ongoing task. Figure 1(b) further shows higher speedups for B OX B OX at matched cumulative model costs. Although HS has a lower total model cost, B OX B OX achieves greater speedup per dollar under our task-level metric. Appendix A.3 provides further analysis. 5.3

A BLATION S TUDIES AND A NALYSIS

We further conduct ablation studies on 14 randomly sampled test tasks to isolate the contributions of key design choices to the effectiveness and efficiency of harness optimization, then analyze how agents perform harness optimization during task execution. Ablation Studies. 8

Table 2: Ablation study of B OX B OX. We examine agent-controlled triggering and full-harness editing in terms of task performance (Speedup@4h), model cost, and cost efficiency (Speedup/$). Method

Speedup@4h

Mean model cost

Speedup/$

1.508 1.349 1.584

$21.58 $16.07 $15.07

0.1338 0.1132 0.3452

w/o Agent Triggering w/o Full Editing Full B OX B OX

Agent-controlled triggering. Following the scheduled refinement in Continual Harness (Karten et al., 2026), the w/o Agent Triggering variant schedules reviews every 30 minutes with full-harness editing. It achieves lower speedup at higher model cost than agent-triggered B OX B OX (Table 2). Scheduled reviews can interrupt the process of proposing, implementing, and evaluating code changes. They also consume model calls even when they conclude that no harness revision is needed. In contrast, B OX B OX lets the agent decide when to allocate budget to harness optimization based on the needs of the ongoing task. Full-harness editing. Following Self-Harness (Zhang et al., 2026b), the w/o Full Editing variant retains agent-controlled triggering but restricts each review to a predefined scope. It achieves lower speedup than B OX B OX (Table 2). Predefined editing boundaries can constrain task-specific adaptations that require changes across components or beyond the permitted scope. Full-harness editing lets the agent decide what to modify and how extensively based on the needs of the ongoing task. Harness Optimization Analysis. Review frequency. B OX B OX invokes a median of five reviews per run, with 92% of runs containing three to six (Figure 5(a)). These results show that boxbox() provides an effective entry point for harness optimization during task execution, allowing agents to review their harnesses based on accumulated trajectories. Budget allocation. Agents allocate most of their time budget to task execution. Harness reviews account for a median of 2.1% of the time budget and 13.8% of model cost per run (Figure 5(b)). Together with B OX B OX’s higher speedup and lower model cost than F ROZEN, these results suggest that agents can effectively balance task execution and harness optimization within a shared budget. Modification scope. Harness modifications span prompt rules, skills, tools, execution logic, and fault handling (Figure 5(c)). These changes show that agents use full-harness editing to adapt both their capabilities and execution process. Cases in Appendix A.4 illustrate how these modifications add capabilities, automate waiting, and revise context management. 60

Median 5

Review share (%)

Runs

40 28 19

20

0

1 0

1

5 2

3

4

5

6

3

2

1

7

8

9

30

16.6%

Skills

20

16.9%

Tool functionality

13.8% 10

0

42.9%

Prompt rules

4.7%

Execution logic

12.7%

Fault fix

2.1% Time

Cost

0

10

20

30

40

50

Harness updates (%)

Reviews per run

(a) Review frequency

40.8%

Review feedback

45 46

(b) Budget allocation

(c) Modification types

Figure 5: Harness optimization analysis. (a) Number of reviews per run across 150 test runs, including agent-triggered and fault-recovery reviews. (b) Per-run review time as a share of the fourhour task budget and review cost as a share of total model cost within that budget; dots represent runs, and boxes show the median and interquartile range. (c) Modification types across a collection of 620 applied, nonempty harness updates from B OX B OX runs. The ablations show that agent-controlled triggering and full-harness editing improve performance and cost efficiency, while the analysis shows that agents adapt their capabilities and execution process while allocating most of their time to task execution. Together, these findings support harness optimization as part of the task-solving agent’s own capabilities. 9

6

C ONCLUSION

In this work, we introduced P ERF R ACE, a benchmark for long-running performance optimization of real-world software, and B OX B OX, which combines agent-triggered, full-harness revision from task-specific experience with kernel-managed hot reloading that preserves execution context. On P ERF R ACE, offline Harness Evolution yields inconsistent gains, whereas B OX B OX achieves higher best-so-far correct speedup and greater time and cost efficiency than the evaluated baselines. It achieves a Speedup@4h of 1.70×, compared with 1.54× for the strongest baseline, and reduces mean model cost by 29.3% relative to Frozen Harness. These results support harness optimization as an agent capability for improving how it solves a task while continuing its execution.

S ECURITY AND R ESPONSIBLE AI L IMITATIONS B OX B OX allows an agent to modify executable harness code during an ongoing task, which introduces security and Responsible AI risks beyond those of a fixed harness. In particular, the agent executes and modifies code that may originate from untrusted repositories, and harness revisions may introduce security regressions, weaken existing safeguards, or create unintended execution behavior. More generally, self-modifying agent mechanisms could be misused to alter controls or expand capabilities in ways not intended by system operators. Our evaluation is conducted in isolated research environments with kernel-managed validation and rollback for harness updates and separate shadow verification for task evaluation. These mechanisms reduce experimental risk but do not constitute a complete security boundary or guarantee safe behavior under adversarial conditions. B OX B OX should therefore be viewed as a research prototype rather than a production-ready self-modifying agent framework. Any production deployment would require additional sandboxing, access control, security review, monitoring, and human oversight of agent-generated harness changes before they are activated.

10

R EFERENCES Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista OpsahlOng, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457, 2025. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. MLEbench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations, 2025. Yizhe Chi, Deyao Hong, Dapeng Jiang, Tianwei Luo, Kaisen Yang, Boshi Zhang, Zhe Cao, Xiaoyan Fan, Bingxiang He, Han Hao, et al. Frontier-Eng: Benchmarking self-evolving agents on realworld engineering tasks with generative optimization. arXiv preprint arXiv:2604.12290, 2026. Evan Chu, Rajan Agarwal, Abishek Thangamuthu, Brendan Graham, and Justus Mattern. FrontierSWE: Benchmarking coding agents at the limits of human abilities. https://www.fronti erswe.com/blog/v1, 2026. Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. Mercury: A code efficiency benchmark for code large language models. In Advances in Neural Information Processing Systems, volume 37, pp. 16601–16622, 2024. doi: 10.52202/079017-0529. Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Gerard Moreno-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, Mengjue Wang, Ian Yu, Amine Benhalloum, Grégoire Mialon, and Thomas Scialom. GAIA2: Benchmarking LLM agents on dynamic and asynchronous environments. arXiv preprint arXiv:2602.11964, 2026. Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Yin Zheng, Zejian Yuan, and Zejun Ma. SWE-Perf: Can language models optimize code performance on real-world repositories? In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Seoul, South Korea, 2026. Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408.08435, 2024. Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M. Zhang. EffiBench: Benchmarking the efficiency of automatically generated code. In Advances in Neural Information Processing Systems, volume 37, pp. 11506–11544, 2024. doi: 10.52202/079017-0367. Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. ALE-Bench: A benchmark for long-horizon objective-driven algorithm engineering. In Advances in Neural Information Processing Systems, volume 38, 2025. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. Seth Karten, Joel Zhang, Tersoo Upaa Jr., Ruirong Feng, Wenzhe Li, Chengshuai Shi, Chi Jin, and Kiran Vodrahalli. Continual harness: Online adaptation for self-improving foundation agents. arXiv preprint arXiv:2605.09998, 2026. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, et al. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. In International Conference on Learning Representations, volume 2024, pp. 54928–54958, 2024. KRAFTON AI and Ludo Robotics. Terminus-KIRA: Boosting frontier model performance on Terminal-Bench with minimal harness, 2026. URL https://github.com/krafton -ai/kira. 11

Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu, Yixing Jiang, Chelsea Finn, Marco Pavone, Ion Stoica, and Azalia Mirhoseini. LLM-as-a-Verifier: A general-purpose verification framework. arXiv preprint arXiv:2607.05391, 2026. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. MetaHarness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026. Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850, 2026. Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluating language models for efficient code generation. In First Conference on Language Modeling, 2024a. Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, Mert Cemri, Zhifei Li, Qiuyang Mang, Ashwin Naren, Ethan Boneh, Audrey Cheng, Melissa Z. Pan, Alexander Du, Kurt Keutzer, Alvin Cheung, Alexandros G. Dimakis, Koushik Sen, Matei Zaharia, and Ion Stoica. EvoX: Metaevolution for automated discovery. arXiv preprint arXiv:2602.23413, 2026. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, 2024b. Bohan Lyu, Yucheng Yang, Siqiao Huang, Jiaru Zhang, Qixin Xu, Xinghan Li, Xinyang Han, Yicheng Zhang, Huaqing Zhang, Runhan Huang, et al. MLS-Bench: A holistic and rigorous assessment of AI systems on building better AI. arXiv preprint arXiv:2605.08678, 2026. Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, and Parthasarathy Ranganathan. SWE-fficiency: Can language models optimize real-world repositories on real workloads? In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Seoul, South Korea, 2026. Qiuyang Mang, Wenhao Chai, Zhifei Li, Huanzhi Mao, Shang Zhou, Alexander Du, Hanchen Li, Shu Liu, Edwin Chen, Yichuan Wang, et al. FrontierCS: Evolving challenges for evolving intelligence. arXiv preprint arXiv:2512.15699, 2025. Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, et al. TerminalBench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general AI assistants. arXiv preprint arXiv:2311.12983, 2023. Microsoft. Dadsv5 size series. Microsoft Learn, 2026. URL https://learn.microsoft. com/en-us/azure/virtual-machines/sizes/general-purpose/dadsv5-s eries. Accessed October 24, 2026. Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2Skill: Distill trajectory-local lessons into transferable agent skills. arXiv preprint arXiv:2603.25158, 2026. Jun Nie, Yonggang Zhang, Jun Song, Qianshu Cai, Dahai Yu, Yike Guo, Xinmei Tian, and Bo Han. TTHE: Test-time harness evolution. arXiv preprint arXiv:2607.08124, 2026. Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. KernelBench: Can LLMs write efficient GPU kernels? arXiv preprint arXiv:2502.10517, 2025. 12

Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. AutoSaddler: Automatic harness optimization with durable updates from agent execution traces. arXiv preprint arXiv:2608.23041, 2026. Ori Press, Brandon Amos, Haoyu Zhao, Yikai Wu, Samuel K. Ainsworth, Dominik Krupke, Patrick Kidger, Touqir Sajed, Bartolomeo Stellato, Jisun Park, Nathanael Bosch, Eli Meril, Albert Steppi, Arman Zharmagambetov, Fangzhao Zhang, David Pérez-Piñeiro, Alberto Mercurio, Ni Zhan, Talor Abramovich, Kilian Lieret, Hanlin Zhang, Shirley Huang, Matthias Bethge, and Ofir Press. AlgoTune: Can language models speed up general-purpose numerical programs? In Advances in Neural Information Processing Systems, volume 38, pp. 118982–119027, 2025. doi: 10.52202/0 85713-3585. Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7957–7968, 2023. Atharva Sehgal, James Hou, Akanksha Sarkar, Ishaan Mantripragada, Swarat Chaudhuri, Jennifer J. Sun, and Yisong Yue. FormulaCode: Evaluating agentic optimization on large codebases. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Seoul, South Korea, 2026. Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, and Ion Stoica. GSO: Challenging software optimization tasks for evaluating SWE-agents. In Advances in Neural Information Processing Systems, volume 38, 2025. Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R. Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improving code edits. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=ix7rLVHXyY. Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, Chris Duroiu, Haoyun Zhang, Jeffrey Lin, et al. Agents’ last exam. arXiv preprint arXiv:2606.05405, 2026. Sydney Von Arx, Lawrence Chan, and Beth Barnes. Recent frontier models are reward hacking. METR Blog, June 2025. URL https://metr.org/blog/2025-06-05-recent-rew ard-hacking/. Siddhant Waghjale, Vishruth Veerendranath, Zhiruo Wang, and Daniel Fried. ECCO: Can we improve model-generated code efficiency without sacrificing functional correctness? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15362– 15376. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.emnlp-main.85 9. Hao Wang, Hanchen Li, Qiuyang Mang, Alvin Cheung, Koushik Sen, and Dawn Song. Do androids dream of breaking the game? systematically auditing AI agent benchmarks with BenchJack. arXiv preprint arXiv:2605.12673, 2026a. Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents. arXiv preprint arXiv:2607.12227, 2026b. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024. Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, Shizhe Diao, Jiaxin Pei, Jinsung Yoon, Hao Zhang, Mengdi 13

Wang, Radha Poovendran, Misha Sra, Alex Pentland, and Zichen Chen. frontier models solve long-horizon auto research and engineering tasks? arXiv:2606.05080, 2026.

AutoLab: Can arXiv preprint

John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, 2024. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 639(8055):609–616, 2025. Guibin Zhang, Leo Lu, Fangzhou Xie, Kang Zhu, Junhao Wang, Zhifei Xie, Zhaochen Yu, Zihang Liu, Zhongxiang Sun, Qiankun Li, Yue Liao, Heng Chang, Xiaobin Hu, Qibing Ren, Wangchunshu Zhou, Chuanrui Hu, Yafeng Deng, and Shuicheng Yan. JIT-Agent: Scaling harness intelligence via just-in-time harness evolution. arXiv preprint arXiv:2608.25593, 2026a. Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-Harness: Harnesses that improve themselves. arXiv preprint arXiv:2606.09498, 2026b. Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 2024. Jiayi Zhang, Yongfeng Gu, Jianhao Ruan, Maojia Song, Yiran Peng, Zhiguang Han, Jinyu Xiang, Zhitao Wang, Caiyin Yang, Yixi Ouyang, Bang Liu, Chenglin Wu, and Yuyu Luo. Harnessing agentic evolution. arXiv preprint arXiv:2605.13821, 2026c. Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models. arXiv preprint arXiv:2510.04618, 2025. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19632–19642, 2024. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In International Conference on Learning Representations, 2023. Deyao Zhu, Xin Zhou, Shengling Qin, Xuekai Zhu, et al. EdgeBench: Unveiling scaling laws of learning from real-world environments. arXiv preprint arXiv:2607.05155, 2026.

14

A

E XPERIMENTAL D ETAILS

We describe the experimental environment and evaluation details in Appendix A.1, followed by the experiments for Harness Evolution and Harness Scaling in Appendices A.2 and A.3. Appendix A.4 examines how B OX B OX harness revisions support task execution, and Appendix A.5 provides additional per-task optimization trajectories. A.1

E XPERIMENTAL S ETUP

Experimental environment. To facilitate replication of our experimental environment, we conduct experiments on an Azure Standard_D96ads_v5 virtual machine (Microsoft, 2026) with 96 vCPUs backed by an AMD EPYC 7763 processor and 384 GiB of system memory, running Ubuntu 24.04 with Docker. Initial harness. All methods start from Terminus-KIRA (KRAFTON AI & Ludo Robotics, 2026), a terminal-based harness that executes shell commands and returns their outputs to the model. F ROZEN retains this harness throughout task execution while the agent iteratively optimizes the project code. Time and cost accounting. Figure 1 and Table 1 use the same 600 test runs: 50 tasks, four methods, and three runs per method and task. For all methods, elapsed task time sums recorded agentexecution and verification phases. Agent execution includes harness review; environment setup is excluded. Only evaluations started within the four-hour budget contribute to Speedup@4h. We calculate model cost from recorded token usage using the rates in Table 3. Reported mean model costs per test run include all recorded model calls; cost and token curves report cumulative usage at each task-budget time point. Table 3: Model token rates used for cost calculation (USD per million tokens). Model GPT-5.6-Luna GPT-5.6-Terra

Uncached input

Cached input

Output

$1.00 $2.50

$0.10 $0.25

$6.00 $15.00

Statistical analysis. The bands in Figure 1 show the 2.5th and 97.5th percentiles of 5,000 resampled benchmark runs over the fixed 50-task set. Each sample independently selects one of the three observed runs per task and aggregates the selected scores using the harmonic mean. The central curves instead use all three runs per task, with the geometric-then-harmonic aggregation defined in Equation 2. In the cost panel, speedup intervals are plotted at the mean cumulative cost for each task-time point; they do not represent uncertainty in cost. A.2

H ARNESS E VOLUTION

Our HE baseline adapts the implementation of Wang et al. (2026b) to P ERF R ACE, retaining its core diagnosis-and-revision workflow. We detail the offline optimization procedure and examine training progress and the generalization of the selected harness to held-out tasks. Setup. We adapt task execution, evaluation, and experiment orchestration to P ERF R ACE, replacing the task-success objective with best-so-far correct speedup under a fixed time budget and tailoring the diagnostic and revision prompts to guide harness improvements that increase speedup and reduce costs. The reference HE experiments use full-batch training on Terminal-Bench 2.1 (Wang et al., 2026b). Publicly released mini-swe-agent trajectories generated with DeepSeek-V4-Flash on this benchmark (Kwok et al., 2026) average 50 agent steps and 12.2 minutes per run, compared with 965 steps for our F ROZEN runs under a four-hour budget on P ERF R ACE. We therefore limit each training run to 45 minutes and partition the 40 training tasks into mini-batches of five to keep the volume of trajectory evidence manageable for diagnosis. Across three reshuffled epochs (24 iterations), we evaluate hi−1 on one mini-batch at iteration i and use the resulting trajectories and evaluation 15

outcomes to produce hi . For validation, we compare h8 , h16 , and h24 using fresh baseline measurements. Table 4 summarizes the HE pipeline and its recorded model costs. Table 4: HE configuration and recorded model costs. Training and validation costs include retries and reruns and are lower bounds where usage records are missing. Stage

Tasks

Training Validation Test Epoch 1 · 1.086×

1.25

Budget per run

Cost ($)

45 min 2h 4h

200.34 1,176.12 3,683.34

24 iterations × 5 tasks 3 harnesses × 3 runs/task h24, 3 runs/task

Epoch 2 · 1.062×

1.32

Epoch 3 · 1.069×

1.289

1.20

Speedup

Speedup

40 10 50

Evaluated runs

1.15 1.10

1.28

1.273 1.244

1.24

1.05 1.00

1.20

1

4

8

12

16

20

Training iteration

24

h8

h16

h24

Harness checkpoint

(b) Validation performance

(a) Training progress

Figure 6: HE training and validation performance. (a) Mini-batch training speedup, with aggregate scores for each epoch. (b) Validation speedup of harness checkpoints h8 , h16 , and h24 . Speedups are aggregated as in Equation 2. Training, validation, and test results. Training performance does not improve consistently across epochs (Figure 6(a)). Aggregate speedup decreases from 1.086× in epoch 1 to 1.062× in epoch 2 and recovers to 1.069× in epoch 3, remaining below its initial level. Validation performance does not improve consistently across iterations (Figure 6(b)), with h24 achieving the highest score among the evaluated checkpoints. We therefore use h24 for testing, where it achieves 1.281×, below Frozen Harness at 1.442× (Table 1). These results show that additional offline revisions do not reliably improve training performance or yield a test-time advantage, consistent with prior findings of limited transfer (Wang et al., 2026b). Including the recorded offline cost raises mean cost to at least $33.73 per test run. Failure analysis. To understand HE’s failure modes, we examine how a revision introduced during training affected execution on a test task. At iteration 5, diagnostic reports identified two patterns: (1) measurements showed faster execution of the baseline code, which had either remained unchanged or been restored after edits were discarded; and (2) records of faster execution did not specify which code version had been measured. These patterns made it difficult to determine whether measured gains reflected actual code improvements or measurement noise. HE therefore added a check to the harness to prevent agents from submitting code without evidence of an actual performance improvement. Specifically, the agent had to record the current and baseline commits, at least three timing measurements for each version, and correctness evidence in .autoresearch/promotion.json. The harness checked that the record matched the current commit, the worktree was clean, the code differed from the baseline, and the measured improvement exceeded a noise threshold. If these requirements were not met, it blocked submission and returned feedback directing the agent to repair. The check read this record and inspected Git state in the working directory selected by the harness (Figure 7). The check enforced fixed requirements based on failures observed during training. However, its assumptions about repository location did not hold for some test tasks, causing it to block submissions before assessing the performance evidence. To examine this failure pattern, we selected mirage-kubernetes-triage for detailed trajectory analysis.

16

Check the repository and read local evidence. + workdir = environment.task_env_config.workdir ... + head = await environment.exec("git rev-parse HEAD", cwd=workdir, timeout_sec=30) + if head.return_code != 0: + errors.append(f"cannot resolve candidate HEAD: {head.stderr.strip()}") + return errors ... + record = await environment.exec( + "cat -- .autoresearch/promotion.json", + cwd=workdir, + timeout_sec=30, + )

Check the measured improvement. + threshold = max(0.05 * baseline_median, 2 * baseline_noise) + improvement = baseline_median - candidate_median ... + if improvement < threshold: + errors.append(f"{PROMOTION_MARGIN_ERROR_PREFIX}: need at least {threshold} seconds")

Block submission when checks fail. + if phase_outcome.action is not PhaseAction.CONTINUE: + gate_errors = await self._promotion_gate_errors(self._session) + if gate_errors: + phase_outcome = PhaseOutcome() + promotion_gate_feedback = ( + "WARNINGS: PROMOTION GATE BLOCKED; no verification was requested.\n" + + "\n".join(f"- {error}" for error in gate_errors) + + "\nCreate or repair .autoresearch/promotion.json, then retry completion." + ) + feedback = "\n".join(filter(None, (feedback, promotion_gate_feedback)))

Figure 7: Selected additions to HE’s harness at iteration 5. The harness checks the repository and performance evidence before submission, blocking submission and returning repair instructions when a check fails.

In the selected run, the agent prepared Git commits and performance evidence inside the editable repository, while the harness ran its submission checks in the parent workspace. The check therefore failed to resolve the candidate commit and returned cannot resolve candidate HEAD, blocking submission before assessing the performance evidence. Repeated attempts to repair the evidence in the repository did not change the directory used by the harness. The trajectory records 473 blocked submission requests spanning approximately 3 hours and 46 minutes. This case illustrates both challenges in Section 1. Difficulty in attributing measured gains motivated stricter submission requirements, but did not establish that these requirements would help the agent find better optimizations. On the selected test task, the check’s repository-location assumptions instead led to repeated repair attempts and blocked submissions, illustrating how a revision intended to address training failures can hinder execution under new task-specific conditions. A.3

H ARNESS S CALING

Our HS baseline adapts the implementation of Wang et al. (2026b) to P ERF R ACE. We describe the execution procedure, analyze model costs, and examine a failure case. 17

Setup. We adapt task execution and evaluation to P ERF R ACE, using best-so-far correct speedup under a fixed time budget as the optimization objective. Each run shares a four-hour budget across task execution, harness optimization, and evaluation. An external controller revises the harness using feedback from each completed attempt and starts a new agent session, preserving project-code progress and resetting the execution context. We record 2,132 HS attempts, with a median of 12 per run. Across 1,983 boundaries, the median interval between one attempt ending and the next beginning is 3.37 minutes, including harness optimization and runtime transition. Model costs. HS’s session resets can lower input-token costs by reducing the accumulated conversation history. Although project-code progress is retained, subsequent attempts may need to reconstruct earlier decisions or repeat experiments whose outcomes are no longer available in the execution context. B OX B OX preserves this context across harness updates, allowing subsequent optimization to build directly on earlier findings. This trade-off helps explain why HS’s lower total model cost does not necessarily translate into greater optimization progress per dollar. Failure analysis. To examine HS’s optimization process, we trace one kitops-minilm-pack run (Figure 8). During one harness optimization step, HS’s controller reported that the agent repeatedly rebuilt the project, ran tests, and reran the workload on the same code version. It therefore added a repetition check to warn after three consecutive matching command batches. Checkpoint 5

Inspect packaging bottlenecks

1.649×

Speedup (×)

1.6 Agent session 113 repeated checks

1.4

Add repetition check

1.2

Warn after 3 consecutive matching command batches

1.0 0

1

2

3

4

Task time (h)

Figure 8: HS optimization trajectory on kitops-minilm-pack. Annotations show selected harness revisions and an agent session containing 113 consecutive checks that produced no new performance measurements. However, the new repetition check did not stop the agent from performing the same checks using different command forms. The agent ran 113 consecutive commands repeating the same checks, without modifying code or collecting new performance measurements. The check only looked for consecutive repetitions of the same commands, so it could miss repeated checks written in different ways. The run reached its best correct speedup of 1.649× at checkpoint 5, with no further improvement in subsequent recorded checkpoints. The repeated checks revealed a need for further harness revision during execution, yet HS performs harness optimization only between agent sessions, leaving the current session to continue under the existing harness. This case illustrates how session-level external control can delay harness revisions needed during ongoing task execution. The revised check missed repeated validation expressed through different commands, and the agent continued spending task time without making code changes or collecting new performance measurements. Allowing the agent to initiate further harness revision when these limitations become apparent would provide an earlier opportunity to address them. A.4

C ASE S TUDY

To analyze how B OX B OX supports effective task execution, we examine two complementary aspects of harness optimization. Appendix A.4.1 traces how agent-triggered reviews address difficulties encountered during ongoing tasks. Appendix A.4.2 examines how full-harness editing can support 18

task performance and reduce model costs through changes to agent guidance, tools, and execution behavior. A.4.1

H ARNESS O PTIMIZATION T RAJECTORIES

We examine two four-hour runs, tracing the difficulties encountered during task execution, the resulting harness revisions, and their use in subsequent optimization. CCC. To reduce CCC’s compilation time on SQLite, the agent profiles compiler stages and tests code changes through repeated compilation runs (Figure 9). During the first review, the agent identified repeated manual processing of timing output and added a helper for repeated runs, result recording, and candidate comparisons. Using this helper, the agent established a baseline and evaluated compiler changes, but still had to extract and compare individual stage timings manually. To automate this step, the agent added stage-timing summaries to the helper in the second review, then used them to investigate optimization passes and code generation. Later, a change to CCC’s source code appeared to reduce compilation time by 9.4% in separately measured batches. Yet when the agent alternated runs of the modified and unchanged compiler versions, their median compilation times were nearly identical, suggesting that the apparent gain came from measurement variation between batches rather than the code change. The agent therefore incorporated this comparison procedure into the helper during the third review to improve measurement reliability, then used it to reject several compiler changes that showed no consistent speedup. Continuing with this helper, the agent developed mnemonic-parsing and operand-splitting changes that reduced official running time from 6.321 to 6.095 and then 5.792 seconds. ccc--compile-speed 2.0 1.8

+0.082× (5.4%)

+0.087× (5.2%)

Reduce parser allocations

Avoid operand allocations

+0.517× (51.7%)

1.748×

Speedup (×)

Batch jump relaxation 1.6 1.4 Retain

1.2 1.0 0.8 0

Repeated runs

Stage timings

Record measurements

Parse timings

1

Paired comparisons Alternate run order 2

3

4

Elapsed task time (h)

Figure 9: B OX B OX optimization trajectory on ccc-compile-speed. Points show correct evaluation results and the line shows best-so-far speedup. Annotations mark harness review outcomes and task-code changes; numerical gains are absolute and relative increases over the preceding bestso-far speedup. FluidSynth. To optimize FluidSynth’s MIDI-to-WAV rendering performance, the agent tests code changes through repeated workload execution and output checks. Figure 10 traces harness revisions during this process. The first review noted that long-running commands required extra model calls to continue waiting, but retained the harness because a proposed waiting tool would duplicate existing functionality. As this pattern recurred, the second review extended waiting inside the existing command runner. During subsequent audio-write batching experiments, a single tool call completed a build, five renders, and output checks, allowing the agent to assess the candidate without an additional waiting call. Comparing batch sizes still required repeated setup of builds, timing loops, and output checks. The third review therefore added benchmark_matrix, reusing the updated command runner to execute labeled experiments and report timings and exit status. The agent then used it for six repeated renders with output-hash checks and subsequent configuration comparisons. Rendering performance improved further after the agent disabled OpenMP in the build configuration. The fourth review re19

tained the harness, judging its existing capabilities sufficient after the agent resolved a build-state issue through a clean rebuild. The run reached a best-so-far speedup of 1.183×. fluidsynth--bagatelles +0.029× (2.7%) Larger write batches

Speedup (×)

1.2

+0.077× (7.7%) Batch audio writes

+0.069× (6.2%) Disable OpenMP

1.183×

1.1 Retain

Retain

1.0 Adaptive waiting Poll to completion

Batch benchmarking Add benchmark_matrix

0.9 0

1

2

3

4

Elapsed task time (h)

Figure 10: B OX B OX optimization trajectory on fluidsynth-bagatelles, using the plotting conventions of Figure 9. Reviews extend command waiting and add batch benchmarking. Together, these cases illustrate how harness optimization can improve task performance and reduce model costs by improving the agent’s performance optimization workflow, even though the reviews do not directly modify project code. A.4.2

R EPRESENTATIVE H ARNESS M ODIFICATIONS

We examine five representative harness updates that change agent guidance, tools, or execution behavior. While Figure 5(c) reports the prevalence of modification categories, Table 5 summarizes the concrete changes analyzed below. Table 5: Representative harness updates grouped by modification type. Case numbers refer to the detailed analyses below; task IDs follow the catalog in Table 7. Case Category

Task ID

Modification

1

Review feedback

n8n-mcp--tool-batch-speed

Redirect the next experiment toward SQLJS conversion and statement overhead.

2

Prompt rules and skills

sum--text-attest

3

Tool functionality

rust-clippy--cargo-package -lint

4

Execution logic

trixi-framework-trixi-j l--self-gravitating-sedov -amr

Require alternating A/B measurements when apparent gains are close to noise. Extend internal command polling to reduce model-directed waiting turns. Add a message-count-based context-compaction path.

5

Fault fix

rust-clippy--cargo-package -lint

Replace an exception on shell exit with a session-recovery call.

The following cases proceed from changes in agent guidance to changes in executable behavior. Cases 1–2 examine review feedback and prompt-based guidance for experiment selection and evaluation. Case 3 examines internal command polling to reduce model-directed waiting. Cases 4–5 turn to context management and terminal recovery within the agent loop. Together, they illustrate how full-harness editing can affect both the direction of the search and the process used to carry it out. Case 1. Review feedback: redirecting an experiment Task: n8n-mcp--tool-batch-speed 20

Background. The task is to reduce the end-to-end running time of the n8n-mcp server on a fixed batch of workflow-building tool calls while preserving response correctness. The agent attempted to reduce logging overhead, but local measurements showed no improvement. The review therefore used profiling evidence to identify a different direction for the next experiment. + The remaining profiler clue is the SQLJS adapter's repeated `all/get` + row/object conversions and statement lifecycle overhead. Treat this as + a hypothesis, not a conclusion: inspect the exact hot path and design + one general correctness-preserving experiment ... + + ... Use a separate branch from main, build, run the exact workload with + paired repeated clean-vs-experiment medians, and compare response hashes. + Reject changes within noise or with any response mismatch.

Review feedback. The feedback directs the agent to stop revisiting logging and investigate SQLJS data-conversion and statement-management overhead. It also specifies an experimental workflow, asking the agent to preserve the current project-code version and test modifications on a separate branch through paired repeated workload measurements and response-hash comparisons. Modifications should be rejected if their measured gains fall within measurement noise or their responses differ. Discussion. This case illustrates how review feedback can support performance optimization by guiding both experiment selection and evaluation. Redirecting effort away from an unsuccessful direction can reduce repeated exploration, while paired measurements and response checks help the agent distinguish useful code changes from measurement noise or incorrect behavior. Case 2. Prompt rules and skills: comparing noisy measurements Task: sum--text-attest Background. The task is to reduce the running time of SUM’s batch text-attestation workflow while preserving its outputs. Across successive experiments, the agent encountered apparent gains that did not hold under more controlled measurements. Earlier reviews had already warned against drawing conclusions from warmup effects and uncontrolled measurement order, yet a later code change again appeared promising based on a single fast run. To address this recurring problem, the review incorporated the measurement guidance into the main prompt for use across subsequent experiments. + When a candidate's apparent gain is comparable to run-to-run noise, do not + infer from one fast first run. Keep the unchanged `main` reference and + candidate branch, warm both in the same controlled procedure, then run + alternating fresh-process A/B repetitions with identical output checks + and compare medians (and spread) before accepting or rejecting the hypothesis. + Align this local procedure with the official workload boundary before + using a tiny timing difference to justify submission.

Harness update. The prompt rule instructs the agent to keep an unchanged reference version and test modified code on a separate branch. It requires the same warmup procedure for both versions, followed by alternating fresh-process workload runs with identical output checks. The agent should compare median running times and their spread before accepting or rejecting the modification, with the measurement procedure aligned with the official workload. Discussion. This case illustrates how prompt revisions can provide consistent guidance for evaluating code changes across successive experiments. More reliable comparisons can help the agent distinguish genuine improvements from measurement noise, reducing the risk of retaining ineffective changes or discarding useful ones based on unstable timing evidence. Case 3. Tool functionality: reducing model calls for command polling Task: rust-clippy--cargo-package-lint Background. During performance optimization on Rust Clippy, the agent builds the project and runs benchmarks to evaluate code changes. The command tool could return before these operations 21

finished, requiring additional model calls simply to continue waiting. The review therefore extended the tool’s internal waiting behavior to reduce these repeated interactions. Harness update. The update extends internal waiting to 180 seconds for ordinary commands, subject to the existing block timeout. The tool continues checking for the completion marker and returns as soon as it appears, or provides a progress snapshot when the waiting window expires. + AUTO_COMMAND_WAIT_SEC = 180 ... + wait_budget = min( + BLOCK_TIMEOUT_SEC, + max(command.duration_sec, AUTO_COMMAND_WAIT_SEC), + ) ... - while time.monotonic() - start < command.duration_sec: + while time.monotonic() - start < wait_budget:

The tool description is updated alongside the implementation, instructing the agent to request an additional wait only when the command outlasts the new window. - "It is always possible to wait again if the prior output has not finished, " - "by running empty keystrokes with a duration on subsequent requests to wait longer. " + "The harness automatically continues waiting when the completion marker has " + "not appeared, up to 180 seconds per tool call, and returns a progress snapshot " + "at that bound. " + "Use an empty-keystroke wait only when a command exceeds that bound; " + "use C-c to interrupt foreground work. "

Discussion. This case illustrates how full-harness editing allows agents to adjust tool behavior adaptively to reduce model calls spent solely on waiting. When an unfinished command returns control to the agent, another model call processes the conversation context and generates a continuation request, incurring input- and output-token costs. Internal polling checks command status without invoking the model, allowing commands that finish within the extended window to return their results without these additional calls. Fewer waiting turns also add fewer messages to subsequent context. The potential cost reduction therefore comes from reducing model interaction during command execution, while commands exceeding the window may still require further calls. Case 4. Execution logic: context compaction Task: trixi-framework-trixi-jl--self-gravitating-sedov-amr Background. During performance optimization on Trixi, the agent accumulated a long interaction history through profiling and repeated experiments. The review reported that late model requests contained approximately 145,000 input tokens, making repeated processing of the accumulated history a substantial source of model cost. The existing summarization mechanism had not sufficiently limited this growth, motivating a revision to the agent loop’s context-management logic. + if ( + original_instruction + and len(chat.messages) >= 64 + and len(chat.messages) >= self._last_forced_compaction_messages + 24 + ): + self._last_forced_compaction_messages = len(chat.messages) + proactive_summary_result = await self._force_proactive_compaction( + chat, original_instruction, self._session, + ) + if proactive_summary_result: + prompt, subagent_refs = proactive_summary_result + self._pending_subagent_refs = subagent_refs + self._pending_handoff_prompt = prompt if original_instruction and self._enable_summarize: ...

22

Harness update. The update adds a message-count trigger before the existing optional summarization check. When the original task instruction is available, the loop attempts compaction if the history contains at least 64 messages and its message count is at least 24 higher than the count recorded at the last attempt. It records the current count before invoking the existing summarizer through a helper with a timeout. Successful results supply the continuation prompt and subagent references; if summarization fails, execution continues with the existing history. Discussion. This case illustrates how full-harness editing allows agents to adapt contextmanagement logic during ongoing tasks. By summarizing accumulated interaction history, the revised loop can reduce the amount of context processed in subsequent model calls and the associated input-token costs. Case 5. Fault fix: terminal recovery Task: rust-clippy--cargo-package-lint Background. While the agent was waiting for a long Rust Clippy build, the persistent terminal shell exited. At the next agent-loop entry, the harness raised an exception before its existing sessionrecovery helper could run. This failure prompted a review of how the harness handles terminal loss during ongoing execution. - if not await self._with_block_timeout(self._session.is_session_alive()): raise RuntimeError("The persistent terminal shell exited") + # The previous episode may have outlived the persistent shell. + # Restart it for the next model turn without replaying any + # command whose completion was uncertain. + await self._with_block_timeout(self.ensure_agent_session())

Harness update. The update replaces the exception with a call to the existing session-recovery helper, which preserves a live terminal or starts a replacement if the shell has exited. This prepares a usable terminal for the next model turn without automatically replaying commands whose completion is uncertain. Such commands may already have modified project files, so replaying them could duplicate their effects. Discussion. This case illustrates how full-harness editing allows agents to revise error-handling logic when runtime faults interrupt ongoing tasks. By restoring terminal access, the revised harness can enable the agent to inspect the current task state and resume performance optimization after a shell failure. Together, these cases illustrate how full-harness editing lets agents adapt guidance, tools, and execution logic to the needs of ongoing tasks. Changes to experiment selection and measurement can support task performance by focusing effort on promising hypotheses and improving the reliability of candidate evaluation. Changes to tool execution and context management can reduce model costs by avoiding unnecessary waiting calls and repeated transmission of long histories, while faulthandling revisions support continued execution. A.5

P ER -TASK O PTIMIZATION T RAJECTORIES

Figures 11–15 show optimization trajectories for all 50 test tasks. Each curve geometrically averages three runs, following the scoring and time-accounting conventions of Figure 1. The vertical scale varies by task.

23

Frozen

HE

HS

aiida-core--battery-import

BOXBOX

arrow-rs--variant-wikidata

1.4

16

1.2

8

1

0 bm25s--trec-covid-index

scc--linux-scan 1.4

1.2

1.2

1.05 1 nextcloud-mcp-server--pdf-ingest

ccc--compile-speed

Speedup (×)

1.5

2.4

1.25

1.6

1 cmtraceopen--hadoop-logs

codediff--neovim-release

1.8

1.8

1.5

1.5

1.2

1.2

core--blob-upload

deepinv--kohler-kernels 1.14

1.2

1.08

1.05

1.02 0

1

2

3

4

0

1

Task time (h)

Figure 11: Per-task optimization trajectories (1/5).

24

2

3

4

Frozen

HE

HS

dftd4--lysozyme-gradient

BOXBOX

drasi-core--gtfs-delay-replay

2

1.4

1.5

1.2

1

1 eucalypt--config-render-speed

eucalypt--eval-speed 2.4

50

1.8

25

1.2 0

Speedup (×)

facet--venta-guild

flexmeasures--smard-price-import

1.12

1.2

1.04

1.05

fluidsynth--bagatelles

geoparquet-io--hilbert-sort

1.2

1.5 1.2

1.05

ifcopenshell--gni-obj-export

javaevolved--site-build

1.35 1.5 1.2 1.25 1.05 1 0

1

2

3

4

0

1

Task time (h)

Figure 12: Per-task optimization trajectories (2/5).

25

2

3

4

Frozen

HE

HS

jsonnet--kubernetes-mixin-speed

BOXBOX

zarr--oceananigans-output

1.2 16

1.12

8

1.04

0 trixi--jeans-instability

kitops--minilm-pack

1.4 2 1.2

1.5

1

1

Speedup (×)

kubb--esi

libcachesim--twitter-lru

1.4 1.6 1.2 1.2 1 mai-gen-videob50--score-images

mini-qr--library-batch 100

5

50

2.5

0 mirage--kubernetes-triage

modflow6--neversink 1.1

8 1.05

4

1 0

1

2

3

4

0

1

Task time (h)

Figure 13: Per-task optimization trajectories (3/5).

26

2

3

4

Frozen

HE

HS

monorepo--crypto-build

BOXBOX

n8n-mcp--tool-batch-speed

1.4

2

1.2

1.5

1

1 onnxscript--bert-optimize

pdal--dtm

1.2

3

1.1

2

1

1 physical-ai-toolchain--aloha

plonky3--keccak-proof

Speedup (×)

2 1.08 1.6 1.04 1.2 1 pytrendy--dow-index

qrisp--modmul-compile 1.4

80

1.2

40

1

0 rspress--react-native-paper

rust-clippy--cargo-package-lint 2

1.12 1.6 1.04

1.2 0

1

2

3

4

0

1

Task time (h)

Figure 14: Per-task optimization trajectories (4/5).

27

2

3

4

Frozen

HE

HS

s3s--kubernetes-sync

BOXBOX

solvespace--printer-mesh

1.35

3

1.2

2

1.05

1 strawberry--tlc-query

suitesparse--livejournal-pagerank

8 1.2 4 1.05

Speedup (×)

sum--text-attest

synalinks--kubernetes-tasks

3 2 2

1.5

1

1 trixi--sedov-amr

xet-core--bert-download 2.4

1.4 1.2

1.6

1 hayabusa--evtx-timeline

zenbu--release-source 2.4

1.5 1.8 1.2 1.2 0

1

2

3

4

0

1

Task time (h)

Figure 15: Per-task optimization trajectories (5/5).

28

2

3

4

B

P ERF R ACE : B ENCHMARK D ETAILS

The P ERF R ACE appendix is organized as follows: Section B.1 compares P ERF R ACE with related benchmarks; Section B.2 characterizes its 100 tasks; Section B.3 reports construction and curation statistics; Section B.4 describes the Pair Verification protocol and quality control; and Section B.5 details the evaluation protocol and analyzes gains from longer optimization budgets. B.1

B ENCHMARK C URATION AND E VALUATION

Table 6 compares P ERF R ACE with representative ultra-long-horizon agent benchmarks and codeoptimization benchmarks along task type, task construction, and evaluation procedure. Existing ultra-long-horizon benchmarks largely assemble fixed suites of hand-crafted or expert-curated tasks. In contrast, P ERF R ACE combines live GitHub discovery with agent–human Pair Verification to continually construct validated optimization tasks from real-world projects and workloads. Compared with existing code-optimization benchmarks, it does not derive tasks from known performance patches and evaluates progress repeatedly rather than only at task completion. A workload specifies the inputs and operations used to exercise a project. A microbenchmark measures a narrowly scoped part of that work, such as a selected API call or operation. For example, SWE-fficiency includes a performance workload that repeatedly times a NumPy einsum call (Ma et al., 2026). An end-to-end workload instead measures a complete project-use workflow. P ER F R ACE adopts this broader measurement boundary, leaving agents to profile the workflow and identify which components to optimize. The distinction is therefore the scope of the measured computation: targeted operations versus an end-to-end workflow, rather than whether the benchmark supplies a workload. Table 6: Comparison with related benchmarks. Microbenchmarks measure targeted operations (for example, selected API calls), whereas end-to-end workloads measure complete project-use workflows. Benchmark

Task type

Task construction

Evaluation

P ERF R ACE

Repository optimization on end-to-end workloads

Live GitHub discovery followed by agent–human Pair Verification

AutoLab (Xu et al., 2026)

Diverse metric-driven optimization tasks

Expert-curated optimization tasks

EdgeBench (Zhu et al., 2026)

Cross-domain iterative problem-solving tasks

Expert-curated day-scale tasks

Iterative correctness and performance feedback from an external Judge Local evaluation during the task; held-out final evaluation Local inner loop; repeated submission-gated Judge feedback Final graded submission after up to 20 hours

FrontierSWE (Chu et al., 2026)

Ultra-long-horizon software engineering and research tasks Terminal-Bench (Merrill et al., 2026) General terminal tasks

Real-world problems curated with academic and industry experts

GSO (Shetty et al., 2025)

Microbenchmark-based repository optimization

Crowdsourced tasks with multi-stage human review Repository commit histories and generated performance tests

SWE-Perf (He et al., 2026)

Microbenchmark-based repository optimization

Historical performance PRs and selected repository tests

SWE-fficiency (Ma et al., 2026)

Microbenchmark-based repository optimization

Performance PRs and manually reconstructed workloads

FormulaCode (Sehgal et al., 2026)

Microbenchmark-based repository optimization

Continuously ingested performance PRs and ASV workloads

B.2

Final-only test-based evaluation Final-only microbenchmark-based evaluation Final-only microbenchmark-based evaluation Final-only microbenchmark-based evaluation Final-only microbenchmark-based evaluation

D ETAILED B ENCHMARK C OMPOSITION

We characterize the 100 P ERF R ACE tasks by project purpose, primary programming language, license, packaged source size, file count, and fixed workload. 29

Projects. The projects represented in P ERF R ACE span 16 primary programming languages: Rust (25 tasks), Python (18), TypeScript (15), C++ (11), C (6), C# (6), Julia (5), Go (4), Java (3), Scala (1), Lua (1), OpenSCAD (1), Solidity (1), MATLAB (1), JavaScript (1), and Zig (1). The packaged project sources range from 42 to 15,869 files and from 471.49 KiB to 689.08 MiB, with medians of 886 files and 14.56 MiB. Primary language

Packaged source size 30

25

Rust

Tasks

18

Python 15

TypeScript 11

C++ 6

C#

0

5

Julia

25

20 10

6

C

27

15 10

<2

2–10

10–25

4

Go

12

10

25–50

50–100

≥100

MiB

3 1

Lua

1

MATLAB

1

OpenSCAD

1

25 20 15

Scala

1

Solidity

1

10

Zig

1

5

0

Packaged source files

30

Tasks

Java JavaScript

5

10

15

20

0

25

24 20 15

17 14 9

<250 250–500 500–1k 1k–2.5k 2.5k–5k

Tasks

≥5k

Files

Figure 16: Distribution of primary programming languages, packaged source sizes, and packaged source file counts across P ERF R ACE projects. Task catalog. Table 7 lists each task’s project, purpose, license, and fixed workload. Table 7: Complete catalog of the 100 P ERF R ACE tasks. Task aiidateam-aiida-cor e--battery-archive-i mport anyrender--openmoji

Project and purpose

aiidateam/aiida-core Scientific workflow and provenance platform DioxusLabs/anyrender Rust 2D drawing abstraction apalache--zcash-sync apalache-mc/apalache Symbolic model checker for TLA+ and Quint arrow-rs--variant-w apache/arrow-rs ikidata Rust implementation of Apache Arrow bevy-voxel-world--t splashdust/bevy_voxe errain-streaming l_world Voxel-world plugin for Bevy bm25s--trec-covid-i xhluca/bm25s ndex BM25 search library boyter-scc--linux-v boyter/scc 6-19-json-scan Source-code counting and complexity tool calldiff--pi-review tanishqkancharla/cal ldiff Function call-stack diff tool cbcoutinho-nextcloud cbcoutinho/nextcloud -mcp-server--ohr-pdf -mcp-server -ingest MCP server for Nextcloud

License

Fixed workload

MIT

AiiDA import of a public battery-cycling provenance archive

Apache-2.0

AnyRender batch rasterization of the OpenMoji SVG corpus Apalache’s bounded safety check for the Zcash v2 block-download scheduler

Apache-2.0

Apache-2.0

apache/arrow-rs while encoding Wikidata entities as a Variant column

Apache-2.0

splashdust/bevy_voxel_world terrain generation and chunk meshing

MIT

bm25s indexing and persistence for the TREC-COVID research collection SCC whole-repository JSON scan time on Linux v6.19

MIT

MIT

Calldiff JSON call-stack diffs for a real Pi coding-agent review

AGPL-3.0

Nextcloud MCP Server’s born-digital PDF ingest and page-aware chunking throughput Continued on next page

30

Table 7 (continued) Task

Project and purpose

License

Fixed workload

ccc--compile-speed

anthropics/claudes-c -compiler Dependency-free C compiler Chaste/Chaste Biomedical simulation framework adamgell/cmtraceopen Cross-platform Windows log viewer cocoonstack/cocoon MicroVM engine esmuellert/codediff. nvim Neovim code-review and diff plugin commitizen-tools/com mitizen Commit and changelog automation tool calimero-network/core Peer-to-peer application platform Cpp2Rust/cpp2rust C++-to-safe-Rust translator deepinv/deepinv Deep-learning inverse-problems library dftd4/dftd4 Atomic-charge-dependent dispersion correction manoelpiovesan/dithe r-guy Image dithering application doc-detective/doc-det ective Documentation testing framework drasi-project/drasi-c ore Continuous-query engine Giorgi/DuckDB.NET .NET bindings for DuckDB lalinsky/dusty Zig HTTP client/server library RenderKit/embree Ray-tracing kernels endee-io/endee Vector database EnzymeAD/Enzyme.jl Julia automatic-differentiation library eole-nlp/eole Language-modeling toolkit curvelogic/eucalypt Configuration language and interpreter curvelogic/eucalypt Configuration language and interpreter Tim-Maes/Facet .NET DTO and mapping source generator FlexMeasures/flexmea sures Energy-management platform FluidSynth/fluidsynth SoundFont software synthesizer

CC0-1.0

CCC compile-to-object latency on the SQLite 3.46.0 amalgamation

BSD-3-Clause

Chaste cardiac simulation for a TTP06 ventricular sheet propagation

MIT

CMTrace Open while loading a real Hadoop/YARN log session

MIT

local import of a supported Cocoon Ubuntu VM OCI image CodeDiff C diff latency for a real Neovim release review

geoparquet/geoparque t-io GeoParquet tooling yawkat/GridFlock Gridfinity baseplate generator

chaste--ttp06-sheet

cmtraceopen--hadoo p-logs cocoon--oci-import codediff--neovim-rel ease

commitizen--check

core--blob-upload

cpp2rust--woff2-com press deepinv--kohler-ker nels dftd4--lysozyme-gra dient dither-guy--photo-b atch doc-detective--redpa nda-transforms

drasi-core--gtfs-del ay-replay duckdb--taxi-ingest dusty--fpindex-inges t embree--chess-render endee--esci enzyme--mnist

eole--gpt2-wikitex t-generate eucalypt--config-ren der-speed eucalypt--eval-speed

facet--venta-guild

flexmeasures--smard -price-import fluidsynth-fluidsynt h--bagatelles-midi-r ender geoparquet-io--hilbe rt-sort gridflock--drawer-s tl

MIT

MIT

Commitizen changelog generation for Angular’s complete Git history

Apache-2.0

Calimero content-addressed blob ingestion for a real OpenStreetMap file upload

MIT

Cpp2Rust’s safe Rust output for public-font WOFF2 compression DeepInverse spatially varying blur-kernel estimation on a real Kohler camera-motion-blurred image DFT-D4 energy, analytic gradient, and virial evaluation for a real lysozyme structure

BSD-3-Clause

LGPL-3.0

GPL-3.0

Dither Guy’s full-resolution photo batch export

AGPL-3.0

Doc Detective dry-run planning for Redpanda transform documentation

Apache-2.0

Drasi Core’s continuous Cypher evaluation for replayed GTFS-Realtime vehicle geofence updates

MIT

DuckDB.NET ingestion of NYC Yellow Taxi trip records Dusty’s HTTP batch-ingestion path for real AcoustID fpindex updates Embree triangle traversal for a public chess-scene visibility render Endee HTTP dense search for a replay of Amazon Shopping Queries product searches Enzyme.jl for Flux’s CPU LeNet MNIST training epoch

MIT Apache-2.0 AGPL-3.0 MIT

MIT MIT OR Apache-2.0

EOLE CPU GPT-2 batch text generation on WikiText-103 prompts the eucalypt interpreter on a config-generation workload

MIT OR Apache-2.0

the eucalypt interpreter on an Advent of Code workload

MIT

Facet package and source-generator compilation time for venta’s Guild service

Apache-2.0

FlexMeasures import of a full year of public SMARD day-ahead prices

LGPL-2.1

FluidSynth offline rendering of a complete CC0 MIDI performance to WAV

Apache-2.0

GeoParquet Hilbert ordering of an Overture Buildings shard

MIT / CC-BY-4.0

GridFlock’s segmented Gridfinity drawer-baseplate STL export Continued on next page

31

Table 7 (continued) Task

Project and purpose

License

Fixed workload

ifcopenshell--gni-b improject-obj-export

IfcOpenShell/IfcOpen Shell IFC library and geometry engine intro-skipper/intro-s kipper Jellyfin intro and credits detector JeroenGar/jagua-rs 2D collision engine for nesting javaevolved/javaevol ved.github.io Java developer documentation site google/jsonnet Data templating language JuliaIO/Zarr.jl Julia implementation of Zarr array storage JuliaLang/julia Julia programming language kitops-ml/kitops OCI packaging tool for ML artifacts kiwifs/kiwifs Markdown filesystem for agents and teams Cosmian/kms Key-management system

LGPL-3.0

IfcOpenShell IfcConvert OBJ export for a lighter GNI BIM Project architectural IFC delivery

GPL-3.0

Intro Skipper’s movie credits black-frame evidence extraction

MPL-2.0

jagua-rs collision collection replayed from a downstream nesting solver the full localized GitHub Pages build for java.evolved

kubb-labs/kubb OpenAPI code-generation framework SakuraMathcraft/LaTe XSnipper Formula OCR and editing tool 1a1a11a/libCacheSim Cache simulation library neomutt/lsp-tree-sit ter Tree-sitter language-server library Nick-bit233/mai-gen-v ideob50 Maimai score-video generation tool Yeraze/meshmonitor Mesh-node monitoring web tool morpho-org/metamorpho Tokenized lending-vault protocol lyqht/mini-qr QR-code generator and scanner strukto-ai/mirage Virtual terminal for coding agents MODFLOW-ORG/modflow6 Groundwater-flow simulator commonwarexyz/monore po Distributed-systems primitives SINTEF-AppliedCompSc i/MRST Reservoir-simulation toolbox systemsomicslab/Msdi alWorkbench Mass-spectrometry analysis workbench hugr-lab/mssql-exten sion DuckDB extension for Microsoft SQL Server czlonkowski/n8n-mcp MCP server for n8n workflow construction

MIT

intro-skipper--credi ts

jagua--nesting javaevolved--site-b uild

jsonnet--kubernete s-mixin-speed juliaio-zarr-jl--oce ananigans-timeserie s-output julialang-julia--tri xi-jeans-instability kitops--minilm-pack

kiwifs--kubernetes-r eindex kms--batch-encrypt

kubb--esi

latexsnipper--formu la-batch libcachesim--twitter -lru lsp-tree-sitter--ter mux-check

mai-gen-videob50--s core-images

meshmonitor--coverag e-grid metamorpho--realloc ate mini-qr--library-bat ch mirage--kubernetes-t riage modflow6--neversink monorepo--crypto-bui ld mrst--norne-import

msdialworkbench--lcm s-alignment

mssql-extension--tax i-copy

n8n-mcp--tool-batch -speed

MIT

Apache-2.0 MIT

MIT Apache-2.0

the Jsonnet C++ interpreter on a Kubernetes monitoring configuration workload Zarr.jl directory-store output for a real Oceananigans tracer time series Julia runtime execution for a published Trixi Jeans gravitational-instability simulation KitOps local ModelKit packaging for a complete all-MiniLM-L6-v2 release snapshot

BUSL-1.1

KiwiFS SQLite FTS5 reindex for Kubernetes documentation

BUSL-1.1

Eviden KMS KMIP BulkData AES-GCM encryption for concurrent application-record batches Kubb regeneration of JitaSpace’s EVE Online ESI client

GPL-3.0

resident CPU formula OCR for a public OpenStax textbook screenshot batch

Apache-2.0

Four-capacity LRU cache-size sweep over a production Twitter request trace Tree-sitter validation of a real Termux package-recipe corpus

GPL-3.0

MIT

batch maimai score-image rendering for a real 50-record archive

BSD-3-Clause

MeshMonitor’s SQLite coverage-grid batch

GPL-2.0

Gas use for a historical MetaMorpho market reallocation

GPL-3.0

Mini QR’s batch SVG export for a public library directory Mirage recursive source search for Kubernetes dual-stack readiness-probe triage

Apache-2.0

CC0-1.0 Apache-2.0

Complete steady-state simulation of the USGS Neversink–Rondout groundwater model the release build latency of commonware-cryptography

GPL-3.0

MRST Norne field-model import and geometry construction

LGPL-3.0

MS-DIAL Console LC-MS peak processing and alignment for a real negative-ion DDA batch

MIT

BCP bulk import of NYC taxi records through the DuckDB MSSQL extension

MIT

n8n-mcp on a fixed MCP tool-call batch

Continued on next page

32

Table 7 (continued) Task

Project and purpose

License

Fixed workload

nerdbank-messagepack --earthquake-fanout

AArnott/Nerdbank.Mes sagePack .NET MessagePack library Open-J-Proxy/ojp Database proxy and JDBC driver ModernRelay/omnigraph Lakehouse-native graph engine microsoft/onnxscript ONNX model authoring and optimization library bzsanti/oxidizePdf Rust PDF library for RAG PDAL/PDAL Point-cloud processing library JuliaParallel/PETSc. jl Julia wrappers for PETSc microsoft/physical-a i-toolchain Physical-AI dataset analysis toolkit Plonky3/Plonky3 Polynomial IOP toolkit polkadot-api/polkado t-api Modular Polkadot API libraries RussellSB/pytrendy Trend-detection library eclipse-qrisp/Qrisp Quantum-algorithm development framework Soomgo-Mobile/react-n ative-code-push React Native OTA update client EnzymeAD/Reactant.jl MLIR/XLA compiler for Julia web-infra-dev/rspress Static-site generator rust-lang/rust-clippy Rust linter s3s-project/s3s S3 service adapter code-yeongyu/senpi Coding-agent runtime ProvableHQ/snarkOS Decentralized operating system for ZK applications ProvableHQ/snarkVM Zero-knowledge virtual machine solvespace/solvespace Parametric 2D/3D CAD

MIT

SignalR MessagePack earthquake snapshot fan-out

Apache-2.0

OJP JDBC export of public NYC TLC taxi records

MIT

OmniGraph’s paginated metabolic clinical-trial refresh and lifecycle dashboard Graph optimization of a BERT-SQuAD FP32 and INT8 model bundle

stoplightio/spectral OpenAPI, JSON, and YAML linter quarylabs/sqruff SQL formatter and linter strawberry-graphql/s trawberry Python GraphQL library DrTimothyAldenDavis/ SuiteSparse Sparse-matrix and graph computation suite OtotaO/SUM Cross-runtime text-attestation library llnl/sundials Nonlinear and differential-equation solver suite

Apache-2.0

Spectral linting of a production OpenAPI batch

Apache-2.0

Sqruff directory linting for Mozilla GLAM ETL SQL Strawberry GraphQL execution for Graphique’s real TLC weekly Parquet column query

ojp--taxi-export

omnigraph--metabolic -trials-refresh onnxscript--bert-opt imize oxidize-pdf--rag-chu nking pdal--dtm petsc--ocean-age-sol ve physical-ai-toolcha in--action-smoothnes s plonky3--ethereum-c alldata-keccak-proof polkadot-api--descr iptor-generation pytrendy--dow-index qrisp--modmul-compi le react-native-code-p ush--mattermost-rel ease reactant--fashion-m nist-train rspress--react-nativ e-paper rust-clippy--cargo-p ackage-lint s3s--kubernetes-sync senpi--image-batch snarkos--cdn-replay

snarkvm--mainnet-exe cution-verify solvespace-solvespac e--printer-assembl y-mesh-export spectral--api-lint

sqruff--glam-etl strawberry-graphql-s trawberry--graphique -tlc-weekly-columns suitesparse--livejou rnal-pagerank

sum--text-attest

sundials--cantera-i gnition

MIT

MIT BSD-3-Clause MIT

oxidize-pdf RAG chunking over a real CC-BY PDF corpus Ground classification and 1 m terrain-model generation from USGS 3DEP LiDAR PETSc.jl GMRES+ILU for a public OCCA ocean-age transport solve

MIT

multi-dataset LeRobot action-smoothness analysis for real ALOHA manipulation recordings

Apache-2.0

Plonky3 Ethereum calldata Keccak trace generation and STARK proving PAPI’s multi-chain TypeScript descriptor generation

MIT

MIT EPL-2.0

PyTrendy trend detection for a real daily Dow Jones Industrial Average price series Qrisp compilation of controlled modular multiplication

MIT

CodePush Android OTA releases for Mattermost Mobile

MIT

Reactant CPU execution of a real Lux Fashion-MNIST training epoch Rspress production builds of the React Native Paper documentation site Clippy lint time for Cargo’s main production package s3s-fs recursive S3 synchronization of Kubernetes pkg source Senpi image attachment preprocessing for a real screenshot batch snarkOS CDN block bundle replay over 150 fixed Aleo Mainnet blocks

MIT Apache-2.0 Apache-2.0 MIT Apache-2.0

Apache-2.0 GPL-3.0

MIT

Apache-2.0 / BSD-2-Clause (used components) Apache-2.0

BSD-3-Clause

snarkVM verification of a batch of accepted Aleo mainnet executions SolveSpace CLI mesh export for a complete Cartesian 3D-printer assembly

Complete PageRank computation on the LiveJournal social graph using GraphBLAS and LAGraph SUM while batch-attesting a corpus of real Wikipedia prose SUNDIALS CVODES for a downstream Cantera methane ignition parameter sweep Continued on next page

33

Table 7 (continued) Task

Project and purpose

License

Fixed workload

svf-tools-svf--nginx -andersen-wpa

SVF-tools/SVF Static value-flow analysis framework SynaLinks/synalinks Neuro-symbolic language-model framework zama-ai/tfhe-rs Fully homomorphic encryption library maptiler/tileserver-g l Vector and raster map-tile server trixi-framework/Trixi .jl Adaptive numerical-simulation framework Myriad-Dreamin/typst .ts Typst runtime for JavaScript proximafusion/vmecpp Magnetohydrodynamic equilibrium solver wiredtiger/wiredtiger Embedded storage engine huggingface/xet-core Content-addressed transfer and reconstruction engine Yamato-Security/haya busa Windows event-log forensics tool JetBrains/youtrackdb Object-oriented graph database zenbu-labs/zenbu.js JavaScript application framework lalinsky/zio Zig asynchronous-I/O framework

AGPL-3.0-orlater

SVF Andersen whole-program pointer analysis on NGINX reverse-proxy bitcode

Apache-2.0

Synalinks Markdown knowledge-base ingestion for Kubernetes task documentation

BSD-3-ClauseClear

TFHE-rs CPU execution for confidential ERC-7984 token transfers

BSD-2-Clause

TileServer GL static PNG rendering for a full Natural Earth country atlas

MIT

a published self-gravitating Sedov blast-wave simulation with adaptive mesh refinement

Apache-2.0

Typst.ts rendering in the Napkin Community production static-site build

MIT

VMEC++ fixed-boundary precise-QH equilibrium solves

GPL-2.0 OR GPL-3.0 Apache-2.0

WiredTiger’s durable import of a real Wikimedia pageview hour local Xet reconstruction of a production BERT model artifact

AGPL-3.0

Hayabusa bulk external attack-EVTX JSONL timeline generation

Apache-2.0

embedded YouTrackDB import of a real OpenStreetMap road network Zenbu.js staging of the Zenbu desktop application’s release source

synalinks--kubernete s-tasks tfhe-rs--erc7984-tra nsfer tileserver-gl--count ry-atlas trixi-framework-tri xi-jl--self-gravita ting-sedov-amr typst--napkin

vmecpp--precise-qh

wiredtiger--pagevie ws xet-core--bert-downl oad yamato-security-hay abusa--external-att ack-evtx-timeline youtrackdb--osm-roa ds zenbu--release-sourc e zio--zrk-typecheck

B.3

MIT

MIT

ZIO’s downstream zrk ReleaseFast type-check latency

C ONSTRUCTION AND C URATION S TATISTICS

This section describes the implementation of Discovery and Filter and reports the candidate counts and curation effort at each construction stage. Discovery implementation. We query the ClickHouse github_events table for opened pull requests and branch pushes by a configured list of coding-agent accounts, then group matching events by repository. For each repository, we retain the latest matching pull-request or branch URL and use the GitHub GraphQL API to record its canonical identity, default-branch revision, stars, primary language, license, disk usage, and repository status. The resulting records are deduplicated by GitHub repository, and projects are excluded when their repository, default branch, or recorded source revision cannot be retrieved. Each step writes its results incrementally, allowing interrupted collection and metadata retrieval to resume without repeating completed work. From GitHub activity recorded between May and August 2026, we identified 118,517 repositories, retrieved metadata for 104,428, and retained 103,475 project candidates after these checks. Table 8: GitHub activity signals used for Discovery. Signal

Repositories

Branch push by a coding-agent account Pull request opened by a coding-agent account

34

112,127 6,390

Filter implementation. We first apply deterministic checks to repository metadata and the recorded Git tree. These checks require an accessible source revision, a license permitting modification and redistribution, 10–4,999 GitHub stars, and a tracked Git tree of at most 500 MiB and 25,000 entries. Projects that pass these checks are collected at the recorded revision with their README, setup documentation, package manifests, license files, and representative source paths. We then use GPT5.6-Luna to assess the project’s setup path, evaluable process, runtime fit, and optimization potential from this source snapshot using a structured response schema. The prompt is provided in Appendix D.1. We require each structured response to address the following criteria. □ Setup path. Repository files support concrete commands for installing dependencies and building or running the project. □ Evaluable process. The project supports a finite process with a fixed input, observable output, performance metric, and correctness check. □ Runtime fit. The process can run in a Linux x86-64 container on commodity CPUs without mandatory special hardware. □ Optimization potential. The project supports a substantial local computation whose cost is expected to increase with input size or repeated execution. Using these assessments, we selected 239 high-potential project candidates for Pair Verification. Table 9 reports the first check that excludes each project from the final Pair Verification selection. Table 9: First failed check during Filter and the final selection result. Result

Projects

Fewer than 10 GitHub stars Accessibility Setup path Runtime fit Optimization potential Evaluable process Assessment error

93,992 1,134 212 359 4,462 3,066 11

Selected for Pair Verification

239

Curation effort. To quantify the effort required to turn selected projects into accepted tasks, Figure 17 reports the agent cost, curation time, and number of agent–human interactions for the 100 accepted tasks. The median task required 4.5 hours, $8.83 in agent cost, and eight agent–human interactions, compared with averages of 9.1 hours, $11.35, and 12 interactions. Curation time 31

31

Agent cost

Median: 4.5 h

40

34

Median: $8.83

15

14

9

20

20

30

23

20

14

11

9

10

Median: 8

40

40 Tasks

20 10

36

30 Tasks

Tasks

30

Agent–human interactions

12

10 1

0

<2

2–4

4–8 Hours

8–16

≥16

0

<5

5–10 10–20 20–40 ≥40 USD

0

1–4

5–8

9–12 13–20 ≥21

Interactions

Figure 17: Distribution of curation time, agent cost, and agent–human interactions across the 100 accepted tasks. 35

B.4

PAIR V ERIFICATION

P ERF R ACE uses Pair Verification to ensure that each task represents meaningful project use, produces reproducible measurements, and rejects shortcut solutions; this section describes the agent– human protocol and analyzes the resulting curation trajectories. B.4.1

P ROTOCOL

Agent

Human reviewer

Explore & Design

Build & Calibrate

Validate Integrity

Finalize

workload, input, utility, and metric

task, verifier, and measurement protocol

invalid and shortcut solutions

complete task package

Realism

Reliability

Integrity

meaningful project use

stable and reproducible

reward reflects real work

ACCEPT

Pair Verification combines agent-led task construction with human review to turn each project candidate into an executable task and assess its realism, reliability, and integrity. The construction skill is provided in Appendix D.2. The agent develops the task in stages and pauses for a reviewer decision after completing the evidence required for each quality criterion.

Figure 18: Pair Verification alternates between agent-led task construction and human review at the Realism, Reliability, and Integrity gates. Explore and Design. The agent first studies the project’s documentation, examples, code paths, and public usage evidence to understand its purpose, users, and performance-relevant operations. The agent searches public release pages for source code, models, and assets, scientific or government data portals, downstream repositories, production archives, and real user exports to identify candidate workload inputs. For each input, the agent records its download location, version, and applicable terms, and accepts it only when an explicit license, public-domain designation, or written authorization permits redistribution with the benchmark. It uses these materials to identify candidate operations, but grounds the workload and scale in a real user or downstream application rather than repository-native tests, benchmarks, examples, or fixtures. The agent compares plausible workload candidates, then traces the selected operation from its user-facing entry point through the relevant project code and demonstrates a small input-to-output execution on the unmodified revision. During this run, it records dependency installation, project build, input preparation, and operation time to check that setup does not dominate the measurement and to estimate whether complete verification can finish within the 1,200-second budget. The agent repeats the operation on identical inputs to detect run-to-run flakiness, including sporadic failures or inconsistent correctness-relevant outputs, and to identify any dependence of runtime on random state, network access, or external services. A candidate is revised or rejected if its stochastic execution remains too variable for reproducible performance measurement after fixing available random seeds and aggregating repeated runs. It then specifies the exact operation, workload and scale available to the agent, held-out correctness cases, observable correctness utility, performance metric, and measurement boundary. Before requesting Realism review, the agent summarizes the project’s purpose and users, the selected execution path, the basis for the input and scale, and the alternatives it considered and rejected. The reviewer approves Realism only when the operation, input, and scale represent meaningful project 36

use, the metric measures the stated utility, and the workload inputs can be redistributed as part of the benchmark. Build and Calibrate. After Realism approval, the agent derives the installation, build, and run commands from upstream documentation, CI configuration, and package manifests, and verifies them on the pinned source revision. The agent implements the verifier, the task’s test program that measures the performance metric on a separate copy of the workload and checks additional held-out cases for correctness. It then constructs the Harbor task files with instruction.md, task.toml, an environment/ directory containing the editable project and workload, and a separate tests/ directory containing the verifier code, workload copy, held-out cases, and expected outputs. The agent pins the project revision, base image, language toolchain, dependencies, workload assets, and expected outputs so that the image can be rebuilt and the task rerun without downloading project source or workload inputs. The task instruction tells the agent how to build and run the project, what it may edit, which behavior it must preserve, and which performance metric to improve. The agent then builds a fresh image and runs the verifier on the unmodified project before calibrating performance measurements. The agent selects the smallest resource profile that completes the workload without CPU, memory, or storage pressure and sets the verifier timeout within 1,200 seconds. The agent uses pilot and repeated runs to choose the workload scale, warmups, repetitions, and aggregation, requiring the final measurements to have a coefficient of variation below 10% while keeping build and setup outside the timed operation. The agent checks that the timing boundary covers the complete operation requested in the task while excluding one-time environment setup and verifier preparation that are unrelated to the optimization objective. For the n timed measurements retained after warmup, x1 , . . . , xn , we compute v u n n u 1 X s 1X CV = , xi , s=t (xi − x̄)2 . x̄ = x̄ n i=1 n − 1 i=1 After fixing the workload, correctness checks, resources, and measurement protocol, the agent builds the final image and evaluates the unmodified baseline and one optimization candidate under the same conditions. The reviewer examines the fresh-build correctness results, repeated measurements and CVs, verification time, resource allocation, and candidate trajectory. The reviewer approves Reliability only when fresh builds and repeated runs return the same correctness verdict and stable performance measurements with the chosen warmups, repetitions, aggregation, and measurement boundary. Validate Integrity. To identify reward-hacking paths that could receive a valid score without completing the task, the agent tests deliberately incorrect, partial, hard-coded, and work-skipping solutions and checks that workload inputs, expected outputs, and verifier-owned data cannot be replaced by agent-controlled files. The agent inspects the evaluation image for leaked expected outputs, verifier code, or construction notes that could reveal the hidden checks. For example, in rust-clippy-cargo-package-lint, removing Cargo dependency tracking made later runs reuse a cached lint result, which the verifier detects by touching clippy-driver, running the workload twice against the same warm target, and requiring each Cargo JSON trace to report a non-fresh compiler artifact for cargo. The reviewer approves Integrity only when every requirement in the task instruction is checked, every verifier check follows from the declared task, and none of the tested shortcut solutions can receive a valid score. Finalize. After all three reviews, the reviewer chooses ACCEPT, REVISE, or REJECT; accepted tasks pass final compatibility checks before publication as a task package comprising an immutable Docker image and task artifact. B.4.2

Q UALITY C ONTROL

We analyze the recorded agent–human Pair Verification trajectories for evidence of the Realism, Reliability, and Integrity of the resulting tasks. Of the 239 candidates, 132 were rejected during Explore and Design, six during Build and Calibrate, none during Validate Integrity, and one during Finalize, leaving 100 accepted tasks. 37

132

Explore & Design 6

Build & Calibrate

Validate Integrity

0

Finalize

1 0

20

40

60

80

100

120

140

Rejected candidates

Figure 19: Candidates rejected at each Pair Verification stage, with candidates lacking a recorded transition into Build & Calibrate assigned to Explore & Design.

Realism. Realism checks cover the workload operation and scale, the source and redistribution terms of its inputs, and the alignment between the declared utility and performance metric. Table 10 groups the workload inputs into released source code, models, or assets (26 tasks), public scientific or government datasets (21), real downstream applications (21), production traces or archives (16), real user exports (6), and other independently grounded inputs (10).

Table 10: Sources of workload inputs across the 100 accepted tasks. Input source

Examples

Tasks

Released source code, model, or asset Public scientific or government dataset Real downstream application Production trace or archive Real user export Other independently grounded input

Linux source tree, OpenMoji assets, released ML models

26

TREC-COVID, NYC taxi records, LC-MS data

21

React Native Paper, Headlamp, Mattermost Mobile Hadoop logs, GTFS delay feeds, GitHub events MIDI collection, BIM project, printer assembly CAD Wikidata variants, Dow index, OSM roads

21 16 6 10

Reliability. All 99 wall-clock tasks have a final unmodified-project CV below 10%; the remaining task, metamorpho-reallocate, measures transaction gas, the deterministic execution cost assigned by the Ethereum Virtual Machine to a fixed smart-contract transaction, and returns exactly 874,508 gas in three fresh containers. Table 11(a) shows that 84 of the 99 wall-clock tasks have a final CV below 6%. Integrity. We validate Integrity across six attack classes: empty or no-op solutions, skipped work, hard-coded outputs, precomputed or cached outputs, modification of protected inputs or tests, and access to verifier-only data. All 100 accepted tasks completed an explicit red-team pass before Integrity approval. Table 11(b) shows that 40 tasks were tested against at least two red-team cases, including two tasks with seven or eight cases. For example, SCC is a source-code statistics tool whose task scans the Linux v6.19 tree and emits JSON statistics; during red-team testing, the agent replaced the scanner with a program that returned the precomputed Linux result instead of scanning the source tree, which passed the workload available to the agent but failed when the verifier asked it to scan a separate pinned checkout of SCC’s own source code. 38

(a) Final measurement CVs CV range

(b) Red-team tests

Tasks

Tests per task

Tasks

20 64 8 3 4 0

1 2 3 4 7 8

60 20 12 6 1 1

0% ≤ CV < 2% 2% ≤ CV < 6% 6% ≤ CV < 7% 7% ≤ CV < 8% 8% ≤ CV < 9% 9% ≤ CV < 10%

Table 11: Reliability and integrity evidence: final CVs for the 99 wall-clock tasks and red-team tests for all 100 tasks, with one test assigned conservatively when an exact count cannot be recovered.

B.5

E VALUATION P ROTOCOL

For each submission, the Judge snapshots the agent’s current container and starts a temporary shadow container from that snapshot. The Judge creates the snapshot with docker compose commit, launches the shadow container under a separate Compose project, and removes its container, local image, and volumes after verification. The agent container remains paused during verification and resumes from the same workspace state after the shadow container is removed. Inside the shadow container, the verifier runs through tests/test.sh and emits the correctness verdict and performance metric as JSON. The verifier code, workload copy, held-out cases, and expected outputs are absent from the agent container and available only inside the shadow container. Each task declares its warmups, repetition count, aggregation rule, and required correctness pass rate in task.toml, and its verifier returns a performance metric only when that pass rate is met. The task clock continues while the shadow is created, executed, and removed, so every submission trades part of budget B for additional correctness and performance feedback. After verification, the Judge returns the correctness verdict and performance metric to the agent as feedback for subsequent optimization. For a correct submission, the Judge computes the speedup score by dividing the unmodified baseline metric by the submission metric. Submissions that fail correctness, exceed the verifier timeout, or do not produce a valid performance metric cannot update the best-so-far trajectory S(t).

Tasks

Five minutes before the deadline, the agent receives a final delivery prompt asking it to leave the project in a coherent state before the Judge performs one final shadow verification of the current workspace. Figure 20 characterizes the time required for one complete Judge evaluation, showing that baseline Judge evaluation remains within the 20-minute limit for all 100 final tasks. 24

Median: 4.9 min

21

Mean: 6.1 min

18

Evaluation limit: 20 min

15 12 9 6 3 0

0

2

4

6

8

10

12

14

16

18

20

Complete baseline Judge evaluation time (minutes)

Figure 20: Distribution of complete baseline Judge evaluation time across the 100 final task packages, with a mean of 6.1 minutes, a median of 4.9 minutes, and a 20-minute evaluation limit.

Longer optimization budgets. To measure the benefit of additional optimization time, we run Terminus-KIRA with GPT-5.6-Luna at high reasoning effort three times on each of the 88 evaluated tasks, maintaining one persistent agent session across Judge submissions during every four-hour run. 39

The aggregate best-so-far correct speedup rises from 1.314× after one hour to 1.377×, 1.396×, and 1.435× after two, three, and four hours, respectively. Across the resulting 264 trajectories, Figure 21 shows that extending Terminus-KIRA’s budget from two to four hours improves the overall score by 4.2%, and the fourth hour alone adds 2.8%. The four-hour score is at least 5% higher on 24 tasks than at two hours and on 12 tasks than at three hours, showing that agents continue to find meaningful improvements late in the task. Among the 228 runs that improve over baseline, the best result is first reached at a median of 88.1% of the four-hour budget, and 51 of the 264 runs produce a new improvement of at least 5% during the final hour. (a) Overall gains from longer runs +4.2%

(b) Task-level gains from longer runs 4

2h to 4h

2h to 4h

17

43

8

16

+2.8%

3h to 4h

8

3h to 4h

0

2

4

6

0

35

20

Aggregate best-so-far gain (%)

33

40

60

5

80

7

100

Share of tasks (%) No gain

0–1%

1–5%

5–10%

>10%

Figure 21: Performance improvements from additional optimization time, based on three four-hour runs for each of 88 P ERF R ACE tasks. Left: increase in the overall score at four hours relative to two and three hours; horizontal lines show 95% bootstrap intervals. Right: number of tasks in each improvement range.

40

C

B OX B OX : M ETHOD D ETAILS

Figure 22 summarizes the task and review loops. The following explains how harness revisions affect subsequent interactions and how review, hot reloading, and recovery preserve task continuity. B OX B OX State: harness H, runtime R, task environment e, task trajectory τ . Review state: environment er and separate trajectory τr . Budget: shared task budget B. 1 while budget B remains do

a ∼ pM,H (· | τ ) if a = boxbox() then 4 Yield control to the kernel; suspend task execution and preserve context 5 Prepare (er , τr ) for independent review under H, with access to τ 6 while true do 7 ar ∼ pM,H (· | τr ) 8 if ar = commit(H ′ ) then break 9 (er , or ) ← ExecuteH (er , ar ) 10 τr ← τr ⊕ (ar , or ) 11 end while 12 if H ′ ̸= H then 13 The kernel validates H ′ and attempts activation of its runtime R′ 14 if successful then select (H, R) ← (H ′ , R′ ) 15 else retain or restore the previous harness and runtime 16 end if 17 Restore continuation context and return control to the selected runtime 18 else 19 (e, o) ← ExecuteH (e, a) 20 τ ← τ ⊕ (a, o) 21 end if 22 end while 2 3

Figure 22: B OX B OX pseudocode. Task execution and independent harness review form nested interaction loops, with kernel-managed hot reloading connecting them. C.1

E FFECTS ON S UBSEQUENT I NTERACTIONS

Equation 1 describes two ways a harness revision can affect subsequent execution while the language model M remains fixed. Changes to how the harness constructs model inputs can alter pM,Hk , influencing the agent’s next action. Changes to ExecuteHk can alter how that action is carried out and how its result is returned as an observation. A revision can affect either or both, so its influence extends to the trajectory generated after activation. The current harness also governs the review loop. Once a revision is activated, subsequent reviews use the updated implementation and can examine the experience accumulated under it. Harness changes can therefore affect both task execution and the process used to produce later revisions. Appendix A.4 illustrates concrete changes to guidance, measurement, command polling, and context handling. C.2

R EVIEW

The task-solving agent requests review through boxbox() without having to propose a patch. The initial harness requires intervening task activity and a 30-minute cooldown after each completed review; once eligible, the agent decides whether and when to request another review. The kernel preserves the task conversation and launches a separate agent review session under the current harness. The reviewer can inspect task trajectories and Git history, but edits only the harness, treating 41

project code as read-only. It may retain the current harness, add guidance, or modify executable behavior, committing any changes before ending review. The default review skill is provided in Appendix D.3. C.3

E XECUTION C ONTEXT

Execution context includes both runtime-owned state and the existing task environment. The former includes the task conversation, review history, and records of which trajectory entries have already been reviewed. These are exported as a state snapshot. The terminal session and project workspace remain in place: shell state, ongoing processes, and project files are preserved through the existing environment connection. When the agent calls boxbox(), the runtime yields control to the kernel, suspending the task interaction loop while review proceeds separately. This pauses the agent’s task decisions, not the entire container or its processes. Before importing state into a candidate runtime, the kernel makes a separate copy, preserving the original snapshot for recovery if the candidate fails. This separates changing the agent implementation from restarting its task environment. C.4

H OT R ELOADING

Harness revisions are tracked as Git commits, while the kernel remains separate from the editable harness. Before switching, the kernel checks the committed candidate, loads its runtime, and verifies that it supports task execution and context transfer. The current runtime keeps its terminal connection until the candidate successfully imports the copied state. The kernel then transfers the same terminal session to the candidate and returns control to it, resuming task interaction from the preserved conversation in the same workspace. If activation fails, the kernel retains or restores the previous runtime and context. For agent-requested reviews, the reviewer can leave concise guidance on using new capabilities and continuing the task. This guidance is preserved across the update, and the resumed agent is prompted to read it before continuing. Task execution and review share the same budget and cost accounting. Table 12 summarizes how execution continues under each outcome. Table 12: Task continuation across review and runtime-update outcomes. Outcome

Continuation

No harness change Successful update

Resume the current runtime with the preserved task context. Continue under the revised harness with transferred context, the existing terminal session, and the same project workspace. Retain or restore the previous runtime and its preserved context. Enter recovery and review using the retained execution records; restart the terminal while preserving project files.

Update fails Runtime fault

C.5

R ECOVERY

The kernel retains a recovery point across harness updates. A newly activated runtime remains provisional until it reaches the next agent-requested review without a runtime fault. If it fails before that boundary, the kernel restores the previous confirmed harness and state and initiates a review using the fault evidence. Faults in a confirmed runtime also trigger review. Runtime-fault recovery starts a fresh terminal, so shell-local state is lost, but project files, harness history, and execution records remain available. The retained records let subsequent reviews examine failed revisions and help the task-solving agent determine how to continue interrupted work.

42

D

P ROMPTS AND S KILLS

This appendix presents the Filter prompt and Pair Verification skill used in benchmark construction, followed by the default B OX B OX self-review skill. D.1

F ILTER P ROMPT

The Filter assesses repository snapshots before Pair Verification. Filter Prompt You are the automated Filter for PerfRace project candidates. Decide whether the supplied repository snapshot provides enough source-grounded evidence to proceed to Pair Verification. Evaluate four gates: 1. Accessible. The repository and revision are available, with an explicit license permitting modification and redistribution. A fork must have an independently maintained product identity. 2. Upstream reproducibility. Documentation, scripts, or manifests support concrete setup/build and minimal smoke commands. Cite the supporting path or manifest field for each command. Dependencies and configuration must come from identifiable public sources. 3. Evaluation potential. Identify a finite project operation with fixed inputs, clear start and end, observable output, a plausible efficiency metric, and a correctness check. Establish a replayable real user or downstream workload, or a self-contained computation with grounded input semantics. Repository tests, benchmarks, demos, fixtures, and generated inputs do not establish a portable workload. Flag reliance on unavailable deployment state, private traces, physical devices, or an unresolved workload source. 4. Resource feasibility. The operation must run in a Linux x86-64 container on commodity CPUs within ordinary resource limits. Reject mandatory special hardware, unavailable private services, and thin remote pass-throughs. Local services or fixed stubs are acceptable only when meaningful project-controlled computation remains. Evidence and optimization potential: - Use only the supplied snapshot; cite metadata or repository paths for every gate. Missing evidence fails the gate. - Describe a preliminary process only when evaluation potential passes. - Describe the local computation and its strongest optimization opportunity. Rate potential as low for tiny or externally dominated work, medium for a clear local hot path, and high for scale-sensitive computation with multiple structural opportunities. - Return concise evidence in the requested structured response.

D.2

PAIR V ERIFICATION S KILL

The skill guides agent-led construction through human reviews of Realism, Reliability, and Integrity, followed by task publication. Pair Verification Skill # PerfRace Pair Verification

43

Pair Verification Skill (continued) Turn one repository into a machine-evaluable task. The agent investigates and executes; the human reviews Realism, Reliability, and Integrity, then decides ACCEPT, REVISE, or REJECT. ## Working agreement Maintain a report linking claims to project sources, runnable probes, and raw results. At each review gate, present the evidence, uncertainty, and recommendation, allow discussion, and pause for the human decision. Reopen the affected reviews when later evidence invalidates an approved claim. ## Stage 1: Explore & Design Understand the project’s users and performance-sensitive workflows. Compare candidate workloads and select an independently grounded real user or downstream operation with reproducible inputs and scale. Upstream examples and benchmarks may guide discovery but must not become the scored workload. Run the unmodified workload end to end and identify the project execution path. Estimate build cost, image size, verification time, and measurement variance before constructing the task. Propose the operation, input provenance and scale, visible workload, independent verifier copy, additional correctness cases, correctness utility, measurement boundary, and metric. Human review: Realism. Explain why the operation, inputs, and scale represent meaningful project use, which alternatives were considered, and what remains uncertain. Obtain approval before building the canonical task. ## Stage 2: Build & Calibrate Package the pinned project, visible workload, instructions, container, and verifier. Keep only the project writable. Use an independent byte-identical scored input and additional held-out correctness cases. Correctness checks enforce the workload’s observable utility, allowing implementation changes that preserve it. Keep candidate compilation and setup outside the timed user operation. Calibrate workload scale, warmups, repetitions, resources, and timeout; then freeze the measurement protocol. Complete verification must fit within 1,200 seconds, and the uncompressed image must not exceed 3 GiB. Target at least three seconds per timed execution; shorter operations require a clear signal and repeated CV at most 10%. Check fresh-image correctness and flakiness. Diagnose variance before increasing workload size or resources. Enforce the same runtime policy for baseline and candidate. Run a baseline and one bounded candidate-agent attempt, retaining the trajectory and raw measurements. One correct, repeatable gain is sufficient; an incorrect or no-gain candidate does not reject the task if its trajectory demonstrates understanding and a reasonable optimization attempt. Human review: Reliability. Present fresh-build correctness, raw measurements, dispersion, verification time, resources, and the candidate trajectory. Ask whether equivalent runs support the same verdict and measurement. ## Stage 3: Validate Integrity

44

Pair Verification Skill (continued)

Audit agreement between instructions and verifier checks, including writable paths, workload copies, expected outputs, caches, and reward inputs. Run explicit red-team tests for leaked-data access, verifier modification, incomplete work, hard-coded answers, workload-specific precomputation, frozen runtime-policy overrides, and reward-path bypasses. Count an attack only when a materially distinct invalid implementation reaches the final verifier. After hardening, replay the attack and record its verdict. Reopen Reliability if changes affect the workload, measurement, resources, or verifier execution. Human review: Integrity. Present the boundary audit and red-team results. Ask whether an invalid or shortcut implementation can receive a valid score. Verifier ownership requires an independent immutable workload copy, not secrecy of the visible workload. ## Stage 4: Finalize Summarize review decisions and limitations, remove construction artifacts from the release package, and run final checks. Confirm that the package contains all required inputs and no credentials, private evidence, or hidden answers. Ask the human to ACCEPT AND PUBLISH, REVISE, or REJECT. Publish the pinned image and verified task package only after acceptance; preserve evidence for rejected candidates.

D.3

S ELF -R EVIEW S KILL

The following listing presents the default review skill with performance-optimization-specific guidance omitted. “Target Agent” denotes the task-solving agent. Self-Review Skill # Self-Review Act as the macro-level controller of the Target Agent’s task-solving process. Review the trajectory to diagnose the task-solving process, then retain the current harness or select a focused-feedback or reusable-ability adaptation that improves subsequent Target progress within the remaining shared budget. ## Review the task-solving process Treat each review as one iteration in a continuing self-review loop. At the start of every review, examine the accumulated trajectory comprehensively before selecting an intervention. Use that broad diagnosis to choose a focused intervention that is supported by the evidence and likely to materially improve subsequent Target progress. Include only the feedback or reusable capabilities justified by the diagnosed problems in task execution and the evidence needed to evaluate their effect. Never be satisfied with the current optimization state while task budget remains. Treat accepted improvements as intermediate evidence and compare the remaining opportunities by their expected contribution.

45

Self-Review Skill (continued)

### Diagnose the task-solving process Treat the Target Agent’s framing, causal model, plans, explanations, and claims as hypotheses, not premises. Use recorded actions, observations, and measurements to identify all material unsupported links, distinguish evidence from inference, and identify plausible explanations not ruled out. Use the complete diagnosis to keep the intervention focused on the related gaps that most materially constrain progress. Treat claims that an experiment completed, improved performance, or excluded a direction as unverified until supported by the trajectory. Explain how each unsupported link affects task-execution cost, experiment selection, or progress. Critique the Target Agent’s choice of actions, not just candidate outcomes. Match each conclusion to the scope the experiments actually tested. Do not generalize beyond the tested workload, configuration, mechanism variant, or measurement, and do not ignore what the evidence establishes within that scope. Distinguish direct findings from broader hypotheses, and identify unsupported assumptions or omitted dimensions that constrained the search. After selecting the intervention focus, review two aspects of macro-level control as ways to advance it: 1. **Task-execution cost.** Examine repeated Target Agent work, which dependencies and evidence requirements have stabilized, and where intermediate model judgment remains valuable. Look for opportunities to reduce, fuse, automate, batch, or reuse work while preserving decision-relevant evidence. 2. **Budget allocation.** Evaluate exploration breadth and exploitation depth by comparing the causal support, remaining leverage, uncertainty, and expected contribution of validated mechanisms and untested hypotheses. ## Decide the intervention After completing the diagnosis, choose among retention, a focused-feedback adaptation, and a reusable-ability adaptation according to their expected contribution to target progress over the remaining budget. Retention means no new Target-facing guidance or other harness change. Use focused feedback for a task-specific concern, unsupported judgment, omission, or unresolved evidence. Identify the concrete observation, the gap, and why it matters. **Be careful about your own claims:** an evidence gap does not establish an alternative explanation or direction; do not turn preferences into concerns or next-step menus. State only the concern and its evidentiary basis; do not require or recommend what the Target Agent should do next. Activate a reusable ability when it directly supports the diagnosis and the trajectory supports repeated use. Also consider a harness adaptation when the Target Agent repeatedly makes the same consequential error and the evidence supports a general way to prevent it; otherwise use focused feedback. For a stable repeated action sequence, fuse or automate the mechanical work while preserving model judgment, decision-relevant evidence, meaningful failure boundaries, and a fallback to the unfused workflow. Validate the ability through its actual Target-facing path and use feedback to explain how to use it and its constraints.

46

Self-Review Skill (continued) Condition each intervention on the current stage of task execution, inferred from the trajectory rather than elapsed time: - **Early:** examine plausible competing hypotheses. - **Developing:** reduce demonstrated repeated work and balance exploration with supported exploitation. - **Saturated:** identify which tested hypothesis families have low remaining value, and flag any claim that the broader search is exhausted unless the trajectory supports that scope. Use the observed downstream effects of prior adaptations when deciding what the current stage needs.

47

Related documents

• Hyper Algorithm Design Agent: Evolving Learnable Optimizer from Zero#1129920
• RL-ABC: Reinforcement Learning for Accelerator Beamline Control#1143735
• MACHINE LEARNING AND MULTI-SCALE OPTIMIZATION FOR CONTROL AND ENERGY MANAGEMENT IN CONNECTED AND AUTOMATED VEHICLE PROPULSION SYSTEMS#1060959
• Physics-Inspired Optimizers beyond the Edge of Stability: Period-Two Orbits and Delayed Instability#1402709
• Learning to evolve by matching velocities#1536278