One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models

One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models Siqi Yang1,2∗ , Qianlan Yang1 , Yu-Xiong Wang1 , Saurabh Pujar2 , Martin Hirzel2 1

University of Illinois Urbana-Champaign 2 IBM

arXiv:2608.10178v1 [cs.SE] 10 Aug 2026

Abstract Self-evolving harnesses are closed-loop systems in which an agent inspects its own rollouts and edits its prompts, tools, and memory. They reliably improve coding agents in evaluations, but prior work reports aggregate gains rather than analyzing what the evolved artifacts encode. It therefore remains unclear whether they encode benchmark-specific adaptations, language-specific engineering knowledge, or compensation for limitations of the underlying model. We disentangle these factors by holding an evolution recipe fixed across a grid of eight programming languages (Multi-SWE-Bench) and three base models, and analyzing the resulting harnesses. The recipe routes every edit through a typed failure signal and records it as a falsifiable contract, making each modification attributable after evolution. Four findings emerge. (1) The loop improves held-out solve rates over both a minimal seed and the manually designed mini-SWE-agent scaffold in most cells, but with two null regions. (2) Gains compensate recoverable execution defects, where defect mass is near zero, and gain is near zero; which defect dominates is cell-specific. A harness closes the gap between what a policy can do and what it does. (3) Evolved harnesses share an abstract playbook across languages but instantiate it with almost disjoint language ecosystem machinery. (4) The shared core transfers and can be distilled into one universal harness, while an ecosystem margin resists both and requires native re-evolution. Together, these results recast the evolved harness as a legible compensation layer, shaped jointly by the language’s engineering demands and the model’s behavioral gaps, rather than an opaque benchmark-tuned scaffold. Code is available at Github

1

Introduction

Coding-agent progress on repository-level and long-horizon software engineering tasks has been driven by both stronger base models and better harnesses—the editable prompts, tools, memory, and workflow scaffolding through which agents act (Jimenez et al. 2024; Yang et al. 2024b; Wang et al. 2025). As base models advance and application surfaces multiply, harness engineering has emerged as an important optimization subject: the same base model can improve substantially depending on the surrounding harness (Lin et al. 2026b; Xia et al. 2025). However, the optimal harness is model- and task-specific, and transfer behavior of such harnesses is insufficiently understood. This motivates a growing ∗

Corresponding author: siqiyang@illinois.edu

line of work on self-evolving harnesses—outer-loop recipes in which an evolving agent inspects inner-loop rollouts and edits the harness autonomously (Hu, Lu, and Clune 2025; Xia et al. 2025; Zhang et al. 2026a; Lin et al. 2026b). Recent work has established that self-evolving harnesses work: they beat both human-designed baselines and promptonly self-improvement methods on standard evaluations (Lin et al. 2026b; Xia et al. 2025; Zhang et al. 2026b). However, a more fundamental question is unanswered: what does a selfevolved harness actually encode? A harness that lifts pass@1 could reflect programming-language knowledge, compensation for the base model’s cognitive limitations, or some combination of the two; it could also merely overfit the evaluated instances. Distinguishing the first two requires holding the evolution recipe fixed while varying the language and the model, and analyzing the resulting artifacts. Without this design, it remains unclear when a harness will transfer, require re-evolution, or which gains are portable. We therefore apply a diagnostic-routed framework across a language × model grid, linking every edit to a specific failure signal. We study programming language and model capability as the two axes within the code-fixing domain of MultiSWE-Bench (Zan et al. 2025). Coding-agent research remains centered on Python benchmarks, while multilingual evaluations reveal substantial performance variation across languages (Zan et al. 2025; Rashid et al. 2025; Baltaji et al. 2025). These gaps may partly reflect language-specific engineering demands that per-language harness evolution can address. Beyond asking whether per-language evolution addresses these language-specific demands, using programming language as the second axis offers analytical advantages: language ecosystems (including build, test, and dependency tooling) impose concrete, comparable constraints, and fixing the code-repair domain reduces task-level variation. This makes language and model effects easier to distinguish. We instantiate this design in a self-evolution framework built for mechanistic legibility, as shown in Fig. 1. An LLM analysis of failure trajectories emits typed signals that route each edit to a specific component and record it as a falsifiable contract, rather than relying on unstructured reflection or a scalar reward. We introduce no new search algorithm; instead, holding the recipe fixed across all 8 × 3 settings lets us attribute structural differences to the (language, model) setting rather than to the search procedure.

Our empirical results support four claims that together reposition self-evolving harnesses from opaque scaffolds to legible compensation layers. (1) The recipe works, and where it does not is explicable. Held fixed across all 8 × 3 cells, the loop improves held-out solve rates over the minimal seed and over the manually designed mini-SWE-agent scaffold (SWE-agent team 2025) in most cells. Two regions are flat, one along each axis: Python under every model, and GPT-5-mini in every language. Both are accounted for by (2) rather than left as exceptions. (2) Gains compensate recoverable execution defects The failures the loop repairs are process defects: editing a test file so that it collides with the hidden gold test patch, shipping source that does not build, submitting without running the target test. Which of these dominates is a property of the cell, not of the method—non-compiling changes account for 70% of C++ rollout defects but under 10% of Go’s, so no single rule is the mechanism, and a static scaffold usually hardcodes one guess for all cells. The two nulls are the two ways that bound can bind: Python’s base policies already follow the disciplines our detectors cover, and GPT-5-mini commits few such defects at all, leaving only localization and repair failures. (3) Evolved harnesses share a playbook but not its instantiation. Coding each harness into abstract concepts and concrete language ecosystem markers reveals moderate cross-language overlap in concepts, but near-none in implementation: harnesses rediscover the same general disciplines yet express them through different commands, test paths, and layout conventions. Overall, 20-40% of each harness is ecosystem-specific. (4) Reuse is limited by the ecosystem margin. The disciplinary core is genuinely portable: a harness transplanted unchanged onto a foreign language helps in 18 of 20 ordered pairs, and a distilled, language-agnostic harness matches native evolution on some ecosystems but not others. But it does not reach the native ceiling: transplants fall short in 14 of 20 pairs, and distillation recovers only 48-68% of native gains on Java, C++, and TypeScript. Three measurements of different kinds — textual, distillation-based, and transplantationbased — converge on the same range. Native re-evolution thus remains necessary for complex ecosystems. Together, these findings quantify the compensation view: the language determines which disciplines a task requires, model capability determines how many of them the base policy already supplies, and evolution installs the difference. Prior work showed that self-evolving harnesses outperform manual designs; we characterize what those gains consist of, when they transfer. This paper makes the following contributions: Framing. We identify programming language and model capability as two independently variable axes along which harness content and performance vary systematically. Instrumented method. We introduce a diagnostic-routed evolution framework that links every edit to a falsifiable failure signal. Holding the recipe fixed makes differences legible and attributable to language and model. A structural and mechanistic account of harness gains.

Across the grid, gains compensate recoverable execution defects, and their size tracks the defect mass a cell exhibits. Which defect binds shifts systematically with the language, so no single rule is the mechanism and a static scaffold must hard-code one guess for all cells. The account also covers its two null regions (Python and GPT-5-mini) and locates 20-40% of each harness in ecosystem-specific machinery. A transfer study. We quantify how far evolved harnesses port: the disciplinary core survives both transplantation onto foreign languages (positive in 18 of 20 ordered pairs) and distillation into a single universal harness, but how much survives depends sharply on the target, leaving an ecosystem margin that neither recovers. This yields practical guidance on when to reuse, distill, or re-evolve a harness.

2 2.1

Related Work

Coding Agent Benchmarks and Harnesses

Repository-level code repair has become a prominent testbed for coding agents because it combines reasoning, planning, tool use, and long-horizon interaction (Jimenez et al. 2024). Performance depends not only on the model but also on the harness: changing the agent–computer interface or scaffold while holding the model fixed can substantially affect solve rates (Yang et al. 2024b; Wang et al. 2025). Yet leaderboard systems often use different, sometimes unreleased, harnesses, obscuring whether gains come from the model or scaffold. Harness effects may also interact with model choice, but this coupling remains under-studied. The miniSWE-agent harness is small yet achieves competitive performance (SWE-agent team 2025). Multi-SWE-bench holds the code-repair domain fixed while spanning multiple programming languages, making language a useful analytical axis (Zan et al. 2025). Existing multilingual evaluations already adapt agents to new ecosystems: Both Multi-SWEbench and SWE-PolyBench ship per-language agent variants (Zan et al. 2025; Rashid et al. 2025). These efforts show that multilingual evaluation often involves language-specific harness changes, but treat those changes as engineering accommodations rather than subjects of systematic analysis.

2.2

Self-Evolving Harness Systems

Self-evolving harness systems use an LLM to propose harness edits, evaluate them on agent rollouts, and retain beneficial variants, but differ in what they evolve and how edits are selected. DGM evolves the agent repository through openended archive search (Zhang et al. 2026a); AHE exposes seven editable component types and binds each edit to a falsifiable prediction (Lin et al. 2026b); HarnessX organizes harnesses along nine composable dimensions (Chen et al. 2026c); HarnessFix diagnoses failures within a seven-layer harness taxonomy before applying scoped repairs (Chen et al. 2026b); and HGM uses descendant performance to guide search (Wang et al. 2026). Related methods evolve cumulative context or skill artifacts. ACE incrementally updates an evolving playbook (Zhang et al. 2026b), SkillOpt applies held-out-gated edits to a skill document (Yang et al. 2026), and MUSE-Autoskill creates and refines reusable skills from

𝐷 editable slots system prompt P

frozen LLM

workflow hooks W lessons memory M tools implementation T

observation

inner loop

harness H

verifier

rollout analyzer

rollouts

layered evidence

evolve agent 

evolution history

thought, action

tool call

diagnostically routed edit

Figure 1: Overview of harness self-evolution recipe.

experience (Lin et al. 2026a). Across these systems, an important design choice is how failures are translated into edits: methods use layer attribution, predicted task-level effects, rollout trajectory diagnosis, or held-out validation rather than relying only on unconstrained reflection. Prior work varies substantially in benchmarks, models, editable components, and search procedures, making cross-setting differences difficult to attribute. We instead hold both the evolution recipe and routing scheme fixed across a language × model grid to study how grid cells affect the resulting harnesses.

Externalized Tuning and Joint Adaptation

A related line optimizes textual and program-level artifacts. DSPy represents LM pipelines as declarative programs (Khattab et al. 2024); OPRO uses candidate–score histories to improve prompts (Yang et al. 2024a); TextGrad propagates textual feedback through compound systems (Yuksekgonul et al. 2025); AutoPDL uses feedback to decide whether to use an agent and how to configure it (Spiess et al. 2025); and GEPA combines reflection with evolutionary selection, outperforming GRPO in its reported evaluations with fewer rollouts (Agrawal et al. 2025). Other work adapts harnesses and model parameters jointly. SIA updates both the scaffold and model weights (Hebbar et al. 2026), while HarnessForge co-evolves harnesses with harness-conditioned policy adapters (Chen et al. 2026a). These methods primarily evaluate optimized artifacts through task performance.

3

This is the checkpoint returned after R rounds, rather than a L global optimum. A disjoint Dtest is reserved for final evaluation and never used to propose or select edits. In controlled comparisons, µ and the evolution protocol are fixed.

3.1 outer loop

2.3

bution incorporates verifier- and budget-side evidence, and typed routing uses the resulting diagnosis to prioritize where the harness changes. For a language ecosystem L, frozen task policy π, and outer-loop driver µ, the finite-round output is
(R) L HL,π,µ = EvolveR H0 , Devo ; π, µ . (1)

TRIAGE: Harness Self-Evolution

Fig. 1 presents TRIAGE (Typed Routing and Instrumented Attribution for Guided Evolution), the instrumented evolution loop we hold fixed across the grid. We design TRIAGE for legibility: typed routing and contract-level records leave every edit attributable to a specific failure signal after evolution, which enables the harness-content analyses of Sec. 4.4 to Sec. 4.6. We build on AHE’s observability-driven backbone (Lin et al. 2026b), retaining its frozen policy, explicit editable harness, layered evidence, and later-round validation. TRIAGE adds two mechanisms: instrumented attri-

Modular Harness Evolution

The harness exposes four versioned, file-level slots, H = (P, W, M, T ): the system prompt P , workflow hooks W , lessons memory M , and tool implementations T . These respectively encode persistent behavior, statedependent guidance, reusable cross-instance lessons, and executable capabilities. We implement them as a lightweight plugin layer on mini-SWE-agent (SWE-agent team 2025); the abstraction requires only inspectable control surfaces around a frozen policy. Skills, sub-agents, or multi-agent orchestration are out of scope. Every setting starts from the same minimal seed H0 : a short prompt, one shell tool, and empty hooks and memory. Memory is capacity-bounded and admits only rules marked as cross-instance reusable, so the cap pressures distillation rather than accumulation; detailed component constraints are given in Appendix A. L , producing At round r, the inner loop runs H (r) on Devo rollout trajectories and verifier outcomes. The rollout analyzer combines these artifacts to form layered evidence. Conditioned on this evidence, the current harness, and the evolution history, the evolve agent µ proposes evidence-backed edits and predicts their task-level improvements and regressions. The next round tests those predictions on the evolution split. We use one linear checkpoint sequence, retaining supported updates and rejecting or reverting regressive ones, without population search or branching. The fixed seed, interface, evidence schema, budget, and selection procedure make the returned harnesses comparable across settings.

3.2

Failure Attribution and Typed Routing

A trajectory records what the task agent attempted, while a scalar verifier outcome records whether the submission passed. Neither alone explains why it failed, particularly when the agent itself believes it succeeded. TRIAGE therefore instruments two complementary evidence channels. Verifier-side evidence includes the scored submission, resulting environment state, and stage-level execution record, including stages that did not run. For example, it can expose interference between a submitted patch and a held-out test patch that appears in the transcript only as an ordinary failure. Budget-side evidence includes termination reasons and resource use, distinguishing a ceiling-interrupted attempt from a completed but incorrect submission. The analyzer represents these records using a fixed, typed attribution schema. Its broad classes comprise verifier in-

teraction, verification, localization, incorrect repair, and efficiency; exact signals appear in Appendix A. These types organize observations but prescribe neither a semantic cause nor a remedy: the analyzer infers the cause from the complete evidence. Routing is advisory. The attribution reaches µ as a dossier—leading cause plus failing stages, repository state, budget trace, and competing signals—not a verdict; µ weighs it and owns the causal claim. Routing therefore prioritises the search rather than fencing it. Persistent guidance typically enters the prompt or memory, state-dependent intervention a hook, and executable support a tool. Because the diagnosis is advisory, a misattribution costs an iteration in the typical case: the resulting edit tends to fail its own prediction and be reverted. Following AHE’s decision-observability contract (Lin et al. 2026b), attribution is treated as a falsifiable hypothesis rather than an oracle. Later rollouts compare predicted task effects with observed outcomes and reject or revert unsupported changes. This validation is checkpoint-level. Detailed schemas, routes, selection criteria, and examples are deferred to Appendix A. Beyond our coding-agent instantiation, TRIAGE’s requirements—a frozen policy, an inspectable harness, structured execution records, and measurable resource use—are not specific to code repair, though we evaluate only that setting.

4

Experiments

We evaluate the self-evolving harness loop of Sec. 3 on Multi-SWE-Bench (Zan et al. 2025) across eight programming languages L ∈ {cpp, c, java, rust, typescript, javascript, go, python} and three rollout models π ∈ {Claude Haiku 4.5, GPT-5-mini, DeepSeek-V4-Flash}, forming an 8×3 grid of ⟨L, π⟩ cells. We evaluate four claims, corresponding to the four results of Sec. 1: (C1) Effectiveness. The loop yields held-out gains over both a minimal seed H0 and mini-SWE-agent across the majority of ⟨L, π⟩ cells, with two null regions—one per axis of the grid—that the mechanism of C2 accounts for (Sec. 4.2). (C2) Compensation Mechanism and Its Ceiling. Gains come from compensating recoverable execution defects: the gap between what a base policy π can do and what it does under a bare scaffold. The harness closes this gap without raising the capability ceiling (Sec. 4.3). (C3) Cell-Specificity. Because which defect dominates differs by cell, evolved harnesses converge on abstract concepts while diverging in concrete instantiation, with 20–40% of each harness bound to its target ecosystem (Sec. 4.4). (C4) Bounded Portability. The shared disciplinary core distills into one universal harness and transfers across languages, while an ecosystem margin limits both—triangulated by three independent measurements (Sec. 4.5–4.6).

4.1

Experimental Setup

Benchmark and Splits. We instantiate the 8 languagespecific subsets of Multi-SWE-Bench (Zan et al. 2025). For each language L, we randomly sample 20 instances for the L L evolution split Devo and hold out 50 instances Dtest for testing, L L yielding two disjoint sets Devo ∩ Dtest = ∅.

Models. For each model configuration, the task policy π and outer-loop driver µ are instantiated by the same underlying model. Cross-model comparisons therefore reflect their joint effect. We evaluate on nmodel = 3 models π spanning distinct capability profiles: two closed-weight models (Claude Haiku 4.5 and GPT-5-mini) and one open-weight model (DeepSeek-V4-Flash). Compared Harnesses. Under identical steps, cost, and rollout budgets, we compare three harnesses per cell: (i) Minimal Seed (H0 , lower bound): a stripped-down ReAct loop with a single bash tool, retaining only task descriptions and commit protocols without memory or hooks. (ii) Mini-SWE-Agent (SWE-agent team 2025): A widely adopted, manually designed static scaffold. (R)

(iii) Evolved Harness (HL,π,µ ): The output of our selfL evolution loop after R = 3 rounds using Devo . Metrics and Protocol. Unless otherwise stated, our primary metric is mean_solve@3: the mean solve rate across L k = 3 independent rollouts per held-out instance in Dtest .

4.2

Main Results: Held-Out Gains

Fig. 2 presents held-out results across all 8 × 3 cells. The (R) evolved harness HL,π,µ improves over the minimal seed H0 in most cells and matches or exceeds the manually designed mini-SWE-agent in 14 of 24 cells; on Claude Haiku 4.5, (R) the ordering (HL,π,µ > mini-SWE-agent > H0 ) is strict on every non-Python language. Two exceptions fall along the two axes of the grid. Along the language axis, Python gains ≈ 0 under all three models; along the model axis, GPT-5-mini gains little in any language. The difference lies within the rollout standard deviation (Appendix C), so we read them as null results rather than regressions. Both nulls have the mechanism we identify in Sec. 4.3.

4.3

Mechanism: What the Harness Compensates

What does the outer loop repair? Among failures to converted successes, many are model-side execution defects, not its understanding of the bug. We call a defect harness-recoverable when (i) the rollout would plausibly have succeeded had the agent behaved differently at a step it was already capable of executing, and (ii) the corrective can be stated as a rule the harness installs ex ante. Three such defects are frequent enough to detect automatically: editing a test file, which collides with the hidden gold test patch (‘git apply’ fails) and zeroes an otherwise-correct source fix (matched discipline: “never modify test files”); shipping non-compiling source, so the test executable is never produced (“verify it compiles before submitting”); and submitting without running the target test, so wrong or incomplete fixes go undetected (“run the failing test before submitting”). These are instances of the class, not a taxonomy of it: the loop also writes lower-frequency, cell-idiosyncratic rules such as repository layout conventions, per-ecosystem invocation forms, and diff-scope constraints. Our claim is accordingly directional: compensating recoverable execution

mean_solve (%)

Haiku 4.5

100

GPT-5-mini

DeepSeek-V4-Flash

75 50 25 0

p cp

t va rus ja

c

js

ts

go

py

p cp

c

t va rus ja

minimal seed

ts

js

go

mini-SWE-agent

py

p cp

c

t va rus ja

ts

js

go

py

TRIAGE (ours)

base-model (round-0) rollouts (%)

Figure 2: Main results. Performance comparison between minimal seed, mini-SWE-agent, and evolved harnesses. 100

ships non-compiling source → rule: “verify it compiles” edits a test file → rule: “never edit test files”

80

10

60 40

52 39

44

20 0

24

70

26

cpp

c

java

18

rust

47

50

js

go

7

ts

7

py

language

Figure 3: Composition of automatically detected execution defects by language. Bars show the relative mix of the two detectors that fire often: shipping non-compiling source vs. editing test files that collide with the hidden gold test patch. The mix shifts from build failures on compiled languages (C++, C) to test-file edits elsewhere. defects is a principal source of the gains we observe. We do not claim that measured defect mass upper-bounds the gain available in a cell; our detectors cover only part of the recoverable space. Which defect dominates is cell-specific. Among the defects our detectors cover, composition shifts sharply across languages (Fig. 3). On compiled languages the dominant mode is a non-compiling change—C++ 70%, C 44%—while test-file editing is rare (≤ 10%); on the scripting and systems targets the reverse holds. No single rule is therefore “the” mechanism. This is the operational argument for per-cell evolution: a harness must be told which defect its cell exhibits. Two null regions, consistent with this account. Both nulls were identified post hoc; we read them as consistent with the mechanism rather than as tests of it. They arrive by different routes, one along each axis. Python (language axis): for all three rollout models, Python’s detector-covered defect rate is the lowest of the eight languages (5.6%), so the effect tracks the language rather than any one model’s competence—the rules the loop would install are ones these policies already follow. Python is the one region where evolution buys nothing: the point estimate lies within the rollout

standard deviation, so we claim only the absence of benefit, not a regression. GPT-5-mini (model axis): our detectors fire seldom, and the loop reaches the same diagnosis independently—the memory.md it evolves contains no testfile rule at all, on languages where every other model’s harness installs one as its first entry. What remains are failures of localizing the defective code and synthesizing a correct fix, which no rule addresses. We do not view GPT-5-mini as a stronger model; in fact, under the same mini-swe-agent, GPT-5-mini is the weakest. This suggests that models possess qualitative properties that are independent of their quantitative capability level. Scope: compensation, not a raised ceiling. Within the failure modes we can detect, the gain is compensation rather than capability extension: we find no recovered instance in which the evolved harness enables a repair the base policy could not otherwise express. We do not claim this holds outside the detected classes. Two practical readings follow. First, a base policy’s measured defect profile is a cheap leading indicator of where harness engineering pays off—though it is an indicator, not an estimate of headroom, since the recoverable space is wider. Second, within the regime where a harness helps at all, an automatically evolved one outperforms a manually designed scaffold (Sec. 4.2), precisely because the binding defect differs per cell and a static scaffold must hard-code one guess for all of them.

4.4

Evolved Harnesses Are Cell-Specific

The main results ship a separate harness per (language, model) cell. Are these genuinely distinct artifacts, or one generic playbook copied around? We answer by inspecting their content. We parse each evolved harness into codeable units and tag every unit at two levels: the abstract concept it expresses (e.g. “localize from the test output”, “minimal diff”) and any concrete ecosystem marker it names (a build tool, a test-path glob); a unit is ecosystem-specific if it names such a marker. Across the evolved language cells, we then measure cross-language Jaccard overlap at each level, for the two rollout models with full coverage. We restrict the structural analysis to the 14 cells where the loop produces a held-out gain. The excluded cells cannot speak to what a working harness encodes (Sec. 4.3).

0.67

0.44

0.55

0.67

cpp

1.00

0.09

0.06

0.08

0.08

0.06

0.06

0.55

c

0.09

1.00

0.10

0.33

0.14

0.09

0.22

java

0.50

0.64

java

0.06

0.10

1.00

0.08

0.08

0.06

0.07

rust

0.67

0.71

0.62

1.00

0.43

0.40

0.55

rust

0.08

0.33

0.08

1.00

0.11

0.08

0.18

ts

0.44

0.43

0.57

0.43

1.00

0.33

0.36

ts

0.08

0.14

0.08

0.11

1.00

0.40

0.08

js

0.55

0.40

0.50

0.40

0.33

1.00

0.73

js

0.06

0.09

0.06

0.08

0.40

1.00

0.06

go

0.67

0.55

0.64

0.55

0.36

0.73

1.00

go

0.06

0.22

0.07

0.18

0.08

0.06

1.00

concept level

0.6

0.4

30

js

go

ts

ru st

c

a ja v

p

Haiku

40

40

31

29

29 24

25

DeepSeek

33 30

30

27

25

24

20

20

18

10

0 cpp

0.2

cp

ja v

cp

js

0.40

0.57

go

0.43

0.62

ts

0.71

1.00

ru st

0.86

0.86

c

1.00

0.60

a

0.50

0.8

c

java

rust

ts

js

go

language 0.0

ecosystem-marker level

Figure 4: Cross-language harness overlap for Claude Haiku 4.5 across the seven evolved languages. Left column: abstract-concept Jaccard—moderate overlap (mean 0.55), a shared generic playbook. Right column: concrete ecosystemmarker Jaccard—near-zero overlap (mean 0.12), the specific build/test/layout tokens each harness names barely co-occur across languages.

Figure 5: Ecosystem-specific fraction of each evolved harness, per language, for Haiku and DeepSeek. generic regime (ρ ≥ 1)

1.4

retention of native gain

0.60

Jaccard similarity

0.50

c

p

language

1.00

ecosystem-specific fraction (%)

1.0

cpp

1.2

ecosystem-heavy regime (ρ < 1) 1.13

1.08 matches native

1.0 0.8

0.71

0.68

0.65

0.6

0.48

0.45

0.4 0.2

Concepts partly converge; instantiations do not. Harnesses evolved for different languages overlap substantially in what they aim to enforce and almost not at all in how they enforce it. At the concept level, mean off-diagonal Jaccard is 0.55 for Haiku (Fig. 4) and 0.57 for DeepSeek (See Appendix C): a sizable shared core of general disciplines, though far from identical—each language also elicits concepts the others do not. At the ecosystem-marker level, the overlap falls by roughly a factor of four, to 0.12 and 0.14: the build commands, test-path globs, and header layouts each harness actually names barely co-occur. The concept-level overlap is high but not near-unity, for two reasons we do not attempt to separate: languages genuinely differ in which disciplines matter (compiled targets elicit build-verification concepts that interpreted ones never raise); and our concept tags come from an LLM rubric with imperfect granularity, which splits near-synonymous concepts and deflates overlap. We therefore rest the claim on the gap between the two levels rather than on the absolute value at either: an evolved harness is not a copy of a universal artifact but a partly shared playbook instantiated on the concrete surface of its ecosystem—the structural form of C2, and the means by which the uniform gains of C1 are delivered by individually specialized harnesses. Fig. 5 sizes the divergent part directly: every evolved harness is 20–40% ecosystem-specific (mean 0.26 Haiku, 0.29 DeepSeek), consistent across two rollout models. A fifth to two-fifths of what the loop writes is therefore unusable outside its target ecosystem.

4.5

Universal Distillation: Isolating the Portable Component

Sec. 4.4 found that harnesses share concepts but not instantiations. If that split is functional, stripping the instantiations should preserve the shared part and lose the rest. We test this directly: using the meta-model, we aggregate the evolved memory rules of all languages into ten generic guidelines. The resulting universal harness is memory-only and names

st ru

ts

p cp

a ja v

c

go

js

0.0

Figure 6: Universal harness vs. native evolution. Fractional retention ρ of each language’s native gain by a single language-agnostic, memory-only harness, across the seven evolved languages.

no build command, test path, or file convention. We deploy it unchanged on DeepSeek’s held-out set and report retention ρ = (u − m)/(n − m), the fraction of that language’s native gain it recovers, where m is the minimal-seed score, n is the natively-evolved score, and u is the transferred score. Two regimes. Fig. 6 separates the seven evolved languages—the same seven characterised textually in Sec. 4.4. On generic targets—Go (1.08), JavaScript (1.13)— the stripped harness is statistically indistinguishable from native: where the gain is purely disciplinary, per-language specialization adds nothing. On ecosystem-heavy targets— Java (0.68), C++ (0.65), TypeScript (0.48)—it recovers only half to two-thirds. The residual 32–52% is an ecosystem margin: gain that exists only when the harness names the target’s own build tools, test runners, and layout conventions. Its magnitude agrees with the textual measurement of Sec. 4.4, where 20–40% of each evolved harness is ecosystem-specific by content—the same quantity, measured once in text and once in held-out solve rate.

4.6

Cross-Language Transfer is Structurally Bounded

Distillation removes the ecosystem component; transplanting a foreign harness keeps one, just the wrong language’s. The two therefore bracket the native ceiling from opposite sides. We deploy the harness evolved natively on source A, unchanged, on the held-out set of target B, and report the same retention ρA→B ; the rollout model is DeepSeek-V4-Flash in every cell, so differences isolate the harness. We use the five languages spanning the two regimes just identified—generic

native

0.95

1.50

0.45

0.88

1.50

java

0.95

native

1.40

0.27

0.75

1.25

ts

1.00

0.48

native

0.18

0.75

1.00 0.75

js

1.32

0.90

-2.30

native

1.00

0.50 0.25 0.00

go

0.32

0.62

1.30

-0.09

native

cpp

java

ts

js

go

retention ρ (1.0 = matches native)

source harness (evolved on)

cpp

−0.25 −0.50

target language

Figure 7: Cross-language transfer retention ρA→B . Rows: the source harness, evolved natively on A. Columns: the target it is deployed on. The grey diagonal is the native harness.

(Go, JS) and ecosystem-heavy (C++, Java, TS)—omitting C (redundant with the other compiled targets), Rust (small native gain makes ρ unstable), and Python (native gain ≤ 0, so ρ is undefined). Most of the discipline travels. Transfer is positive in 18/20 off-diagonal cells, with mean retention 0.63 (0.79 excluding the single strongly negative cell): a harness carried over from an unrelated language recovers a substantial share of the target’s gain with no adaptation at all. This is the behavioural counterpart of the shared concept-level core, and it is what makes evolved harnesses reusable for bootstrapping a language the loop has never seen. But native remains a ceiling. Retention falls below 1 in 14/20 cells, and full recovery occurs in only 6/20. The shortfall is not confined to weak sources: the strongest source, C++ (row mean 0.95), reaches native on just one of its four targets. Roughly a third of what native evolution installs is thus supplied by none of the five sources we tested. Where full recovery does occur, it is structured, not random—it concentrates where the source already carries the discipline the target demands, most visibly for compiled-language sources deployed on TypeScript (C++ 1.50, Java 1.40, Go 1.30). Transfer completes when there is no language-specific gap left to bridge. Exporting and importing are different properties. The two marginals are nearly uncorrelated, as the mechanism of Sec. 4.3 predicts: a harness exports the discipline matched to its dominant defect, and a target is recoverable only if that is also the defect it exhibits. JavaScript makes the point twice over. As a source it exports well into typed targets (C++ 1.32, Go 1.00, Java 0.90); as a target it is the least recoverable language in the matrix (column mean 0.20), with no foreign harness recovering even half its native gain. The one wall: a discipline the ecosystem does not enforce for free. The single strongly negative cell is JS→TS (−2.30), and its asymmetry with JS→C++ (1.32) identifies

what transfer actually fails to carry. A JavaScript-evolved harness contains the language-general rule “run the failing test before submitting” but no compile-verification step, since interpreted JavaScript has nothing to compile. On C++ and Java, that omission is harmless: executing the target test entails building the project, so the generic rule silently supplies compile verification and the gap never opens—which is why compiled targets are the easiest to import into. On TypeScript, the entailment breaks. Test runners routinely transpile without type-checking, so a patch can run its tests green while tsc rejects it, and the grader type-checks. Verification must therefore be named explicitly in the harness or it does not happen at all, and a harness that never learned to name it ships patches that never build. The boundary a portable harness cannot cross is not interpreted-versus-typed as such, but the point at which a discipline stops being a free consequence of the ecosystem’s own execution chain—exactly the compensation role of Sec. 4.3, seen from its failure side. The two experiments agree. On ecosystem-heavy targets, the foreign specialized harness recovers strictly more than the stripped universal one (C++ 0.90 vs. 0.65; Java 0.74 vs. 0.68) yet still falls short of native, giving universal ≤ transplant < native; on generic targets the order reverses (Go 0.84 vs. 1.08, JS 0.20 vs. 1.13), the foreign ecosystem baggage being dead weight or worse. The total gain therefore decomposes into a large portable disciplinary component and a remainder that neither reuse strategy recovers (C4): externalized harness engineering is a genuine reusable asset, but not a substitute for native evolution.

5

Conclusion

We studied what self-evolved harnesses encode by applying a fixed, instrumented evolution recipe across eight programming languages and three base models. Typed failure signals and falsifiable edit contracts make the resulting harnesses traceable. The evolved harness improves over both a minimal seed and mini-SWE-agent in most settings. Gains come from correcting recoverable execution defects. Python and GPT-5-mini form two null regions, in which the defects our detectors cover are uncommon, and evolution has little left to install. Across settings, harnesses learn similar behavioral disciplines but express them through language-specific conventions. This shared core is portable but incomplete: a distilled language-agnostic harness matches native evolution on generic targets yet recovers part of the native gain on ecosystem-heavy ones. These results characterize the evolved harness as a compensation layer: it improves how a model applies its existing capabilities, with the required guidance shaped jointly by model behavior and language ecosystem. Under testing conditions, existing harnesses appear useful starting points, but cannot always replace language-specific evolution. Our evidence comes from repository-level repair under one evolution recipe with a single run per setting, so we read aggregate structure; whether the same decomposition into a portable disciplinary component and an ecosystembound remainder survives at longer horizons remains open.

References Agrawal, L. A.; Tan, S.; Soylu, D.; Ziems, N.; Khare, R.; Opsahl-Ong, K.; Singhvi, A.; Shandilya, H.; Ryan, M. J.; Jiang, M.; et al. 2025. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457. 3 Baltaji, R.; Pujar, S.; Mandel, L.; Hirzel, M.; Buratti, L.; and Varshney, L. 2025. Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study. Transactions on Machine Learning Research (TMLR). 1 Chen, M.; Lv, C.; Zhang, G.; Chang, H.; and Zhou, S. 2026a. HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems. arXiv:2606.01779. 3 Chen, M.; Wang, J.; Liu, Z.; Wang, Y.; Zheng, H.; and Wang, Q. 2026b. From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws. arXiv:2606.06324. 2, 18 Chen, T.; Lu, S.; Zhao, K.; Meng, W.; Teng, H.; Li, T.; Li, C.; Liu, X.; Liang, J.; Zhang, Z.; et al. 2026c. HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry. arXiv preprint arXiv:2606.14249. 2 Hebbar, P.; Manawat, Y.; Verboomen, S.; Ivanova, A.; Palanimalai, S.; Bhatia, K.; and Baskaran, V. 2026. SIA: Self Improving AI with Harness & Weight Updates. arXiv:2605.27276. 3 Hu, S.; Lu, C.; and Clune, J. 2025. Automated Design of Agentic Systems. In International Conference on Learning Representations (ICLR). 1 Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In International Conference on Learning Representations (ICLR). 1, 2 Khattab, O.; Singhvi, A.; Maheshwari, P.; Zhang, Z.; Santhanam, K.; A, S. V.; Haq, S.; Sharma, A.; Joshi, T. T.; Moazam, H.; Miller, H.; Zaharia, M.; and Potts, C. 2024. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. In International Conference on Learning Representations (ICLR). 3 Lin, H.; Li, P.; Song, J.; Jiang, F.; and Zhang, T. 2026a. MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation. arXiv:2605.27366. 3 Lin, J.; Liu, S.; Pan, C.; Lin, L.; Dou, S.; Xi, Z.; Huang, X.; Yan, H.; Han, Z.; Gui, T.; et al. 2026b. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850. 1, 2, 3, 4, 18 Rashid, M. S.; Bock, C.; Zhuang, Y.; Buccholz, A.; Esler, T.; Valentin, S.; Franceschi, L.; Wistuba, M.; Sivaprasad, P. T.; Kim, W. J.; Deoras, A.; Zappella, G.; and Callot, L. 2025. SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents. https://arxiv. org/abs/2504.08703. 1, 2 Spiess, C.; Vaziri, M.; Mandel, L.; and Hirzel, M. 2025. AutoPDL: Automatic Prompt Optimization for LLM Agents. In Conference on Automated Machine Learning (AutoML). 3

SWE-agent team. 2025. mini-SWE-agent: The Minimal AI Software Engineering Agent. https://github.com/SWEagent/mini-swe-agent. Software repository. 2, 3, 4 Wang, W.; Piękos, P.; Nanbo, L.; Laakom, F.; Chen, Y.; Ostaszewski, M.; Zhuge, M.; and Schmidhuber, J. 2026. Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine. In The Fourteenth International Conference on Learning Representations. 2 Wang, X.; Li, B.; Song, Y.; Xu, F. F.; Tang, X.; Zhuge, M.; Pan, J.; Song, Y.; Li, B.; Singh, J.; Tran, H. H.; Li, F.; Ma, R.; Zheng, M.; Qian, B.; Shao, Y.; Muennighoff, N.; Zhang, Y.; Hui, B.; Lin, J.; Brennan, R.; Peng, H.; Ji, H.; and Neubig, G. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In International Conference on Learning Representations (ICLR). 1, 2 Xia, C. S.; Wang, Z.; Yang, Y.; Wei, Y.; and Zhang, L. 2025. Live-SWE-agent: Can Software Engineering Agents SelfEvolve on the Fly? arXiv preprint arXiv:2511.13646. 1 Yang, C.; Wang, X.; Lu, Y.; Liu, H.; Le, Q. V.; Zhou, D.; and Chen, X. 2024a. Large Language Models as Optimizers. In International Conference on Learning Representations (ICLR). 3 Yang, J.; Jimenez, C. E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K.; and Press, O. 2024b. SWE-Agent: Agentcomputer Interfaces Enable Automated Software Engineering. In Conference on Neural Information Processing Systems (NeurIPS), 50528–50652. 1, 2 Yang, Y.; Gong, Z.; Huang, W.; Yang, Q.; Zhou, Z.; Huang, Z.; Li, Y.; Gao, X.; Dai, Q.; Liu, B.; et al. 2026. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv preprint arXiv:2605.23904. 2 Yuksekgonul, M.; Bianchi, F.; Boen, J.; Liu, S.; Lu, P.; Huang, Z.; Guestrin, C.; and Zou, J. 2025. Optimizing generative AI by backpropagating language model feedback. Nature, 639(8055): 609–616. 3 Zan, D.; Huang, Z.; Liu, W.; Chen, H.; Xin, S.; Zhang, L.; Liu, Q.; Li, A.; Chen, L.; Zhong, X.; Liu, S.; Xiao, Y.; Chen, L.; Zhang, Y.; Su, J.; Liu, T.; Long, R.; Ding, M.; and Xiang, L. 2025. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. In NeurIPS Datasets and Benchmarks Track. 1, 2, 4 Zhang, J.; Hu, S.; Lu, C.; Lange, R.; and Clune, J. 2026a. Darwin Gödel Machine: Open-Ended Evolution of SelfImproving Agents. In International Conference on Learning Representations (ICLR). 1, 2 Zhang, Q.; Hu, C.; Upasani, S.; Ma, B.; Hong, F.; Kamanuru, V.; Rainton, J.; Wu, C.; Ji, M.; Li, H.; Thakker, U.; Zou, J.; and Olukotun, K. 2026b. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. In International Conference on Learning Representations (ICLR). 1, 2

A

TRIAGE Implementation Details

This appendix specifies the parts of TRIAGE that Sec. 3 summarises: the harness slots and their admission constraints (Sec.A.1), the typed attribution schema (Sec. A.2), the routing table (Sec. A.3), the edit-contract and selection machinery (Sec. A.4), the meta-model roles and their inputs (Sec. A.5), and one complete round traced end to end (Sec. A.6).

A.1

Harness Slot Specifications and Constraints

The harness H = (P, W, M, T ) is a directory of versioned files. Every slot is admitted through a smoke test before it can enter a round: a proposed edit that fails validation is rejected and the round proceeds with the previous contents, so a malformed edit costs an iteration but cannot corrupt the incumbent. P prompt: System and per-instance templates (plus observation and format-error templates). Free-form text; unbounded in length. W workflow hooks: Python module exposing at most the two hook points before_query(agent, messages) and after_observation(agent, message, outputs); must import and execute without error. M memory: Bounded list of cross-instance lessons: at most 12 entries, each at most 400 characters, and any entry containing an instance-specific token (task identifier, container or repository path) is rejected. T tools: Declarative shell-tool records (name, description, in-context example, body). Each record is validated independently; invalid records are dropped and the remainder kept. Tools are upserted by name and rendered into the system prompt as a ## Tools section. Why memory is bounded. Memory and prompt text both reach the policy through context, so the distinction between them is not one of mechanism but of constraint. The cap is the mechanism: once it binds, a new lesson can be added only by merging or pruning existing ones, which forces distillation rather than accumulation, whereas an unbounded prompt can always absorb one more sentence. The instance-token filter enforces the other half of the contract: memory is replayed verbatim on unseen tasks, so it must not carry instancespecific content. These two constraints are what make the memory slot the portable component isolated in Sec. 4.5.

1. model_incompat — the rollout failed at the API layer (message ordering or fields the endpoint rejects, context overflow). This blocks the rollout entirely, so it outranks everything else. 2. budget_exceeded — the rollout terminated at the step, cost, or time ceiling. An observable budget failure, not an incorrect answer. 3. patch_broke_tests — the grader’s fix stage captured zero tests and the submitted patch touches a test path. Grading applies the held-out gold test patch on top of the agent’s diff, so editing the same test file makes that application fail and the target test never runs. This is invisible in the pass/fail boolean and mimics a clean incorrect fix, which is why it outranks the heuristics below. 4. unwinnable — the reference patch is very large (many files or bytes), i.e. the instance is out of scope for a single rollout. 5. patch_pollution — the scored patch carries build artifacts, scratch files, or frozen snapshots: a harnesspipeline self-injury rather than an agent error. 6. no_build — the command audit shows the project was never successfully built. 7. wrong_fix — default: the patch applied, the target test ran, and the behaviour is still wrong. The analyzer’s soft vocabulary is loop, format_violation, tool_error, gave_up, wrong_fix, crash, budget_exceeded, patch_pollution, patch_broke_tests, model_incompat, undetermined, novel, other. Alongside the bucket, each rollout carries a verification audit (whether the project was built, whether the target test was executed before submission) and a budget trace (steps, model calls, cost, termination reason). What the buckets actually fire on. Table 1 gives the distribution over all diagnosed rollouts in the grid. Two entries are worth noting. wrong_fix, the class we deliberately do not route, is the single largest at 38.7%: the loop leaves more than a third of failures untouched by construction. And model_incompat is vanishingly rare (1.0%), which is why no cell ever evolved a middleware. Bucket

A.2

Typed Attribution Schema: Signal Definitions

Each rollout receives two attributions. The deterministic bucket is computed from artifacts alone—no model is involved—and is the categorical signal the router and our analyses use. The issue type is a soft tag emitted by the analyzer for aggregation only; the analyzer’s authoritative output is a free-form root cause with artifact-anchored evidence, and it is instructed to prefer undetermined over a fabricated diagnosis and novel when the narrative is clear but no existing category fits. The deterministic bucket is assigned by the first matching rule in the following fixed priority order; the order encodes which failure blocks the others from being observable.

Rollouts

Share

wrong_fix no_build patch_broke_tests budget_exceeded unwinnable model_incompat patch_pollution other

1775 1514 606 525 99 44 9 10

38.7% 33.0% 13.2% 11.5% 2.2% 1.0% 0.2% 0.2%

Total

4582

100%

Table 1: Deterministic bucket distribution over all diagnosed rollouts in the grid.

A.3

Routing Table: Signal to Component

Routing is advisory: the attribution reaches the evolve agent as a dossier—dominant cause, failing stages, repository state, budget trace, and competing signals—rather than as a verdict, and the agent owns the causal claim. Table 2 gives the map it is shown. Dominant signal

Prioritised component

budget_exceeded

Concise memory (greps, no re-reads, minimal edits) + context trim. Raise step/cost as last resort. Format-error template. Observation template. Agent code. Verification/boundary discipline in template or hook.

format_violation loop, missing info Unpromptable behavior wrong_fix, patch_broke_tests, verification Missing guidance

Non-regression gate. The harness shipped to the held-out set is the best-scoring checkpoint, and it is floored at the seed: if no checkpoint matches or beats H0 on the evolution split, the loop returns H0 . The reported gain is therefore lowerbounded by zero by construction. This one-sided guarantee is a property of editing an inspectable artifact—a file can be reverted at file granularity—and is not available to weightspace updates.

A.5

Meta-Model Roles and Inputs

Three meta-model roles share one underlying model per configuration (the same model that serves as the task policy π).

Instance or system template.

Table 2: Advisory routing from dominant signal to harness component. Two constraints accompany the table. First, a soft-prompt rule: verification is expressed as encouragement, never as a hard gate that blocks submission, because a gate conditioned on a check the agent may be unable to satisfy (“do not submit until the hidden test passes”) produces non-terminating rollouts. Second, an anti-monoculture guard: if recent rounds edited only the instance template without a confirmed gain, the next round is required to edit a different component. At most two components are edited per round.

A.4

Rollback is deliberately not strict, since a single-sample dip would otherwise discard useful work: a round within a tolerance band of 0.12 is kept without counting against patience; the loop reverts to the incumbent only after 2 non-improving rounds or on a catastrophic drop of 0.25.

Edit Contracts, Prediction Format, and the Non-Regression Gate

Contract format. Every round writes a machine-readable change manifest with five fields: failure_evidence (the specific instances and artifacts the diagnosis rests on), root_cause, targeted_fix (the concrete edit), predicted_impact (an explicit list of instances the edit should fix, and any it may regress), and changed_fields. The prediction is the falsifiable part: it commits to task-level outcomes before they are observed. Validation. The next round re-runs the evolution split and writes a change evaluation against the manifest: predicted_fixes_confirmed, predicted_fixes_missed, anticipated_regressions, unattributed_regressions, and a verdict. Validation is checkpoint-level: it scores the round’s edit set as a whole, not each line independently. Selection. Let sr be the round-r score on the evolution split and ρr the fraction of instances solved by the incumbent Hbest that round r no longer solves. The selection score is s̃r = sr − λ ρr ,

λ = 2.0,

so a round that raises the aggregate while silently breaking previously-solved instances cannot become the incumbent.

Rollout analyzer. Runs once per failed rollout as a small tool-using agent under a read budget of 20 steps, with each read capped at 9,000 characters, so that the diagnosis is grounded in retrieved artifacts rather than a single long context dump. Its tools expose the trajectory (message index, ranged reads, regex search), the submitted patch, the raw test output, the per-target-test evaluation summary, and— critically—the grader’s own stage-level verdict (run/test/fix passed/failed/skipped counts and the broken-test-build flag). It emits the free-form root cause, the soft issue type, secondary patterns, and a self-reported confidence. Evolve agent. Receives the layered evidence (per-rollout diagnoses aggregated by dominant cause), the current harness contents, the evolution history with each past edit’s verdict, and the advisory routing table. It emits the change manifest of Sec. A.4. Seeing past verdicts is what prevents it from re-proposing an already-falsified edit. Distillation (Sec. 4.5 only). A separate pass that reads the evolved memory of all languages and returns a single bounded list of generic lessons, subject to the same admission constraints as any memory slot plus an explicit instruction to name no build command, test path, or file-layout convention. The result is the universal harness evaluated in Sec. 4.5.

A.6

A Worked Round: Evidence to Diagnosis to Contract to Validation

We trace the C++ × Claude Haiku 4.5 cell, whose evolutionsplit trajectory is 10.0 → 62.5 → 72.5 → 72.5 and whose held-out score moves from 12.9% to 87.1%. Round 0 — evidence. The seed harness solves 10%. The deterministic buckets place 16 of 18 failures in patch_broke_tests: the grader’s fix stage captured zero tests while the run stage had produced results. Diagnosis. Reading the artifacts, the analyzer reports that in these instances the agent’s source edit matches the reference fix, but the submission also appends cases to existing test files under test/ and tests/. Grading applies the gold test patch on top of that diff, the application conflicts,

no target test is ever executed, and an otherwise correct fix is scored zero. Note the shape of the error: the transcript ends with the agent believing it has verified its work. Contract. The evolve agent proposes a boundary rule in the instance template and a memory lesson naming C++ testpath conventions, and commits to a prediction: a named list of instances that should now pass. Changed fields: instance template and memory. Validation (round 2). Re-running the split confirms the direction—the score moves 10.0 → 62.5—and the change evaluation records which predicted instances actually flipped and which did not, with no unattributed regressions; the verdict is confirmed. Note that the contract is scored on its own terms: several predicted instances did not flip, and that is recorded rather than absorbed into the aggregate. Subsequent rounds. With test-file collisions suppressed, the dominant bucket shifts to build and verification failures, and round 2 adds a hook that builds and runs the specific failing test before submission (62.5 → 72.5). Round 3 proposes extending the boundary rule to approval-baseline files; it does not improve the split, and the best-checkpoint rule keeps round 2 as the shipped harness. The two mechanisms in this single cell—a boundary rule and a verification step— are the two whose relative frequency varies by language in Sec. 4.3, which is why no single one of them is “the” mechanism.

A.7

round is run, in two directions: those the edit should fix and those it may break. The following round scores the manifest against observed outcomes and writes a change evaluation recording predicted_fixes_confirmed, predicted_fixes_missed, anticipated_regressions, unattributed_regressions, and a verdict. A real contract. Below is the verbatim manifest from round 1 of the C++ × Haiku cell traced in Sec. A.6, abridged only in the instance lists. 1 2

{ "failure_evidence": "16/18 failures are patch_broke_tests: catchorg__Catch2__pr-2128, pr-2187, pr-2394, pr-2723, fmtlib__fmt__pr -1171, ... nlohmann__json__pr-545. In every case the agent’s source fix matches gold exactly but also edits test files (test/, tests/, test/src/), causing git apply of the gold test_patch to fail with a conflict -> broke_test_build=True -> zero tests graded -> automatic loss.", "root_cause": "The agent appends its own test cases to existing test files to ’demonstrate’ or ’verify’ its fix. The grading harness applies a gold test_patch on top of the agent’ s diff via ‘git apply‘. Because the agent already modified the same test files, git apply fails with a conflict, the test build is broken, and the fix-stage captures zero test results. The source fix is often correct -- it ’s never evaluated because the test file collision blocks everything.", "targeted_fix": "Add a prominent, specific instruction in instance_template: DO NOT modify test files (files in test/, tests/, or any *test*.cpp/cc files). The harness applies its own test patch on top of your changes; if you touch test files, a merge conflict will break the entire evaluation. Only modify source/implementation files. Also add memory lessons for durability across rollouts.", "predicted_impact": { "should_pass": ["catchorg__Catch2__pr-2128", " catchorg__Catch2__pr-2187", " catchorg__Catch2__pr-2394", " catchorg__Catch2__pr-2723", "fmtlib__fmt__pr -1171", /* ... 10 more ... */ " nlohmann__json__pr-545"], "at_risk": ["catchorg__Catch2__pr-1608", " fmtlib__fmt__pr-2292"] }, "changed_fields": ["instance_template", "memory "]

3

Edit-Contract Schema and a Filled Instance

The evolve agent’s entire output is a single JSON object; nothing else it writes is read. The schema is fixed and is the mechanism by which an edit becomes falsifiable: three fields record the reasoning, one field commits to observable consequences, and one records what was touched. failure_evidence str — the specific failing task(s) and trace evidence targeted root_cause str — why it failed, not just what failed targeted_fix str — what the edit changes, and why that addresses the cause predicted_impact {"should_pass": [task_ids], "at_risk": [task_ids]} edits object keyed by editable field; only fields actually changed

edits may name any of the versioned slots: system_template, instance_template, observation_template, format_error_template, memory, hooks, tools. Template fields are supplied as full replacement text rather than as patches, so an edit cannot silently corrupt a template it did not intend to touch; the result is re-rendered and must parse before adoption. The memory field is additionally constrained: the agent is shown the current memory and must return the complete updated list, bounded to roughly a dozen entries, so adding a lesson past the cap forces it to merge or prune rather than append. predicted_impact is the part that makes the contract falsifiable. It names instances by identifier, before the next

4

5 6

7 8 9 10

}

Three properties of this artifact are worth noting, because they are what the instrumentation is for. The diagnosis is anchored: it names sixteen instances and the exact grader signature (broke_test_build) rather than asserting a general tendency. The prediction is two-sided: the agent volunteers two instances it expects to harm, which is what allows a regression to be classified as anticipated rather than unattributed. And the causal claim is checkable against the

artifact, not against the rollout’s narration—the agent in those rollouts believed it had verified its fix. The following round’s evaluation of this contract recorded one confirmed fix among the named instances, four predicted fixes that did not materialise, no unattributed regressions, and a verdict of confirmed: the direction held (the split moved 10.0 → 62.5) even though most individual predictions did not. We report this rather than a cleanly-met prediction because it is the typical case, and because it shows the contract is scored on its own terms instead of being absorbed into the aggregate.

A.8

A Hook and a Prompt Delta

The coding rubric of Sec. B.5 treats a memory lesson, a hook, and a sentence of the prompt delta as equivalent codeable units. Sec. G.1 shows the memory slot; this section shows the other two, from the same C++ × Haiku harness, so that all three slot types can be inspected against the rubric. The verification hook (round 2). This is the edit that moves the evolution split 62.5 → 72.5 in Sec. A.6. It is reproduced verbatim. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

def before_query(agent, messages) -> str | None: """Nudge agent to run the failing test before submitting if it hasn’t done so.""" recent = messages[-6:] if len(messages) >= 6 else messages recent_text = ’ ’.join(...) # flatten the last few turns to text about_to_submit = any(kw in recent_text.lower () for kw in [ ’patch.txt’, ’complete_task_and_submit’, ’ git diff’, ’submit’, ’i have fixed’, ’the fix is complete’]) if not about_to_submit: return None has_run_test = any(kw in recent_text.lower() for kw in [ ’ctest’, ’make test’, ’passed’, ’failed’, ’error:’, ’ok]’, ’[ ok ]’, ’running’, ’test result’, ’assertions’, ’ catch2’]) if not has_run_test: return ("Before submitting: have you run the failing test to verify your fix? " "Try: find the build directory, run ‘make -j4 2>&1 | tail -20‘ to " "compile, then ‘ctest -R < relevant_test_name> -V 2>&1 | tail -40‘ to " "check. If you see approval/ snapshot test failures, you may also need " "to update baseline files.") return None

The hook illustrates why this slot exists as something distinct from the prompt. The rule it enforces—verify before submitting—is already stated in the prompt, but a prompt states it once, at the start, whereas the hook fires at the moment the behaviour is about to occur and only then. It is state-dependent in both directions: it triggers on evidence that the agent is preparing to submit, and it suppresses itself

if the transcript already shows a test run. It is also, as the routing constraint of Sec. A.3 requires, a nudge and not a gate: it returns a string that is appended to the context, and the agent remains free to submit anyway. Under the rubric, this single unit codes to the concept run the target test before submitting and carries three ecosystem markers—make -j4, ctest -R, catch2—so it is classified as ecosystem-specific. The prompt delta. The seed’s instance template is 325 characters; the evolved one is 1,730. The added text, abridged, is: ## CRITICAL RULES DO NOT modify test files (files under test/, tests/, or any file whose name contains test or spec). The grading system applies its own test patch. . . MINIMIZE your diff scope. Only touch files that are strictly necessary for the fix. The fewer files you modify, the less likely you are to conflict. . . Focus your changes on: • Source files (e.g. src/, include/, lib/) that implement the buggy behavior • Build files only if strictly necessary to add new source files • Do NOT add new test cases, even to verify your fix ## Verification Before submitting, try to run the failing test to confirm your fix works: 1. Find the failing test name from the issue or test output 2. Build and run it: e.g. cd build && make -j4 2>&1 | tail -20 && ctest -R <test_name> -V 2>&1 | tail -30 3. If a test still fails, investigate further before submitting 4. If the repo uses approval/snapshot tests (*.approved.txt, *.snap), check whether your behavior change requires updating those baseline files

Coded under the rubric, this delta contributes units for never modify test files, minimal diff, localize from the failing test, and run the target test before submitting—the same abstract concepts the Go harness reaches (Sec. G.2)— while its concrete tokens (make -j4, ctest -R, src/, include/, *.approved.txt, *.snap) share nothing with Go’s (go build ./..., go vet ./..., *_test.go, git add). This is the two-level structure of Sec. 4.4 visible inside a single slot rather than aggregated over harnesses. Note finally that the same discipline appears in all three slots of this one harness: as a rule in the prompt, as a lesson in memory, and as a hook that fires at submission time. The loop does not choose between them—it installs the general statement in text and the situational trigger in code.

B B.1

Experimental Setup and Measurement Instruments Benchmark Splits

We use the eight language-specific subsets of Multi-SWEBench. For each language L we draw a disjoint evolution L L L L split Devo and held-out split Dtest , Devo ∩ Dtest = ∅. The

evolution split is fixed at 20 instances per language, which is what each round is scored on. The held-out split is fixed at 50 instances per language The held-out split is never used to propose, score, or select an edit; it is evaluated once, at the end, on the promoted checkpoint. All transfer and distillation experiments (Sec. 4.5–4.6) L reuse the same Dtest , so a transplanted harness is scored on exactly the instances the native harness was scored on.

B.2

Model Versions and Budgets

Three rollout models span distinct capability profiles: Claude Haiku 4.5, GPT-5-mini, and DeepSeek-V4-Flash. Within a configuration the same underlying model serves as the task policy π and as the outer-loop driver µ, so crossmodel comparisons reflect their joint effect. Every rollout, in every arm, runs under identical limits: a step ceiling of 80 agent steps and a cost ceiling of $1.5 per rollout. These are hard ceilings enforced by the runtime— a rollout that reaches either terminates and is recorded as budget_exceeded (§A.2)—and they apply equally to the minimal seed, mini-SWE-agent, and the evolved harness, so no arm can buy performance with a larger budget. The evolution loop runs 3 rounds.

B.3

Baseline Harness Configurations

Minimal seed H0 . A stripped ReAct loop: a two-sentence system prompt stating that the agent can interact with a shell, an instance template carrying the task text plus the submission protocol (git diff > patch.txt, then an explicit completion sentinel), one bash tool, and empty hooks and memory files. It contains no guidance about tests, builds, verification, or diff scope—this is deliberate, since every such rule that appears in an evolved harness must then have been derived from observed rollouts rather than inherited. mini-SWE-agent. The manually designed scaffold, run unmodified at its released configuration, under the same step and cost ceilings, the same containers, and the same grader as the other two arms. It is the reference for “what a careful human scaffold achieves on this model”, and in Sec. 4.3 its margin over H0 is used as a coarse indicator of how much of a cell’s failure mass is addressable by harness design at all.

B.4

Execution-Defect Detectors: Definitions and Precision

Two automatic detectors are used in Sec. 4.3 and §D.1. Both are computed from artifacts only, on the minimal-seed rollouts of the held-out set. • Test-file edit. The submitted patch touches a test path. Detection is by a language-aware matcher covering test directories (test/, tests/, spec/, __tests__/), Python test_*.py and conftest.py, Go C/C++/Rust and *_test.go, *_test.* *test*.cpp, Java CamelCase FooTest.java, and JS/TS *.test.* and *.spec.*. • Build break. The grader’s own record shows the fix stage captured zero tests while the run stage produced results— i.e. applying the submission prevented the test executable from being produced at all.

Why we report these separately rather than as one “pollution” rate. The two signals are not interchangeable, and a single combined rate is not comparable across languages. The build break is a consequence signal: it fires when the test binary fails to exist, which a test-file edit can cause on a compiled language but which non-compiling source causes just as often. Its precision as a proxy for test-file editing therefore varies by cell—close to 1.0 on Haiku, where nearly every break is a test edit; between 0.00 and 1.00 across DeepSeek’s languages, where breaks on C and C++ are dominated by non-compiling source; and low on GPT-5-mini, which breaks builds without editing tests. Most importantly, on interpreted languages the build break is structurally unable to fire—Python’s rate is 0% under all three models— even though agents demonstrably do edit Python test files. We therefore use the language-agnostic test-edit matcher wherever a cross-language comparison is made, and report the build break as a separate mode rather than folding the two into one number. These two detectors cover part of the recoverable space, not all of it. The loop also installs cell-idiosyncratic rules (repository layout conventions, invocation forms, diff-scope constraints) that no detector of ours counts. Claims built on the detectors are accordingly directional: they establish that detector-covered defects account for a large share of what is recovered (§D.1), not that they exhaust it.

B.5

Concept and Ecosystem-Marker Coding Rubric

For the content analysis of Sec. 4.4, each evolved harness is parsed into codeable units: a memory lesson, a hook, or a semantically self-contained sentence of the prompt delta relative to H0 . Each unit receives two independent tags. • Abstract concept — what the unit tries to enforce, independent of language: for example do not modify test files, localize from the failing test rather than the issue prose, keep the diff minimal, verify the change builds, run the target test before submitting, stage newly created files. • Ecosystem marker — any concrete token naming the target’s machinery: a build or test command (make -j4, ctest -R, go build ./..., mvn test -Dtest=), a test-path glob (*_test.go, *test*.cpp), a language-specific file or layout convention (single_include/, *.approved.txt, conftest.py), or an idiom tied to the language (errors.Is). A unit is ecosystem-specific if it names at least one marker. Cross-language overlap is then measured at each level as the Jaccard similarity between the two languages’ tag sets, and the ecosystem-specific fraction of a harness is the share of its units carrying a marker. Two properties of this instrument bound what it can show. Concept tags come from an LLM rubric with imperfect granularity, which splits near-synonymous concepts and therefore deflates concept overlap; and marker extraction is lexical, so a discipline expressed without naming any concrete token is counted as generic even if it was learned from one ecosystem. Both biases push the two levels toward each other, so

the gap between concept overlap and marker overlap—which is what Sec. 4.4 rests on—is if anything understated. We do not interpret the absolute value at either level.

C C.1

Full Results

Held-Out mean_solve@3 per Cell

Table 3 gives every number behind Fig. 2: the minimal seed H0 , mini-SWE-agent, and the evolved harness, for all 8 × 3 cells. All values are held-out mean_solve@3. Model

Lang

H0

Haiku 4.5

C++ Java C JS TS Rust Go Python

12.9 2.9 30.0 31.5 34.6 11.1 13.0 75.0

61.9 51.4 75.0 68.5 65.4 31.5 25.9 72.6

87.1 82.9 81.7 75.9 86.5 72.2 37.0 73.2

+74.3 +80.0 +51.7 +44.4 +51.9 +61.1 +24.1 −1.8

C++ Java C JS TS Rust Go Python

22.9 22.9 28.3 33.3 46.2 24.1 22.2 67.9

27.6 16.2 46.7 37.0 36.5 31.5 27.8 65.5

27.1 28.6 36.7 31.5 44.2 20.4 22.2 67.9

+4.3 +5.7 +8.3 −1.8 −2.0 −3.7 ±0.0 ±0.0

C++ Java C JS DS-V4-Flash TS Rust Go Python

34.3 24.3 38.3 38.9 44.2 18.5 18.5 73.2

60.0 35.7 65.0 46.3 65.4 44.4 27.8 71.4

65.7 54.3 65.0 59.3 63.5 35.2 33.3 66.1

+31.4 +30.0 +26.7 +20.4 +19.3 +16.7 +14.8 −7.1

GPT-5-mini

mini-SWE TRIAGE ∆ vs H0

Table 3: Held-out mean_solve (%) per cell. Bold marks the best of the three arms.

Stability and Significance

Table 4 reports, per cell, the aggregate ± its standard devia√ tion across resamples (σ = s/ n), the paired improvement, and the paired statistic z = ∆/σ∆ . Stars denote |z| ≥ 1.64 (∗ ), ≥ 1.96 (∗∗ ), ≥ 2.58 (∗∗∗ ). The pattern is the one the mechanism predicts. Every nonPython cell is significant for the two models with substantial defect mass (Haiku all ∗∗∗ , z = 3.3–11.7; DeepSeek z = 1.7–4.6), and no GPT-5-mini cell reaches significance (|z| < 1.3)—which is why we read that region as a null rather than as a gain. Python is null or slightly negative under all three models.

C.3

Cross-Language Jaccard: DeepSeek-V4-Flash

Fig. 8 repeats the content analysis of Sec. 4.4 on the second model with full language coverage. The two-level structure replicates: mean off-diagonal concept overlap 0.57 (Haiku: 0.55) against ecosystem-marker overlap 0.14 (Haiku: 0.12),

TRIAGE

∆ (pp)

z

Claude Haiku 4.5 C++ 12.9 ± 5.2 Java 2.9 ± 2.0 C 30.0 ± 7.8 Go 13.0 ± 6.3 Rust 11.1 ± 6.2 TS 34.6 ± 8.7 JS 31.5 ± 8.1 Python 75.0 ± 7.5

87.1 ± 4.7 82.9 ± 6.1 81.7 ± 6.6 37.0 ± 8.7 72.2 ± 8.2 86.5 ± 4.4 75.9 ± 7.2 73.2 ± 7.9

+74.3 ± 7.2 +10.30∗∗∗ +80.0 ± 6.9 +11.66∗∗∗ +51.7 ± 8.8 +5.87∗∗∗ +24.1 ± 7.2 +3.32∗∗∗ +61.1 ± 9.0 +6.80∗∗∗ +51.9 ± 8.5 +6.08∗∗∗ +44.4 ± 8.6 +5.18∗∗∗ −1.8 ± 6.5 −0.27

GPT-5-mini C++ 22.9 ± 6.3 Java 22.9 ± 6.3 C 28.3 ± 6.6 Go 22.2 ± 7.7 Rust 24.1 ± 7.2 TS 46.2 ± 9.2 JS 33.3 ± 9.2 Python 67.9 ± 8.2

27.1 ± 6.9 28.6 ± 6.6 36.7 ± 7.6 22.2 ± 7.7 20.4 ± 6.1 44.2 ± 8.5 31.5 ± 8.5 67.9 ± 8.2

+4.3 ± 4.8 +5.7 ± 4.9 +8.3 ± 6.8 ±0.0 ± 4.6 −3.7 ± 7.5 −1.9 ± 7.1 −1.9 ± 5.0 ±0.0 ± 3.6

+0.90 +1.16 +1.22 0.00 −0.49 −0.27 −0.37 0.00

DeepSeek-V4-Flash C++ 34.3 ± 6.7 Java 24.3 ± 6.9 C 38.3 ± 7.5 Go 18.5 ± 7.1 Rust 18.5 ± 7.1 TS 44.2 ± 8.9 JS 38.9 ± 9.0 Python 73.2 ± 7.9

65.7 ± 7.0 54.3 ± 7.5 65.0 ± 7.6 33.3 ± 9.2 35.2 ± 7.9 63.5 ± 8.1 59.3 ± 9.3 66.1 ± 8.2

+31.4 ± 6.8 +30.0 ± 6.9 +26.7 ± 7.5 +14.8 ± 7.5 +16.7 ± 9.6 +19.2 ± 8.8 +20.4 ± 8.1 −7.1 ± 4.2

+4.61∗∗∗ +4.37∗∗∗ +3.57∗∗∗ +1.99∗∗ +1.73∗ +2.18∗∗ +2.51∗∗ −1.69∗

Table 4: Per-cell stability and paired significance. Values are mean ± resample standard deviation; ∆ is the paired improvement over H0 with its own σ∆ .

a factor of four. That the gap reproduces under a different rollout model is the reason we read it as a property of the language axis rather than of one model’s writing style.

C.4 C.2

H0

Lang

Ecosystem-Specific Fraction: Per-Cell Breakdown

Table 5 gives the per-cell values summarised in Fig. 5. Every evolved harness falls in the 20–40% band, with cell means of 26% (Haiku) and 29% (DeepSeek). The ordering across languages is not identical between the two models—C is the most ecosystem-specific cell for DeepSeek (40%) but mid-range for Haiku (25%)—so we read the band, not the per-language ranking, as the stable quantity. Model

C++

Java

C

Go

Rust

TS

JS

mean

Haiku DS

29 24

29 25

25 40

24 30

31 33

27 30

18 20

26 29

Table 5: Ecosystem-specific fraction (%) of each evolved harness.

0.50

cpp

1.00

0.20

0.07

0.27

0.06

0.07

0.14

c

0.71

1.00

0.88

0.57

0.50

0.44

0.55

c

0.20

1.00

0.17

0.20

0.12

0.17

0.14

java

0.62

0.88

1.00

0.50

0.60

0.56

0.64

java

0.07

0.17

1.00

0.12

0.09

0.11

0.10

rust

0.80

0.57

0.50

1.00

0.50

0.43

0.40

rust

0.27

0.20

0.12

1.00

0.10

0.12

0.25

ts

0.44

0.50

0.60

0.50

1.00

0.56

0.80

ts

0.06

0.12

0.09

0.10

1.00

0.20

0.18

js

0.38

0.44

0.56

0.43

0.56

1.00

0.60

js

0.07

0.17

0.11

0.12

0.20

1.00

0.10

go

0.50

0.55

0.64

0.40

0.80

0.60

1.00

go

0.14

0.14

0.10

0.25

0.18

0.10

1.00

0.6

0.4

concept level

js

0.0

go

ts

ru st

c

a ja v

p

0.2

cp

js

ecosystem-marker level

Figure 8: Cross-language harness overlap for DeepSeekV4-Flash. Left: abstract concepts (mean off-diagonal 0.57). Right: concrete ecosystem markers (mean off-diagonal 0.14).

D D.1

Relation to the ceiling claim. This decomposition is about recovered instances and therefore says nothing about instances that remained unsolved. It supports the directional claim (compensating recoverable execution defects is a principal source of the gains) and not the stronger one we explicitly decline in Sec. 4.3 (that detected defect mass bounds the gain available in a cell).

Mechanism Analyses D.2

Failure-Type Decomposition of Recovered Instances

Sec. 4.3 claims that the gains compensate recoverable execution defects. The most direct test is to take the instances the evolved harness actually recovered—those it solves in a strictly larger fraction of its k rollouts than the seed did— and ask what the seed’s failure looked like on exactly those instances. We classify each recovered instance by the seed’s behaviour using the two detectors of Sec. B.4. Across the grid, 256 held-out instances are recovered. Of these, 88% carried a detector-covered defect under the seed: 81% had a submitted patch touching a test path, and a further 7% broke the test build without touching a test file. The remaining 12% show neither signature and are recoveries we cannot attribute to a detected defect. Model

Two readings follow. For the two models with substantial gains, recovery is overwhelmingly the suppression of a selfinflicted defect: 95% (Haiku) and 91% (DeepSeek) of recovered instances were failing for a reason the harness can state as a rule. For GPT-5-mini the picture inverts—a majority (52%) of its few recoveries show neither signature—which is the same fact reported in Sec. 4.3 from the other direction: with little detector-covered defect mass to remove, what little it gains is not compensation of the kind we can attribute.

recovered

test-edit

build-break

neither

Haiku 4.5 DS-V4-Flash GPT-5-mini

142 81 33

94% 81% 21%

1% 10% 27%

4% 9% 52%

Pooled

256

81%

7%

12%

Gain Against Defect Mass

Fig. 10 plots, for each of the 24 cells, the seed’s detectorcovered defect rate against the net gain of the evolved harness. The association is strong (r = 0.81), and it is the relationship the compensation account predicts: cells whose base policy sabotages itself often are the cells where installing a rule pays. 100

evolved gain over minimal seed (pp)

ja v

cp

0.8

Jaccard similarity

0.38

go

0.44

ts

0.80

ru st

0.62

c

0.71

a

1.00

p

language

1.0

cpp

Haiku 4.5 GPT-5-mini

java

DeepSeek-V4-Flash gain = defect mass

80

cpp

60

40

20 cpp py

0

py py

0

20

40

rust

60

80

100

recoverable defect mass of the minimal seed (%)

Table 6: What the minimal seed was doing on the instances the evolved harness recovered.

edited a test file broke the test build Haiku 4.5 (n=142)

neither (localisation / repair)

94%

DeepSeek-V4-Flash (n=81)

81%

GPT-5-mini (n=33)

21% 0

10%

27% 20

9%

52% 40

60

80

100

recovered held-out instances (%)

Figure 9: Seed-side failure signature of recovered held-out instances, by model.

Figure 10: Seed defect mass against evolved gain, one point per ⟨L, π⟩ cell, coloured by model. The diagonal marks equality; the shaded region is where no cell falls. Three qualifications matter, and we state them rather than smoothing them over. (i) The relationship is not a bound we claim. No cell converts more of its defect mass than it has, but our detectors cover only part of the recoverable space (Sec. B.4), so the x-axis is a measured indicator and not an estimate of headroom. (ii) Conversion is not constant. The gain per unit of defect mass differs by model—about 0.58 for Haiku, 0.34 for DeepSeek, 0.12 for GPT-5-mini— so defect mass alone does not determine the gain; how much of it a policy can be induced to give up is a second factor. (iii) Within-model association varies. It is strong for Haiku (r = 0.85) and moderate for DeepSeek (r = 0.54), but absent for GPT-5-mini (r = 0.12), where the gain range is too compressed (−3.7 to +8.3 pp) for any relationship to be

visible. The pooled r = 0.81 is therefore carried by the two models that have gains at all.

Meta-Agent Driver

Held-Out

Minimal Seed H0

18.5

D.3

GPT-OSS-20B DeepSeek-V4-Flash Claude Sonnet 4.6

34.2 33.3 35.6

What the Loop Encodes: A Planted-Defect Probe

The analyses above are observational: they describe what the loop wrote when left to find its own targets. This probe asks a sharper question—can the loop discover and persist an operational fact that is not already part of general good practice? We construct a controlled defect that is observable and non-shortcut-able. A gate is injected into the environment such that the obvious action (running the failing test) fails with an explicit error naming the remedy, and the same gate is applied by the grader, so the failure is real rather than cosmetic. Crucially, no amount of generic discipline helps: localizing from the traceback, minimising the diff, or running the test are all blocked by the gate rather than assisted by it. The only remedy is the specific operational fact, which surfaces only after running once and reading the error, and which persists through a new non-test file so that no testpollution collision confounds the measurement. The result is negative. Across the probe conditions, the loop did not converge on the planted fact: held-out scores moved 78.3% → 76.1% in the control and 76.1% → 71.7% when the environment additionally surfaced the error text, both within noise and both non-positive. Instead of encoding the specific operational fact, the loop wrote generic verification discipline of the kind it writes everywhere else. We report this because it bounds the claim of the paper in a useful direction. The loop is effective at recall—recognising a recurring failure mode and installing the discipline that suppresses it—and we have no evidence that it performs discovery of novel, environment- specific operational knowledge. That is consistent with everything else we measure: the content is a playbook of general disciplines instantiated on a concrete ecosystem (Sec. 4.4), the portable part distills cleanly (Sec. 4.5), and what does not transfer is ecosystem machinery rather than insight.

E E.1

Ablations

Disentangling Meta-Agent Capability from Harness Gains

A potential concern is whether the performance gains come from external domain knowledge or superior reasoning injected by the strong Meta-Agent rather than the self-evolution mechanism itself. To strictly isolate this factor, we conduct a controlled ablation on a representative, medium-complexity cell: (Go, DeepSeek-V4-Flash). Holding the rollout base policy (DeepSeek-V4-Flash) fixed, we evaluate the harness evolution loop under three distinct Meta-Agent drivers spanning a wide spectrum of model capabilities: 1. Strong Meta-Agent: Claude Sonnet 4.6 (a strong model). 2. Medium-Weak Meta-Agent: DeepSeek-V4-Flash (a native, same-model self-evolution loop). 3. Ultra-Weak Meta-Agent: GPT-OSS-20B (an openweights 20B-parameter model).

Table 7: Meta-Agent driver ablation on the (Go, DeepSeek-V4-Flash) cell (k = 3 rollouts). Even an ultra-weak 20B open Meta-Agent recovers the performance gains delivered by the frontier model, confirming that harness evolution is governed by structural diagnostic routing rather than external knowledge injection.

As presented in Table 7, all three Meta-Agents consistently suppress self-inflicted test pollution. Qualitative inspection of the evolved artifacts confirms that even the 20B model successfully extracts the correct behavioral contracts. These results provide evidence that the primary driver of capability recovery is not the extra external knowledge or advanced reasoning of a superior Meta-Agent, but rather the diagnostic-routed feedback loop and falsifiable contract system. By converting raw execution traces into explicit, typed failure signals (Sec. 3), the task complexity of harness editing is drastically simplified—allowing even an open 20B model to reliably identify failure modes and write effective behavioral constraints. To be clear, we do not claim that the Meta-Agent’s capability is entirely irrelevant. A baseline reasoning threshold is inherently required for the Meta-Agent to follow instruction schemas, digest structural telemetry, and write syntactically sound harness edits. However, our findings demonstrate that once the Meta-Agent satisfies this minimal operational threshold (e.g., at the level of standard open 20B models), the structural design of the evolution framework—namely, diagnostic failure routing and falsifiable contract enforcement—becomes the primary governing factor.

E.2

Capability Tools Are Reachable Behaviourally

We observe that the loop essentially never writes tools. One reading is that tools are unnecessary; another is that our substrate makes them hard to reach. We test the distinction by supplying the tool ourselves. We take a task family where a custom capability is plausibly useful—detecting duplicated code regions during repair—implement find_duplicates as a first-class tool, and pre-install it in the harness so the loop begins with the capability rather than having to invent it. We then evolve with and without the tool present, holding everything else fixed. The tool is net-negative: supplying the capability up front does not merely fail to help, it costs 12.9 pp (Haiku & C++) relative to evolving without it. The mechanism is visible in the traces—the tool competes for steps and attention with the behavioural disciplines that actually carry the gain, and the loop spends rounds integrating it instead of installing rules. We draw a narrow conclusion: within this substrate

Arm

seed

evolved

∆

Haiku, no tool Haiku, tool pre-installed

11.1 11.1

22.2 9.3

+11.1 −1.8

Table 8: Held-out mean_solve (%) with and without a preinstalled capability tool.

and budget, the absence of evolved tools reflects their low marginal value against behavioural edits, not an inability of the loop to reach them.

F

Qualitative Evolution Trace

The trace below plots the evolution-split score by round, annotated with what that round changed, the verdict the following round assigned to it, and the concrete machinery the harness named. Together they illustrate the claim of Sec. 4.3 that which defect binds is a property of the cell: the same loop writes a boundary rule where test pollution dominates and a verification step where it does not. 100 R2 + hooks +10 [confirmed] verify via C++ build/test before submitting: cd build && make -j4 → ctest -R <test_name> -V

held-out evolved 87.1%

train mean_solve@2 (%)

80

60 R1 prompt + memory +52.5 [confirmed] “never edit test files” — C++ globs: test/ tests/ *test*.cpp *test*.cc *spec*.cpp → kills test pollution [compensation]

40

R3 + hooks +0 [regressed → discarded] Catch2 approval baselines *.approved.txt / *.snap over-edited → new pollution; best-checkpoint keeps R2 (shipped harness: 87.1% held-out)

seed (R0): 10% agent appends its own cases to a *test*.cpp file → grader git-applies the gold test patch on top → conflict, fix=[0,0,0] (self-inflicted)

20

held-out seed 12.9%

0

R0

R1

R2 evolution round

R3

Figure 11: C++ × Haiku. Round 1 installs the test-file boundary and moves the split 10.0 → 62.5; round 2 adds buildand-run-the-target-test verification; round 3 over-extends the boundary to approval baselines, does not improve, and is discarded by the best-checkpoint rule. Held-out: 12.9 → 87.1.

G

Example Evolved Harnesses

We reproduce two evolved harnesses in full, chosen to make the Sec. 4.4 claim inspectable: they come from the same rollout model and differ only in language, so any divergence is attributable to the ecosystem. Both are dominated by the same abstract concepts—do not touch tests, minimise the diff, localize from the failing test, verify before submitting— while sharing almost no concrete token.

G.1

test build, and scores your correct source fix as zero. - MINIMIZE your diff scope: only touch files strictly necessary for the fix. The grading harness also modifies some source files via its test patch -- touching extra files increases the risk of a merge conflict. - Run the failing test before submitting. Build first (e.g. make -j4), then run the specific failing test (e.g. ctest -R <test_name> -V). - Localize from the failing test or traceback first; do not guess from issue prose alone. - When a repo has approval/snapshot baseline files (*.approved.txt, *.snap, *.baseline), changing behavior may require updating those baselines too. - When fixing a bug that involves control flow, trace the full call graph to understand when each function runs; an existing call may need to be removed before adding new ones. - When a PR adds a new feature, cover the full scope of the PR, not just one aspect. - After making a fix, verify it compiles: cd <build_dir> && make -j4. - If conditional output behavior is involved (e.g. an ensure_ascii flag), handle all relevant branches, not just the default case.

C++ plus Haiku (memory slot, full text)

- NEVER modify test files (test/, tests/, *test*.cpp, *test*.cc, *spec*.cpp). The grading harness applies its own test patch on top of your diff -- editing test files causes a merge conflict, breaks the

G.2

Go × Haiku (memory slot, full text)

- NEVER modify *_test.go files or test fixture files (test/fixtures/*). The grading harness applies its own gold test patch on top of your diff -if you edited those files, git apply will conflict and your fix will never be graded. - If your fix requires creating NEW files, run git add <newfile> before generating the patch; plain git diff only shows already-tracked files. - Localize from the failing test name and error trace, not just the issue prose. - Make the minimal correct change; avoid refactoring surrounding code unnecessarily. - Use targeted reads: grep for the relevant symbol first, then read only the pertinent lines. Avoid cat-ing whole large files. - Prefer errors.Is() over != for error comparisons in Go; use the exact output format strings expected by existing tests.

- When adding a feature, follow an existing parallel function/pattern exactly -- naming, signature, error-handling style. - After making a change, verify it compiles: go build ./... or go vet ./.... - Submit with git add -A && git diff HEAD > patch.txt, then the completion sentinel. This captures both modified and new files.

Read side by side, the two harnesses make the twolevel structure concrete. The abstract concepts recur almost one for one. The instantiations share nothing: make -j4 and ctest -R against go build ./... and go vet ./...; *test*.cpp against *_test.go; C++ approval baselines against Go’s untracked-file staging problem; and one language-idiom rule (errors.Is) with no C++ counterpart. Note also that the Go harness spends a lesson on token efficiency (targeted reads) that C++ does not—Go’s cell hits the budget ceiling more often—which is the efficiency channel of Sec. A.2 appearing in the artifact.

G.3

The Distilled Universal Harness (ten guidelines)

This is the artifact evaluated in Sec. 4.5. It is memory-only and, by construction, names no build command, test path, or file-layout convention. 1. Never modify test files; the grading harness applies its own test patch on top of your diff, and any conflict causes a correct source fix to score zero. 2. Make the minimal diff: touch only the source files strictly necessary for the fix. 3. Localize from the failing test output and stack trace first, not from the issue prose alone. 4. Before submitting, verify your fix compiles or typechecks by running the language’s build step; an uncompiled fix is unverified. 5. Run the specific failing test after applying your fix to confirm it passes before submitting. 6. If you create new files, explicitly stage them before generating your patch; untracked files are invisible to a plain diff. 7. Never rename or move existing files; the grader’s patch targets original paths. 8. When an issue covers multiple behaviors, implement all required changes—partial fixes fail the remaining cases. 9. When fixing control-flow or state-management bugs, trace the full call graph to understand ordering; adding a call may require removing or guarding an existing one. 10. Before finalizing, re-read the failing test assertions to confirm the implementation matches exactly what the test expects.

Comparing this list with the two harnesses above shows what distillation keeps and what it drops. The concepts survive nearly intact. What is removed is every concrete token— and that removal is what Fig. 6 prices: on generic targets it costs nothing, and on ecosystem-heavy ones it costs 32–52% of the native gain.

H

Limitations

Task scope, and the parts of the design it leaves untested. Our evidence comes entirely from Multi-SWE-Bench-style tasks: one well-specified repair goal per instance, resolved in a single episode and verified by a fixed test suite. This regime does not exercise multi-goal or long-horizon behaviour, and two components of the evolvable space are correspondingly under-exercised. Persistent memory is tested only as distilled cross-instance discipline, never as within-episode state carried across many subgoals; sub-agent decomposition has little to do in tasks whose plan is typically a handful of steps. The loop is not specialized to this format—nothing in diagnostic routing or the contract system assumes single-goal tasks—but we do not demonstrate that it transfers to longer horizons, and we expect the balance between behavioural and structural edits to shift if it does. Dependence on the grading protocol. What counts as a recoverable defect (Sec. 4.3) is partly defined by how the benchmark scores. Editing a test file is fatal here because the grader applies a hidden gold test patch on top of the agent’s diff; in a deployment without that convention the same edit would be merely questionable, not automatically zero. Two of the three measured defects—shipping non-building source, submitting without running the target test—are protocol-independent engineering failures, but the magnitude of the first is specific to SWE-Bench-style grading. We therefore read the absolute gain as partly reflecting what the harness learns about the evaluation protocol, and the comparison across harnesses, which holds the protocol fixed, as the reliable quantity. No head-to-head comparison with related harnessoptimization methods. We do not claim novelty for the general idea of automatically editing an agent’s scaffold; our design shares its core with concurrent work (Lin et al. 2026b; Chen et al. 2026b), and our contribution is a lightweight, plug-in instantiation together with the analysis of when and why such optimization pays off. We nonetheless do not run those methods as baselines: each requires its own evolution budget per cell, and reproducing them across a 8 × 3 grid was outside our compute budget. Two further comparisons are missing for the same reason: stronger manually designed scaffolds than mini-SWE-agent, and a compute-matched inference-time baseline that spends the evolution budget on additional rollouts instead. The last is the most informative gap—we show the harness beats a fixed scaffold at equal inference cost, not that harness evolution is the best use of a fixed total budget. Limited component-level attribution. We isolate one design axis—the capability of the meta-agent driving the loop (Sec. E.1)—and find that gains survive an ultra-weak driver, which rules out knowledge injection from a frontier model as the source of the improvement. We do not ablate the remaining elements individually: diagnostic routing, the falsifiablecontract format, the non-regression gate, and two-axis editing are evaluated only as a bundle. The evidence therefore supports the claim that this bundle works and that its benefit is not attributable to the meta-agent’s strength, but it does not

apportion credit among the bundle’s parts. Statistical power and search budget. Cost constraints bound the protocol in four ways. Splits are small (20 evolution / 50 held-out instances per language), so per-cell numbers are individually noisy and we interpret aggregate structure— marginals, sign patterns, regime boundaries—rather than single values. Evolution runs once per cell, so we report rollout variance but not variance over the loop’s own stochasticity; the cell-specificity of Sec. 4.4 is consequently a lower bound on agreement, since run-to-run variation is folded into the cross-language differences. The loop runs three rounds of greedy hill-climbing rather than a population-based search, so the harnesses we report are reachable improvements, not optima. Finally, our defect detectors and concept/ecosystem rubric are automated instruments with imperfect precision; we rely on them for relative composition across cells rather than for absolute defect rates.

I I.1

Computational Resources and Cost

Scale of the Experiments

The unit of compute is a rollout: one agent episode on one instance, bounded at 80 steps and $1.5 (Sec. B.2), followed by containerised grading with the benchmark’s own harness. Table 9 accounts for every rollout executed for this paper, recovered from the per-rollout records rather than estimated. In total the study executed roughly 15k agent episodes comprising 747k agent steps and a comparable number of model calls, at a mean of 41.3 steps per rollout. Within the main grid the split between the two phases is 4,233 rollouts spent inside the evolution loop and 2,820 on held-out evaluation, i.e. evaluation is roughly as expensive as evolution—a consequence of scoring three arms at k = 3 on every held-out instance of every cell. The rollout budget is dominated by the grid rather than by any single analysis: the 8 × 3 main grid plus its miniSWE-agent reference accounts for 57.5% of all rollouts. The transfer study is the second largest item because it is quadratic in languages—each ordered pair requires a full held-out evaluation of the target—which is the practical reason Sec. 4.6 reports a 5 × 5 subset rather than the full matrix in the main text.

I.2

Cost Accounting

Costs decompose into two channels. Rollout cost is the frozen task policy π acting on instances; meta-model cost is the outer loop—the rollout analyzer reading trajectories and the evolve agent writing edits. Metered meta-model spend across all experiment logs is $256, against $3,667 of metered rollout spend, so the outer loop is roughly 5% of the total. This ratio is the practical argument for the approach: the expensive part of self-evolution is not thinking about the harness; it is running the agent often enough to tell whether an edit helped. Cost per unit of result. At the observed mean of 41.3 steps per rollout, one evolved harness—three evolution rounds on 20 instances plus a k = 3 held-out evaluation—costs on the order of a few hundred rollouts, or single-digit dollars of meta-model spend per cell. This is the sense in which harness

evolution is cheap relative to weight-space adaptation: no gradient step is taken, the policy is never loaded for training, and the artifact produced is a few kilobytes of text.

I.3

Infrastructure

All experiments ran on a single machine: 64 CPU cores and 251 GB of RAM, with no GPU, since every model is accessed as a remote API and the local work is container orchestration rather than inference. Each rollout executes inside the benchmark’s own per-repository Docker image, and grading runs in the same image, so the environment a patch is written in is the environment it is scored in.

Component

Rollouts

Agent steps

Metered cost

Main grid, 8 × 3 cells (§I.1) — evolution loop — held-out evaluation Cross-language transfer (Sec. 4.6) mini-SWE-agent baseline Universal distillation (Sec. 4.5) Planted-defect probe (§D.3) Tool ablation (§E.2)

7,053 4,233 2,820 4,792 1,606 578 701 324

330,060

$1,415

237,964 72,226 31,649 50,571 14,718

$1,088 $357 $615 — $192

Total

15,054

737,188

$3,667

Table 9: Compute accounting for all experiments reported in this paper.

Related documents

• Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer#1142973
• On the Evolving Biology of Language#101589
• Language Evolution#470287
• Training Large Language Models for Self-Explanation Faithfulness#394434
• Métaphore et comparaison, ellipses de l’analogie : convergences en linguistique et en traitement automatique du langage naturel#26285