What Do Evolutionary Coding Agents Evolve?
Nico Pelleriti1,2 Sree Harsha Nelaturu1 Zhanke Zhou3 Zongze Li3 Max Zimmer1 Bo Han3,4 Sebastian Pokutta1,2 1
arXiv:2605.20086v1 [cs.NE] 19 May 2026
3
Zuse Institute Berlin 2 Technical University of Berlin Hong Kong Baptist University 4 RIKEN Center for Advanced Intelligence Project
Abstract Recent work pairs LLMs with evolutionary search to iteratively generate, modify, and select code using task-specific feedback. These systems have produced strong results in mathematical discovery and algorithm design, yet a fundamental question remains: what do they actually evolve? Progress is typically summarized by the best score a run reaches under a task-specific evaluator, but that score can reflect several different mechanisms: new algorithmic structure, re-tuning an existing strategy, recombining ideas already in the model’s internal knowledge, or overfitting to the evaluator. Distinguishing these mechanisms requires inspecting the search process itself, not only its final outcome. We introduce EvoTrace, a dataset of evolutionary coding traces spanning four evolutionary frameworks, reasoning and non-reasoning models, and 16 tasks across mathematics and algorithm design. To analyze these traces, we develop EvoReplay, a replay-based methodology that reconstructs the local search states behind high-scoring solutions and tests controlled interventions, including adjusting constants, removing program components and substituting models or prompting contexts. We annotate every code edit in EvoTrace with one of nine recurring edit types using an LLM-as-judge pipeline validated against blind human re-annotation. Across EvoTrace, most score gains come from a small subset of these edit types. We further find a deterministic cycling pattern: about 30% of code lines added during search are byte-identical re-introductions of previously-deleted lines, present throughout nearly every run. These results show that benchmark gains in evolutionary coding agents can arise from qualitatively different mechanisms, only some of which correspond to new algorithmic structure. EvoTrace enables more diagnostic evaluation of evolutionary coding agents beyond final benchmark scores.
1
Introduction
Large Language Model (LLM)-driven evolutionary code search has rapidly emerged as a promising paradigm for automated scientific and engineering discovery. In this setting, LLMs propose program mutations within search loops that are guided by executable feedback [1–7]. This paradigm has produced strong results across mathematical construction, systems optimization, algorithm design, and GPU kernel engineering, including improved bounds for combinatorial problems, better packing constructions, compiler and scheduling heuristics [1–3, 8–11]. In this study, we define an evolutionary coding agent as a system with a task specification and executable evaluator, a population or archive of candidate programs, one or more LLMs that generate code mutations, recombinations, or refinements, and a search procedure that selects which programs and contexts feed future generations. A search trace is the full record produced by such a system: generated programs, scores, execution feedback, parent-child relations, prompts, model choices, and intermediate artifacts. Despite rapid interest and empirical progress, the internal dynamics of evolutionary coding agents remain poorly understood. Existing work often reports final best scores, aggregate success rates, Preprint.
Bug fix
External dependency
+ degeneracy guard
+ scipy.spatial.ConvexHull
14-gon → 2 concentric 7-gons
trial = selected + [cand_idx]
import numpy as np
- # regular 14-gon
cur_min = min_area(lattice[trial])
+ from scipy.spatial import ConvexHull
+ if cur_min <= 1e-12: +
continue if cur_min > best_min: best_min = cur_min
...
Architectural change
- angles = np.arange(14) * 2*pi/14 - pts = R * stack((cos a, sin a))
- return min_area(pts) / hull_const
+ # two concentric heptagons
+ hull = ConvexHull(pts)
+ outer = R * heptagon(angles_7)
+ return min_area(pts) / hull.volume
+ for f in np.arange(.20,.91,.01): +
inner = f*R*heptagon(angles_7+pi/7)
+
pts = vstack((outer, inner))
Composition
Local refinement
hill-climb + SA acceptance
unit radius → unit area
−48 lines, delegate to baseline
+ start_temp = max(cur*0.5, 1e-8)
angles = linspace(0, 2*pi, n)
- cols = ceil(sqrt(n))
points = stack((cos(a), sin(a)))
- radius = min(w, h) / (2*cols)
for step in range(n_iters): +
temp = start_temp*(1 - step/n_iters)
+ # scale to unit hull area
- centers, radii = [], []
if cand > cur:
+ hull_area = (n/2)*sin(2*pi/n)
- # ... 40+ lines of placement
+ points *= sqrt(1.0 / hull_area)
- return centers, radii, sum_radii
cur = cand +
Pruning
else:
return points
+ return baseline_packing(instance)
Refactor
Efficiency
Hyperparameter tuning
extract helper function
hoist combinations() out of loop
single literal change
- def heilbronn_convex14():
+ COMBOS = np.array(combinations(N, 3))
+
p = exp((cand - cur)/temp)
+
if rng() < p: cur = cand
-
# 80-line monolithic body
-
...
+ def _heptagon_layout():
-
idx = np.array(combinations(N,3))
+
... # extracted
-
p1,p2,p3 = pts[idx[:,0]], ...
+ def heilbronn_convex14():
+
p1,p2,p3 = pts[COMBOS[:,0]], ...
+
// solver budget - double time_limit = 1.85;
def min_area(pts):
+ double time_limit = 1.90; solver.set_time_limit(time_limit);
return _heptagon_layout()
Figure 1: A taxonomy of edits performed by evolutionary coding agents. Each panel shows a representative parent–child diff (added lines in green, deleted lines in red) drawn from EvoTrace runs and labeled with one of nine recurring categories: Bug fix, External dependency, Architectural change, Composition, Local refinement, Pruning, Refactor, Efficiency, and Hyperparameter tuning. The categories range from minimal numeric edits (a single literal change) to structural rewrites (replacing a 14-gon with two concentric heptagons), and they form the basis of the LLM-as-judge edit annotation used throughout the paper. Edits are typically multi-label; we examine prevalence and per-edit utility in §5.1.
or a handful of illustrative trajectories, but such endpoints obscure the pathways and mechanisms by which improvements arise. While evolutionary coding agents sometimes demonstrate clear advantages over baselines such as independent sampling, greedy refinement, or beam-style search, these gains are highly sensitive to choices in task design, initialization, evaluator specification, or model configuration. There is consequently no clear consensus on what these systems are actually evolving: whether progress comes from discovering new program structure, tuning parameters of already-known strategies, recombining concepts already present in the model, or preserving early biases in the population. This motivates the central question we address: What do evolutionary coding agents evolve, and how do their search dynamics produce improvements? To address this question, we introduce a dataset of evolutionary coding traces collected across multiple frameworks, models, and task families, covering mathematical constructions and algorithmic programming tasks. Rather than treating each run only by its final score, we study the full trajectory of generated programs: which solutions are explored, which lineages produce major improvements, and how stable those improvements are. The resulting dataset is intended to make evolutionary coding agents analyzable as dynamic systems, not just benchmark submissions. Analyzing these traces is non-trivial: a single run may generate hundreds of unique programs, and each candidate can differ from its ancestors through non-local structural changes, small numerical edits, prompt-driven rewrites, or evaluator-specific hacks. To make runs comparable, we develop a unified trace representation together with an annotation and measurement pipeline. The representation exposes the search graph, candidate programs, evaluations, and lineage information, enabling analyses of population structure, diversity, best-lineage stability, counterfactual model or context changes, and the decomposition of structural versus parametric gains. 2
We use this framework as a diagnostic tool for understanding both progress and failure in evolutionary coding. Our analysis covers four diagnostics: static measures of program complexity and lineage utilization (§5.1), deterministic detection of cycling, the re-introduction of previously-deleted lines (§5.2), replay-based stability tests on score-improving edits (§5.3), and a tuning-gap baseline that runs Bayesian optimization over the hyperparameters of a single program (§5.4). These diagnostics are meant to be practical: the same trace representation and measurements can be integrated into existing open-source evolutionary coding frameworks to reveal whether a run is discovering new algorithmic structure, retuning known patterns, or becoming trapped Figure 2: EvoTrace and EvoReplay. EvoTrace records each evoby its own history. Overall, our lutionary run as a structured object: programs, parent–child graph, results suggest that progress in prompts and context, scores, and evaluator metadata. EvoReevolutionary coding is not ex- play reconstructs local search states from these traces and reruns plained by final scores alone, but controlled interventions, including same-prompt replay, Bayesiandepends jointly on the task, the optimization retuning, static analysis, cycling detection, ablation, search procedure, the evaluator, repair, context substitution, and model substitution. and the underlying model family. By measuring the dynamics of search traces, we take a first step toward a systematic understanding of what these agents change over time, which changes matter, and why some runs keep improving while others stall. Contributions. We contribute two artifacts and a set of trace-level findings that use them. (1) EvoTrace1 , a dataset of 121 evolutionary coding-search runs across four frameworks on 16 benchmarks spanning Python mathematical constructions and C++ competitive programming problems, with 10,672 unique programs, 18,400 LLM calls, full parent–child graphs, prompts and contexts, scores, and evaluator metadata, normalized into a unified replayable schema (§3). (2) EvoReplay2 , a methodology and accompanying open-source package that reconstructs local search states from EvoTrace traces and reruns controlled interventions (same-prompt replay, Bayesian-optimization retuning, static analysis, deterministic cycling detection, ablation, repair, and context or model substitution), so that mechanistic claims about a search trajectory can be tested rather than asserted (§4). (3) Findings from applying these tools to characterize how evolutionary coding agents actually behave: which edit types drive score gains, how often runs end up adding back code they had previously deleted (a cycling pattern present throughout the trajectory in nearly all runs), how reliably score-improving programs can be reproduced by re-running the same prompt, how well public scores generalize to held-out evaluation on competitive programming tasks, and how much of a math run’s headline gain is recoverable by tuning the hyperparameters of a single mid-run program (§5).
2
Related Work
2.1
LLM-Guided Evolutionary Coding Approaches
LLMs can act as mutation operators inside evolutionary loops over executable programs. FunSearch [1] and AlphaEvolve [2, 3] established this paradigm, and a growing collection of open-source frameworks now extend it, including OpenEvolve [12], GEPA [13], ShinkaEvolve [4], GigaEvo [14], CodeEvolve [5], the FM Agent [15], and AIDE [16]. A second wave targets the search procedure itself by adapting strategies, models, or signals during the run [6, 7, 17–19], or by changing the unit of evolution to solution spaces, strategies, skills, prompt groups, populations, or concept trees [20–31]. 1 Available at https://huggingface.co/datasets/ZIB-IOL/EvoTrace. 2 Available at https://github.com/ZIB-IOL/EvoReplay.
3
A particularly active line of work targets GPU kernel optimization, where wall-clock runtime provides a tight reward signal [8–11, 32–36]. These frameworks have also been applied to compiler heuristics, computer architecture, cosmology, swarm-intelligence design, symbolic regression, retrieval, recommendation, hyperparameter optimization, code optimization, agentic reasoning, autonomous data science, and broader scientific discovery [37–49]. This line builds on classical evolutionary computation and quality-diversity search [50–54]; because the evolved objects are programs, recent work develops program-aware notions of diversity and similarity [55, 56]. Our work is complementary: rather than proposing another framework, we analyze traces from four of them (OpenEvolve, GEPA, ShinkaEvolve, EvoX) to study the mechanisms by which programs improve, stagnate, diversify, or collapse. 2.2
Analyzing Evolutionary Coding Agents
A separate body of work studies evolutionary coding systems themselves. Surveys situate the broader space [57, 58], benchmarks evaluate agents across ML competitions, long-horizon algorithm engineering, frontier research science, and production deployments [59–64], and Gideoni et al. [65] show that simple baselines can match elaborate evolutionary pipelines. Closer to our methodology, several papers analyze search behavior rather than only final scores: trajectory analyses [66], fitnesslandscape characterization [67], failure modes of iterative LLM optimization [68], exploration deficits [69], output homogeneity [70], emergent risks in self-evolving agents [71], and a taxonomy of multiagent failures [72]. Our work adopts a similar diagnostic perspective but focuses on full evolutionary coding traces, which we use to characterize how populations evolve, how improvement propagates through lineages, and how search dynamics produce both successes and failure modes.
3
EvoTrace
To understand what evolutionary coding agents evolve, we construct EvoTrace, a dataset of structured search traces from LLM-driven evolutionary coding systems. EvoTrace contains the artifacts needed to inspect and replay parts of the search process: generated programs, parent-child relations, prompts and retrieved context, evaluator outputs, scores, execution logs, and environment metadata. The dataset covers mathematical constructions and competitive programming tasks. These domains were chosen to capture distinct forms of code improvement: mathematical tasks reward new search algorithms, while competitive programming tasks require generated programs to compile and pass external judging, with limited access to the evaluator. Different frameworks also log different parts of the search state, making direct cross-system comparison difficult. To address these issues, EvoTrace treats evolutionary search traces not merely as logs to annotate after the fact, but as structured computational objects that can be normalized, replayed, and intervened on. The collection and replay infrastructure is built on top of SkyDiscover [49], a flexible framework for AI-driven scientific and algorithmic discovery; we extend it with a unified cross-backend schema, replay environments, and the analysis tooling described in §4. 3.1
Data Collection Across Tasks, Frameworks, and Models
EvoTrace covers 16 different tasks across two language–domain pairs: 6 Python mathematicaldiscovery tasks (circle packing, Heilbronn placement, autocorrelation and uncertainty inequalities, signal processing) and 10 C++ competitive programming problems from ALE-bench Lite [60] (AtCoder Heuristic Contest), each with a judge-defined score. We measure four evolutionary coding systems (Table 1): OpenEvolve [12], GEPA [13], EvoX [7], and ShinkaEvolve [4]. They share the propose–evaluate–feedback pattern but differ in selection, context, diversity, and adaptation. We employ 5 different LLMs (Table 2) to generate the mutations, and 100 search iterations per run, with a total of 121 runs and over 10,000 recorded program edits. 3.2
Trace schema and design choices
EvoTrace normalizes each run, regardless of which backend produced it, into a unified JSONL schema. The schema covers run-level metadata, candidate programs (full source, byte-identical), evaluator outputs (raw execution logs, errors, timings, task-specific metrics), parent–child edges with 4
Table 1: Evolutionary coding frameworks in EvoTrace. Framework
Search strategy
Distinctive feature
OpenEvolve GEPA EvoX ShinkaEvolve
Archive-based, MAP-Elites style Reflective prompt evolution Meta-evolution Sample-efficient bandit search
Open-source reference baseline Natural-language lessons drive mutation Adapts the search strategy during the run Parent sampling, novelty rejection, model routing
Table 2: Models used in EvoTrace. Model
Role
deepseek-reasoner claude-sonnet-4-6 claude-haiku-4-5 gemini-3-flash-preview deepseek-chat
Primary generator Frontier alternative Small reasoning model Cross-vendor alternative Non-reasoning baseline
operator labels, the prompts and contexts the LLM saw at generation time, and the replay environment needed to rerun selected candidates against their original evaluator. Recording the full source rather than a score-only log is what enables literal extraction for the BO baseline (§5.4), the cycling classifier (§5.2), and same-prompt replay (§5.3). Replayability is treated as a collection criterion: traces we cannot rerun against their original evaluator are excluded. The schema supports two complementary uses: aggregate analysis (cross-framework comparison of population sizes, score progression, diversity, validity, lineage depth, best-so-far trajectories) and local reconstruction by EvoReplay (§4). The full per-field schema is given in Appendix A.1.
4
EvoReplay
EvoTrace records what happened during a run; EvoReplay is the Python package we built on top of it (and on top of SkyDiscover [49]) to ask why. By treating each candidate program as an executable artifact attached to its evaluator, parent context, and search position, EvoReplay turns a passive log into an experimental object: any point in the search graph can be re-executed, perturbed, retuned, or re-judged, and the outcome compared against what the original run produced. This section describes the four capabilities EvoReplay provides. Each experimental section in the rest of the paper uses one of them, and the package is the common substrate that makes our cross-framework, cross-model results comparable. (a) Static analysis of traces. EvoReplay normalizes runs from different backends into a common per-edit table (parent, child, prompt, score, and a unified diff between parent and child source), so that aggregate measurements are defined once and applied across frameworks. The static analyses in §5.1 (hyperparameter-literal counts, lineage depth, best-so-far trajectories) and the deterministic cycling classifier in §5.2 both operate on this normalized representation, with no framework-specific code path. (b) LLM-as-judge annotation of edit types. The 9-edit-type taxonomy referenced in the abstract and §5.1 was developed by the authors through manual inspection of parent–child edits sampled across frameworks, languages, and models, grouping recurring patterns and iterating until no new categories emerged. EvoReplay’s pipeline then applies this taxonomy at scale: for every parent–child diff, it requests a structured judgment from an LLM judge, returning a category for the edit and tags for the lines that drove the score change. The package handles batching, retries, schema validation, and caching, so the same trace can be re-annotated under different prompts or judge models without re-running the underlying search. 5
0
2 × 10
0
10
best hparams ÷ seed hparams
best LOC ÷ seed LOC
3 × 10
ale (n=62) math (n=59)
0
0.0
0.2
0.4
0.6
0.8
1.0
4 × 10
0
3 × 10
0
2 × 10
0
10
ale (n=62) math (n=59)
0
0.0
normalised iteration
0.2
0.4
0.6
0.8
1.0
normalised iteration
Figure 3: Program size and numeric-literal hyperparameter count over a run. Best-so-far program length (LOC, left) and numeric-literal count (right), each normalized by the run’s seed value, plotted against normalized iteration. Solid line = cross-run median; shaded band = inter-quartile range; dashed gray line marks the seed value. Math runs (n=59) accumulate modest LOC and hp growth (median final ratios 1.33× and 1.70×); ALE runs (n=62) refine large seeds in place (final ratio ≈ 1.0× on both axes). (c) Bayesian optimization to study hyperparameter tuning. EvoReplay implements the BO baseline of §5.4: a single LLM call identifies tunable numeric constants in a target program (full prompt in Appendix B.9), the package rewrites the program with a top-level parameter block, and gp_minimize runs the same evaluator harness with 24 calls per target. This isolates the structuralvs-parametric component of an evolutionary gain on a fixed seed structure, and lets us quantify how ⋆ much of fevo a single-seed hyperparameter sweep already recovers. (d) Stability analysis of breakthroughs. EvoReplay can re-execute the saved generating prompt for any candidate under the original or a substituted model and report the distribution over children. The replay-stability results of §5.3 use this capability with n=10 resamples per target across samemodel and cross-model conditions; the resulting (parse success) × (evaluation success) × (score conditional on success) triple is the right summary because failure modes turn out to be bimodal rather than Gaussian. Together, these four capabilities make EvoTrace more than a collection of examples: they let us ask which improvements are reproducible, which are parametric, which are structural, and how each framework’s edit composition shifts across models and prompting modes.
5
Results
We analyze 121 evolutionary runs across the four frameworks of Table 1, 16 tasks spanning Python mathematical constructions and C++ ALE-bench problems (ahc008–ahc046), and 5 LLMs varied with and without diff-based generation (whether the LLM emits a unified diff or a full rewritten program). Each run consists of 100 search iterations. Aggregate program-, call-, and token-level statistics are reported in Appendix B.1. 5.1
Static analysis: what gets evolved?
We characterize the typical behaviour of evolutionary coding frameworks across our 121 runs. Programs do change shape during search (Figure 3): math runs accumulate modest LOC and numericliteral growth, while ALE runs are refined at near-constant size from already-large seeds. The figure serves as backdrop for the math-comparison difficulty discussed next and for the BO-ceiling probe of §5.4; the more interesting question is what these changes contribute, which we address through edit-level analysis below. Lineage depth and budget utilization. The chain of parents from each final-best program back to the seed is short in both domains, but is somewhat shorter on ALE (median lineage depth 4, vs. 6 on math). On math the median best-so-far is reached around iteration 0.75 of the budget, while 6
External dependency Hyperparameter tuning
Efficiency
Local refinement
Architectural change
edit label
edit label
Architectural change Bug fix Composition Efficiency
Bug fix Refactor Composition
Pruning
Hyperparameter tuning
Refactor
Pruning
External dependency 0.0
Local refinement
0.1
0.2
0.3
0.4
share of all edits with label
10
0.5
0
2 × 10
0
3 × 10
0
odds ratio for positive score delta
(a) Prevalence
(b) Helpfulness (odds ratio)
Figure 4: Edit-taxonomy: frequency vs. per-edit utility across all programs in EvoTrace. (a) Frequency of each label: Hyperparameter tuning dominates the search distribution. (b) Per-edit odds ratio for positive normalized score change: External dependency, Efficiency, and Architectural change are the most helpful categories on a per-edit basis. The categories that most often improve a single edit are not the categories the search spends most of its effort on. Best-so-far and final-best-lineage enrichment views, plus per-domain and per-backend breakdowns, are in Appendix B.2 and Appendix B.3. on ALE it lands earlier (median normalized iteration 0.49). In both cases the dominant pattern is jackpot-then-flat: most of the per-run iteration budget is spent on dead branches that do not contribute to the final best. Public-vs-private generalization on ALE. ALE-bench public scores are not the held-out judging metric. Re-scoring every ALE run’s public best-so-far chain on the private test set used by AtCoder (n=30 run/problem pairs across the four main backends), two of the four frameworks overfit on at least 30% of the problems they were scored on, and the same problem can flip generalization sign across frameworks: on ahc024, OpenEvolve found a +1,606 rating-point private gain while ShinkaEvolve, on the same problem, lost 1,610 rating points despite a positive public score change. The public best-so-far chain is therefore unreliable as a single-number summary on ALE; full per-problem and per-framework tables are in Appendix B.7. Edit taxonomy via LLM-as-judge. We use EvoReplay’s LLM-as-judge pipeline (§4, capability (b)) to annotate every parent–child edit with one or more categories from a 9-label taxonomy (Hyperparameter tuning, Local refinement, Architectural change, Composition, Efficiency, Bug fix, External dependency, Pruning, and Refactor, applied to every parent–child edit in EvoTrace across the four backends. Agreement between the judge and a blind human re-annotation on a stratified sample of 200 edits is substantial overall (macro κ = 0.77, micro-F1 = 0.90, exact-match accuracy 74.5%), with per-category breakdowns and the one failure case (external_dependency) reported in Appendix B.4. The picture splits cleanly into a frequency view and a per-edit utility view (Figure 4). By frequency, Hyperparameter tuning is the single most prevalent label, consistent with the cycling pattern of §5.2 and the tuning-gap analysis of §5.4. By per-edit utility, however, the strongest categories are different: External dependency edits have a 3.58× odds ratio for positive normalized score change (n=104), Efficiency 1.61× (n=464), and Architectural change 1.55× (n=1,075). The frequency–utility gap propagates to successful trajectories: best-so-far updates and final-best lineages are both enriched in Efficiency, External dependency, and Hyperparameter tuning relative to the all-edits base rate (Appendix B.2). Edits are typically multi-label (67.4% have ≥ 2 labels), so these categories should not be read as mutually exclusive modes; the most common compound patterns are Hyperparameter tuning + Local refinement and Composition + Hyperparameter tuning. Finding. The categories that most often improve a single edit (External dependency, Efficiency, Architectural change) are not the categories evolutionary search spends most of its effort on (Hyperparameter tuning, Local refinement). Most score gains come from a small subset of edit types, and that subset is rare in the search distribution. 7
Table 3: Replay summary across 36 breakthrough events. Aggregate medians and four illustrative targets covering recurring replay patterns. “Replay/Original” is the median replayed score divided by the original program’s score. Target
Parse Eval Exact Replay/Orig. Pattern
Median across 36 targets
1.00
1.00
0.00
0.76
—
circle_packing_iter68_improve second_autocorr_iter79_improve circle_packing_iter32_neutral heilbronn_convex_iter100_regress
1.00 1.00 0.50 1.00
1.00 1.00 0.50 1.00
≥0 0.00 0.00 0.00
≈ 1.00 0.96 — 1.11
tight reproduction parent-revert bimodal failure replays exceed regress.
5.2
Cycling: re-introducing previously deleted code
While manually inspecting traces to derive the edit taxonomy, we repeatedly observed lineages re-introducing lines they had earlier deleted, so we operationalized this as a deterministic check: for each parent–child diff, how often is an added line byte-identical to a line that the same lineage has already deleted in an earlier iteration? Across all 121 runs, the median share of added lines that are such re-introductions is ∼ 30%, and this rate grows monotonically over the run in 118 of 121 cases (median per-iteration slope +0.0030). Cycling is present throughout the trajectory of essentially every run we measure, not a late-run pathology, and is dominated by short-span churn (median 5 iterations between deletion and re-introduction); the signal is stable across all four frameworks, both languages, and all 5 generator models we tested. A walk-through of one short-span cycle is given in Appendix B.8, with additional analyses (a finer three-way recycling classifier, a model- and prompt-dependence breakdown, and a null result on post-breakthrough cycling) in Appendix B.6. Finding. Roughly 30% of code lines added during evolutionary search are byte-identical to lines previously deleted in the same lineage, and the cycling rate grows monotonically over the run in 118 of 121 cases. Search budget is partly spent re-introducing material the run has already discarded, a deterministic and reproducible signal that is present throughout each run and across all frameworks, languages, and generator models we tested. 5.3
Replay reproducibility: structural, not lexical
When evolutionary search reaches a new best-so-far program, can we reproduce that breakthrough by re-running the same prompt? For each of 36 best-so-far events across the four backends, we re-prompted an LLM 10 times with the exact context the original run had used and asked four questions of each replayed program: does it run? does the evaluator accept it? does it match the original program byte-for-byte? does it match the original score? Table 3 reports the medians and four illustrative targets covering the recurring patterns. Finding. Replays almost always produce a runnable program (median parse and evaluator success 1.00) but essentially never the original program (median exact-match 0.00). They nevertheless recover a median 0.76 of the original score from a different program: the score gain is broadly reproducible from the same prompt context even though the specific program is not. 5.4
The tuning gap: how much is just hyperparameter search?
We separate a program p into a structure s and a hyperparameter vector θ ∈ Θs it exposes, writing p = s(θ). Holding s0 fixed and running Bayesian optimization over θ yields a tuning ceiling ⋆ ⋆ ⋆ fBO (s0 ) = maxθ f (s0 (θ)), and the tuning gap ∆(s0 ) = fevo − fBO (s0 ) measures how much of the evolutionary gain reflects structural discovery rather than parametric search. We operationalize ⋆ fBO with one deepseek-reasoner call that proposes per-knob log/linear intervals, an automatic rewrite to a top-level PARAMS block, and a 24-call gp_minimize (8 random +16 BO acquisitions; full pipeline in Appendix B.9).
8
BO matches the evolutionary run’s final-best on Table 4: BO outcomes on 36 mid-run promost intermediate programs. On 36 intermediate grams (median 6 knobs per program; 24 programs sampled across runs, frameworks, and models, evaluator calls each). BO improves over the program’s original score in 22 of Outcome # targets 36 cases (Table 4). When compared against the run’s fiBO improves over original 22 nal-best score (rather than the program’s original score), No change 8 BO matches or exceeds it on 13 of 15 intermediate proBO regresses 6 grams (median delta +0.025). The largest individual gain is on heilbronn_tri_dsr_nodiff, where the Total 36 evo run reached 0.521 in 100 iterations and BO on an intermediate program from the same run reached 0.886 ⋆ (1.70× the evo final-best). The strong dependence of fBO (s0 ) on s0 ’s exposed knobs complicates math-benchmark cross-framework comparison: two frameworks with similar topology but different knob exposures give different headline scores even with identical search behaviour, so the defensible ⋆ per-target summary is the pair f (p0 ), fBO (s0 ) . Finding. A 24-call Bayesian-optimization pass on a single intermediate program’s exposed hyperparameters improves over the program’s score in 22 of 36 probed targets, and matches or exceeds the evolutionary run’s final-best score on 13 of 15 intermediate programs (median delta +0.025). On these targets, late evolutionary iterations on math are largely matched by post-hoc hyperparameter tuning of an earlier program.
6
Discussion
Looking at the traces themselves rather than at final scores, our diagnostics surface several recurring inefficiencies in current LLM-driven evolutionary code search. A non-trivial share of the search budget is spent re-introducing material the run has already discarded: ∼ 30% of added lines are byte-identical to previously-deleted ones, and this share grows steadily across the trajectory in 118 of 121 runs. Breakthrough events are also not crisp, repeatable artifacts: same-prompt replays almost never reproduce the original program byte-for-byte, yet typically recover a substantial fraction of its score from a different program. The trajectory carries the structural gain, while the specific program is one draw from a wider distribution. Lineages back to the seed are short, so most of the per-run budget is spent on branches that do not contribute to the final best. On math benchmarks, a Bayesian-optimization pass over a single intermediate program’s exposed knobs often matches or exceeds the run’s final-best score, suggesting that on math the parametric refinement evolutionary search performs late in a run is largely substitutable by post-hoc tuning. On ALE, two of four frameworks overfit on at least 30% of their problems. These patterns hold across four frameworks, two languages, and five LLMs. Implications. On math, the BO finding suggests a natural decomposition of the work an evolutionary run does: the structural changes it makes early in a run, and the parametric refinement of those structures, which can often be done post-hoc by hyperparameter tuning of an intermediate program. A practical consequence is that math-benchmark headline scores should be reported alongside the ⋆ single-program tuning ceiling fBO (s0 ) so this decomposition is visible; on ALE, public scores should additionally be paired with a private-test re-score to surface overfitting. For system design, cycling growth and lineage shallowness suggest that interventions which prevent the search from re-doing discarded work (lineage-aware credit assignment, deletion-aware novelty filters, prompting strategies that expose a parent’s deletion history) are promising directions to explore, complementary to extending the search budget. Scope and open questions. The trace-level view we develop here opens several natural directions. The single-program tuning ceiling, currently reported on math, can be extended to other domains and evaluator types; replay-based reproducibility can be probed with larger samples and richer perturbations of the local search state; and the dynamics we surface (cycling, lineage shallowness, and the frequency–utility gap across edit categories) can be re-examined under different selection rules, prompting strategies, and underlying model families. Because EvoTrace records full source and replay environments, each of these follow-ups can be posed as a controlled intervention on the same trace, without re-running the original search. 9
Acknowledgements This research was partially supported by the DFG Cluster of Excellence MATH+ (EXC-2046/1, project id 390685689) funded by the Deutsche Forschungsgemeinschaft (DFG) as well as by the German Federal Ministry of Research, Technology and Space (fund number 16IS23025B).
10
References [1] Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475, January 2024. ISSN 0028-0836, 1476-4687. doi: 10.1038/s41586-023-06924-6. [2] Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. AlphaEvolve: A coding agent for scientific and algorithmic discovery, June 2025. https://arxiv.org/abs/2506.13131. [3] Bogdan Georgiev, Javier Gómez-Serrano, Terence Tao, and Adam Zsolt Wagner. Mathematical exploration and discovery at scale, December 2025. https://arxiv.org/abs/2511.02864. [4] Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution, September 2025. https://arxiv.org/abs/2509. 19349. [5] Henrique Assumpção, Diego Ferreira, Leandro Campos, and Fabricio Murai. CodeEvolve: An open source evolutionary coding agent for algorithmic discovery and optimization, March 2026. https://arxiv.org/abs/2510.14150. [6] Mert Cemri, Shubham Agrawal, Akshat Gupta, Shu Liu, Audrey Cheng, Qiuyang Mang, Ashwin Naren, Lutfi Eren Erdogan, Koushik Sen, Matei Zaharia, Alex Dimakis, and Ion Stoica. AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization, February 2026. https: //arxiv.org/abs/2602.20133. [7] Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, Mert Cemri, Zhifei Li, Qiuyang Mang, Ashwin Naren, Ethan Boneh, Audrey Cheng, Melissa Z. Pan, Alexander Du, Kurt Keutzer, Alvin Cheung, Alexandros G. Dimakis, Koushik Sen, Matei Zaharia, and Ion Stoica. EvoX: MetaEvolution for Automated Discovery, March 2026. https://arxiv.org/abs/2602.23413. [8] Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Shubham Agarwal, Mert Cemri, Bowen Wang, Alexander Krentsel, Tian Xia, Jongseok Park, Shuo Yang, Jeff Chen, Lakshya Agrawal, Ashwin Naren, Shulu Li, Ruiying Ma, Aditya Desai, Jiarong Xing, Koushik Sen, Matei Zaharia, and Ion Stoica. Let the Barbarians In: How AI Can Accelerate Systems Performance Research, December 2025. https://arxiv.org/abs/2512.14806. [9] Ping Guo, Chenyu Zhu, Siyuan Chen, Fei Liu, Xi Lin, Zhichao Lu, and Qingfu Zhang. EvoEngineer: Mastering Automated CUDA Kernel Code Evolution with Large Language Models, October 2025. https://arxiv.org/abs/2510.03760. [10] Shiyi Cao, Ziming Mao, Joseph E. Gonzalez, and Ion Stoica. K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model, February 2026. https://arxiv.org/abs/2602. 19128. [11] Nina Wiedemann, Quentin Leboutet, Michael Paulitsch, Diana Wofk, and Benjamin Ummenhofer. KernelFoundry: Hardware-aware evolutionary GPU kernel optimization, March 2026. https://arxiv.org/abs/2603.12440. [12] Asankhaya Sharma. Openevolve: an open-source evolutionary coding agent, 2025. URL https://github.com/algorithmicsuperintelligence/openevolve. [13] Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista OpsahlOng, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, February 2026. https://arxiv.org/abs/2507.19457. 11
[14] Valentin Khrulkov, Andrey Galichin, Denis Bashkirov, Dmitry Vinichenko, Oleg Travkin, Roman Alferov, Andrey Kuznetsov, and Ivan Oseledets. GigaEvo: An Open Source Optimization Framework Powered By LLMs And Evolution Algorithms, November 2025. https://arxiv.org/abs/2511.17592. [15] Annan Li, Chufan Wu, Zengle Ge, Yee Hin Chong, Zhinan Hou, Lizhe Cao, Cheng Ju, Jianmin Wu, Huaiming Li, Haobo Zhang, Shenghao Feng, Mo Zhao, Fengzhi Qiu, Rui Yang, Mengmeng Zhang, Wenyi Zhu, Yingying Sun, Quan Sun, Shunhao Yan, Danyu Liu, Dawei Yin, and Dou Shen. The FM Agent, February 2026. https://arxiv.org/abs/2510.26144. [16] Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. AIDE: AI-Driven Exploration in the Space of Code, February 2025. https: //arxiv.org/abs/2502.13138. [17] Minghao Yan, Bo Peng, Benjamin Coleman, Ziqi Chen, Zhouhang Xie, Shuo Chen, Zhankui He, Noveen Sachdeva, Isabella Ye, Weili Wang, Chi Wang, Ed H. Chi, Fernando Pereira, Wang-Cheng Kang, Derek Zhiyuan Cheng, and Beidou Wang. PACEvolve: Enabling LongHorizon Progress-Aware Consistent Evolution, January 2026. https://arxiv.org/abs/ 2601.10657. [18] Pretam Ray, Pratik Prabhanjan Brahma, Zicheng Liu, and Emad Barsoum. AdaptEvolve: Improving Efficiency of Evolutionary AI Agents through Adaptive Model Selection, February 2026. https://arxiv.org/abs/2602.11931. [19] Yongqiang Chen, Chenxi Liu, Zhenhao Chen, Tongliang Liu, Bo Han, and Kun Zhang. CausalEvolve: Towards Open-Ended Discovery with Causal Scratchpad, March 2026. https: //arxiv.org/abs/2603.14575. [20] Yi Zhai, Zhiqiang Wei, Ruohan Li, Keyu Pan, Shuo Liu, Lu Zhang, Jianmin Ji, Wuyang Zhang, Yu Zhang, and Yanyong Zhang. \(X\)-evolve: Solution space evolution powered by large language models, August 2025. https://arxiv.org/abs/2508.07932. [21] Sichun Luo, Yi Huang, Haochen Luo, Fengyuan Liu, Guanzhi Deng, Lei Li, Qinghua Yao, Zefa Hu, Junlan Feng, and Qi Liu. SeaEvo: Advancing Algorithm Discovery with Strategy Space Evolution, April 2026. https://arxiv.org/abs/2604.24372. [22] Haoran Ye, Xuning He, Vincent Arak, Haonan Dong, and Guojie Song. Meta Context Engineering via Agentic Skill Evolution, February 2026. https://arxiv.org/abs/2601.21557. [23] Tiancheng Li, Yuhang Wang, Zhiyang Chen, Zijun Wang, Liyuan Ma, and Guo-jun Qi. CEvolve: Consensus-based Evolution for Prompt Groups, September 2025. https://arxiv. org/abs/2509.23331. [24] Yanzhi Zhang, Yitong Duan, Zhaoxi Zhang, Jiyan He, and Shuxin Zheng. Population-Evolve: A Parallel Sampling and Evolutionary Method for LLM Math Reasoning, December 2025. https://arxiv.org/abs/2512.19081. [25] Timothee Leleu, Sudeera Gunathilaka, Federico Ghimenti, and Surya Ganguli. Contrastive Concept-Tree Search for LLM-Assisted Algorithm Discovery, February 2026. https:// arxiv.org/abs/2602.03132. [26] Julien Pourcel, Cédric Colas, and Pierre-Yves Oudeyer. Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGI, March 2026. https: //arxiv.org/abs/2507.14172. [27] Yiping Wang, Shao-Rong Su, Zhiyuan Zeng, Eva Xu, Liliang Ren, Xinyu Yang, Zeyi Huang, Xuehai He, Luyao Ma, Baolin Peng, Hao Cheng, Pengcheng He, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. ThetaEvolve: Test-time Learning on Open Problems, November 2025. https://arxiv.org/abs/2511.23473. [28] Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, Yejin Choi, James Zou, Carlos Guestrin, and Yu Sun. Learning to Discover at Test Time, February 2026. https://arxiv.org/abs/2601.16175. 12
[29] Ruiying Ma, Chieh-Jan Mike Liang, Yanjie Gao, and Francis Y. Yan. MetaMuse: Algorithm Generation via Creative Ideation, October 2025. https://arxiv.org/abs/2510.03851. [30] Shivam Singhal, Priyadarsi Mishra, Eran Malach, and Tomer Galanti. LLM Priors for ERM over Programs, February 2026. https://arxiv.org/abs/2510.14331. [31] Wei Liu, Siya Qi, Yali Du, and Yulan He. Self-play only evolves when self-synthetic pipeline ensures learnable information gain, 2026. https://arxiv.org/abs/2603.02218. [32] Weihua Du, Jingming Zhuo, Yixin Dong, Andre Wang He, Weiwei Sun, Zeyu Zheng, Manupa Karunaratne, Ivan Fox, Tim Dettmers, Tianqi Chen, Yiming Yang, and Sean Welleck. AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation, April 2026. https://arxiv.org/abs/2604.16625. [33] Daniel Nichols, Konstantinos Parasyris, Caetano Melone, Tal Ben-Nun, Giorgis Georgakoudis, and Harshitha Menon. Record-Remix-Replay: Hierarchical GPU Kernel Optimization using Evolutionary Search, April 2026. https://arxiv.org/abs/2604.11109. [34] Anjiang Wei, Tianran Sun, Yogesh Seenichamy, Hang Song, Anne Ouyang, Azalia Mirhoseini, Ke Wang, and Alex Aiken. Astra: A Multi-Agent System for GPU Kernel Performance Optimization, December 2025. https://arxiv.org/abs/2509.07506. [35] Hongyuan Su, Yu Zheng, and Yong Li. ContextEvolve: Multi-Agent Context Compression for Systems Code Optimization, February 2026. https://arxiv.org/abs/2602.02597. [36] Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia, Mert Cemri, Jongseok Park, Shuo Yang, Jeff Chen, Lakshya Agrawal, Aditya Desai, Jiarong Xing, Koushik Sen, Matei Zaharia, and Ion Stoica. Barbarians at the Gate: How AI is Upending Systems Research, October 2025. https://arxiv.org/abs/2510.06189. [37] Hongzheng Chen, Alexander Novikov, Ngân Vũ, Hanna Alam, Zhiru Zhang, Aiden Grossman, Mircea Trofin, and Amir Yazdanbakhsh. Magellan: Autonomous Discovery of Novel Compiler Optimization Heuristics with AlphaEvolve, January 2026. https://arxiv.org/abs/2601. 21096. [38] Raghav Gupta, Akanksha Jain, Abraham Gonzalez, Alexander Novikov, Po-Sen Huang, Matej Balog, Marvin Eisenberger, Sergey Shirobokov, Ngân Vũ, Martin Dixon, Borivoje Nikolić, Parthasarathy Ranganathan, and Sagar Karandikar. ArchAgent: Agentic AI-driven Computer Architecture Discovery, February 2026. https://arxiv.org/abs/2602.22425. [39] Tianyi Li, Shihui Zang, and Moritz Münchmeyer. MadEvolve: Evolutionary Optimization of Cosmological Algorithms with Large Language Models, February 2026. https://arxiv. org/abs/2602.15951. [40] Shipeng Cen and Ying Tan. Beyond Algorithm Evolution: An LLM-Driven Framework for the Co-Evolution of Swarm Intelligence Optimization Algorithms and Prompts, December 2025. https://arxiv.org/abs/2512.09209. [41] Zhuo-Yang Song, Zeyu Cai, Shutao Zhang, Jiashen Wei, Jichen Pan, Shi Qiu, Qing-Hong Cao, Tie-Jiun Hou, Xiaohui Liu, Ming-xing Luo, and Hua Xing Zhu. Iterated Agent for Symbolic Regression, October 2025. https://arxiv.org/abs/2510.08317. [42] Jinming Nian, Fangchen Li, Dae Hoon Park, and Yi Fang. RankEvolve: Automating the Discovery of Retrieval Algorithms via LLM-Driven Evolution, February 2026. https:// arxiv.org/abs/2602.16932. [43] Haochen Wang, Yi Wu, Daryl Chang, Li Wei, and Lukasz Heldt. Self-Evolving Recommendation System: End-To-End Autonomous Model Optimization With LLM Agents, February 2026. https://arxiv.org/abs/2602.10226. [44] Fabio Ferreira, Lucca Wobbe, Arjun Krishnakumar, Frank Hutter, and Arber Zela. Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch, April 2026. https://arxiv.org/abs/2603.24647. 13
[45] Tu Hu, Ronghao Chen, Shuo Zhang, Jianghao Yin, Mou Xiao Feng, Jingping Liu, Shaolei Zhang, Wenqi Jiang, Yuqi Fang, Sen Hu, Huacan Wang, and Yi Xu. Controlled Self-Evolution for Algorithmic Code Optimization, February 2026. https://arxiv.org/abs/2601.07348. [46] Zhanke Zhou, Chentao Cao, Xiao Feng, Xuan Li, Zongze Li, Xiangyu Lu, Jiangchao Yao, Weikai Huang, Tian Cheng, Jianghangfan Zhang, Tangyu Jiang, Linrui Xu, Yiming Zheng, Brando Miranda, Tongliang Liu, Sanmi Koyejo, Masashi Sugiyama, and Bo Han. AlphaApollo: A System for Deep Agentic Reasoning, March 2026. https://arxiv.org/abs/2510. 06261. [47] Xu Yang, Xiao Yang, Shikai Fang, Yifei Zhang, Jian Wang, Bowen Xian, Qizheng Li, Jingyuan Li, Minrui Xu, Yuante Li, Haoran Pan, Yuge Zhang, Weiqing Liu, Yelong Shen, Weizhu Chen, and Jiang Bian. R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science, October 2025. https://arxiv.org/abs/2505.14738. [48] Zhaotian Weng, Antonis Antoniades, Deepak Nathani, Zhen Zhang, Xiao Pu, and Xin Eric Wang. Group-Evolving Agents: Open-Ended Self-Improvement via Experience Sharing, February 2026. https://arxiv.org/abs/2602.04837. [49] Shu Liu, Mert Cemri, Shubham Agarwal, Alexander Krentsel, Ashwin Naren, Qiuyang Mang, Zhifei Li, Akshat Gupta, Monishwaran Maheswaran, Audrey Cheng, Melissa Pan, Ethan Boneh, Kannan Ramchandran, Koushik Sen, Alexandros G. Dimakis, Matei Zaharia, and Ion Stoica. SkyDiscover: A flexible framework for AI-driven scientific and algorithmic discovery, 2026. URL https://skydiscover-ai.github.io/blog.html. [50] John R. Koza. Genetic programming as a means for programming computers by natural selection. Statistics and Computing, 4(2), June 1994. ISSN 0960-3174, 1573-1375. doi: 10.1007/BF00175355. [51] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population Based Training of Neural Networks, November 2017. https://arxiv.org/abs/1711.09846. [52] Chao Qian, Ke Xue, and Ren-Jian Wang. Quality-Diversity Algorithms Can Provably Be Helpful for Optimization, May 2024. https://arxiv.org/abs/2401.10539. [53] Giorgia Nadizar, Francesco Rusin, Eric Medvet, and Gabriela Ochoa. The Role of Stepping Stones in MAP-Elites: Insights from Search Trajectory Networks. In Bing Xue, Luca Manzoni, and Illya Bakurov, editors, Genetic Programming, volume 15609, pages 224–239. Springer Nature Switzerland, Cham, 2025. ISBN 978-3-031-89990-4 978-3-031-89991-1. doi: 10.1007/ 978-3-031-89991-1_14. [54] Edward Hughes, Michael Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rocktaschel. Open-Endedness is Essential for Artificial Superhuman Intelligence, June 2024. https://arxiv.org/abs/2406.04268. [55] Dan Friedman and Adji Bousso Dieng. The Vendi Score: A Diversity Evaluation Metric for Machine Learning, July 2023. https://arxiv.org/abs/2210.02410. [56] Rui Zhang and Zhichao Lu. Rethinking Code Similarity for Automated Algorithm Design with LLMs, March 2026. https://arxiv.org/abs/2603.02787. [57] Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu, Siwei Liu, Zihao Li, Zhaochun Ren, Nikos Aletras, Xi Wang, Han Zhou, and Zaiqiao Meng. A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems, August 2025. https://arxiv.org/abs/ 2508.07407. [58] Qiujie Xie, Yixuan Weng, Minjun Zhu, Fuchen Shen, Shulin Huang, Zhen Lin, Jiahui Zhou, Zilan Mao, Zijie Yang, Linyi Yang, Jian Wu, and Yue Zhang. How Far Are AI Scientists from Changing the World?, August 2025. https://arxiv.org/abs/2507.23276. 14
[59] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. ˛ MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering, February 2025. https://arxiv.org/abs/2410.07095. [60] Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering, October 2025. https://arxiv.org/abs/2506.09050. [61] Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka, Alberto Pepe, Alexis Audran-Reiss, Muna Aghamelu, Nicolas Baldwin, Lucia Cipolina-Kun, JeanChristophe Gagnon-Audet, Chee Hau Leow, Sandra Lefdal, Hossam Mossalam, Abhinav Moudgil, Saba Nazir, Emanuel Tewolde, Isabel Urrego, Jordi Armengol Estape, Amar Budhiraja, Gaurav Chaurasia, Abhishek Charnalia, Derek Dunfield, Karen Hambardzumyan, Daniel Izcovich, Martin Josifoski, Ishita Mediratta, Kelvin Niu, Parth Pathak, Michael Shvartsman, Edan Toledo, Anton Protopopov, Roberta Raileanu, Alexander Miller, Tatiana Shavrina, Jakob Foerster, and Yoram Bachrach. AIRS-Bench: A Suite of Tasks for Frontier AI Research Science Agents, February 2026. https://arxiv.org/abs/2602.06855. [62] Haotian Ye, Haowei Lin, Jingyi Tang, Yizhen Luo, Caiyin Yang, Chang Su, Rahul Thapa, Rui Yang, Ruihua Liu, Zeyu Li, Chong Gao, Dachao Ding, Guangrong He, Miaolei Zhang, Lina Sun, Wenyang Wang, Yuchen Zhong, Zhuohao Shen, Di He, Jianzhu Ma, Stefano Ermon, Tongyang Li, Xiaowen Chu, James Zou, and Yuzhi Xu. Evaluation-driven Scaling for Scientific Discovery, April 2026. https://arxiv.org/abs/2604.19341. [63] Melissa Z. Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya A. Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, Shu Liu, Tianneng Shi, Xiaoyuan Liu, Jared Quincy Davis, Emmanuele Lacavalla, Alessandro Basile, Shuyi Yang, Paul Castro, Daniel Kang, Joseph E. Gonzalez, Koushik Sen, Dawn Song, Ion Stoica, Matei Zaharia, and Marquita Ellis. Measuring Agents in Production, February 2026. https://arxiv.org/abs/ 2512.04123. [64] Jingsheng Zheng, Jintian Zhang, Yujie Luo, Yuren Mao, Yunjun Gao, Lun Du, Huajun Chen, and Ningyu Zhang. Can We Predict Before Executing Machine Learning Agents?, January 2026. https://arxiv.org/abs/2601.05930. [65] Yonatan Gideoni, Sebastian Risi, and Yarin Gal. Simple Baselines are Competitive with Code Evolution, February 2026. https://arxiv.org/abs/2602.16805. [66] Xinhao Zhang, Xi Chen, François Portet, and Maxime Peyrard. What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search, April 2026. https://arxiv.org/abs/2604.19440. [67] Fei Liu, Qingfu Zhang, Jialong Shi, Xialiang Tong, Kun Mao, and Mingxuan Yuan. Fitness Landscape of Large Language Model-Assisted Automated Algorithm Search, August 2025. https://arxiv.org/abs/2504.19636. [68] Allen Nie, Xavier Daull, Zhiyi Kuang, Abhinav Akkiraju, Anish Chaudhuri, Max Piasevoli, Ryan Rong, YuCheng Yuan, Prerit Choudhary, Shannon Xiao, Rasool Fakoor, Adith Swaminathan, and Ching-An Cheng. Understanding the Challenges in Iterative Generative Optimization with LLMs, March 2026. https://arxiv.org/abs/2603.23994. [69] Lan Pan, Hanbo Xie, and Robert C. Wilson. Large Language Models Think Too Fast To Explore Effectively, May 2025. https://arxiv.org/abs/2501.18009. [70] Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Alon Albalak, and Yejin Choi. Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond), October 2025. https://arxiv.org/abs/ 2510.22954. [71] Shuai Shao, Qihan Ren, Chen Qian, Boyi Wei, Dadi Guo, Jingyi Yang, Xinhao Song, Linfeng Zhang, Weinan Zhang, Dongrui Liu, and Jing Shao. Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents, March 2026. https://arxiv.org/abs/2509.26354. 15
[72] Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why Do Multi-Agent LLM Systems Fail?, October 2025. https://arxiv.org/abs/2503.13657.
16
A
Additional EvoTrace Details
A.1
Per-field trace schema
EvoTrace normalizes each run into six object types stored as JSONL tables, each motivated by a mechanistic question that storing only iteration-vs-score traces would foreclose. Table 5 summarises the six object types. Table 5: EvoTrace per-field schema. Each object type is recorded because at least one analysis in the main paper requires the corresponding raw artifact rather than a score-only summary. Object
Fields
Why recorded
Runs
Task, framework, model configuration, evaluator command, seed artifact, search budget, run-time environment. Full program source (byte-identical), iteration index, parent identifiers, prompt context, validity status, evaluator score. Execution outputs, error messages, timing, correctness signals, taskspecific metrics. Parent–child relations with the operator that produced the child (mutation, recombination, refinement, repair). Prompts, retrieved examples, population summaries, and lineage information seen by the LLM at generation time (byte-identical). Evaluator command, dependencies, timeouts, hardware assumptions, raw artifacts.
Makes any reported number sliceable by framework or model.
Candidates
Evaluations
Edges
Contexts
Replay environments
B
Additional Experimental Details
B.1
Experiment scale and cost
The full source enables literal extraction for the BO baseline (§5.4, Appendix B.9), the cycling classifier (§5.2), and post-hoc simplification or repair. The bimodal failure picture in §5.3 is invisible to score-only logging. Lineage-depth and dead-branch statistics in §5.1 require the full edge table, not the best-so-far trajectory. Stability replays in §5.3 pass the saved input back to the original or a substituted model.
Replayability is a collection criterion: traces we cannot rerun against the original evaluator are excluded.
The dataset spans 121 evolutionary runs that together propose 10,672 unique programs (including 1,708 explicitly rejected ones), make 18,400 LLM calls, and consume 274.7 M prompt and 80.3 M completion tokens (of which 42.8 M are reasoning tokens). A typical 100-iteration run produces about 100 programs, makes about 134 LLM calls, and uses ∼ 1.7 M prompt and ∼ 535 K completion tokens. ALE runs use approximately 2.8× more prompt tokens than math runs because of their much larger seeds. Tables 6, 7, and 8 report full breakdowns. Table 6: Experiment scale by backend. “Edits” counts programs with a non-null parent; LLM calls and tokens are aggregated from each run’s call log. backend
runs
programs
accepted
rejected
edits
LLM calls
prompt tok
compl. tok
openevolve evox gepa shinka
44 30 29 18
4,267 2,396 2,137 1,872
4,267 2,396 429 1,872
0 0 1,708 0
4,223 2,366 2,108 1,782
5,064 4,700 4,194 4,442
84.9 M 71.5 M 78.1 M 40.2 M
21.1 M 14.4 M 19.7 M 25.1 M
total
121
10,672
8,964
1,708
10,479
18,400
274.7 M
80.3 M
17
Table 7: Experiment scale by domain (ALE vs. math). domain
runs
programs
edits
LLM calls
prompt tok
compl. tok
ale math
56 65
5,269 5,403
5,173 5,306
9,341 9,059
201.7 M 73.0 M
42.8 M 37.5 M
total
121
10,672
10,479
18,400
274.7 M
80.3 M
Table 8: Per-run cost (medians and right tails) over the 121 runs.
B.2
per-run metric
median
75th pct
max
programs edits (parent → child) LLM calls prompt tokens completion tokens
101 99 134 1.7 M 535,582
107 106 165 3.3 M 811,225
118 117 512 7.3 M 2.7 M
Edit-taxonomy: aggregate enrichment views
The main paper (Figure 4) features the two-panel prevalence vs. helpfulness view. The companion enrichment panels are reported here. Relative to the all-edits base rate, best-so-far updates are enriched in Efficiency (1.49×), External dependency (1.34×), and Hyperparameter tuning (1.32×); final-best lineages retain a similar mix, with Efficiency (1.42×), Hyperparameter tuning (1.27×), and Composition (1.21×) overrepresented (Figures 5 and 6). The same broad set of categories is enriched on both intermediate-improvement events and on the lineages that produce the eventual winner, so the frequency–utility gap surfaced in the main figure is not an artefact of conditioning on a particular subset of edits.
Efficiency External dependency Hyperparameter tuning
edit label
Composition Local refinement Architectural change Bug fix Refactor Pruning 6 × 10
−1
10
0
enrichment ratio vs all edits
Figure 5: Best-so-far enrichment of edit labels (aggregate). Enrichment of each taxonomy label among best-so-far updates relative to the all-edits base rate. The categories most overrepresented on successful intermediate steps (Efficiency, External dependency, Hyperparameter tuning, Composition) are not identical to the most frequent labels in Figure 4 (a). B.3
Edit-taxonomy breakdowns by domain and backend
The aggregate edit-taxonomy results reported in §5.1 (Figure 4) combine ALE and math runs across all four backends. This section reports the same analyses split by domain and, for the helpfulness view, by backend, so that readers can verify that the headline patterns survive these slices. The underlying labeled corpus covers all programs in EvoTrace. 18
Efficiency Hyperparameter tuning Composition
edit label
Architectural change External dependency Local refinement Bug fix Refactor Pruning 6 × 10
−1
10
0
enrichment ratio vs all edits
Figure 6: Final-best-lineage enrichment (aggregate, robustness check). Enrichment of each label along the lineage from each run’s final best program back to the seed. Efficiency, Hyperparameter tuning, and Composition remain overrepresented relative to the all-edits base rate, supporting the best-so-far view in Figure 5.
Hyperparameter tuning
Architectural change
Hyperparameter tuning
Architectural change
Composition
edit label
Local refinement
Bug fix
edit label
Local refinement
Composition Refactor
Bug fix Efficiency
Efficiency
Pruning
Pruning
External dependency
External dependency 0.0
Refactor 0.1
0.2
0.3
0.4
0.5
0.0
share of all edits with label
0.1
0.2
0.3
0.4
0.5
share of all edits with label
(a) ALE
(b) Math
Figure 7: Edit-label prevalence by domain. Frequency of each taxonomy label among labeled edits, split by domain. Hyperparameter tuning dominates in both domains, but Composition is more prominent on math while structural categories shift their relative weights between domains.
B.4
LLM-as-judge validation
Taxonomy origin. The 9-category edit taxonomy used in §5.1 was derived inductively from EvoTrace runs rather than imposed top-down. The first author sampled several classified runs and proposed an initial set of edit categories; these were discussed and refined with co-authors over multiple iterations on further sampled traces, with categories merged or split until the label set stabilised on the nine used to prompt the LLM judge: hyperparameter_tuning, local_refinement, architectural_change, composition, efficiency, bug_fix, pruning, refactor, and external_dependency. Table 9 gives a one-line working definition for each label; curated example diffs are provided in Appendix B.5. Inter-rater reliability against the LLM judge. To validate the LLM-as-judge classifier we conducted a blind multi-label inter-rater reliability study on a stratified sample of 200 parent→child edits drawn from three classified runs. The first author labelled each edit without seeing the model’s output; agreement was then computed against the deepseek-chat judge. 19
External dependency
External dependency
Efficiency
Composition
Architectural change
Architectural change Efficiency
edit label
edit label
Bug fix Refactor Local refinement
Local refinement Hyperparameter tuning
Pruning
Bug fix
Hyperparameter tuning
Refactor
Composition
Pruning 10
0
2 × 10
0
3 × 10
0
10
odds ratio for positive score delta
0
odds ratio for positive score delta
(a) ALE
(b) Math
Figure 8: Per-edit helpfulness (odds ratio for positive normalized score change) by domain. On ALE, External dependency, Efficiency, and Architectural change are the strongest positive categories. On math, External dependency is even stronger and Composition plays a larger role than on ALE.
Composition
Local refinement
Hyperparameter tuning
External dependency
Local refinement
edit label
External dependency
edit label
Efficiency Hyperparameter tuning
Bug fix Composition
Architectural change Efficiency
Refactor
Bug fix
Architectural change
Pruning
Pruning
Refactor 6 × 10
−1
10
0
10
enrichment ratio vs all edits
0
enrichment ratio vs all edits
(a) ALE
(b) Math
Figure 9: Best-so-far enrichment of edit labels by domain. Enrichment of each label among bestso-far updates relative to the all-edits base rate, split by domain. The qualitative signal, a small set of categories (notably Efficiency, External dependency, and Hyperparameter tuning) overrepresented on successful intermediate steps, is consistent with the aggregate view in Figure 4, with domain-specific shifts in magnitude. Across the nine categories we observed substantial overall agreement: macro Cohen’s κ = 0.77, mean Jaccard = 0.86, micro-F1 = 0.90, and exact-match accuracy = 74.5%. B.5
Curated examples per edit type
To make the taxonomy labels of §5.1 concrete, we illustrate each of the nine categories with one curated example diff drawn from the labeled corpus. Examples are selected for visual clarity rather than for the highest score delta; some carry more than one label (e.g., Pruning, Composition, Bug fix, Efficiency, and External dependency are each accompanied by other labels in the cleanest paper-ready example), which is itself part of the empirical story analyzed in Appendix B.3: most edits in the corpus are multi-label. Each example below lists run identity, iteration, label set, and score delta; full unified diffs and metadata are bundled with the dataset release. Hyperparameter tuning. Single cooling-rate change. openevolve_native / heilbronn_triangle, iter 40, labels {hyperparameter_tuning}, ∆s = +0.0153. The cleanest possible single-knob example: one numeric literal changes; the surrounding algorithm is unchanged. @@ -74,7 +74,7 @@
20
Hyperparameter tuning
Hyperparameter tuning
Bug fix
Refactor
Local refinement
External dependency
edit label
Composition
edit label
Efficiency
Composition
Efficiency
Architectural change
Local refinement
External dependency
Architectural change
Pruning
Bug fix
Refactor
Pruning 6 × 10
−1
10
0
10
−1
10
enrichment ratio vs all edits
0
enrichment ratio vs all edits
(a) ALE
(b) Math
Figure 10: Final-best-lineage enrichment by domain. Robustness check for Figure 9, restricted to edits that lie on the lineage of each run’s final best program. The enriched categories overlap heavily with the best-so-far view in both domains, with Efficiency and Hyperparameter tuning retaining their overrepresentation. 0.6
0.5
0.5
share of edits
share of edits
0.4 0.4 0.3 0.2
0.2 0.1
0.1 0.0
0.3
0
1
2
3
0.0
4+
labels assigned to edit
0
1
2
3
4+
labels assigned to edit
(a) ALE
(b) Math
Figure 11: Distribution of labels per edit, by domain. Most edits in both domains are multilabel: 52.4% of edits aggregate-wide carry exactly two labels and only 32.4% are single-label. The categories of Figure 4 should therefore be read as overlapping rather than mutually exclusive modes.
+
num_restarts = 25 steps_per_run = 200000 T0 = 0.12 cooling_rate = 0.99992 cooling_rate = 0.99993 best_min = 0.0 best_points = None
# more restarts to escape local minima # longer runs for better convergence # higher initial temperature # slightly slower cooling # slightly slower cooling (compensate more steps)
Pruning. Delete final global-shake phase. openevolve_native / heilbronn_triangle, iter 38, labels {hyperparameter_tuning, pruning}, ∆s = +0.0670. A whole final optimization phase is removed (not commented out or renamed); the diff also retunes several literals, so this is a pruning-plus-tuning compound example. @@ -71,10 +71,10 @@ return np.min(area)
+ + + +
# Simulated Annealing parameters num_restarts = 35 # increased restarts to escape local minima steps_per_run = 250000 # longer runs for better convergence T0 = 0.15 # higher initial temperature for more exploration cooling_rate = 0.99992 # slightly slower cooling num_restarts = 25 # balanced restarts and run length steps_per_run = 280000 # more steps for deeper exploration T0 = 0.12 # initial temperature cooling_rate = 0.99993 # slightly slower cooling (compensate more steps)
21
External dependency
20.0 17.5 15.0 12.5 10.0 7.5 5.0 2.5
Architectural change
edit label
Composition Local refinement Pruning Refactor Efficiency
odds ratio for positive score delta
Bug fix
Hyperparameter tuning
ve
vol
E pen
oX
PA
Ev
O
framework
ve
vol
GE
aE ink
Sh
Figure 12: Per-edit helpfulness by backend. Odds ratio for positive normalized score change broken down by the four evolutionary backends. Some categories (notably External dependency) are consistently positive across backends, while others vary in magnitude. openevolve_native contributes only 2 runs to this corpus, so its column should be interpreted with a wider implicit confidence band; we include it for completeness. Table 9: Working definitions of the nine edit categories. One curated example diff per category is given in Appendix B.5. Label
Definition
Hyperparameter tuning
Change to one or more numeric literals or configuration values; control flow and surrounding algorithm unchanged. Targeted edit within an existing routine; the routine’s role and structure are preserved. A core algorithmic block is replaced by a substantively different approach. A new component (operator, phase, branch) is added alongside an existing one; existing logic is retained. Same input–output behaviour reimplemented at lower asymptotic or constant cost (e.g. batching, vectorization, partial sorts, caching). Correction of a latent defect such as a missing guard, wrong sign, off-by-one, or mishandled sentinel. A code path, phase, or feature is removed; the remaining program is kept intact. Behaviour-preserving restructuring: renaming, reordering, extracting helpers, or moving declarations. Introduction or removal of an import or external library.
Local refinement Architectural change Composition Efficiency Bug fix Pruning Refactor External dependency
best_min = 0.0 best_points = None @@ -169,24 +169,6 @@ best_min = current_min_ref best_points = points_ref.copy() # Final global shake of best configuration to escape narrow local minima if best_points is not None: ... [truncated, see full diff file] ...
22
Architectural change. Brute-force candidate search replaced by closed-form selection. evox / ale_bench_ahc016, iter 28, labels {architectural_change, local_refinement}, ∆s = +1.37 × 107 . The program stops scanning many candidate graph sizes and switches to a derived closed-form rule. @@ -203,40 +203,49 @@ double epsilon_noise_rate; std::cin >> M_graphs >> epsilon_noise_rate; double best_score = -1e100; int best_N = 4; Strategy best_strat = Strategy::GED; // Evaluate GED strategies (N=4,5,6) for (int Ncand : {4,5,6}) { if (Ncand == 6 && M_graphs > 156) continue; if (Ncand == 5 && M_graphs > 34) continue; if (Ncand == 4 && M_graphs > 11) continue; double score = estimate_ged_score(Ncand, M_graphs, epsilon_noise_rate); if (score > best_score) { ... } } // Evaluate edge-count strategies (N=4..100) for (int Ncand = 4; Ncand <= 100; ++Ncand) { double score = estimate_edge_score(Ncand, M_graphs, epsilon_noise_rate); if (score > best_score) { ... } } + int N_for_GED_strat; + if (M_graphs <= 11) N_for_GED_strat = 4; ... [truncated, see full diff file] ...
Local refinement. Add boundary bonus inside the existing scoring heuristic. evox / ale_bench_ahc015, iter 94, labels {local_refinement}, ∆s = +4,295. Small targeted change to a heuristic formula; surrounding algorithm intact. @@ -283,6 +283,10 @@ if ((candy.c == 0 && min_target_c == 0) || (candy.c == GRID_SIZE - 1 && max_target_c == GRID_SIZE - 1)) { bonus_val += PER_CANDY_BONUS_FACTOR; + } + // Additional bonus for being at the boundary of the target column range + if (candy.c == min_target_c || candy.c == max_target_c) { + bonus_val += 0.5; } } }
Composition. Add swap mutation operator on top of existing plan search. evox / ale_bench_ahc026, iter 57, labels {hyperparameter_tuning, composition}, ∆s = +9,252. Main search intact; an additional mutation operator is layered on, alongside several constant retunings. @@ -18,11 +18,11 @@ // Constants for heuristic evaluation -const double HEURISTIC_EMPTY_STACK_BONUS_SCORE = 1500.0; +const double HEURISTIC_EMPTY_STACK_BONUS_SCORE = 1000.0; const double STACK_HEIGHT_PENALTY_FACTOR = 0.1; const int HEURISTIC_LOOKAHEAD_WINDOW = 5; -const double HEURISTIC_COVER_CRITICAL_PENALTY_PER_BOX_ABOVE = 4.0; -const double HEURISTIC_MIN_LABEL_IN_DEST_FACTOR = 0.03; +const double HEURISTIC_COVER_CRITICAL_PENALTY_PER_BOX_ABOVE = 5.0; +const double HEURISTIC_MIN_LABEL_IN_DEST_FACTOR = 0.05; @@ -473,7 +467,7 @@ double op_choice_rand = RGen.a_double(0.0, 1.0); if (op_choice_rand < 0.35 && N_CONST > 0) { + if (op_choice_rand < 0.25 && N_CONST > 0) { ... [truncated, see full diff file] ...
Bug fix. Check strategy success and INF_COST sentinel. evox / ale_bench_ahc046, iter 99, labels {bug_fix, hyperparameter_tuning}, ∆s = +56.2. Parent silently ignored a helper’s return value and invalid-cost sentinel; child explicitly guards against both failure modes. @@ -378,7 +378,7 @@ -const int GREEDY_REOPTIMIZE_SUBSET_SIZE = 40; // Balanced between exploration and speed +const int GREEDY_REOPTIMIZE_SUBSET_SIZE = 170; // Full scan for best strategy per segment @@ -798,12 +798,16 @@ current_sa_choices[k] = current_best_strategy_code_for_k;
23
+ +
+ + + +
SegmentExecResult final_segment_res_for_k_build; apply_combined_strategy(current_best_strategy_code_for_k, bool success_seg = apply_combined_strategy( current_best_strategy_code_for_k, player_pos_sim_build, target_P_k, greedy_grid_sim_build, final_segment_res_for_k_build, true); if (!success_seg || final_segment_res_for_k_build.turns == INF_COST) { possible_greedy = false; break; }
Efficiency. std::sort replaced by std::nth_element. evox / ale_bench_ahc027, iter 46, labels {efficiency, hyperparameter_tuning}, ∆s ≈ 9.2 × 1018 . Top-k selection semantics are preserved; full sort is replaced by a partial-ordering primitive. @@ -266,8 +266,6 @@ TMP_CELL_DIRT_INFOS_LIST_GLOBAL_BUFFER.push_back(...); } } std::sort(TMP_CELL_DIRT_INFOS_LIST_GLOBAL_BUFFER.begin(), TMP_CELL_DIRT_INFOS_LIST_GLOBAL_BUFFER.end()); @@ -275,6 +273,13 @@ // Stochastic: pick uniformly from the top 20% of cells int num_candidates = std::max(1, (int)TMP_CELL_DIRT_INFOS_LIST_GLOBAL_BUFFER.size() / 5); + // Use nth_element to partition: the first num_candidates elements are the largest + std::nth_element(TMP_CELL_DIRT_INFOS_LIST_GLOBAL_BUFFER.begin(), + TMP_CELL_DIRT_INFOS_LIST_GLOBAL_BUFFER.begin() + num_candidates, + TMP_CELL_DIRT_INFOS_LIST_GLOBAL_BUFFER.end(), + [](const CellDirtInfo& a, const CellDirtInfo& b) { + return a.weighted_dirt_contribution + > b.weighted_dirt_contribution; + }); @@ -600,7 +605,7 @@ const int POST_MAX_ATTEMPTS = 500; + const int POST_MAX_ATTEMPTS = 200;
External dependency. Introduce JAX and Optax optimization pipeline. shinkaevolve / first_autocorr_ineq, iter 94, labels {architectural_change, external_dependency}, ∆s = +0.991. The strongest paper example of external_dependency: the imports of jax and optax are unambiguous and visually prominent. Note that the same diff also constitutes a large architectural change, illustrating that this label often appears in combination with others. @@ -1,3 +1,127 @@ # EVOLVE-BLOCK-START -# Paste your original program here. Ensure the evolve block markers are present. +import jax +import jax.numpy as jnp +import optax +import numpy as np +from dataclasses import dataclass + +@dataclass +class Hyperparameters: + num_intervals: int = 600 + learning_rate: float = 0.005 + end_lr_factor: float = 1e-4 + num_steps: int = 40000 + warmup_steps: int = 2000 + +class AutocorrelationOptimizer: + def __init__(self, hypers: Hyperparameters): + self.hypers = hypers + self.domain_width = 0.5 + self.dx = self.domain_width / self.hypers.num_intervals ... [truncated, see full diff file] ...
Refactor. Move SimulationResult definition before use. shinkaevolve / ale_bench_ahc026, iter 68, labels {refactor}, ∆s = +9,248. The cleanest refactor diff in the corpus: a type definition is moved to a more appropriate place without changing the algorithm or introducing new logic. 24
@@ -137,7 +137,13 @@ } }; -// Forward declaration for helper functions +// Define SimulationResult before any function that uses it +struct SimulationResult { + long long energy_cost; + std::vector<std::pair<int, int>> ops_history; +}; + +// Forward declaration for helper functions (SimulationResult is now defined) std::pair<State, long long> simulate_up_to_k( const std::vector<std::vector<int>>& init, const std::vector<int>& plan, @@ -319,11 +325,6 @@ return run_simulation_from_intermediate_state( std::move(st), plan, 0, N, M, record_all); } -struct SimulationResult { long long energy_cost; std::vector<std::pair<int, int>> ops_history; -}; int main() {
B.6
Cycling: additional analyses
The body section §5.2 reports only the headline cycling result. This appendix covers (i) a finer three-way classifier on the unified diff, (ii) a model- and prompt-dependence breakdown of the tuning share, and (iii) a null result on post-breakthrough cycling that did not survive cross-run aggregation. Three-way classifier. Each parent–child edit is classified deterministically using three categories on the unified diff between pt and pt . Literal recycling: the added line is byte-identical to a line previously removed elsewhere in the lineage. Tuning recycling: the added line’s number-collapsed skeleton matches a previously-removed line, but the numeric values differ (coefficient churn). Trivial recycling: comment-only or whitespace-only changes. The body cycling rate aggregates all three; per-category rates are emitted by the classifier. Edit composition is model- and prompt-dependent. Restricting to code-changing lines, the median per-run share that the paired-skeleton classifier marks as tuning recycling is 8%, but the range is wide (2–44%). Holding the task fixed at ahc015, the tuning share varies sharply with the generator (Table 10). The diff-vs-no-diff axis is the strongest single predictor: the same model (deepseek-reasoner) drops from 20% to 2% tuning share when its diff-based generation is turned off. Table 10: Tuning share of code-changing lines on ahc015, holding task fixed. The diff-vs-no-diff axis dominates model identity. Generator
Tuning share
gflash claude-haiku-4-5 deepseek-reasoner (diff) deepseek-reasoner (no diff)
44% 26% 20% 2%
Negative result on post-breakthrough cycling. We initially hypothesized that cycling spikes immediately after a best-so-far event (a refractory period in which the search churns over surrounding code). The change in cycling rate in the 5 iterations after each best-so-far event has mean −0.005, median −0.021, and range [−0.39, +0.23] across 26 runs. Some runs show large positive spikes, others large drops. The hypothesis does not survive cross-run aggregation and we report it as a null. 25
B.7
Public-vs-private generalization on ALE
ALE-bench public scores are not the held-out judging metric. We re-score every ALE run’s public best-so-far chain on the private test set used by AtCoder (n=30 run/problem pairs across the four main backends, covering 10 ALE problems). Two of the four frameworks overfit on at least 30% of the problems they were scored on, and the same problem can flip generalization sign between frameworks (Table 11). On ahc024, openevolve found a +1,606 rating-point private gain (aligned), while shinkaevolve, on the same problem, lost 1,610 rating points despite a positive public score change. On ahc027, three of four frameworks (evox, gepa, openevolve) overfit; only shinkaevolve generalized. Per-framework counts are summarized in Table 12: evox aligned 0 of 8 scored runs, gepa 1 of 7, openevolve 4 of 9, shinkaevolve 4 of 6. The public best-so-far chain is therefore unreliable as a single-number summary of an ALE run; in our data, problem identity is a stronger predictor of overfit than framework identity. Table 11: Public vs. private generalization across frameworks on ALE. Each cell reports the change in AtCoder rating-point performance from seed to the final public-best program along the run’s public best-so-far chain, when re-scored on the held-out private test set. Bold cells flag overfitting (public ↑, private ↓); “—” indicates a single-event lineage, an unscorable seed, or a missing private metric. problem
evox
gepa
openevolve
shinka
ahc008 ahc011 ahc015 ahc016 ahc024 ahc025 ahc026 ahc027 ahc039 ahc046
+0 −3 +0 −1 +0 +0 +0 −45 +0 +0
+0 — +0 +5 — +0 +0 −489 +0 +0
+37 — +0 +2 +1,606 −16 +1,735 −69 +0 −1,995
+38 — +2,194 +7 −1,610 — +10 +30 −32 +0
Table 12: Per-framework counts of aligned vs. overfitting public→private trajectories on ALE. A run is aligned if both public and private scores improved from seed to final, mild overfit if private worsened by ≤ 200 rating points despite a public gain, and severe overfit if by more than 200 points; the remainder is no movement on private.
B.8
framework
runs scored
aligned
overfit (mild)
overfit (severe)
evox gepa openevolve shinka
8 7 9 6
0 1 4 4
3 0 2 1
0 1 1 1
A walk-through of one short-span cycle
To make the deterministic cycling classifier of §5.2 concrete, we trace one short-span cycle in a single openevolve_native / heilbronn_triangle run. At iteration i the parent contains a handtuned annealing schedule with cooling_rate = 0.99992; the child at i+1 rewrites the schedule, deletes the line, and replaces it with cooling_rate = 0.99988. By iteration i+5 the search has produced a child whose diff against its own parent re-adds the byte-identical line cooling_rate = 0.99992, classified as literal recycling by the paired-skeleton classifier (the added line matches a previously-removed line in the lineage exactly, including the trailing comment). The same constant is deleted again at i+9 and re-introduced at i+14, and so on; the cycle is short-span (median across all classified runs: 5 iterations between deletion and re-introduction) and accumulates over the run, contributing to the monotonic per-iteration cycling-rate growth reported in §5.2. 26
B.9
Bayesian optimization baseline
This section documents the BO baseline used for the tuning-gap analysis in §5.4. The goal is to ⋆ estimate fBO (s0 ) for a fixed seed structure s0 by tuning only its embedded numeric constants, with no further structural search. The pipeline has three stages: (i) a single LLM call that identifies tunable knobs and proposes intervals, (ii) a deterministic rewrite that exposes those knobs as a top-level parameter block, and (iii) a short gp_minimize run that calls the original evaluator harness. Knob identification (one LLM call). We send the program source to deepseek-reasoner via an OpenAI-compatible chat endpoint and ask for a JSON list of candidate knobs. There is no agentic loop and no retry: a single structured-output call returns a list of objects with fields name, source_literal, context_line, default, low, high, scale (linear or log), kind (int or float), and a one-sentence rationale. The system prompt restricts candidates to solver tolerances, iteration/sample budgets, step sizes and learning rates, soft penalty/reward weights, cooling rates, and threshold dispatches; problem constants (population sizes √ that the evaluator reads, grid dimensions, the n in n-circle packing), mathematical identities ( 2, π), array shapes, and boolean flags are explicitly excluded. Range guidelines are calibrated by parameter family (log-scale spans of 102 for step sizes and weights, 103 for tolerances; linear [0, 1] for probabilities and acceptance ratios; etc.), with a hard rule that any range with high/low ≥ 100 uses a log scale. The prompt caps the proposal at 8 knobs, since BO budget scales poorly past that, and instructs the model to skip any literal that appears multiple times in the source (ambiguous replacement target). Validation and rewrite. Each returned spec is validated: the source_literal must appear byteidentically in the source, and the context_line must match a line in the file. For Python targets, a PARAMS = {...} dict is injected at the top of the file (after shebang/encoding/docstring) and each accepted literal is replaced in its context line by PARAMS["name"]. For C++ targets, a block of #define _BO_NAME value macros is inserted after the last #include and the literal is replaced by the corresponding macro. Replacement uses a regex that excludes longer-number neighbors (so 0.1 does not match inside 0.123). Specs whose literal can be found in the source but not in the proposed context line are dropped without retry; the BO then runs over the surviving knobs only, so the effective per-target knob count (median 6 in the runs reported in §5.4) is generally smaller than the LLM’s nominal proposal. BO loop. Tuning runs use skopt.gp_minimize (scikit-optimize) with 24 evaluator calls per target: 8 random initial points followed by 16 BO acquisitions over a Gaussian-process surrogate. Each call substitutes the current parameter vector into the rewritten source (regenerating the PARAMS dict or rewriting the #define block) and invokes the same evaluator harness used during the original ⋆ ⋆ evolutionary run, so fBO (s0 ) and fevo are directly comparable. Reported numbers in §5.4 (Table 4) are best-of-24 minus the seed program’s score f (p0 ). Knob-identification prompt. The system prompt sent to deepseek-reasoner is reproduced verbatim below; the user message is just the program source wrapped in a fenced block tagged with the language. You are an expert at identifying tunable hyperparameters in heuristic optimisation code. Your task: read a program and return a JSON list of numeric constants that are good candidates for Bayesian-optimisation tuning, WITH SENSIBLE RANGES. GOOD candidates: - Solver tolerances / convergence thresholds (e.g. 1e-6, 0.001) - Iteration / restart / sample budgets (e.g. 1000, 50, 10) - Step sizes, perturbation scales, learning rates - Soft penalty / reward weights (e.g. 0.5, 2.0) - Cooling / annealing rates (e.g. 0.95, 0.99) - Threshold dispatches (e.g. ‘if depth < 30: return X else return Y‘) BAD candidates (do NOT include):
27
- Problem constants (n=26, n_circles=26, dim=2, GRID_SIZE=10) - Mathematical identities (sqrt(2), pi, e) - Array shapes / index bounds - Evaluator-facing constants (timeout values that the harness reads) - Constants inside identities the model should preserve (vertices, basis vectors) - True boolean flags (use=True) For EACH selected knob, return: - name : a unique snake_case identifier (e.g. "step_size") - source_literal : the EXACT numeric literal as it appears in the source (e.g. "0.1", "1e-6", "0.95"). Must be byte-identical. - context_line : the line of code where the literal appears (verbatim, used as a disambiguator for replacement) - default : the default value (= source_literal as a number) - low / high : interval bounds for BO (see RANGE GUIDELINES) - scale : "linear" or "log" (see RANGE GUIDELINES) - kind : "int" or "float" - rationale : one short sentence RANGE GUIDELINES -- cover orders of magnitude when the parameter family warrants it. Don’t propose timid +-20% intervals: BO can only discover what the search space contains. Use these defaults: parameter family -------------------------learning rate / step size temperature / amplitude tolerance / threshold penalty / reward weight cooling factor (0<x<1) acceptance prob / sigmoid restart count / pop size iteration budget sample count small-int categorical
scale -----log log log log linear linear log log log linear
range relative to default --------------------------------[default/100, default*100] [default/100, default*100] [default/1000, default*1000] [default/100, default*100] [0.5, 0.99999] (always wide) [0.0, 1.0] [max(1, default/10), default*10] [default/5, default*5] [max(1, default/10), default*10] [1, max(default*5, 20)]
If a parameter doesn’t fit a family above, default to: - log scale if default is a positive non-integer < 1 or > 100 - linear scale otherwise - range that spans at least one order of magnitude (high/low >= 10) HARD RULES: - If high/low >= 100, scale MUST be "log" (BO converges much faster on log-scale priors over wide intervals). - Cooling factors and probabilities stay within their natural [0, 1] range even if that constrains low/high. - For integers: low = max(1, floor(low_proposed)); high = ceil(high_proposed). CONSERVATIVE RULES: - Pick at most 8 knobs. Fewer is fine -- quality over quantity. - If a literal appears multiple times in the source, skip it (ambiguous). - If unsure whether a constant is tunable, skip it. Return ONLY a JSON object of the form: {"hparams": [ { ...one knob... }, ... ]} No commentary, no markdown fences.
28