Humanize MLE Agents: Multi-Agent Session Alternation Beyond Single-Agent Limits

Preprint

H UMANIZE MLE AGENTS : M ULTI -AGENT S ESSION A LTERNATION B EYOND S INGLE -AGENT L IMITS Changye Li1 , Zijian Zhang1,2 , Muheng Li1 , Yiduo Pang3 , Tingjun Wu1 Menghan Li1 , Yitong Liu1 , Hongkai Guo1 , Mengchi Chen1 , Chenyu He3 Kaisheng Ma1 , Wei Xu1 , Song Han2,4 , Sihao Liu2 , Ligeng Zhu2

arXiv:2610.13494v1 [cs.AI] 9 Oct 2026

1

Tsinghua University , 2 NVIDIA , 3 Zhejiang University, 4 MIT

A BSTRACT Existing LLM-based agent approaches for long-horizon machine learning engineering (MLE) often use context engineering, implemented through custom harnesses that manage tools, skills, and accumulated experience. However, we find that as single-session execution lengthens, the agent’s reasoning intensity declines alongside slowing progress, summarized as long-horizon deceleration, and the benefits of these custom harnesses vanish as the underlying model improves, summarized as harness staleness. These findings suggest that their benefits may not persist across versions, which motivates a workflow-level approach in which agents build on one another’s work across sessions. We model these alternating agent sessions as a context-reset semi-Markov decision process (CR-SMDP) and establish sufficient conditions for alternation to exceed single-agent reference limits, derive an explicit experiment-budget bound, and characterize when session caps preserve this advantage. Guided by this analysis, we introduce Humanize MLE Agents (HMA), which alternates two agents on a shared workspace, limits each session to at most k experiments and resets the context between sessions, without additional task-specific harness or prior knowledge supplements. Across 75 Kaggle tasks in MLE-bench, HMA reaches a state-of-the-art any-medal rate of 78.2%, exceeding the strongest open-source baseline by 8.0 points with a quarter of its time budget. Moreover, among six configurations spanning five heterogeneous agent pairs, HMA improves any-medal rate over the constituent baselines by 6.4 points on average, demonstrating cross-pair generality. A supplementary case on kernel optimization reaches 878 cycles, within 1.6% of the community leaderboard leader, demonstrating applicability beyond MLE. Code: https://github.com/humanfia/hma

1

I NTRODUCTION

LLM-based agents tackle machine learning engineering (MLE) through repeated experiments: writing training code, running it, and evaluating candidate solutions (Chan et al., 2025; Toledo et al., 2025; Karpathy, 2026). For long-horizon tasks, producing a working solution is only the beginning: the agent must keep finding ways to improve it. As further gains become harder to find, the agent is expected to spend more tokens reasoning about the limitations of the current solution and deciding the next action. In our experiment across six models, however, mean output tokens per response—a proxy for reasoning intensity—decline over time as score gains diminish (Figure 2). We refer to this phenomenon as long-horizon deceleration and analyze its implications for prior agent design. Existing approaches often seek to sustain improvement through context engineering, supplying Markdown instructions, skills, and accumulated knowledge to guide an agent’s next steps (Zhu et al., 2026; Du et al., 2026; Karpathy, 2026; Rajasekaran et al., 2025). However, guidance that helps one model need not help its successor. For example, OpenAI recommends rewriting AGENTS.md, skills, and task prompts when upgrading models (Provencher, 2026), while Anthropic advises removing instructions for checks that newer models can already perform on their own (Anthropic, 2026a). As 1

Preprint

(A) Prior Failure Phenomena 1. Long-horizon Deceleration

ℐi: Initiation Set πi: Agent Policy βi: Termination Rule

Single-agent goal run

X0

X1

X2

(B) HMA: Alternative Optimization with Fixed k-Interruption 1. System Overview: Agent Session as Option oi = (ℐi, πi, βi) o B1

Overall policy μ : oA1 → oB1 → ⋯

…

o A2

B

∇A

Declining Response Length and Score 100

Switch agent + reset context

Option A 0 Time

2

4

6

Mean output tokens per response

Score

A

Option B Agent B

o A1

E

E: Reserved Candidate; ci: Agent Context; W: Shared Workspace; ℰ : All Candidates {Ei}Ni=1 w/ agent harness

Final Result E ̂

Review(ℰ, E−1, tend ) At budget end

Workflow design generalizes across model generations.

2. Theoretical View: Context-Reset Semi-MDP Formulation

Helpful

Option A

No gain / harmful

Model Capability

Weaker Model

Stronger Model

Harness gains vanish as models improve.

o B2

Task state Xi = (Wi, Ei, ci)

2. Harness Staleness Native agent

Archives all evaluated candidates ℰt

Agent A

Reasoning effort declines as research proceeds.

Score

∇B

Interrupt after k steps

50 0

Xi

Humanize Framework

X0

X1

…

Option B

Xk+1

Xk

…

Alternating agents: Continued exploration Single Agent: Rapid Convergence

oA1

… Early endpoint

…

X2k

…

XN

Beyond single-model endpoint

New State for A

oB1

Repeat to budget N

oAN

…

oBN

Complementary models can open new search states.

Figure 1: HMA overview. (A) Declining output tokens per response accompany diminishing score gains, indicating decelerating single-agent exploration; harness gains can vanish or reverse as models improve. (B) HMA alternates native agents after at most k experiments, resetting context while preserving shared work and candidates for final review. The motor analogy illustrates options A and b The CR-SMDP view captures B alternately energizing shared state Xi toward the final result E. how complementary options can extend exploration beyond single-agent limits.

model capabilities improve, guidance that once compensated for their limitations can thus become redundant or counterproductive. We observe a related pattern in our comparison of three custom MLE harnesses—systems that manage a model’s tools, context, and execution loop (Zhu et al., 2026; Du et al., 2026; Zhao et al., 2026). Nevertheless, our comparison shows that the relative benefits of specialized harnesses diminish or reverse as the base model improves, even under longer execution budgets (Table 1). We refer to this decline in relative performance as harness staleness, which indicates the limitations of custom harness design. These limitations motivate workflow-level coordination across agent sessions, each defined as an agent’s execution from context initialization to termination. In MLE, code, trained models, and evaluated candidates persist in a shared workspace, allowing successive agents to build on prior work (Zhao et al., 2026; Young, 2025). This persistence enables complementary roles: the agent that discovers a useful candidate need not be the one best suited to refine it, and subsequent changes can provide a new starting point when the first agent returns. We therefore ask: Can structured alternation convert complementary agent capabilities into a consistent advantage over the best single agent under a matched budget? To address this question, we represent alternating agent sessions as a context-reset semi-Markov decision process (CR-SMDP). We establish sufficient conditions and derive experiment-budget bounds under which alternation exceeds single-agent reference limits, and identify when fixed-k interruption retains this advantage (Sections 3.1–3.3). Building on these results, we introduce Humanize MLE Agents (HMA), which coordinates two native agents—complete model—harness pairs using official harnesses—through a shared workspace (Figure 1). Each session is capped at k experiments; after each handoff, the other agent continues with the shared work and a reset context. The workflow leaves each agent’s tools, context management, and execution loop unchanged (Bolin, 2026; Anthropic, 2026c), and provides no task-specific knowledge at initialization. Because coordination depends on completed experiments and persistent task artifacts, application to other iterative optimization tasks is a natural question. In summary, our contributions are: 2

Preprint

• We identify single-agent limitations and show how HMA sustains exploration. We characterize long-horizon deceleration across six models and harness staleness across three custom MLE harnesses in Sections 2. In contrast, from the early to the late phase of execution, HMA retains 86.4% of its mean output tokens per response, compared with 60.7% for matched single-agent baselines, indicating more persistent exploration across sessions. • Alternation and workflow design extend single-agent limits. Our CR-SMDP analysis establishes sufficient conditions and experiment-budget bounds for exceeding single-agent reference limits (Theorem 3.3), and characterizes when fixed-k interruption preserves this advantage Theorem 3.5). Algorithm 1 instantiates these principles with context reset, alternating agents, and final review of candidates (Sections 3.1–3.4). • HMA achieves state-of-the-art any-medal performance with cross-pair generality. On MLE-bench, HMA achieves a state-of-the-art any-medal rate of 78.2%, exceeding the strongest published open-source workflow by 8.0 points. Across the six heterogeneous agent-pair configurations, HMA improves the any-medal rate over the constituent baselines by 6.4 points on average, with gains in every configuration (Sections 4.2). We also explore applicability beyond MLE-bench through a supplementary 48-hour kernel-optimization case study (Section 4.5).

M OTIVATION : L IMITS OF S INGLE -AGENT E XPLORATION Mean output tokens per response

2

1400

GPT-5.6-sol Claude Opus 5 DeepSeek V4 Flash Kimi K3 GLM-5.3 DeepSeek V4.1 Flash Avg.

Table 1: Specialized harnesses become less effective as the backbone improves. We re1000 port medal rates on the 16-task MLE-bench subset, compared on DeepSeek-V4-Flash with 800 DeepSeek-V4.1-Flash. The DeepSeek Harness 600 row is the native baseline. Bold and underlined 400 entries indicate the largest and second-largest rates within each model column. With the weaker 200 backbone, some specialized harnesses match or 0 0 1 2 3 4 5 6 exceed the native baseline, whereas the stronger Active runtime (h) backbone consistently favors the native harness. This gap persists despite the longer execution budFigure 2: Single-session exploration deceler- gets of the specialized workflows, showing the ates over time. Each curve reports mean out- empirical pattern we name harness staleness. put tokens per recorded model response for one Time DeepSeek-V4-Flash DeepSeek-V4.1-Flash /goal configuration, pooling responses across Harness (h) Medals (%) Medals (%) 75 tasks and repeats. Across configurations, DeepSeek Harness 6 16.7 77.1 24 25.0 +8.3 68.8 −8.3 mean response length falls by 29.8%–45.0% ScienceFlow 24 6.3 −10.4 37.5 −39.6 from hours 0–2 to hours 3–5, indicating slower ML-Master 2.0 MLEvolve 12 18.8 +2.1 50.0 −27.1 single-session exploration over time. 1200

In this section, we investigate two limitations of single-agent MLE workflows: diminishing exploration over extended execution and limited transfer of harness improvements across model versions. These observations motivate HMA, which coordinates alternating agents from different providers to extend exploration beyond the capabilities of an individual model–harness system. Long-horizon deceleration. We examine six single-session configurations under /goal mode. Figure 2 reports mean output tokens per response in a trailing 30-minute window, used as a proxy for the agent’s reasoning intensity. Across configurations, response length decreases by 29.8%–45.0% from hours 0–2 to hours 3–5. Together with the rapid saturation of prefix gains in Figure 3, this reveals diminishing returns from continued single-agent exploration; detailed measurement definitions and supporting statistics are given in Appendix D.1. Harness staleness. To counter diminishing single-agent exploration, existing harnesses introduce planning, review, and context engineering (Young, 2025; Rajasekaran et al., 2025). With taskspecific cold-start knowledge disabled for fairness, our comparison of recent MLE harnesses (Zhu 3

Preprint

et al., 2026; Du et al., 2026; Zhao et al., 2026) reveals harness staleness: their benefits relative to the official harness diminish as the backbone improves (Table 1). The stronger performance of the official harness is consistent with a possible benefit from model–harness alignment acquired during training. This motivates treating each agent as an integrated model–harness system and studying how agents from different providers collaborate through workflow-level coordination. We present detailed configurations and results in Appendix D.2.

3

M ETHOD

In this section, we formulate Humanize MLE Agents (HMA) as a context-reset semi-MDP (Section 3.1), derive conditions and experiment budgets for alternation to exceed single-agent reference limits (Section 3.2), analyze fixed-k interruption and empirical cap selection (Section 3.3), and present the complete HMA workflow (Section 3.4). 3.1

HMA P RELIMINARIES

Workflow policy and task state. A multi-agent workflow that alternates between agent sessions is governed by a fixed policy over options µ (Sutton et al., 1999). Without loss of generality, we consider two agents, A and B, working on a shared task. The state x = (W, E, c(u) ) comprises the shared workspace W , retained result E, and context c(u) of the active agent u ∈ {A, B}. Session i of (u) (u) (u) (u) the workflow, executed by agent u, is modeled as an option oi = (Ii , πi , βi ), with initiation (u) (u) (u) (u) set Ii := {x : µ(oi | x) > 0}, internal policy πi , and termination rule βi . Termination creates an option boundary, where µ selects the next option. Task initialization and result quality. For a general MLE task, N is the total experiment budget of the workflow; each primitive step completes one experiment and updates (W, E) within an option. The initial state x0 = (W0 , E0 , ∅) ∼ ν0 is drawn from the task-state distribution ν0 , with workspace W0 , empty result E0 , and empty context. Let En be the result retained after n experiments, with normalized quality q(x) := q(E) ∈ [0, 1]. Expected quality and target-success probability are Perf µ (N ) := Eµ,ν0 q(EN ) and pµ (N ; τ ) := Prµ,ν0 (q(EN ) ≥ τ ) for τ ∈ [0, 1]. (u )

(u )

Workflow trajectory, rewards, and values. The trajectory is (x0 , o0 0 , x1 , o1 1 , . . .), where (u ) session i starts at xi and selects oi i ∼ µ(· | xi ) with ui ∈ {A, B}. The option kernel is P (u) (x, x′ ) := Pr(xi+1 = x′ | xi = x, ui = u). With a discount factor γ = 1 and reward rn+1 := q(En+1 ) − q(En ). Let r̄(u) (x) be the expected option reward from x. Values V µ (x) and (u) (u) Qµ (x, oi ) are expected future quality increments, with Qµ executing oi first. The semi-MDP Bellman recursion (Bradtke & Duff, 1994; Dietterich, 2000) gives X X (u) (u) (u) Qµ (x, oi ) = r̄(u) (x) + P (u) (x, x′ )V µ (x′ ), V µ (x) = µ(oi | x)Qµ (x, oi ). (1) ′ x

u

Context-Reset SMDP. The HMA workflow is formulated as a context-reset SMDP (CR-SMDP), whose option boundaries preserve (W, E) and reset context to ∅. For integer cap k ≥ 1, remaining (u) experiment budget t, and natural duration Ti (xi ) measured in experiments, option i completes (u) (u) di = min{Ti (xi ), k, t} experiments before natural termination, fixed-k interruption, or budget (u) (u) exhaustion. If Ti (xi ) > k almost surely and t ≥ k, then βi (ℓ) = 1{ℓ≥k} and di = k, where ℓ counts completed experiments within the option. Further formulation details and early natural termination analysis appear in Appendix A.1. 3.2

A LTERNATION B EYOND S INGLE -AGENT L IMITS

For u ∈ {A, B}, let B(u) be a finite set of single-agent workflow variants of u with ordinary or specified stronger harnesses. Define the reference workflow family BA,B := B(A) ∪ B(B) . All workflows initialize from x0 ∼ ν0 and satisfy the following retention assumption. Assumption 3.1 (Retention of the best result). q(En+1 ) ≥ q(En ) for all n. 4

Preprint

We adopt the fixed-k termination rule from Section 3.1. For simplicity, assume the alternating policy µk = (A, B, u0 , k) starts with u0 ∈ arg maxu∈{A,B} Perf (A,B,u,k) (N ), where Ā = B and B̄ = A. (u)

With option kernels Pk , including context reset, define the cycle kernel and reference ceiling: (u )

(ū )

Mµk = Pk 0 Pk 0 ,

U∗ := max

lim Perf b (N ).

b∈BA,B N →∞

(2)

On each task, let Xref and Xµalt contain states whose shared components (W, E) are reachable by k the references and alternation, respectively. For τ > U∗ , the target set T := {x ∈ Xµalt \ Xref : k q(x) ≥ τ } contains high-quality states beyond reference reachability; the next assumption requires persistent cooperative access to them. Assumption 3.2 (Cooperative access and use). Fix µk and τ ∈ (U∗ , 1]. In cycle m, agents u and ū start at x2m and x2m+1 , respectively. There exist constants α, β ∈ (0, 1] such that, for every m with Prµk (q(x2m ) < τ ) > 0, Prµk (x2m+1 ∈ G | q(x2m ) < τ ) ≥ α,

G := {x : q(x) < τ,

X x′ ∈T

Pk (x, x′ ) ≥ β}. (ū)

(3)

Here α bounds u’s access to states useful to ū, and β bounds ū’s probability of reaching T . Theorem 3.3 (Joint alternation exceeds the reference limits). Under Assumptions 3.1 and 3.2, let a = αβ and F0 = Prµk (q(x0 ) < τ ). For finite k and integers m ≥ 0, Perf µk (2km) ≥ τ [1 − F0 (1 − a)m ], Uµk := lim Perf µk (N ) ≥ τ > U∗ . (4) N →∞

Proof sketch. With Fm = Prµk (q(x2m ) < τ ), persistent access and quality preservation give Fm+1 ≤ (1 − a)Fm . Iteration and q ≥ 0 imply the bound; monotone convergence gives the limit. Corollary 3.4 (Experiment budget). Under Theorem 3.3, with F0 = 1, fixed p0 ∈ (0, 1), and 0 < a ≪ 1, a sufficient budget for pµk (N ; τ ) ≥ p0 has the leading form  2k 1  Nsuff (k, p0 , τ ) = log 1 + O(a) . (5) a 1 − p0 Taking p0 = (U∗ + η)/τ , 0 < η < τ − U∗ guarantees mean quality U∗ + η. The budget form also applies to a single agent with a uniform conditional success probability of at least a > 0. Appendix A.2 gives the proofs, exact budgets, and startup analysis. 3.3

F IXED - K I NTERRUPTION AND E MPIRICAL C HOICE

In this section, we study when fixed-k interruption maintains the advantage in Theorem 3.3 and motivate short caps empirically. We use Natural-Termination Alternation (NTA), µ∞ := (A, B, u0 , ∞), (u) (u) as the reference without fixed-k interruption, with di = min{Ti (xi ), t}. Let xi,ℓ be the state after ℓ experiments within option i, with entry xi,0 = xi and final state xi,−1 := xi,d(ui ) before context reset. At a boundary, xi+1,0 = f (xi,−1 ), where f (W, E, c) := (W, E, ∅). We i

(u )

define Aµ∞ (xi,−1 ) := V µ∞ (xi+1,0 ) − Qµ∞ (xi,−1 , oi i ). Here Qµ∞ continues the current option, whereas V µ∞ starts the reset option; the interruption theorem (Sutton et al., 1999, Theorem 2) gives nondegradation when Aµ∞ (xi,−1 ) ≥ 0. Pi (u ) For the capped policy µk , define the interrupted option indices Jµk (N ) := {i ≥ 0 : j=0 dj j < (u )

(u )

i N, d = k < Ti i (xi,0 )}. The expected net gain and NTA margin are Aµk (N ) := i P Eµk i∈Jµ (N ) Aµ∞ (xi,−1 ) and ∆µ∞ (N ) := Perf µ∞ (N ) − U∗ . k

Theorem 3.5 (Expected advantage under fixed-k interruption). For the optimal initial agent u0 , a sufficient condition to exceed U∗ is ∆µ∞ (N ) + Aµk (N ) > 0 =⇒ Perf µk (N ) > U∗ . (6) Proof sketch. Telescoping the option-level Bellman recursion gives Perf µk (N ) − Perf µ∞ (N ) = Aµk (N ). Substitution proves the claim; see Appendix A.3. Corollary 3.6 (Upper bound on the required cap). Suppose ∆µ∞ (N ) > 0 and interruption losses decay geometrically with k, then some finite 1 ≤ k ≤ k0 satisfies Perf µk (N ) > U∗ . 5

Preprint

Observed gain retained (%)

Empirically selected cap range: k ≤ 5. Figure 3 compares six goal and five NTA configurations on n = 75 MLE-bench tasks (Chan et al., 2025). The prefix curves show that k = 5 preserves most observed gains with fewer within-option experiments, motivating candidate caps k ≤ 5. Detailed analysis are presented in Appendix A.3; statistical and empirical details are provided in Appendix B. 100 k=1 63.15%

75

k=3 93.35%

Algorithm 1 HMA with Fixed-k Interruption

k=5 97.83%

50 25 0

0

24

.76

54

.88

73

.77

100

Mean within-option experiments retained (%)

Figure 3: Short prefixes retain most observed gains. The horizontal axis reports the mean percentage of within-option experiments retained by the first k experiments, and the vertical axis reports the corresponding percentage of observed gains retained. Faint curves represent individual configurations, the dark curve averages gain retention equally across configurations, and guides mark k = 1, 3, 5. Across the eleven configurations, k = 5 retains 97.83% of observed gains with a mean within-option experiment retention of 73.77%, supporting candidate caps k ≤ 5. 3.4

Require: Workflow µk = (A, B, u0 , k); task distribution ν0 ; total time ttotal ; final review time trev . b Ensure: Final candidate E. 1: Stage 1: Workflow Initialization 2: x0 = (W0 , E0 , ∅) ∼ ν0 . 3: tend ← ttotal ; ts ← tend − trev . 4: i ← 0; u ← u0 ; E ← ∅. 5: Stage 2: Alternate Options 6: while t < ts do 7: ℓ ← 0; xi,0 ← xi . (u) 8: while βi (ℓ, t; k) = 0 do (u) 9: xi,ℓ+1 ← πi (xi,ℓ ). 10: Append (Ei,ℓ+1 , u) to E. 11: ℓ ← ℓ + 1. 12: end while 13: xi+1 ← f (xi,−1 ). 14: i ← i + 1; u ← ū. 15: end while 16: Stage 3: Final Review b ← E−1 . 17: E 18: if |E| ≥ 2 ∧ t < tend then b ← R EVIEW(ū−1 ) (E, E−1 , tend ). 19: E 20: end if b 21: return E.

HMA W ORKFLOW OVERVIEW

In this section, we describe the implementation of the HMA workflow, including alternating agent execution and final review, as summarized in Algorithm 1. Agent Execution. Algorithm 1 implements µk = (A, B, u0 , k) from x0 ∼ ν0 . Here t is elapsed time, tend = ttotal is the final deadline, and ts = tend − trev ends exploration. Agent u executes (u) (u) (u) πi until βi (ℓ, t; k) = 1{ℓ ≥ min{Ti (xi ), k} ∨ t ≥ ts } = 1. Then xi,−1 = xi,ℓ and xi+1 = f (xi,−1 ) reset context for agent ū. Final review. The archive E stores evaluated candidates and their authors in order; (E−1 , u−1 ) is b← the latest pair. We reserve the final trev for the other agent ū−1 to review the archive and select E (ū−1 ) R EVIEW (E, E−1 , tend ), with E−1 as the fallback. This cross-agent check seeks agreement on the best archived candidate without test-set scores, approximating the result-retention objective of Assumption 3.1 at final selection.

4

E XPERIMENT

4.1

E XPERIMENT S ETUP

Implementation details. We implement HMA on Humanize (Liu et al., 2026), following Algorithm 1 with GPT, Kimi, GLM, DeepSeek, and Claude agents, each using their native harness and max reasoning effort. For MLE-bench, each task run uses one NVIDIA A10 GPU and a six-hour budget: agents alternate under k = 5, reset context at each handoff, and reserve the final 15 minutes for final review. Exact model–harness mappings, hardware, baseline configurations, and framework details are given in Appendix C. 6

Preprint

Benchmark. Our main evaluation uses MLE-bench (Chan et al., 2025), comprising 75 Kaggle competitions requiring agents to develop machine learning solutions. We report the percentage of tasks earning any medal (All) or a gold medal (Gold), supplemented by complexity-specific and leaderboard-based metrics in Table 2. Test-set scores are unavailable to agents and used only for offline evaluation. Section 4.5 supplements this evaluation with an exploratory application to kernel optimization. 4.2

M AIN C OMPARISON

Table 2: HMA exceeds the stronger constituent agent in overall medal rate. All performance metrics are percentages. Lite, Medium, and High report medal rates by task complexity. Med+ measures the percentage outperforming the human leaderboard median; Mean Percentile averages leaderboard percentiles across tasks. Avg. of Single averages the two constituent six-hour goal baselines, and signed gains measure HMA’s improvement over this average in percentage points. HMA starts with the first listed model. Bold and underlined medal rates mark the best and secondbest results. Statistical details and available HMA standard deviations appear in Appendix C.3. Workflow

Time (h)

Medal Rate (%) Lite (22)

Medium (38)

High (15)

Overall Metrics (%) All (75)

Gold

Mean Percentile (%)

Med+

Single-Agent Baselines with Native Harnesses (/goal) Claude Opus 5 GPT-5.6-sol GLM-5.3 DeepSeek-V4.1-Flash Kimi K3 DeepSeek-V4-Flash

6 6 6 6 6 6

80.3 ± 1.5 78.1 ± 2.3 46.7 ± 0.0 72.4 ± 1.6 50.7 ± 0.8 85.3 ± 1.3 80.3 ± 1.5 69.3 ± 2.3 46.7 ± 6.7 68.0 ± 1.3 47.6 ± 2.5 81.3 ± 0.8 77.3 ± 0.0 57.0 ± 0.9 40.0 ± 0.0 59.6 ± 0.4 38.2 ± 2.2 75.1 ± 1.2 77.3 ± 2.6 61.4 ± 6.3 37.8 ± 2.2 61.3 ± 4.1 37.8 ± 0.9 74.2 ± 2.4 77.3 ± 2.6 45.6 ± 2.3 42.2 ± 2.2 54.2 ± 1.6 34.7 ± 1.3 73.3 ± 2.3 54.5 ± 6.9 24.6 ± 1.8 24.4 ± 5.9 33.3 ± 3.5 15.6 ± 1.6 50.2 ± 3.2

84.8 ± 0.6 81.3 ± 1.3 75.7 ± 0.9 74.7 ± 2.3 73.8 ± 1.6 51.8 ± 2.8

Open-Source Agent Baselines ScienceFlow DeepSeek-V4-Flash-Preview

MLEvolve Gemini-3.1-Pro-preview

ML-Master 2.0 DeepSeek-V3.2-Speciale

24

80.3 ± 1.5 74.6 ± 0.9 44.4 ± 2.2 70.2 ± 1.2

–

–

–

12

80.3 ± 1.5 64.0 ± 0.9 46.7 ± 0.0 65.3 ± 0.8 34.7 ± 0.0 76.0 ± 2.3

–

24

75.8 ± 1.5 50.9 ± 3.5 42.2 ± 2.2 56.4 ± 2.5 19.6 ± 0.9 63.1 ± 1.2

–

HMA (Ours) Claude Opus 5 ↔ GPT-5.6-sol Avg. of Single 6 80.3 ± 1.1 73.7 ± 1.6 46.7 ± 3.3 70.2 ± 1.0 49.1 ± 1.3 83.3 ± 0.8 HMA (ours) 6 84.8+4.5 80.7+7.0 62.2+15.6 78.2+8.0 53.8+4.7 90.7+7.3

83.0 ± 0.7 89.0+5.9

GPT-5.6-sol ↔ Claude Opus 5 Avg. of Single 6 80.3 ± 1.1 73.7 ± 1.6 46.7 ± 3.3 70.2 ± 1.0 49.1 ± 1.3 83.3 ± 0.8 HMA (ours) 6 86.4+6.1 76.3+2.6 46.7+0.0 73.3+3.1 50.7+1.6 85.3+2.0

83.0 ± 0.7 85.4+2.3

GPT-5.6-sol ↔ DeepSeek-V4.1-Flash Avg. of Single 6 78.8 ± 1.5 65.4 ± 3.4 42.2 ± 3.5 64.7 ± 2.1 42.7 ± 1.3 77.8 ± 1.2 HMA (ours) 6 81.8+3.0 76.3+10.9 46.7+4.5 72.0+7.3 50.7+8.0 85.3+7.6

78.0 ± 1.3 84.4+6.4

GPT-5.6-sol ↔ Kimi K3 Avg. of Single 6 HMA (ours) 6

78.8 ± 1.5 57.5 ± 1.6 44.4 ± 3.5 61.1 ± 1.0 41.1 ± 1.4 77.3 ± 1.2 81.8+3.0 71.1+13.6 53.3+8.9 70.7+9.6 50.7+9.6 85.3+8.0

77.6 ± 1.0 83.9+6.4

GPT-5.6-sol ↔ GLM-5.3 Avg. of Single 6 HMA (ours) 6

78.8 ± 0.8 63.2 ± 1.2 43.3 ± 3.3 63.8 ± 0.7 42.9 ± 1.7 78.2 ± 0.7 77.3−1.5 73.7+10.5 46.7+3.4 69.3+5.6 50.7+7.8 82.7+4.4

78.5 ± 0.8 81.5+2.9

DeepSeek-V4.1-Flash ↔ Kimi K3 Avg. of Single 6 77.3 ± 1.9 53.5 ± 3.4 40.0 ± 1.6 57.8 ± 2.2 36.2 ± 0.8 73.8 ± 1.6 HMA (ours) 6 77.3+0.0 63.2+9.6 40.0+0.0 62.7+4.9 38.7+2.4 80.0+6.2

74.3 ± 1.4 78.1+3.8

HMA outperforms single-agent and specialized workflow baselines. Table 2 compares HMA with single agents using native harnesses, evaluated in our experiments, and specialized workflows 7

Preprint

using results reported in their original papers (Zhu et al., 2026; Du et al., 2026; Zhao et al., 2026). Opus-first HMA achieves 78.2% overall medals, exceeding our strongest single-agent baseline at 72.4% and the highest published workflow baseline at 70.2%. HMA and our single-agent baselines share a six-hour budget, whereas the published workflows use 12–24 hours. HMA achieves these gains without task-specific knowledge bases or additional prompt engineering, while retaining the agents’ existing internal harnesses. HMA improves heterogeneous agent pairs across capability levels. Across five model pairs with constituent medal rates of 54.2%–72.4%, all six HMA configurations improve overall medal rate over the stronger constituent, and Gold, Med+, and mean percentile over constituent averages. For Opus–GPT, both starting orders retain the advantage, reaching 78.2% and 73.3% medals versus the stronger constituent’s 72.4%, indicating that the observed advantage is not tied to a single favorable initial agent. 4.3

E XPERIMENT A NALYSIS AND A BLATION S TUDY (b) Performance versus time

(a) Output per response over time 80

Single-agent goal mean HMA mean

1200

Medal rate (%)

Mean output tokens per response

1500

900

600

60

40

20

300

Single-agent goal mean HMA mean 0

0 0

1

2

3

4

5

0

6

1

2

3

4

5

6

Active runtime (h)

Active runtime (h)

Figure 4: HMA sustains response length and continued improvement. (a) Mean output tokens per response and (b) medal rate, both plotted against active runtime. Bold red and dashed blue curves report weighted means for HMA and its matched constituent goal references, respectively; each HMA configuration receives equal weight, and each goal reference is the equally weighted mean of its two constituent agents. Faint curves show individual configurations.

HMA sustains exploration and continued improvement. The response-level curve in Figure 4 (a) uses the same 30-minute trailing window. From hours 0–2 to hours 3–5, HMA retains 86.4% of its mean output tokens per response, versus 60.7% for matched goal references. In Figure 4 (b), the six HMA configurations reach 42.7%–60.0% medals within two hours and continue improving through six hours, contrasting with the single-agent deceleration in Section 2. Final review improves historical any-medal retention. Holding exploration fixed, final review changes 8.9% of selections and increases any-medal outcomes from 311 to 319, with no losses relative to the pre-review candidate. Among the 450 task runs, 330 produce at least one medal-winning exploration candidate; final selection retains a medal in 319 of these runs (96.7%), compared with 311 (94.2%) before final review. These results quantify how terminal review supports the best-result retention objective in Assumption 3.1. Additional analyses and ablations. Appendix E details the weighting and aggregation used to construct the mean curves in Figure 4, together with trajectory construction and task-level comparisons. Appendix F further examines starting-order effects, model time allocation, and partner participation (Appendix F.1); the interruption-parameter ablations(Appendix F.2); natural termination and the fixed-duration approximation (Appendix F.3); and final review’s effects on final selection and historical any-medal retention (Appendix F.4). 8

5

6 7

Gold: 3,971,366 Top 1: 3,438,191 1,123,915

Late options (linear scale) 40k

1M

34,234 35k 3.2

100k

4.4

5.6

Best goal: 63,991

30k 0

1

2

3

4

5

Active runtime (h)

1500

10

15

1000

500

0

6

1

GPT-5.6-sol

→ Opus → GPT

4

→ GPT

3

→ Opus

10M

(b) Output tokens per response

2

→ Opus

1

→ GPT

(a) Test performance Mean output tokens / response

Test MAE (log scale; lower is better)

Preprint

5

20

25 27

Experiment number

Opus 5

Figure 5: HMA GPT–Opus alternation sustains improvement on INGV. (a) Test MAE across 27 experiments, using a logarithmic axis with a linear inset. (b) Mean output tokens per response for each experiment. Blue and purple regions identify GPT and Opus; dotted lines mark handoffs, and horizontal lines show reference scores. 4.4

C ASE S TUDY

On INGV Volcanic Eruption Prediction, GPT and Opus improve the historical best MAE in five of six post-handoff options (Figure 5). Their final MAE of 34,234 is 46.5% below the best of six constituent goal runs. Mean response length also increases on four of five returns to a previously used model, illustrating continued refinement of shared task state alongside renewed output. Appendix G provides detailed measurements and additional cross-model cases. 4.5

E XPLORATORY A PPLICATION B EYOND MLE- BENCH (a) Best performance Mean output tokens / response

Best cycles (lower is better)

(b) Output tokens per response

147,734

100k 1k 910 900 890

878

878 0

12

24

36

8k

6k

4k

2k

0

48

1

Trunk runtime (h)

15

30

45

60

77

Experiment number GPT-6-astra

Opus 5.5

Figure 6: HMA Opus–GPT alternation on AOPT. (a) Best accepted cycle count over 48 hours, using a broken vertical axis (upper: logarithmic; lower: linear) to resolve later improvements. (b) Mean output tokens per response for each experiment with available transcripts; gaps indicate missing records. Blue and purple regions identify GPT and Opus; dotted lines mark option boundaries. The preceding case illustrates continued refinement within MLE. We next explore the workflow on Anthropic’s Original Performance Take-Home (AOPT) (Anthropic, 2026b), a kernel-optimization task measured in simulated execution cycles. Claude Opus 5.5 and GPT-6-astra alternate with a cap of three accepted submissions per session (cap 3). The selected 48-hour run reaches 878 cycles at 34 hours 33 minutes and retains this score in its final candidate. Figure 6 pairs the performance trajectory with response-length traces, following the presentation in Figure 5. For context, 878 cycles matches the 10th-ranked score on the Paradigm community leaderboard as of October 8, 2026 (Paradigm, 2026), just 14 cycles (1.62%) above the leading score of 864 cycles. The leaderboard uses a separate nine-seed judge, so this is a comparison of reported scores rather 9

Preprint

than a verified leaderboard placement. This single-run case provides preliminary evidence of the workflow’s applicability to kernel optimization beyond MLE-bench over a longer horizon.

5

R ELATED W ORK

MLE agents and harness-level systems. MLE-bench evaluates autonomous agents on Kaggle competitions (Chan et al., 2025). Recent MLE systems improve individual agents through search, reflection, memory, branch reuse, recoverable workspaces, or task-conditioned interfaces (Toledo et al., 2025; Chen et al., 2026; Du et al., 2026; Zhu et al., 2026; Zhao et al., 2026; Yu et al., 2026; Zhang et al., 2026; Ye et al., 2026). These methods modify how one model acts within its harness. HMA instead leaves each native model–harness pair unchanged and coordinates complete agents at the workflow level. Multi-agent and long-horizon workflows. Prior workflows sustain long-horizon execution through reusable experience, reflection, tree search, workflow search, communication, role assignment, checkpointing, or independent review (Shinn et al., 2023; Zhao et al., 2024; Zhou et al., 2024; Zhang et al., 2025; Wu et al., 2024; Qian et al., 2024; Li et al., 2025). Recent handoff studies examine candidate transfer, learned control exchange, and the quality cost of continuing another model’s trajectory (Luo et al., 2026; Li et al., 2026; Ganz et al., 2026). HMA studies fixed-cap alternation of complete native agents on one shared workspace, with a context reset at every handoff. Options and semi-MDPs. Semi-Markov decision processes model actions with variable duration, while options represent such actions through an internal policy and termination rule (Bradtke & Duff, 1994; Dietterich, 2000). The interruption theorem characterizes when terminating an option early preserves or improves continuation value (Sutton et al., 1999); subsequent work studies learned and off-policy termination (Bacon et al., 2017; Harutyunyan et al., 2018). HMA models each agent session as an option and applies this framework to fixed-cap alternation with context reset.

6

C ONCLUSION

HMA alternates native agents, resetting context while preserving shared task state. Our CR-SMDP analysis gives sufficient conditions and budget bounds for exceeding single-agent reference limits and conditions for preserving this advantage under fixed-cap interruption. Across six MLE-bench configurations, HMA outperforms the stronger constituent in medal rate under equal six-hour budgets, reaching 78.2% at best. A supplementary AOPT case illustrates application to kernel optimization over 48 hours; establishing comparative gains beyond MLE-bench remains future work. Together, these results motivate further study of workflow-level coordination for sustained exploration across iterative optimization tasks. ACKNOWLEDGMENTS We thank Zijian Huang, Chengxing, Jing Xiong, Zhengkai Lin, Boshi Zhang, Yan Hu, Dongyun Zou, and Chang Liu for their participation in the early exploration of this project and their contributions to workflow design and experiments. AI USE STATEMENT In this work, we used generative AI tools for language polishing, grammar checking, and generating portions of the code. The research ideas, methodology, experimental design, and interpretation of results were developed by the authors. We reviewed all AI-assisted text to ensure that it accurately conveyed our intended meaning. LLM-generated code was verified and tested for correctness by two authors. We take full responsibility for the final content of this work, including all text, claims, and artifacts produced with the assistance of generative AI. 10

Preprint

R EFERENCES Anthropic. Prompting Claude Opus 5. Claude Platform documentation, 2026a. URL https://platform.claude.com/docs/en/build-with-claude/ prompt-engineering/prompting-claude-opus-5. Accessed September 25, 2026. Anthropic. Anthropic’s Original Performance Take-Home. GitHub repository, 2026b. https://github.com/anthropics/original_performance_takehome. cessed October 8, 2026.

URL Ac-

Anthropic. How Claude Code works. Claude Code documentation, 2026c. URL https: //code.claude.com/docs/en/how-claude-code-works. Documentation version updated September 26, 2026. Accessed September 26, 2026. Pierre-Luc Bacon, Jean Harb, and Doina Precup. The Option-Critic architecture. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), February 2017. ISSN 2159-5399. doi: 10.1609/aaai.v31i1.10916. URL http://dx.doi.org/10.1609/aaai.v31i1.10916. Michael Bolin. Unrolling the Codex agent loop. OpenAI Engineering, January 2026. URL https://openai.com/index/unrolling-the-codex-agent-loop/. Published January 23, 2026. Accessed September 21, 2026. Steven Bradtke and Michael Duff. Reinforcement learning methods for continuous-time Markov decision problems. In G. Tesauro, D. Touretzky, and T. Leen (eds.), Advances in Neural Information Processing Systems, volume 7. MIT Press, 1994. URL https://proceedings.neurips.cc/paper_files/paper/1994/file/ 07871915a8107172b3b5dc15a6574ad3-Paper.pdf. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. ˛ MLE-bench: Evaluating machine learning agents on machine learning engineering, 2025. URL https://arxiv.org/abs/2410.07095. Jiefeng Chen, Bhavana Dalvi Mishra, Jaehyun Nam, Rui Meng, Tomas Pfister, and Jinsung Yoon. MARS: Modular agent with reflective search for automated AI research, 2026. URL https: //arxiv.org/abs/2602.02660. DeepSeek-AI. DeepSeek Harness: Everything is a plugin. deepseek-ai/deepseek-harness, 2026.

https://github.com/

T. G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13:227–303, November 2000. ISSN 1076-9757. doi: 10.1613/jair.639. URL http://dx.doi.org/10.1613/jair.639. Shangheng Du, Xiangchao Yan, Jinxin Shi, Zongsheng Cao, Shiyang Feng, Zichen Liang, Boyuan Sun, Tianshuo Peng, Yifan Zhou, Xin Li, Jie Zhou, Liang He, Bo Zhang, and Lei Bai. MLEvolve: A self-evolving framework for automated machine learning algorithm discovery, 2026. URL https://arxiv.org/abs/2606.06473. Roy Ganz, Mor Shpigel Nacson, Adi Kalyanpur, and Ron Litman. The handoff tax: Continuing nonnative trajectories in LLM agents, 2026. URL https://arxiv.org/abs/2608.24358. Aurélien Garivier and Olivier Cappé. The KL-UCB algorithm for bounded stochastic bandits and beyond. In Sham M. Kakade and Ulrike von Luxburg (eds.), Proceedings of the 24th Annual Conference on Learning Theory, volume 19 of Proceedings of Machine Learning Research, pp. 359–376, Budapest, Hungary, 09–11 Jun 2011. PMLR. URL https://proceedings.mlr. press/v19/garivier11a.html. Anna Harutyunyan, Peter Vrancx, Pierre-Luc Bacon, Doina Precup, and Ann Nowé. Learning with options that terminate off-policy. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018. ISSN 2159-5399. doi: 10.1609/aaai.v32i1.11740. URL http://dx.doi. org/10.1609/aaai.v32i1.11740. 11

Preprint

Andrej Karpathy. autoresearch: AI agents running research on single-GPU nanochat training automatically. GitHub, 2026. URL https://github.com/karpathy/autoresearch. Released March 2026. Accessed September 25, 2026. Jiayu Li, Enpei Zhang, Dawei Zhou, Elynn Chen, and Yujun Yan. Learning to hand off: Provably convergent workflow learning under interface constraints, 2026. URL https://arxiv.org/ abs/2605.19140. Yang Li, Siqi Ping, Xiyu Chen, Xiaojian Qi, Zigan Wang, Ye Luo, and Xiaowei Zhang. AgentGit: A version control framework for reliable and scalable LLM-powered multi-agent systems, 2025. URL https://arxiv.org/abs/2511.00628. Sihao Liu, Ligeng Zhu, Zijian Zhang, Dongyun Zou, Zhengyang Zhang, Changye Li, Song Bian, Song Han, and Tony Nowatzki. Humanize: Judgement engineering for agentic coding, 2026. URL https://arxiv.org/abs/2610.08900. Sichun Luo, Yi Huang, Guanzhi Deng, Haibo Wang, Haochen Luo, Lei Li, Zefa Hu, Junlan Feng, and Qi Liu. Relay, don’t route: Adaptive population handoff for cost-efficient LLM-driven evolution, 2026. URL https://arxiv.org/abs/2608.05651. Moonshot AI. Kimi Code CLI. GitHub, 2026. URL https://github.com/MoonshotAI/ kimi-code. Official open-source software. Accessed September 21, 2026. Paradigm. Anthropic Take-Home Challenge: Community Leaderboard. Paradigm Puzzles, 2026. URL https://www.paradigm.xyz/puzzles/anthropic-challenge. Accessed October 8, 2026. Eric Provencher. Rethinking skills and prompts for GPT-6 Astra. OpenAI Developers Blog, September 2026. URL https://developers.openai.com/blog/ rethinking-skills-and-prompts-for-gpt-6-astra. Accessed September 25, 2026. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative agents for software development. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15174–15186, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.810. URL https://aclanthology.org/2024.acl-long.810/. Prithvi Rajasekaran, Ethan Dixon, Carly Ryan, and Jeremy Hadfield. Effective context engineering for AI agents. Anthropic Engineering, September 2025. URL https://www.anthropic. com/engineering/effective-context-engineering-for-ai-agents. Published September 29, 2025. Accessed September 21, 2026. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 8634–8652. Curran Associates, Inc., 2023. doi: 10.52202/ 075280-0377. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/file/1b44b878bb782e6954cd888628510e90-Paper-Conference.pdf. Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(12):181–211, August 1999. ISSN 0004-3702. doi: 10.1016/s0004-3702(99)00052-1. URL http://dx.doi.org/10.1016/S0004-3702(99)00052-1. Edan Toledo, Karen Hambardzumyan, Martin Josifoski, Rishi Hazra, Nicolas Baldwin, Alexis Audran-Reiss, Michael Kuchnik, Despoina Magka, Minqi Jiang, Alisia Maria Lupidi, Andrei Lupu, Roberta Raileanu, Kelvin Niu, Tatiana Shavrina, Jean-Christophe Gagnon-Audet, Michael Shvartsman, Shagun Sodhani, Alexander H. Miller, Abhishek Charnalia, Derek Dunfield, CaroleJean Wu, Pontus Stenetorp, Nicola Cancedda, Jakob Nicolaus Foerster, and Yoram Bachrach. AI research agents for machine learning: Search, exploration, and generalization in MLE-bench, 2025. URL https://arxiv.org/abs/2507.02554. 12

Preprint

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id= BAakY1hNKS. Zihao Ye, Yingyi Huang, Hongyi Jin, Bohan Hou, Junru Shao, Zhongming Yu, Jinqi Chen, Meghan Cowan, Shiyi Cao, Shanli Xing, Hanfeng Chen, Vinod Grover, Tianqi Chen, and Luis Ceze. CAKE: Compiler-agent co-design for frontier kernel evolution, 2026. URL https://arxiv. org/abs/2608.12629. Justin Young. Effective harnesses for long-running agents. Anthropic Engineering, November 2025. URL https://www.anthropic.com/engineering/ effective-harnesses-for-long-running-agents. Published November 26, 2025. Accessed September 21, 2026. Zhaochen Yu, Yingcheng Wu, Zhenfei Yin, Kaiyuan Chen, Zhe Zhao, Mengdi Wang, Shuicheng Yan, and Ling Yang. Recursive experiential-working memory evolution for long-horizon agent harnesses, 2026. URL https://arxiv.org/abs/2608.24876. Guibin Zhang, Leo Lu, Fangzhou Xie, Kang Zhu, Junhao Wang, Zhifei Xie, Zhaochen Yu, Zihang Liu, Zhongxiang Sun, Qiankun Li, Yue Liao, Heng Chang, Xiaobin Hu, Qibing Ren, Wangchunshu Zhou, Chuanrui Hu, Yafeng Deng, and Shuicheng Yan. JIT-Agent: Scaling harness intelligence via just-in-time harness evolution, 2026. URL https://arxiv.org/abs/2608. 25593. Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id= z5uVAKwmjf. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19632–19642, March 2024. ISSN 2159-5399. doi: 10.1609/aaai.v38i17.29936. URL http://dx.doi.org/10.1609/aaai.v38i17.29936. Mingming Zhao, Jiqian Dong, Kangping Xu, Zadid Hasan, Chengrui Fan, Shan Jiang, Shuai Mao, Yating Ling, Linyi Zou, Tailin Zhou, Yun Hin Chan, Wenkai Zhang, Zhanhong Zhou, Guowei Huang, Hongliang Li, Wenjing Cun, Zhitang Chen, Mingxuan Yuan, and Yanhui Geng. ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyond, 2026. URL https://arxiv.org/abs/2608.14354. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 62138–62160. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/zhou24r.html. Xinyu Zhu, Yuzhu Cai, Zexi Liu, Bingyang Zheng, Cheng Wang, Rui Ye, Yuzhi Zhang, Linfeng Zhang, Weinan E, Siheng Chen, and Yanfeng Wang. Toward ultra-long-horizon agentic science: Cognitive accumulation for machine learning engineering, 2026. URL https://arxiv.org/ abs/2601.10402.

13

Preprint

A PPENDIX C ONTENTS A Theoretical Details and Proofs

15

A.1 Model Details and Exact Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

A.2 Alternation Guarantee and Experiment Budget . . . . . . . . . . . . . . . . . . . . . . . . .

15

A.3 Interruption Guarantee and Sufficient Caps . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

B Prefix Retention and Empirical Cap Selection

19

B.1 Retention Statistics and Confidence Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

B.2 Configurations and Observed Prefix Results . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

C Experimental Setup and Reproducibility

21

C.1 Humanize and Execution Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

C.2 Agent and Baseline Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

C.3 Evaluation Protocol and Repeats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

D Additional Evidence for the Motivation

23

D.1 Declining Response Length in Single-Agent Runs . . . . . . . . . . . . . . . . . . . . . . .

23

D.2 Harness Staleness across Model Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23

E Additional Main-Comparison Analyses

25

F Ablation Studies

26

F.1

Starting-Order Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

F.2

Effects of the Fixed Interruption Cap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

F.3

Natural Termination before the Cap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

F.4

Final Review and Historical Any-Medal Retention . . . . . . . . . . . . . . . . . . . . . . .

28

G Case Studies of Heterogeneous Alternation

30

G.1 INGV Volcanic Eruption Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

G.2 Google QUEST Q&A Labeling: GPT–DeepSeek . . . . . . . . . . . . . . . . . . . . . . . .

30

G.3 PetFinder Pawpularity: GLM–Kimi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

G.4 Google Smartphone Decimeter: DeepSeek–Kimi . . . . . . . . . . . . . . . . . . . . . . . .

32

14

Preprint

A

T HEORETICAL D ETAILS AND P ROOFS

We organize the formal supplement in the same order as the Method section: Appendix A.1 defines the finite-budget CR-SMDP and exact performance curves; Appendix A.2 proves the cooperative alternation guarantee and budget bound; Appendix A.3 handles NTA, fixed-k interruption, and sufficient-cap conditions. A.1

M ODEL D ETAILS AND E XACT P ERFORMANCE

We supplement Section 3.1 with finite-budget definitions, the general option recursion, and exact performance expressions used in the alternation proof. Definition A.1 (State and finite budget). The state from Section 3.1 implicitly includes agent identity, option age, and any scheduling history needed for the Markov property (Dietterich, 2000, Sec(u ) tion 3.2). For session oi i at xi , let ni be the completed experiment count and ti the remaining budget. Then ni :=

i−1 X

dh ,

ti := N − ni ,

(u )

di = min{Ti i (xi ), k, ti }.

(7)

h=0 (u)

Here n0 = 0, and Ti (xi ) is the positive, almost-surely finite natural duration, which may depend on the entry state. Thus early natural termination and a final partial option are included; k = ∞ gives NTA. Values track the remaining budget, vanish when it is exhausted, and no option starts at ti = 0. Definition A.2 (Option rewards and transitions). For γ ∈ (0, 1] and the experiment rewards in Section 3.1, define "d −1 # i X r̄(u) (x) := E γ ℓ rni +ℓ+1 xi = x, ui = u . (8) ℓ=0 (u)

Let Φk,t (x′ , d | x) be the joint law of duration and next option-entry state, including context reset, P (u) ′ (u) (u) (u) and Pk,t (x, x′ ) := := Pk,∞ ; finite t d Φk,t (x , d | x). The complete-option kernel is Pk truncates the same option. Suppressing fixed k, t, the general semi-MDP recursion (Sutton et al., 1999; Bradtke & Duff, 1994) is X (u) Qµ (x, oi ) = r̄(u) (x) + γ d Φ(u) (x′ , d | x)V µ (x′ ). (9) d,x′

At γ = 1, this reduces to equation 1, and total reward is q(EN ) − q(E0 ). Proposition A.3 (Exact performance expressions). Let Pµ be the one-experiment kernel under µ on the augmented state, including option boundaries, and Sτ := {x : q(x) ≥ τ }. After workflow termination, retain the terminal state. Then pµ (N ; τ ) = ν0 PµN 1Sτ ,

Perf µ (N ) = ν0 PµN q,

(10)

where 1Sτ indicates target attainment. These expressions give the probability of reaching Sτ and the expected final quality, respectively, with Perf µ (N ) ≥ τ pµ (N ; τ ). Proof of Proposition A.3. The endpoint law is ν0 PµN ; applying it to 1Sτ and q gives equation 10. Taking expectations of q ≥ τ 1Sτ yields the inequality. A.2

A LTERNATION G UARANTEE AND E XPERIMENT B UDGET

We follow Section 3.2: cooperative access yields a failure recursion, which proves the alternation guarantee and the experiment budget; we then discuss startup effects. 15

Preprint

Definition A.4 (Reference ceiling and cycle budget). bounded quality and Assumption 3.1 give U∗ := max Ub , b∈BA,B

For the reference family in Section 3.2,

Ub := lim Perf b (N ) = Eb,ν0 N →∞

h

i lim q(EN ) .

N →∞

(11)

(u)

Fix µk and τ > U∗ . Write u for the first agent in each cycle: cycle m executes o2m followed by (ū) o2m+1 , with kernel Mµk . Before total-budget truncation, the experiments used by m cycles are P2m−1 (u ) Cµk ,m := i=0 min{Ti i (xi ), k}, so Cµk ,m ≤ 2km for finite k. Write Fm := Prµk (q(x2m ) < τ ), where Prµk and Eµk include the initial draw from ν0 . Lemma A.5 (Cooperative success and failure recursion). Under Assumptions 3.1 and 3.2, each cycle has conditional success probability at least a := αβ whenever Fm > 0, and Fm+1 ≤ (1 − a)Fm for all m ≥ 0. Proof of Lemma A.5. The u-option enters G with conditional probability at least α, and the ū-option reaches T ⊆ Sτ from each x ∈ G with probability at least β. By the Markov property, for Fm > 0, Prµk (x2m+2 ∈ Sτ | x2m ∈ / Sτ ) ≥ β Prµk (x2m+1 ∈ G | x2m ∈ / Sτ ) ≥ αβ = a.

(12)

Quality retention makes Sτ absorbing, so Fm+1 ≤ Fm (1 − a). If Fm = 0, it also gives Fm+1 = 0. Theorem A.6 (Joint alternation exceeds the reference limits; Theorem 3.3 restated). Under Assumptions 3.1 and 3.2, take a = αβ as a lower bound on conditional success per cycle, and F0 = Prµk (q(x0 ) < τ ). For finite k, Perf µk (2km) ≥ τ [1 − F0 (1 − a)m ],

Uµk := lim Perf µk (N ) ≥ τ > U∗ . N →∞

(13)

Proof of Theorem A.6. Iterating Lemma A.5 gives Fm ≤ F0 (1 − a)m . Since Cµk ,m ≤ 2km, quality retention and Proposition A.3 yield Perf µk (2km) ≥ τ pµk (2km; τ ) ≥ τ (1 − Fm ) ≥ τ [1 − F0 (1 − a)m ].

(14)

Bounded monotone convergence ensures that Uµk exists; letting m → ∞ gives Uµk ≥ τ > U∗ . The same proof permits early natural termination when Assumption 3.2 holds for the resulting kernels, since Cµk ,m ≤ 2km still holds. Corollary A.7 (Experiment budget; Corollary 3.4 restated). Under the theorem’s conditions, let F0 = 1 and fix p0 ∈ (0, 1). For 0 < a ≪ 1, a sufficient budget for pµk (N ; τ ) ≥ p0 has the leading form Nsuff (k, p0 , τ ) =

 2k 1  log 1 + O(a) . a 1 − p0

(15)

Proof of Corollary A.7. For 0 < a < 1, the bound pµk (2km; τ ) ≥ 1 − (1 − a)m reaches p0 once m ≥ log(1/(1 − p0 ))/[− log(1 − a)]. Hence the exact rounded sufficient budget is   log(1/(1 − p0 )) Nsuff (k, p0 , τ ) = 2k . (16) − log(1 − a) Using − log(1 − a) = a + O(a2 ) and a rounding error below 2k gives (2k/a) log(1/(1 − p0 )) + O(k), equivalent to equation 15 for fixed p0 . Finally, p0 = (U∗ + η)/τ , 0 < η < τ − U∗ , gives Perf µk (N ) ≥ U∗ + η by Proposition A.3. Startup and initial-agent effects. Startup denotes the early cycles before cooperative progress stabilizes. The initial agent can affect access to G and finite-budget quality. If the two starts later have similar conditional success rates, their subsequent budget scales 2k/a are similar, while their startup progress can still differ. 16

Preprint

A.3

I NTERRUPTION G UARANTEE AND S UFFICIENT C APS

We expand Section 3.3. Corollary A.8 extends the alternation guarantee to NTA. We define interruption advantages in Definition A.9 and compare the workflows through option-level Bellman recursions. Theorem A.10 compares fixed-k interruption with NTA; Corollaries A.11–A.12 give a finite candidate condition and the explicit cap bound underlying Corollary 3.6. Corollary A.8 (Natural termination and quality limits). For Natural-Termination Alternation (NTA), use µ∞ = (A, B, u0 , ∞). Under Assumptions 3.1 and 3.2 and almost-surely finite option durations, let Mτ := inf{m ≥ 0 : q(x2m ) ≥ τ }. Then Cµ∞ ,Mτ < ∞ almost surely and Uµ∞ ≥ τ > U∗ . If NTA and a finite cap satisfy access for every τ ∈ (U∗ , U † ), where U † ∈ (U∗ , 1] bounds both reachable quality spaces, then Uµk = Uµ∞ = U † . Proof of Corollary A.8. Applying equation 12 to NTA gives Prµ∞ (Mτ > m) ≤ F0 (1 − a)m → 0, with a = αβ. Thus Mτ and the sum of its option durations are almost surely finite. Monotone quality yields Uµ∞ ≥ τ . Combining this argument with Theorem A.6 for every τ ∈ (U∗ , U † ) and using the common upper bound gives equality for both limits. Definition A.9 (Interruption advantage and reference margin). Fix µk = (A, B, u0 , k) and N . The workflows µk and µ∞ = (A, B, u0 , ∞) share internal policies and boundary updates and differ only in option termination. Following Section 3.3, xi,0 = xi is option i’s entry, xi,−1 = xi,d(ui ) its i

endpoint before reset, and xi+1,0 = f (xi,−1 ) the next entry, with q(xi+1,0 ) = q(xi,−1 ). At an added cap interruption, NTA would continue from xi,−1 with the current context. Ending (u ) the option instead gives advantage Aµ∞ (xi,−1 ) := V µ∞ (xi+1,0 ) − Qµ∞ (xi,−1 , oi i ). Both values use NTA with the same remaining experiment budget; the next entry activates the other agent with empty context. The interruption contribution is zero at natural termination or budget exhaustion. The added interruptions are indexed by  i  X (u ) (u ) (u ) Jµk (N ) := i ≥ 0 : dj j < N, di i = k < Ti i (xi,0 ) .    

j=0

Thus natural termination takes precedence when it coincides with the cap. The expected net gain is P Aµk (N ) := Eµk i∈Jµ (N ) Aµ∞ (xi,−1 ), and the NTA margin is ∆µ∞ (N ) := Perf µ∞ (N ) − U∗ . k

Theorem A.10 (Expected advantage under fixed-k interruption; Theorem 3.5 restated). For the selected initial agent u0 , a sufficient condition to exceed U∗ is ∆µ∞ (N ) + Aµk (N ) > 0

=⇒

Perf µk (N ) > U∗ .

(17)

Proof of Theorem A.10. Let M be the number of options executed under µk within budget N , with xM,0 terminal, and let ιi := 1{i∈Jµk (N )} . For each i < M , the option-level Bellman recursion (Sutton et al., 1999, proof of Theorem 2) gives Eµk [q(xi+1,0 ) − q(xi,0 ) + V µ∞ (xi+1,0 ) | xi,0 ] − V µ∞ (xi,0 ) = Eµk [ιi Aµ∞ (xi,−1 ) | xi,0 ] .

(18)

Since V µ∞ (xM,0 ) = 0 and x0,0 = x0 ∼ ν0 , summing telescopes: Perf µk (N ) − Perf µ∞ (N ) = Eµk [q(xM,0 ) − q(x0,0 ) − V µ∞ (x0,0 )] X = Eµk Aµ∞ (xi,−1 ) = Aµk (N ). i∈Jµk (N )

Thus Perf µk (N ) − U∗ = ∆µ∞ (N ) + Aµk (N ), which proves equation 17. 17

(19)

Preprint

(A)

(B)

(B)

(A)

Initial-agent dependence of interruption gains. The kernels Pk Pk and Pk Pk can induce different early task states, changing both ∆µ∞ (N ) and Aµk (N ). Consequently, the preferred initial agent can depend on k and N ; the associated startup budget effects are discussed in Appendix A.2. Corollary A.11 (A sufficient cap in a finite candidate range). Fix A, B, u0 , N with ∆µ∞ (N ) > 0 and integer K ≥ 1. For each candidate cap 1 ≤ k ≤ K, let µk = (A, B, u0 , k) and suppose there is εµk ≥ 0 with Aµk (N ) ≥ −εµk Eµk |Jµk (N )|. Then min ⌊N/k⌋εµk < ∆µ∞ (N )

1≤k≤K

=⇒

∃ 1 ≤ k ≤ K : Perf µk (N ) > U∗ .

(20)

Proof of Corollary A.11. Each added interruption uses a distinct block of k experiments, so |Jµk (N )| ≤ ⌊N/k⌋ and Aµk (N ) ≥ −⌊N/k⌋εµk . Choose a minimizing cap in equation 20 and apply Theorem A.10. Corollary A.12 (Explicit cap under geometric loss decay; quantitative form of Corollary 3.6). Fix A, B, u0 , N with ∆µ∞ (N ) > 0. Under the expected loss condition of Corollary A.11, assume εµk ≤ Cρk for every µk = (A, B, u0 , k), k ≥ 1, with constants C > 0 and 0 < ρ < 1. The policy µk0 := (A, B, u0 , k0 ) with the following cap preserves at least half the NTA margin:    log(2N C/∆µ∞ (N )) k0 := max 1, , − log ρ (21) 1 Perf µk0 (N ) ≥ U∗ + 2 ∆µ∞ (N ) > U∗ . Proof of Corollary A.12. The expected loss is at most ⌊N/k⌋Cρk ≤ N Cρk . The choice of k0 makes this at most ∆µ∞ (N )/2; substitution in equation 19 proves the claim. Taking k = k0 for the selected initial agent gives a sufficient cap in [1, k0 ], proving Corollary 3.6. Uniform C, ρ and a positive margin lower bound across budgets give k0 = O(log N ).

18

Preprint

B

P REFIX R ETENTION AND E MPIRICAL C AP S ELECTION

We quantify the observed gains retained by short prefixes and provide the simultaneous confidence construction used to motivate candidate caps in Section 3.3. Executed fixed-k outcomes are analyzed separately in Appendix F. B.1

R ETENTION S TATISTICS AND C ONFIDENCE B OUNDS

We establish the statistical guarantee supporting Section 3.3. Definition B.1 specifies task-level prefix retention, Assumption B.2 states the sampling condition, and Proposition B.3 gives simultaneous 1 − δ lower confidence bounds. Definition B.1 (Observed prefix gain). For policy µ and task j, define (u )

Oµ,j := {i : oi i is recorded on task j under µ}. For each session index i ∈ Oµ,j , let li be its recorded experiment count and qi (d) its best quality after d experiments, with entry quality at d = 0. For integer k ≥ 0, define P gi (k) i∈O gi (k) = qi (min{k, li }) − qi (0), ζµ,j (k) = P µ,j . (22) i∈Oµ,j gi (li ) The ratio is defined for positive total gain; nµ counts these tasks. Assumption B.2 (Independent task observations). For each prespecified policy µ, cap k, and statistic, let Yj ∈ [0, 1] be ζµ,j (k) or 1{ζµ,j (k)≥0.9} . Conditional on defined ratios, assume independence across the nµ tasks. Dependence across configurations and caps is allowed. Proposition B.3 (Simultaneous P P lower confidence bounds). Under Assumption B.2, set Ȳ = −1 n−1 Y and µ = n j Y µ µ j j EYj . For Mcmp prespecified comparisons and δ ∈ (0, 1), define µY ≤ Ȳ by nµ dKL (Ȳ ∥µY ) = log(Mcmp /δ),

(23)

with µY = 0 when Ȳ = 0. Then Pr(µY ≤ µY for all comparisons) ≥ 1 − δ. Proof of Proposition B.3. The exponential-moment bound (Garivier & Cappé, 2011, Lemma 9), Jensen’s inequality, and Chernoff optimization give Pr(Ȳ ≥ z) ≤ e−nµ dKL (z∥µY ) for µY < z ≤ 1. Inverting at failure level δ/Mcmp gives equation 23; a union bound over the comparisons proves the claim. B.2

C ONFIGURATIONS AND O BSERVED P REFIX R ESULTS

This subsection documents the MLE-bench configurations and results supporting empirical cap selection in Section 3.3 and Figure 3. Table 3 lists six goal and five NTA configurations, all using max reasoning effort. A goal task run is one continuous session; NTA alternates at natural termination, starting with the first listed model. Harness documentation is provided by Bolin (2026); Anthropic (2026c); Moonshot AI (2026); DeepSeek-AI (2026). For MLE-bench, we use q = 1 − (position − 1)/(leaderboard size) on 75 tasks, including 75 measured Opus goal runs. Each goal task run is one option; NTA follows recorded boundaries, giving 1,065 options. At each k, Figure 3 plots the mean of min{k, li }/li across options horizontally and gain retention averaged across tasks within each configuration, then equally across configurations vertically, both as percentages. Individual curves use each configuration’s data, starting at (0, 0) and ending at (100, 100) at its maximum option length. 19

Preprint

Table 3: Model and harness composition of the eleven MLE-bench prefix-analysis configurations. NTA arrows specify the starting order of the repeating pair. Family Goal Goal Goal Goal Goal Goal NTA NTA NTA NTA NTA

Model(s), in starting order GPT-5.6-sol Kimi K3 GLM-5.3 DeepSeek-V4-Flash DeepSeek-V4.1-Flash Claude Opus 5 GPT-5.6-sol → Kimi K3 Kimi K3 → GPT-5.6-sol GLM-5.3 → DeepSeek-V4-Flash DeepSeek-V4.1-Flash → GLM-5.3 Claude Opus 5 → GPT-5.6-sol

Harness(es), same order Codex Kimi Code Claude Code DeepSeek Harness DeepSeek Harness Claude Code Codex → Kimi Code Kimi Code → Codex Claude Code → DeepSeek Harness DeepSeek Harness → Claude Code Claude Code → Codex

At k = 1, 3, 5, mean gain retention is 63.15%, 93.35%, and 97.83%, with experiment fractions 24.76%, 54.88%, and 73.77%. At k = 5, individual configurations retain 96.11%–99.20% (Opus: 98.74%), motivating candidates 1 ≤ k ≤ 5. For eleven configurations, fifty caps (k = 1, . . . , 50), and two statistics, Mcmp = 1100. At δ = 0.05 and k = 5, 87.84%–98.67% of tasks retain at least 90% of gains; Appendix B.1 gives simultaneous lower bounds of 65.08%–83.30%.

20

Preprint

C

E XPERIMENTAL S ETUP AND R EPRODUCIBILITY

We collect the execution environment, agent configurations, baseline settings, evaluation rules, and repeat protocol used throughout the experiments. Empirical findings are separated into Appendices D–G. C.1

H UMANIZE AND E XECUTION E NVIRONMENT

Terminology. A configuration specifies the agent or agent pair, starting order, cap, and budget. A task run is one complete execution on one task; a repeat evaluates the full task suite. Each agent session is modeled as an option, with experiments counted within it; a model response is one recorded output from a model call. Framework. We implement the workflow on Humanize (Liu et al., 2026), which coordinates agent execution in a shared workspace, fixed-k interruption, and final review as specified in Algorithm 1. The model harness supplies the internal agent policy; the workflow controls option boundaries and preserves shared task workspace. HMA uses k = 5 and ttotal = 21600 seconds per task, including trev = 900 seconds reserved for the other agent’s final review. Thus exploration ends at ts = 20700 seconds. Hardware. Our execution environment exposes 30 vCPUs backed by an Intel Xeon Platinum 8358 processor, 220 GiB RAM, and one NVIDIA A10 GPU. The GPU reports 24 GiB of available device memory. These resources are shared by the two alternating agents within a single task run. C.2

AGENT AND BASELINE C ONFIGURATIONS

Agents. Table 4 lists the full model names and native harnesses. In analyses and figures, GPT, Kimi, GLM, and Opus abbreviate GPT-5.6-sol, Kimi K3, GLM-5.3, and Claude Opus 5; DeepSeek-V4 and DeepSeek-V4.1 abbreviate the corresponding Flash models. DeepSeek alone denotes the version specified for the configuration. All agents in our runs use max reasoning effort. Codex, Claude Code, Kimi Code, and DeepSeek Harness provide the model execution interfaces (DeepSeek-AI, 2026; Bolin, 2026; Anthropic, 2026c; Moonshot AI, 2026). Appendix B.2 separately lists the goal and NTA configurations used for prefix analysis. Table 4: Model–harness mappings and reasoning effort for our runs. Model GPT-5.6-sol Kimi K3 GLM-5.3 DeepSeek-V4-Flash DeepSeek-V4.1-Flash Claude Opus 5

Harness Codex Kimi Code Claude Code DeepSeek Harness DeepSeek Harness Claude Code

Reasoning effort max max max max max max

Comparison baselines. We include ScienceFlow, MLEvolve, and ML-Master 2.0 using their reported results (Zhao et al., 2026; Du et al., 2026; Zhu et al., 2026). Table 5 records their original configurations; the max reasoning setting applies to our runs, while these published results retain their reported inference settings. C.3

E VALUATION P ROTOCOL AND R EPEATS

Evaluation metrics. Lite, Medium, High, and All denote the 22-, 38-, 15-, and 75-task sets, respectively. Lite, Medium, and High report medal percentages within each complexity group; All reports the medal percentage over the full 75-task suite. For repeated evaluations, rates are calculated from per-repeat medal counts before averaging. Table 2 reports all performance metrics to one decimal place. Within a model pair, Avg. of Single averages the two constituent goal means with equal weight; HMA gains are computed against this mean before rounding. Gold is the percentage of tasks attaining a gold medal. Med+ is the percentage outperforming the human leaderboard 21

Preprint

Table 5: Configurations of the three comparison baselines, as reported in their original papers. Resource allocations and time budgets differ across methods. Baseline ScienceFlow

Model and time budget DeepSeek-V4-FlashPreview; 24 h

Execution resources Key configuration 16–32 logical CPU cores, 256 Recoverable workspaces, GB RAM, at most 2 GPUs ESTRA re-anchoring, and resource-aware execution control. MLEvolve Gemini-3.1-Pro-preview; 21 vCPUs, 234 GB RAM, 1 Temperature 1.0; 500 ex12 h NVIDIA H200 pansion steps; 3 parallel branches and 3 initial drafts. cognitive ML-Master 2.0 DeepSeek-V3.2-Speciale; 36 AMD EPYC vCPUs and Hierarchical DeepSeek-V3.2 (think- 2 RTX 4090 GPUs per agent; caching; prior wisdom iniing) for context promo- 1008 GB RAM and 1 TB SSD tialized from 407 Kaggle competitions excluding shared across 4 tasks tion; 24 h MLE-bench.

median. Mean percentile averages the task-level leaderboard percentile over the evaluated suite, using the rank convention in Appendix B.2. All metrics use the workflow’s returned candidate; for HMA this is the final review selection or its fallback in Algorithm 1. Test-set scores are computed offline. Repeated evaluations. Each configuration is evaluated in three independent repeats over the fixed 75-task suite. Reported values average the three repeats; published baseline uncertainties retain their original definitions.

22

Preprint

D

A DDITIONAL E VIDENCE FOR THE M OTIVATION

We provide the detailed evidence behind the two motivation claims in Section 2: response-length decay in uninterrupted runs and harness staleness across model versions. D.1

D ECLINING R ESPONSE L ENGTH IN S INGLE -AGENT RUNS

Configurations. We document Figure 2 using the six configurations in Table 4. The /goal protocol runs one agent until natural termination or exhaustion of six hours of active runtime, without model alternation or fixed-cap interruption. Each of the six configurations is evaluated on 75 tasks with three independent repeats and max reasoning effort. Recorded cohorts and prompt policies are retained. Response-level moving average. Figure 2 reports mean output tokens per recorded model response. At one-minute active-runtime checkpoints, we divide the sum of provider-reported output tokens by the number of responses in the preceding 30 minutes, pooling the 75 tasks and repeats within each configuration. Near the origin, the window uses the elapsed prefix; windows without responses are undefined. This response-weighted statistic excludes tool-return text, input tokens, and coordinator output, while retaining model responses that request tools. Early versus late responses. Table 6 compares responses in hours 0–2 and 3–5 using raw event totals divided by response counts in each interval. All six configurations show a reduction of 29.8%– 45.0%; these interval statistics are computed before temporal smoothing. Table 6: Mean output tokens per recorded response. Change is relative to hours 0–2. The same task suite and repeats are used as in Figure 2. Model GPT-5.6-sol Claude Opus 5 GLM-5.3 DeepSeek-V4.1-Flash Kimi K3 DeepSeek-V4-Flash

D.2

Hours 0–2 tokens/response

Hours 3–5 tokens/response

Change (%)

336.5 1008.3 658.4 1137.5 585.7 682.2

213.6 554.8 388.5 696.1 390.2 479.2

−36.5 −45.0 −41.0 −38.8 −33.4 −29.8

H ARNESS S TALENESS ACROSS M ODEL V ERSIONS

Comparison setup. We compare ML-Master 2.0 (Zhu et al., 2026), MLEvolve (Du et al., 2026), and ScienceFlow (Zhao et al., 2026) with the official DeepSeek Harness on a 16-task MLE-bench subset (Chan et al., 2025), using DeepSeek-V4-Flash and DeepSeek-V4.1-Flash. We retain the research harnesses’ original workflow designs and experimental settings while disabling task-specific cold-start knowledge wherever present. Execution budgets are reported in Table 1. DeepSeek Harness runs each model under /goal with max reasoning effort for six hours; its reported medal rate is averaged over three repeats. Result accounting. Medal rates use all 16 tasks as the denominator. The disqualified ScienceFlow run on Chaii with DeepSeek-V4.1-Flash is counted as no medal because its checkpoint had seen the test answers. Table 1 reports medal rates and percentage-point differences from the corresponding DeepSeek Harness baseline. Table 7 provides the task-level outcomes. These 16-task DeepSeek Harness references are distinct from the full 75-task results in Table 2.

23

Preprint

Table 7: Task-level outcomes for the harness comparison. DeepSeek Harness entries report the number of runs earning a medal out of three; its medal total is the mean count. The last row divides each medal total by 16, matching Table 1. Other columns report medal classes from the retained run: G, S, and B denote gold, silver, and bronze; — denotes no medal, × denotes the disqualified ScienceFlow V4.1 Chaii run. V4 and V4.1 denote DeepSeek-V4-Flash and DeepSeek-V4.1-Flash. When a run did not nominate a final artifact, the recorded score-independent rule selects its latest eligible submission. Workflow DeepSeek Harness ScienceFlow ML-Master 2.0 MLEvolve Task

V4

V4.1

V4

V4.1

V4

V4.1

V4

V4.1

3D Object Detection APTOS Blindness Detection Toxic Comment Classification Leaf Classification Billion Word Imputation Cdiscount Image Classification Chaii Question Answering Freesound Audio Tagging iMet iWildCam Kuzushiji Recognition Learning Agency Essay Scoring LMSYS Chatbot Arena PetFinder Pawpularity Tweet Sentiment Extraction US Patent Phrase Matching

1/3 0/3 0/3 1/3 0/3 0/3 0/3 1/3 0/3 2/3 2/3 0/3 0/3 0/3 0/3 1/3

2/3 3/3 3/3 3/3 3/3 1/3 1/3 1/3 1/3 3/3 3/3 3/3 2/3 3/3 2/3 3/3

S S — S — — — — — — B — — — — —

G B G S G — × — S G — G S G S —

— — — — — — — — — — — — — — — S

S G — — — — — — — S G G — S — —

— — B B — — — — — G — — — — — —

— S S G — — — S — S — — — S S S

Medals Medal rate (%)

2.7 16.7

12.3 77.1

4 25.0

11 68.8

1 6.3

6 37.5

3 18.8

8 50.0

24

Preprint

E

A DDITIONAL M AIN -C OMPARISON A NALYSES

We supplement the main comparison with the aggregation rules, response and performance trajectories, matched constituent weighting, and task-level comparisons used for Figure 4. Analysis configurations. We analyze the five participating goal models and six HMA configurations in Table 2, using the protocol in Appendix C. The trajectory aggregation and matched constituent construction follow the definitions below. The separate scheduling ablations appear in Appendix F. Performance and response trajectories. At each active-runtime checkpoint, we aggregate the medal status of the latest accepted candidate on each task, assigning zero before a valid submission. Final points use the returned candidate under the recorded selection rule. Scores use the corrected answer keys. Receipt processing at the nominal deadline is included in the final point. Figure 4 (a) uses the response-level statistic in Appendix D.1: at each one-minute checkpoint, total actor output tokens in the preceding 30-minute trailing window are divided by the number of recorded responses, pooling tasks and repeats within each configuration and including HMA’s final review. Near the origin, the window uses the elapsed prefix; empty windows are undefined. Figure 4 (b) retains the unsmoothed medal-rate trajectory. Token counts measure output volume across different tokenizers, not a common monetary or computational cost. Neither panel extrapolates trajectories or uses best-so-far score selection. Matched group-average curves. Figure 4 assigns weight 1/6 to each HMA configuration. For configuration j with constituent models Aj , Bj , its goal reference is (YAgoal (t) + YBgoal (t))/2, where j j Y denotes mean output tokens per response or medal rate, computed within each configuration as above. We average these six references at each common runtime checkpoint. The resulting matched model weights are reported in Table 8; the same weights are used for Goal and HMA, so each model contributes equally to both references. DeepSeek-V4-Flash is excluded because it appears in none of the six HMA configurations. This matches the mean-of-constituents reference used for the gains in Table 2; the weights describe configuration composition rather than measured execution-time shares. Both panels use active runtime horizontally, with mean output tokens per response in (a) computed over a 30-minute trailing window and mean medal rate in (b). Faint curves retain the participating configurations. For the early/late comparison, we divide raw output-token totals by response counts separately in hours 0–2 and 3–5 within each configuration, then apply the same group weights. The resulting means are 610.9 and 527.6 tokens per response for HMA, and 650.3 and 394.9 for goal, retaining 86.4% and 60.7%, respectively; these interval statistics are computed before temporal smoothing. Table 8: Matched workflow weights. The matched weight is used for both Goal and HMA, so every model contributes equally to both references. Model GPT Opus DeepSeek-V4.1 Kimi GLM

Table 9: Early and final medals. Any-medal rates are reported at two hours and at the sixhour endpoint for each HMA order.

Matched weight 5/12 1/6 1/6 1/6 1/12

Order

2h

6h

Opus/GPT GPT/Opus GPT/DeepSeek-V4.1 GPT/Kimi GPT/GLM DeepSeek-V4.1/Kimi

57.3 58.7 54.7 60.0 50.7 42.7

78.2 73.3 72.0 70.7 69.3 62.7

Early progress and task-level gains. Table 9 reports the any-medal rates at two hours and at the six-hour endpoint. The six HMA configurations attain 57.3%, 58.7%, 54.7%, 60.0%, 50.7%, and 42.7% medals at two hours, with corresponding final rates of 78.2%, 73.3%, 72.0%, 70.7%, 69.3%, and 62.7%. Across matched GPT goal comparisons, GPT–Kimi HMA gains medals on 5, 4, and 6 tasks and loses them on 4, 3, and 2, yielding net gains of 1, 1, and 4. These successful tasks remain within the evaluated goal-task union, characterizing gains achieved within the six-hour budget.

25

Preprint

F

A BLATION S TUDIES

We supplement Section 4.3 with starting-order effects, executed fixed-k comparisons, naturaltermination statistics, and a selection-only final review audit. We discuss order before cap size because model exposure within a finite time budget depends on both. F.1

S TARTING -O RDER E FFECTS

We compare GPT-first and Kimi-first alternation under NTA and HMA with k = 5, then relate the endpoint differences to model-session time and partner participation. Configurations. Each configuration covers the same 75 tasks with a six-hour budget and max reasoning effort. The additional Kimi-first HMA run records 283 exploration options: among the 75 initial options, 52 reach the cap, 22 end naturally, and one reaches the deadline. The GPT-first k = 5 run records 277 exploration options, with 66 initial cap closures and nine natural exits. Corrected answer keys give 42/75 medals for Kimi-first NTA, including three corrected outcomes. For the Kimi-first HMA archive, the terminal window is 21,700 seconds; all accepted candidates arrive within 21,600 seconds and its six-hour medal count matches the returned result. Final quality and finite-budget exposure. Figure 7(a) compares both GPT–Kimi starting orders under the six-hour budget. NTA reaches 77.3% and 56.0% medals with GPT and Kimi first, respectively; HMA with k = 5 reaches 70.7% and 66.7%, narrowing the observed gap from 21.3 to 4.0 percentage points. The six-hour horizon limits sustained participation by both agents under NTA: by 5.75 hours, the second agent has produced a recorded response on only 19/75 GPT-first tasks and 47/75 Kimi-first tasks. Under NTA, the nominal first model occupies about 81–83% of the budget. Because the GPT goal baseline is stronger than the Kimi baseline (68.0% versus 54.2%), the longer uninterrupted GPT exposure in the GPT-first archive is associated with the higher NTA result. GPT-first HMA instead allocates 51.1% to GPT and 46.3% to Kimi, whereas Kimi-first HMA still allocates 81.3% to Kimi and only 16.2% to GPT. These finite-budget differences accord with the startup effects in Appendix A.2; improvement from interruption remains conditional on continuation advantage (Theorem 3.5). (a) Starting-order ablation

Medal rate (%)

70

77.3 70.7

60

NTA HMA, k=5 66.7

70

Medal rate (%)

80

(b) Fixed-cap ablation 80

56.0

50 40 30

66.7 (50/75)

k=1

k=3

70.7 (53/75)

60 50 40 30

20

20

10

10

0

65.3 (49/75)

0 GPT → Kimi

Kimi → GPT

k=5

Figure 7: Fixed-cap interruption: starting-order and cap ablations. Both panels use the same 75-task, six-hour GPT-5.6-sol–Kimi K3 suite. (a) Starting-order comparison: NTA and HMA with k = 5; the cap narrows the observed order gap. (b) GPT-first HMA with k ∈ {1, 3, 5}; the new A10 runs produce 49/75 and 50/75 medals at k = 1 and k = 3, while the existing k = 5 run produces 53/75. Values are final medal rates, with counts shown in the cap panel. Model time allocation. We reconstruct occupied model-session intervals from native session and controller logs over the first six hours. Native creation and completion records determine session endpoints. For an interrupted session without a completion record, we use the next session or option start, or the active attempt’s end, whichever occurs first. Intervals are intersected with active-runtime windows; pause gaps and auxiliary subagent sessions are excluded, and final review is attributed to 26

Preprint

(u)

its executing model. If wj is the resulting model-u duration on task j, its mean budget share is P75 (u) 100 j=1 wj /(75 · 21600). All returns to a model are accumulated. Unallocated time averages 0.07% and 0.15% for GPT-first and Kimi-first NTA, and 2.63% and 2.49% for the corresponding HMA runs. These shares include tool execution and waiting within sessions. Recorded starting-order exceptions. Workflow labels retain the archived nominal configurations. In the GPT-first NTA archive, seven of 75 tasks have Kimi as the earliest recorded native session; the other NTA configuration and both HMA runs match their nominal initial models on all 75 tasks. Runtime attribution follows the recorded model, and all tasks remain included. Thus the historical NTA groups are configuration comparisons rather than perfectly controlled starting-order interventions. Designated-partner participation. We measure participation from the first recorded response by the nominal second agent (Kimi in the GPT-first configuration and GPT in the Kimi-first configuration), excluding turn-review sessions. This measures observable access to search, rather than an exact controller launch time. Tasks with no such response are right-censored at six hours; counts retain all 75 tasks in the denominator. Table 10 reports these response-based timing statistics separately from the session-duration estimates above. With GPT first, the second agent’s mean within-task share of output tokens before hour 5.75 rises from 13.0% under NTA to 36.1% under HMA. A cap limits experiments per option; it does not impose a fixed wall-clock duration on an experiment. Table 10: The cap advances access to the second agent. All counts are out of 75 tasks and exclude final review. By the six-hour deadline, 22 GPT-first NTA tasks and 18 Kimi-first NTA tasks still have no recorded response from the second agent. Tasks with a second-agent response Workflow

By 3 h

By 5 h

By 5.75 h

17 15 55

18 27 69

19 47 75

GPT → Kimi NTA Kimi → GPT NTA GPT → Kimi HMA

F.2

Median first response (h)

Final medals (%)

5.97 5.56 1.89

77.3 56.0 70.7

E FFECTS OF THE F IXED I NTERRUPTION C AP

We hold GPT as the initial agent and compare executed HMA policies with k ∈ {1, 3, 5}, complementing the observed-prefix analysis in Appendix B. Configurations and results. Each cap contributes one repeat over 75 tasks with the same model pair, max reasoning effort, and nominal six-hour budget including final review. The k = 1 and k = 3 archives use a 21,700-second terminal window. Table 11 reports the final medal counts and rates, also plotted in Figure 7(b). These cap configurations and the reverse-order configuration are used for scheduling ablations; the main comparison retains its six configurations. Table 11: k = 5 gives the highest medal rate among the tested finite caps. GPT-first HMA alternates GPT-5.6-sol and Kimi K3 on the same 75 tasks under the nominal six-hour budget. Cap k 1 3 5

Medals / 75

Medal rate (%)

49 50 53

65.3 66.7 70.7

Empirical selection within the candidate range. The prefix statistics in Appendix B motivate candidates 1 ≤ k ≤ 5. Among the executed GPT-first runs, k = 1, 3, 5 yield 65.3%, 66.7%, and 70.7% medals, respectively, against the GPT goal mean of 68.0%. These results support selecting 27

Preprint

k = 5 within the tested finite caps for this order and horizon. Theorem 3.5 gives the corresponding conditional advantage criterion; the prefix statistics guide candidate selection without directly estimating its continuation values. Interaction with time and starting order. A larger cap permits longer uninterrupted experiment blocks before handoff, while the six-hour deadline limits the partner’s subsequent search. Given GPT’s stronger standalone performance, the GPT-first cap trend and higher GPT-first NTA result are consistent with a benefit from preserving more continuous GPT search. Conversely, k = 5 improves the Kimi-first result and reduces sensitivity to the initial agent. The limited horizon can therefore leave the pair’s complementary capabilities incompletely exploited, making cap size and initial-agent choice jointly relevant. The recorded starting-order exceptions in Appendix F.1 delimit this interpretation. F.3

NATURAL T ERMINATION BEFORE THE C AP

We report option-closure outcomes from terminal controller receipts and evaluator ledgers for the six main-table HMA configurations. One complete repeat per configuration, using the first Opus-first GPT repeat, yields 450 task runs with the same answer keys and final-candidate rule as Table 2. Option closure. Table 12 separates the first option from later options and distinguishes reaching five accepted experiments, natural termination before five, and closure at the exploration deadline. Final review sessions are excluded. Initial options contain 57 natural exits among 446 closures before the deadline (12.8%); later options contain 598 among 992 (60.3%). Across all options, 655/1,438 closures before the deadline are natural (45.5%). Of the 655 natural exits, 248 produce no accepted experiment. These are retained in the audit. One GPT–DeepSeek deadline receipt omits the cap field following a controller timeout; its accepted count is zero, and the recorded standing submission agrees with the adopted fallback. All 1,888 recorded option counts are at most five. Table 12: Observed option-closure reasons at k = 5. Each configuration has 75 initial options. Natural-exit percentages exclude deadline closures because the latter censor whether an option would reach the cap or terminate naturally. The first listed model starts each workflow. Initial option

F.4

Later options

Configuration

Cap Natural Deadline Cap Natural Deadline

Opus → GPT GPT → Opus GPT → DeepSeek-V4.1 GPT → Kimi GPT → GLM DeepSeek-V4.1 → Kimi

61 72 71 66 54 65

13 2 4 9 19 10

1 1 0 0 2 0

56 89 88 56 71 34

121 69 132 71 36 169

74 74 75 75 73 75

All configurations

389

57

4

394

598

446

F INAL R EVIEW AND H ISTORICAL A NY-M EDAL R ETENTION

We evaluate retention of historically attained any-medal outcomes on the same 450 task runs as Appendix F.3, supplementing the mechanism in Section 3.4. Selection-only comparison. We hold each recorded exploration trajectory fixed and compare the returned candidate with the candidate standing immediately before final review. The reserved final review time is not reallocated to search, so this measures the effect of final selection rather than a rerun with a longer exploration budget. Historical any-medal retention. For each task run, historical any-medal status is one if any accepted exploration candidate earns bronze or better, and zero otherwise. We compute this status from offline evaluator records before final review; repeated receipts for a reviewed candidate add no new historical outcome. Historical any-medal retention is the fraction of historically medal-positive task runs whose selected candidate earns a medal. Table 13 compares this fraction before and after 28

Preprint

final review. Of 450 task runs, 330 produce a medal-winning candidate; final review raises retained medals from 311/330 (94.2%) to 319/330 (96.7%), recovering eight medals without losing any held immediately before final review. Table 13: Final review improves retention of historical any-medal outcomes. Each configuration contains 75 task runs. Historical counts runs with at least one medal-winning exploration candidate; retention divides selected medal counts by Historical. Before and after refer to the standing candidate and final review’s returned candidate. Missed counts historical medal outcomes not retained after final review. Configuration

Changed

Historical Selected medals Retention (%) Missed any medal before → after before → after after

Opus → GPT GPT → Opus GPT → DeepSeek-V4.1 GPT → Kimi GPT → GLM DeepSeek-V4.1 → Kimi

4 6 11 6 9 4

59 57 56 54 54 50

58 → 58 55 → 55 51 → 54 52 → 53 48 → 52 47 → 47

98.3 → 98.3 96.5 → 96.5 91.1 → 96.4 96.3 → 98.1 88.9 → 96.3 94.0 → 94.0

1 2 2 1 2 3

All configurations

40

330

311 → 319

94.2 → 96.7

11

Across all 450 runs, the returned candidate matches the historical any-medal status in 439 (97.6%), including 120 runs that never produce a medal. The conditional retention rate of 96.7% excludes these 120 runs. Final review changes 40 selections (8.9%); 311/319 final medals (97.5%) were already held before final review. Thirty-five GLM-pair final reviews are silent and one DeepSeekpair final review fails; their recorded fallback candidates remain included. This audit measures historical any-medal retention at final selection. Medal recovery and remaining misses. On MBM-01 with GPT–DeepSeek-V4.1, the first GPT option reaches 0.84671 without a medal, and DeepSeek reaches bronze at 0.85445. The returning GPT replaces the standing result with 0.75795; final review restores the archived DeepSeek candidate and recovers the medal. On MBM-20 with GPT–GLM, GPT produces a gold-medal candidate at 0.84835, but the final GLM candidate scores 0.25403; final review restores 0.84835. In aggregate, final review reduces missed historical medals from 19 to 11, supporting its role in retaining search outcomes at final selection.

29

Preprint

G

C ASE S TUDIES OF H ETEROGENEOUS A LTERNATION

We examine four model pairs across five model families: GPT–Opus, GPT–DeepSeek, GLM–Kimi, and DeepSeek–Kimi. The INGV case in Section 4.4 is followed by examples from text prediction, image regression, and satellite positioning. Selection and measurement. We selected task runs with inspectable artifacts, controller logs, evaluator receipts, and native response records. GPT–Opus, GPT–DeepSeek, and DeepSeek–Kimi occur in the main comparison; GLM–Kimi is an additional combination using the same k = 5 workflow. Following the trajectory analysis of Zhao et al. (2026), we align each accepted candidate with its producing option using controller counts and receipt times. Test scores are evaluated offline and were hidden from the agents; all accepted scores, including regressions, are plotted. Goal references use three runs per constituent. The appendix figures compare disjoint first- and last-20response windows; options with fewer than 40 responses show their counts instead. Final review is excluded from token statistics and preserves the standing candidate in all four cases. Response renewal is compared within the same model, since tokenizers and policies differ. These cases illustrate observed mechanisms rather than estimating their average causal effect. G.1

INGV VOLCANIC E RUPTION P REDICTION

Repeated refinement. On INGV (MBH-07), GPT and Opus complete seven exploration options with 5, 5, 5, 2, 5, 1, and 4 accepted experiments and 77, 263, 251, 59, 110, 49, and 48 responses, respectively. Options 1, 2, 3, and 5 reach the cap; options 4 and 6 end naturally, and option 7 reaches the deadline. Their best MAEs are 1,123,915, 41,214, 39,061, 39,878, 37,422, 36,934, and 34,234: five of the six post-handoff options improve the historical best, while the fourth option regresses. Shared features and cached predictions support successive refinements. The trace records GPT reusing and pruning Opus’s retrieval features, Opus adding consistency constraints for paired windows, GPT introducing an uncertainty-based residual gate, Opus refining its calibration, and GPT enforcing joint temporal feasibility. Score comparisons and plot conventions. After the first Opus session reaches 41,214 MAE, subsequent alternation lowers it to 34,234, a further 16.9% improvement and 46.5% below the best of six constituent goal runs (63,991). Native MAE resolves progress beyond the already-gold goal results. Figure 5(a) retains all 27 accepted scores, including regressions, without smoothing; it uses a logarithmic axis with a linear inset for late options. Horizontal lines mark the gold threshold (3,971,366), the minimum MAE in MLE-bench’s 621-entry reference leaderboard (3,438,191), and the best constituent goal endpoint. Blue and purple regions identify GPT and Opus, and dotted vertical lines mark handoffs. Response measurement and renewal. In Figure 5(b), each point averages output tokens per response from the preceding accepted submission or option start to the current acceptance. The 27 intervals contain 826 responses, excluding 31 trailing responses; final review is excluded. Comparing the first experiment after each return with the same model’s preceding last experiment, mean response length increases on four of five returns by a factor of 1.6–3.6; the final Opus return has a ratio of 0.94. G.2

G OOGLE QUEST Q&A L ABELING : GPT–D EEP S EEK

Repeated reuse of text-model predictions. On Google QUEST Q&A Labeling (MBM-10), GPT5.6-sol and DeepSeek-V4.1-Flash complete seven options, five of which produce 23 accepted candidates. GPT initializes sparse-text and encoder models; DeepSeek completes additional DeBERTa fits and combines their saved predictions. Subsequent sessions reuse these artifacts for target-specific blending, calibration, and larger-encoder refinement. The first five options achieve best Spearman correlations of 0.45664, 0.47124, 0.48741, 0.50381, and 0.50760. The returned candidate scores 0.50747, versus 0.47252 for the best of six constituent goal runs (Figure 8). At the first returns, GPT’s mean response length rises from 168.9 to 529.1 tokens and DeepSeek’s from 647.2 to 1060.9; GPT’s next return instead falls from 616.7 to 339.2. The final two options produce no accepted experiment, so neither response renewal nor useful progress occurs at every handoff. 30

Preprint

(a) Test performance

Gold threshold

0.42 0.40

1061

1025

DS 2

GPT 3

600 400 200

n=5

0.45664

0.44

800

n=17

0.46

1000

339 327

0.50381

Last 20

657

0.47124

Best observed goal

529 617

0.48

First 20

1200

647

0.50

1400

936

GPT 3 DS 4 GPT 5 0.50747

169

Spearman correlation ↑

0.52

(b) Output tokens per response

DS 2

Mean output tokens / response

GPT 1

DS 6

GPT 7

0 0

1

2

3

4

5

6

GPT 1

Active runtime (h)

DS 4

GPT 5

Exploration option GPT-5.6-sol

DeepSeek-V4.1-Flash

Figure 8: GPT–DeepSeek refinement on Google QUEST. (a) All 23 accepted scores, including regressions. Shading identifies the active model; reference lines mark the best constituent goal endpoint and gold threshold. (b) First- and last-20-response means per option. Options 6 and 7 have only 17 and 5 responses and no accepted experiments; their response counts replace overlapping windows. Final review retains the standing candidate and is excluded from token statistics. G.3

P ET F INDER PAWPULARITY: GLM–K IMI

Extending a shared prediction pool. On PetFinder Pawpularity (MBM-27), GLM-5.3 builds CLIP and DINOv2 feature caches, regressors, and calibrated predictions before Kimi K3 takes over at 2.40 hours. Kimi reuses the out-of-fold predictions in a constrained stack and adds concatenatedfeature ridge and MLP members, improving the best test RMSE from 16.87437 to 16.78333. Returning GLM explores further combinations and reaches 16.77953; the returned candidate scores 16.85639 and earns gold (Figure 9). Neither constituent earns a medal in its three goal runs. The three productive options contain 5, 5, and 3 experiments; the final Kimi option adds none. GLM’s returning first 20 responses average 724.5 tokens, compared with 503.8 in its preceding option’s last 20. The final score regression remains visible: artifact reuse improves the search trajectory without ensuring that the returned candidate is its best. (a) Test performance

(b) Output tokens per response Kimi 2

20.0

Kimi 4

Late candidates (linear scale)

20.08411

17.0

19.5

RMSE ↓

GLM 3

16.9

19.0

16.77953

16.8

18.5

2.5

4.0

18.0

5.5

16.85639

17.5

16.78333

Best observed goal

17.0

Gold threshold

0

1

1400 First 20

Last 20

1200 949

1000 817

800 600

724 504

470 347

400 200

n=12

GLM 1

Mean output tokens / response

20.5

0 2

3

4

5

6

Active runtime (h)

GLM 1

Kimi 2

GLM 3

Kimi 4

Exploration option GLM-5.3

Kimi K3

Figure 9: GLM–Kimi cooperation on Pawpularity. (a) Every accepted test RMSE, with a linear inset resolving the late candidates. The dotted and dashed lines show the best of six constituent goal endpoints and the gold threshold. (b) Response-length windows for the three productive options; the last option has only 12 responses. The final candidate remains below the gold threshold despite regressing from the best observed RMSE.

31

Preprint

G.4

G OOGLE S MARTPHONE D ECIMETER : D EEP S EEK –K IMI

Late correction after an unsuccessful refinement. On the Google Smartphone Decimeter Challenge 2022 (MBH-12), DeepSeek-V4.1-Flash builds a GNSS fusion pipeline with Doppler constraints and robust optimization, reaching 4.33091 mean haversine error. Kimi K3 improves it to 4.28914, but DeepSeek’s next option leaves a worse candidate at 5.35529. Returning at 5.08 hours, Kimi retains the shared solver, checks the mean-error objective, and replaces straight-line extrapolation through an end-of-log outage with a direction-matched route from public training trajectories. Its accepted candidate reaches 2.77310 and earns bronze, a 42.1% reduction from the best of six constituent goal endpoints (4.78536), none of which earns a medal (Figure 10). Kimi’s first 20 returning responses average 847.9 tokens, below the preceding Kimi option’s last-20 mean of 1010.0. This late quality gain therefore does not require a response-length rebound. (b) Output tokens per response

3.5 Bronze threshold

2.77310

3.0 2.5

1000 750 500 250

n=9

4.28914

848

4.33091

1250

601

4.0

1500

Last 20

1314

Best observed goal

4.5

First 20

1750

1283

5.0

2000

1010

5.5

Kimi 4

637

DS 3 5.35529

1503

Kimi 2

1440

DS 1

Mean output tokens / response

Mean haversine distance (m) ↓

(a) Test performance

0 0

1

2

3

4

5

6

DS 1

Active runtime (h)

Kimi 2

DS 3

Kimi 4

DS 5

Exploration option DeepSeek-V4.1-Flash

Kimi K3

Figure 10: DeepSeek–Kimi correction in smartphone positioning. (a) All ten accepted meanhaversine scores. The fourth option repairs the late outage and crosses the bronze threshold after the third option regresses. The dotted line is the best constituent goal endpoint. (b) First- and last20-response means; the final DeepSeek option contains nine responses and no accepted experiment. Final review retains Kimi’s repaired candidate.

32

Related documents

• Humanize MLE Agents: Multi-Agent Session Alternation Beyond Single-Agent Limits#1297846
• Learning Reusable Agent Subroutines from Experience#1344605
• SkillGym: Training Skill-Use Agents with Automatic Verifiable Environment Generation#1141860
• Using Process Mining to Generate AI Agents from Software Engineering Process Records#1199899
• Tokalator: A Context Engineering Toolkit for Artificial Intelligence Coding Assistants#2664