OpenMOSS
AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers Tianyu Huai2,3,4 , Tingshuo Fan1,4 , Xinchi Chen1,4,† , Yining Zheng1,2,4 , Yuxin Wang1,2,4 , Shuang Chen1,2,4 , Jie Zhou3 , Xuanjing Huang1,2 1 Fudan University 2 Shanghai Innovation Institute 3 East China Normal University 4 OpenMOSS
arXiv:2607.29626v1 [cs.AI] 31 Jul 2026
Abstract As LLMs evolve from code completion systems into autonomous scientific agents, evaluating their ability to conduct experiments has become increasingly important. Existing benchmarks typically focus on static code generation, paper replication, or final answer correctness, but do not directly assess whether agents can interpret experimental evidence and use it to guide subsequent hyperparameter decisions. To address this gap, we introduce AgentHPOBench, a sequential benchmark comprising 30 executable machine learning tasks across seven research categories. Each task begins with a validated baseline run, after which an agent performs several sequential interventions. At each step, the agent observes the accumulated configurations, metrics, and logs before proposing the next valid configuration. We evaluate 12 widely used agents and conventional HPO baselines under a unified protocol. The results show that current agents exhibit measurable experimental optimization ability across domains, but still face clear limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance. GitHub: https://github.com/OpenMOSS/AgentHPOBench
1
Introduction
As LLMs improve in reasoning, use of long contexts, and tool interaction, agents are becoming capable of completing increasingly complex workflows that form part of empirical research. Recent benchmarks have therefore moved beyond static question answering and isolated code generation toward executable research environments. MLGym [1] and MLE-Dojo [2] already provide interactive environments in which agents execute experiments and refine solutions through feedback. PaperBench [3], AutoExperiment [4], RE-Bench [5], MLR-Bench [6], and AIRS-Bench [7] further extend evaluation toward paper replication, research engineering, and broader stages of the ML research lifecycle. These benchmarks demonstrate the value of iterative evaluation, but they generally assess broad research or engineering workflows in which improvements may arise from data processing, code modification, architecture design, debugging, hyperparameter tuning, or combinations of these actions. Traditional HPO benchmarks address a different aspect of the problem. They provide controlled tabular, surrogate, or executable objectives for comparing optimization algorithms [8, 9], but typically abstract away the logs, configurations, and procedural context of research repositories. This leaves a specific capability † Corresponding authors.
1
Figure 1 Conceptual overview of AgentHPOBench. Benchmark
Interface
HPO-B [10] Tabular LCBench [11] Tabular HPOBench [8] Mixed YAHPO Gym [9] Surrogate JAHS-Bench-201 [12] Surrogate DACBench [13] Executable AgentHPOBench
Task Unit Objective Learning curve Objective Objective Search space Environment
Decision Space
Observation
Primary Target
Hyperparameters Metric Hyperparameters Metric curve Hyperparameters Metric Hyperparameters Metric Architecture + hyperparameters Metrics Dynamic configuration State + reward
Executable Research repository
Hyperparameters
Metrics + logs
Optimizer Optimizer Optimizer Optimizer Optimizer Policy HPO agent
Table 1 Comparison with representative HPO and algorithm configuration benchmarks.
insufficiently examined: whether an agent can interpret evidence from completed repository experiments and convert it into the next effective hyperparameter configuration. In a typical ML workflow, researchers execute a baseline, inspect target and auxiliary metrics together with execution logs, and decide which parameters to adjust in the next experiment. We therefore ask a focused question: can an autonomous agent improve real ML experiments through a sequence of hyperparameter interventions guided by empirical feedback? To address this question, we introduce AgentHPOBench, a dedicated and controlled benchmark for evaluating sequential HPO by agents in executable research repositories, as shown in Figure 1. AgentHPOBench contains 30 executable tasks constructed from recent ML research repositories across seven categories: natural language processing, computer vision, time series forecasting, graph learning, reinforcement learning, large language modeling, and structured learning. By focusing on recent research repositories, the task suite captures contemporary training pipelines, configuration interfaces, and evaluation practices that are often abstracted away by conventional HPO benchmarks. Table 1 summarizes the differences between AgentHPOBench and representative HPO benchmarks. AgentHPOBench comprises three components. First, the task construction protocol maps each repository to an executable optimization task with a reference baseline, a target metric, a constrained intervention space, and a paper or repository anchor. Second, the unified execution harness validates proposed configurations, executes experiments, and records the complete sequence of configurations, metrics, logs, and decisions. Third, the scoring and auditing layer verifies task traces and converts heterogeneous task results into mean bounded normalized score, baseline win rate, and mean anchor attainment. These metrics distinguish relative improvement over the reference baseline from absolute attainment of reported performance. Together, these components implement a common sequential evaluation protocol. Each task begins with a validated reference baseline that is shared by all agents and methods evaluated under the same budget setting. After the baseline run, the agent receives a fixed number of intervention opportunities. At each intervention, it 2
observes the accumulated configurations, target and auxiliary metrics, and execution logs, and proposes a new configuration within the intervention space derived from the official training scripts, configuration files, or repository documentation. The dataset, data split, target metric, evaluation code, and benchmark metadata remain fixed. The audited result after the final intervention is used for scoring. AgentHPOBench therefore isolates the ability to convert experimental feedback into the next valid configuration. We evaluate models with open weights and API agents, together with conventional HPO methods under the same baseline and number of intervention opportunities. The results show that current agents can improve reference baselines and that several agents obtain stronger aggregate results than conventional HPO methods. However, this advantage is not consistent across task categories or evaluation settings. Evaluation under the full training budget improves both mean bounded normalized score and mean anchor attainment, but does not uniformly increase baseline win rate. The feedback ablation shows that removing intermediate experimental evidence reduces optimization performance, while trajectory analysis reveals that later interventions may plateau or discard earlier gains. These findings show that discovering a useful configuration and reliably refining it through feedback are distinct capabilities. In a nutshell, our contributions are as follows: • We introduce AgentHPOBench for evaluating whether agents can convert experimental feedback into effective hyperparameter decisions in executable research repositories. The benchmark contains 30 tasks from recent ML repositories across seven research categories. • We develop a unified evaluation framework with intervention spaces and an execution harness that validates, executes, and records agent interventions under shared reference baselines and fixed task definitions. The framework measures both improvement over the baseline and attainment of reported reference performance. • We evaluate several agents together with conventional HPO methods on AgentHPOBench. The results provide a systematic view of the capabilities and limitations of current agents in repository based HPO.
2
Related Work
Benchmarks for language agents. A growing body of work evaluates agents in ML research workflows. MLAgentBench [14], ML-Bench [15], and MLE-bench [16] examine codebase use, model training, and ML engineering, while CORE-Bench [17], RE-Bench [5], and PaperBench [3] focus on reproducibility, research engineering, and paper replication. MLGym [1], MLE-Dojo [2], MLR-Bench [6], and AIRS-Bench [7] broaden evaluation to more complete ML research workflows. These benchmarks assess general research execution or reproduction, whereas AgentHPOBench isolates whether agents can convert repository metrics and logs into the next valid hyperparameter configuration. HPO methods and benchmarks. OpenML benchmark suites [18] and HPOBench [8] provide standardized tasks for comparing optimization algorithms, while NAS-Bench-101 [19] provides reusable objectives for architecture search. Common HPO approaches include random search [20], methods that use surrogate models [21–23], resource allocation methods such as Hyperband [24] and BOHB [25], and Population Based Training [26]. These methods and benchmarks generally assume a predefined search space, a structured objective interface, and relatively clean numerical feedback. LLMs and agents for optimization. Recent work also uses language models as optimizers or HPO assistants. OptFormer learns an optimizer from tuning traces [27], LLAMBO incorporates LLMs into Bayesian optimization [28], and other studies investigate HPO decisions guided by LLMs and optimization frameworks based on agents [29–31]. Rather than proposing another optimization method, AgentHPOBench evaluates how reliably agents convert accumulated feedback from executable research repositories into valid configurations and empirical improvement.
3
AgentHPOBench RL
TS
Graph
NLP CV
Auditing & Scoring
Trace Recording
LLM
SL
��,� = {��,� , ��,� , ℓ�,� }
Agent
��,� = ��,� ��,�
Validity & Completeness Result Schema
Logs Metrics History
PASS
30 Real ML Repositories
···
Pre-Scoring Audit
Trial Execution
Agent Observation Sequential Optimization Loop
Task Specification Reference Baseline Target Metric
Open-weight Agents
Compare & Diagnose
Paper / Repo Anchor Intervention Space
Execution Scripts Evaluation Interface
API-based Agents
Intervention Proposal new
old
Executable Task Interface
��,� ∈ Ω�
Revise Config
Result Protocol
ℋ�−1 = {��,0 , …, ��,�−1 }
Figure 2 Overview of AgentHPOBench. Each task is constructed from an executable ML repository with a reference baseline, target metric, intervention space, anchor, and standardized task interface. At each step, the agent observes the accumulated metrics and logs and proposes a valid configuration for the next run. The harness validates and executes the proposal, records the resulting trace, and audits the completed trajectory before scoring.
3
Method
We introduce AgentHPOBench, a benchmark and evaluation harness for assessing whether agents can improve executable ML experiments through sequential hyperparameter interventions, as shown in Figure 2. Unlike agent benchmarks that evaluate general execution across multiple steps, code modification, or paper reproduction, AgentHPOBench isolates a specific experimental capability: converting the logs and metrics of an executed repository experiment into a valid configuration for the next run. The agent selects values only from the predefined intervention space, while the harness validates and executes the proposed configuration and returns the resulting observations. This process is repeated for a predefined number of intervention steps. Table 1 compares AgentHPOBench with representative HPO benchmarks. Most prior HPO benchmarks provide controlled tabular, surrogate, or wrapped objectives for comparing optimization algorithms, but abstract away repository execution details, textual logs, and sequential interaction. AgentHPOBench instead exposes real research repositories as executable task units and evaluates sequential optimization using task metrics tied to results reported by the corresponding papers or repositories.
3.1
AgentHPOBench
AgentHPOBench consists of 30 tasks constructed from 30 executable ML repositories on GitHub. Task construction follows three principles. First, each task must require a substantive experimental decision within an executable training or evaluation pipeline. Second, each task must provide measurable feedback after every intervention, allowing the benchmark to evaluate how the agent uses previous outcomes to inform subsequent decisions. Third, each task must remain close to real research practice by preserving the scripts, dependencies, logs, and failure modes of the original repository whenever possible. Each task is based on an executable experiment from the original repository or its official reproduction environment. An executable task interface specifies how the harness launches the experiment, extracts the target metric, and records the outputs under a standardized result protocol. For each task, we construct the intervention space from the hyperparameters and valid values exposed by the official training scripts, configuration files, or repository documentation. We retain only fields that affect the execution or outcome of 4
the experiment. Agents may modify only these predefined fields, while the dataset, data split, target metric, evaluation code, and benchmark metadata remain fixed.
3.2
Problem Formulation
We formulate autonomous hyperparameter optimization over executable research repositories as a sequential decision problem. Each task 𝑡 is defined by a repository specific experimental objective, a constrained intervention space Ω𝑡 , an evaluation protocol, and a scalar performance metric. For task 𝑡 under budget setting 𝑟, let 𝑥 𝑡,0 ∈ Ω𝑡 denote the reference baseline configuration. This configuration is selected and validated during task construction and is shared by all agents and conventional optimizers. Executing 𝑥 𝑡,0 under 𝑟 produces the reference baseline performance 𝑦𝑡,0 = 𝑓𝑡,𝑟 (𝑥 𝑡,0 ).
(1)
The same 𝑦𝑡,0 serves as the reference baseline for all agents and conventional optimizers evaluated under the same budget setting. The configuration 𝑥 𝑡,0 remains fixed across budget settings, but executing it under different budget may produce different baseline performance. The agent first receives the reference baseline configuration 𝑥 𝑡,0 and observes its performance 𝑦𝑡,0 . The harness records each completed run, including the baseline, as a trace entry 𝜏𝑡,𝑘 = {𝑥 𝑡,𝑘 , 𝑦𝑡,𝑘 , ℓ 𝑡,𝑘 },
(2)
where ℓ 𝑡,𝑘 contains the execution logs and auxiliary metrics made available to the agent. At intervention step 𝑘 ∈ {1, . . . , 𝐾}, the agent observes the accumulated trace history ℋ𝑡,𝑘−1 = {𝜏𝑡,0 , 𝜏𝑡,1 , . . . , 𝜏𝑡,𝑘−1 },
(3)
uses this history to form an implicit experimental state, and proposes a new valid configuration 𝑥 𝑡,𝑘 ∈ Ω𝑡 . After validating the proposal, the harness executes 𝑥 𝑡,𝑘 under budget setting 𝑟 and obtains 𝑦𝑡,𝑘 = 𝑓𝑡,𝑟 (𝑥 𝑡,𝑘 ),
(4)
where the direction of improvement is specified by the task metric. The resulting trace 𝜏𝑡,𝑘 is then added to the history, and the corresponding feedback is returned to the agent before the next intervention. After 𝐾 sequential interventions, this process produces the trajectory (𝑥 𝑡,0 , 𝑦𝑡,0 ), (𝑥 𝑡,1 , 𝑦𝑡,1 ), . . . , (𝑥 𝑡,𝐾 , 𝑦𝑡,𝐾 ).
(5)
This formulation differs from conventional HPO benchmarks in two respects. First, the objective is embedded in an executable research repository rather than exposed through a clean objective interface. Second, the agent must interpret experimental evidence, including logs, metrics, task constraints, and prior execution outcomes, and convert this evidence into a valid configuration for the next experiment.
3.3
Evaluation Harness
To make the benchmark executable and comparable across agents, we implement a unified evaluation harness. The harness standardizes the interaction between agents and heterogeneous research repositories while preserving the original execution logic of each task. It launches experiments, provides task context to the agent, validates proposed configurations, and records the resulting outputs under a standardized result schema. For each task, the harness maintains the agent visible trace, together with an internal execution record containing the proposed configurations, extracted metrics, execution logs, runtime metadata, and completion 5
Agent / Method
Mean bounded normalized score
Overall metrics
NLP (3) CV (5) TS (7) Graph (2) RL (3) LLM (4) SL (6) Overall BWR (%) MAA (%)
Conventional HPO baselines Random search TPE BOHB variant
0.034 0.020 -0.062
-0.135 0.299 -0.110 0.005 -0.006 0.235
-0.463 -0.406 -0.824
0.124 0.192 0.260
-0.292 -0.219 -0.236
-0.136 -0.034 -0.302 -0.113 0.153 0.018
48.9 40.0 45.6
62.6 62.4 65.3
0.014 0.008 -0.001 0.019 0.047 0.060
-0.103 -0.223 -0.140 -0.142 -0.077 -0.120
0.233 0.316 0.233 0.253 0.278 0.285
-0.459 -0.333 -0.271 -0.525 -0.448 -0.302
0.005 0.074 0.158 0.125 0.306 0.349
-0.171 -0.271 -0.219 -0.144 -0.098 -0.120
0.066 0.157 0.124 0.172 0.430 0.485
-0.001 0.018 0.024 0.030 0.130 0.148
55.6 54.4 53.3 44.4 63.3 60.0
64.2 64.1 65.1 66.3 66.9 69.1
-0.088 -0.053 -0.088 -0.029 -0.028 -0.025
-0.108 -0.106 0.119 -0.184 -0.033 0.191
0.267 0.277 0.266 0.346 0.375 0.403
-1.000 -1.000 -1.000 0.094 0.281 0.877
0.032 0.466 0.061 0.408 0.821 0.710
-0.258 -0.173 -0.285 0.036 0.092 0.120
0.713 0.556 0.689 0.378 0.565 0.691
0.080 0.110 0.112 0.175 0.305 0.407
56.7 56.7 60.0 63.3 66.7 76.7
67.0 70.4 67.8 70.1 76.7 79.5
Open-weight agents Gemma2-2B DeepSeek-R1-Qwen-14B Qwen3-8B Llama-3.1-8B Phi-4-14B Qwen3-32B API agents GLM-5.1 Kimi-2.6 GLM-4.7 DeepSeek-V4-Pro GPT-5.5 Claude Sonnet 4.6
Table 2 AgentHPOBench results under the limited budget protocol for conventional HPO methods, open-weight agents, and API agents. Category columns report mean bounded normalized score. BWR and MAA are computed over all 30 tasks. Bold indicates the best performance across all evaluated methods.
status. These records preserve the complete sequence of interventions and support consistent auditing across tasks and agents. The harness also enforces comparability between models with open weights and API agents. All agents are evaluated using the same task definitions, budget setting, number of interventions, allowed intervention space, scoring rules, and result schema. The different backends receive the same task information through interfaces specific to each backend. Their outputs are parsed into a common configuration schema and validated against Ω𝑡 before execution. Before scoring, the harness audits all task records. The audit verifies the presence of the baseline and the required intervention records, compliance with the result schema, correct metric extraction, and consistency with the reference baseline and anchor. Only complete records that pass these checks are included in aggregate scoring.
3.4
Scoring
Because the benchmark tasks use heterogeneous metrics, we evaluate each task using the audited result after the final intervention. For task 𝑡 under budget setting 𝑟, let 𝑏 𝑡,𝑟 = 𝑦𝑡,0 denote the reference baseline, 𝑠 𝑡,𝑟 = 𝑦𝑡,𝐾 denote the final result, and 𝑎 𝑡 denote the repository anchor. Before scoring, these values are expressed on the same numerical scale and oriented so that larger values indicate better performance. We denote the oriented values by 𝑏˜ 𝑡,𝑟 , 𝑠˜ 𝑡,𝑟 , and 𝑎˜ 𝑡 . We use mean bounded normalized score and baseline win rate as the primary metrics. The normalized score is 𝑠˜ 𝑡,𝑟 − 𝑏˜ 𝑡,𝑟 NS𝑡,𝑟 = , (6) 𝑎˜ 𝑡 − 𝑏˜ 𝑡,𝑟 where the audit ensures that the denominator is positive and nonzero. To reduce the influence of tasks with a
6
small baseline to anchor gap, we bound each task score to [−1, 1]: BNS𝑡,𝑟 = min{1, max{−1, NS𝑡,𝑟 }}.
(7)
The mean bounded normalized score (MBNS) is 𝑇
MBNS𝑟 =
1Õ BNS𝑡,𝑟 , 𝑇
(8)
𝑡=1
where 𝑇 is the number of evaluated tasks. A bounded score of 0 matches the baseline, 1 reaches or exceeds the anchor, and a negative value indicates degradation. The baseline win rate (BWR) is defined as 𝑇
BWR𝑟 =
1Õ 1 𝑠˜ 𝑡,𝑟 > 𝑏˜ 𝑡,𝑟 . 𝑇
(9)
𝑡=1
It measures the proportion of tasks for which the final result improves over the reference baseline. As a secondary metric, we report anchor attainment in the original metric direction:
( AA𝑡,𝑟 =
𝑠 𝑡,𝑟 /𝑎 𝑡 , 𝑎 𝑡 /𝑠 𝑡,𝑟 ,
for higher is better metrics, for lower is better metrics.
(10)
Mean anchor attainment (MAA) is defined as 𝑇
MAA𝑟 =
1Õ AA𝑡,𝑟 . 𝑇
(11)
𝑡=1
An anchor attainment of 100% indicates that the final result matches the repository anchor, while values above indicate that it exceeds the anchor.
4
Experiments
Experimental Setup and Metrics. We evaluate agents on AgentHPOBench across seven categories: NLP, CV, TS, Graph, RL, LLM, and SL. Unless otherwise stated, we use the limited budget protocol. The harness first executes the reference baseline and then requests five sequential interventions from the agent. Under this protocol, both the baseline and each intervention use approximately 10% of the training budget of the corresponding original experiment. This setting retains executable training and empirical feedback for each task while making broad evaluation across repositories and agents computationally feasible. Before each intervention, the agent receives the accumulated trace, including previous configurations, target and auxiliary metrics, and execution logs. All task scores use the result after the final intervention rather than the best intermediate result. We report mean bounded normalized score (MBNS), baseline win rate (BWR), and mean anchor attainment (MAA) as the metrics. Category scores average the tasks within each category, whereas overall scores are computed directly over all 30 tasks, giving each task equal weight. Agents and Implementation Details. We evaluate both models with open weights and API agents. The models with open weights include Qwen3-8B and Qwen3-32B [32], Gemma2-2B [33], DeepSeek-R1-Qwen-14B [34], Phi-4-14B [35], and Llama-3.1-8B-Instruct [36]. The API agents include DeepSeek-V4-Pro, GPT-5.5, GLM-4.7, GLM-5.1, Kimi-2.6, and Claude Sonnet 4.6. We additionally compare these agents with random search [20], TPE [22], and a BOHB-style method [25] as conventional HPO baselines. All agents and HPO baselines use the same task definitions, reference baselines, intervention spaces, number of configuration evaluations, budget 7
Mean bounded normalized score
Agents
Overall metrics
NLP (3) CV (5) TS (7) Graph (2) RL (3) LLM (4) SL (6) Overall BWR (%) MAA (%)
DeepSeek-R1-Qwen-14B Qwen3-32B Claude Sonnet 4.6
0.029 0.029 0.722
-0.125 0.369 0.145 0.293 0.266 0.455
-0.094 -0.594 0.502
0.185 0.488 0.754
-0.145 -0.015 0.184
0.448 0.442 0.582
0.151 0.191 0.472
50.0 56.7 76.7
78.6 81.8 89.1
Table 3 Full-budget final-step AgentHPOBench results.
Harness
Mean bounded normalized score
Overall metrics
NLP (3) CV (5) TS (7) Graph (2) RL (3) LLM (4) SL (6) Overall BWR (%) MAA (%)
Claude Sonnet 4.6 Claude Code CLI AgentHPOBench
0.090 -0.025
0.120 0.191
0.295 0.403
0.518 0.877
0.800 0.710
-0.170 0.120
0.712 0.691
0.332 0.407
76.7 76.7
82.2 79.5
0.050 -0.028
0.108 0.318 -0.033 0.375
0.565 0.281
0.666 0.821
-0.145 0.092
0.418 0.565
0.266 0.305
70.0 66.7
78.7 76.7
GPT-5.5 Codex CLI AgentHPOBench
Table 4 Harness ablation under the limited-budget protocol.
settings, and scoring rules. The conventional HPO baselines operate on previously evaluated configurations and target metric values, whereas agents additionally receive task context, auxiliary metrics, and execution logs. We repeat all locally executable open-weight agents and conventional HPO baselines using three experiment seeds, {0, 1, 42}. For these methods, the implementation, model checkpoint or optimization algorithm, runtime environment, and random state can be explicitly controlled, allowing the repetitions to measure sensitivity to benchmark stochasticity. API agents are evaluated once on the canonical benchmark instance. Further details are provided in Appendix. Main Results. Table 2 summarizes the results under the limited-budget protocol. Claude Sonnet 4.6 achieves the highest overall MBNS of 0.407, the highest BWR of 76.7%, and the highest MAA of 79.5%. Among open-weight agents, Qwen3-32B obtains the highest overall MBNS (0.148) and MAA (69.1%), whereas Phi-4-14B achieves the highest BWR (63.3%). The conventional HPO methods generally trail the stronger agents in overall performance under the same five intervention opportunities. Among these baselines, the BOHB variant obtains the highest overall MBNS (0.018) and MAA (65.3%), while random search achieves the highest BWR (48.9%). Performance varies substantially across task categories, and no method performs best in every category. Qwen3-32B achieves the highest MBNS on NLP. Claude Sonnet 4.6 leads on CV, TS, Graph, and LLM, GPT-5.5 leads on RL, and GLM-5.1 leads on SL. These results demonstrate that the relative effectiveness of agents and conventional HPO methods depends strongly on the experimental domain. The category-level HPO results also show that conventional optimization remains competitive in specific domains despite its lower overall performance. Aggregate leadership does not imply consistent improvement across individual tasks. Although Claude Sonnet 4.6 obtains the strongest aggregate results, its BWR of 76.7% means that its final configurations exceed the reference baseline on 23 of the 30 tasks. Qwen3-32B obtains a mean BWR of 60.0% across the three controlled seeds. The remaining failures, together with the negative and near-zero category scores, show that even strong agents can produce final configurations that do not improve the baseline. These findings indicate that the main challenge is not only to identify useful interventions, but also to refine and preserve their benefits throughout the sequential optimization process.
8
0.60
Mean bounded normalized score (MBNS)
Mean bounded normalized score (MBNS)
0.30
Qwen3-32B
0.15 Phi-4-14B
Qwen3-8B
0.00 DeepSeek-R1-14B
Llama-3.1-8B
-0.15 Gemma2-2B
-0.30 Baseline
Int. 1
Int. 2
Int. 3
Int. 4
Claude Sonnet 4.6
0.40
GPT-5.5 DeepSeek-V4-Pro Kimi-2.6
0.20
GLM-4.7 GLM-5.1
0.00
-0.20 Baseline
Int. 5
Int. 1
Int. 2
Int. 3
Int. 4
Sequential trial
Sequential trial
(a) Open-weight agents
(b) API-based agents
Int. 5
Figure 3 Sequential optimization trajectories under the limited budget protocol. Lines show MBNS from the reference baseline through five interventions, and shaded regions indicate cross-task standard error. Feedback setting Standard feedback No intermediate feedback
Mean bounded normalized score
Overall metrics
NLP (3) CV (5) TS (7) Graph (2) RL (3) LLM (4) SL (6) Overall BWR (%) MAA (%) 0.060 0.021
-0.120 0.285 -0.227 0.211
-0.302 -0.333
0.349 0.346
-0.120 -0.234
0.485 0.286
0.148 0.052
60.0 50.0
69.1 65.2
Table 5 Ablation of intermediate experimental feedback for Qwen3-32B under the limited budget protocol.
Full Budget Results. To examine how the training budget affects sequential optimization, we evaluate three selected agents using the full training budget of the original repositories, as shown in Table 3. The full budget increases both MBNS and MAA for all three agents. The MBNS values of Qwen3-32B, DeepSeek-R1-Qwen-14B, and Claude Sonnet 4.6 increase from 0.148, 0.018, and 0.407 under the limited-budget protocol to 0.191, 0.151, and 0.472, respectively. Claude Sonnet 4.6 also achieves the highest full-budget MAA of 89.1%. However, BWR does not improve uniformly: Claude Sonnet 4.6 remains at 76.7%, while Qwen3-32B and DeepSeek-R1-Qwen-14B decrease from 60.0% and 54.4% to 56.7% and 50.0%, respectively. These changes are not contradictory because the three metrics capture different aspects of performance. Full training can change both the baseline performance and the final performance obtained by an agent. Consequently, an agent may finish closer to the anchor while achieving a larger average normalized improvement, yet exceed the corresponding full-budget baseline on fewer tasks. The limited- and full-budget settings therefore provide complementary evidence about optimization under different computational constraints. Harness Ablation. We examine how the execution harness affects performance by comparing the native AgentHPOBench harness with corresponding CLI harnesses under the same tasks, intervention spaces, budgets, and scoring rules. The results are shown in Table 4. For Claude Sonnet 4.6, the native harness achieves a higher MBNS than Claude Code CLI (0.407 vs. 0.332), while both harnesses obtain the same BWR of 76.7%. Claude Code CLI achieves a higher MAA (82.2% vs. 79.5%). Similarly, for GPT-5.5, the native harness achieves a higher MBNS than Codex CLI (0.305 vs. 0.266), whereas Codex CLI obtains a higher BWR (70.0% vs. 66.7%) and MAA (78.7% vs. 76.7%). These results show that the native harness produces greater relative improvement over the reference baseline for both agents, while the CLI harnesses achieve higher absolute anchor attainment. However, no harness consistently dominates across all metrics or task categories. Harness choice can therefore affect the configurations produced during sequential optimization and the resulting performance. This sensitivity motivates the use of a common harness for the main comparison and the separate reporting of harness effects.
9
Sequential Optimization Trajectories. Figure 3 reports the MBNS obtained after the reference baseline and each subsequent intervention. Claude Sonnet 4.6 and Qwen3-32B achieve substantial gains during the first two interventions, while GPT-5.5 improves more gradually across the trajectory. Other agents exhibit less stable behavior. DeepSeek-R1-Qwen-14B initially falls below the baseline before recovering, whereas Gemma2-2B and Llama-3.1-8B fluctuate around the baseline for most of the trajectory. Several agents also show nonmonotonic refinement. For example, Phi-4-14B loses part of its early improvement before recovering at the final intervention, while Kimi-2.6 declines at the fourth intervention and subsequently rebounds. In contrast, Claude Sonnet 4.6 and GPT-5.5 achieve their highest MBNS at the final intervention, showing that later feedback can still produce useful refinements. Overall, current agents can identify beneficial configurations, but their ability to preserve and improve earlier gains remains inconsistent across models. Intermediate Feedback Ablation. We isolate the contribution of intermediate experimental feedback by evaluating Qwen3-32B under an otherwise identical protocol. The baseline observation and all other task inputs remain available, but the metrics and execution logs produced by each intervention are withheld from subsequent decisions. As shown in Table 5, removing intermediate feedback reduces overall MBNS from 0.148 to 0.052 and BWR from 60.0% to 50.0%, while MAA decreases from 69.1% to 65.2%. Standard feedback yields higher MBNS in all seven task categories. These results provide evidence that access to intermediate outcomes improves sequential optimization for Qwen3-32B under the limited-budget protocol.
5
Conclusion
In this work, we present AgentHPOBench, a benchmark for evaluating whether agents can improve executable ML experiments through sequential hyperparameter interventions. AgentHPOBench moves beyond conventional HPO benchmarks with clean black box objective interfaces by requiring agents to interpret metrics and logs from research repositories and convert this evidence into valid configurations. Experiments across 30 tasks show that current agents can improve reference baselines, but performance remains uneven across task domains, resource settings, and execution harnesses, and often remains below the reported reference performance. These findings establish HPO in research repositories as a challenging setting and provide a foundation for developing agents with stronger capabilities for experimental diagnosis and sequential decision making.
10
References [1] Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Mikhail Plekhanov, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, et al. Mlgym: A new framework and benchmark for advancing ai research agents. In Second Conference on Language Modeling, 2025. [2] Rushi Qiang, Yuchen Zhuang, Yinghao Li, Sagar V. K. Dingu, Rongzhi Zhang, Changhao Li, Ian Shu-Hei Wong, Sherry Yang, Percy Liang, Chao Zhang, and Bo Dai. MLE-Dojo: Interactive environments for empowering LLM agents in machine learning engineering. In Advances in Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=5W5mFU4oMO. Datasets and Benchmarks Track. [3] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s ability to replicate AI research. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 56843–56873. PMLR, 2025. URL https: //proceedings.mlr.press/v267/starace25a.html. [4] Gyeongwon James Kim, Alex Wilf, Louis-Philippe Morency, and Daniel Fried. From reproduction to replication: Evaluating research agents with progressive code masking. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=qBcHWGBnIb. [5] Hjalmar Wijk, Tao Roa Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Joshua M Clymer, Jai Dhyani, et al. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts. In International Conference on Machine Learning, pages 66772–66832. PMLR, 2025. [6] Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. MLR-Bench: Evaluating AI agents on open-ended machine learning research. In Advances in Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=JX9DE6colf. Datasets and Benchmarks Track. [7] Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka, Alberto Pepe, Alexis AudranReiss, Muna Aghamelu, Nicolas Baldwin, Lucia Cipolina-Kun, Jean-Christophe Gagnon-Audet, Chee Hau Leow, Sandra Lefdal, Hossam Mossalam, Abhinav Moudgil, Saba Nazir, Emanuel Tewolde, Isabel Urrego, Jordi Armengol Estape, Amar Budhiraja, Gaurav Chaurasia, Abhishek Charnalia, Derek Dunfield, Karen Hambardzumyan, Daniel Izcovich, Martin Josifoski, Ishita Mediratta, Kelvin Niu, Parth Pathak, Michael Shvartsman, Edan Toledo, Anton Protopopov, Roberta Raileanu, Alexander Miller, Tatiana Shavrina, Jakob Foerster, and Yoram Bachrach. AIRS-Bench: A suite of tasks for frontier AI research science agents, 2026. URL https://arxiv.org/abs/2602.06855. arXiv preprint arXiv:2602.06855. [8] Katharina Eggensperger, Philipp Mueller, Neeratyoy Mallik, Matthias Feurer, Rene Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. URL https://openreview.net/forum?id=1k4rJYEwda-. [9] Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl. YAHPO gym - an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. In Proceedings of the First International Conference on Automated Machine Learning, Proceedings of Machine Learning Research. PMLR, 2022. URL https://proceedings.mlr.press/v188/pfisterer22a.html. [10] Sebastian Pineda Arango, Hadi Samer Jomaa, Martin Wistuba, and Josif Grabocka. Hpo-b: A large-scale reproducible benchmark for black-box hpo based on openml. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [11] Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-PyTorch: Multi-fidelity metalearning for efficient and robust AutoDL. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):3079–3090, 2021. URL https://ieeexplore.ieee.org/document/9382913. [12] Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, and Frank Hutter. JAHS-Bench-201: A foundation for research on joint architecture and hyperparameter search. In Advances in Neural Information Processing Systems, 2022. URL https://papers.nips.cc/paper_files/paper/2022/hash/ fd78f2f65881c1c7ce47e26b040cf48f-Abstract-Datasets_and_Benchmarks.html.
11
[13] Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, and Marius Lindauer. DACBench: A benchmark library for dynamic algorithm configuration. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 1668–1674, 2021. doi: 10.24963/IJCAI.2021/230. [14] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 20271–20309. PMLR, 2024. URL https://proceedings.mlr. press/v235/huang24y.html. [15] Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. ML-Bench: Evaluating large language models and agents for machine learning tasks on repository-level code, 2024. URL https://arxiv.org/abs/2311.09835. arXiv preprint arXiv:2311.09835. [16] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. Mle-bench: Evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representations (ICLR 2025), 2025. URL https://dblp.org/rec/conf/iclr/ChanCJASMSLMPMW25.html. [17] Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE-Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=BsMMc4MEGS. [18] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan N. van Rijn, and Joaquin Vanschoren. OpenML benchmarking suites. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. URL https://openreview.net/forum? id=OCrD8ycKjG. [19] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS-Bench-101: Towards reproducible neural architecture search. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7105–7114. PMLR, 2019. URL https: //proceedings.mlr.press/v97/ying19a.html. [20] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281–305, 2012. URL https://jmlr.org/papers/v13/bergstra12a.html. [21] Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization, volume 6683 of Lecture Notes in Computer Science, pages 507–523. Springer, 2011. doi: 10.1007/978-3-642-25566-3_40. [22] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011. [23] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, volume 25, 2012. URL https://papers.nips.cc/paper/ 4522-practical-bayesian-optimization-of-machine-learning-algorithms. [24] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. The journal of machine learning research, 18(1):6765–6816, 2017. [25] Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1437–1446. PMLR, 2018. URL https://proceedings.mlr.press/v80/falkner18a.html. [26] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks, 2017. URL https://arxiv.org/abs/1711.09846. arXiv preprint arXiv:1711.09846. [27] Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg
12
Kochanski, Arnaud Doucet, Marc’Aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Processing Systems, 35:32053–32068, 2022. [28] Tennison Liu, Nicolas Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=OOxotBmGol. [29] Michael R. Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization, 2024. URL https://arxiv.org/abs/2312.04528. arXiv preprint arXiv:2312.04528. [30] Siyi Liu, Chen Gao, and Yong Li. AgentHPO: Large language model agent for hyper-parameter optimization. In Conference on Parsimony and Learning, volume 280 of Proceedings of Machine Learning Research, pages 1146–1169. PMLR, 2025. URL https://proceedings.mlr.press/v280/liu25c.html. [31] Kanan Mahammadli and Seyda Ertekin. Sequential large language model-based hyper-parameter optimization, 2025. URL https://arxiv.org/abs/2410.20302. arXiv preprint arXiv:2410.20302. [32] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. arXiv preprint arXiv:2505.09388. [33] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. [34] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [35] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. URL https://arxiv.org/abs/2412.08905. arXiv preprint arXiv:2412.08905. [36] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [37] Ivan Drokin. Kolmogorov-Arnold convolutions: Design principles and empirical studies, 2024. URL https: //arxiv.org/abs/2407.01092. arXiv preprint arXiv:2407.01092. [38] Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, and Yuqing Yang. Agent lightning: Train any AI agents with reinforcement learning, 2025. URL https://arxiv.org/abs/2508.03680. arXiv preprint arXiv:2508.03680. [39] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 53140–53164. PMLR, 2024. URL https: //proceedings.mlr.press/v235/woo24a.html. [40] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 10148–10167. PMLR, 2024. URL https://proceedings.mlr.press/v235/das24c. html.
13
[41] Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2526–2547, 2025. [42] Keller Jordan. 94% on CIFAR-10 in 3.29 seconds on a single GPU, 2024. URL https://arxiv.org/abs/2404. 00498. arXiv preprint arXiv:2404.00498. [43] Zhonghao Wang, Danyu Sun, Sheng Zhou, Haobo Wang, Jiapei Fan, Longtao Huang, and Jiajun Bu. NoisyGL: A comprehensive benchmark for graph neural networks under label noise. In Advances in Neural Information Processing Systems, volume 37, pages 38142–38170, 2024. URL https://openreview.net/forum?id=brxBxj4Dv3. [44] Ricardo Knauer, Marvin Grimm, and Erik Rodner. PMLBmini: A tabular classification benchmark suite for data-scarce applications. In AutoML 2024 Workshop, 2024. URL https://openreview.net/forum?id=XctSyEsBzx. [45] Vitaliy Kinakh and Slava Voloshynovskiy. Tabular data generation using binary diffusion. In Table Representation Learning Workshop at NeurIPS 2024, 2024. URL https://openreview.net/forum?id=lv234j8Hq7. [46] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. Advances in Neural Information Processing Systems, 37:107547–107603, 2024. [47] James Harrison, John Willes, and Jasper Snoek. Variational bayesian last layers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Sx7BIiPzys. [48] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing tabular deep learning with parameterefficient ensembling. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=Sd4wYYOhmY. [49] Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yunzhong Qiu, Jianmin Wang, and Mingsheng Long. TimeXer: Empowering transformers for time series forecasting with exogenous variables. In Advances in Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=INAeUQ04lT. [50] Yu-Xuan Huang, Wen-Chao Hu, En-Hao Gao, and Yuan Jiang. ABLkit: A python toolkit for abductive learning. Frontiers of Computer Science, 18(6):186354, 2024. doi: 10.1007/s11704-024-40085-7. [51] Yuankai Luo, Lei Shi, and Xiao-Ming Wu. Classic GNNs are strong baselines: Reassessing GNNs for node classification. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=xkljKdGe4E. [52] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, 2025. URL https://arxiv.org/abs/2409.19256. [53] Alexia Jolicoeur-Martineau, Kilian Fatras, and Tal Kachman. Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 1288–1296. PMLR, 2024. URL https: //proceedings.mlr.press/v238/jolicoeur-martineau24a.html. [54] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In Advances in Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=gojL67CfS8. [55] Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, Mohan Wang, Josef Dai, Tianyi Qiu, Hua Xu, Dong Li, Weipeng Chen, Jun Song, Bo Zheng, and Yaodong Yang. Align anything: Training all-modality models to follow instructions with language feedback, 2024. URL https://arxiv.org/abs/2412.15838. arXiv preprint arXiv:2412.15838. [56] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. Chronos: Learning the language of time series, 2024. URL https://arxiv.org/abs/2403.07815.
14
[57] Ahmad Bdeir, Kristian Schwethelm, and Niels Landwehr. Fully hyperbolic convolutional neural networks for computer vision. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=ekz1hN5QNh. [58] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. iTransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=JePfAI8fah. [59] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and Jun Zhou. TimeMixer: Decomposable multiscale mixing for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=7oLshfEIC2. [60] Pin-Yen Huang, Szu-Wei Fu, and Yu Tsao. RankUp: Boosting semi-supervised regression with an auxiliary ranking classifier. In Advances in Neural Information Processing Systems, 2024. URL https://openreview.net/forum? id=d2lPM1Aczs. [61] Shengsheng Lin, Weiwei Lin, Wentai Wu, Haojun Chen, and Junjie Yang. SparseTSF: Modeling long-term time series forecasting with 1k parameters. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 30211–30226. PMLR, 2024. URL https://proceedings.mlr. press/v235/lin24n.html.
15
# Repository
Task
Cat.
Dataset / Task
Metric
Direction Anchor
1 llm.c FineWeb Pretraining NLP FineWeb sample + HellaSwag validation loss lower 2 torch-conv-kan ConvKAN CIFAR-10 CV CIFAR-10 accuracy higher 3 open-r1 Open-R1 MATH-500 LLM MATH-500 exact match higher 4 agent-lightning Room Selector Tuning RL APO room selector validation accuracy higher 5 uni2ts Uni2TS ETTh1 Forecasting TS ETTh1 LSF forecast error lower 6 timesfm TimesFM Long Horizon TS ETTh1 long horizon WAPE lower 7 ModernBERT ModernBERT MNLI NLP GLUE MNLI accuracy higher 8 cifar10-airbench AirBench CIFAR-10 CV CIFAR-10 accuracy higher 9 NoisyGL NoisyGL Cora GCN Graph Cora + 30% label noise accuracy higher 10 TabMini TabMini Promoters SL molecular biology promoters AUC higher 11 binary-diffusion-tabular Adult Tabular Diffusion SL Adult test accuracy higher 12 xlstm xLSTM Parity NLP formal language Parity scaled accuracy higher 13 vbll VBLL Yacht Regression SL UCI Yacht RMSE lower 14 tabm TabM California Housing SL California Housing RMSE lower 15 TimeXer TimeXer PJM Forecasting TS EPF / PJM MSE lower 16 ABLkit ABLkit HWF Reasoning SL HWF reasoning accuracy higher 17 tunedGNN tunedGNN Cora GCN Graph Cora accuracy higher 18 verl verl GRPO GSM8K LLM GSM8K accuracy higher 19 ForestDiffusion ForestDiffusion Iris SL Iris F1_fake higher 20 VAR VAR ImageNet 256 CV ImageNet-1K 256 FID lower 21 align-anything RAGEN Bandit Alignment RL RAGEN Bandit success rate higher 22 chronos-forecasting Chronos Weather Forecasting TS monash weather WQL lower 23 HyperbolicCV HyperbolicCV CIFAR-100 CV CIFAR-100 accuracy higher 24 iTransformer iTransformer ETTm2 Forecasting TS ETTm2 horizon 96 MSE lower 25 TimeMixer TimeMixer ETTm2 Forecasting TS ETTm2 horizon 96 MSE lower 26 ART ART 2048 RL 2048 game rollouts win rate higher 27 open-r1-multimodal Multimodal Open-R1 MathVista LLM MathVista-mini accuracy higher 28 semi-supervised-regression RankUp UTKFace Regression CV UTKFace MAE lower 29 SparseTSF SparseTSF ETTm1 Forecasting TS ETTm1 horizon 96 MSE lower 30 simpleRL-reason SimpleRL MATH-500 LLM MATH-500 accuracy higher
3.425 84.170 83.100 0.721 0.375 0.509 90.400 94.010 71.060 0.930 85.740 1.000 0.860 0.441 0.093 99.200 85.100 86.100 0.970 3.550 1.000 0.148 78.070 0.180 0.175 0.600 51.600 4.851 0.314 34.400
Table 6 AgentHPOBench task suite. Each task is built from one executable repository and one scoreable target metric.
6
Appendix
This supplementary material provides additional details on the experimental protocol, task suite, reference baselines, and implementation settings. It also presents complete results, supplementary analyses, and qualitative examples that complement the findings in the main paper.
7
AgentHPOBench Task Suite
AgentHPOBench contains 30 tasks drawn from 30 distinct executable ML repositories associated with recent papers when available [37–61]. Table 6 lists the full task suite. The anchor is taken from a paper, an official repository report, or a documented full setting reproduction. Table 7 reports the reference baseline performance used for scoring under the limited- and full-budget settings. The baseline configuration of each task is fixed across settings. The full-budget protocol expands the task-specific training budget where applicable. For pretrained inference tasks and tasks without a distinct scalable training stage, the evaluation protocol remains unchanged, so identical baseline values are expected. For example, ART 2048 uses the same fixed evaluation of 25 games with at most 70 moves per game in both settings, and its baseline heuristic wins none of these games, yielding a win rate of zero. For the full-budget study, each selected agent executes the fixed baseline configuration, and the median of the three observations is used as the common full-budget reference for that task. Values are reported in the metric units defined in Table 6 and rounded to three decimal places. All scores are computed from the corresponding full-precision values.
8
Agent Prompt Template
AgentHPOBench standardizes the agent interface through a structured decision schema. At each intervention step, the task script constructs a task-specific context block that contains the task description, target metric, metric direction, current configuration, allowed search space, paper or repository anchor, baseline result, and previous trial history when available. This context is then passed to the decision backend, and the returned 16
Task
Limited
Full
Task
Limited
Full
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI
4.344 28.970 1.563 0.414 1.155 0.551 38.961
3.698 35.500 2.200 0.379 1.144 0.551 63.495
97.450 83.500 84.685 0.901 70.713 0.168 0.372
97.600 83.500 84.230 0.901 70.713 0.178 0.372
AirBench CIFAR-10 NoisyGL Cora GCN
89.090 70.000
93.680 70.000
16.390 0.183
56.630 0.183
TabMini Promoters
0.928
0.928
0.178
0.178
Adult Tabular Diffusion xLSTM Parity
74.106 0.011
85.519 0.067
0.000 48.800
0.000 48.800
VBLL Yacht Regression
1.487
0.467
34.542
6.427
TabM California Housing
0.536
0.533
0.341
0.340
TimeXer PJM Forecasting
0.133
0.114
ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
15.800
5.200
(𝑟)
Table 7 Reference baseline performance 𝑦𝑡,0 under the limited and full budget settings. Each value is reported in the original metric unit of the corresponding task.
intervention is parsed into a structured configuration before execution. Figure 4 shows the shared prompt template used for local open-weight agents, while Figure 5 shows the compact decision prompt used for API-based agents. Figure 6 provides an example task-specific context, illustrating how concrete trial evidence and the allowed search space are supplied before requesting the next intervention. We use “search space” to denote the task-specific allowed intervention space, namely the configurable fields exposed to the agent and the valid values for each field, rather than the decoding hyperparameters of the agent. The local and API templates preserve the interfaces used by their respective agent implementations rather than enforcing identical surface wording. Prompt construction, response parsing, configuration validation, retries, and other harness behavior are therefore part of the evaluated agent system. The resulting comparisons should be interpreted as comparisons between complete agent pipelines, not as isolated rankings of the underlying language models.
9
Implementation and Reproducibility Details
Common execution protocol. Unless stated otherwise, all results use the limited-budget protocol. Each task first executes a fixed reference baseline and then permits five sequential interventions. The baseline and every intervention use the same task-specific data split, metric, intervention space, and execution budget. The limited setting uses approximately 10% of the corresponding original training or evaluation budget. The full-budget study changes only this execution budget and retains the task definition, prompt, intervention space, five-decision protocol, result schema, and scoring pipeline. Each decision must return one complete configuration in the structured NEW_CONFIG format. Omitted fields retain their current values, and configurations are validated and clamped to the task-specific discrete intervention space before execution. The reported task result is the metric after intervention five, rather than the best intermediate metric. The benchmark optimizes the objective exposed by each upstream repository. Some repositories report a testset metric, or a test metric at the checkpoint selected by validation performance, and this repository-defined objective is visible during sequential decision making. AgentHPOBench therefore evaluates optimization 17
Local open-weight agent prompt 1 You are an expert machine learning researcher helping optimize hyperparameters for model
training. 2 3 ## Task 4 Optimize hyperparameters to {optimization_goal} the target metric within the remaining budget 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
. - Metric Direction: {metric_direction} ## Current Training Progress - Current Epoch: {epoch} - Budget Remaining: {budget_remaining} epochs out of {total_budget} total - Decision Number: {decision_number} ## Recent Training Logs {training_log} ## Current Hyperparameters {current_config} ## Hyperparameter Search Space {search_space} ## Instructions - Be conservative unless the run is clearly plateauing or unstable. - Only return values inside the search space. - If training is improving normally, keeping the current config is acceptable. - Output plain text only. - Do not use markdown code fences, XML tags, or extra sections. - Follow the exact headers below. ## Response Format REASONING: <short explanation> NEW_CONFIG: {"learning_rate": 0.05, "batch_size": 128, "optimizer": "sgd"} CONFIDENCE: 0.8
Figure 4 Prompt template used for local open-weight agents. The optimization goal is set to maximize for higher-is-better metrics and minimize for lower-is-better metrics.
API-based agent prompt 1 2 3 4 5 6 7 8 9 10 11 12 13
Task: choose the next hyperparameter config for {task_name}. Return exactly one JSON object containing only proposed hyperparameter keys and values. Current config: {current_config} Search space: {search_space} Budget remaining: {budget_remaining} of {total_budget}. Training history: {training_log}
Figure 5 Compact decision prompt used for API-based agents.
of an observable experimental objective. It does not provide a separate hidden test set and should not be interpreted as estimating generalization after adaptive model selection.
18
Example task-specific context 1 You are choosing one hyperparameter configuration for a budget-limited reproduction benchmark
. 2 3 Task: SparseTSF ETTm1 multivariate long-term forecasting, pred_len=96. 4 Metric: test_mse_at_best_val over the fixed epoch budget, lower is better. 5 Paper anchor: ICML 2024 SparseTSF official long-term forecasting table reports ETTm1 horizon
-96 MSE=0.314. 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Budget baseline: test_mse_at_best_val=0.340816, config={...} Trial history: Trial 0 baseline: config={...} test_mse=0.340816 val_mse=0.440882 Trial 1 intervention_1: config={...} test_mse=0.340816 val_mse=0.440882 Trial 2 intervention_2: config={...} test_mse=0.318475 val_mse=0.407163 Search space: {...} Return exactly one line: NEW_CONFIG: {"lr": 0.02, "weight_decay": 0.0, "batch_size": 256, ...} Choose the config most likely to reduce test_mse_at_best_val under the fixed budget. Do not change epochs, data split, dataset, seq_len, pred_len, model family, or metric.
Figure 6 Example task-specific context used to request the next intervention.
Open-weight decision models. Table 8 lists the exact checkpoints and realized generated-token usage for the six open-weight agents under the limited-budget protocol. We load all checkpoints with Hugging Face Transformers using the checkpoint’s chat template and bfloat16 weights. Agent decoding is deterministic: temperature is 0, sampling is disabled, and optional thinking output is disabled so that generation begins with the requested structured decision. Token counts are computed from the generated decision text using the tokenizer associated with each checkpoint. They exclude input-prompt tokens. Each checkpoint contributes 450 decisions from 30 tasks, five interventions, and three experiment seeds. API-based agents. We evaluate DeepSeek-V4-Pro, GPT-5.5, GLM-4.7, GLM-5.1, Kimi-2.6, and Claude Sonnet 4.6 using the same five logical decisions, task contexts, output schema, and configuration validation as the open-weight agents. The request temperature is set to 0 when the endpoint exposes this control. For every decision, the result trace records the provider and model identifier returned by the harness, the raw response, timestamp, token-usage fields when available, and retry or error metadata. Each API agent is evaluated with one complete, timestamped, audited 30-task run. This choice is not presented as a controlled seed replicate: hosted endpoints do not expose an immutable checkpoint build, serving replica, batching state, or a reproducible end-to-end random seed, and these service-side states may change independently of the benchmark. Repeating an API request would therefore measure a mixture of model and serving changes rather than the benchmark stochasticity isolated by the controlled local runs. Table 9 reports the provider-recorded usage of the API evaluation campaign. 19
Hugging Face checkpoint Qwen/Qwen3-8B google/gemma-2-2b-it meta-llama/Meta-Llama-3.1-8B-Instruct deepseek-ai/DeepSeek-R1-Distill-Qwen-14B microsoft/Phi-4 Qwen/Qwen3-32B
Decisions
Total tokens
Mean
Median
P95
450 450 450 450 450 450
60,636 40,857 53,700 153,673 91,327 54,771
134.7 90.8 119.3 341.5 202.9 121.7
127.0 85.5 123.0 130.0 214.0 125.0
383.5 142.5 209.0 1,380.2 338.5 191.5
Table 8 Realized generated-token usage of the open-weight decision models under the limited-budget protocol. P95 denotes the 95th percentile across individual decisions. API agent Claude Sonnet 4.6 GPT-5.5 DeepSeek-V4-Pro GLM-5.1 Kimi-2.6 GLM-4.7
Requests Total tokens Tokens/request 1,695 1,418 1,321 1,296 1,237 1,231
2,498,302 2,132,497 2,000,752 1,978,744 1,922,535 1,866,319
1,474 1,504 1,515 1,527 1,554 1,516
Table 9 Provider-recorded API usage during the limited-budget evaluation campaign. Request counts include retries and harness-level calls in addition to the five logical decisions per task.
Each API agent makes 150 accepted logical decisions across the 30 tasks, while the campaign records 1,231– 1,695 requests, or approximately 8.2–11.3 requests per accepted decision. The request totals include retries, parsing or validation recovery, and other harness-level calls and therefore are not additional intervention opportunities. Provider-recorded token counts include both prompt and generated tokens and are not directly comparable with the generated-output-only counts in Table 8. Seeds, repetitions, and aggregation. We run every open-weight agent and each conventional HPO baseline with experiment seeds 𝑠 ∈ {0, 1, 42}. The same three seeds are used consistently across the benchmark harness, optimizer, and stochastic task execution, while dataset splits fixed by an upstream protocol remain unchanged. Random search, TPE, and the fixed-budget BOHB-style method use the same reference baseline, discrete intervention space, and five configuration evaluations as the agents. Their proposal rules are detailed in Appendix 11.1. For each controlled method, the bounded normalized score, baseline win indicator, and anchor attainment are computed independently for every task and seed. Category and overall metrics are then computed for each seed and reported as the arithmetic mean across the three seeds. Table 10 additionally reports the corresponding sample standard deviation. This preserves equal weight for every benchmark task. The local agent checkpoints, decoding settings, runtime environment, and task random states are explicitly controlled, so these repetitions quantify sensitivity to benchmark execution stochasticity rather than provider-side variation. No intermediate feedback ablation. The no intermediate feedback condition is paired with the standard feedback run at the task and replicate-seed level. It uses the exact baseline configuration, target metric, and baseline observation recorded by the corresponding standard-feedback run rather than independently re-evaluating the baseline. The baseline metric is therefore available to the agent. However, for all five decisions, the visible history contains only this fixed baseline observation. Metrics, auxiliary outputs, and logs produced by interventions one through four are withheld. The proposed interventions are still executed and recorded normally. Thus, the ablation isolates access to intermediate experimental evidence while holding the initial observation, evaluator seed, task budget, search space, and decision model fixed.
20
Execution environment and reproducibility resources. The controlled experiments run on Linux development machines equipped with NVIDIA H200 GPUs with 143,771 MiB of visible memory. The orchestration layer uses Python 3.10 or newer. Because the 30 tasks depend on heterogeneous upstream repositories, each task adapter invokes its repository-specific Conda environment rather than forcing all tasks into one dependency stack. Model weights and datasets are downloaded before execution, and the reported runs use offline Hugging Face modes. For reproducibility, we record the upstream repository version, benchmark adaptations, environment and asset-preparation requirements, model checkpoint, task seeds, baseline configuration, metric extraction rule, intervention space, execution budget, and scoring reference for every task. Detailed setup instructions and machine-readable task specifications will be provided in the public GitHub repository. Result Validation. We include only task runs that complete the reference baseline and all five interventions, with a valid configuration and metric recorded at every step. Interrupted or malformed runs are excluded and rerun at the task level. All reported aggregates are computed from complete task records. The public GitHub repository includes the corresponding validation and aggregation utilities.
10
Scoring and Aggregation Details (𝑟)
For a fixed budget setting 𝑟, let 𝑦𝑡,0 denote the reference baseline performance for task 𝑡 reported in Table 7, (𝑟)
let 𝑦𝑡,𝑘 denote the result after intervention 𝑘, and let 𝑎 𝑡 denote the paper or repository anchor. The main task (𝑟)
result is 𝑦𝑡,𝐾 , which is obtained after the final intervention rather than selected as the best intermediate result. This convention measures whether an optimizer preserves or refines its improvements after observing the complete trajectory. Before scoring, every baseline, intervention result, and anchor is converted to a common numeric unit within its task. For example, an accuracy represented as a fraction is converted to a percentage when the corresponding anchor is reported as a percentage. Let 𝑔𝑡 (𝑧) = 𝑧 for a metric in which higher values are better and 𝑔𝑡 (𝑧) = −𝑧 for a metric in which lower values are better. We define the normalized score as
NS𝑡,𝑟 =
(𝑟)
(𝑟)
𝑔𝑡 𝑦𝑡,𝐾 − 𝑔𝑡 𝑦𝑡,0
(𝑟)
𝑔𝑡 (𝑎 𝑡 ) − 𝑔𝑡 𝑦𝑡,0
.
(12)
A value of 0 matches the reference baseline, a value of 1 matches the anchor, and a negative value indicates degradation. Because a small baseline to anchor gap can give one task disproportionate influence, we bound each task score before aggregation: BNS𝑡,𝑟 = min{1, max{−1, NS𝑡,𝑟 }} .
(13)
For the task set 𝒯 , the mean bounded normalized score is MBNS𝑟 =
1 Õ BNS𝑡,𝑟 . |𝒯 |
(14)
𝑡∈𝒯
The baseline win rate reports the percentage of tasks for which the final result strictly improves over the reference baseline: i 100 Õ h (𝑟) (𝑟) BWR𝑟 = . (15) 𝕀 𝑔𝑡 𝑦𝑡,𝐾 > 𝑔𝑡 𝑦𝑡,0 |𝒯 | 𝑡∈𝒯
Ties are not counted as wins. Anchor attainment measures absolute performance relative to the anchor in the
21
original metric direction:
( (𝑟) AA𝑡,𝑟 =
𝑦𝑡,𝐾 /𝑎 𝑡 ,
if higher values are better,
(𝑟) 𝑎 𝑡 /𝑦𝑡,𝐾 ,
if lower values are better.
The mean anchor attainment is MAA𝑟 =
(16)
100 Õ AA𝑡,𝑟 . |𝒯 |
(17)
𝑡∈𝒯
Thus, MBNS measures bounded improvement over the common reference baseline, BWR measures the coverage of positive improvements, and MAA measures absolute attainment of reported reference performance. MAA may exceed 100% when a result surpasses its anchor. The reference baseline configuration is fixed across budget settings, but its measured performance can change (𝑟) with the execution budget. Within each budget setting, the same 𝑦𝑡,0 is used for all agents and conventional (𝑟)
optimizers. For the intervention level tables below, BNS𝑡,𝑟,𝑘 and AA𝑡,𝑟,𝑘 are computed by replacing 𝑦𝑡,𝐾 with (𝑟)
𝑦𝑡,𝑘 in the definitions above.
11
Additional Experimental Protocols and Results
This section provides implementation details and supporting results for the conventional HPO baselines and the intermediate feedback ablation reported in the main paper. All methods use the task-specific intervention spaces defined by the benchmark task specifications, begin from the same limited-budget reference baseline, receive five intervention opportunities, and are scored using the result after the fifth intervention. Table 6 summarizes the corresponding tasks, objectives, metric directions, and anchors.
11.1
Conventional HPO Baselines
Implementation. Random search samples every configurable field independently and uniformly from its allowed discrete values. We evaluate random search with seeds 0, 1, and 42. The TPE baseline uses the observations available before each intervention to rank previous configurations in the direction of the target metric. Proposals are sampled uniformly until at least two executed observations are available. TPE then assigns the best ⌈0.35𝑛⌉ of the 𝑛 observations to the good set and samples each field according to its smoothed good to bad frequency ratio. The additive smoothing constant is 1.0, and a proposal uses uniform exploration with probability 0.15 for each field. Because every intervention in the limited budget protocol receives the same training budget, standard multi fidelity resource allocation is not available. We therefore implement a fixed budget adaptation of BOHB. Proposals are sampled uniformly until at least two executed observations are available. Later proposals rank the observations, retain the best ⌈𝑛/3⌉ as the current elite set, select a parent with rank based weights, and mutate each field with probability 0.35. A mutated field is sampled from the smoothed good to bad frequency ratio with probability 0.5 and uniformly otherwise. Random search, TPE, and this BOHB adaptation are each evaluated with seeds {0, 1, 42}. Thus, the BOHB result evaluates its configuration selection policy under the common five intervention protocol, rather than the resource allocation component of standard BOHB. The conventional optimizers operate on configurations and scalar target values, while the agents additionally process the task description and experimental feedback through their native decision interface. Accordingly, these baselines compare complete optimization systems under a common execution budget. All methods are evaluated under the same five-intervention execution budget. Because every trial receives the same resource budget, the BOHB-style baseline evaluates its configuration-proposal component in this setting. Multi-fidelity resource scheduling is outside the scope of the protocol.
22
Mean bounded normalized score
Agent / Method NLP (3)
CV (5)
TS (7)
Graph (2)
RL (3)
Overall metrics LLM (4)
SL (6)
Overall
BWR (%)
MAA (%)
Conventional HPO baselines Random search TPE BOHB variant
0.034 ± 0.039 −0.135 ± 0.075 0.299 ± 0.023 −0.463 ± 0.556 0.124 ± 0.192 −0.292 ± 0.102 −0.136 ± 0.097 −0.034 ± 0.051 48.9 ± 10.2 62.6 ± 0.3 0.020 ± 0.006 −0.110 ± 0.105 0.005 ± 0.101 −0.406 ± 0.526 0.192 ± 0.184 −0.219 ± 0.029 −0.302 ± 0.161 −0.113 ± 0.068 40.0 ± 3.3 62.4 ± 1.8 −0.062 ± 0.011 −0.006 ± 0.194 0.235 ± 0.111 −0.824 ± 0.305 0.260 ± 0.127 −0.236 ± 0.025 0.153 ± 0.348 0.018 ± 0.056 45.6 ± 7.7 65.3 ± 2.1
Open-weight agents Gemma2-2B 0.014 ± 0.011 −0.103 ± 0.130 0.233 ± 0.088 −0.459 ± 0.505 0.005 ± 0.293 −0.171 ± 0.128 DeepSeek-R1-Qwen-14B 0.008 ± 0.014 −0.223 ± 0.061 0.316 ± 0.101 −0.333 ± 0.577 0.074 ± 0.045 −0.271 ± 0.082 Qwen3-8B −0.001 ± 0.050 −0.140 ± 0.048 0.233 ± 0.076 −0.271 ± 0.638 0.158 ± 0.042 −0.219 ± 0.109 Llama-3.1-8B 0.019 ± 0.143 −0.142 ± 0.147 0.253 ± 0.051 −0.525 ± 0.463 0.125 ± 0.238 −0.144 ± 0.218 Phi-4-14B 0.047 ± 0.032 −0.077 ± 0.138 0.278 ± 0.013 −0.448 ± 0.508 0.306 ± 0.233 −0.098 ± 0.107 Qwen3-32B 0.060 ± 0.054 −0.120 ± 0.104 0.285 ± 0.014 −0.302 ± 0.606 0.349 ± 0.037 −0.120 ± 0.060
0.066 ± 0.059 −0.001 ± 0.015 55.6 ± 5.1 64.2 ± 2.5 0.157 ± 0.174 0.018 ± 0.035 54.4 ± 5.1 64.1 ± 0.8 0.124 ± 0.171 0.024 ± 0.069 53.3 ± 3.3 65.1 ± 1.0 0.172 ± 0.190 0.030 ± 0.033 44.4 ± 10.2 66.3 ± 2.0 0.430 ± 0.075 0.130 ± 0.062 63.3 ± 3.3 66.9 ± 1.9 0.485 ± 0.175 0.148 ± 0.059 60.0 ± 3.3 69.1 ± 0.6
Table 10 Three-seed results for conventional HPO baselines and open-weight agents under the limited-budget protocol. Entries report mean ± sample standard deviation over seeds {0, 1, 42}. Category columns report MBNS. BWR and MAA are computed independently for each seed before aggregation.
Random seed sensitivity. Table 10 reports the variability of all locally executable methods. Each entry is the arithmetic mean and sample standard deviation of the corresponding seed level metric over seeds {0, 1, 42}.
11.2
Statistical Robustness
We additionally assess sensitivity to the composition of the benchmark task suite using 20,000 categorystratified paired bootstrap resamples. Each resample preserves the number of tasks in every research category, and the same sampled tasks are used for all methods. For open-weight agents and conventional HPO baselines, the task-level BNS and anchor attainment values are first averaged over the three controlled seeds and then resampled. The resulting intervals therefore quantify uncertainty associated with task composition, whereas the standard deviations in Table 10 quantify variation across controlled executions. For API agents, which are evaluated once, these intervals reflect sensitivity to benchmark task composition only and should not be interpreted as uncertainty across repeated API executions. Interval definitions. A task-bootstrap 95% confidence interval is obtained by repeatedly resampling tasks with replacement within each research category, recomputing the aggregate metric for each resample, and taking the 2.5th and 97.5th percentiles of the resulting distribution. It measures the sensitivity of an aggregate result to the composition of the benchmark task suite. A paired MBNS difference is computed as MBNS(𝐴) − MBNS(𝐵) using the same resampled tasks for both methods. A positive difference favors Method A, while a negative difference favors Method B. If its 95% confidence interval includes zero, the observed ordering is not stable under variation in task composition. The bootstrap intervals show that benchmark composition contributes non-negligible uncertainty, particularly for methods whose gains are concentrated in a small number of categories or tasks. Claude Sonnet 4.6 retains the strongest point estimates across the four reported summaries. Qwen3-32B and Phi-4-14B have similar task-composition uncertainty among open-weight agents. The three conventional HPO intervals are also broad under the final-step criterion. These intervals are intended as a robustness diagnostic rather than a multiple-comparison significance test. Small differences between methods should therefore be interpreted together with the paired comparisons reported below.
11.3
Best-So-Far Performance
The main results use the configuration produced at the fifth intervention, which evaluates whether a method preserves and refines improvements throughout the full trajectory. As a complementary diagnostic, Table 12 reports the best result observed among interventions one through five. For each task, the best intervention according to the task-specific metric direction is scored against the same common reference baseline used in the main results. The reference baseline itself is not included among the candidate interventions. For methods evaluated with three seeds, scoring is performed independently for each task and seed before averaging across seeds and tasks, matching the aggregation used in the main results. 23
Agent / Method
MBNS [95% CI]
MAA (%) [95% CI]
Median BNS
BWR (%)
−0.034 [−0.153, 0.082] −0.113 [−0.265, 0.045] 0.018 [−0.122, 0.161]
62.6 [52.5, 72.2] 62.4 [51.9, 72.3] 65.3 [55.5, 74.5]
0.009 −0.030 0.012
48.9 40.0 45.6
−0.001 [−0.154, 0.143] 0.018 [−0.141, 0.169] 0.030 [−0.118, 0.183] 0.024 [−0.127, 0.172] 0.130 [0.001, 0.256] 0.148 [0.016, 0.283]
64.2 [55.0, 73.2] 64.1 [54.1, 73.9] 66.3 [57.3, 75.0] 65.1 [55.8, 74.2] 66.9 [57.8, 75.9] 69.1 [60.3, 77.7]
0.003 0.007 −0.041 −0.010 0.066 0.058
55.6 54.4 44.4 53.3 63.3 60.0
0.080 [−0.084, 0.240] 0.110 [−0.054, 0.260] 0.112 [−0.049, 0.268] 0.175 [−0.017, 0.352] 0.305 [0.174, 0.433] 0.407 [0.264, 0.541]
67.0 [57.4, 76.4] 70.4 [61.0, 79.8] 67.8 [58.2, 77.2] 70.1 [61.6, 78.5] 76.7 [66.7, 86.7] 79.5 [66.6, 93.6]
0.036 0.012 0.076 0.109 0.103 0.382
56.7 56.7 60.0 63.3 66.7 76.7
Conventional HPO baselines Random search TPE BOHB variant Open-weight agents Gemma2-2B DeepSeek-R1-Qwen-14B Llama-3.1-8B Qwen3-8B Phi-4-14B Qwen3-32B API agents GLM-5.1 Kimi-2.6 GLM-4.7 DeepSeek-V4-Pro GPT-5.5 Claude Sonnet 4.6
Table 11 Task-composition robustness of the evaluated methods under the limited-budget protocol. Intervals are percentile 95% confidence intervals from 20,000 category-stratified paired bootstrap resamples of the 30 tasks. Median BNS and BWR are point estimates.
This analysis distinguishes the quality of the best configuration discovered within the intervention budget from the ability to retain it at the final step. Claude Sonnet 4.6 remains the strongest method overall, reaching an MBNS of 0.469, a BWR of 90.0%, and an MAA of 80.5%. It also leads on CV, Graph, LLM, and SL, while GPT-5.5 leads on RL, TPE on TS, and random search on NLP. Compared with their final-step results, the overall MBNS of the three conventional HPO baselines improves by 0.273–0.411 under best-so-far selection, indicating that they often discover useful configurations but do not consistently finish with them. Under this incumbent-style view, random search, TPE, and the BOHB variant obtain overall MBNS values of 0.325, 0.298, and 0.291, respectively, exceeding all evaluated open-weight agents. Phi-4-14B and Qwen3-32B reach 0.197 and 0.195. This ranking reversal clarifies that the final-step metric combines search quality with the ability to preserve or refine an earlier gain, whereas standard HPO commonly returns the incumbent. The strongest API agents remain competitive under best-so-far selection, with Claude Sonnet 4.6 and GPT-5.5 reaching 0.469 and 0.347. The corresponding seed variation and task-bootstrap intervals are reported below. Table 13 separates execution variation from sensitivity to benchmark composition. The three conventional HPO methods have overlapping best-so-far task-bootstrap intervals, as do Phi-4-14B and Qwen3-32B. Table 14 directly bootstraps task-level differences between selected methods. Under the final-step criterion, Claude Sonnet 4.6 has a positive paired difference relative to GPT-5.5, and Qwen3-32B has a positive difference relative to the BOHB variant. The interval for Phi-4-14B versus Qwen3-32B includes zero. Under best-so-far selection, Claude Sonnet 4.6 remains ahead of GPT-5.5, while the interval for GPT-5.5 versus random search includes zero. Random search has a positive paired difference relative to Phi-4-14B, whereas its differences from TPE and the BOHB variant are inconclusive. These results support reporting both selection criteria without treating small point-estimate differences as definitive rankings.
11.4
Representative Decision Traces
Table 15 compares five agents on the same tunedGNN Cora GCN task. For the three open-weight agents, we show the seed-0 trace from the controlled three-seed evaluation so that each row represents one executable 24
Agent / Method
Mean bounded normalized score
Overall metrics
NLP (3) CV (5) TS (7) Graph (2) RL (3) LLM (4) SL (6) Overall BWR (%) MAA (%)
Conventional HPO baselines Random search TPE BOHB variant
0.089 0.088 0.015
0.179 0.138 0.225
0.629 0.682 0.570
0.011 -0.192 -0.167
0.481 0.426 0.305
-0.097 -0.090 -0.061
0.515 0.447 0.538
0.325 0.298 0.291
71.1 74.4 67.8
72.7 71.4 70.4
0.020 0.018 0.016 0.019 0.074 0.050
0.170 0.051 0.123 0.045 0.070 0.123
0.259 0.320 0.303 0.291 0.289 0.282
-0.459 -0.333 -0.271 -0.177 -0.302 -0.281
0.125 0.404 0.194 0.299 0.375 0.413
-0.091 -0.206 -0.038 -0.035 -0.079 -0.064
0.142 0.181 0.180 0.215 0.508 0.459
0.089 0.112 0.125 0.134 0.195 0.197
62.2 63.3 63.3 56.7 66.7 70.0
66.3 67.1 66.7 68.7 70.4 69.0
-0.088 -0.050 -0.088 -0.027 -0.028 -0.013
0.171 0.122 0.197 0.096 0.164 0.258
0.286 0.277 0.283 0.510 0.375 0.519
0.031 0.031 0.000 0.094 0.281 0.877
0.032 0.486 0.061 0.621 0.822 0.710
0.009 -0.052 -0.044 0.052 0.119 0.206
0.713 0.684 0.689 0.544 0.589 0.746
0.235 0.261 0.228 0.316 0.347 0.469
70.0 70.0 66.7 73.3 73.3 90.0
69.1 71.4 69.6 72.8 77.5 80.5
Open-weight agents Gemma2-2B DeepSeek-R1-Qwen-14B Qwen3-8B Llama-3.1-8B Qwen3-32B Phi-4-14B API agents GLM-5.1 Kimi-2.6 GLM-4.7 DeepSeek-V4-Pro GPT-5.5 Claude Sonnet 4.6
Table 12 Best-so-far AgentHPOBench results under the limited-budget protocol. For each task, the best observed result among interventions one through five is selected and scored. Category columns report MBNS, while BWR and MAA are computed over all 30 tasks. This diagnostic complements the final-step results in the main paper. Agent / Method
Best-so-far MBNS
Seed SD
Task-bootstrap 95% CI
0.325 0.298 0.291
0.082 0.048 0.040
[0.199, 0.447] [0.179, 0.417] [0.187, 0.396]
0.089 0.112 0.125 0.134 0.197 0.195
0.008 0.052 0.061 0.028 0.039 0.060
[−0.071, 0.241] [−0.010, 0.234] [−0.016, 0.262] [0.003, 0.267] [0.065, 0.327] [0.066, 0.327]
0.235 0.261 0.228 0.316 0.347 0.469
– – – – – –
[0.086, 0.381] [0.139, 0.380] [0.076, 0.374] [0.198, 0.433] [0.221, 0.470] [0.343, 0.588]
Conventional HPO baselines Random search TPE BOHB variant Open-weight agents Gemma2-2B DeepSeek-R1-Qwen-14B Qwen3-8B Llama-3.1-8B Phi-4-14B Qwen3-32B API agents GLM-5.1 Kimi-2.6 GLM-4.7 DeepSeek-V4-Pro GPT-5.5 Claude Sonnet 4.6
Table 13 Uncertainty estimates for best-so-far MBNS. Seed SD is the sample standard deviation over seeds {0, 1, 42} and is reported only for locally controlled methods. Task-bootstrap intervals use the same 20,000 category-stratified paired resamples.
configuration trajectory. The API-agent rows show their single evaluation. All displayed traces start
25
Selection criterion
Method A – Method B
Paired MBNS difference [95% CI]
Final step Final step Final step
Claude Sonnet 4.6 – GPT-5.5 Phi-4-14B – Qwen3-32B Qwen3-32B – BOHB variant
0.102 [0.007, 0.208] −0.018 [−0.071, 0.034] 0.131 [0.024, 0.238]
Best so far Best so far Best so far Best so far Best so far Best so far
Claude Sonnet 4.6 – GPT-5.5 GPT-5.5 – Random search Random search – Phi-4-14B Random search – TPE Random search – BOHB variant Phi-4-14B – Qwen3-32B
0.122 [0.055, 0.207] 0.022 [−0.116, 0.156] 0.127 [0.029, 0.232] 0.026 [−0.029, 0.089] 0.033 [−0.018, 0.091] 0.003 [−0.048, 0.053]
Table 14 Selected paired method comparisons under the final-step and best-so-far criteria. Each entry is Method A minus Method B. The intervals use 20,000 category-stratified paired task-bootstrap resamples. Controlled methods are first averaged over their three matched seeds at the task level. Agent
Configurations selected at interventions 1–5
Test accuracy (%)
Qualitative pattern
Claude Sonnet 𝐼1 : lr=0.005, hidden=256, dropout=0.5, layer norm=True; 4.6 𝐼2 : retain 𝐵 except lr=0.002; 𝐼3 : retain 𝐼2 and set residual=True; 𝐼4 –𝐼5 : retain 𝐼3
82.2 → 83.8 → 85.2 → 85.2 → 85.2
Recovers from a harmful joint change, isolates one architectural change, and retains the improved configuration.
GPT-5.5
𝐼1 : lr=0.005, dropout=0.5; 𝐼2 : retain 𝐵 except lr=0.002; 𝐼3 : retain 𝐼2 except layers=2; 𝐼4 –𝐼5 : retain 𝐼3
82.2 → 83.8 → 84.4 → 84.4 → 84.4
Uses a conservative correction, tests a single change to model depth, and retains the improved configuration.
Qwen3-32B
𝐼1 : lr=0.002, hidden=256, layers=4; 𝐼2 : retain 𝐵 except lr=0.002; 𝐼3 –𝐼5 : retain 𝐼2
79.4 → 83.8 → 83.8 → 83.8 → 83.8
Recovers from an aggressive first proposal but stops exploring after a modest improvement.
Llama-3.1-8B
𝐼1 : lr=0.002, hidden=256, dropout=0.5, weight decay=0; 𝐼2 : return to 𝐵; 𝐼3 : return to 𝐼1 ; 𝐼4 : retain 𝐼1 except lr=0.001; 𝐼5 : retain 𝐼1 except layers=4
85.0 → 83.5 → 85.0 → 84.5 → 81.0
Finds a strong configuration early but continues exploring and discards it at the final intervention.
Gemma2-2B
𝐼1 –𝐼5 : retain 𝐵
83.5 → 83.5 → 83.5 → 83.5 → 83.5
Repeats the baseline configuration without testing an alternative.
Table 15 Decision traces on tunedGNN Cora GCN under the limited budget protocol. 𝐵 denotes the shared baseline, and 𝐼 𝑖 denotes intervention 𝑖. The accuracy column reports the outcomes of 𝐼1 –𝐼5 . For compactness, configurations are described relative to 𝐵 or a referenced earlier intervention.
from the shared baseline 𝐵 = (lr = 0.001, hidden = 512, layers = 3, dropout = 0.7, weight decay = 0.0005, layer norm = False, residual = False), which obtains 83.5% test accuracy at the epoch with the highest validation accuracy. Holding the task, baseline, and intervention space fixed reveals distinct responses to the same type of experimental feedback. Claude Sonnet 4.6 and GPT-5.5 recover from harmful initial changes and preserve subsequent improvements. Qwen3-32B also recovers but stops exploring after a modest gain, whereas Llama-3.1-8B discards its strongest configuration at the final intervention. Gemma2-2B repeatedly executes the baseline without testing an alternative.
12
Metric Sensitivity
Metric sensitivity. Some tasks have a small gap between the limited-budget reference baseline and the reported anchor. In such cases, modest execution noise can produce a relatively large normalized change. Bounding each task score to [−1, 1] prevents an arbitrarily large contribution from one task, but does not remove sensitivity near a small denominator and can cause score saturation. We therefore report BWR, MAA, median BNS, task-level results, and task-composition bootstrap intervals alongside MBNS. The current benchmark does not separately estimate a noise-based minimum meaningful difference from repeated no-op
26
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
Q8
Q32
Gem
DS14
Phi
Llama
DSV4
GPT
G4.7
G5.1
Kimi
Claude
6.053 26.883 2.200 0.195 1.141 0.511 39.803 89.170 67.733 0.896 85.560 0.017 1.754 0.519 0.127 97.350 81.267 83.902 0.954 88.906 0.686 0.372 9.360 0.183 0.177 0.320 45.933 115.639 0.339 5.800
5.868 23.183 17.600 0.379 0.398 0.509 35.446 89.303 67.900 0.958 85.560 -0.031 1.776 0.519 0.127 97.333 73.400 83.573 0.954 83.735 0.730 0.372 5.177 0.185 0.177 0.320 48.800 68.706 0.339 5.800
5.931 24.917 15.133 0.161 0.398 0.524 41.481 89.580 67.467 0.800 85.560 0.021 1.754 0.519 0.127 97.500 81.267 84.155 0.954 70.713 0.664 0.372 16.313 0.185 0.177 0.227 48.800 50.213 0.339 5.800
5.931 27.930 2.200 0.057 1.151 0.509 40.051 89.170 67.733 0.896 85.560 0.017 1.754 0.519 0.127 97.733 81.267 83.472 0.954 75.917 0.723 0.372 19.433 0.185 0.177 0.320 48.800 74.628 0.339 5.800
6.057 33.660 21.600 0.448 0.398 0.511 50.382 89.720 66.400 0.930 85.918 -0.009 1.754 0.519 0.127 95.800 80.167 84.054 0.954 81.729 0.531 0.385 7.010 0.185 0.177 0.227 39.967 100.742 0.339 5.800
6.053 17.647 18.133 0.276 1.179 0.511 35.446 88.757 67.600 0.868 85.560 0.021 1.754 0.519 0.127 97.317 80.833 84.205 0.947 70.713 0.506 0.372 8.853 0.185 0.177 0.320 48.633 52.047 0.353 5.800
5.931 38.370 45.000 0.517 0.375 0.509 35.446 89.190 70.000 0.925 87.224 -0.021 1.663 0.530 0.127 97.450 82.400 84.610 0.979 70.713 0.669 0.385 12.900 0.184 0.177 0.280 46.300 358.546 0.337 5.800
5.931 26.250 45.000 1.000 0.375 0.509 35.446 89.010 69.200 0.928 86.548 -0.021 0.530 0.530 0.127 97.050 82.200 85.216 0.979 72.862 0.687 0.380 27.070 0.184 0.177 0.000 48.700 733.655 0.339 5.800
5.931 39.730 1.800 0.000 1.150 0.551 35.446 89.220 70.000 0.928 85.519 0.011 0.657 0.530 0.127 96.550 82.900 84.913 0.979 70.713 0.694 0.372 23.820 0.185 0.177 0.280 40.200 37.693 0.341 5.800
5.931 25.180 1.800 0.000 1.248 0.551 35.446 89.220 70.000 0.928 85.519 0.011 0.657 0.530 0.127 96.900 82.900 85.064 0.979 70.713 0.646 0.372 25.560 0.185 0.177 0.280 40.200 129.974 0.341 5.800
5.931 27.370 2.200 0.414 1.131 0.551 35.446 89.090 70.000 0.928 85.519 0.011 0.657 0.530 0.133 97.350 82.900 85.064 0.979 70.713 0.646 0.372 17.910 0.185 0.178 0.280 48.600 32.836 0.341 5.800
5.931 13.950 46.000 0.862 0.375 0.509 35.446 88.570 68.600 0.956 87.454 0.021 0.465 0.530 0.127 97.150 82.200 84.761 0.979 75.143 0.690 0.385 12.550 0.182 0.177 0.000 40.200 259.580 0.337 5.800
Table 16 Per-task raw metric values after the first intervention in the main limited budget evaluation.
executions.
13
Detailed Experimental Results
This section reports the complete per-task results for the main limited budget evaluation. The corresponding reference baseline values are provided in Table 7. Tables 16–20 report the raw metric observed after each of the five sequential interventions. For open-weight agents, raw metrics are averaged over seeds {0, 1, 42} at each task and intervention, while bounded normalized scores and anchor attainment values are computed per seed and then averaged. API-agent entries correspond to their single audited evaluation. Values are shown in the same unit and direction as the corresponding task anchor in Table 6. Raw values are rounded to three decimal places, whereas all derived scores are computed from the full precision values in the recorded traces. These tables show whether each agent improves, plateaus, or degrades across interventions instead of only reporting a final aggregate score. The fifth intervention provides the final result used for aggregate scoring. Tables 21–25 convert the intervention results into bounded normalized scores using the common limited budget baseline for each task. Tables 26–30 report the corresponding anchor attainment values. Repeated raw values across adjacent interventions can indicate that an agent retained the same configuration or obtained the same rounded metric after execution. Nonmonotonic trajectories are expected because each intervention is evaluated as a new repository experiment rather than as a best result retained across previous trials.
27
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
Q8
Q32
Gem
DS14
Phi
Llama DSV4
GPT
5.443 21.687 17.600 0.218 1.151 0.511 38.553 89.980 67.733 0.896 85.560 0.014 1.754 0.519 0.127 97.283 80.067 84.483 0.954 80.579 0.696 0.371 9.953 0.185 0.177 0.320 48.733 215.894 0.339 5.267
5.310 30.760 45.000 0.356 0.383 0.509 40.499 89.393 67.467 0.962 83.405 0.015 1.542 0.519 0.127 97.517 80.667 83.523 0.954 83.735 0.730 0.372 7.573 0.185 0.177 0.320 48.800 84.479 0.339 4.800
5.517 29.730 15.133 0.184 0.398 0.521 41.464 89.433 67.467 0.800 85.560 0.015 1.754 0.519 0.127 97.583 81.267 83.826 0.954 70.713 0.512 0.372 16.090 0.185 0.177 0.227 48.800 66.062 0.339 5.733
5.414 20.980 2.200 0.000 1.151 0.509 40.353 89.263 67.733 0.896 85.560 0.017 1.816 0.519 0.127 97.650 81.267 83.397 0.954 75.917 0.705 0.372 19.930 0.185 0.177 0.320 48.800 38.123 0.339 5.133
5.567 26.040 21.600 0.471 0.398 0.511 35.446 89.863 67.733 0.955 85.611 0.013 1.754 0.519 0.127 97.600 81.267 84.205 0.952 81.729 0.525 0.385 9.620 0.185 0.177 0.320 48.800 59.138 0.339 4.933
5.578 24.923 45.800 0.207 0.918 0.506 37.694 89.037 68.167 0.924 85.560 0.012 2.502 0.519 0.127 72.767 79.233 84.180 0.954 70.713 0.523 0.372 8.830 0.185 0.177 0.227 48.800 67.808 0.339 4.800
5.414 5.450 27.540 45.080 45.000 1.800 1.000 0.034 0.375 0.375 0.509 0.509 37.830 35.446 89.220 88.650 70.000 66.700 0.928 0.968 86.471 85.534 0.011 0.017 0.530 0.657 0.499 0.530 0.127 0.127 97.650 95.450 83.800 82.900 84.761 84.534 0.979 0.979 70.713 70.713 0.999 0.654 0.368 0.368 21.790 23.200 0.183 0.185 0.177 0.177 0.000 0.280 41.300 48.600 70.937 1305.460 0.339 0.341 5.600 5.200
5.414 13.240 45.000 0.586 0.375 0.509 35.446 89.220 69.800 0.928 87.300 0.011 9.076 0.530 0.127 96.950 83.800 84.534 0.979 72.393 0.863 0.379 17.310 0.186 0.177 0.120 48.700 28.823 0.339 4.000
G4.7
G5.1
Kimi Claude
5.450 23.830 1.800 0.069 0.375 0.509 35.446 88.650 66.700 0.968 85.534 0.017 0.657 0.530 0.127 95.350 83.100 85.595 0.964 70.713 0.665 0.368 22.120 0.185 0.177 0.280 48.600 469.664 0.341 5.800
5.414 26.710 2.200 0.241 0.375 0.551 38.462 88.650 66.700 0.968 85.534 0.017 0.657 0.530 0.127 97.550 83.200 84.913 0.979 70.713 0.713 0.372 16.470 0.185 0.177 0.280 48.600 65.221 0.341 5.400
5.414 36.460 46.000 0.690 0.375 0.509 35.446 89.270 70.000 0.982 87.485 0.021 0.465 0.513 0.144 97.550 83.800 85.519 0.981 66.253 0.997 0.372 22.010 0.184 0.176 0.000 48.600 21.816 0.337 5.200
Table 17 Per-task raw metric values after the second intervention in the main limited budget evaluation.
14
Full Budget and Harness Ablation Details
The full budget evaluation uses the same task definitions, agent interface, result schema, scoring rules, and five intervention protocol as the limited budget evaluation. Each baseline and intervention is instead executed with the full training or evaluation budget of the original repository. The complete intervention level raw results are shown in Tables 31–35. The bounded normalized scores are shown in Tables 36–40, and the anchor attainment values are shown in Tables 41–45. Raw metrics are displayed to three decimal places, while all derived scores use the corresponding full precision values. The harness ablation keeps the AgentHPOBench task suite, intervention budget, result schema, and scoring pipeline fixed while changing the external execution harness used by the backbone agent. For Claude Sonnet 4.6, the comparison is between the native AgentHPOBench harness and Claude Code CLI. For GPT-5.5, the comparison is between the native AgentHPOBench harness and Codex CLI. These experiments isolate the effect of the execution interface from the benchmark task definitions and scoring code. The complete intervention level raw results are shown in Tables 46–50. The bounded normalized scores are shown in Tables 51–55, and the anchor attainment values are shown in Tables 56–60.
28
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
Q8
Q32
Gem
DS14
Phi
Llama DSV4
GPT
G4.7
G5.1
Kimi Claude
4.992 23.653 17.600 0.218 0.899 0.511 41.729 89.870 67.733 0.896 85.560 0.012 1.754 0.519 0.127 97.650 80.567 84.180 0.954 85.507 0.698 0.372 16.567 0.185 0.177 0.320 47.967 47.520 0.339 6.000
4.866 27.330 45.000 0.356 0.383 0.509 40.316 89.370 67.733 0.962 85.867 -0.007 1.581 0.519 0.127 97.333 81.267 83.877 0.954 87.969 0.709 0.372 14.320 0.185 0.177 0.320 48.800 48.510 0.339 5.733
5.066 22.153 15.133 0.149 0.398 0.524 40.285 89.913 67.467 0.800 85.560 0.021 1.754 0.519 0.127 97.050 81.267 83.927 0.964 70.713 0.535 0.372 15.123 0.185 0.177 0.227 48.800 97.179 0.339 5.000
4.983 26.757 2.200 0.402 1.147 0.509 40.598 89.283 67.733 0.896 85.560 0.017 1.754 0.519 0.127 97.517 81.233 83.851 0.954 75.917 0.706 0.372 21.383 0.185 0.177 0.320 48.800 41.854 0.339 5.267
5.147 20.983 21.600 0.379 0.398 0.511 52.304 89.833 68.133 0.958 83.451 0.011 1.884 0.519 0.127 97.117 80.767 84.205 0.954 87.222 0.532 0.372 15.457 0.185 0.177 0.227 48.800 69.264 0.339 4.933
5.030 26.917 46.267 0.207 0.390 0.508 42.272 88.573 68.133 0.868 85.560 0.020 1.754 0.519 0.127 97.417 78.667 84.205 0.954 72.862 0.527 0.372 5.880 0.185 0.177 0.227 48.800 60.480 0.339 5.133
4.983 21.540 45.000 1.000 0.375 0.509 35.446 89.220 69.300 0.928 86.548 0.011 0.530 0.496 0.127 97.200 84.400 84.913 0.979 71.188 0.999 0.365 29.180 0.183 0.177 0.000 48.800 67.191 0.388 4.600
5.064 31.980 1.800 0.069 0.375 0.509 35.446 88.930 66.700 0.968 85.534 0.017 0.657 0.530 0.127 95.800 83.500 85.064 0.979 70.713 0.689 0.373 23.240 0.185 0.177 0.280 46.300 18.461 0.341 4.600
5.064 25.350 1.800 0.069 0.375 0.509 35.446 88.920 66.700 0.968 85.534 0.017 0.657 0.530 0.127 94.600 83.600 85.216 0.979 70.713 0.612 0.373 22.150 0.185 0.177 0.280 46.300 18.119 0.341 4.600
4.965 15.630 2.200 0.448 0.375 0.551 35.446 89.270 66.700 0.968 85.534 0.017 0.657 0.530 0.133 95.350 83.600 85.140 0.979 70.713 0.706 0.372 15.970 0.185 0.177 0.280 48.800 44.994 0.341 5.400
4.983 41.980 45.000 0.586 0.375 0.509 37.565 88.880 70.000 0.925 87.300 0.009 7.080 0.530 0.127 97.500 83.800 84.003 0.979 70.713 0.795 0.379 5.030 0.183 0.177 0.280 48.700 70.980 0.337 5.600
Table 18 Per-task raw metric values after the third intervention in the main limited budget evaluation.
29
4.983 47.950 46.000 0.897 0.375 0.509 36.943 89.090 70.600 0.970 87.638 0.021 0.333 0.496 0.127 97.300 85.200 85.823 0.964 70.713 0.994 0.593 22.450 0.183 0.177 0.000 48.800 19.940 0.335 6.000
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
Q8
Q32
Gem
DS14
Phi
Llama
DSV4
GPT
G4.7
G5.1
Kimi Claude
4.529 26.723 17.600 0.184 0.635 0.511 40.628 90.287 67.733 0.896 81.941 -0.009 1.754 0.519 0.127 97.517 80.967 84.837 0.954 80.579 0.671 0.446 16.130 0.185 0.177 0.320 48.667 74.567 0.339 5.267
4.414 30.077 47.200 0.356 0.383 0.509 40.156 89.063 67.300 0.962 86.164 0.014 1.581 0.519 0.127 97.783 81.400 83.422 0.954 87.969 0.722 0.374 16.603 0.185 0.177 0.320 48.800 113.477 0.339 4.667
4.645 28.067 15.133 0.138 0.398 0.521 40.645 89.760 67.467 0.800 83.185 0.013 1.754 0.519 0.127 97.517 81.267 84.534 0.954 70.713 0.684 0.372 16.453 0.185 0.177 0.227 48.800 56.755 0.339 5.267
4.513 18.270 2.200 0.126 0.635 0.509 39.168 89.170 67.733 0.896 85.560 0.017 1.754 0.519 0.127 97.317 81.267 84.180 0.954 75.917 0.699 0.372 21.570 0.183 0.177 0.320 48.800 54.772 0.339 5.700
4.723 29.077 21.600 0.483 0.398 0.511 50.053 90.067 67.900 0.963 83.216 0.005 5.210 0.519 0.127 96.517 81.100 84.382 0.954 87.222 0.565 0.372 15.967 0.185 0.177 0.227 48.800 44.233 0.339 5.133
4.732 22.863 36.133 0.379 0.375 0.510 35.446 88.690 68.167 0.870 85.560 -0.009 1.754 0.519 0.127 72.900 78.733 84.104 0.954 72.862 0.525 0.372 9.647 0.185 0.177 0.227 48.800 138.505 0.349 5.000
4.513 27.510 44.600 0.483 0.375 0.509 37.310 89.250 70.000 0.915 87.300 0.011 0.797 0.530 0.127 97.550 83.800 83.927 0.979 70.713 0.812 0.366 12.690 0.185 0.177 0.280 48.800 49.257 0.337 5.600
4.513 16.210 45.000 1.000 0.375 0.509 36.577 89.270 70.000 0.928 86.379 0.011 0.530 0.497 0.127 97.600 84.400 84.837 0.979 70.713 0.998 0.604 29.230 0.183 0.177 0.000 48.800 17.757 0.339 5.600
4.648 41.580 1.800 0.034 0.375 0.509 35.446 88.930 66.700 0.968 85.534 0.017 0.657 0.530 0.127 96.800 82.400 85.064 0.979 70.713 0.655 0.593 24.160 0.185 0.177 0.280 48.400 21.168 0.341 4.800
4.648 27.860 1.800 0.000 0.375 0.509 35.446 88.650 66.700 0.968 85.534 0.017 0.657 0.530 0.127 97.100 82.400 84.913 0.979 70.713 0.671 0.593 25.130 0.185 0.177 0.280 48.400 21.229 0.341 4.400
4.588 23.350 2.200 0.069 0.375 0.509 35.446 89.120 66.700 0.968 85.534 0.017 1.487 0.530 0.133 95.450 82.400 84.837 0.979 70.713 0.706 0.372 16.540 0.185 0.178 0.280 48.400 30.868 0.341 5.400
Table 19 Per-task raw metric values after the fourth intervention in the main limited budget evaluation.
30
4.513 20.740 46.000 0.897 0.375 0.509 39.918 89.250 70.800 0.979 87.239 -0.035 0.333 0.497 0.127 97.450 85.200 84.837 0.979 73.648 0.998 0.382 22.340 0.186 0.176 0.080 48.700 16.276 0.340 5.600
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
Q8
Q32
Gem
DS14
Phi
Llama DSV4
GPT
G4.7
G5.1
Kimi Claude
4.377 25.413 17.600 0.195 0.636 0.511 41.729 90.287 67.733 0.896 85.560 -0.009 1.754 0.519 0.127 97.233 80.567 84.433 0.954 85.507 0.709 0.378 16.627 0.185 0.177 0.320 45.533 87.674 0.339 5.400
4.274 31.240 47.200 0.379 0.383 0.509 45.138 89.147 67.733 0.962 86.522 -0.007 1.581 0.519 0.127 97.600 81.433 84.028 0.954 87.969 0.688 0.372 16.477 0.185 0.177 0.320 48.800 75.314 0.339 5.067
4.357 26.400 15.133 0.161 0.398 0.524 41.484 89.783 67.467 0.800 83.185 0.017 1.754 0.519 0.127 97.567 81.267 84.306 0.954 70.713 0.517 0.372 16.213 0.185 0.177 0.227 48.800 67.726 0.339 4.967
4.344 16.160 2.200 0.057 0.626 0.509 39.925 89.283 67.733 0.896 85.560 0.017 1.754 0.519 0.127 97.583 81.267 83.927 0.954 75.917 0.742 0.372 21.043 0.183 0.177 0.320 48.800 103.021 0.337 5.467
4.533 25.390 21.600 0.448 0.398 0.511 56.529 90.113 65.967 0.963 85.929 0.016 1.754 0.519 0.127 97.550 79.533 84.534 0.954 87.222 0.523 0.372 15.293 0.185 0.177 0.227 48.800 51.079 0.339 5.900
4.419 22.883 45.600 0.276 0.383 0.506 45.617 88.333 67.833 0.922 85.560 0.020 2.339 0.519 0.127 97.000 78.733 83.902 0.954 72.862 0.539 0.372 15.763 0.185 0.177 0.227 48.633 48.053 0.345 4.667
4.406 29.590 45.000 1.000 0.375 0.509 38.125 89.270 70.000 0.928 86.548 0.011 0.530 0.496 0.127 97.400 84.400 85.368 0.979 70.713 0.998 0.365 29.140 0.183 0.177 0.280 48.800 47.044 0.339 3.800
4.530 38.780 1.800 0.000 0.375 0.509 35.446 88.930 66.700 0.968 85.534 0.017 0.657 0.530 0.127 97.600 80.800 85.292 0.979 70.713 0.765 0.389 23.780 0.185 0.177 0.280 39.000 24.783 0.341 5.200
4.530 35.970 1.800 0.000 0.375 0.509 35.446 88.930 66.700 0.968 85.534 0.017 0.657 0.530 0.127 97.850 80.800 85.444 0.979 70.713 0.691 0.389 21.780 0.185 0.177 0.280 39.000 56.042 0.341 5.200
4.480 47.030 2.200 0.517 0.375 0.509 38.034 89.170 66.700 0.968 85.534 0.017 0.657 0.530 0.127 96.200 80.000 84.534 0.979 70.713 0.662 0.372 24.070 0.185 0.177 0.280 48.800 67.671 0.341 4.800
4.406 32.210 45.000 0.483 0.375 0.509 38.095 89.220 70.000 0.915 87.485 0.009 0.797 0.530 0.127 97.800 83.800 84.989 0.979 70.713 0.610 0.366 15.980 0.180 0.177 0.280 48.800 85.941 0.390 4.600
Table 20 Per-task raw metric values after the fifth intervention in the main limited budget evaluation.
31
4.406 37.620 46.000 0.862 0.375 0.509 38.074 89.250 70.800 0.970 87.423 0.021 0.333 0.517 0.127 97.350 85.200 85.444 0.979 75.143 0.996 0.371 30.450 0.185 0.176 0.080 48.800 16.543 0.326 4.600
Task
Q8
Q32
Gem
DS14
Phi
Llama DSV4
GPT
G4.7
G5.1
Kimi Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
-1.000 -0.038 0.008 -0.711 0.018 0.953 0.016 0.016 0.000 -0.667 0.985 0.006 -0.364 0.185 0.151 -0.057 -0.667 -0.554 0.728 -0.271 0.623 0.000 -0.114 0.000 0.250 0.533 -0.333 -0.851 0.075 -0.538
-1.000 -0.105 0.197 -0.112 0.971 0.991 -0.068 0.043 0.000 1.000 0.985 -0.042 -0.390 0.185 0.151 -0.067 -1.000 -0.601 0.728 -0.194 0.675 0.000 -0.182 -0.642 0.246 0.533 0.000 -0.504 0.075 -0.538
-1.000 -0.073 0.166 -0.782 0.971 0.635 0.049 0.100 -0.252 -1.000 0.985 0.011 -0.364 0.185 0.151 0.029 -0.667 -0.375 0.728 0.000 0.596 0.000 -0.001 -0.445 0.377 0.378 0.000 -0.419 0.075 -0.538
-1.000 -0.019 0.008 -1.000 0.005 0.991 0.021 0.016 0.000 -0.667 0.985 0.006 -0.364 0.185 0.151 0.162 -0.667 -0.834 0.728 -0.077 0.667 0.000 0.049 -0.445 0.243 0.533 0.000 -0.626 0.075 -0.538
-1.000 0.085 0.246 0.112 0.971 0.953 0.222 0.128 -0.824 0.333 0.997 -0.020 -0.364 0.185 0.151 -0.448 -0.729 -0.447 0.728 -0.164 0.437 -0.057 -0.152 -0.445 0.242 0.378 -1.000 -1.000 0.075 -0.538
-1.000 -0.205 0.203 -0.449 -0.031 0.953 -0.068 -0.068 -0.491 -0.912 0.985 0.010 -0.364 0.185 0.151 -0.076 -0.354 -0.339 0.659 0.000 0.406 0.000 -0.122 -0.445 0.241 0.533 -0.060 -0.590 -0.452 -0.538
-1.000 -0.049 0.533 1.000 1.000 0.990 -0.068 -0.016 -0.755 0.000 1.000 -0.033 1.000 0.066 0.151 -0.229 -0.813 0.375 1.000 -0.032 0.623 -0.035 0.173 -0.278 0.370 0.000 -0.036 -1.000 0.056 -0.538
-1.000 0.195 0.003 -1.000 0.006 0.000 -0.068 0.026 0.000 0.000 0.981 0.000 1.000 0.066 0.151 -0.514 -0.375 0.161 1.000 0.000 0.633 0.000 0.120 -0.445 0.239 0.467 -1.000 -0.106 0.000 -0.538
-1.000 -0.069 0.003 -1.000 -0.119 0.000 -0.068 0.026 0.000 0.000 0.981 0.000 1.000 0.066 0.151 -0.314 -0.375 0.268 1.000 0.000 0.575 0.000 0.149 -0.445 0.263 0.467 -1.000 -1.000 0.000 -0.538
-1.000 -0.029 0.008 0.000 0.031 0.000 -0.068 0.000 0.000 0.000 0.981 0.000 1.000 0.066 0.000 -0.057 -0.375 0.268 1.000 0.000 0.575 0.000 0.025 -0.445 0.000 0.467 -0.071 0.057 0.000 -0.538
-1.000 0.170 0.533 0.337 1.000 0.991 -0.068 0.020 0.000 -1.000 1.000 -0.033 -0.281 0.066 0.151 0.000 -0.688 -0.054 1.000 0.000 0.602 -0.057 -0.057 -0.339 0.252 0.467 -0.893 -1.000 0.138 -0.538
-1.000 -0.272 0.545 1.000 1.000 0.991 -0.068 -0.106 -1.000 1.000 1.000 0.011 1.000 0.066 0.151 -0.171 -0.813 0.054 1.000 -0.066 0.627 -0.057 -0.062 0.350 0.248 0.000 -1.000 -1.000 0.151 -0.538
Table 21 Per-task bounded normalized scores after the first intervention in the main limited budget evaluation. Scores are bounded to [−1, 1].
32
Task
Q8
Q32
Gem
DS14
Phi
Llama DSV4
GPT
G4.7
G5.1
Kimi Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
-1.000 -0.132 0.197 -0.636 0.006 0.953 -0.008 0.181 0.000 -0.667 0.985 0.004 -0.364 0.185 0.151 -0.095 -0.542 -0.143 0.728 -0.147 0.635 0.007 -0.104 -0.445 0.248 0.533 -0.024 -1.000 0.075 -0.566
-1.000 0.032 0.533 -0.187 0.990 0.991 0.030 0.062 -0.252 1.000 0.756 0.004 -0.087 0.185 0.151 0.038 -0.604 -0.780 0.728 -0.194 0.676 0.000 -0.143 -0.445 0.260 0.533 0.000 -0.973 0.075 -0.591
-1.000 0.014 0.166 -0.708 0.971 0.667 0.049 0.070 -0.252 -1.000 0.985 0.004 -0.364 0.185 0.151 0.076 -0.667 -0.601 0.728 0.000 0.413 0.000 -0.005 -0.445 0.242 0.378 0.000 -0.484 0.075 -0.541
-1.000 -0.145 0.008 -1.000 0.005 0.991 0.027 0.035 0.000 -0.667 0.985 0.006 -0.364 0.185 0.151 0.114 -0.667 -0.869 0.728 -0.077 0.645 0.000 0.057 -0.445 0.248 0.533 0.000 -0.121 0.076 -0.573
-1.000 -0.053 0.246 0.187 0.971 0.953 -0.068 0.157 0.000 1.000 0.947 0.002 -0.364 0.185 0.151 0.086 -0.667 -0.339 0.699 -0.164 0.429 -0.057 -0.110 -0.445 0.247 0.533 0.000 -0.616 0.075 -0.584
-1.000 -0.073 0.543 -0.674 0.304 1.000 -0.025 -0.011 0.000 -0.245 0.985 0.001 -1.000 0.185 0.151 -0.505 -0.667 -0.351 0.728 0.000 0.427 0.000 -0.123 -0.445 0.243 0.378 0.000 -0.712 0.075 -0.591
-1.000 -0.026 0.533 1.000 1.000 0.990 -0.022 0.026 0.000 0.000 1.000 0.000 1.000 0.389 0.151 0.114 0.188 0.054 1.000 0.000 0.999 0.020 0.088 0.023 0.166 0.000 -1.000 -1.000 0.056 -0.548
-1.000 0.292 0.003 -1.000 1.000 0.991 -0.068 -0.089 -1.000 1.000 0.982 0.006 1.000 0.066 0.151 -1.000 -0.375 -0.107 1.000 0.000 0.585 0.020 0.110 -0.445 0.265 0.467 -0.071 -1.000 0.000 -0.570
-1.000 -0.093 0.003 -1.000 1.000 0.991 -0.068 -0.089 -1.000 1.000 0.982 0.006 1.000 0.066 0.151 -1.000 -0.250 0.643 0.917 0.000 0.598 0.020 0.093 -0.445 0.250 0.467 -0.071 -1.000 0.000 -0.538
-1.000 -0.041 0.008 -0.561 1.000 0.000 -0.010 -0.089 -1.000 1.000 0.982 0.006 1.000 0.066 0.151 0.057 -0.188 0.161 1.000 0.000 0.656 0.000 0.001 -0.445 0.229 0.467 -0.071 -1.000 0.000 -0.559
-1.000 -0.285 0.533 0.561 1.000 0.991 -0.068 0.026 -0.189 0.000 1.000 0.000 -1.000 0.066 0.151 -0.286 0.188 -0.107 1.000 -0.025 0.836 -0.032 0.015 -0.781 0.261 0.200 -0.036 0.193 0.069 -0.634
-1.000 0.136 0.545 0.898 1.000 0.991 -0.068 0.037 0.000 1.000 1.000 0.011 1.000 0.247 -0.300 0.057 0.188 0.590 1.000 0.066 0.997 0.000 0.091 -0.216 0.547 0.000 -0.071 0.429 0.151 -0.570
Table 22 Per-task bounded normalized scores after the second intervention in the main limited budget evaluation. Scores are bounded to [−1, 1].
33
Task
Q8
Q32
Gem
DS14
Phi
Llama DSV4
GPT
G4.7
G5.1
Kimi Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
-0.705 -0.096 0.197 -0.636 0.328 0.953 0.054 0.159 0.000 -0.667 0.985 0.001 -0.364 0.185 0.151 0.114 -0.542 -0.357 0.728 -0.220 0.637 -0.002 0.003 -0.445 0.232 0.533 -0.298 -0.365 0.075 -0.527
-0.568 -0.030 0.533 -0.187 0.990 0.991 0.026 0.057 0.000 1.000 0.992 -0.017 -0.088 0.185 0.151 -0.067 -0.604 -0.572 0.728 -0.257 0.651 -0.002 -0.034 -0.445 0.241 0.533 0.000 -0.403 0.075 -0.541
-0.784 -0.123 0.166 -0.782 0.971 0.635 0.026 0.167 -0.252 -1.000 0.985 0.011 -0.364 0.185 0.151 -0.229 -0.667 -0.536 0.874 0.000 0.441 0.000 -0.021 -0.445 0.266 0.378 0.000 -0.533 0.075 -0.581
-0.694 -0.040 0.008 -0.037 0.010 0.991 0.032 0.039 0.000 -0.667 0.985 0.006 -0.364 0.185 0.151 0.038 -0.667 -0.548 0.728 -0.077 0.646 0.000 0.081 -0.520 0.240 0.533 0.000 -0.120 0.083 -0.566
-0.873 -0.145 0.246 -0.112 0.971 0.953 0.259 0.151 0.000 1.000 0.762 0.001 -0.570 0.185 0.151 -0.190 -0.812 -0.339 0.728 -0.246 0.438 0.000 -0.015 -0.445 0.250 0.378 0.000 -0.214 0.075 -0.584
-0.746 -0.037 0.548 -0.674 0.980 0.994 0.064 -0.105 0.000 -0.912 0.985 0.009 -0.364 0.185 0.151 -0.019 -0.354 -0.339 0.728 -0.032 0.432 0.000 -0.170 -0.445 0.250 0.378 0.000 -0.487 0.075 -0.573
-0.694 -0.135 0.533 1.000 1.000 0.990 -0.068 0.026 -0.660 0.000 1.000 0.000 1.000 0.420 0.151 -0.143 0.563 0.161 1.000 -0.007 0.998 0.032 0.207 0.023 0.344 0.000 0.000 -1.000 -1.000 -0.602
-0.782 0.055 0.003 -1.000 1.000 0.991 -0.068 -0.033 -1.000 1.000 0.982 0.006 1.000 0.066 0.151 -0.943 0.000 0.268 1.000 0.000 0.627 -0.005 0.111 -0.445 0.260 0.467 -0.893 0.542 0.000 -0.602
-0.782 -0.066 0.003 -1.000 1.000 0.991 -0.068 -0.035 -1.000 1.000 0.982 0.006 1.000 0.066 0.151 -1.000 0.062 0.375 1.000 0.000 0.533 -0.005 0.093 -0.445 0.285 0.467 -0.893 0.553 0.000 -0.602
-0.675 -0.242 0.008 0.112 1.000 0.000 -0.068 0.037 -1.000 1.000 0.982 0.006 1.000 0.066 0.000 -1.000 0.062 0.322 1.000 0.000 0.647 0.000 -0.007 -0.445 0.245 0.467 0.000 -0.352 0.000 -0.559
-0.694 0.236 0.533 0.561 1.000 0.991 -0.027 -0.043 0.000 -1.000 1.000 -0.002 -1.000 0.066 0.151 0.029 0.188 -0.482 1.000 0.000 0.754 -0.032 -0.184 -0.037 0.259 0.467 -0.036 -1.000 0.138 -0.548
-0.694 0.344 0.545 1.000 1.000 0.991 -0.039 0.000 0.566 1.000 1.000 0.011 1.000 0.420 0.151 -0.086 1.000 0.804 0.917 0.000 0.992 -0.984 0.098 0.105 0.210 0.000 0.000 0.492 0.215 -0.527
Table 23 Per-task bounded normalized scores after the third intervention in the main limited budget evaluation. Scores are bounded to [−1, 1].
34
Task
Q8
Q32
Gem
DS14
Phi
Llama DSV4
GPT
G4.7
G5.1
Kimi Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
-0.201 -0.041 0.197 -0.748 0.667 0.953 0.032 0.243 0.000 -0.667 0.673 -0.020 -0.364 0.185 0.151 0.038 -0.542 0.107 0.728 -0.147 0.605 -0.328 -0.004 -0.445 0.247 0.533 -0.048 -0.709 0.075 -0.566
-0.076 0.020 0.560 -0.187 0.990 0.991 0.023 -0.005 -0.252 1.000 0.992 0.003 -0.088 0.185 0.151 0.190 -0.604 -0.834 0.728 -0.257 0.666 -0.008 0.003 -0.445 0.245 0.533 0.000 -1.000 0.075 -0.599
-0.327 -0.016 0.166 -0.857 0.971 0.667 0.033 0.136 -0.252 -1.000 0.780 0.003 -0.364 0.185 0.151 0.038 -0.667 -0.107 0.728 0.000 0.621 0.000 0.001 -0.445 0.251 0.378 0.000 -0.644 0.075 -0.566
-0.183 -0.194 0.008 -0.932 0.667 0.991 0.004 0.016 0.000 -0.667 0.985 0.006 -0.364 0.185 0.151 -0.076 -0.667 -0.357 0.728 -0.077 0.638 0.000 0.084 0.000 0.249 0.533 0.000 -0.436 0.075 -0.543
-0.411 0.002 0.246 0.224 0.971 0.953 0.216 0.199 -0.252 1.000 0.741 -0.006 -0.697 0.185 0.151 -0.352 -0.667 -0.155 0.728 -0.246 0.477 0.000 -0.007 -0.445 0.240 0.378 0.000 -0.326 0.075 -0.573
-0.421 -0.111 0.424 -0.112 1.000 0.978 -0.068 -0.081 0.000 -0.912 0.985 -0.020 -0.364 0.185 0.151 -0.429 -0.458 -0.405 0.728 -0.032 0.430 0.000 -0.109 -0.622 0.248 0.378 0.000 -0.963 -0.302 -0.581
-0.183 -0.231 0.533 1.000 1.000 0.990 -0.046 0.037 0.000 0.000 1.000 0.000 1.000 0.417 0.151 0.086 0.563 0.107 1.000 0.000 0.998 -1.000 0.208 0.023 0.370 0.000 0.000 0.565 0.056 -0.548
-0.330 0.228 0.003 -1.000 1.000 0.991 -0.068 -0.033 -1.000 1.000 0.982 0.006 1.000 0.066 0.151 -0.371 -0.688 0.268 1.000 0.000 0.586 -0.984 0.126 -0.445 0.242 0.467 -0.143 0.450 0.000 -0.591
-0.330 -0.020 0.003 -1.000 1.000 0.991 -0.068 -0.089 -1.000 1.000 0.982 0.006 1.000 0.066 0.151 -0.200 -0.688 0.161 1.000 0.000 0.604 -0.984 0.142 -0.445 0.248 0.467 -0.143 0.448 0.000 -0.613
-0.265 -0.102 0.008 -1.000 1.000 0.991 -0.068 0.006 -1.000 1.000 0.982 0.006 0.000 0.066 0.000 -1.000 -0.688 0.107 1.000 0.000 0.647 0.000 0.002 -0.445 0.000 0.467 -0.143 0.124 0.000 -0.559
-0.183 -0.026 0.528 0.224 1.000 0.991 -0.032 0.033 0.000 -1.000 1.000 0.000 1.000 0.066 0.151 0.057 0.188 -0.536 1.000 0.000 0.775 0.027 -0.060 -0.445 0.238 0.467 0.000 -0.496 0.138 -0.548
-0.183 -0.149 0.545 1.000 1.000 0.991 0.019 0.033 0.755 1.000 1.000 -0.046 1.000 0.417 0.151 0.000 1.000 0.107 1.000 -0.044 0.998 -0.046 0.096 -0.889 0.585 0.133 -0.036 0.615 0.021 -0.548
Table 24 Per-task bounded normalized scores after the fourth intervention in the main limited budget evaluation. Scores are bounded to [−1, 1].
35
Task
Q8
Q32
Gem
DS14
Phi
Llama DSV4
GPT
G4.7
G5.1
Kimi Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
-0.036 -0.064 0.197 -0.711 0.665 0.953 0.054 0.243 0.000 -0.667 0.985 -0.020 -0.364 0.185 0.151 -0.124 -0.542 -0.179 0.728 -0.220 0.650 -0.026 0.004 -0.445 0.256 0.533 -0.333 -0.662 0.075 -0.559
0.077 0.041 0.560 -0.112 0.990 0.990 0.120 0.012 0.000 1.000 1.000 -0.017 -0.088 0.185 0.151 0.086 -0.604 -0.464 0.728 -0.257 0.626 0.000 0.001 -0.445 0.235 0.533 0.000 -0.395 0.075 -0.577
-0.014 -0.047 0.166 -0.782 0.971 0.635 0.049 0.141 -0.252 -1.000 0.780 0.006 -0.364 0.185 0.151 0.067 -0.667 -0.268 0.728 0.000 0.420 0.000 -0.003 -0.445 0.242 0.378 0.000 -0.605 0.075 -0.582
0.000 -0.232 0.008 -1.000 0.678 0.991 0.019 0.039 0.000 -0.667 0.985 0.006 -0.364 0.185 0.151 0.076 -0.667 -0.536 0.728 -0.077 0.690 0.000 0.075 0.000 0.246 0.533 0.000 -0.922 0.149 -0.556
-0.205 -0.065 0.246 0.112 0.971 0.953 0.342 0.208 -0.333 1.000 0.975 0.005 -0.364 0.185 0.151 0.057 -0.562 -0.107 0.728 -0.246 0.427 0.000 -0.018 -0.445 0.242 0.378 0.000 -0.264 0.075 -0.532
-0.081 -0.110 0.540 -0.449 0.990 1.000 0.129 -0.154 -0.050 -0.245 0.985 0.009 -0.364 0.185 0.151 -0.257 -1.000 -0.458 0.728 -0.032 0.446 0.000 -0.010 -0.445 0.247 0.378 -0.060 -0.406 -0.173 -0.599
-0.067 0.011 0.533 1.000 1.000 0.990 -0.016 0.037 0.000 0.000 1.000 0.000 1.000 0.420 0.151 -0.029 0.563 0.482 1.000 0.000 0.997 0.032 0.207 0.023 0.370 0.467 0.000 -0.421 0.056 -0.645
-0.202 0.178 0.003 -1.000 1.000 0.991 -0.068 -0.033 -1.000 1.000 0.982 0.006 1.000 0.066 0.151 0.086 -1.000 0.429 1.000 0.000 0.718 -0.077 0.120 -0.445 0.241 0.467 -1.000 0.329 0.000 -0.570
-0.202 0.127 0.003 -1.000 1.000 0.991 -0.068 -0.033 -1.000 1.000 0.982 0.006 1.000 0.066 0.151 0.229 -1.000 0.536 1.000 0.000 0.628 -0.077 0.087 -0.445 0.247 0.467 -1.000 -0.724 0.000 -0.570
-0.147 0.327 0.008 0.337 1.000 0.991 -0.018 0.016 -1.000 1.000 0.982 0.006 1.000 0.066 0.151 -0.714 -1.000 -0.107 1.000 0.000 0.594 0.000 0.125 -0.445 0.242 0.467 0.000 -1.000 0.000 -0.591
-0.067 0.059 0.533 0.224 1.000 0.991 -0.017 0.026 0.000 -1.000 1.000 -0.002 1.000 0.066 0.151 0.200 0.188 0.214 1.000 0.000 0.532 0.027 -0.007 1.000 0.255 0.467 0.000 -1.000 -1.000 -0.602
-0.067 0.157 0.545 1.000 1.000 0.991 -0.017 0.033 0.755 1.000 1.000 0.011 1.000 0.204 0.151 -0.057 1.000 0.536 1.000 -0.066 0.996 0.007 0.228 -0.461 0.585 0.133 0.000 0.606 0.547 -0.602
Table 25 Per-task bounded normalized scores after the fifth intervention in the main limited budget evaluation. Scores are bounded to [−1, 1].
36
Task
Q8
Q32 Gem DS14
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
56.6 31.9 2.6 27.1 32.9 99.6 44.0 94.9 95.3 96.4 99.8 1.7 50.7 85.1 73.5 98.1 95.5 97.4 98.4 4.3 68.6 39.7 12.0 98.2 98.7 53.3 89.0 5.4 92.7 16.9
58.4 27.5 21.2 52.6 94.2 99.9 39.2 95.0 95.6 103.1 99.8 -3.1 55.9 85.1 73.5 98.1 86.3 97.1 98.4 4.2 73.0 39.7 6.6 97.1 98.7 53.3 94.6 11.5 92.7 16.9
57.7 29.6 18.2 22.3 94.2 97.2 45.9 95.3 94.9 86.0 99.8 2.1 50.7 85.1 73.5 98.3 95.5 97.7 98.4 5.0 66.4 39.7 20.9 97.5 99.0 37.8 94.6 13.7 92.7 16.9
Phi
57.7 56.5 33.2 40.0 2.6 26.0 8.0 62.2 32.6 94.2 99.9 99.6 44.3 55.7 94.9 95.4 95.3 93.4 96.4 100.0 99.8 100.2 1.7 -0.9 50.7 50.7 85.1 85.1 73.5 73.5 98.5 96.6 95.5 94.2 96.9 97.6 98.4 98.4 4.7 4.3 72.3 53.1 39.7 38.4 24.9 9.0 97.5 97.5 98.7 98.7 53.3 37.8 94.6 77.5 8.1 4.8 92.7 92.7 16.9 16.9
Llama DSV4 GPT G4.7 G5.1 Kimi Claude 56.6 21.0 21.8 38.3 31.8 99.6 39.2 94.4 95.1 93.4 99.8 2.1 50.7 85.1 73.5 98.1 95.0 97.8 97.6 5.0 50.6 39.7 11.3 97.5 98.7 53.3 94.3 9.4 89.0 16.9
57.7 45.6 54.2 71.7 100.0 99.9 39.2 94.9 98.5 99.5 101.7 -2.1 51.7 83.3 73.5 98.2 96.8 98.3 100.9 5.0 66.9 38.4 16.5 97.7 98.7 46.7 89.7 1.4 93.1 16.9
57.7 31.2 54.2 138.7 100.0 99.9 39.2 94.7 97.4 99.8 100.9 -2.1 162.3 83.3 73.5 97.8 96.6 99.0 100.9 4.9 68.7 38.9 34.7 97.8 98.9 0.0 94.4 0.7 92.5 16.9
57.7 47.2 2.2 0.0 32.6 92.3 39.2 94.9 98.5 99.8 99.7 1.1 130.9 83.3 73.5 97.3 97.4 98.6 100.9 5.0 69.4 39.7 30.5 97.5 98.7 46.7 77.9 12.9 92.1 16.9
57.7 29.9 2.2 0.0 30.0 92.3 39.2 94.9 98.5 99.8 99.7 1.1 130.9 83.3 73.5 97.7 97.4 98.8 100.9 5.0 64.6 39.7 32.7 97.5 98.8 46.7 77.9 3.7 92.1 16.9
57.7 32.5 2.6 57.4 33.2 92.3 39.2 94.8 98.5 99.8 99.7 1.1 130.9 83.3 70.2 98.1 97.4 98.8 100.9 5.0 64.6 39.7 22.9 97.5 98.3 46.7 94.2 14.8 92.1 16.9
57.7 16.6 55.4 119.6 100.0 99.9 39.2 94.2 96.5 102.8 102.0 2.1 184.8 83.3 73.5 97.9 96.6 98.4 100.9 4.7 69.0 38.4 16.1 98.9 98.7 0.0 77.9 1.9 93.2 16.9
Table 26 Per-task anchor attainment after the first intervention in the main limited budget evaluation, reported as percentages.
37
Task
Q8
Q32 Gem DS14
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
62.9 25.8 21.2 30.3 32.6 99.6 42.6 95.7 95.3 96.4 99.8 1.4 50.7 85.1 73.5 98.1 94.1 98.1 98.4 4.5 69.6 39.9 12.7 97.5 98.7 53.3 94.4 3.1 92.7 15.3
64.5 36.5 54.2 49.4 98.1 99.9 44.8 95.1 94.9 103.5 97.3 1.5 61.7 85.1 73.5 98.3 94.8 97.0 98.4 4.2 73.0 39.7 9.7 97.5 98.8 53.3 94.6 6.0 92.7 14.0
62.1 35.3 18.2 25.5 94.2 97.9 45.9 95.1 94.9 86.0 99.8 1.5 50.7 85.1 73.5 98.4 95.5 97.4 98.4 5.0 51.2 39.7 20.6 97.5 98.7 37.8 94.6 12.5 92.7 16.7
Phi
Llama DSV4 GPT G4.7 G5.1 Kimi Claude
63.3 61.5 61.4 24.9 30.9 29.6 2.6 26.0 55.1 0.0 65.4 28.7 32.6 94.2 52.7 99.9 99.6 100.7 44.6 39.2 41.7 95.0 95.6 94.7 95.3 95.3 95.9 96.4 102.7 99.3 99.8 99.8 99.8 1.7 1.3 1.2 49.7 50.7 34.7 85.1 85.1 85.1 73.5 73.5 73.5 98.4 98.4 73.4 95.5 95.5 93.1 96.9 97.8 97.8 98.4 98.2 98.4 4.7 4.3 5.0 70.5 52.5 52.3 39.7 38.4 39.7 25.5 12.3 11.3 97.5 97.5 97.5 98.7 98.7 98.7 53.3 53.3 37.8 94.6 94.6 94.6 13.1 9.7 8.1 92.7 92.7 92.7 14.9 14.3 14.0
63.3 15.7 54.2 81.3 100.0 99.9 39.2 94.9 98.2 99.8 101.8 1.1 9.5 83.3 73.5 97.7 98.5 98.2 100.9 4.9 86.3 39.0 22.2 96.9 98.8 20.0 94.4 16.8 92.6 11.6
63.3 32.7 54.2 138.7 100.0 99.9 41.8 94.9 98.5 99.8 100.9 1.1 162.3 88.4 73.5 98.4 98.5 98.4 100.9 5.0 99.9 40.2 27.9 98.3 98.6 0.0 80.0 6.8 92.5 16.3
62.8 53.6 2.2 4.8 100.0 99.9 39.2 94.3 93.9 104.1 99.8 1.7 130.9 83.3 73.5 96.2 97.4 98.2 100.9 5.0 65.4 40.2 29.7 97.5 98.8 46.7 94.2 0.4 92.1 15.1
62.8 28.3 2.2 9.6 100.0 99.9 39.2 94.3 93.9 104.1 99.8 1.7 130.9 83.3 73.5 96.1 97.6 99.4 99.4 5.0 66.5 40.2 28.3 97.5 98.7 46.7 94.2 1.0 92.1 16.9
63.3 31.7 2.6 33.5 100.0 92.3 42.5 94.3 93.9 104.1 99.8 1.7 130.9 83.3 73.5 98.3 97.8 98.6 100.9 5.0 71.3 39.7 21.1 97.5 98.7 46.7 94.2 7.4 92.1 15.7
63.3 43.3 55.4 95.7 100.0 99.9 39.2 95.0 98.5 105.6 102.0 2.1 184.8 86.1 64.4 98.3 98.5 99.3 101.1 5.4 99.7 39.7 28.2 97.9 99.2 0.0 94.2 22.2 93.2 15.1
Table 27 Per-task anchor attainment after the second intervention in the main limited budget evaluation, reported as percentages.
38
Task
Q8
Q32 Gem DS14
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
68.6 28.1 21.2 30.3 53.1 99.6 46.2 95.6 95.3 96.4 99.8 1.2 50.7 85.1 73.5 98.4 94.7 97.8 98.4 4.4 69.8 39.7 21.2 97.5 98.7 53.3 93.0 12.0 92.7 17.4
70.4 32.5 54.2 49.4 98.0 99.9 44.6 95.1 95.3 103.5 100.1 -0.7 61.0 85.1 73.5 98.1 95.5 97.4 98.4 4.0 70.9 39.7 18.3 97.5 98.7 53.3 94.6 14.3 92.7 16.7
67.7 26.3 18.2 20.7 94.2 97.2 44.6 95.6 94.9 86.0 99.8 2.1 50.7 85.1 73.5 97.8 95.5 97.5 99.4 5.0 53.5 39.7 19.4 97.5 98.8 37.8 94.6 8.2 92.7 14.5
Phi
Llama DSV4 GPT G4.7 G5.1 Kimi Claude
68.7 66.5 68.1 31.8 24.9 32.0 2.6 26.0 55.7 55.8 52.6 28.7 32.7 94.2 96.1 99.9 99.6 100.2 44.9 57.9 46.8 95.0 95.6 94.2 95.3 95.9 95.9 96.4 103.0 93.4 99.8 97.3 99.8 1.7 1.1 2.0 50.7 47.0 50.7 85.1 85.1 85.1 73.5 73.5 73.5 98.3 97.9 98.2 95.5 94.9 92.4 97.4 97.8 97.8 98.4 98.4 98.4 4.7 4.1 4.9 70.6 53.2 52.7 39.7 39.7 39.7 27.4 19.8 7.5 97.4 97.5 97.5 98.7 98.7 98.7 53.3 37.8 37.8 94.6 94.6 94.6 17.0 13.8 12.7 92.7 92.7 92.7 15.3 14.3 14.9
68.7 49.9 54.2 81.3 100.0 99.9 41.6 94.5 98.5 99.5 101.8 0.9 12.1 83.3 73.5 98.3 98.5 97.6 100.9 5.0 79.5 39.0 6.4 98.2 98.8 46.7 94.4 6.8 93.1 16.3
68.7 25.6 54.2 138.7 100.0 99.9 39.2 94.9 97.5 99.8 100.9 1.1 162.3 88.9 73.5 98.0 99.2 98.6 100.9 5.0 99.9 40.5 37.4 98.3 98.9 0.0 94.6 7.2 81.0 13.4
67.6 38.0 2.2 9.6 100.0 99.9 39.2 94.6 93.9 104.1 99.8 1.7 130.9 83.3 73.5 96.6 98.1 98.8 100.9 5.0 68.9 39.6 29.8 97.5 98.8 46.7 89.7 26.3 92.1 13.4
67.6 30.1 2.2 9.6 100.0 99.9 39.2 94.6 93.9 104.1 99.8 1.7 130.9 83.3 73.5 95.4 98.2 99.0 100.9 5.0 61.2 39.6 28.4 97.5 98.8 46.7 89.7 26.8 92.1 13.4
69.0 18.6 2.6 62.2 100.0 92.3 39.2 95.0 93.9 104.1 99.8 1.7 130.9 83.3 70.2 96.1 98.2 98.9 100.9 5.0 70.6 39.7 20.5 97.5 98.7 46.7 94.6 10.8 92.1 15.7
68.7 57.0 55.4 124.3 100.0 99.9 40.9 94.8 99.4 104.3 102.2 2.1 258.1 88.9 73.5 98.1 100.1 99.7 99.4 5.0 99.4 25.0 28.8 98.4 98.7 0.0 94.6 24.3 93.7 17.4
Table 28 Per-task anchor attainment after the third intervention in the main limited budget evaluation, reported as percentages.
39
Task
Q8
Q32 Gem DS14
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
75.6 31.7 21.2 25.5 75.8 99.6 44.9 96.0 95.3 96.4 95.6 -0.9 50.7 85.1 73.5 98.3 95.1 98.5 98.4 4.5 67.1 34.8 20.7 97.5 98.7 53.3 94.3 8.0 92.7 15.3
77.6 35.7 56.8 49.4 98.1 99.9 44.4 94.7 94.7 103.5 100.5 1.4 61.0 85.1 73.5 98.6 95.7 96.9 98.4 4.0 72.2 39.6 21.3 97.5 98.7 53.3 94.6 4.3 92.7 13.6
73.8 33.3 18.2 19.1 94.2 97.9 45.0 95.5 94.9 86.0 97.0 1.3 50.7 85.1 73.5 98.3 95.5 98.2 98.4 5.0 68.4 39.7 21.1 97.5 98.7 37.8 94.6 9.7 92.7 15.3
Phi
Llama DSV4 GPT G4.7 G5.1 Kimi Claude
75.9 72.5 72.4 21.7 34.5 27.2 2.6 26.0 43.5 17.5 67.0 52.6 77.5 94.2 100.0 99.9 99.6 99.8 43.3 55.4 39.2 94.9 95.8 94.3 95.3 95.6 95.9 96.4 103.6 93.5 99.8 97.1 99.8 1.7 0.5 -0.9 50.7 27.2 50.7 85.1 85.1 85.1 73.5 73.5 73.5 98.1 97.3 73.5 95.5 95.3 92.5 97.8 98.0 97.7 98.4 98.4 98.4 4.7 4.1 4.9 69.9 56.5 52.5 39.7 39.7 39.7 27.6 20.5 12.4 98.2 97.5 97.2 98.7 98.7 98.7 53.3 37.8 37.8 94.6 94.6 94.6 10.4 11.3 4.5 92.7 92.7 90.0 16.6 14.9 14.5
75.9 32.7 53.7 67.0 100.0 99.9 41.3 94.9 98.5 98.3 101.8 1.1 108.0 83.3 73.5 98.3 98.5 97.5 100.9 5.0 81.2 40.4 16.3 97.5 98.7 46.7 94.6 9.8 93.1 16.3
75.9 19.3 54.2 138.7 100.0 99.9 40.5 95.0 98.5 99.8 100.7 1.1 162.3 88.9 73.5 98.4 99.2 98.5 100.9 5.0 99.8 24.5 37.4 98.3 98.9 0.0 94.6 27.3 92.5 16.3
73.7 49.4 2.2 4.8 100.0 99.9 39.2 94.6 93.9 104.1 99.8 1.7 130.9 83.3 73.5 97.6 96.8 98.8 100.9 5.0 65.5 25.0 30.9 97.5 98.7 46.7 93.8 22.9 92.1 14.0
73.7 33.1 2.2 0.0 100.0 99.9 39.2 94.3 93.9 104.1 99.8 1.7 130.9 83.3 73.5 97.9 96.8 98.6 100.9 5.0 67.1 25.0 32.2 97.5 98.7 46.7 93.8 22.9 92.1 12.8
74.6 27.7 2.6 9.6 100.0 99.9 39.2 94.8 93.9 104.1 99.8 1.7 57.8 83.3 70.2 96.2 96.8 98.5 100.9 5.0 70.6 39.7 21.2 97.5 98.3 46.7 93.8 15.7 92.1 15.7
75.9 24.6 55.4 124.3 100.0 99.9 44.2 94.9 99.6 105.3 101.7 -3.5 258.1 88.9 73.5 98.2 100.1 98.5 100.9 4.8 99.8 38.7 28.6 96.7 99.3 13.3 94.4 29.8 92.3 16.3
Table 29 Per-task anchor attainment after the fourth intervention in the main limited budget evaluation, reported as percentages.
40
Task
Q8
Q32 Gem DS14
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
78.3 30.2 21.2 27.1 75.8 99.6 46.2 96.0 95.3 96.4 99.8 -0.9 50.7 85.1 73.5 98.0 94.7 98.1 98.4 4.4 70.9 39.2 21.3 97.5 98.8 53.3 88.2 8.2 92.7 15.7
80.1 37.1 56.8 52.6 98.0 99.9 49.9 94.8 95.3 103.5 100.9 -0.7 61.0 85.1 73.5 98.4 95.7 97.6 98.4 4.0 68.8 39.7 21.1 97.5 98.7 53.3 94.6 10.1 92.7 14.7
78.6 31.4 18.2 22.3 94.2 97.2 45.9 95.5 94.9 86.0 97.0 1.7 50.7 85.1 73.5 98.4 95.5 97.9 98.4 5.0 51.7 39.7 20.8 97.5 98.7 37.8 94.6 9.3 92.7 14.4
Phi
Llama DSV4 GPT G4.7 G5.1 Kimi Claude
78.8 75.6 77.5 19.2 30.2 27.2 2.6 26.0 54.9 8.0 62.2 38.3 77.7 94.2 98.1 99.9 99.6 100.7 44.2 62.5 50.5 95.0 95.9 94.0 95.3 92.8 95.5 96.4 103.5 99.2 99.8 100.2 99.8 1.7 1.6 2.0 50.7 50.7 45.0 85.1 85.1 85.1 73.5 73.5 73.5 98.4 98.3 97.8 95.5 93.5 92.5 97.5 98.2 97.4 98.4 98.4 98.4 4.7 4.1 4.9 74.2 52.3 53.9 39.7 39.7 39.7 27.0 19.6 20.2 98.2 97.5 97.5 98.7 98.7 98.7 53.3 37.8 37.8 94.6 94.6 94.3 5.4 12.1 10.9 93.2 92.7 90.9 15.9 17.2 13.6
77.7 38.3 54.2 67.0 100.0 99.9 42.1 94.9 98.5 98.3 102.0 0.9 108.0 83.3 73.5 98.6 98.5 98.7 100.9 5.0 61.0 40.4 20.5 100.1 98.8 46.7 94.6 5.6 80.5 13.4
77.7 35.2 54.2 138.7 100.0 99.9 42.2 95.0 98.5 99.8 100.9 1.1 162.3 88.9 73.5 98.2 99.2 99.1 100.9 5.0 99.8 40.5 37.3 98.3 98.9 46.7 94.6 10.3 92.5 11.0
75.6 46.1 2.2 0.0 100.0 99.9 39.2 94.6 93.9 104.1 99.8 1.7 130.9 83.3 73.5 98.4 94.9 99.1 100.9 5.0 76.5 38.0 30.5 97.5 98.7 46.7 75.6 19.6 92.1 15.1
75.6 42.7 2.2 0.0 100.0 99.9 39.2 94.6 93.9 104.1 99.8 1.7 130.9 83.3 73.5 98.6 94.9 99.2 100.9 5.0 69.1 38.0 27.9 97.5 98.7 46.7 75.6 8.7 92.1 15.1
76.5 55.9 2.6 71.7 100.0 99.9 42.1 94.9 93.9 104.1 99.8 1.7 130.9 83.3 73.5 97.0 94.0 98.2 100.9 5.0 66.2 39.7 30.8 97.5 98.7 46.7 94.6 7.2 92.1 14.0
77.7 44.7 55.4 119.6 100.0 99.9 42.1 94.9 99.6 104.3 102.0 2.1 258.1 85.4 73.5 98.1 100.1 99.2 100.9 4.7 99.6 39.9 39.0 97.5 99.3 13.3 94.6 29.3 96.3 13.4
Table 30 Per-task anchor attainment after the fifth intervention in the main limited budget evaluation, reported as percentages.
41
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
3.548 41.390 22.000 0.379 0.375 0.509 35.450 93.420 70.000 0.928 85.520 0.067 0.334 0.504 0.121 96.700 82.900 84.460 0.979 70.710 0.996 0.372 54.370 0.185 0.175 0.280 40.200 7.690 0.340 5.000
3.548 36.860 2.200 0.069 0.375 0.509 35.450 93.560 69.800 0.968 85.530 0.067 0.334 0.504 0.121 96.800 82.900 84.000 0.979 70.710 0.997 0.372 56.200 0.185 0.175 0.280 40.200 8.910 0.340 4.800
3.529 35.500 46.000 1.000 0.375 0.509 35.450 93.460 70.400 0.982 87.480 0.999 0.294 0.504 0.121 96.000 82.300 83.240 0.979 72.680 0.997 0.385 78.170 0.182 0.175 0.000 40.200 5.790 0.340 5.800
Table 31 Per-task raw metric values after the first intervention in the full budget evaluation.
42
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
3.508 35.580 22.000 0.379 0.375 0.509 35.450 93.610 69.800 0.968 85.530 0.298 0.334 0.504 0.121 95.350 83.100 84.530 0.979 70.710 0.996 0.368 57.140 0.185 0.174 0.280 48.600 6.060 0.340 4.400
3.508 60.550 2.200 0.000 0.375 0.509 35.450 93.560 69.800 0.968 85.530 1.000 0.334 0.504 0.121 95.350 83.200 84.690 0.979 70.710 0.998 0.368 53.200 0.185 0.175 0.280 48.600 6.070 0.340 4.800
3.466 48.220 46.000 0.690 0.375 0.509 35.450 93.730 69.000 0.970 87.640 1.000 0.341 0.494 0.122 97.200 83.800 85.370 0.981 64.000 0.997 0.368 76.580 0.185 0.180 0.000 48.700 5.610 0.341 7.500
Table 32 Per-task raw metric values after the second intervention in the full budget evaluation.
43
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
3.488 29.200 22.000 0.379 0.375 0.509 35.450 93.680 69.800 0.968 85.530 0.298 0.334 0.504 0.121 95.500 82.600 84.990 0.979 70.710 0.994 0.373 54.680 0.185 0.175 0.280 46.300 5.450 0.340 2.800
3.488 50.820 2.200 0.414 0.375 0.509 59.790 93.560 69.800 0.968 85.530 0.298 0.334 0.504 0.121 95.500 82.700 86.050 0.979 70.710 0.996 0.373 54.640 0.185 0.175 0.280 46.300 6.450 0.340 4.400
3.435 41.630 45.800 1.000 0.375 0.509 67.010 93.680 70.400 0.982 87.420 1.000 0.302 0.493 0.114 97.450 83.900 85.290 0.979 78.770 0.997 0.372 75.090 0.184 0.175 0.080 48.600 6.220 0.325 5.400
Table 33 Per-task raw metric values after the third intervention in the full budget evaluation.
44
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
3.476 32.950 22.000 0.379 0.375 0.509 35.450 93.680 69.800 0.968 85.530 0.298 0.334 0.504 0.121 97.750 82.400 84.760 0.979 70.710 0.995 0.593 50.450 0.185 0.174 0.280 48.400 17.310 0.340 2.800
3.476 49.040 2.200 0.138 0.375 0.509 35.450 93.560 69.800 0.968 85.530 0.298 0.334 0.504 0.121 96.850 82.400 85.670 0.979 70.710 0.997 0.593 52.380 0.185 0.176 0.280 48.400 22.490 0.340 5.400
3.415 39.630 45.800 1.000 0.375 0.509 67.540 93.740 70.900 0.965 87.240 1.000 0.404 0.493 0.119 96.100 83.900 86.660 0.979 80.490 0.998 0.372 74.820 0.186 0.175 0.160 46.300 5.570 0.335 5.400
Table 34 Per-task raw metric values after the fourth intervention in the full budget evaluation.
45
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
3.468 59.570 22.000 0.379 0.375 0.509 35.450 93.680 69.800 0.968 85.530 0.298 0.334 0.504 0.121 97.500 81.600 85.600 0.979 70.710 0.997 0.389 57.770 0.185 0.175 0.280 39.000 6.150 0.340 4.200
3.468 33.280 2.200 0.069 0.375 0.509 35.450 93.490 69.800 0.968 85.530 0.298 0.334 0.504 0.121 97.550 83.500 84.990 0.979 70.710 0.998 0.389 54.460 0.183 0.174 0.280 39.000 6.270 0.340 5.600
3.403 28.470 46.000 1.000 0.375 0.509 67.970 93.730 70.800 0.982 87.420 1.000 0.356 0.505 0.114 97.450 83.900 84.690 0.979 82.390 0.996 0.370 74.380 0.185 0.174 0.160 48.800 5.380 0.323 3.800
Table 35 Per-task raw metric values after the fifth intervention in the full budget evaluation.
46
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
0.550 0.121 0.245 0.000 1.000 0.991 -1.000 -0.788 0.000 0.000 0.000 0.000 0.339 0.308 -0.331 -0.562 -0.375 0.122 1.000 0.000 0.995 0.000 -0.105 -0.445 0.869 0.467 -1.000 -0.803 0.000 -0.007
0.550 0.028 0.000 -0.908 1.000 0.991 -1.000 -0.364 -0.189 1.000 0.069 0.000 0.339 0.308 -0.331 -0.500 -0.375 -0.122 1.000 0.000 0.996 0.000 -0.020 -0.445 0.847 0.467 -1.000 -1.000 0.000 -0.014
0.619 0.000 0.541 1.000 1.000 0.991 -1.000 -0.667 0.377 1.000 1.000 0.999 0.440 0.308 -0.331 -1.000 -0.750 -0.527 1.000 -0.029 0.996 -0.057 1.000 0.350 0.897 0.000 -1.000 0.404 -0.006 0.021
Table 36 Per-task bounded normalized scores after the first intervention in the full budget evaluation. Scores are bounded to [−1, 1].
47
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
0.698 0.002 0.245 0.000 1.000 0.991 -1.000 -0.212 -0.189 1.000 0.069 0.247 0.339 0.308 -0.331 -1.000 -0.250 0.162 1.000 0.000 0.996 0.020 0.024 -0.445 1.000 0.467 -0.071 0.236 0.000 -0.027
0.698 0.515 0.000 -1.000 1.000 0.991 -1.000 -0.364 -0.189 1.000 0.069 1.000 0.339 0.308 -0.331 -1.000 -0.188 0.243 1.000 0.000 0.997 0.020 -0.160 -0.445 1.000 0.467 -0.071 0.229 0.000 -0.014
0.850 0.261 0.541 0.908 1.000 0.991 -1.000 0.152 -0.943 1.000 1.000 1.000 0.321 0.425 -0.381 -0.250 0.188 0.608 1.000 0.100 0.997 0.020 0.931 -0.528 -0.821 0.000 -0.036 0.517 -0.027 0.079
Table 37 Per-task bounded normalized scores after the second intervention in the full budget evaluation. Scores are bounded to [−1, 1].
48
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
0.769 -0.129 0.245 0.000 1.000 0.991 -1.000 0.000 -0.189 1.000 0.069 0.247 0.339 0.308 -0.331 -1.000 -0.563 0.406 1.000 0.000 0.993 -0.005 -0.091 -0.445 1.000 0.467 -0.893 0.620 0.000 -0.082
0.769 0.315 0.000 0.101 1.000 0.991 -0.138 -0.364 -0.189 1.000 0.069 0.247 0.339 0.308 -0.331 -1.000 -0.500 0.973 1.000 0.000 0.995 -0.005 -0.093 -0.445 1.000 0.467 -0.893 -0.012 0.000 -0.027
0.965 0.126 0.539 1.000 1.000 0.991 0.131 0.000 0.377 1.000 1.000 1.000 0.422 0.429 0.000 -0.094 0.250 0.568 1.000 -0.120 0.997 0.000 0.861 -0.216 1.000 0.133 -0.071 0.129 0.572 0.007
Table 38 Per-task bounded normalized scores after the third intervention in the full budget evaluation. Scores are bounded to [−1, 1].
49
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
0.813 -0.052 0.245 0.000 1.000 0.991 -1.000 0.000 -0.189 1.000 0.069 0.247 0.339 0.308 -0.331 0.094 -0.688 0.284 1.000 0.000 0.994 -0.984 -0.288 -0.445 1.000 0.467 -0.143 -1.000 0.000 -0.082
0.813 0.278 0.000 -0.706 1.000 0.991 -1.000 -0.364 -0.189 1.000 0.069 0.247 0.339 0.308 -0.331 -0.469 -0.688 0.771 1.000 0.000 0.997 -0.984 -0.198 -0.445 0.791 0.467 -0.143 -1.000 0.000 0.007
1.000 0.085 0.539 1.000 1.000 0.991 0.150 0.182 0.849 1.000 1.000 1.000 0.161 0.436 -0.252 -0.937 0.250 1.000 1.000 -0.146 0.998 0.000 0.848 -0.889 1.000 0.267 -0.893 0.545 0.189 0.007
Table 39 Per-task bounded normalized scores after the fourth intervention in the full budget evaluation. Scores are bounded to [−1, 1].
50
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
0.841 0.495 0.245 0.000 1.000 0.991 -1.000 0.000 -0.189 1.000 0.069 0.247 0.339 0.308 -0.331 -0.062 -1.000 0.730 1.000 0.000 0.996 -0.077 0.053 -0.445 0.916 0.467 -1.000 0.178 0.000 -0.034
0.841 -0.046 0.000 -0.908 1.000 0.991 -1.000 -0.576 -0.189 1.000 0.069 0.247 0.339 0.308 -0.331 -0.031 0.000 0.406 1.000 0.000 0.997 -0.077 -0.101 0.000 1.000 0.467 -1.000 0.098 0.000 0.014
1.000 -0.144 0.541 1.000 1.000 0.991 0.166 0.152 0.755 1.000 1.000 1.000 0.283 0.301 0.000 -0.094 0.250 0.243 1.000 -0.174 0.995 0.007 0.828 -0.461 1.000 0.267 0.000 0.667 0.646 -0.048
Table 40 Per-task bounded normalized scores after the fifth intervention in the full budget evaluation. Scores are bounded to [−1, 1].
51
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
96.5 49.2 26.5 52.6 100.0 99.9 39.2 99.4 98.5 99.8 99.7 6.7 257.5 87.5 77.0 97.5 97.4 98.1 100.9 5.0 99.6 39.7 69.6 97.5 99.8 46.7 77.9 63.1 92.4 14.5
96.5 43.8 2.6 9.6 100.0 99.9 39.2 99.5 98.2 104.1 99.8 6.7 257.5 87.5 77.0 97.6 97.4 97.6 100.9 5.0 99.6 39.7 72.0 97.5 99.8 46.7 77.9 54.5 92.4 14.0
97.0 42.2 55.4 138.7 100.0 99.9 39.2 99.4 99.1 105.6 102.0 99.9 292.0 87.5 77.0 96.8 96.7 96.7 100.9 4.9 99.7 38.4 100.1 98.9 99.8 0.0 77.9 83.8 92.3 16.9
Table 41 Per-task anchor attainment after the first intervention in the full budget evaluation, reported as percentages.
52
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
97.6 42.3 26.5 52.6 100.0 99.9 39.2 99.6 98.2 104.1 99.8 29.8 257.5 87.5 77.0 96.1 97.6 98.2 100.9 5.0 99.6 40.2 73.2 97.5 100.4 46.7 94.2 80.1 92.4 12.8
97.6 71.9 2.6 0.0 100.0 99.9 39.2 99.5 98.2 104.1 99.8 100.0 257.5 87.5 77.0 96.1 97.8 98.4 100.9 5.0 99.8 40.2 68.1 97.5 100.1 46.7 94.2 80.0 92.4 14.0
98.8 57.3 55.4 95.7 100.0 99.9 39.2 99.7 97.1 104.3 102.2 100.0 251.9 89.4 76.3 98.0 98.5 99.1 101.1 5.5 99.7 40.2 98.1 97.3 97.2 0.0 94.4 86.4 92.2 21.8
Table 42 Per-task anchor attainment after the second intervention in the full budget evaluation, reported as percentages.
53
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
98.2 34.7 26.5 52.6 100.0 99.9 39.2 99.6 98.2 104.1 99.8 29.8 257.5 87.5 77.0 96.3 97.1 98.7 100.9 5.0 99.4 39.6 70.0 97.5 100.2 46.7 89.7 89.0 92.4 8.1
98.2 60.4 2.6 57.4 100.0 99.9 66.1 99.5 98.2 104.1 99.8 29.8 257.5 87.5 77.0 96.3 97.2 99.9 100.9 5.0 99.6 39.6 70.0 97.5 100.1 46.7 89.7 75.2 92.4 12.8
99.7 49.5 55.1 138.7 100.0 99.9 74.1 99.6 99.1 105.6 102.0 100.0 285.0 89.4 81.7 98.2 98.6 99.1 100.9 4.5 99.7 39.7 96.2 97.9 100.1 13.3 94.2 77.9 96.6 15.7
Table 43 Per-task anchor attainment after the third intervention in the full budget evaluation, reported as percentages.
54
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
98.5 39.1 26.5 52.6 100.0 99.9 39.2 99.6 98.2 104.1 99.8 29.8 257.5 87.5 77.0 98.5 96.8 98.4 100.9 5.0 99.5 25.0 64.6 97.5 100.4 46.7 93.8 28.0 92.4 8.1
98.5 58.3 2.6 19.1 100.0 99.9 39.2 99.5 98.2 104.1 99.8 29.8 257.5 87.5 77.0 97.6 96.8 99.5 100.9 5.0 99.7 25.0 67.1 97.5 99.7 46.7 93.8 21.6 92.4 15.7
100.3 47.1 55.1 138.7 100.0 99.9 74.7 99.7 99.8 103.8 101.7 100.0 212.7 89.5 78.0 96.9 98.6 100.6 100.9 4.4 99.8 39.7 95.8 96.7 100.2 26.7 89.7 87.1 93.7 15.7
Table 44 Per-task anchor attainment after the fourth intervention in the full budget evaluation, reported as percentages.
55
Task
Q32
DS-R1
Claude
FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
98.7 70.8 26.5 52.6 100.0 99.9 39.2 99.6 98.2 104.1 99.8 29.8 257.5 87.5 77.0 98.3 95.9 99.4 100.9 5.0 99.7 38.0 74.0 97.5 99.9 46.7 75.6 78.9 92.4 12.2
98.7 39.5 2.6 9.6 100.0 99.9 39.2 99.4 98.2 104.1 99.8 29.8 257.5 87.5 77.0 98.3 98.1 98.7 100.9 5.0 99.8 38.0 69.8 98.2 100.3 46.7 75.6 77.3 92.4 16.3
100.6 33.8 55.4 138.7 100.0 99.9 75.2 99.7 99.6 105.6 102.0 100.0 241.6 87.4 81.7 98.2 98.6 98.4 100.9 4.3 99.6 39.9 95.3 97.5 100.4 26.7 94.6 90.2 97.2 11.0
Table 45 Per-task anchor attainment after the fifth intervention in the full budget evaluation, reported as percentages.
56
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
5.931 13.950 46.000 0.862 0.375 0.509 35.450 88.570 68.600 0.956 87.450 0.021 0.465 0.530 0.127 97.150 82.200 84.760 0.979 75.140 0.690 0.385 12.550 0.182 0.177 0.000 40.200 259.580 0.337 5.800
5.771 32.050 45.000 1.000 0.375 0.509 35.450 88.930 70.100 0.982 87.440 -0.021 0.465 0.530 0.127 97.900 82.800 83.620 0.979 72.860 0.728 0.385 12.530 0.184 0.176 0.000 40.200 394.830 0.335 5.800
5.931 26.250 45.000 1.000 0.375 0.509 35.450 89.010 69.200 0.928 86.550 -0.021 0.530 0.530 0.127 97.050 82.200 85.220 0.979 72.860 0.686 0.380 27.070 0.184 0.177 0.000 48.700 733.660 0.339 5.800
5.859 19.970 45.000 1.000 0.375 0.509 35.450 89.010 69.200 0.928 87.240 -0.021 0.697 0.530 0.127 97.400 83.800 83.700 0.979 72.860 0.746 0.368 13.240 0.184 0.177 0.000 40.200 1376.850 0.339 5.800
Table 46 Per-task raw metric values after the first intervention in the harness ablation evaluation. Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
5.414 36.460 46.000 0.690 0.375 0.509 35.450 89.270 70.000 0.982 87.480 0.021 0.465 0.513 0.144 97.550 83.800 85.520 0.981 66.250 0.997 0.372 22.010 0.184 0.176 0.000 48.600 21.820 0.337 5.200
5.284 13.800 45.000 0.690 0.375 0.509 42.590 89.250 70.400 0.970 87.300 0.001 0.465 0.513 0.144 94.350 83.100 84.230 0.979 72.680 0.995 0.368 20.730 0.184 0.177 0.000 48.700 15.810 0.335 5.400
5.414 27.540 45.000 1.000 0.375 0.509 37.830 89.220 70.000 0.928 86.470 0.011 0.530 0.499 0.127 97.650 83.800 84.760 0.979 70.710 0.999 0.368 21.790 0.183 0.177 0.000 41.300 70.940 0.339 5.600
5.224 18.820 45.000 1.000 0.375 0.509 37.620 89.090 70.000 0.928 87.240 0.011 0.321 0.511 0.127 97.050 84.100 83.780 0.981 75.920 0.998 0.372 28.590 0.183 0.177 0.360 48.600 27.780 0.339 5.200
Table 47 Per-task raw metric values after the second intervention in the harness ablation evaluation.
57
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
4.983 47.950 46.000 0.897 0.375 0.509 36.940 89.090 70.600 0.970 87.640 0.021 0.333 0.496 0.127 97.300 85.200 85.820 0.964 70.710 0.994 0.593 22.450 0.183 0.177 0.000 48.800 19.940 0.335 6.000
4.761 42.400 47.200 0.966 0.375 0.509 44.330 88.790 69.000 0.982 87.440 0.021 0.348 0.497 0.127 97.250 83.800 84.000 0.979 78.770 0.997 0.368 22.500 0.186 0.176 0.360 48.600 15.440 0.337 5.200
4.983 21.540 45.000 1.000 0.375 0.509 35.450 89.220 69.300 0.928 86.550 0.011 0.530 0.496 0.127 97.200 84.400 84.910 0.979 71.190 0.999 0.365 29.180 0.183 0.177 0.000 48.800 67.190 0.388 4.600
4.761 22.880 45.000 1.000 0.375 0.509 35.450 89.220 70.800 0.928 87.480 0.009 0.479 0.507 0.127 97.600 82.200 82.870 0.979 66.250 0.995 0.365 29.780 0.183 0.177 0.120 48.700 16.500 0.339 5.600
Table 48 Per-task raw metric values after the third intervention in the harness ablation evaluation. Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
4.513 20.740 46.000 0.897 0.375 0.509 39.920 89.250 70.800 0.979 87.240 -0.035 0.333 0.497 0.127 97.450 85.200 84.840 0.979 73.650 0.999 0.382 22.340 0.186 0.176 0.080 48.700 16.280 0.340 5.600
4.347 24.420 47.200 1.000 0.375 0.509 46.500 89.220 70.400 0.970 87.350 0.021 0.333 0.495 0.127 97.850 82.900 83.850 0.971 80.490 1.000 0.504 30.270 0.183 0.177 0.160 46.300 45.040 0.326 7.300
4.513 16.210 45.000 1.000 0.375 0.509 36.580 89.270 70.000 0.928 86.380 0.011 0.530 0.497 0.127 97.600 84.400 84.840 0.979 70.710 0.998 0.604 29.230 0.183 0.177 0.000 48.800 17.760 0.339 5.600
4.340 27.060 47.200 1.000 0.375 0.509 37.750 89.220 70.800 0.925 87.480 0.011 0.321 0.496 0.127 97.350 84.100 83.320 0.979 64.000 0.999 0.364 27.710 0.183 0.177 0.560 48.800 74.350 0.339 6.700
Table 49 Per-task raw metric values after the fourth intervention in the harness ablation evaluation.
58
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
4.406 37.620 46.000 0.862 0.375 0.509 38.070 89.250 70.800 0.970 87.420 0.021 0.333 0.517 0.127 97.350 85.200 85.440 0.979 75.140 0.997 0.370 30.450 0.185 0.176 0.080 48.800 16.540 0.326 4.600
4.199 21.930 47.200 1.000 0.375 0.509 44.180 89.220 70.900 0.982 87.470 0.021 0.297 0.500 0.127 97.250 83.800 83.620 0.979 82.060 0.999 0.365 33.060 0.183 0.182 0.240 48.800 16.690 0.319 6.700
4.406 29.590 45.000 1.000 0.375 0.509 38.130 89.270 70.000 0.928 86.550 0.011 0.530 0.496 0.127 97.400 84.400 85.370 0.979 70.710 0.998 0.365 29.140 0.183 0.177 0.280 48.800 47.040 0.339 3.800
4.194 23.950 47.200 1.000 0.375 0.509 38.270 89.090 70.800 0.925 87.530 0.011 0.321 0.497 0.127 97.750 84.100 83.930 0.964 67.260 0.997 0.454 25.760 0.183 0.177 0.000 48.800 21.900 0.339 4.600
Table 50 Per-task raw metric values after the fifth intervention in the harness ablation evaluation.
59
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
-1.000 -0.272 0.545 1.000 1.000 0.991 -0.068 -0.106 -1.000 1.000 1.000 0.011 1.000 0.066 0.151 -0.171 -0.813 0.054 1.000 -0.066 0.627 -0.057 -0.062 0.350 0.248 0.000 -1.000 -1.000 0.151 -0.538
-1.000 0.056 0.533 1.000 1.000 0.991 -0.068 -0.033 0.094 1.000 1.000 -0.033 1.000 0.066 0.151 0.257 -0.437 -0.750 1.000 -0.032 0.673 -0.057 -0.063 -0.216 0.530 0.000 -1.000 -1.000 0.215 -0.538
-1.000 -0.049 0.533 1.000 1.000 0.990 -0.068 -0.016 -0.755 0.000 1.000 -0.033 1.000 0.066 0.151 -0.229 -0.813 0.375 1.000 -0.032 0.623 -0.035 0.173 -0.278 0.370 0.000 -0.036 -1.000 0.056 -0.538
-1.000 -0.163 0.533 1.000 1.000 0.991 -0.068 -0.016 -0.755 0.000 1.000 -0.033 1.000 0.066 0.151 -0.029 0.188 -0.697 1.000 -0.032 0.694 0.020 -0.051 -0.278 0.369 0.000 -1.000 -1.000 0.056 -0.538
Table 51 Per-task bounded normalized scores after the first intervention in the harness ablation evaluation. Scores are bounded to [−1, 1].
60
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
-1.000 0.136 0.545 0.898 1.000 0.991 -0.068 0.037 0.000 1.000 1.000 0.011 1.000 0.247 -0.300 0.057 0.188 0.590 1.000 0.066 0.997 0.000 0.091 -0.216 0.547 0.000 -0.071 0.429 0.151 -0.570
-1.000 -0.275 0.533 0.898 1.000 0.991 0.071 0.033 0.377 1.000 1.000 -0.010 1.000 0.247 -0.300 -1.000 -0.250 -0.322 1.000 -0.029 0.994 0.020 0.070 -0.278 0.245 0.000 -0.036 0.631 0.215 -0.559
-1.000 -0.026 0.533 1.000 1.000 0.990 -0.022 0.026 0.000 0.000 1.000 0.000 1.000 0.389 0.151 0.114 0.188 0.054 1.000 0.000 0.999 0.020 0.088 0.023 0.166 0.000 -1.000 -1.000 0.056 -0.548
-0.957 -0.184 0.533 1.000 1.000 0.991 -0.026 0.000 0.000 0.000 1.000 0.000 1.000 0.267 0.151 -0.229 0.375 -0.643 1.000 -0.077 0.997 0.001 0.198 0.023 0.370 0.600 -0.071 0.228 0.056 -0.570
Table 52 Per-task bounded normalized scores after the second intervention in the harness ablation evaluation. Scores are bounded to [−1, 1].
61
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
-0.694 0.344 0.545 1.000 1.000 0.991 -0.039 0.000 0.566 1.000 1.000 0.011 1.000 0.420 0.151 -0.086 1.000 0.804 0.917 0.000 0.992 -0.984 0.098 0.105 0.210 0.000 0.000 0.492 0.215 -0.527
-0.453 0.243 0.560 1.000 1.000 0.991 0.104 -0.061 -0.943 1.000 1.000 0.011 1.000 0.417 0.151 -0.114 0.188 -0.482 1.000 -0.120 0.996 0.020 0.099 -0.998 0.530 0.600 -0.071 0.643 0.151 -0.570
-0.694 -0.135 0.533 1.000 1.000 0.990 -0.068 0.026 -0.660 0.000 1.000 0.000 1.000 0.420 0.151 -0.143 0.563 0.161 1.000 -0.007 0.998 0.032 0.207 0.023 0.344 0.000 0.000 -1.000 -1.000 -0.602
-0.453 -0.110 0.533 1.000 1.000 0.991 -0.068 0.026 0.755 0.000 1.000 -0.002 1.000 0.310 0.151 0.086 -0.813 -1.000 1.000 0.066 0.994 0.032 0.217 0.023 0.168 0.200 -0.036 0.608 0.056 -0.548
Table 53 Per-task bounded normalized scores after the third intervention in the harness ablation evaluation. Scores are bounded to [−1, 1].
62
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
-0.183 -0.149 0.545 1.000 1.000 0.991 0.019 0.033 0.755 1.000 1.000 -0.046 1.000 0.417 0.151 0.000 1.000 0.107 1.000 -0.044 0.998 -0.046 0.096 -0.889 0.585 0.133 -0.036 0.615 0.021 -0.548
-0.002 -0.082 0.560 1.000 1.000 0.991 0.147 0.026 0.377 1.000 1.000 0.011 1.000 0.440 0.151 0.229 -0.375 -0.590 1.000 -0.146 1.000 -0.588 0.225 0.153 0.254 0.267 -0.893 -0.354 0.547 -0.457
-0.183 -0.231 0.533 1.000 1.000 0.990 -0.046 0.037 0.000 0.000 1.000 0.000 1.000 0.417 0.151 0.086 0.563 0.107 1.000 0.000 0.998 -1.000 0.208 0.023 0.370 0.000 0.000 0.565 0.056 -0.548
0.004 -0.035 0.560 1.000 1.000 0.991 -0.024 0.026 0.755 -1.000 1.000 0.000 1.000 0.420 0.151 -0.057 0.375 -0.965 1.000 0.100 0.998 0.037 0.184 0.023 0.369 0.933 0.000 -1.000 0.056 -0.489
Table 54 Per-task bounded normalized scores after the fourth intervention in the harness ablation evaluation. Scores are bounded to [−1, 1].
63
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
-0.067 0.157 0.545 1.000 1.000 0.991 -0.017 0.033 0.755 1.000 1.000 0.011 1.000 0.204 0.151 -0.057 1.000 0.536 1.000 -0.066 0.996 0.007 0.228 -0.461 0.585 0.133 0.000 0.606 0.547 -0.602
0.158 -0.128 0.560 1.000 1.000 0.991 0.101 0.026 0.849 1.000 1.000 0.011 1.000 0.385 0.151 -0.114 0.188 -0.750 1.000 -0.169 0.999 0.032 0.270 0.087 -1.000 0.400 0.000 0.601 0.805 -0.489
-0.067 0.011 0.533 1.000 1.000 0.990 -0.016 0.037 0.000 0.000 1.000 0.000 1.000 0.420 0.151 -0.029 0.563 0.482 1.000 0.000 0.997 0.032 0.207 0.023 0.370 0.467 0.000 -0.421 0.056 -0.645
0.163 -0.091 0.560 1.000 1.000 0.991 -0.013 0.000 0.755 -1.000 1.000 0.000 1.000 0.417 0.151 0.171 0.375 -0.536 0.917 0.051 0.997 -0.364 0.152 0.023 0.370 0.000 0.000 0.426 0.056 -0.602
Table 55 Per-task bounded normalized scores after the fifth intervention in the harness ablation evaluation. Scores are bounded to [−1, 1].
64
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
57.7 16.6 55.4 119.6 100.0 99.9 39.2 94.2 96.5 102.8 102.0 2.1 184.8 83.3 73.5 97.9 96.6 98.4 100.9 4.7 69.0 38.4 16.1 98.9 98.7 0.0 77.9 1.9 93.2 16.9
59.4 38.1 54.2 138.7 100.0 99.9 39.2 94.6 98.6 105.6 102.0 -2.1 184.9 83.3 73.5 98.7 97.3 97.1 100.9 4.9 72.8 38.4 16.0 97.9 99.2 0.0 77.9 1.2 93.7 16.9
57.7 31.2 54.2 138.7 100.0 99.9 39.2 94.7 97.4 99.8 100.9 -2.1 162.3 83.3 73.5 97.8 96.6 99.0 100.9 4.9 68.7 38.9 34.7 97.8 98.9 0.0 94.4 0.7 92.5 16.9
58.5 23.7 54.2 138.7 100.0 99.9 39.2 94.7 97.4 99.8 101.7 -2.1 123.5 83.3 73.5 98.2 98.5 97.2 100.9 4.9 74.6 40.2 17.0 97.8 98.9 0.0 77.9 0.4 92.5 16.9
Table 56 Per-task anchor attainment after the first intervention in the harness ablation evaluation, reported as percentages.
65
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
63.3 43.3 55.4 95.7 100.0 99.9 39.2 95.0 98.5 105.6 102.0 2.1 184.8 86.1 64.4 98.3 98.5 99.3 101.1 5.4 99.7 39.7 28.2 97.9 99.2 0.0 94.2 22.2 93.2 15.1
64.8 16.4 54.2 95.7 100.0 99.9 47.1 94.9 99.1 104.3 101.8 0.1 184.9 86.1 64.4 95.1 97.6 97.8 100.9 4.9 99.5 40.2 26.6 97.8 98.7 0.0 94.4 30.7 93.7 15.7
63.3 32.7 54.2 138.7 100.0 99.9 41.8 94.9 98.5 99.8 100.9 1.1 162.3 88.4 73.5 98.4 98.5 98.4 100.9 5.0 99.9 40.2 27.9 98.3 98.6 0.0 80.0 6.8 92.5 16.3
65.6 22.4 54.2 138.7 100.0 99.9 41.6 94.8 98.5 99.8 101.7 1.1 267.7 86.4 73.5 97.8 98.8 97.3 101.1 4.7 99.8 39.8 36.6 98.3 98.9 60.0 94.2 17.5 92.5 15.1
Table 57 Per-task anchor attainment after the second intervention in the harness ablation evaluation, reported as percentages.
66
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
68.7 57.0 55.4 124.3 100.0 99.9 40.9 94.8 99.4 104.3 102.2 2.1 258.1 88.9 73.5 98.1 100.1 99.7 99.4 5.0 99.4 25.0 28.8 98.4 98.7 0.0 94.6 24.3 93.7 17.4
71.9 50.4 56.8 133.9 100.0 99.9 49.0 94.4 97.1 105.6 102.0 2.1 247.0 88.9 73.5 98.0 98.5 97.6 100.9 4.5 99.6 40.2 28.8 96.5 99.2 60.0 94.2 31.4 93.2 15.1
68.7 25.6 54.2 138.7 100.0 99.9 39.2 94.9 97.5 99.8 100.9 1.1 162.3 88.9 73.5 98.0 99.2 98.6 100.9 5.0 99.9 40.5 37.4 98.3 98.9 0.0 94.6 7.2 81.0 13.4
71.9 27.2 54.2 138.7 100.0 99.9 39.2 94.9 99.6 99.8 102.0 0.9 179.6 87.1 73.5 98.4 96.6 96.2 100.9 5.4 99.5 40.5 38.1 98.3 98.6 20.0 94.4 29.4 92.5 16.3
Table 58 Per-task anchor attainment after the third intervention in the harness ablation evaluation, reported as percentages.
67
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
75.9 24.6 55.4 124.3 100.0 99.9 44.2 94.9 99.6 105.3 101.7 -3.5 258.1 88.9 73.5 98.2 100.1 98.5 100.9 4.8 99.8 38.7 28.6 96.7 99.3 13.3 94.4 29.8 92.3 16.3
78.8 29.0 56.8 138.7 100.0 99.9 51.4 94.9 99.1 104.3 101.9 2.1 257.9 89.2 73.5 98.6 97.4 97.4 100.1 4.4 100.0 29.4 38.8 98.5 98.8 26.7 89.7 10.8 96.3 21.2
75.9 19.3 54.2 138.7 100.0 99.9 40.5 95.0 98.5 99.8 100.7 1.1 162.3 88.9 73.5 98.4 99.2 98.5 100.9 5.0 99.8 24.5 37.4 98.3 98.9 0.0 94.6 27.3 92.5 16.3
78.9 32.1 56.8 138.7 100.0 99.9 41.8 94.9 99.6 99.5 102.0 1.1 267.7 88.9 73.5 98.1 98.8 96.8 100.9 5.5 99.9 40.7 35.5 98.3 98.9 93.3 94.6 6.5 92.5 19.5
Table 59 Per-task anchor attainment after the fourth intervention in the harness ablation evaluation, reported as percentages.
68
Task FineWeb Pretraining ConvKAN CIFAR-10 Open-R1 MATH-500 Room Selector Tuning Uni2TS ETTh1 Forecasting TimesFM Long Horizon ModernBERT MNLI AirBench CIFAR-10 NoisyGL Cora GCN TabMini Promoters Adult Tabular Diffusion xLSTM Parity VBLL Yacht Regression TabM California Housing TimeXer PJM Forecasting ABLkit HWF Reasoning tunedGNN Cora GCN verl GRPO GSM8K ForestDiffusion Iris VAR ImageNet 256 RAGEN Bandit Alignment Chronos Weather Forecasting HyperbolicCV CIFAR-100 iTransformer ETTm2 Forecasting TimeMixer ETTm2 Forecasting ART 2048 Multimodal Open-R1 MathVista RankUp UTKFace Regression SparseTSF ETTm1 Forecasting SimpleRL MATH-500
AH-Claude
CC-CLI
AH-GPT
Codex-CLI
77.7 44.7 55.4 119.6 100.0 99.9 42.1 94.9 99.6 104.3 102.0 2.1 258.1 85.4 73.5 98.1 100.1 99.2 100.9 4.7 99.6 39.9 39.0 97.5 99.3 13.3 94.6 29.3 96.3 13.4
81.6 26.1 56.8 138.7 100.0 99.9 48.9 94.9 99.8 105.6 102.0 2.1 289.2 88.3 73.5 98.0 98.5 97.1 100.9 4.3 99.9 40.5 42.3 98.4 96.2 40.0 94.6 29.1 98.4 19.5
77.7 35.2 54.2 138.7 100.0 99.9 42.2 95.0 98.5 99.8 100.9 1.1 162.3 88.9 73.5 98.2 99.2 99.1 100.9 5.0 99.8 40.5 37.3 98.3 98.9 46.7 94.6 10.3 92.5 11.0
81.7 28.5 56.8 138.7 100.0 99.9 42.3 94.8 99.6 99.5 102.1 1.1 267.7 88.9 73.5 98.5 98.8 97.5 99.4 5.3 99.7 32.6 33.0 98.3 98.9 0.0 94.6 22.1 92.5 13.4
Table 60 Per-task anchor attainment after the fifth intervention in the harness ablation evaluation, reported as percentages.
69