ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization

Preprint

ACTIVE S ADDLER : AUTOMATED C URRICULUM L EARN ING FOR AGENT H ARNESS O PTIMIZATION Sungho Park1∗ Wonjoong Kim2∗ Jue Zhang3† Wook-Shin Han1† Pengfei Gao3 Chanyoung Park2 Yongqiang Yao3 Rao Fu3 Elsie Nallipogu3 Qingwei Lin3 Victor Rühle3 POSTECH 2 KAIST 3 Microsoft juezhang@microsoft.com wshan@dblab.postech.ac.kr

A BSTRACT Automated harness optimization can substantially improve LLM agents by iteratively updating their prompts, tool interfaces, and control logic from execution feedback. However, existing methods primarily optimize how the harness is updated while largely fixing which training scenarios generate the feedback that drives those updates. As the harness evolves, the scenarios most useful for further optimization can change, suggesting that the training curriculum itself should adapt alongside the harness. We formulate this missing dimension of harness optimization as an automated curriculum learning problem and introduce ActiveSaddler. ActiveSaddler models the evolving curriculum as a non-stationary bandit with dynamically instantiated optimization targets. It abstracts recurring failures into reusable failure-pattern arms, estimates the potential learning progress from further targeting each pattern, and adaptively balances revisiting known weaknesses with exploring unseen scenarios for new ones. Optimization outcomes continually update both the set of discovered failure patterns and their priorities, allowing the curriculum to co-evolve with the harness. Experiments on GAIA2 and TerminalBench 2.0 show that ActiveSaddler consistently discovers stronger harnesses, improving test Pass@1 by 4.4 and 7.5 percentage points over the same harness optimizer using a scenario order fixed before optimization, respectively. Ablations further show that these gains depend on dynamically constructing optimization targets, estimating their evolving utility, and balancing continued optimization with new failure discovery. Together, these results establish automated curriculum learning as a new crucial optimization dimension for harness optimization. Project website and code will be available at https://aka.ms/ActiveSaddler-website.

70

ActiveSaddler GEPA Meta-Harness AutoSaddler AutoSaddler w/ Category Acc. Order AutoSaddler w/ Scenario Acc. Order

65

100

Best Dev Set Accuracy (%)

Best Dev Set Accuracy (%)

arXiv:2610.00906v1 [cs.AI] 1 Oct 2026

1

60 55 50

0

200

400

600

800

1000

Total Scenarios Executed

1200

90 85 80 75 70 65 60

1400

(a) GAIA2.

ActiveSaddler GEPA Meta-Harness AutoSaddler AutoSaddler w/ Category Acc. Order AutoSaddler w/ Scenario Acc. Order

95

0

100

200

300

Total Scenarios Executed

400

(b) Terminal-Bench 2.0.

Figure 1: Optimization efficiency across benchmarks. Best-so-far development accuracy versus cumulative task-agent rollouts for ActiveSaddler and baseline methods.

1

I NTRODUCTION

Large Language Models (LLMs) have become increasingly capable of powering autonomous agents, yet reliable performance on multi-step and long-horizon tasks still depends heavily on the external ∗ †

Work done during an internship at Microsoft. Corresponding authors.

1

Preprint

harness surrounding the model. A harness specifies how the agent is prompted, which tools it can use, and how its execution is controlled, monitored, and corrected. While well-designed harnesses can substantially improve agent robustness (Zhou et al., 2026; Anthropic, 2025; OpenAI, 2026; LangChain, 2026), their effectiveness can degrade as models, task distributions, or deployment environments change. This has motivated automated harness optimization methods that execute training scenarios, diagnose deficiencies from the resulting execution traces, and patch prompts, tool interfaces, or runtime control logic (Lee et al., 2026; Park et al., 2026). These methods thus focus primarily on how to update the harness, whereas the training scenarios used to generate feedback are usually predetermined, either as the full training set or as mini-batches scheduled before optimization begins. As the harness evolves, however, the usefulness of these training scenarios can change, creating two limitations. First, unresolved failures may benefit from further optimization, whereas repaired ones may require little additional attention. A fixed schedule cannot respond to this shift and may therefore move on from unresolved failures too early or spend rollouts on scenarios with little remaining value. Indeed, under fixed-order optimization, many failures observed during training remain unresolved by the final optimized harness (Figure 3b). Second, the relative value of revisiting known failures versus evaluating unseen scenarios also changes over optimization. Revisiting is more valuable when actionable unresolved failures remain, whereas exploring unseen scenarios becomes more valuable when few worthwhile targets remain among the observed failures. A fixed schedule cannot adapt this balance. These limitations are particularly costly under a limited rollout budget, where optimization opportunities must be allocated selectively. This perspective is consistent with prior work on curriculum and environment design, which shows that learning can benefit from adapting what data, tasks, or environments are presented to the learner (Clune, 2019; Wang et al., 2019; Graves et al., 2017; Matiisen et al., 2017). This suggests a complementary dimension for harness optimization: as the harness evolves, the curriculum can also adapt which training scenarios are used to drive subsequent updates. In this work, we formulate scenario selection for budgeted offline harness optimization as an automated curriculum-learning problem and introduce ActiveSaddler, which proactively adapts the optimization curriculum as the harness evolves. We model curriculum selection as a non-stationary bandit whose arms are instantiated online from observed failures: each failure-pattern arm represents a potentially repairable harness deficiency shared across failed trajectories. At each iteration, ActiveSaddler either evaluates unseen scenarios to discover new failure patterns or revisits an existing arm, prioritizing arms by their estimated potential for further improvement. The resulting optimization outcomes update both the priorities of existing arms and the set of discovered patterns. Through this feedback loop, ActiveSaddler co-evolves the curriculum with the harness to allocate a fixed rollout budget toward more valuable optimization opportunities. We evaluate ActiveSaddler on GAIA2 (Froger et al., 2026) and Terminal-Bench 2.0 (Merrill et al., 2026) against existing harness optimizers and fixed curriculum baselines. Across both benchmarks, ActiveSaddler achieves the strongest final harness performance, improving over the same harness optimizer with a scenario order fixed before optimization by 4.4 and 7.5 percentage points, respectively. Our ablation analyses reveal three design principles for automated curriculum learning in harness optimization: i) failure-pattern arms provide an effective optimization abstraction: category-level arms can conflate distinct failures, whereas scenario-level arms fragment recurring failures across individual scenarios; ii) adaptive arm prioritization is crucial because the value of revisiting a failure changes as the harness evolves; iii) adaptive exploration is necessary because the relative value of revisiting observed failures versus evaluating unseen scenarios changes over optimization. Our main contributions are summarized as follows: • We formulate scenario selection for offline harness optimization as an automated curriculumlearning problem, where training scenarios are selected adaptively as the harness evolves. • We introduce ActiveSaddler, which combines failure-pattern arms, adaptive arm prioritization, and adaptive exploration to construct this evolving curriculum. • We show consistent gains on GAIA2 and Terminal-Bench 2.0 over existing harness optimizers and fixed curricula under the same rollout budget. 2

Preprint

2

R ELATED W ORK

Automatic Harness Optimization. Recent work extends automatic optimization beyond prompts to the broader agent harness, including tools, memory, skills, and runtime control logic (Zhou et al., 2026). Despite implementation differences, these methods generally execute the current harness, use rollout feedback to propose updates, and evaluate candidate harnesses. Broadly, automatic harness optimization (AHO) spans search-based and learning-based approaches. Search-based methods explore alternative harness implementations or configurations through evolutionary or Bayesian search (Lee et al., 2026; Lin et al., 2026; Zhang et al., 2026; Sengupta & Wang, 2026). Learningbased methods instead treat external harness state as the object being learned, repeatedly updating it from rollout feedback on training mini-batches, analogous to model training but without updating model weights (Park et al., 2026; Yang et al., 2026b). GEPA (Agrawal et al., 2026) lies between these styles, coupling reflective updates from training mini-batches with evolutionary Pareto search over validation performance. Recent work further extends AHO by changing the optimization signal, such as recalibrating historical experience or replacing labeled validation with self-preference (Guo et al., 2026; Pan et al., 2026), and by jointly optimizing harnesses and model weights (Hebbar et al., 2026; Chen et al., 2026; Kim et al., 2026). Orthogonally, offline approaches optimize the harness before deployment, often using training data and held-out validation (Park et al., 2026; Yang et al., 2026b), whereas online or test-time approaches continue adapting during sequential interaction or evaluation (Karten et al., 2026; Wei et al., 2026; Nie et al., 2026). We focus on offline, learningbased AHO under a standard train/dev/test protocol, where ActiveSaddler leaves the harness-update mechanism unchanged and instead adapts which training scenarios provide evidence for each update. Automated Curriculum Learning. Curriculum learning studies how training data, tasks, or environments are scheduled to improve learning (Bengio et al., 2009). Early work prioritized learning opportunities according to competence or learning progress (Oudeyer et al., 2007; Lopes & Oudeyer, 2012), while subsequent automated curricula formalized this choice through non-stationary bandits, teacher-student task selection, and progress-driven environment sampling (Graves et al., 2017; Matiisen et al., 2017; Portelas et al., 2019). These methods estimate task utility from observable learner-side signals, such as loss reduction, competence progress, or TD error (Graves et al., 2017; Matiisen et al., 2017; Jiang et al., 2021). ActiveSaddler adopts the same principle that training opportunities should be prioritized according to their evolving value to the learner. Harness optimization, however, makes both the curriculum target and its progress implicit. A training scenario does not specify a fixed harness weakness: the weakness becomes apparent only after execution, the same scenario may expose different failures as the harness changes, and the same weakness may recur across different scenarios. Moreover, a diagnosed weakness has no direct loss or other target-level learning signal indicating whether it remains unresolved or worth another intervention; this must instead be inferred from the current harness, prior optimization attempts, and execution history. ActiveSaddler therefore constructs failure-pattern arms from diagnosed failures, continually re-estimates their optimization value, and allocates rollouts between revisiting known weaknesses and exploring unseen scenarios for new ones. Adaptive Task Selection for Agent Optimization. Recent work has begun to adapt task allocation at different stages of agent optimization. For model-weight optimization, CoEvolve (Yang et al., 2026a) uses forgetting and uncertainty observed in agent rollouts to generate new training tasks targeting behaviors the current agent still struggles with. In harness optimization, Task-CoEvolve (Miyai et al., 2026) adaptively selects validation tasks that best distinguish candidate harnesses, while HarnessLens (Xu et al., 2026) selects verification tasks relevant to each proposed harness modification. These methods adapt task selection on the evaluation side of harness optimization, after candidate modifications have been proposed. In contrast, ActiveSaddler adapts the training scenarios that generate evidence for harness updates throughout optimization, repeatedly deciding which known weaknesses to revisit and when to execute unseen scenarios to discover new ones.

3

P RELIMINARIES

Harness Optimization. Let Hθ denote an agent harness parameterized by θ ∈ Θ, where θ may include prompts, tool interfaces, and runtime control logic. For a task scenario (x, y ∗ ) drawn from a task distribution T , where x denotes the task input and y ∗ its reference answer, executing the agent 3

Preprint

with Hθ produces an execution trace τ and a final answer ŷ according to (τ, ŷ) ∼ Pθ (· | x), where Pθ denotes the stochastic execution distribution induced by the model and harness. Let µ(ŷ, y ∗ ) be a task-level evaluation metric. The population performance of a harness is   J(θ) = E(x,y∗ )∼T E(τ,ŷ)∼Pθ (·|x) µ(ŷ, y ∗ ) . (1) For a finite dataset D, we define the empirical harness performance as X 1 JbD (θ) = µ(ŷi , yi∗ ), (τi , ŷi ) ∼ Pθ (· | xi ). |D| ∗

(2)

(xi ,yi )∈D

Following the learning-based offline harness-optimization setting of AutoSaddler (Park et al., 2026), we assume fixed and mutually disjoint training, development, and test sets, Dtrain , Ddev , and Dtest . At iteration t, let θt denote the current harness parameters and write Ht := Hθt . Given a selected set of training scenarios Xt ⊆ Dtrain , executing Ht produces a collection of execution records Et = {(x, y ∗ , τ, ŷ) | (x, y ∗ ) ∈ Xt , (τ, ŷ) ∼ Pθt (· | x)} .

(3)

A harness optimizer O uses these records to diagnose deficiencies in the current harness, construct and evaluate candidate updates, and return the harness retained for the next iteration:
θt+1 ∼ O θt , Et , Ddev , (4) where Ddev supports candidate selection, regression checks, and generalization checks during optimization, while Dtest is reserved for final evaluation. Thus, Xt determines which evidence is collected, while O determines how it is used to update the harness. Automated Curriculum Learning for Harness Optimization. Automated curriculum learning adapts the selection of training data to the evolving state of a learner (Graves et al., 2017). In harness optimization, we regard the current harness as the evolving learner and formulate the selection of Xt as a curriculum-learning problem. Let I<t denote the information available before the curriculum decision at iteration t, including the current harness and the execution and update history accumulated over previous iterations. A curriculum policy π selects the next training-scenario set according to Xt ∼ π(· | I<t ),

Xt ⊆ Dtrain .

(5)

The selected scenarios are executed under Ht , and the resulting harness-optimization step provides feedback for subsequent curriculum decisions. We hold the optimizer O and its candidate-evaluation procedure fixed and study how the curriculum policy adapts as the harness evolves. Let θ̂π denote the final harness selected by development-set performance from the optimization trajectory induced by policy π. Under a fixed rollout budget shared across curriculum policies, the automated curriculum-learning objective is h i π ∗ ∈ arg max E J(θ̂π ) . (6) π

After optimization, we report the performance of the selected harness on the test set as JbDtest (θ̂π ).

4

P ROPOSED M ETHOD : ACTIVE S ADDLER

We now present ActiveSaddler, an iterative curriculum-learning framework for harness optimization. We first introduce the problem formulation and overall workflow, then describe key modules in detail. Problem Formulation and Design Rationale. Harness optimization operates under a limited rollout budget, requiring the curriculum to decide at each iteration which training target should receive the next optimization opportunity. This naturally leads to a sequential resource-allocation problem, which we formulate as a multi-armed bandit: each arm represents an optimization target, and selecting an arm directs the next round of optimization toward that target (Graves et al., 2017; Matiisen et al., 2017). To instantiate these optimization targets, we define each arm around a harness weakness rather than an individual failed execution, since a patch typically addresses an underlying deficiency that may recur 4

Preprint

Task Agent (LLM + Harness) Exploration Controller

𝑯𝒕

𝑰<𝒕

𝑨𝒕 𝑼 𝒕

Unseen Exploration

Arm Prioritizer

𝑨𝒕

𝑯𝒕+𝟏 𝑰<𝒕+𝟏𝑨𝒕+𝟏𝑼𝒕+𝟏

𝑯𝒕

Arm A

Arm B Arm C

Failure Pattern

Failure Pattern

𝑼𝒕+𝟏 Arm A

Arm Selection

Unseen Scenario Pool

Arm Set

Scenario Batch

Unseen Scenario Sampling

𝑼𝒕

Update!

𝑨𝒕+𝟏

Optimization History

Training Scenario

𝑰<𝒕

𝝓𝒕 𝑨 = 𝟎. 𝟏𝟏

Arm B

𝝓𝒕 𝑩 = 𝟎. 𝟖𝟔

Arm C

𝝓𝒕 𝑪 = 𝟎. 𝟎𝟑

Stochastic Selection

𝑯𝒕 𝑿𝒕

Execute

Harness Optimization FailurePattern Extractor

𝑯𝒕+𝟏 Failed Execution Records

Arm D

Accumulate

𝑰<𝒕+𝟏

Harness Optimizer

Execution Records

Figure 2: Overview of ActiveSaddler. At each iteration, the curriculum chooses between exploring unseen scenarios and revisiting an instantiated arm, optimizes the harness on the selected scenario batch, and updates its state based on the resulting execution and optimization outcomes.

across multiple scenarios and traces. Because such weaknesses are not directly observable, we infer them through failure diagnosis and group failures attributed to the same deficiency into a common failure-pattern arm. This lets related failures contribute evidence toward a shared optimization target. This formulation introduces two practical challenges. First, the arm set is not known in advance: as the harness evolves and executions vary stochastically, new failure patterns may emerge. We therefore construct the arm set during optimization and expand it as new failure patterns are diagnosed. Second, each iteration typically executes only a subset of the training set (Park et al., 2026; Agrawal et al., 2026), leaving some scenarios and their potential failure patterns unobserved. The curriculum must therefore balance exploitation, by prioritizing among known failure-pattern arms, with exploration, by executing unobserved scenarios that may reveal new weaknesses. These challenges motivate the three core components of ActiveSaddler: a Failure-Pattern Extractor, which identifies recurring harness weaknesses and organizes them into failure-pattern arms; an Arm Prioritizer, which selects among known arms for further optimization; and an Exploration Controller, which decides when to explore unseen scenarios that may reveal new weaknesses. We next describe how these components interact within the iterative harness-optimization loop. Workflow Overview. As illustrated in Figure 2, each iteration t begins with the current harness Ht , accumulated optimization history I<t , instantiated arm set At , and unseen scenario pool Ut . The Exploration Controller first decides whether the next optimization step should explore previously unseen scenarios or continue optimizing a known failure-pattern arm ( 1 ). If exploration is selected, ActiveSaddler constructs the next scenario batch Xt from Ut ( 2 ); otherwise, the Arm Prioritizer selects an arm at ∈ At , and Xt is constructed from the scenarios associated with that arm ( 3 ). The selected scenarios are then executed under Ht , and the resulting execution records Et are passed to the harness optimizer O to produce the updated harness Ht+1 ( 4 ). From failed executions, the Failure-Pattern Extractor identifies recurring harness weaknesses, matches them to existing failurepattern arms or instantiates new ones, and updates the arm set for subsequent iterations ( 5 ). The resulting Ht+1 , optimization history I<t+1 , arm set At+1 , and unseen scenario pool Ut+1 define the state for the next curriculum decision ( 6 ). In this way, the training curriculum co-evolves with the harness as its unresolved weaknesses change over optimization. The full ActiveSaddler algorithm and its instantiation with AutoSaddler are in Appendix B. Failure-Pattern Extraction. The Failure-Pattern Extractor converts failure evidence produced during each optimization step into reusable failure-pattern arms. Let Ft denote the failed execution records made available by the underlying harness optimizer at iteration t, and let δt (e) denote the associated diagnostic evidence for each e ∈ Ft . The exact composition of Ft depends on the 5

Preprint

optimizer. In our AutoSaddler instantiation, it includes both failures diagnosed before patching and failures that remain or newly emerge during post-patch evaluation. The extractor operates in two stages: symptom extraction and pattern normalization. It first independently abstracts each failure into a candidate symptom, ct (e) = gext (e, δt (e)), without consulting the existing arm set. After all candidates are extracted, they are normalized against the current arms: At+1 = gnorm (At , {ct (e)}e∈Ft ) ,

(7)

where gnorm links each candidate to an existing atomic pattern, composes multiple existing patterns when needed, or instantiates a new arm for a previously unseen failure pattern. Each arm a ∈ At+1 maintains a pattern description, a supporting scenario set St+1 (a) ⊆ Dtrain , and the associated harness, execution, and diagnostic evidence under which the pattern was observed. When an arm is selected, its supporting scenarios form the pool for the next optimization batch, subject to the per-iteration execution limit. Detailed extraction instructions are in Appendix K. Arm Prioritization. When the Exploration Controller chooses to continue optimization over the instantiated arm set, the Arm Prioritizer determines which a ∈ At should receive the next optimization attempt. Ideally, this decision would be based on the expected population-performance change induced by selecting arm a. However, this population objective is not directly observable during optimization; we therefore use an LLM to estimate a learning-progress score ϕt (a) = gselect (a, I<t ) ∈ [0, 1],

(8)

where gselect denotes the LLM-based arm-scoring procedure and a larger ϕt (a) indicates greater estimated benefit from allocating the next optimization step to arm a. To estimate this score, the LLM is instructed to jointly consider four factors: whether the failure pattern remains active under the current harness (severity), whether it appears addressable through a harness intervention (fixability), how broadly a successful repair may generalize (breadth), and the risk that such an intervention introduces regressions (side-effect risk). The scores are recomputed from the latest I<t at each arm-selection step, allowing arm priorities to adapt as the harness evolves. The resulting scores define a stochastic selection distribution over the current arm set: exp(ϕt (a)/τsel ) qt (a) = X , exp(ϕt (a′ )/τsel )

a ∈ At ,

(9)

a′ ∈At

where τsel > 0 controls the stochasticity of arm selection. The curriculum then samples at ∼ Categorical(qt )

(10)

and constructs the next scenario batch from the selected arm, Xt ⊆ St (at ). This stochastic selection rule favors arms with greater estimated learning progress while preserving the possibility of revisiting lower-scored arms as their utility changes over optimization. Detailed scoring instructions and definitions of the four factors are in Appendix K. Exploration Control. The Exploration Controller determines whether the next optimization step should exploit a previously discovered failure-pattern arm or explore unseen scenarios that may reveal new failure patterns. Given the current harness Ht , instantiated arm set At , unseen scenario pool Ut , and optimization history I<t , we instantiate an agent session for a binary decision: dt = gexplore (Ht , At , Ut , I<t ) ∈ {unseen, arm}.

(11)

The policy makes this decision based on the current optimization context, weighing whether the known arms contain a worthwhile unresolved optimization target against exploring the remaining unseen scenarios for previously undiscovered weaknesses. If dt = unseen, i.e., a D RAW decision, ActiveSaddler samples Xt ⊆ Ut without replacement and removes the selected scenarios from the unseen pool. If dt = arm, i.e., a P ULL decision, Arm Prioritizer selects at ∈ At , and the next scenario batch is constructed as Xt ⊆ St (at ). When At = ∅ and Ut ̸= ∅, the policy selects unseen; when Ut = ∅ and At ̸= ∅, it selects arm. Detailed decision criteria and instructions are in Appendix K. To complement the regression and generalization checks of the underlying harness optimizer, ActiveSaddler also guards against regressions on previously successful training scenarios. Once all 6

Preprint

Table 1: Test Pass@1 on GAIA2 and Terminal-Bench 2.0 (mean ± std. over three test-time executions). Parentheses show task counts; bold indicates the best result. GAIA2 (Pass@1) Harness (Type)

Terminal-Bench 2.0 (Pass@1) Test-Split (300)

Default Agent (manual) GEPA Meta-Harness AutoSaddler AutoSaddler w/ Category Acc. Order AutoSaddler w/ Scenario Acc. Order ActiveSaddler

53.6 ± 1.1 54.2 ± 2.2 54.2 ± 1.2 55.4 ± 1.2 55.9 ± 1.3 55.7 ± 1.2 59.8 ± 1.0

w/o Failure Pattern Based Arm (Category Arm) w/o Failure Pattern Based Arm (Scenario Arm) w/o Arm Prioritizer w/o Exploration Controller

56.8 ± 0.8 56.2 ± 0.2 55.3 ± 2.1 55.3 ± 0.9

Harness (Type)

Test-Split (40)

Terminus 2 (manual) Terminus-KIRA (manual) GEPA Meta-Harness AutoSaddler AutoSaddler w/ Category Acc. Order AutoSaddler w/ Scenario Acc. Order ActiveSaddler

64.2 ± 2.9 69.2 ± 3.8 65.8 ± 5.2 66.7 ± 5.2 72.5 ± 0.0 70.8 ± 1.4 73.3 ± 1.4 80.0 ± 2.5

w/o Failure Pattern Based Arm (Category Arm) w/o Failure Pattern Based Arm (Scenario Arm) w/o Arm Prioritizer w/o Exploration Controller

69.2 ± 2.9 73.3 ± 3.8 72.5 ± 2.5 71.7 ± 1.4

offline training scenarios have been explored and every arm has since been visited, prior successes that did not instantiate an arm become eligible for exploration again, analogous to additional epochs over seen examples in conventional offline learning. If such a scenario now fails, the Failure-Pattern Extractor processes it normally, enabling regression rediscovery without re-evaluating all prior successes after every harness update.

5

E XPERIMENTS

5.1

E XPERIMENTAL S ETUP

Benchmarks and Base Harness Systems. We evaluate ActiveSaddler on two diverse benchmarks: GAIA2 (Froger et al., 2026) and Terminal-Bench 2.0 (TB2) (Merrill et al., 2026). GAIA2 evaluates general-purpose assistant capabilities in a simulated smartphone environment spanning 10 distinct user environments, referred to as Universes; we use its default ReAct-based agent as the base harness. TB2 comprises 89 realistic tasks across domains including system administration, machine learning, and cybersecurity, and introduces Terminus 2, which we adopt as the base harness. Implementation and Baseline Methods. We instantiate ActiveSaddler on AutoSaddler (Park et al., 2026) by adding three curriculum components: the Failure-Pattern Extractor, Arm Prioritizer, and Exploration Controller (Algorithm 2). These components use the GitHub Copilot SDK (GCSDK) (GitHub, 2026) and a shared pattern registry for dynamically instantiated failure-pattern arms; agent instructions and registry details are provided in Appendix K and Appendix A. We compare ActiveSaddler against GEPA (Agrawal et al., 2026), Meta-Harness (Lee et al., 2026), and AutoSaddler (Park et al., 2026), which optimize prompts or harnesses without an explicit curriculum. We also construct fixed easy-to-hard curricula (Bengio et al., 2009; Zaremba & Sutskever, 2014; Wu & Tian, 2017) by ranking categories or scenarios by empirical accuracy from five evaluations of the base harness on the training set. For reference, we additionally report Terminus-KIRA (KRAFTON AI & Ludo Robotics, 2026), a manually engineered TB2 harness. Experimental Protocol and Data Splits. Unless otherwise noted, both the task agent and optimizer use gpt-5.5, with reasoning effort set to medium for the task agent and xhigh for GC-SDK optimizer sessions. Following AutoSaddler (Park et al., 2026), we use out-of-distribution train/dev/test splits with disjoint task groups to evaluate harness generalization. Detailed train/dev/test splits and hyperparameters are provided in Appendix A. We evaluate each optimized harness over three test-time executions and report mean Pass@1 with standard deviation. 5.2

M AIN R ESULTS

As shown in Table 1, ActiveSaddler achieves the strongest test performance on both GAIA2 and TB2, reaching 59.8% and 80.0% Pass@1, respectively. Under the same AutoSaddler optimizer, ActiveSaddler improves over a fixed randomly shuffled scenario order by +4.4 pp on GAIA2 and +7.5 pp on TB2. It also outperforms difficulty-based fixed curricula by +3.9 pp and +4.1 pp over category- and scenario-level ordering on GAIA2, and by +9.2 pp and +6.7 pp on TB2, respectively. 7

Preprint

Other

P26

P23

P24

P15

P12

P10

P8

P4

P2

P5

ActiveSaddler

0.75

AutoSaddler

75.0

80

0.50 0.25 0.00

w/o Arm Prioritizer

100

Fix Rate (%)

Pull probability

1.00

60 40 20

5

12

24

Optimization iteration

36

48

(a) Evolution of pull-probability mass (GAIA2).

0

50.0 36.4 33.3

30.0 21.7

GAIA2

TB2

(b) Failure-to-success conversion rate.

Figure 3: Effect of adaptive arm scoring. Adaptive scoring dynamically reallocates optimization priority and yields more effective repair of observed failures.

Together, these results show that adapting scenario selection during optimization improves harness quality over fixed curricula under the same optimizer. We further examine these gains through an independent optimization run, a different offline optimizer, and alternative curriculum strategies. In an independent GAIA2 run, ActiveSaddler again outperforms all fixed-order baselines, showing robustness to optimization stochasticity (Appendix C). Applying ActiveSaddler to GEPA improves its average Pass@1 from 54.2% to 57.2% (+3.0 pp), showing that the benefit of adaptive training-scenario selection is not limited to AutoSaddler (Appendix E). We also compare against non-LLM alternatives: EMA-based failure-persistence scoring and count-based UCB-AIR achieve results 3.1 pp and 4.2 pp below ActiveSaddler, respectively (Appendix D). As shown in Figure 1, ActiveSaddler achieves stronger development performance with fewer taskagent rollouts on both benchmarks. We further analyze whether this efficiency persists after accounting for optimizer-side LLM overhead by combining it with task-agent execution cost. Although ActiveSaddler incurs additional optimizer overhead, it achieves a substantially better end-to-end cost–accuracy trade-off on both benchmarks. On GAIA2, ActiveSaddler reaches 58.5% development accuracy at $298, whereas AutoSaddler and category ordering require $1,360 and $673, respectively, to reach the same accuracy, and scenario ordering reaches only 56.9% despite spending $1,698. On TB2, ActiveSaddler reaches 78.9% at $128, while AutoSaddler and scenario ordering require $220 and $363, respectively, to reach the same accuracy, and category ordering reaches only 73.7% despite spending $548. Thus, the additional cost of curriculum adaptation is offset by more effective allocation of expensive task-agent rollouts; detailed cost analyses are provided in Appendix F. 5.3

A BLATION S TUDIES AND A NALYSIS

RQ1: How do failure-pattern arms enable optimization to track and retarget weaknesses as the harness evolves? We first examine how the choice of arm representation affects optimization. Specifically, we replace the failure-pattern arms in ActiveSaddler with category- or scenario-level arms while keeping the curriculum procedure unchanged. As shown in Table 1, category and scenario arms reduce test Pass@1 by 3.0 pp and 3.6 pp on GAIA2, and by 10.8 pp and 6.7 pp on TB2, respectively. These results show that arm representation is critical to harness quality. Analyses in Appendix G further support the benefit of failure-pattern arms. The case studies show that failure-pattern arms preserve unresolved weaknesses as optimization targets, allowing them to be revisited and unsuccessful patches to be iteratively refined. In contrast, category arms are too coarse, grouping distinct weaknesses together and allowing successes on some scenarios to mask unresolved weaknesses in others. Scenario arms are too fine-grained, separating the same weakness across individual scenarios and making it harder to revisit under a limited budget. Consistent with these cases, failure-pattern arms more frequently sample scenarios on which the current harness still fails, exceeding category and scenario arms by 16.5 pp and 35.3 pp on GAIA2, and by 29.9 pp and 23.8 pp on TB2, respectively. The discovered failure pattern arms are listed in Appendix H. RQ2: How does adaptive arm scoring prioritize weaknesses as the harness evolves? We examine whether prioritizing failure-pattern arms by their estimated learning potential improves optimization. Specifically, we compare ActiveSaddler with a w/o Arm Prioritizer variant that assigns 8

Preprint

Iteration 2: DRAW (i.e., unseen) 2 arms in pool | 72 unseen left best dev 55.4 → 56.9%

Only two known arms exist and both come from the same scenario ... the failure surface is poorly mapped, so exploring unseen scenarios is more valuable for this iteration.

Distinct failure patterns discovered

while several high-activity known arms remain genuinely active and broad ... the known-arm set now contains multiple severe, plausibly fixable population-level weaknesses ... another DRAW would likely defer repairs that are already well diagnosed.

30

60

20

55

10

50

24 arms in pool | 33 unseen left best dev 61.5 → 63.1%

Best dev accuracy (%)

8 arms in pool | 60 unseen left best dev 56.9 → 58.5%

Iteration 30: PULL (i.e., arm)

65

Iteration 7: PULL (i.e., arm)

the latest evaluation exposed a fresh, severe, trace-backed known arm ... older known arms are inactive or already resolved, and the trace-less timeout arm is severe but much less diagnosable ... the current known arm provides a clearer expected learning opportunity. Iteration 24: DRAW (i.e., unseen) 16 arms in pool | 45 unseen left best dev 58.5 → 61.5%

the proven C7/C22 executor fixes already cover most previously pulled arms ... arms are either fixed, recently all-pass, or Optimization iteration repeatedly inactive on later candidates ... Initial seeding (pool empty) Best dev accuracy improves discovery value is higher than spending this DRAW: probe an unseen scenario Dev accuracy (shaded) iteration on a mostly exhausted PULL: exploit an existing arm Best dev so far known-arm set. 0

10

20

30

40

50

Figure 4: Adaptive exploration over the optimization trajectory. The policy chooses D RAW when discovery is more valuable and P ULL when promising repair opportunities remain. the same score to every arm while keeping the remaining curriculum procedure unchanged. As shown in Table 1, removing adaptive arm scoring reduces test Pass@1 by 4.5 pp on GAIA2 and 7.5 pp on TB2, showing the value of prioritizing by learning potential under limited rollouts. To understand why adaptive arm scoring improves optimization, Figure 3 examines both how optimization priorities evolve and how effectively observed failures are repaired. As shown in (a), pull-probability mass continually shifts across failure-pattern arms rather than remaining concentrated on a fixed subset, indicating that the curriculum reallocates optimization budget as the harness and accumulated evidence evolve. In (b), we measure the fraction of scenarios observed to fail during optimization that are subsequently converted to success by the optimized harness. ActiveSaddler achieves the highest conversion rate, followed by w/o Arm Prioritizer and AutoSaddler. Further analyses in Appendix I provide representative case studies of resolved and recurring weaknesses, examine probability redistribution across the full arm pool, and analyze stochastic selection as new arms are discovered. Together, these results show that adaptive arm scoring improves optimization by reprioritizing weaknesses as their learning potential changes. RQ3: How does adaptive exploration balance the continued repair of known weaknesses with the discovery of new ones? We examine whether adapting the timing of unseen-scenario exploration improves optimization over a fixed schedule. Specifically, we compare ActiveSaddler with a w/o Exploration Controller variant that explores unseen scenarios every five iterations while keeping the remaining curriculum procedure unchanged. As shown in Table 1, replacing adaptive exploration with this fixed schedule reduces test Pass@1 by 4.5 pp on GAIA2 and 8.3 pp on TB2. These results show that when to explore is critical under a limited optimization budget. Figure 4 illustrates how exploration decisions adapt to the evolving optimization state on GAIA2. Representative decisions show that the policy selects D RAW when known arms offer limited remaining learning value and P ULL when actionable weaknesses remain. This behavior also appears globally: the D RAW rate decreases from 44% in the first half, with relatively few instantiated arms, to 20% in the second half, with a larger arm pool. To quantify this behavior, we compare how many arms remain unresolved according to their most recent observation at P ULL and D RAW decisions. As detailed in Appendix J, ActiveSaddler has 2.0× and 6.9× as many unresolved arms at P ULL as at D RAW decisions on GAIA2 and TB2, respectively. In contrast, the fixed schedule yields ratios close to one (0.8× and 1.3×), indicating that its P ULL and D RAW decisions are largely insensitive to how many unresolved arms remain. Together, these results show that adaptive exploration favors repair when worthwhile unresolved weaknesses remain and discovery when few such weaknesses remain among the known arms.

6

C ONCLUSION

We introduced ActiveSaddler, an automated curriculum-learning approach for harness optimization that adapts which training scenarios generate the execution feedback used for each harness update. Rather than fixing scenario selection before optimization, ActiveSaddler formulates curriculum construction as a non-stationary bandit that dynamically instantiates failure-pattern arms, prioritizes 9

Preprint

them by their evolving learning potential, and balances revisiting known weaknesses with exploring unseen scenarios. Across GAIA2 and Terminal-Bench 2.0, ActiveSaddler improves test Pass@1 by 4.4 and 7.5 percentage points over the same harness optimizer with a fixed scenario order, respectively, while also outperforming existing harness optimizers and fixed curricula. Our ablations further show that these gains rely on failure-pattern arms, adaptive arm prioritization, and adaptive exploration. Overall, these results establish automated curriculum learning as a complementary dimension of harness optimization: performance depends not only on how feedback is used for updates, but also on which training scenarios generate that feedback. AI U SE S TATEMENT We used generative AI tools for writing assistance, including revising sentences and paragraphs to improve clarity, readability, and overall presentation. We also used generative AI tools for retrieval and discovery, including identifying potentially relevant related work and supporting literature searches. All AI-assisted content was carefully reviewed and verified by the authors, including the wording of the manuscript, technical claims, numerical results, and citations. The authors take full responsibility for the final manuscript. R EPRODUCIBILITY S TATEMENT We provide the implementation details, prompts, data splits, and evaluation protocol needed to reproduce our experiments. Section 5.1 specifies the benchmark and base-harness configurations, baseline methods, model settings, construction of the fixed curricula, and test-time evaluation protocol. Appendix A further describes the Pattern Registry and its pattern CLI interface, as well as the data-split construction and task counts for both benchmarks. Algorithm 1 provides the complete optimization procedure, including curriculum decisions, and development-set-based harness selection. Finally, Appendix K provides the full session prompt templates used for failure-pattern extraction, arm scoring, and unseen-scenario exploration.

R EFERENCES Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: reflective prompt evolution can outperform reinforcement learning. In Carl Vondrick, Bharath Hariharan, Colin Raffel, Lerrel Pinto, Diyi Yang, and Aleksandra Faust (eds.), The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. proceedings.iclr.cc, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/ 0e9e708b6f48e14fd0ac29e167413f76-Abstract-Conference.html. Anthropic. Effective harnesses for long-running agents. https://www.anthropic.com/ engineering/effective-harnesses-for-long-running-agents, 2025. Accessed 2026-04-21. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman (eds.), Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pp. 41–48. ACM, 2009. doi: 10.1145/1553374.1553380. URL https://doi.org/10.1145/ 1553374.1553380. Zhengyu Chen, Teng Xiao, Huaisheng Zhu, Yige Yuan, Luan Zhang, and Jingang Wang. Co-harness: Co-evolving harnesses and model weights for LLM agents. CoRR, abs/2607.22688, 2026. doi: 10. 48550/ARXIV.2607.22688. URL https://doi.org/10.48550/arXiv.2607.22688. Jeff Clune. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. CoRR, abs/1905.10985, 2019. URL http://arxiv.org/abs/1905.10985. 10

Preprint

Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Gerard Moreno-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, Mengjue Wang, Ian Yu, Amine Benhalloum, Grégoire Mialon, and Thomas Scialom. Gaia2: Benchmarking LLM agents on dynamic and asynchronous environments. CoRR, abs/2602.11964, 2026. doi: 10.48550/ARXIV. 2602.11964. URL https://doi.org/10.48550/arXiv.2602.11964. GitHub. Github copilot sdk. https://docs.github.com/en/copilot/how-tos/ copilot-sdk, 2026. Accessed 2026-09-23. Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1311–1320. PMLR, 2017. URL http://proceedings.mlr.press/v70/graves17a.html. Hanghui Guo, Weijie Shi, Zhangze Chen, Shengxiang Xu, Yishu Wang, Yimei Zhang, Wangze Ni, Jia Zhu, and Shimin Di. Drevo: Distilling recalibrated historical experience for harness self-evolution. CoRR, abs/2607.26722, 2026. doi: 10.48550/ARXIV.2607.26722. URL https: //doi.org/10.48550/arXiv.2607.26722. Prannay Hebbar, Yogendra Manawat, Samuel Verboomen, Alesia Ivanova, Selvam Palanimalai, Kunal Bhatia, and Vignesh Baskaran. SIA: self improving AI with harness & weight updates. CoRR, abs/2605.27276, 2026. doi: 10.48550/ARXIV.2605.27276. URL https://doi.org/ 10.48550/arXiv.2605.27276. Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 4940–4950. PMLR, 2021. URL http://proceedings.mlr.press/v139/ jiang21b.html. Seth Karten, Joel Zhang, Tersoo Upaa Jr, Ruirong Feng, Wenzhe Li, Chengshuai Shi, Chi Jin, and Kiran Vodrahalli. Continual harness: Online adaptation for self-improving foundation agents. CoRR, abs/2605.09998, 2026. doi: 10.48550/ARXIV.2605.09998. URL https://doi.org/ 10.48550/arXiv.2605.09998. Haechan Kim, Yoonho Lee, Gisang Lee, Chelsea Finn, and Kangwook Lee. Whale: A simple recipe for joint harness-weight optimization. arXiv preprint arXiv:2609.00196, 2026. KRAFTON AI and Ludo Robotics. Terminus-kira: Boosting frontier model performance on terminalbench with minimal harness. https://github.com/krafton-ai/kira, 2026. LangChain. Improving deep agents with harness engineering. https://www.langchain.com/ blog/improving-deep-agents-with-harness-engineering, 2026. Accessed 2026-04-21. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Metaharness: End-to-end optimization of model harnesses. CoRR, abs/2603.28052, 2026. doi: 10. 48550/ARXIV.2603.28052. URL https://doi.org/10.48550/arXiv.2603.28052. Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Xuanjing Huang, Hang Yan, Zhenhua Han, and Tao Gui. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. CoRR, abs/2604.25850, 2026. doi: 10.48550/ARXIV.2604.25850. URL https://doi.org/10.48550/arXiv.2604.25850. Manuel Lopes and Pierre-Yves Oudeyer. The strategic student approach for life-long exploration and learning. In 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics, ICDL-EPIROB 2012, San Diego, CA, USA, November 7-9, 2012, pp. 1–8. IEEE, 2012. doi: 10.1109/DEVLRN.2012.6400807. URL https://doi.org/10.1109/DevLrn. 2012.6400807. 11

Preprint

Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. CoRR, abs/1707.00183, 2017. URL http://arxiv.org/abs/1707.00183. Mike A. Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, Estefany Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis-Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Kumar Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. CoRR, abs/2601.11868, 2026. doi: 10.48550/ARXIV.2601.11868. URL https://doi.org/10. 48550/arXiv.2601.11868. Atsuyuki Miyai, Kiyoharu Aizawa, and Toshihiko Yamasaki. Task-coevolve: Efficient harness optimization via adaptive validation task selection. CoRR, abs/2608.20169, 2026. doi: 10.48550/ ARXIV.2608.20169. URL https://doi.org/10.48550/arXiv.2608.20169. Jun Nie, Yonggang Zhang, Jun Song, Qianshu Cai, Dahai Yu, Yike Guo, Xinmei Tian, and Bo Han. TTHE: test-time harness evolution. CoRR, abs/2607.08124, 2026. doi: 10.48550/ARXIV.2607. 08124. URL https://doi.org/10.48550/arXiv.2607.08124. OpenAI. Harness engineering: leveraging codex in an agent-first world. https://openai.com/ index/harness-engineering/, 2026. Accessed 2026-04-21. Pierre-Yves Oudeyer, Frédéric Kaplan, and Verena Vanessa Hafner. Intrinsic motivation systems for autonomous mental development. IEEE Trans. Evol. Comput., 11(2):265–286, 2007. doi: 10.1109/TEVC.2006.890271. URL https://doi.org/10.1109/TEVC.2006.890271. Wenbo Pan, Shujie Liu, Chin-Yew Lin, Jingying Zeng, Xianfeng Tang, Xiangyang Zhou, Yan Lu, and Xiaohua Jia. Evolving agents in the dark: Retrospective harness optimization via selfpreference. CoRR, abs/2606.05922, 2026. doi: 10.48550/ARXIV.2606.05922. URL https: //doi.org/10.48550/arXiv.2606.05922. Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Autosaddler: Automatic harness optimization with durable updates from agent execution traces. CoRR, abs/2608.23041, 2026. doi: 10.48550/ARXIV.2608.23041. URL https://doi. org/10.48550/arXiv.2608.23041. Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep RL in continuously parameterized environments. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pp. 835–853. PMLR, 2019. URL http: //proceedings.mlr.press/v100/portelas20a.html. Biswa Sengupta and Jinhua Wang. HARBOR: automated harness optimization. CoRR, abs/2604.20938, 2026. doi: 10.48550/ARXIV.2604.20938. URL https://doi.org/10. 48550/arXiv.2604.20938. Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. CoRR, abs/1901.01753, 2019. URL http://arxiv.org/abs/1901.01753. 12

Preprint

Wenyi Wang, Piotr Piekos, Nanbo Li, Firas Laakom, Yimeng Chen, Mateusz Ostaszewski, Mingchen Zhuge, and Jürgen Schmidhuber. Huxley-gödel machine: Human-level coding agent development by an approximation of the optimal self-improving machine. CoRR, abs/2510.21614, 2025. doi: 10. 48550/ARXIV.2510.21614. URL https://doi.org/10.48550/arXiv.2510.21614. Yizao Wang, Jean-Yves Audibert, and Rémi Munos. Algorithms for infinitely many-armed bandits. In Daphne Koller, Dale Schuurmans, Yoshua Bengio, and Léon Bottou (eds.), Advances in Neural Information Processing Systems 21, Proceedings of the Twenty-Second Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 8-11, 2008, pp. 1729–1736. Curran Associates, Inc., 2008. URL https://proceedings.neurips.cc/ paper/2008/hash/49ae49a23f67c759bf4fc791ba842aa2-Abstract.html. Tianxin Wei, Zhan Shi, Minhua Lin, Bing He, Zewen Liu, Yisi Sang, Yuanchen Bei, Xuying Ning, Jiaru Zou, Ting-Wei Li, Xiao Lin, Yanjun Zhao, Chi Wang, Benoît Dumoulin, Dakuo Wang, Jingrui He, and Hanqing Lu. Evo-harness: Context-to-harness skill compilation for self-evolving agents. CoRR, abs/2608.15071, 2026. doi: 10.48550/ARXIV.2608.15071. URL https://doi.org/10.48550/arXiv.2608.15071. Yuxin Wu and Yuandong Tian. Training agent for first-person shooter game with actor-critic curriculum learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=Hk3mPK5gg. Jinghan Xu, Yikai Zhang, Aili Chen, Weiyuan Li, Jiaqing Liang, and Deqing Yang. Verify smarter, evolve further: Efficient harness evolution through behavior-aware verification. CoRR, abs/2608.27311, 2026. doi: 10.48550/ARXIV.2608.27311. URL https://doi.org/10. 48550/arXiv.2608.27311. Shidong Yang, Ziyu Ma, Tongwen Huang, Yiming Hu, Yong Wang, and Xiangxiang Chu. Coevolve: Training LLM agents via agent-data mutual evolution. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.), Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, pp. 23015–23036. Association for Computational Linguistics, 2026a. doi: 10.18653/V1/2026.ACL-LONG.1055. URL https://doi.org/10.18653/v1/2026. acl-long.1055. Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, and Chong Luo. Skillopt: Executive strategy for self-evolving agent skills. CoRR, abs/2605.23904, 2026b. doi: 10.48550/ ARXIV.2605.23904. URL https://doi.org/10.48550/arXiv.2605.23904. Wojciech Zaremba and Ilya Sutskever. Learning to execute. CoRR, abs/1410.4615, 2014. URL http://arxiv.org/abs/1410.4615. Luan Zhang, Ruochen Zhou, Dandan Song, Zhengyu Chen, Yuhang Tian, Jun Yang, Huipeng Ma, Chenhao Li, Guangyuan Feng, Xudong Li, Yizhou Jin, and Yan Xu. Harnesscompass: Guiding automatic harness evolution toward generalizable and effective agent harnesses. CoRR, abs/2608.01918, 2026. doi: 10.48550/ARXIV.2608.01918. URL https://doi.org/10. 48550/arXiv.2608.01918. Chenyu Zhou, Huacan Chai, Wenteng Chen, Zihan Guo, Rong Shan, Yuanyi Song, Tianyi Xu, Yingxuan Yang, Aofan Yu, Weiming Zhang, Congming Zheng, Jiachen Zhu, Zeyu Zheng, Zhuosheng Zhang, Xingyu Lou, Changwang Zhang, Zhihui Fu, Jun Wang, Weiwen Liu, Jianghao Lin, and Weinan Zhang. Externalization in LLM agents: A unified review of memory, skills, protocols and harness engineering. CoRR, abs/2604.08224, 2026. doi: 10.48550/ARXIV.2604.08224. URL https://doi.org/10.48550/arXiv.2604.08224.

13

Preprint

A

A DDITIONAL D ETAILS ON E XPERIMENT S ETUP

Implementation of Pattern Registry. The Pattern Registry stores each failure pattern, i.e., each arm, as a symptom-level label with its tagged (harness, trace, scenario) tuples, root-cause evidence, activity observations, and learning-progress scores. The registry can be implemented straightforwardly, but providing effective agent access to it is more challenging: as arms accumulate observation histories and evidence across iterations, serializing the registry into the prompt is neither compact nor selective. We therefore introduce the pattern command-line interface (CLI) to facilitate interaction between the GC-SDK and the Pattern Registry. The interface provides on-demand access to scoped views of the arm population, including the canonical pattern table, per-pattern evidence, score rankings, and the complete pull history of an arm with its patch intents; it is also the only channel through which the agent registers arms, tags tuples, and records scores and decisions. Because sessions differ in what they are permitted to do, the outer loop installs the CLI with a per-session capability manifest determined by the sampling strategy, and commands outside it are refused. The full command set is summarized in Table 2. Table 2: Commands exposed by the pattern CLI, grouped into read operations for scoped views of the arm population and write operations for per-session registry updates. Each session receives only the subset of commands in its capability manifest. Command

Purpose

When to use

pattern list

Canonical arm table: score, observation history, scenario count, last observation, label

Start of session; orienting to the full arm population

pattern show <id>

Pattern details: label, score, tagged tuples, per-scenario root causes, observations

Inspecting a specific arm before rating or patching

pattern score [-top-k <n>]

Top-k arms ranked by current score

Quick triage of the highestpriority arms

pattern scenarios -pattern-id <id>

Scenarios tagged to an arm, with cooccurring patterns

Checking an arm’s coverage and overlap

pattern history [-pattern-id <ids>] [-last-k <K>]

Complete per-arm pull history: patched attempts, all-pass skips, and failed attempts, with diagnoses, patch intents, dev-set impact, and per-scenario reflections

Before scoring or re-selecting an arm; assessing what prior repairs already attempted

pattern register -label "..."

Create a new arm from a symptomlevel label; returns its pattern ID

Pattern Extraction Session, for a newly observed failure type

pattern tag -pattern-id <ids> -harness <idx> -trace <dir> -scenario <sid> [-root-cause "..."]

Bind a (harness, trace, scenario) tuple to one or more patterns and record its root-cause evidence

Pattern Extraction Session, after normalizing a candidate against existing arms

pattern rate -pattern-id <id> -severity -fixability -breadth -side-effect [-rationale]

Record the cardinal learningprogress score for the current iteration from the severity, fixability, breadth, and side-effect axes

Arm Scoring Session, once per arm; an unrated arm defaults to the lowest priority

pattern decide -action pull|draw [-rationale]

Record the exploration decision: exploit a known arm (pull) or explore the unseen-scenario pool for a new failure type (draw)

Unseen-Scenario Exploration Session, exactly once per iteration

Read operations

Write operations

14

Preprint

Data Splits across Benchmarks. To evaluate whether optimized harnesses generalize beyond the tasks seen during optimization, we use the same train, development, and test splits as AutoSaddler (Park et al., 2026) for each benchmark, as summarized in Table 3. For benchmarks with a natural grouping structure, we split by task group rather than by individual tasks. In GAIA2, the split axis is the Universe, where each Universe corresponds to a distinct persona and simulated digital environment. This design ensures that the test set contains tasks from groups unseen during optimization, providing a stronger measure of out-of-distribution generalization than random task-level splits. For Terminal-Bench 2.0, tasks span diverse domains but do not provide a natural grouping axis; we therefore use a uniform random partition into train, dev, and test sets. Table 3: Data splits across the two benchmarks. For GAIA2, we evaluate generalization under distribution shift by partitioning the benchmark such that train, development, and test sets contain tasks from disjoint Universes (personas), rather than using random task-level splits. TerminalBench 2.0 contains only 89 tasks across diverse domains and offers no natural grouping axis, so we adopt a uniform random partition. Benchmark

GAIA2

Terminal-Bench 2.0

Split Axis

Universe (persona)

Random†

Split

Task Group

# Tasks

Train

Universe 29

75

Dev

Universe 30

65

Test

Universe 21 Universe 22 Universe 27

107 112 81

Train

—

30

Dev

—

19

Test

—

40

†

Only 89 tasks across heterogeneous domains (system administration, machine learning, and cybersecurity); no natural axis for distribution-shifted splits exists.

Hyperparameters. We set the shared rollout budget to K = 10 (|Dtrain | + |Ddev |), corresponding to 1,400 rollouts on GAIA2 and 490 on Terminal-Bench 2.0. We choose this budget to match ten full passes over the Meta-Harness training set: following the configuration used in AutoSaddler (Park et al., 2026), Meta-Harness treats Dtrain ∪ Ddev as its training set, and Meta-Harness (Lee et al., 2026) runs optimization for 10 iterations. For all methods, we limit each optimization iteration to at most three scenarios, matching the mini-batch size used in AutoSaddler (Park et al., 2026). We set the arm-selection temperature to τsel = 0.15.

B

A LGORITHM OF ACTIVE S ADDLER

General Algorithm. Algorithm 1 operationalizes the curriculum policy defined in Sections 3 and 4. At each iteration, the Exploration Controller first decides whether to explore scenarios from the current exploration pool or continue optimization of an instantiated failure-pattern arm. If an arm is chosen, the Arm Prioritizer scores the current arms and samples the next optimization target. After the fixed harness optimizer O updates the harness, the Failure-Pattern Extractor converts the resulting failure evidence into matched, composed, or newly instantiated failure-pattern arms. Thus, the curriculum determines which training scenarios form Xt , while O determines how the resulting execution evidence is translated into harness updates and uses Ddev for candidate selection, regression checks, and generalization checks. The regression-rediscovery rule in Section 4 is handled within the Exploration Controller. Once all training scenarios have been explored and every arm has since been visited, previously successful scenarios that did not instantiate an arm become eligible for exploration again. This changes only the eligibility of scenarios for the unseen branch and does not introduce an additional curriculum action. Following AutoSaddler (Park et al., 2026), all task-level executions incurred during optimization, including those used by the underlying optimizer to evaluate candidate harnesses, count toward the fixed rollout budget K, and the final harness is selected based on development-set performance. 15

Preprint

Algorithm 1 ActiveSaddler for automated curriculum learning in harness optimization Require: Initial harness parameters θ0 , training set Dtrain , development set Ddev , rollout budget K, fixed harness optimizer O, arm-selection temperature τsel , and procedures gexplore , gselect , gext , gnorm 1: H0 ← Hθ0 2: Initialize optimization history I<0 with H0 and arm set A0 ← ∅ 3: U0 ← Dtrain ; t ← 0 4: while rollout budget remains do Exploration Controller if Ut = ∅, all scenarios in Dtrain have been explored, and every instantiated arm has since been visited 5: then 6: Add to Ut previously successful scenarios that did not instantiate a failure-pattern arm 7: end if 8: if At = ∅ and Ut ̸= ∅ then 9: dt ← unseen 10: else if Ut = ∅ and At ̸= ∅ then 11: dt ← arm 12: else if At ̸= ∅ and Ut ̸= ∅ then 13: dt ← gexplore (Ht , At , Ut , I<t ) 14: else 15: break 16: end if 17: if dt = unseen then 18: Sample Xt ⊆ Ut without replacement, subject to the per-iteration execution limit 19: Ut ← Ut \ Xt 20: else Arm Prioritizer 21: for all a ∈ At do 22: ϕt (a) ← gselect (a, I<t ) 23: end for exp(ϕt (a)/τsel ) 24: qt (a) ← P for all a ∈ At ′ ′ a ∈At exp(ϕt (a )/τsel ) 25: at ∼ Categorical(qt ) 26: Construct Xt ⊆ St (at ), subject to the per-iteration execution limit 27: end if Harness Optimizer 28: Et ← E XECUTE(Ht , Xt ) 29: θt+1 ∼ O(θt , Et , Ddev ); Ht+1 ← Hθt+1 30: Obtain failed execution records Ft and diagnostic evidence {δt (e)}e∈Ft made available by O Failure-Pattern Extractor 31: for all e ∈ Ft do 32: ct (e) ← gext (e, δt (e)) ▷ Symptom extraction 33: end for 34: At+1 ← gnorm (At , {ct (e)}e∈Ft ) ▷ Pattern normalization 35: Update the supporting scenario sets and associated harness, execution, and diagnostic evidence for matched, composed, or newly instantiated arms Curriculum-State Update 36: Ut+1 ← Ut 37: Update I<t+1 with Ht+1 , Xt , the execution and optimizer outcomes, and the curriculum decision 38: t←t+1 39: end while 40: Select θ̂π by development-set performance 41: return Hθ̂π

AutoSaddler Instantiation. Algorithm 2 instantiates the same ActiveSaddler curriculum on AutoSaddler. The three curriculum components, the Exploration Controller, Arm Prioritizer, and Failure-Pattern Extractor, are unchanged from Algorithm 1. We repeat the full procedure for line-by-line comparison, with only the AutoSaddler-specific optimizer operations highlighted in blue. The AutoSaddler-specific difference lies in how the selected execution records are converted into a harness update and which failures are exposed to the Failure-Pattern Extractor. After executing Ht on Xt , the Diagnosis–Patch Session diagnoses the observed failures and derives a structured patch, yielding a candidate harness Ht′ . The candidate is then executed on the same scenario batch 16

Preprint

Algorithm 2 ActiveSaddler instantiated on AutoSaddler (AutoSaddler-specific operations highlighted in blue) Require: Initial harness parameters θ0 , training set Dtrain , development set Ddev , rollout budget K, AutoSaddler as the fixed harness optimizer O, arm-selection temperature τsel , and procedures gexplore , gselect , gext , gnorm 1: H0 ← Hθ0 2: Initialize optimization history I<0 with H0 and arm set A0 ← ∅ 3: U0 ← Dtrain ; t ← 0 4: while rollout budget remains do Exploration Controller if Ut = ∅, all scenarios in Dtrain have been explored, and every instantiated arm has since been visited 5: then 6: Add to Ut previously successful scenarios that did not instantiate a failure-pattern arm 7: end if 8: if At = ∅ and Ut ̸= ∅ then 9: dt ← unseen 10: else if Ut = ∅ and At ̸= ∅ then 11: dt ← arm 12: else if At ̸= ∅ and Ut ̸= ∅ then 13: dt ← gexplore (Ht , At , Ut , I<t ) 14: else 15: break 16: end if 17: if dt = unseen then 18: Sample Xt ⊆ Ut without replacement, subject to the per-iteration execution limit 19: Ut ← Ut \ Xt 20: else Arm Prioritizer 21: for all a ∈ At do 22: ϕt (a) ← gselect (a, I<t ) 23: end for exp(ϕt (a)/τsel ) 24: qt (a) ← P for all a ∈ At ′ a′ ∈At exp(ϕt (a )/τsel ) 25: at ∼ Categorical(qt ) 26: Construct Xt ⊆ St (at ), subject to the per-iteration execution limit 27: end if AutoSaddler Optimizer 28: Et ← E XECUTE(Ht , Xt ) 29: if FAILED(Et ) ̸= ∅ then 30: Diagnose failures in Et and derive a structured patch, yielding candidate harness Ht′ 31: Etpost ← E XECUTE(Ht′ , Xt ) 32: If Ht′ improves over Ht on Xt , additionally evaluate Ht′ on Ddev 33: Reflect on the pre-/post-patch outcomes; record fixed, regressed, still-failing, and still-passing cases; and update EvoDAG 34: Ht+1 ← E VOLVE(EvoDAG) 35: Ft ← FAILED(Et ) ∪ FAILED(Etpost ), with diagnostic evidence {δt (e)}e∈Ft from the Diagnosis– Patch and Reflection sessions 36: else 37: Ht+1 ← Ht ; Ft ← ∅ 38: end if Failure-Pattern Extractor 39: for all e ∈ Ft do 40: ct (e) ← gext (e, δt (e)) ▷ Symptom extraction 41: end for 42: At+1 ← gnorm (At , {ct (e)}e∈Ft ) ▷ Pattern normalization Update the supporting scenario sets and associated harness, execution, and diagnostic evidence for 43: matched, composed, or newly instantiated arms Curriculum-State Update 44: Ut+1 ← Ut 45: Update I<t+1 with Ht+1 , Xt , the execution and optimizer outcomes, and the curriculum decision 46: t←t+1 47: end while 48: Select θ̂π by development-set performance 49: return Hθ̂π

17

Preprint

Xt and, if it improves over Ht on this batch, is additionally evaluated on Ddev . The Reflection Session compares the pre- and post-patch executions, records fixed, regressed, still-failing, and still-passing cases, and updates EvoDAG. The Evolution Session then synthesizes Ht+1 from the updated EvoDAG. For this instantiation, Ft contains failures diagnosed before patching together with failures that remain or newly emerge during post-patch evaluation, with diagnostic evidence supplied by the Diagnosis–Patch and Reflection sessions. The same Failure-Pattern Extractor then processes Ft exactly as in the general algorithm.

C

ROBUSTNESS TO O PTIMIZATION S TOCHASTICITY

Table 4: Robustness to optimization stochasticity. Pass@1 on three held-out GAIA2 universes for harnesses obtained from two independent optimization runs of AutoSaddler, its difficulty-based fixed curricula, and ActiveSaddler. Performance on each universe is reported as mean ± standard deviation over three test-time executions, and the average is computed across all test scenarios. GAIA2 Universe (Pass@1)

Harness (Type) Default Agent (manual) AutoSaddler (Run 1) AutoSaddler (Run 2) AutoSaddler w/ Category Acc. Order (Run 1) AutoSaddler w/ Category Acc. Order (Run 2) AutoSaddler w/ Scenario Acc. Order (Run 1) AutoSaddler w/ Scenario Acc. Order (Run 2) ActiveSaddler (Run 1) ActiveSaddler (Run 2)

Avg.

21 (107)

22 (112)

27 (81)

54.5 ± 0.5 57.0 ± 0.9 57.6 ± 1.4 56.1 ± 3.4 57.0 ± 3.4 57.0 ± 1.9 56.1 ± 3.7 60.4 ± 1.4 62.0 ± 4.4

50.0 ± 1.5 51.5 ± 2.2 53.0 ± 1.0 53.9 ± 1.0 53.6 ± 0.9 53.3 ± 1.0 52.7 ± 2.4 57.7 ± 2.6 60.4 ± 1.9

57.2 ± 1.4 58.8 ± 0.7 60.1 ± 2.9 58.4 ± 1.9 58.4 ± 0.7 57.2 ± 3.8 60.1 ± 1.9 61.7 ± 1.2 62.6 ± 2.6

53.6 ± 1.1 55.4 ± 1.2 56.6 ± 1.0 55.9 ± 1.3 56.1 ± 1.0 55.7 ± 1.2 55.9 ± 0.7 59.8 ± 1.0 61.6 ± 2.9

To assess robustness to optimization stochasticity, we conduct two independent GAIA2 optimization runs for AutoSaddler, its category- and scenario-level difficulty-based fixed curricula, and ActiveSaddler. Each resulting harness is evaluated on three held-out universes (Universes 21, 22, and 27; 300 scenarios in total) with three repeated test-time executions. Both runs of ActiveSaddler outperform all fixed-order baselines on each held-out universe. Across the three universes, ActiveSaddler achieves 59.8% and 61.6% Pass@1 in Runs 1 and 2, exceeding the fixed-order baselines by 3.9 to 4.4 and 5.0 to 5.7 percentage points, respectively. These consistent gains across independent optimization runs and held-out universes indicate that the improvement of ActiveSaddler is not specific to a single optimization trajectory.

D

A LTERNATIVE A RM P RIORITIZATION AND E XPLORATION S TRATEGIES

Table 5: Comparison with alternative curriculum strategies. Pass@1 on three held-out GAIA2 universes when replacing either the Arm Prioritizer or Exploration Controller in ActiveSaddler with an alternative strategy while keeping the remaining curriculum components unchanged. Performance on each universe is reported as mean ± standard deviation over three test-time executions. Method

GAIA2 Universe (Pass@1)

Avg.

21 (107)

22 (112)

27 (81)

Default Agent (manual) AutoSaddler ActiveSaddler ActiveSaddler w/ EMA Scoring ActiveSaddler w/ UCB-AIR Controller

54.5 ± 0.5 57.0 ± 0.9 60.4 ± 1.4 57.9 ± 2.5 57.9 ± 3.4

50.0 ± 1.5 51.5 ± 2.2 57.7 ± 2.6 53.0 ± 3.4 50.9 ± 1.5

57.2 ± 1.4 58.8 ± 0.7 61.7 ± 1.2 60.1 ± 0.7 58.8 ± 3.1

53.6 ± 1.1 55.4 ± 1.2 59.8 ± 1.0 56.7 ± 1.5 55.6 ± 1.4

w/o Arm Prioritizer w/o Exploration Controller

57.3 ± 1.4 56.4 ± 2.2

51.2 ± 3.1 51.5 ± 0.5

58.4 ± 4.3 59.3 ± 3.3

55.3 ± 2.1 55.3 ± 0.9

To examine alternative design choices for the two curriculum decisions in ActiveSaddler, we replace the Arm Prioritizer and Exploration Controller, one at a time, with simple non-LLM alternatives while 18

Preprint

keeping the Failure-Pattern Extractor, arm-selection distribution, and underlying harness optimizer unchanged. First, ActiveSaddler w/ EMA Scoring replaces the LLM-based learning-progress estimate ϕt (a) = gselect (a, I<t ) with an exponential moving-average (EMA) score, following the EMA-based curriculum scoring used by TSCL (Matiisen et al., 2017). In our setting, the score tracks the residual failure rate of each arm after patching:  ϕt (a) = X̄t (a),

X̄t (a) =

(1 − η)X̄t−1 (a) + η activet (a), X̄t−1 (a),

if a is observed at t, otherwise,

where activet (a) is the fraction of evaluated scenarios associated with a that still exhibit the same failure pattern after patching. Thus, recurring failures that remain unresolved receive higher scores and are prioritized for further optimization. Following TSCL (Matiisen et al., 2017), we set η = 0.1. Newly instantiated arms are initialized with X̄t (a) = 1. Second, ActiveSaddler w/ UCB-AIR Controller retains the LLM-based Arm Prioritizer but replaces the Exploration Controller dt = gexplore (Ht , At , Ut , I<t ) with a count-based arm-creation schedule derived from UCB-AIR for infinitely many-armed bandits (Wang et al., 2008):  dt =

unseen, arm,

Ntα ≥ |At |, otherwise,

where Nt is the cumulative number of scenario executions so far. We set α = 0.6, following the UCB-AIR configuration adopted by HGM (Wang et al., 2025). This rule periodically explores unseen scenarios as optimization proceeds, using only the number of previous executions and instantiated arms rather than the current optimization state. Table 5 reports the results. EMA scoring achieves 56.7% average Pass@1, improving by 1.4 percentage points over removing the Arm Prioritizer (55.3%), showing that even a simple failurepersistence signal provides useful prioritization. However, it remains 3.1 points below the 59.8% achieved by ActiveSaddler. By comparison, the UCB-AIR controller achieves 55.6%, only 0.3 points above the fixed exploration schedule (55.3%) and 4.2 points below ActiveSaddler. Together, these results show that, among the tested alternatives, the LLM-based modules are more effective for the two key curriculum decisions: which known failure pattern to revisit and when to explore unseen scenarios for new ones.

E

A PPLYING ACTIVE S ADDLER TO GEPA

Table 6: Applying ActiveSaddler to GEPA. Pass@1 on three held-out GAIA2 universes for the default agent, GEPA, and GEPA augmented with the ActiveSaddler curriculum. Performance on each universe is reported as mean ± standard deviation over three test-time executions. The average is computed across all test scenarios in the three universes. Method Default Agent (manual) GEPA GEPA w/ ActiveSaddler

GAIA2 Universe (Pass@1) 21 (107)

22 (112)

27 (81)

54.5 ± 0.5 56.4 ± 1.4 58.6 ± 0.5

50.0 ± 1.5 50.0 ± 2.4 54.2 ± 1.0

57.2 ± 1.4 57.2 ± 3.6 59.7 ± 0.7

Avg. 53.6 ± 1.1 54.2 ± 2.2 57.2 ± 0.4

Our main experiments instantiate ActiveSaddler on AutoSaddler. To examine whether its curriculum benefit extends beyond AutoSaddler, we additionally apply ActiveSaddler to GEPA, another offline optimizer. At each iteration, ActiveSaddler selects the training-scenario batch provided to GEPA, while GEPA’s underlying prompt-update and candidate-selection procedure remains unchanged. We follow the same GAIA2 train/development/test split and test-time evaluation protocol as in the main experiments. 19

Preprint

Table 7: Optimizer-side cost per generated patch. Runtime, monetary cost, LLM calls, and token usage are averaged per generated patch. GAIA2 Time Generated Rejected Accepted (s)

Method AutoSaddler w/ Category Acc. Order w/ Scenario Acc. Order ActiveSaddler

32 20 30 37

14 7 18 18

Cost ($) LLM Calls Output Tokens Cache-Read Input Tokens

18 943 7.87 13 889 6.71 12 1,142 7.75 19 1,581 11.44

95.4 91.7 100.3 142.2

57,751 59,945 61,791 89,525

5,956,240 5,717,811 6,087,970 8,704,858

Terminal-Bench 2.0 Method

Generated Rejected Accepted

AutoSaddler w/ Category Acc. Order w/ Scenario Acc. Order ActiveSaddler

26 20 14 35

12 12 5 17

Time (s)

Cost ($) LLM Calls Output Tokens Cache-Read Input Tokens

14 977 6.77 8 801 6.71 9 875 7.21 18 1,438 11.37

79.2 80.2 82.9 121.5

47,389 49,387 52,445 75,257

4,513,280 4,641,920 4,796,855 6,873,936

Table 8: Average task-agent evaluation cost per rollout. GAIA2 Method AutoSaddler w/ Category Acc. Order w/ Scenario Acc. Order ActiveSaddler

LLM Calls

Input Tokens

Output Tokens

Time (s)

18.9 16.5 16.4 17.5

467,996 381,948 410,934 410,960

13,678 11,629 11,948 12,340

231.6 201.4 199.8 212.0

Terminal-Bench 2.0 Method AutoSaddler w/ Category Acc. Order w/ Scenario Acc. Order ActiveSaddler

LLM Calls

Input Tokens

Output Tokens

Time (s)

11.8 14.8 14.9 13.4

204,258 331,167 321,888 307,070

10,689 12,124 12,198 11,553

305.4 365.5 369.6 322.6

Table 6 reports the results. Applying ActiveSaddler improves GEPA on all three held-out universes, from 56.4% to 58.6% on Universe 21, from 50.0% to 54.2% on Universe 22, and from 57.2% to 59.7% on Universe 27. Aggregated over the three universes, GEPA w/ ActiveSaddler achieves 57.2% Pass@1, compared with 54.2% for GEPA, an improvement of 3.0 percentage points. These results show that the benefit of adapting which training scenarios drive optimization also extends beyond the AutoSaddler instantiation used in our main experiments.

F

E ND - TO -E ND O PTIMIZATION C OST C HARACTERIZATION

We profile optimizer-side computation and task-agent execution separately, and then combine them to characterize end-to-end optimization cost. Optimizer-side cost includes AutoSaddler’s diagnosis, patch generation, reflection, and candidate selection; for ActiveSaddler, it additionally includes the curriculum-layer operations for failure-pattern extraction, arm scoring, and unseen-scenario exploration. This decomposition allows us to examine how the additional computation introduced by the curriculum layer affects overall optimization efficiency. Cost Breakdown. We first examine optimizer-side and task-agent costs separately to understand where the end-to-end optimization cost arises. Table 7 reports optimizer-side resource usage per generated patch, including AutoSaddler’s diagnosis, patch generation, reflection, and candidate selection; for ActiveSaddler, it additionally includes the curriculum-layer operations for failurepattern extraction, arm scoring, and unseen-scenario exploration. On GAIA2, ActiveSaddler costs $11.44 per generated patch, compared with $7.87 for AutoSaddler, $6.71 for category ordering, and $7.75 for scenario ordering. On Terminal-Bench 2.0, the corresponding costs are $11.37, $6.77, $6.71, and $7.21. In contrast, Table 8 shows that the average cost of an individual task-agent rollout 20

Preprint

Cost

LLM calls

Input tokens

Output tokens

GAIA2 best dev acc. (%)

64 62 60 58 56 54

0

500

0

200

1000

1500

2000 0

400

600

800 0.0

10

20

30

0

200

400

7.5

10.0

0

100

200

600

800

0

5

10

15

20

2

4

6

8

Terminal-Bench 2.0 best dev acc. (%)

90

80

70

60

Cost ($) ActiveSaddler

2.5

5.0

LLM calls (k) AutoSaddler

300

Input tokens (M)

AutoSaddler w/ Category Acc. Order

400

0

Output tokens (M) AutoSaddler w/ Scenario Acc. Order

Figure 5: End-to-end optimization efficiency. Best-so-far development accuracy against cumulative monetary cost, LLM calls, input tokens, and output tokens on GAIA2 (top) and Terminal-Bench 2.0 (bottom). Cumulative usage includes both optimizer-side and task-agent computation.

is broadly comparable across methods on each benchmark. This suggests that differences in total task-agent cost are driven primarily by how rollouts are allocated over the optimization trajectory rather than by substantially cheaper individual executions. End-to-End Cost Efficiency. We next combine optimizer-side and task-agent usage along each optimization trajectory and measure the best development accuracy achieved at each cumulative resource level. For the two accuracy-ordered AutoSaddler variants, the five full training-set evaluations used to construct the ordering are counted toward both the shared rollout budget and the cumulative optimization cost. Figure 5 reports cumulative monetary cost, LLM calls, input tokens, and output tokens, with every method truncated at the shared rollout budget. After accounting for both optimizer-side and task-agent computation, ActiveSaddler achieves a better end-to-end cost–accuracy trade-off on both benchmarks. On GAIA2, ActiveSaddler reaches 58.5% development accuracy at $298, 61.5% at $730, and 63.1% at $1,178. In contrast, AutoSaddler reaches 58.5% only at $1,360 and remains at that accuracy after $1,838, category ordering reaches 58.5% at $673 and remains there after $1,479, and scenario ordering does not exceed 56.9% after $1,698. On Terminal-Bench 2.0, ActiveSaddler reaches 78.9% at $128 and 84.2% at $181, whereas AutoSaddler and scenario ordering require $220 and $363, respectively, to reach their peak of 78.9%, and category ordering does not exceed 73.7% after $548. The same qualitative advantage holds for cumulative LLM calls and token usage. Overall, these results show that, despite its additional optimizer-side overhead, ActiveSaddler achieves better end-to-end cost efficiency by allocating task-agent rollouts to more useful optimization targets.

G

A DDITIONAL A NALYSIS FOR RQ1

Case Studies. Figure 6 illustrates how retargeting the same failure-pattern arm enables iterative refinement of an initially unsuccessful repair. In this task, the agent should email two friends to confirm their attendance and, if either responds positively within four minutes, immediately reply that the party starts at 10 PM. The agent completed the interaction but produced an affirmative event-start reply that omitted the required closing and user-name signature. This failure instantiated a failure-pattern arm capturing that affirmative event-start replies were not canonicalized to the required acknowledgement structure. The first repair at Iteration 3 added an executor hook that detects such replies and appends the required closing and real-name signature. However, the hook was designed to trigger only when the model-generated reply was a single-line draft. Because the model instead produced a multiline reply, the hook did not trigger and the scenario remained unsuccessful. Because the weakness remained explicitly represented as a failure-pattern arm, it was retargeted at Iteration 5. 21

Preprint

Failure Pattern: Affirmative event-start email reply content is not canonicalized to the required acknowledgement structure Task: Email two friends to confirm their party attendance, and if either replies positively within 4 minutes, immediately reply tha t the party starts at 10 PM. Expected Email (Oracle) Step 1: Email both friends to confirm attendance

Golden Steps

Step 2: Monitor their replies for 4 minutes Step 3: Immediately tell positive respondents the party starts at 10 PM

Thanks for confirming! The party will start at 10 PM. Looking forward to seeing you then! Best, Rohan Kumar Actual Email (Agent) Great, thanks for confirming! The party will start at 10 PM.

User-name signature omitted Supported Scenario

Patch: Added an executor hook that detects affirmative event-start replies and automatically appends the required closing and real-name signature.

Patch: Revised the hook to recognize multiline affirmative replies and overwrite them with a canonical reply including the required closing and real-name signature.

Result: The detection rule only accepted single-line drafts, so the hook never triggered on the model’s multiline reply → Still failed.

Result: The hook now triggered on the model’s actual draft and enforced the required format → Passed

First Attempt (Iter 3)

Second Attempt (Iter 5)

Figure 6: Case study on tnxtee (GAIA2), where a failure-pattern arm instantiated from the initial failure enables an unsuccessful repair to be retargeted and refined in a subsequent iteration. The second repair extended the trigger condition to multiline affirmative replies, allowing the hook to correctly append the required closing and real-name signature; the scenario then passed. In the category-arm variant, tnxtee was grouped with 10 other scenarios under the much broader category-time arm. After the first sampled time batch passed, the arm’s severity dropped sharply from 0.62 to 0.12, and its median rank was only fourth among the five category arms. Although the arm was selected three times, tnxtee itself was never sampled. Thus, successes on other timerelated scenarios reduced the estimated value of the entire category before this still-active weakness was observed, preventing the optimizer from obtaining an opportunity to repair it. Failure Pattern: Batch event replacement scheduling fails to partition ambiguous and unambiguous dates Task: Find and cancel the events tagged either 'Social' or 'Personal'. Additionally, for each day with at least one cancelled event, schedule a new "Volunteer Social Work" event at the same time as the cancelled event. Ask if unclear! Golden Steps

Step 1: Retrieve & delete events for tagged Social or Personal (8 events across Nov 2-7)

Step 2: Create "Volunteer Social Work" events only for days with exactly one cancelled event (Nov 4, 5, 7 → 3 events)

Step 3: Request clarification for days with multiple cancelled events (Nov 2: 2 events, Nov 6: 3 events)

Step 1: Success

Step 2: Created 8 events instead of 3

Step 3: Unreached

Supported Scenario Patch: Added a hook before sending the clarification message to auto-create missing replacements for unambiguous dates.

Patch: Moved the intervention to calendar-event pre-registration and blocked replacements for dates with multiple cancellations.

Result: Invalid replacements were already registered during calendar creation, while the hook ran only before the clarification message, too late to intervene → Still failed.

Result: Invalid replacements were blocked before registration, leaving only the 3 unambiguous events → Passed

First Attempt (Iter 24)

Second Attempt (Iter 25)

Figure 7: Case study on 4bytkx (GAIA2), where retargeting the same failure-pattern arm corrects an executor hook that was initially inserted too late in the execution flow. Figure 7 illustrates a complementary form of repair refinement, where the first repair addresses the correct weakness but places the executor hook too late in the execution flow. In this task, the agent should cancel all events tagged Social or Personal, create replacement events only for dates with a single cancelled event, and request clarification for dates with multiple cancellations. Instead, it created replacement events for all eight cancelled events, including dates whose replacement times were ambiguous. This failure instantiated a failure-pattern arm capturing that batch event replacement failed to distinguish ambiguous from unambiguous dates. The first repair at Iteration 24 added an executor hook immediately before the clarification message so that valid replacements could be created while ambiguous dates were left for clarification. However, by the time this hook ran, the invalid replacement events had already been registered during calendar creation. Because the same weakness remained represented as a failure-pattern arm, it was retargeted at Iteration 25. The second repair moved the executor hook to calendar-event pre-registration, allowing invalid replacements to be blocked before creation while retaining the three valid replacements; the scenario then passed. In the scenario-arm variant, 4bytkx was also successfully repaired at Iteration 31, but this repair was later lost as subsequent harness updates accumulated. The scenario-level representation maintained 75 independent arms, more than twice the maximum of 35 failure-pattern arms, making individual scenarios less likely to be revisited under the same optimization budget. Moreover, a scenario 22

Preprint

arm identifies an executable instance but does not explicitly represent which weakness or repair in the current harness that scenario corresponds to. After the repair disappeared, 4bytkx was not re-executed for more than ten iterations, leaving the scorer without fresh evidence about its status under the changed harness. It instead concluded that “This arm was recently fixed,” causing its severity to drop from 0.80 to 0.15 and its rank from 6th to 52nd. This deprioritization further reduced the likelihood of revisiting the scenario, and it was not selected again before optimization ended. Together, these cases expose complementary limitations of predefined arm definitions. Category arms are too coarse: heterogeneous weaknesses share a single optimization state, so successes on unrelated scenarios can obscure a weakness that remains unresolved. Scenario arms are precise in identifying executable instances, but their optimization state is fragmented across many arms and does not explicitly encode the relationship between a scenario, its underlying weakness, and the repairs retained in the current harness. Under a limited optimization budget, this makes frequent re-observation difficult and can leave arm scores based on stale evidence as the harness evolves. Failure-pattern arms instead represent observed harness weaknesses as explicit optimization targets, providing a granularity that is specific enough to isolate repair opportunities while compact enough to keep their status trackable. This allows unsuccessful or subsequently lost repairs to remain available for retargeting and further refinement.

Failure Hit Rate (%)

100 80 60 40

Failure Pattern Category Scenario 60.4 52.9 36.4

30.5 17.6

20 0

36.6

GAIA2

TB2

Figure 8: Failure hit rate by arm definition. Fraction of optimization iterations whose sampled mini-batch contains at least one scenario with an unresolved failure. Budgets are 1,400 rollouts for GAIA2 and 490 for TB2. Failure Hit Rate. Figure 8 measures how often each arm definition directs an optimization iteration toward a scenario whose failure remains unresolved. A scenario is considered unresolved if it most recently failed or if a repair that previously made it pass is no longer retained in the current harness. An iteration is a failure hit when its sampled mini-batch contains at least one such scenario, and the failure hit rate is the fraction of iterations that satisfy this condition. We use equal budgets of 1,400 rollouts on GAIA2 and 490 rollouts on TB2. On GAIA2, failure-pattern arms achieve a failure hit rate of 52.9%, compared with 36.4% for category arms and 17.6% for scenario arms. On TB2, failure-pattern arms achieve 60.4%, compared with 30.5% for category arms and 36.6% for scenario arms. Failure-pattern arms are induced from observed failures, so their pulls directly target weaknesses the harness has already exhibited. Category arms instead mix multiple scenarios under one label, while scenario arms spread optimization across the full training set. As a result, failure-pattern arms devote a larger share of the optimization budget to weaknesses that remain unresolved as the harness evolves.

23

Preprint

H

D ISCOVERED FAILURE PATTERNS Table 9: Failure patterns induced by ActiveSaddler on GAIA2 ID

Iter.

Induced Failure Pattern

Supporting Scenarios

1

1

No-attachment path defaults serialize as null instead of the oracle’s empty path string

2

1

3

3

4

4

5

4

6

4

7

6

8

6

9

7

10 11

10 10

12

15

13 14

15 20

15

21

16

21

17

24

18

26

19

26

20 21 22

26 28 28

23 24

28 29

25

33

26

34

27

35

28

40

29 30

40 40

31

42

32

42

33

42

34

42

35

46

Over-specific event notification wording diverges from concise oracle-style message content Affirmative event-start email reply content is not canonicalized to the required acknowledgement structure Multi-ambiguity user clarifications omit explicit questions for all unresolved decision dimensions Time-sensitive repeated actions drift against deadlines and stop before the required count is complete Lookup tools reject raw identifiers that downstream action tools accept, causing resolution loops in deadline-bound tasks Partial filesystem discovery causes incomplete multi-file deletion for filename-prefix tasks Incomplete source enumeration before cross-app aggregation produces wrong numeric answers Relative same-time-tomorrow reschedules anchor to the referenced event date instead of the task receipt date Ambiguous contact-dependent state changes execute before identity clarification Duplicate first-name contact clarification uses downstream app matches instead of source contact ambiguity Missing target event replacement ambiguity produces over-broad clarification messages Explicit separate-email requests are collapsed into one multi-recipient email event Property-viewing confirmation emails omit canonical subject and full property address details Picky-recipient gift shopping tasks add cart items before resolving ambiguous product variants Direct personal gift-notification messages omit required recipient first-name greeting before event registration Batch event replacement scheduling fails to partition ambiguous and unambiguous dates Ambiguous existing calendar event references are guessed and mutated instead of deferred for clarification Email composition drops one purpose from compound thank-you-and-materials requests Unresolved ambiguity in one subtask blocks independent required actions Unrequested optional calendar metadata on indirectly described events Premature notification waiting blocks required user update at multi-phase task boundaries Trace-less execution timeout in broad multi-step workflows Follow-up availability notifications trigger unrequested reply confirmations in multi-phase workflows Notification-summary reliance misses authoritative same-thread follow-up state before conditional downstream actions Premature downstream actions before required checkpoint notifications in conditional workflows Global shopping option-term narrowing undercounts per-product variant ambiguity in clarification prompts Duplicate-contact clarification canonicalization drops full-name options and task-specific action context Filtered save-only selection is misread as permission to remove existing saved items Stale contact addresses are trusted for location-dependent service booking despite relocation uncertainty Correct cab-conflict handling leaves replacement ride writes with non-canonical location and exact-time arguments Phase-blind write canonicalization corrupts preliminary multi-phase scheduling outreach before reply context exists Schedule proposal selection ignores same-category calendar commitments when applying availability constraints Descriptor-based contact lookup lacks profile-field coverage and triggers fragile downstream message composition Timed reply-gathering tasks proceed before the collection window is complete

8hgfug, aa11lh, tnxtee 8hgfug

24

tnxtee 2qw4nk, qrxrry 71j6lf 71j6lf xcv49b anc8y4, bv0gwh, m1momj, mp5r62 2qw4nk dpbafp, rkdpfj dpbafp, rkdpfj b6v3y8 b6v3y8 rkdpfj kao12j kao12j 4bytkx lxe62a lxe62a ja2axd 8c9am2 8c9am2, fqdxfe 3rljy6 fqdxfe fqdxfe fqdxfe kao12j dpbafp dpbafp r4pl5k 096dyu i6hy8j i6hy8j n9ewlm tf35kd

Preprint

Table 10: Failure patterns induced by ActiveSaddler on Terminal-Bench 2.0 ID

Iter.

Induced Failure Pattern

Supporting Scenarios

1

1

overfull-hbox

2

1

3

2

4

4

5

4

6

4

7

6

8

9

Premature completion on constrained file edits without rule-aware final validation Long-running probabilistic computation lacks staged dependency setup, diagnostic runs, and progress-aware output validation Security sanitizer tasks finalize after weak smoke tests without adversarial and clean-preservation validation Binary extraction solutions finalize after superficial output smoke tests without independent address-space convention validation Agent self-validation leaves stale runtime artifacts that change verifier timing and hide required fresh-run output Completion-only validation gates do not help long build/debug tasks that time out before attempting finalization Missing role-by-role exact source-component validation for multi-part generated artifacts Superficial VM/frame validators accept nonsemantic dummy outputs as success

9

10

10

10

11

12

12

16

13

23

14

23

15

41

Unbounded manual ML artifact creation lacks executable training and metric validation gates VM/frame validators conflate runtime display geometry with saved artifact geometry Temporal leaderboard queries use current live data with date filters instead of reconstructing historical snapshots Missing bounded decoder and completion gate for text embedded in structured geometry artifacts Oversized inline setup payloads exceed exec argument limits before the task loop

Transient validation artifacts left in verifier-checked output directories violate clean artifact contracts Unsafe top-level shell exits in persistent terminal sessions cause harness-level crashes before recovery

mcmc-sampling-stan filter-js-from-html extract-elf make-doom-for-mips make-doom-for-mips protein-assembly make-doom-for-mips, make-mips-interpreter train-fasttext make-doom-for-mips, make-mips-interpreter mteb-leaderboard gcode-to-text bn-fit-modify, constraints-scheduling, gcode-to-text, polyglot-c-py polyglot-c-py make-doom-for-mips

Table 9 and Table 10 present the failure-pattern arms discovered during the GAIA2 and TerminalBench 2.0 optimization runs, respectively. These patterns are induced from diagnosed execution failures during optimization rather than predefined by a failure taxonomy. For each pattern, we report its discovery iteration and the set of supporting scenarios. The induced representation exhibits a many-to-many relationship between scenarios and failure patterns. Some patterns consolidate the same weakness across multiple scenarios (e.g., GAIA2 P8 and TB2 P13 each span four scenarios/tasks), while a single scenario can expose multiple distinct patterns (e.g., dpbafp in GAIA2 and make-doom-for-mips in TB2). Appendix K details the two-stage extraction and normalization procedure. The complete SKILL.md files for symptom-extract and symptom-normalize are provided in the anonymous repository linked in the Reproducibility Statement.

25

Preprint

65

0.75

60

0.50

55

0.25

50

0.00

0

12

Priority ϕ Arm pulled

24

Patch accepted No patch: all passed

36

Dev accuracy (%)

Priority ϕ

P3 arm 1.00

48

Dev accuracy (shaded) Best dev so far

Figure 9: Case study of a resolved weakness (P3 of GAIA2). After an accepted patch at Iteration 5, the arm’s priority drops sharply. A later pull finds all supporting scenarios successful and produces no further patch, after which the arm remains at low priority.

65

0.75

60

0.50

55

0.25

50

0.00

0

12

Priority ϕ Arm pulled Scenario joins the pattern

24

Patch accepted No patch: all passed

36

Dev accuracy (%)

Priority ϕ

P8 arm 1.00

48

Dev accuracy (shaded) Best dev so far

Figure 10: Case study of a recurring weakness (P8 of GAIA2). Early pulls produce no patch because the associated scenarios succeed when re-executed, yet the arm remains highly prioritized. The same failure pattern later appears in additional scenarios, providing new evidence that the weakness remains active; a subsequent pull produces an accepted patch. Cross markers indicate scenarios newly associated with P8.

I

A DDITIONAL A NALYSIS FOR RQ2

Case Studies. To illustrate how LLM-based arm scoring responds to evolving optimization evidence, Figure 9 and Figure 10 show two representative failure-pattern arms with contrasting trajectories. For P3, the first pull at Iteration 5 produces an accepted patch, after which the arm’s priority drops sharply. A later revisit finds that all supporting scenarios pass and requires no additional patch, providing further evidence that the corresponding weakness has been resolved. The arm consequently remains at low priority, allowing optimization budget to shift toward other weaknesses. P8 exhibits the opposite behavior. Its first three pulls produce no patch because the associated scenarios pass when re-executed, but these transient successes provide limited evidence that the underlying weakness has been resolved: agent executions are stochastic, and no patch directly addressing the failure pattern has yet been applied. Accordingly, the scorer continues to assign substantial priority to P8. The same failure pattern subsequently appears in additional scenarios (Figure 10), strengthening the evidence that the weakness remains relevant beyond its originally observed cases. P8 is eventually selected again, and the resulting accepted patch increases the best development-set accuracy from 61.5% to 63.1%. Together, these cases illustrate how arm scoring distinguishes evidence of resolution from evidence of continued relevance. A successfully repaired weakness is deprioritized, whereas a recurring weakness can retain or regain priority as new evidence accumulates. The following analyses examine how this evidence-dependent reprioritization manifests across the full arm pool. 26

Live arms

Preprint

35 6 P8

P8

P8

P10

Pull probability

1.00 0.75 0.50 0.25 0.00

5

10

15

20

25

30

Optimization iteration

35

40

45

50

P10 Ambiguous contact-dependent state changes execute… P8 Incomplete source enumeration before cross-app… P4 Multi-ambiguity user clarifications omit explicit… P2 Over-specific event notification wording diverges… P5 Time-sensitive repeated actions drift against… Scenario joins the arm (colour = arm)

Other arms (up to 25) P26 Premature downstream actions before required… P23 Trace-less execution timeout in broad multi-step… P24 Follow-up availability notifications trigger… P15 Picky-recipient gift shopping tasks add cart… P12 Missing target event replacement ambiguity…

Figure 11: Evolution of pull-probability mass across failure-pattern arms. The ten arms with the highest average pull probability over the run are shown individually, while the remaining 25 arms are aggregated into Other arms. The upper panel shows the number of instantiated arms, which grows from 6 to 35 as new failure patterns are discovered. Pull probabilities are shown only at the 34 iterations where arm selection is performed.

Evolution of Arm Priorities. To further examine the curriculum-level dynamics summarized in Figure 3a, we analyze the pull-probability trajectories of all failure-pattern arms throughout the GAIA2 optimization run. Arm scores and pull probabilities are recomputed only when the curriculum selects an instantiated arm for optimization. Accordingly, Figure 11 covers the 34 arm-selection iterations between Iterations 1 and 51; iterations without arm selection are omitted. These decisions are made over a continually evolving target space: the number of instantiated failure-pattern arms grows from 6 to 35 as previously unseen failure patterns are discovered. Thus, the curriculum must continually reassess priorities while both the harness and the set of observed weaknesses evolve. We first examine how probability mass is distributed according to the latest observed outcomes of each arm’s supporting scenarios. As optimization proceeds, previously observed weaknesses may be addressed such that the scenarios supporting an arm subsequently succeed when re-evaluated. At Iteration 51, 24 of the 35 arms (68.6%) have had all of their supporting scenarios re-evaluated after association with the arm, with every such scenario succeeding in its latest valid evaluation. Despite comprising more than two thirds of the arm pool, these arms account for only 26.2% of the total pull-probability mass. For reference, under uniform allocation, they would receive 68.6% of the probability mass, making their observed share only 0.38× this uniform baseline. The other 11 arms include at least one supporting scenario without a subsequent successful evaluation and together receive 73.8% of the probability mass, compared with 31.4% under uniform allocation, or 2.35× their uniform share. These results indicate that, as optimization progresses and previously observed weaknesses are successfully addressed on their supporting scenarios, their corresponding arms tend to receive progressively less pull probability. However, arm priorities can also increase when new evidence indicates that a previously observed weakness remains active. The cross markers for P8 and P10 in Figure 11 indicate iterations at which a failure pattern already represented by an existing arm is observed again in an additional scenario, which is then added to that arm’s supporting scenario set. For P8, these support-set expansions are followed by an average increase of 4.8 percentage points in pull probability, while the corresponding P10 event is followed by a 9.4 percentage-point increase. These cases show that observing the same failure pattern in additional scenarios can increase the priority of an existing arm by providing new evidence that the corresponding weakness remains relevant. Taken together, these observations show that the priority assigned to a failure-pattern arm changes with the evidence accumulated about the corresponding weakness. Arms receive less probability as 27

No-attachment path defaults serialize… P1 Over-specific event notification… P2 Affirmative event-start email reply… P3 Multi-ambiguity user clarifications… P4 Time-sensitive repeated actions drift… P5 Lookup tools reject raw identifiers… P6 Partial filesystem discovery causes… P7 Incomplete source enumeration before… P8 Relative same-time-tomorrow reschedules… P9 Ambiguous contact-dependent state… P10 Duplicate first-name contact… P11 Missing target event replacement… P12 Explicit separate-email requests are… P13 Property-viewing confirmation emails… P14 Picky-recipient gift shopping tasks add… P15 Direct personal gift-notification… P16 Batch event replacement scheduling… P17 Ambiguous existing calendar event… P18 Email composition drops one purpose… P19 Unresolved ambiguity in one subtask… P20 Unrequested optional calendar metadata… P21 Premature notification waiting blocks… P22 Trace-less execution timeout in broad… P23 Follow-up availability notifications… P24 Notification-summary reliance misses… P25 Premature downstream actions before… P26 Global shopping option-term narrowing… P27 Duplicate-contact clarification… P28 Filtered save-only selection is misread… P29 Stale contact addresses are trusted for… P30 Correct cab-conflict handling leaves… P31 Phase-blind write canonicalization… P32 Schedule proposal selection ignores… P33 Descriptor-based contact lookup lacks… P34 Timed reply-gathering tasks proceed… P35

0.5

0.4

0.3

Pull probability

Arms, oldest at the top

Preprint

0.2

0.1

5 7 8 9 11 12 13 14 16 17 20 22 23 25 27 30 31 32 33 34 35 36 37 38 39 41 43 44 45 47 48 49 50 51

0.0

Optimization iteration

Arm pulled at this iteration

Arm not yet in the pool

Figure 12: Pull-probability trajectories for all failure-pattern arms. Rows show all 35 discovered arms ordered by instantiation time, and columns correspond to the 34 iterations in which arm selection is performed. Cell intensity denotes pull probability, orange markers indicate the arm sampled at each iteration, and gray cells denote iterations before an arm entered the pool.

their supporting scenarios subsequently succeed, while new observations of the same failure pattern across additional scenarios can increase their priority again. This evidence-dependent prioritization explains the continual redistribution of probability mass across failure-pattern arms observed in Figure 11.

Stochastic Selection over the Expanding Arm Pool. Figure 12 complements the aggregate probability-mass view in Figure 11 by showing the pull probability and realized selection of every failure-pattern arm at each arm-selection step. Across 34 pulls, the sampler selects 19 of the 35 discovered failure patterns. The selected arm has a median priority rank of third, with 62% of pulls drawn from the top three arms, while selections extend as far as rank 20. Thus, the stochastic selection rule strongly favors arms assigned high current priority without reducing the curriculum to greedy selection of the highest-scoring arm. Lower-ranked arms retain opportunities to be sampled as their relative priorities change over optimization. The gray staircase also makes explicit that these selections are made over an arm pool that expands throughout the run. Consequently, an arm’s priority is always relative to the set of weaknesses known at that point in optimization rather than to a fixed collection of targets defined in advance. As new failure patterns enter the pool and the evidence associated with existing arms changes, the set of arms receiving substantial probability and the arms actually selected both change over time. Together with the probability-mass analysis in Figure 11, this shows that LLM-based arm scoring produces a focused but non-greedy curriculum: most optimization opportunities are directed toward currently high-priority weaknesses, while stochastic selection preserves the possibility of targeting other arms as their relative optimization value evolves. 28

Active Arms (Pull / Draw)

Preprint

8

ActiveSaddler w/o Exploration Controller 6.9×

6 4 2 0

2.0×

1.3×

0.8×

TB2

GAIA2

Figure 13: Dependence of exploration decisions on unresolved weaknesses. Ratio of the average number of unresolved arms at P ULL to D RAW decisions. Values above one indicate more unresolved weaknesses at P ULL, while values near one indicate little difference between the two decision types.

J

A DDITIONAL A NALYSIS FOR RQ3

Dependence of Exploration Decisions on Unresolved Weaknesses. To quantify whether exploration decisions respond to unresolved weaknesses, we compare how many arms remain unresolved at P ULL and D RAW decisions. An arm is considered unresolved when its associated weakness remains unsolved under the current harness. We report the ratio of the average number of unresolved arms at P ULL to that at D RAW: values above one indicate that P ULL tends to occur when more unresolved weaknesses remain, whereas values near one indicate little difference between the two decision types. As shown in Figure 13, ActiveSaddler has 2.0× as many unresolved arms at P ULL as at D RAW decisions on GAIA2 (1.62 vs. 0.81) and 6.9× as many on Terminal-Bench 2.0 (0.76 vs. 0.11). In contrast, the fixed exploration schedule yields ratios close to one: 0.8× on GAIA2 (0.21 vs. 0.25) and 1.3× on Terminal-Bench 2.0 (1.14 vs. 0.86), indicating that its P ULL and D RAW decisions are largely insensitive to how many unresolved arms remain. Thus, ActiveSaddler tends to continue repair when more unresolved weaknesses remain and to explore when fewer remain, whereas the fixed schedule shows little such dependence on the current optimization state.

K

I NSTRUCTIONS U SED I N ACTIVE S ADDLER

To make the curriculum decisions in ActiveSaddler reproducible and interpretable, we provide the prompt templates used by the three curriculum-specific LLM components introduced in the main method: the Failure-Pattern Extractor, Arm Prioritizer, and Exploration Controller. Each prompt provides the optimization context required for its corresponding decision and constrains the agent to a focused analysis procedure. The Failure Pattern Extraction Prompt (Figure 14) converts failed execution evidence into reusable failure-pattern arms. It first abstracts each failure independently into a candidate symptom without consulting the existing arm set, and then normalizes the extracted candidates by linking them to existing atomic patterns, composing multiple patterns when needed, or instantiating new arms. The Arm Scoring Prompt (Figure 15) implements the Arm Prioritizer by estimating a learningprogress score ϕt (a) for every instantiated failure-pattern arm. Using the latest optimization history, the agent assesses severity, fixability, breadth, and side-effect risk. The resulting scores are used by the stochastic arm-selection rule to prioritize the next optimization target. The Unseen Scenario Exploration Prompt (Figure 16) implements the Exploration Controller. Given the current optimization context, the agent decides whether to continue optimizing a known failure-pattern arm or explore scenarios that may reveal previously undiscovered weaknesses, based on whether a worthwhile unresolved target remains among the known arms and the remaining value of further exploration.

29

Preprint

Failure Pattern Extraction Prompt ## Mandatory Skills You MUST read and follow the SKILL.md for each skill listed below. Do NOT skip or summarize any skill -- execute the full procedure described in each one. These skills are installed at ‘.claude/skills/<name>/SKILL.md‘ in the current worktree. | Order | Skill | When | Why | |-----|-----|----|---| | 1 | ‘symptom-extract‘ | For each failed scenario | Symptom extraction from diagnosis/reflection results -- generates candidate symptoms | | 2 | ‘symptom-normalize‘ | After extracting all candidates | 3-way normalization against existing patterns -- links, composes, or creates patterns to maintain atomic pattern integrity | > Enforcement: You MUST process every failed scenario. Do NOT skip the extraction step for any failed scenario. During extraction (Skill 1: ‘symptom-extract‘), do NOT examine existing patterns; consult them only during normalization (Skill 2: ‘symptom-normalize‘). This separation is required to avoid anchoring bias. ## Goal Extract failure patterns from diagnosis and reflection results, then tag each failed (harness, trace, scenario) tuple with the identified pattern(s). Failure patterns are symptom-level abstractions -- more abstract than individual root causes, but specific enough to distinguish different failure types. They are the arms of the curriculum sampler. At each outer-loop iteration, each arm (i.e., failure pattern) is assigned a learning-progress score estimating the expected improvement in the current harness’s population performance that would result from selecting the arm and successfully addressing its corresponding failure pattern in the next optimization step. ## Context - Iteration: {iteration} - Candidate: C{candidate_idx} - Current worktree: ‘{worktree_path}‘ - Session root: ‘{session_root}‘ - Pre-patch trace dir: ‘{before_output_dir}‘ - Post-patch trace dir: ‘{after_output_dir}‘ ## Failed Scenarios to Process ### Pre-patch failures (from diagnosis) These scenarios failed BEFORE any patch was applied. Each entry provides the batch-level diagnosis-and-patching step diagnosis and detailed reasoning-file location from before patching, followed by the reviewed underlying root cause and post-hoc analysis of how the patch affected that cause. {pre_patch_failures} ### Post-patch failures (from reflection) These scenarios STILL FAILED or REGRESSED after the patch was applied. Each entry provides the status change, the batch-level patch approach, and the reviewed root cause and post-hoc patch-effect explanation. For ‘still_failing‘, the root cause is the remaining or revised failure cause; for ‘regressed‘, it is the newly observed regression cause or failure mechanism. {post_patch_failures} ## Workflow ### 1. Extract candidate symptoms For each failed scenario listed above: 1. For pre-patch failures, review the diagnosis/reasoning and reflection step’s reviewed cause and patch-effect analysis. For post-patch failures, also

30

Preprint

(continued) use the status change and patch approach to distinguish a remaining cause from a newly introduced regression mechanism. 2. Follow the ‘symptom-extract‘ skill to generate a candidate symptom label. Do NOT look at existing patterns during this step. 3. Record every candidate symptom and its associated root cause in ‘symptom_candidates.md‘ at the working directory root. Complete ‘symptom_candidates.md‘ for all failures before moving to normalization. ### 2. Normalize against existing patterns After all candidate symptoms are extracted: 1. Read ‘symptom_candidates.md‘ from the working directory root. 2. Run ‘pattern list‘ to retrieve all existing patterns. 3. For each candidate symptom in the file, follow the ‘symptom-normalize‘ skill to perform a 3-way judgment: - (a) Same: the candidate matches an existing pattern → link to it - (b) Composition: the candidate is a combination of existing atomic patterns → tag with multiple existing patterns - (c) New: genuinely novel failure type → register as a new pattern ### 3. Tag tuples For each resolved pattern assignment: 1. Register new patterns if needed: “‘ pattern register -label "DESCRIPTIVE SYMPTOM LABEL" “‘ 2. Tag the (harness, trace, scenario) tuple: “‘ pattern tag -pattern-id <id> -harness {candidate_idx} -trace <dir> -scenario <sid> -root-cause "..." “‘ For compositions, repeat ‘-pattern-id‘ for each atomic pattern. > IMPORTANT: Always use ‘-harness {candidate_idx}‘ (the current > candidate) for ALL tagging commands -- both pre-patch and post-patch > failures. The ‘-trace‘ directory distinguishes before/after. The > harness identifies which iteration this observation belongs to. > Note: The failure-pattern extraction step only needs to execute ‘pattern register‘ and ‘pattern tag‘. The agent does not need to calculate or record observation values separately. For each previously known pattern, the outer loop automatically calls ‘PatternRegistry.observe()‘ with the scenarios associated with that pattern and evaluated in the mini-batch, the subset still tagged after patching, and the resulting activity fraction. ## Important Notes - Root cause is preserved as evidence, not used directly as the pattern. The pattern label should be an abstraction that generalizes across multiple scenarios. - Patterns should be atomic -- a single patch should be able to fix one pattern. Complex failures are expressed as compositions (multi-tagging). - Do NOT over-merge: if two failures have different root causes, they should be different patterns even if they look superficially similar. - Do NOT over-split: if the same behavioral issue manifests with minor surface variations, it’s the same pattern.

Figure 14: Failure Pattern Extraction Prompt: the agent independently abstracts failed execution evidence into candidate symptoms and then normalizes them against the existing arm set by linking to existing atomic patterns, composing multiple patterns when needed, or instantiating new failurepattern arms.

31

Preprint

Arm Scoring Prompt ## Mandatory Skills You MUST read and follow the SKILL.md for each skill listed below. Do NOT skip or summarize any skill -- execute the full procedure described in each one. These skills are installed at ‘.claude/skills/<name>/SKILL.md‘ in the current worktree. | Order | Skill | When | Why | |-----|-----|----|---| | 1 | ‘history-analysis‘ | Before any scoring (Step 1) | Understand the Prepared harness and evolution history | | 2 | ‘progress-scoring‘ | For every candidate pattern (Steps 3-5) | Estimate each arm’s cardinal learning-progress score | ## Goal For every candidate failure-pattern arm, estimate a cardinal learning-progress score ϕ ∈ [0, 1] representing the expected improvement in the prepared harness’s population performance from allocating the next optimization step to that arm. The estimate should reflect the likely outcome of executing the arm’s supporting scenarios and attempting to address the harness weakness represented by the pattern, including both the potential performance gain and the probability of obtaining a broadly effective, dev-preserving update. Base each score on the prepared harness, the arm’s complete execution and optimization history, prior dev-set outcomes, and the available evidence about the pattern’s severity, fixability, breadth, and risk of side effects. Record every cardinal score with ‘pattern rate‘. This is a read-only analysis session: do NOT modify the codebase. Only inspect the harness, failure patterns, supporting scenarios, and optimization history, then record the scores. ## Context - Iteration: {iteration} - Prepared harness: C{prepared_candidate_idx} - Prepared worktree: ‘{prepared_worktree_path}‘ - Prepared commit: ‘{prepared_commit}‘ - Provisional parent: C{provisional_parent_idx} (‘{provisional_parent_commit}‘) - Selection JSON: ‘{selection_session_json_path}‘ - Session root: ‘{session_root}‘ ### Prepared Harness Summary {prepared_harness_summary} ## Candidate Failure Patterns (arms to score) Score every pattern below. {candidate_patterns} ### Candidate Table Columns - ‘Pattern‘: Stable failure-pattern ID used by the ‘pattern‘ and ‘evo-dag‘ CLIs. - ‘Activity‘: Rested EMA of the raw observations, seeded at ‘1.00‘. This is a mechanical severity reference, not the Agent learning-progress score ϕ. - ‘Observations‘: Complete per-iteration observation history. Each entry uses ‘iteration:tagged/evaluated (active=...), tagged=[...], evaluated=[...]‘. ‘evaluated‘ lists the scenarios associated with this pattern that were evaluated in that iteration’s mini-batch, and ‘tagged‘ lists the subset that was still tagged with the pattern after patching. ‘active‘ is the corresponding

32

Preprint

(continued) ‘tagged/evaluated‘ fraction. For legacy observations whose scenario IDs were not recorded, the counts appear as ‘?/¿ and both lists as ‘(not recorded)‘; ‘(none)‘ means the arm has not been selected again since the pattern was most recently observed in a scenario, so no subsequent activity observation has been recorded. - ‘#Scen‘: Number of unique scenarios currently tagged with the pattern. - ‘LastObs‘: Most recent iteration in which the arm was selected and an activity observation was recorded, or ‘0‘ if the arm has not been selected again since the pattern was most recently observed in a scenario. - ‘Label‘: Symptom-level failure-pattern description. - ‘Scenarios‘: Complete list of scenario IDs currently owned by the arm. ## Workflow ### 1. Understand the current harness (history-analysis) Run the ‘history-analysis‘ skill. Inspect the current prepared node with ‘evo-dag show node {prepared_candidate_idx}‘ and its provisional parent with ‘evo-dag show node {provisional_parent_idx}‘. Read the actual prepared-harness change with ‘git diff {provisional_parent_commit} {prepared_commit}‘. You need this context to assess all four scoring axes: *severity* (whether the failure pattern is likely still active under the prepared harness), *fixability* (whether and where a patch can address it), *breadth* (how broadly a successful repair may generalize), and *side-effect risk* (which components or behaviors a patch may affect or regress). ### 2. Review prior pulls for each candidate arm Read the histories of all arms that have previously been selected: “‘bash pattern history “‘ With no ‘-pattern-id‘, the command shows every historically pulled arm. If the full output is too large to review efficiently, you may optionally narrow it to a subset in one call with ‘pattern history -pattern-id <id1> <id2>‘, or use ‘-last-k <K>‘ to show only the K most recent entries for each selected arm during initial triage. These options are convenience filters, not substitutes for complete review: inspect an arm’s full history before rating it whenever older attempts could change your judgment. The output distinguishes patched attempts, all-pass skips, and failed attempts and includes diagnoses, patch intents, dev-set impact, and per-scenario reflections. ### 3. Inspect each candidate pattern For every pattern in the table: - ‘pattern show <pattern_id>‘ -- label, tagged scenarios, evidence (root causes). - ‘evo-dag show scenario <sid>‘ -- for each of the pattern’s scenarios, read its failure history and relate it to the current harness change. ### 4. Estimate learning progress (progress-scoring) Follow the ‘progress-scoring‘ skill. For each pattern, judge the four considerations (severity, fixability, breadth, side-effect), then rate all four axes carefully in [0, 1]. The final score ϕ is the mean of those four axes with side-effect inverted -‘(severity + fixability + breadth + (1 - side_effect)) / 4‘. Your per-axis ratings (not a single holistic judgment) determine the result. **Use the per-arm pull history as primary evidence, and weigh the dev-set, not just the mini-batch.** A scenario merely passing the mini-batch in a recent pull does NOT by itself mean the pattern is resolved -- learning progress is measured on the dev/held-out distribution: - If a prior patch fixed the scenario but dropped dev-set accuracy (so it was

33

Preprint

(continued) rejected and reverted), a better dev-preserving fix is still needed → progress may remain high. - If the scenario is genuinely resolved (passing with no dev headroom left), or its failure is intrinsically unfixable without regressing dev → progress is low. - If recent pulls show the scenarios repeatedly passing, lower the score, **but do not force it to 0** -- the pass may be intermittent, and the sampler’s minimum-probability floor still re-checks low-scored arms periodically to catch silent regressions. ### 5. Record every score > Reminder: ϕ is computed from your four axes (see step 4). ‘-side-effect‘ is a RISK -- higher means worse. For each candidate pattern: “‘bash pattern rate -pattern-id <id> \ -severity <s> -fixability <f> -breadth <b> -side-effect <e> \ -rationale "..." “‘ CRITICAL: rate every pattern in the table before finishing. An un-rated pattern is treated as low priority (score 0.0) -- if you skip one it gets deprioritized rather than reflecting your judgment, so do not leave any un-rated. ## What Happens Next The outer loop samples one pattern with probability proportional to ‘softmax(ϕ / τ )‘ and sends its scenarios to the diagnosis-and-patching step.

Figure 15: Arm Scoring Prompt: the agent estimates a learning-progress score ϕt (a) for every instantiated failure-pattern arm by considering severity, fixability, breadth, and side-effect risk. The resulting scores define the stochastic prioritization of known arms. Unseen Scenario Exploration Prompt ## Mandatory Skills You MUST read and follow the SKILL.md for each skill listed below. Do NOT skip or summarize the skill -- execute the full procedure described in it. This skill is installed at ‘.claude/skills/<name>/SKILL.md‘ in the current worktree. | Order | Skill | When | Why | |-----|-----|----|---| | 1 | ‘history-analysis‘ | Before the decision (Step 1) | Understand the Prepared harness and evolution history | ## Goal Decide whether this iteration should exploit a known failure-pattern arm (PULL) or explore unseen scenarios that may reveal a new failure type (DRAW). Record exactly one decision with ‘pattern decide‘. This is an exploration-exploitation trade-off in arm space: - PULL spends this iteration on an already-discovered pattern. arm-scoring step then scores every known pattern so the sampler can choose an arm to repair. - DRAW probes scenarios that have never been executed. This may reveal a failure pattern not yet represented by the known arms; Arm scoring is skipped.

34

Preprint

(continued) There is no fixed formula or threshold for this decision. Do not decide mechanically from ‘|P_t|‘, ‘|U_t|‘, Activity, or any single statistic. Weigh all available evidence and make one holistic decision. Here, ‘P_t‘ is the set of failure patterns discovered by iteration ‘t‘, and ‘U_t‘ is the set of scenarios still unseen at that iteration; ‘|...|‘ denotes the size of a set. This is a read-only analysis session: do NOT modify the codebase and do NOT rate arms. ## Context - Iteration: {iteration} - Prepared harness: C{prepared_candidate_idx} - Prepared worktree: ‘{prepared_worktree_path}‘ - Prepared commit: ‘{prepared_commit}‘ - Provisional parent: C{provisional_parent_idx} (‘{provisional_parent_commit}‘) - Selection JSON: ‘{selection_session_json_path}‘ - Session root: ‘{session_root}‘ - Discovered failure patterns: |P_t| = {num_arms} - Unseen scenarios remaining: |U_t| = {unseen_pool_size} ### Prepared Harness Summary {prepared_harness_summary} ## Candidate Failure Patterns (known arms) Use the patterns below to judge whether a worthwhile known target exists. {candidate_patterns} ### Candidate Table Columns - ‘Pattern‘: Stable failure-pattern ID used by the ‘pattern‘ and ‘evo-dag‘ CLIs. - ‘Activity‘: Rested EMA of the raw observations, seeded at ‘1.00‘. This is a mechanical severity reference, not the Agent learning-progress score ϕ. - ‘Observations‘: Complete per-iteration observation history. Each entry uses ‘iteration:tagged/evaluated (active=...), tagged=[...], evaluated=[...]‘. ‘evaluated‘ lists the scenarios associated with this pattern that were evaluated in that iteration’s mini-batch, and ‘tagged‘ lists the subset that was still tagged with the pattern after patching. ‘active‘ is the corresponding ‘tagged/evaluated‘ fraction. For legacy observations whose scenario IDs were not recorded, the counts appear as ‘?/¿ and both lists as ‘(not recorded)‘; ‘(none)‘ means the arm has not been selected again since the pattern was most recently observed in a scenario, so no subsequent activity observation has been recorded. - ‘#Scen‘: Number of unique scenarios currently tagged with the pattern. - ‘LastObs‘: Most recent iteration in which the arm was selected and an activity observation was recorded, or ‘0‘ if the arm has not been selected again since the pattern was most recently observed in a scenario. - ‘Label‘: Symptom-level failure-pattern description. - ‘Scenarios‘: Complete list of scenario IDs currently owned by the arm. ## Workflow ### 1. Understand the current harness (history-analysis) Run the ‘history-analysis‘ skill. Inspect the current prepared node with ‘evo-dag show node {prepared_candidate_idx}‘ and its provisional parent with ‘evo-dag show node {provisional_parent_idx}‘. Read the actual prepared-harness change with ‘git diff {provisional_parent_commit} {prepared_commit}‘. Use this context to judge whether known-arm evidence still applies to the

35

Preprint

(continued) prepared harness and whether its weaknesses remain reachable by another patch. ### 2. Review prior pulls for each candidate arm Read the histories of all arms that have previously been selected: “‘bash pattern history “‘ With no ‘-pattern-id‘, the command shows every historically pulled arm. If the full output is too large for initial triage, narrow it with ‘pattern history -pattern-id <id1> <id2>‘ or apply ‘-last-k <K>‘ separately to each selected arm. These are convenience filters, not substitutes for complete review when older attempts could change the PULL/DRAW decision. The output distinguishes patched attempts, all-pass skips, and failed attempts and includes patch approaches, dev-set impact, and per-scenario reflections. ### 3. Inspect promising known patterns Use ‘pattern show <pattern_id>‘ for any pattern that appears impactful. Read its label, tagged scenarios, root-cause evidence, and observations. You do not need to assign a learning-progress score in this session; make a coarse judgment of whether at least one strong known target exists. ### 4. Weigh the PULL/DRAW considerations Use all two lenses below. They guide attention but do not form a formula. 1. Is there a clearly worthwhile known pattern to fix? A severe, plausibly fixable, broad pattern argues for PULL. If known patterns look near-resolved, unreachable, or exhausted by repeated low-yield attempts, diminishing returns argue for DRAW. 2. How complete is coverage of the failure surface? Few discovered arms and a large unseen pool suggest that known patterns do not yet represent the harness’s weaknesses. A small or exhausted unseen pool leaves little discovery value and favors PULL. ### 5. Make and record one decision The discovery value of DRAW is a prediction: unseen scenario contents are not known. Do not over-explore late when a strong fix is available, and do not over-exploit early when the failure surface is poorly mapped. Record exactly one action: “‘bash pattern decide -action pull \ -rationale "why a known pattern is the best use of this iteration" # OR pattern decide -action draw \ -rationale "why discovery is more valuable than fixing known patterns now" “‘ Do NOT call ‘pattern rate‘ in this session. ## What Happens Next - After PULL, arm scoring runs on the same prepared harness and scores every arm. - After DRAW, arm scoring is skipped and the outer loop samples unseen scenarios.

Figure 16: Unseen Scenario Exploration Prompt: the agent decides whether to continue optimizing a known failure-pattern arm or explore scenarios that may reveal new failure patterns by weighing the value of unresolved known targets against the remaining value of further exploration. 36

Preprint

L

L IMITATIONS AND R ESPONSIBLE D EPLOYMENT

ActiveSaddler is a research exploration of automated curriculum learning for offline harness optimization and is not intended or evaluated as a production-ready agent system. Our experiments are conducted entirely in benchmark environments using public and synthetic task data; no real user data are used in the study. Accordingly, the empirical results characterize optimization effectiveness under these controlled settings rather than production safety, security, privacy, or governance readiness. Improved benchmark performance should therefore not be interpreted as evidence that an optimized harness is safe or compliant for unrestricted deployment. This distinction is important for harness optimization because updates may modify prompts, tool interfaces, middleware, and runtime control logic, thereby changing not only task behavior but also how an agent accesses resources and executes actions. Moreover, ActiveSaddler prioritizes optimization opportunities according to observed task failures and the objectives provided by the underlying harness optimizer; it does not by itself determine whether the resulting behavior satisfies independent safety, security, privacy, or policy requirements. Safety and responsible AI. Because ActiveSaddler is evaluated as a research optimization method rather than a production agent system, our experiments do not include dedicated evaluations of agent misuse, cyber-abuse behavior, or other unsafe deployment scenarios. We also do not introduce an independent safety filter or a dedicated mechanism for detecting reward hacking or specification gaming, where an optimizer improves the measured objective without satisfying its intended semantics. Accordingly, the task-level objectives and failure signals used by ActiveSaddler should not be interpreted as a complete specification of acceptable agent behavior. A production deployment would need to augment task-level optimization with independent safety criteria, adversarial misuse and policy-violation testing, and safety regression checks throughout the optimization and release process. Safety-relevant signals should be treated as independent constraints or review criteria rather than inferred solely from improvements in task performance. Security. Our study does not evaluate production security risks such as unsafe command execution, privilege misuse, unauthorized access to sensitive files, or escalation through tools. These risks are particularly relevant to harness optimization because an automatically generated update may modify prompts, tool interfaces, middleware, command construction, tool-selection logic, or runtime control flow even when the underlying model weights remain unchanged. In a production setting, such changes should therefore be evaluated within explicit security boundaries, including least-privilege tool access, sandboxing, sensitive-resource isolation, and adversarial testing of tool and command execution. Updates that modify permissions, exposed capabilities, tool interfaces, or other securitycritical behavior should require explicit review and should not be promoted to production solely on the basis of task-performance improvements. Deployment mechanisms should additionally support rollback when security regressions are detected. Privacy. Because our experiments use public/synthetic benchmarks and do not involve real user data, they do not exercise the privacy risks that can arise when optimization is applied to production execution traces. In real deployments, however, traces used for failure diagnosis and harness optimization may contain personally identifiable information (PII), credentials, proprietary content, or other sensitive data. Such deployments should therefore minimize the data collected for optimization and apply PII and secret detection, sensitive-content filtering, access controls, and anonymization or redaction where appropriate before traces are persisted or consumed by the optimization pipeline. They should also define an explicit, purpose-limited retention period consistent with the applicable data classification and privacy requirements, rather than retaining optimization traces indefinitely. These production privacy controls are outside the scope of the present benchmark study. Governance and deployment controls. Automatic acceptance or promotion of an update by ActiveSaddler should not itself constitute approval for production deployment. Organizations applying automated harness optimization in production should conduct the applicable AI impact and risk assessments and maintain governance processes that are independent of the optimizer’s task-level objective. Depending on the deployment context, these processes may include human-in-the-loop approval for high-impact or security-sensitive changes, auditable records of generated, evaluated, accepted, and rejected updates, staged evaluation before release, explicit safety-review and risk-acceptance checkpoints, and defined rollback procedures when regressions are detected. The required degree of oversight should depend on the capabilities exposed to the agent, the sensitivity of the affected resources, and the potential impact of its actions. More generally, optimizing task performance does not remove the need for deployment-specific safety, security, privacy, and compliance review. 37

Related documents

• ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization#1155841
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses#141551
• LiteEvo: Automated, Cost-Efficient Harness Evolution for Generalization to Unseen Tasks#1117858
• Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable#412818
• Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable#370397