arXiv CS open access

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

2026 · arxiv_cs

arXiv CS · Papers · 2026

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks Fanzhe Meng* , Guoxin Chen* , Jiale Zhao, Shuang Sun, Zhiyu Lin, Wayne Xin Zhao† , Ruihua Song† , † Ji-Rong Wen and Kai Jia

Training terminal agents requires executable and verifiable tasks that are not merely solvable, but appropriately challenging for learning. Executable validation establishes feasibility, yet does not reveal how a task behaves relative to a given solver setting. In this paper, we present CalibForge, an autonomous terminal-task synthesis system that uses verified solver behavior to revise candidate tasks through adversarial solver calibration. Multi-solver calibration targets disagreement within a heterogeneous solver pool, whereas contrastive solver calibration targets a designated strong-pass/weak-fail relation; both operationalize a solver-relative learnable zone anchored in demonstrated solvability. Using CalibForge, we construct 5,431 calibrated terminal tasks. Our ablations show that both strategies yield more effective supervision than authoring and validation alone or ordinary single-solver feedback. Models trained on the full collection achieve 32.58% and 47.57% on Terminal-Bench 2.0. The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. Together, these results support solver-relative learnability as a practical target for constructing effective and transferable agent training data. ∗ Equal Contributions. † Corresponding authors.

Date: Aug. 07, 2026.

(1) Adversarial Solver Calibration Targets the Learnable Zone Contrastive Solver Calibration

Multi-Solver Calibration Learnable Zone

...

too hard / task issue

RETAIN

All Pass

PASS

...

Stronger Solver

All Fail

calibration

arXiv:2608.06352v1 [cs.LG] 6 Aug 2026

1 Gaoling School of Artificial Intelligence, Renmin University of China, 2 Independent Researcher, 3 AweAI Team

too easy

FAIL

Stronger Weaker

Both Pass

Learnable Zone

too easy

Both Fail

Ablation of Solver Feedback and Calibration (Qwen3-30B-A3B-Instruct)

50 40 30

calibration Stronger Weaker

too hard / FAIL task issue

20 10

potential task issue

FAIL Weaker Solver

No Solver

Single Solver

22.47 +1.87 24.34

Multi Solver +6.74 29.21

Contrast Solver +8.6231.09

PASS

(2) Solver-Calibrated Tasks Yield Gains on Terminal and Software-Engineering Benchmarks Terminal-Bench 2.0 Base Model Termigen Qwen3-30B-A3B-Instruct

50 40 30 20 10

+ 24.71

32.58 23.60 26.22 7.87

TerminalTraj CalibForge Qwen3.5-35B-A3B + 8.47

39.10

40.07

40.82

47.57

SWE-Bench Pro (OOD Eval) 50 40 30 20 10

Qwen3-30BA3B-Instruct +27.68

Doc2Repo (OOD Eval)

Qwen3.5-35B-A3B +3.03

41.29

44.32

30.94

3.26

50 40 30 20 10

Qwen3-30BA3B-Instruct +30.04

Qwen3.5-35B-A3B

44.92+3.85 48.77

35.98

5.94

Figure 1. Overview of CalibForge. Top: Multi-solver and contrastive calibration target a solverrelative learnable zone through cross-solver disagreement and a strong-pass/weak-fail relation, respectively; both outperform authoring and validation alone and single-solver feedback on Qwen330B-A3B-Instruct. Bottom: Training on trajectories distilled from 5,431 calibrated tasks improves two Qwen backbones on Terminal-Bench 2.0, SWE-bench Pro, and Doc2Repo.

Emails: {mengfanzhe16, gx.chen.chn, batmanfly, jiakai0419}@gmail.com, songruihua_bloon@outlook.com

Dataset

GitHub

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

1. Introduction Large language model agents increasingly tackle complex computing tasks through terminal interfaces (OpenAI, 2025; Anthropic, 2026; Cursor Research et al., 2026). Recent synthesis systems have made substantial progress in constructing and validating terminal tasks at scale (Zhu et al., 2026; Pi et al., 2026; Gandhi et al., 2026; Tang et al., 2026). Yet executable validity alone does not reveal whether a task is appropriately challenging for learning rather than trivial or effectively unsolvable. Independent attempts by solver agents provide behavioral evidence relevant to this task: their verified outcomes reveal which attempts succeed or fail, while their trajectories help explain why. This raises a natural question: how can such solver behavior guide terminal-task construction toward more effective training data? Our key idea is to turn task construction into a constrained adversarial author–solver loop, in which solver behavior serves as construction-time feedback. Solver agents seek successful completion of the current candidate, while the authoring agent uses the resulting behavioral evidence to revise the task toward a desired outcome pattern. Crucially, a validated candidate is not treated as fixed: the authoring agent may return to external research, reconsider the technical direction, or revise any task component. Each revised version is revalidated and re-probed before it can be retained. The loop is constrained by demonstrated solvability, since the target outcome pattern always requires at least one successful solver attempt. We call this multi-round process environment-level behavioral calibration. We present CalibForge, an autonomous task synthesis system that realizes this process through adversarial solver calibration. Starting from a clue, CalibForge researches concrete engineering problems, jointly authors the task instruction, execution environment, and verification tests, and validates the resulting candidate through structural checks and self-solving. It then applies one of two calibration strategies, using solver outcomes and trajectories to guide multi-round revision and re-probing. Multi-solver calibration targets disagreement within a heterogeneous solver pool, retaining a candidate when at least one solver succeeds and at least one fails. Contrastive solver calibration targets a strong-pass/weak-fail relation, retaining a candidate when a designated stronger solver succeeds and a designated weaker solver fails. Together, these criteria operationalize a solver-relative learnable zone: retained candidates are demonstrably solvable, yet not uniformly solved under the specified solver setting. Our experiments show that whether a candidate lies within a solver-relative learnable zone is neither guaranteed by executable validity alone nor fixed at initial construction. Adversarial solver calibration makes this status observable and actionable: although all candidates entering contrastive calibration have passed structural validation and self-solving, only 19% initially satisfy the target relation, while revision and re-probing increase cumulative acceptance to 96%. Our ablation shows the downstream value of the learnable zone: multi-solver and contrastive calibration achieve 29.21% and 31.09% accuracy on Terminal-Bench 2.0 (Merrill et al., 2026), compared with 22.47% after authoring and validation alone and 24.34% with single-solver feedback. Trained on the full collection of 5,431 tasks, CalibForge-30B-A3B and CalibForge-35B-A3B achieve 32.58% and 47.57%, surpassing the strongest baselines by 6.36 and 6.75 percentage points. The benefit extends to outof-distribution software-engineering benchmarks: the 35B model achieves 44.32% on SWE-bench Pro (Deng et al., 2025) for long-horizon issue resolution and 48.77% on Doc2Repo (Chen et al., 2026b) for full-repository generation, the highest scores among the evaluated training-data sources for the same backbone. Our contributions are summarized as follows:

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Adversarial Solver Calibration

Task Authoring & Validation Feedback

task specification inputs & expected failure tools & dependencies edge cases outcome modes feasibility env dependencies tool behavior files instruction build viability tests pass fail two-stage validation structural Docker fail-first checks pass self-solving revise self-solving

Multi-Solver Calibration

pass / fail steps count finish / timeout self-assessment failure diagnosis

All Fail ...

too hard / task issue

Learnable Zone ...

RETAIN

All Pass ...

too easy

trajectories ... Candidate Task instruction.md task.toml Dockerfile initial files tests

Stronger Solver

wide c3 search

adversarial revise

clue

Contrastive Solver Calibration Weaker Solver FAIL PASS Learnable PASS Zone too easy RETAIN FAIL

potential task issue

too hard / task issue

...

Figure 2. Overview of CalibForge. Starting from a clue, CalibForge authors a candidate task, subjects it to structural validation and self-solving, and then probes it using one of two adversarial solver-calibration strategies: multi-solver or contrastive calibration. The solvers’ pass/fail outcomes provide the retention signal, while feedback summaries and full interaction trajectories guide revisions at any stage: the authoring agent may return to web research or revise the task’s instruction, environment, or verification tests before revalidation and probing the candidate task again. • We formulate environment-level behavioral calibration, a construction principle that treats task learnability as solver-relative and uses verified solver behavior to determine retention and guide multi-round revision. • We introduce CalibForge, a terminal-task synthesis system that realizes this principle through a constrained adversarial author–solver loop. Its multi-solver and contrastive calibration strategies target cross-solver disagreement and a designated strong-pass/weak-fail relation. • We construct 5,431 calibrated terminal tasks and show that adversarial solver calibration yields more effective and transferable supervision than authoring and validation alone or single-solver feedback.

2. Methodology 2.1. Overview Starting from a clue, CalibForge’s authoring agent constructs a candidate terminal task τ, validates its structure, and attempts to solve it; V (τ ) indicates whether both checks pass. A valid candidate then enters an author–solver loop for adversarial solver calibration, governed by a calibration specification γ, which defines the solver setting and behavioral retention criterion Cγ . Each calibration round returns structured feedback summaries containing verified pass/fail outcomes, together with full interaction trajectories. When the outcomes do not satisfy Cγ , the authoring agent uses this feedback to revise and revalidate the task before the next round. The task is retained once Cγ is satisfied and discarded if the criterion remains unmet after Rmax rounds. Figure 2 illustrates the overall task-construction process, while Algorithm 1 specifies its iterative control flow.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Algorithm 1 The Pipeline of CalibForge Require: clue c, calibration specification γ, maximum calibration rounds Rmax I. Task Authoring & Validation 1: specification ← W I D E S E A R C H A N D S P E C I F Y (c ) 2: τ ← C O N S T R U C T T A S K (specification) 3: while ¬V (τ ) do

τ ← R E PA I R (τ ) 5: end while II. Adversarial Solver Calibration 6: for r = 1, . . . , Rmax do 7: feedback ← P R O B E A N D V E R I F Y (τ; γ) 8: outcomes ← O U T C O M E S (feedback) 9: if Cγ (outcomes) = 1 then 10: return τ 11: end if 12: τ ← R E V I S E (τ; feedback) 13: while ¬V (τ ) do 14: τ ← R E PA I R (τ ) 15: end while 16: end for 17: return D I S C A R D 4:

▷ validation failed

▷ criterion unmet ▷ validation failed

2.2. Candidate Task Authoring and Validation CalibForge places the authoring agent in a draft sandbox and equips it with tools for web research, shell command execution, and file editing. Starting from the clue, the agent searches external technical sources, including official documentation, GitHub repositories and issue discussions, and Stack Overflow questions, for concrete engineering problems such as version-specific bugs, dependency conflicts, configuration pitfalls, and reproducible edge cases. Based on this evidence, it develops several task directions and selects one based on its relevance to the clue, distinctiveness, and implementation feasibility. The agent then turns the selected direction into a task specification that records the required tools and dependencies, inputs and edge cases, expected final behaviors, and target failure modes. Before authoring the task files, it uses the draft sandbox to test whether required packages can be installed, external resources can be accessed, and relevant tools behave as expected, revising the specification when an assumption does not hold. Guided by this specification, the agent jointly constructs the task instruction, an execution environment defined by the Dockerfile and initial files, and verification tests that constitute the candidate task. Before validation, the agent checks that the initial environment does not expose solution artifacts and that the verification tests match the task instruction without imposing unstated requirements (Bercovich, 2026). Before entering solver calibration, each candidate undergoes a two-stage validation process comprising structural validation and self-solving. During structural validation, CalibForge checks that the required build-context files are present, builds and initializes the execution environment, runs the verifier, and confirms that all tests fail in the initial state. Once the structural checks pass, CalibForge instantiates an isolated test sandbox from the candidate Dockerfile. Within this sandbox, the authoring agent attempts to solve the task and runs the verifier against the resulting environment state, checking that the intended solution is executable and that the instruction, environment,

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

and verification tests are mutually consistent. If either stage fails, the authoring agent repairs the candidate and reevaluates it through both stages. Accordingly, V (τ ) = 1 if and only if the candidate passes both structural validation and self-solving; only such candidates proceed to adversarial solver calibration.

2.3. Adversarial Solver Calibration Rather than using solver behavior only to evaluate a finished task, CalibForge uses it to revise the candidate toward satisfying Cγ . Calibration Loop. In each calibration round, CalibForge instantiates an isolated sandbox from the candidate’s Dockerfile for every solver subagent, assigns the solvers according to γ, and provides each with the same task instruction. After an attempt, CalibForge runs the task verifier against the final sandbox state, producing a verified outcome yi ∈ {0, 1}, where yi = 1 indicates that all verification tests pass. Each attempt returns a structured feedback summary containing the verified pass/fail outcome, step count, completion status, self-assessment, and failure diagnosis, together with the full interaction trajectory. The pass/fail outcomes determine whether the candidate satisfies Cγ , while the feedback summary and full interaction trajectories guide task revision. Multi-Solver Calibration. Under γmulti , CalibForge dispatches K solver subagents using different models to attempt the same candidate independently. Let y = (y1 , . . . , yK ) denote their verified outcomes. The retention criterion requires disagreement across solver models: at least one attempt must pass, while at least one must fail. Formally, the multi-solver retention criterion is " # K

Cmulti (y) = 1 0 < ∑ yi < K .

(1)

i =1

An all-pass round may indicate that the task is too easy, for example because it admits a shallow solution path, whereas an all-fail round may indicate excessive difficulty, underspecification, or a broken task. The feedback summaries and trajectories help the authoring agent diagnose these outcome patterns and revise the task before another calibration round. By using disagreement across solver models as its retention criterion, multi-solver calibration reduces reliance on any single solver and captures a broader range of solution paths and failure modes. Contrastive Solver Calibration. Under γcon , CalibForge runs a designated stronger solver and a designated weaker solver on the same candidate. Writing ys and yw for their verified outcomes, respectively, Cγ is instantiated as Ccon (ys , yw ) = 1[ ys = 1 ∧ yw = 0 ] .

(2)

The weaker solver’s failure shows that the task lies beyond the weaker setting, while the stronger solver’s success confirms that it remains solvable under the stronger setting; together, these outcomes place the candidate within the capability interval defined by the two solver settings. If both solvers pass, the authoring agent searches for shortcuts or insufficient difficulty; if both fail, it checks solvability, specification quality, and verification; if the weaker solver passes while the stronger solver fails, the authoring agent inspects the task for leakage, nondeterminism, or misleading formulation. The specifications γmulti and γcon therefore instantiate γ with different solver settings and retention criteria while sharing the same author–solver loop. 5

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

3. Experiments 3.1. Experimental Setup Task Construction and Calibration. CalibForge uses DeepSeek-V4-Pro (Xu et al., 2026) as the authoring agent. For multi-solver calibration, three solver subagents using DeepSeek-V4-Flash, GLM-5 (Zeng et al., 2026), and Kimi K2.5 (Team et al., 2026), respectively, attempt each candidate independently. For contrastive solver calibration, DeepSeek-V4-Pro and DeepSeek-V4-Flash serve as the designated stronger and weaker solvers, respectively. Each solver attempt is limited to 100 interaction steps and 30 minutes. For each candidate, the author–solver loop runs for at most Rmax = 50 calibration rounds and terminates early once the target retention criterion is met. Trajectory Collection and Training. Retained tasks are stored as Harbor-style task instances (Harbor Framework Team, 2026). We distill SFT trajectories with DeepSeek-V4-Pro (reasoning effort high) under C A L I B F O R G E -E VA L , a minimal code-agent scaffold that exposes only bash, file-editing, and finish tools and follows the DeepSeek-V4 evaluation setting (Xu et al., 2026). For both CalibForge and baseline task sets, each task is attempted twice with a 200-step limit and a one-hour timeout, and test-passing trajectories are retained and subsequently filtered for length, invalid tool calls, and tokenizer-unsafe special tokens. This shared protocol controls for the teacher model and rollout budget across data sources. We fine-tune Qwen3-30B-A3B-Instruct and Qwen3.5-35B-A3B (Team, 2025; Qwen Team, 2026) using full-parameter SFT and train for 10 epochs. Baselines. We compare with open-source terminal-task synthesis methods that release their task sets: Endless Terminals, SETA-Env, CLI-Gym, TermiGen, and TerminalTraj (Gandhi et al., 2026; Shen et al., 2026b; Lin et al., 2026; Zhu et al., 2026; Wu et al., 2026). For SETA-Env, we use its January 2026 public release containing 1,375 tasks. For each baseline, we re-distill its full released task set using the same teacher protocol and apply the same training recipe. On Qwen3.5-35B-A3B, we retrain TermiGen and TerminalTraj, the two best-performing baselines on Qwen3-30B-A3B-Instruct. Benchmarks and Evaluation. We use Terminal-Bench 2.0 (Merrill et al., 2026) as the primary benchmark for terminal-agent problem solving. To evaluate out-of-distribution transfer, we additionally use the 731-task public set of SWE-bench Pro (Deng et al., 2025) for complex, long-horizon software engineering in existing repositories and Doc2Repo (Chen et al., 2026b; Zhao et al., 2026b) for full-repository generation from natural-language specifications. We use C A L I B F O R G E -E VA L to evaluate Terminal-Bench 2.0 with a 500-step limit and the same one-hour per-task timeout used for trajectory collection. Each evaluation sandbox is capped at 16 CPUs and 32 GB RAM. SWE-bench Pro and Doc2Repo use their official evaluation scaffolds. We report task accuracy on Terminal-Bench 2.0, Resolved Rate on SWE-bench Pro, and Pass Rate on Doc2Repo. Terminal-Bench 2.0 and Doc2Repo results are reported as mean ± standard error of the mean (SEM) over three runs, while SWE-bench Pro is evaluated once. Benchmark Decontamination. We decontaminate the training tasks against all three evaluation benchmarks. Following prior work (Pi et al., 2026), we first remove candidates whose prompts have 14-gram overlap with any evaluation instance. We then compute 5-shingle Jaccard similarity over normalized instructions and available verifier or test code, and combine it with shared output paths, overlapping test functions, and high-risk task-family matches. Any candidate flagged against Terminal-Bench 2.0, SWE-bench Pro, or Doc2Repo is discarded; the supplementary material gives the full matching rule.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Training Data

TB2 SWE-Pro Doc2Repo Acc. (%) Resolved (%) Pass Rate (%) Qwen3-30B-A3B-Instruct

Base Model 7.87 ± 0.00 Endless Terminals 19.48 ± 3.00 CLI-Gym 23.22 ± 2.70 SETA-Env 23.22 ± 0.75 TermiGen 23.60 ± 1.12 TerminalTraj 26.22 ± 0.75 CalibForge 32.58 ± 1.12

3.26 21.84 28.81 29.91 27.77 26.28 30.94

5.94 ± 0.88 18.26 ± 2.39 29.91 ± 1.53 24.91 ± 1.84 34.11 ± 2.64 24.36 ± 0.96 35.98 ± 1.82

Qwen3.5-35B-A3B Base Model TermiGen TerminalTraj CalibForge

39.10 ± 1.09 40.07 ± 0.99 40.82 ± 0.75 47.57 ± 0.99

41.29 43.37 43.91 44.32

44.92 ± 1.14 44.54 ± 1.58 47.20 ± 1.89 48.77 ± 0.90

Table 1. Main results on Terminal-Bench 2.0 and out-of-distribution software-engineering benchmarks. Blue -shaded SWE-bench Pro and Doc2Repo columns denote out-of-distribution evaluations. Results are reported as mean ± SEM over three runs except for SWE-bench Pro, which is evaluated once. Bold and underlined denote the best and second-best result within each backbone block.

3.2. Main Results Terminal-Task Performance. On Terminal-Bench 2.0, CalibForge-30B-A3B and CalibForge-35BA3B achieve 32.58% and 47.57%, outperforming the strongest baselines under the shared training protocol by 6.36 and 6.75 percentage points, respectively. Because all task sets use the same distillation and training protocol, the training-data source is the primary experimental difference. Figure 3 further shows that the gains extend across Terminal-Bench 2.0 categories rather than being driven by a narrow subset, with both models improving or matching their base models in every category. Cross-Benchmark Transfer. On SWE-bench Pro, CalibForge-30B-A3B and CalibForge-35B-A3B improve over their respective base models by 27.68 and 3.03 percentage points; on Doc2Repo, they improve by 30.04 and 3.85 points, respectively. Takeaway. Across two backbones, CalibForge tasks provide more effective supervision than the evaluated terminal-task datasets. Their gains on repository-level generation and issue resolution further suggest that the learned capabilities transfer beyond the terminal-task distribution used for training.

3.3. Analysis of Synthesized Data Scale and Domain Coverage. The resulting 5,431 tasks comprise 1,263 tasks from multi-solver calibration and 4,168 from contrastive solver calibration. The tasks span all 16 categories in our domain taxonomy. To characterize domain coverage, Figure 4 compares CalibForge-generated tasks with prior terminal-task sets under a common taxonomy. Several prior datasets are dominated by one or two categories: system administration accounts for 74.6% of SETA-Env and 49.9% of TerminalTraj, debugging accounts for 67.0% of CLI-Gym, and file operations accounts for 40.6% of Endless-Terminals. In contrast, the largest category in the final CalibForge task set is software engineering at 25.5%, with substantial coverage also allocated to system administration, scientific 7

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

SWE (26) SysAdmin (9) Security (8) Debugging (5) Data Sci. (8) Model Train. (4) Data Proc. (4) SciComp (8) Math (4) ML (3) Personal Asst. (1) Optimization (1) Games (1) File Ops. (5) Data Query (1) Video Proc. (1)

+2 +1 +1 +1

+6 +5

+2 +3

Base CalibForge

+1 0

2 4 8 10 6 TB2 tasks solved (pass@3)

+1 +2 +1 +1 +1 +1 0

(a) CalibForge-30B-A3B

+1 +2

Base CalibForge

2 4 8 10 6 TB2 tasks solved (pass@3)

(b) CalibForge-35B-A3B

Figure 3. Per-category Terminal-Bench 2.0 tasks solved (pass@3) for (a) CalibForge-30B-A3B and (b) CalibForge-35B-A3B, compared with their respective base backbones. + N gives the net increase in solved tasks. SWE, SysAdmin, SciComp, and ML denote software engineering, system administration, scientific computing, and machine learning, respectively. Both variants improve or match their base backbones in every category. 31.7%

Endless Term.

40.6%

9.2%

SETA-Env

74.6% 23.5%

CLI-Gym

67.0%

19.7%

12.0%

21.8%

TerminalTraj

0 Software Engineering Data Science

8.1%

13.2%

17.0% 10.3%

49.9%

25.5%

CalibForge

6.5%

8.4%

16.1%

TermiGen

20.7%

11.9% 20 System Administration Debugging

9.4% 40

8.6%

7.3%

60 % of tasks Scientific Computing Data Processing

8.2%

6.4% 5.7%

17.0%

80 Security Other

100

File Operations

Figure 4. Domain composition of CalibForge-generated tasks and prior terminal-task datasets under a common 16-category taxonomy. Categories accounting for at least 5% of CalibForge-generated tasks are shown separately, with the remainder grouped as Other. computing, security, file operations, data science, debugging, and data processing. This composition indicates that the synthesized data broaden terminal supervision beyond the narrow domain concentrations present in several existing task sets. Capability Diversity. Beyond broad domain categories, we examine diversity in the tools, techniques, and problem-solving capabilities exercised by individual tasks. Following the TerminalBench 2.0 annotation scheme (Merrill et al., 2026), we represent these properties as capability tags. As shown in Figure 5, we observe 3,885 distinct capability tags, with a median of five tags per task. Their rank–frequency distribution is long-tailed: 51.6% of the distinct tags occur in only one task, and 82.2% occur in at most five tasks. This long tail shows that the domain breadth is accompanied 8

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Number of tasks

103

102

101

100 100

101

102 Capability-tag rank

103

Figure 5. Rank–frequency distribution of capability tags in the CalibForge task collection. Metric Environment Initial artifacts Distinct file types Distinct environment dependencies Verification Verifier test functions

Collection

Median

IQR

P90

19,911 362 615

2 1 2

1–4 1–2 1–4

8 4 7

45,953

5–11

Table 2. Collection-level and per-task statistics of environment and verification components. IQR denotes the interval between the first and third quartiles. by many specialized capabilities rather than repeated combinations of a small capability vocabulary. Environment and Verification Statistics. Figure 6 presents the per-task distributions of initial artifacts, distinct file types, environment dependencies, and verifier test functions, while Table 2 summarizes the corresponding statistics. A median task contains two initial artifacts, one distinct file type, two environment dependencies, and seven verifier test functions; the corresponding 90th-percentile values are eight, four, seven, and 15. These distributions show that the collection varies not only in task topic, but also in the amount of environment state and verification logic that a solver must handle. Takeaway. Terminal-task diversity is multidimensional: domain coverage, capability variety, and environment and verification structure capture complementary properties of a training collection. Task count or domain breadth alone is therefore insufficient to characterize the range of behaviors represented by synthesized terminal tasks.

Trajectory Characteristics. We next compare per-trajectory interaction steps and teacher thinking tokens across task sets re-distilled under the identical teacher protocol (Figure 7). CLI-Gym derives repository-grounded environment-repair tasks by inverting runnable Python project environments (Lin et al., 2026). Such tasks typically require dependency or configuration diagnosis followed by repeated execution and verification, so their task structure naturally produces longer interaction traces. Consistent with this task structure, CLI-Gym has the highest median interaction length at 28 steps, whereas CalibForge has a median of 21 steps; the ordering reverses for thinking tokens, where CalibForge has the highest median at 5.3k compared with 4.0k for CLI-Gym.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Median 2 P90 8

Median 1 P90 4

40 Fraction of tasks (%)

Fraction of tasks (%)

20 15 10 5 0

30 20 10 0

10 Files per task

Median 2 P90 7

(b) Distinct file types Median 7 P90 15

20 Fraction of tasks (%)

Fraction of tasks (%)

(a) Initial artifacts

2 4 File types per task

15 10 5 0

0.0

2.5

5.0 7.5 Packages per task

10.0

12.5

10 20 Verifier test functions per task

(d) Verifier test functions

Figure 6. Per-task distributions of environment and verification statistics. Solid and dashed vertical lines denote the median and 90th percentile, respectively. CalibForge

CalibForge

CLI-Gym

TermiGen

SETA-Env

TerminalTraj

Endless Term. 0

Endless Term. 20

4.0

TermiGen

SETA-Env

5.3

CLI-Gym

40 80 60 Steps per trajectory

(a) Interaction steps

100

3.2 1.6 1.3 0.9 0

10 20 15 Thinking tokens (k)

(b) Teacher thinking tokens

Figure 7. Trajectory characteristics across task sets re-distilled under the identical teacher protocol: (a) interaction steps and (b) teacher thinking tokens. Numbers below the boxes denote medians. CLI-Gym produces the longest trajectories by interaction count, whereas CalibForge elicits the largest teacher reasoning budget.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Takeaway. Terminal tasks may require long-horizon reasoning, yet interaction length is also shaped by execution mechanics. CLI-Gym’s environment-repair tasks naturally involve repeated execution, debugging, and verification; under the same teacher protocol, CalibForge elicits more thinking tokens with fewer interaction steps. Step count alone is therefore insufficient to characterize the reasoning depth or quality of a terminal trajectory.

3.4. Effect of Solver Calibration Ablation Variants. No Solver omits external solver feedback. Single Solver uses a separate solver subagent powered by the same DeepSeek-V4-Pro model as the authoring agent. It isolates ordinary solver feedback from multi-solver and contrastive calibration: the authoring agent may revise the candidate using this subagent’s pass/fail outcome and trajectory, but receives neither cross-model disagreement nor a designated strong–weak relation. Multi Solver and Contrast Solver instantiate the two retention criteria introduced in the Methodology section. Adversarial solver calibration provides larger gains than single-solver feedback. Table 3 compares the four variants under a matched task-count setting. Single-solver feedback improves TerminalBench 2.0 accuracy by 1.87 points, whereas multi-solver and contrastive calibration improve it by 6.74 and 8.62 points, respectively. These improvements cannot be explained by trajectory volume alone: multi-solver calibration produces fewer retained SFT trajectories than the no-solver variant yet improves accuracy substantially. Takeaway. The advantage of adversarial solver calibration is not explained by adding solver feedback or collecting more trajectories alone. The larger gains from multi-solver and contrastive calibration suggest that locating candidate difficulty relative to the selected solver settings provides a more useful construction target than a single solver’s pass/fail signal.

To examine whether solver feedback during contrastive calibration merely filters candidates or actively reshapes them, we trace all contrastive-calibration runs from their first verified solver outcome to their eventual retention outcome. Beyond task validity, solver feedback revises rather than merely filters candidates. Figure 8 classifies each candidate by its first verified solver probe: only 19% exhibit the target strongpass/weak-fail relation, while 81% do not, despite all candidates having passed structural validation and self-solving. The dominance of both-pass outcomes shows that, in this setting, the most common gap left by validity checks is insufficient separation between the solver settings rather than task unsolvability. Structural validation and self-solving establish executability and solvability, but do not control where a task lies within the target capability interval defined by the two solver settings. After feedback-driven revision and re-probing, 96% of candidates ultimately satisfy the contrastive retention criterion. Feedback-driven revision corrects many mismatches early, while harder candidates require sustained calibration. Figure 9 groups candidates by the total number of recorded solver probes in the completed calibration run. Under this run-level measure, 15% complete a retained run with one recorded probe, 53% within five probes, and 93% within twenty. The early gains show that many mismatches can be corrected with few revisions, while the long tail shows that harder cases require sustained probing. The calibration budget therefore affects not only construction cost but also which recoverable candidates enter the training set: a short horizon favors candidates whose mismatches can be corrected with only a few revisions.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Collected SFT TB2 ∆ vs. Tasks Trajectories Acc. No Solver

Calibration Mode No Solver Single Solver

1,300 1,300

2,466 2,493

22.47 24.34

– +1.87

Multi Solver Contrast Solver

1,300 1,300

2,425 2,561

29.21 31.09

+6.74 +8.62

Table 3. Effect of solver feedback and behavioral calibration on Qwen3-30B-A3B-Instruct. blue rows denote our multi-solver and contrastive calibration strategies. Each variant contains 1,300 tasks and uses the same teacher-distillation and SFT recipe; SFT trajectory counts reflect trajectory filtering. feedback → revision → re-probing First calibration round

Final status

Target relation

19% Both fail 16% Accepted 96% Both pass 61%

Inverted: 4%

Discarded: 4%

Figure 8. First-probe solver outcomes and final retention under contrastive calibration. The left nodes classify the first verified solver probe: 19% exhibit the target strong-pass/weak-fail relation. The right nodes report final run-level retention, which reaches 96% after revision and re-probing. Remaining Start 100%

Retained 0%

Round 1 85%

15%

Round 5 47%

53%

Round 10 24%

76%

Round 20

93%

Final

96%

Figure 9. Progressive contrastive-calibration funnel. Width denotes the remaining candidates; right-hand labels show the cumulative fraction retained in runs completed within the indicated number of solver probes.

Takeaway. A non-target solver outcome is often a correctable mismatch rather than an intrinsic defect in the candidate. Treating such outcomes as diagnostic evidence allows calibration to revise recoverable tasks toward the target relation, while the long tail shows that some mismatches require sustained calibration rather than only a few revisions.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

4. Related Work Terminal Agents and Benchmarks. Execution-grounded agents operate in interactive software environments, spanning code and repository tasks as well as stateful command-line workspaces (Yang et al., 2023; Jimenez et al., 2024; Yang et al., 2024; Yao et al., 2022; Liu et al., 2024; Xie et al., 2024; Siegel et al., 2024; Song et al., 2026; Sun et al., 2026; Bui, 2026; Ren et al., 2026; Ivison et al., 2026; Wang et al., 2025; Chen et al., 2026a; Cheng et al., 2026a). Terminal-Bench, TerminalWorld, and OpenThoughts-TBLite evaluate terminal agents using executable tasks and verifiers (Merrill et al., 2026; Chu et al., 2026; OpenThoughts-Agent team, 2026; Feng et al., 2026; Chen et al., 2026c; Li et al., 2026). Such tasks couple instructions, initial files, dependencies, and verification tests, making their joint construction and validation essential. Verifiable Terminal-Task Synthesis. Existing methods synthesize terminal tasks from specifications and capability taxonomies (Zhu et al., 2026; Pi et al., 2026; Gandhi et al., 2026; Shen et al., 2026a; Hua et al., 2026; Peng et al., 2026; Lai et al., 2026; Zhao et al., 2026c; Chen et al., 2026d), from software artifacts, repositories, and agent trajectories (Lin et al., 2026; Wu et al., 2026; Yang et al., 2026; Zhao et al., 2026a), or from structured agent skills and skill graphs (Cheng et al., 2026b; Fan et al., 2026). These methods combine environment construction with execution-grounded validation, whereas CalibForge studies how solver behavior can guide candidate revision. Behavioral Feedback for Data Construction. Prior work uses model behavior to adapt benchmarks or curricula (Kiela et al., 2021; Dennis et al., 2020), while Reflexion uses trajectory feedback to improve a solver’s subsequent attempts (Shinn et al., 2023). CalibForge instead applies solver behavior to the task under construction: verified pass/fail outcomes determine retention, while feedback summaries and full interaction trajectories guide task revision.

5. Conclusion We presented CalibForge, an autonomous terminal-task synthesis system that turns solver behavior into construction-time feedback through two adversarial solver-calibration strategies: multi-solver and contrastive calibration. Fine-tuning two Qwen backbones on trajectories distilled from 5,431 calibrated tasks yields gains on Terminal-Bench 2.0 and transfer to repository-level software-engineering benchmarks. Matched ablations attribute these gains to the two calibration strategies rather than trajectory volume or single-solver feedback. Together, these results show that solver behavior can guide terminal-task construction toward more effective training data.

References Anthropic. Claude Code. https://code.claude.com/docs/en/overview, 2026. AweAI Team. AweAgent: A unified, composable framework to build, evaluate, and train agents. https://github.com/AweAI-Team/AweAgent, 2026. GitHub repository. I. Bercovich. What makes a good terminal-agent benchmark task: A guideline for adversarial, difficult, and legible evaluation design. arXiv preprint arXiv:2604.28093, 2026. N. D. Bui. Building effective ai coding agents for the terminal: Scaffolding, harness, context engineering, and lessons learned. arXiv preprint arXiv:2603.05344, 2026. G. Chen, J. Chen, L. Chen, J. Zhao, F. Meng, W. X. Zhao, R. Song, C. Chen, J.-R. Wen, and K. Jia. Toward autonomous long-horizon engineering for ml research. arXiv preprint arXiv:2604.13018, 2026a. 13

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

G. Chen, F. Meng, J. Zhao, M. Li, D. Cheng, H. Song, J. Chen, Y. Lin, H. Chen, X. Zhao, et al. Beyondswe: Can current code agent survive beyond single-repo bug fixing? arXiv preprint arXiv:2603.03194, 2026b. S. Chen, L. Wang, X. Yang, Z. Liu, Y. Cong, Y. Ji, F. Zhou, X. Zhang, F. Yang, and B. Zeng. Tua-bench: A benchmark for general-purpose terminal-use agents. arXiv preprint arXiv:2606.28480, 2026c. X. Chen, Z. Qiao, G. Chen, L. Su, Z. Zhang, X. Wang, P. Xie, F. Huang, J. Zhou, Y. Jiang, and T. Chen. Expanding the capability frontier of LLM agents with ZPD-guided data synthesis. In The Fourteenth International Conference on Learning Representations, 2026d. URL https://openrevi ew.net/forum?id=c5bf47nDx1. D. Cheng, S. Huang, Y. Gu, H. Song, G. Chen, L. Dong, W. X. Zhao, J.-R. Wen, and F. Wei. Llm-insandbox elicits general agentic intelligence. arXiv preprint arXiv:2601.16206, 2026a. Z. Cheng, H. Wang, Z. Liu, X. Wang, X. Zhu, Y. Guo, W. Lin, J. Z. Pan, and Y. Wang. Terminal-world: Scaling terminal-agent environments via agent skills. arXiv preprint arXiv:2605.20876, 2026b. Z. Chu, J. Hu, X. Jiang, P. Zou, H. Li, C. Peng, P. O’Hearn, E. T. Barr, M. Harman, F. Sarro, et al. Terminalworld: Benchmarking agents on real-world terminal tasks. arXiv preprint arXiv:2605.22535, 2026. Cursor Research, A. Chan, A. Shalaby, A. Wettig, A. Sanger, A. Zhai, A. Ajay, A. Nair, C. Snell, C. Lu, et al. Composer 2 Technical Report. arXiv preprint arXiv:2603.24477, 2026. URL https: //arxiv.org/abs/2603.24477. X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025. M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:13049–13061, 2020. Z. Fan, T. Yu, Y. Cai, J. Guan, Y. Yang, D. Hu, J. Zhou, X. Wu, Z. Han, F. Zhang, et al. Toward scalable terminal task synthesis via skill graphs. arXiv preprint arXiv:2604.25727, 2026. Y. Feng, J. Sun, Z. Yang, J. Ai, C. Li, Z. Li, F. Zhang, K. He, R. Ma, J. Lin, et al. Longcli-bench: A preliminary benchmark and study for long-horizon agentic programming in command-line interfaces. In Findings of the Association for Computational Linguistics: ACL 2026, pages 29952–29963, 2026. K. Gandhi, S. Garg, N. D. Goodman, and D. Papailiopoulos. Endless terminals: Scaling rl environments for terminal agents. arXiv preprint arXiv:2601.16443, 2026. Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026. URL https://doi.org/10.5281/zenodo.20953922. Z. Hua, Y. Yao, W. Xie, Y. Zhao, M. Liu, R. Qiu, Z. Huang, Z. Wang, Y. Ji, Y. Ye, et al. Cli-universe: Towards verifiable task synthesis engine for terminal agents. arXiv preprint arXiv:2606.22883, 2026. H. Ivison, J. O. Yin, R. Shao, T. Xiao, N. Lambert, and H. Hajishirzi. Tmax: A simple recipe for terminal agents. arXiv preprint arXiv:2606.23321, 2026.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024. D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, et al. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104.14337, 2021. Y. Lai, P. Xia, H. Ji, K. Xiong, K. Zeng, J. Liu, F. Wu, J. Zhong, Z. Zheng, C. Xie, et al. Clawforge: Generating executable interactive benchmarks for command-line agents. arXiv preprint arXiv:2605.14133, 2026. Z. Li, Z. Li, Y. Shi, R. Wang, J. Yang, Z. Liu, X. Wu, A. Li, Y. Yu, N. Liu, et al. Long-horizon-terminalbench: Testing the limits of agents on long-horizon terminal tasks with dense reward-based grading. arXiv preprint arXiv:2607.08964, 2026. Y. Lin, H. Wang, S. Wu, L. Fan, F. Pan, S. Zhao, and D. Tu. Cli-gym: Scalable cli task generation via agentic environment inversion. arXiv preprint arXiv:2602.10999, 2026. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, volume 2024, pages 52989–53046, 2024. M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. OpenAI. Introducing Codex. https://openai.com/index/introducing-codex/, May 2025. B. L. OpenThoughts-Agent team, Snorkel AI. OpenThoughts-TBLite: A High-Signal Benchmark for Iterating on Terminal Agents. https://www.openthoughts.ai/blog/openthoughts-tblite, Feb. 2026. X. Peng, K. Zhang, X. Lu, B. Cao, Y. Lu, H. Lin, X. Han, and L. Sun. LiteCoder-Terminal: Scaling long-horizon terminal environments for learning language agents. arXiv preprint arXiv:2605.29559, 2026. R. Pi, G. Lam, M. Shoeybi, P. Jannaty, B. Catanzaro, and W. Ping. On data engineering for scaling llm terminal capabilities. arXiv preprint arXiv:2602.21193, 2026. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/ blog?id=qwen3.5. J. Ren, S. Wu, Y. Li, K. Zhu, S. Xu, B. Feng, R. Yuan, W. Zhang, R. Batista-Navarro, J. Yang, et al. A self-evolving framework for efficient terminal agents via observational context compression. arXiv preprint arXiv:2604.19572, 2026. Q. Shen, Z. Huang, V. Kamanuru, A. Aliev, J. Rainton, A. Awelkair, Z. Zeng, J. Li, S. Dong, Y. Yuan, et al. Seta: Scaling environments for terminal agents. arXiv preprint arXiv:2607.10891, 2026a. URL https://arxiv.org/abs/2607.10891. Q. Shen, J. Rainton, A. Aliev, A. Awelkair, B. Ma, Z. Huang, Y. Mao, W. Fan, P. Torr, B. Ghanem, C. Hu, U. Thakker, and G. Li. SETA-Env: Scaling Environments for Terminal Agents. https: //github.com/camel-ai/seta-env, Jan. 2026b. Public release containing 1,375 terminal tasks.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023. Z. S. Siegel, S. Kapoor, N. Nadgir, B. Stroebl, and A. Narayanan. Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. Transactions on Machine Learning Research, 2024. H. Song, L. Huang, S. Sun, J. Jiang, R. Le, D. Cheng, G. Chen, Y. Hu, Z. Chen, Y. Jia, et al. Swemaster: Unleashing the potential of software engineering agents via post-training. arXiv preprint arXiv:2602.03411, 2026. S. Sun, H. Song, L. Huang, J. Jiang, R. Le, Z. Lv, Z. Chen, Y. Hu, W. Luo, W. X. Zhao, et al. Swe-world: Building software engineering agents in docker-free environments. arXiv preprint arXiv:2602.03419, 2026. X. Tang, H. Peng, G. Chen, Y. Shi, Z. Su, P. Liu, W. X. Zhao, Y. Li, and Z. Xue. Agent systems with harness engineering, 2026. URL https://openreview.net/pdf?id=nM5tDHrQsx. K. Team, T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Y. Charles, H. Che, C. Chen, G. Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. Q. Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume 2025, pages 65882–65919, 2025. S. Wu, Y. Li, Y. Song, W. Zhang, Y. Wang, R. Batista-Navarro, X. Yang, M. Tang, B. Dai, J. Yang, et al. Large-scale terminal agentic trajectory generation from dockerized environments. arXiv preprint arXiv:2602.01244, 2026. T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024. A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36: 23826–23854, 2023. J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. S. Yang, C. Tao, J. Chen, T. Yu, R. Wang, Y. Jiang, Y. Du, W. Xu, J. Xiong, T. Wu, et al. What makes interaction trajectories effective for training terminal agents? arXiv preprint arXiv:2606.03461, 2026. S. Yao, J. Zhao, D. Yu, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. J. Zhao, G. Chen, F. Meng, M. Li, J. Chen, H. Xu, Y. Sun, X. Zhao, R. Song, Y. Zhang, et al. Immersion in the GitHub universe: Scaling coding agents to mastery. arXiv preprint arXiv:2602.09892, 2026a. J. Zhao, G. Chen, F. Meng, W. X. Zhao, R. Song, J.-R. Wen, and K. Jia. Denovoswe: Scaling long-horizon environments for generating entire repositories from scratch. arXiv preprint arXiv:2606.10728, 2026b. J. Zhao, Z. Lei, Z. Xi, R. Zheng, H. Yan, J. Zhou, Q. Chen, and L. He. Nexforge: Scaling executable agent tasks via requirement-first synthesis. arXiv preprint arXiv:2607.14186, 2026c. Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, Z. Feng, and Y. Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http://arxiv.org/abs/2403.13372. K. Zhu, Y. Nie, Y. Li, Y. Huang, J. Wu, J. Liu, X. Sun, Z. Yin, L. Wang, Z. Liu, et al. Termigen: High-fidelity environment and robust trajectory synthesis for terminal agents. arXiv preprint arXiv:2602.07274, 2026.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

A. From a Clue to a Calibrated Task The following example traces how CalibForge uses web research to develop a clue into a candidate terminal task and retains it through contrastive solver calibration. 1 ○

Input c — Clue

Domain Scope Capability hints Tool hints

Data Processing & ETL Pipelines, transformation, schema evolution, and log analysis Data I/O; Binary Parsing; Data Recovery xxd; hexdump

↓ 2 ○

Wide Search and Direction Selection

The authoring agent issued 24 search calls spanning the major directions below. Direction

Representative searches

Authoring agent’s assessment

Tabular and columnar data

CSV edge cases; corrupted Parquet metadata

Parquet repair appeared too shallow.

Database and repository recovery

SQLite/WAL salvage; damaged Git objects

Git recovery was too close to existing tasks; SQLite/WAL recovery was too tool-specific.

Archive and text recovery

Truncated gzip; mixed encodings; malformed NDJSON

Format-specific repair risked collapsing into a single utility invocation.

Media and network formats

PCAP extraction; EXIF/JPEG repair; MP3/ID3 recovery

PCAP was judged too security-oriented.

Legacy and heterogeneous data

Fixed-width/COBOL records; multi-format ETL

Legacy-format ETL required substantial setup without a comparably clear verification target.

Binary sensor logs

IoT formats; sync markers; checksums; interrupted writes

Selected. Reason: grounded, multi-step, not solvable by one command, and deterministically verifiable.

Search-grounded direction. Recover environmental readings from a corrupted binary sensor log using record synchronization and checksum validation.

↓ 3 ○

Output I — Task Instruction

An EnviroLog-2000 monitoring station stores temperature, humidity, pressure, and CO2 readings in a custom binary format. A power failure corrupted /app/sensor_data.bin; the format specification is provided in /app/FORMAT_SPEC.md. Recover as many valid readings as possible and write /app/recovered_readings.csv with the columns timestamp, temperature, humidity, pressure, and co2, sorted by timestamp. Do not trust the header record count, and discard records with invalid checksums or impossible sensor values.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

↓ 4 ○

Output E — Environment and Initial Artifacts

/app/sensor_data.bin

Corrupted binary log containing intact, truncated, and overwritten records. Record layout, sync marker, CRC-16 procedure, and physical validity ranges. Ubuntu 24.04 environment with Python and the two task files placed under /app.

/app/FORMAT_SPEC.md Dockerfile

↓ 5 ○

Output V — Verification Scope

✓

The output CSV exists and has the required schema.

✓

Recovered records have the correct count and chronological order.

✓

The overwritten interval is detected and excluded.

✓

Timestamps and sensor values satisfy the expected ranges and boundary cases.

✓

Selected recovered records match their expected values.

Implementation: 11 verifier tests.

↓ 6 ○

Contrastive Calibration Feedback

Role

Solver model

Verified outcome

Stronger solver Weaker solver

DeepSeek-V4-Pro DeepSeek-V4-Flash

PASS FAIL

Solver self-assessments Role

Difficulty

Condensed assessment

Stronger

Easy–medium

Weaker

Medium

The format specification was clear; an inconsistent example CRC required validation against actual records. Binary-structure parsing and combined CRC/range validation were the main challenges.

B. How Solver Feedback Revises Tasks The following cases show how structured feedback and full interaction trajectories guide task revision during adversarial solver calibration. Each case presents the initial pass/fail outcomes, the feedback used by the authoring agent, the resulting task revision, and the outcomes after revalidation and re-probing. 19

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

B.1. Removing Procedural Hints after Both Solvers Pass Case C1: Recovering Corrupted Transaction Data

Contrastive Calibration

Task. Repair corrupted fixed-width transaction records and export a normalized CSV file. Initial Outcome Both Pass in the first calibration round. Role

Solver model

Stronger Weaker

DeepSeek-V4-Pro DeepSeek-V4-Flash

Outcome

Steps

Difficulty of self-assessment

PASS PASS

8 17

Easy Medium

Solver Evidence The stronger solver rated the task easy because the instruction exposed the exact field positions, corruption patterns, and a worked date-repair example. The weaker solver identified fixed-width extraction and handling the two disclosed corruption types as the main work. Their assessments indicated that success depended on implementing an explicit repair procedure rather than diagnosing the malformed records. Author Revision BEFORE AFTER REVISION Two types of corruption have been identified: extra Corrupted records may have structural issues or leading whitespace shifts every field, and invalid invalid field values. The solver must inspect the dates contain swapped month and day compo- data and layout specification, determine what nents. went wrong, and repair the records. Outcome after Re-probing Stronger solver: PASS

Weaker solver: FAIL

RETAINED AFTER RE-PROBING

The initial all-pass pattern exposed an overly explicit solution path rather than merely low task difficulty. Removing the disclosed repair path preserved the required output while making success depend on diagnosing the corrupted records.

B.2. Clarifying Comparison Semantics after All Solvers Fail Case C2: Comparing Customer Database Exports

Multi-Solver Calibration

Task. Compare two customer-database exports and report record-level and schema-level differences after normalization. Initial Outcome All Fail in the first calibration round. Solver model GLM-5 Kimi-K2.5 DeepSeek-V4-Flash

Outcome

Steps

Difficulti of self-assessment

FAIL FAIL FAIL

50 15 26

Medium–hard Medium Medium

Solver Evidence Task. The task provides two snapshots of the same customer database: a reference export and a current export. The solver must normalize the two files and produce a comparison report that separates schema-level differences from modifications to individual customer records.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Header Record

Reference export

Current export

id,name,email 42,Ana,a@x.com

id,name,email,tier 42,Ana,a@x.com,gold

Solver output versus verifier expectation. Report component

All three solvers reported

Verifier expected

Schema differences Modified records

Added column tier.

Customer id=42 was modified because the current export contains the additional field tier=gold.

No modified record because the shared fields id, name, and email are unchanged.

Because all three solvers produced the same extra record-level classification, the shared all-fail pattern exposed an underspecified comparison rule rather than independent implementation errors. Author Revision BEFORE AFTER REVISION Report schema changes and modified customer Determine whether a customer record is modirecords. Either export may contain columns ab- fied by comparing only fields shared by both exsent from the other. ports. Treat columns present in only one export as schema changes, not record modifications. Outcome after Re-probing Mixed Outcomes after re-probing. GLM-5: PASS

Kimi-K2.5: FAIL

DeepSeek-V4-Flash: PASS

RETAINED AFTER RE-PROBING

The shared failures identified ambiguity in the task instruction rather than a need to simplify the underlying data-processing problem. The revision resolved the ambiguity without making the task uniformly easy: two solvers passed after re-probing, while one still failed, satisfying the multi-solver retention criterion.

B.3. Generalizing an Overly Prescriptive Verifier after an Inverted Outcome Case C3: Securing a Legacy Password Vault

Contrastive Calibration

Task. Repair cryptographic vulnerabilities in a command-line password manager while preserving its interface and keeping entries from the legacy vault readable. Initial Outcome Inverted Outcome in the first calibration round. Role

Solver model

Stronger Weaker

DeepSeek-V4-Pro DeepSeek-V4-Flash

Outcome

Steps

Difficulty of self-assessment

FAIL PASS

40 38

Medium Medium

Solver Evidence The task required all sensitive fields to be encrypted while keeping existing vault entries readable. The stronger solver combined the password, notes, and other sensitive fields into one record and encrypted the record as a whole using authenticated encryption. However, the verifier expected these fields to contain separately encrypted ciphertext and therefore rejected this valid alternative layout. The weaker solver followed the verifier’s expected field-wise layout and passed. The inverted outcome

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

therefore exposed an overly prescriptive verifier rather than an invalid solution. Author Revision AFTER REVISION BEFORE The verifier expected named fields to contain sepa- The verifier discovers the encrypted payload inderately encoded nonce:ciphertext:tag values and pendently of its field name and tests authenticated swapped the password fields to test context bind- encryption, context binding, and legacy migration ing. without prescribing the vault’s object layout. Outcome after Re-probing Stronger solver: PASS

Weaker solver: FAIL

RETAINED AFTER RE-PROBING

The inverted outcome exposed a brittle verifier rather than an invalid solution. Generalizing the verifier preserved the required security properties while allowing alternative valid storage layouts. After the verifier was revised, a subsequent calibration round produced a stronger-solver pass and a weaker-solver failure, satisfying the contrastive retention criterion. Summary. Verified pass/fail outcomes determine whether a candidate satisfies its retention criterion, but do not explain why the observed outcome pattern occurs. Across these cases, feedback summaries and full interaction trajectories reveal problems such as overly explicit instructions, underspecified requirements, and overly prescriptive verifiers, thereby guiding task revision.

C. CalibForge-Eval C.1. Tool Interface CalibForge-Eval is a minimal code-agent scaffold implemented using AweAgent (AweAI Team, 2026) for trajectory distillation and Terminal-Bench 2.0 evaluation. Following the DeepSeek-V4 code-agent evaluation setting (Xu et al., 2026), it exposes only bash, file-editing, and finish tools under a compact prompt. Tool

Parameters

Function

execute_bash

command: string (required); timeout: number (optional)

Executes one Bash command in the persistent task runtime and returns its output and execution status.

str_replace_editor

command: view, create, str_replace, or insert; path: string (both required); operation-specific file_text, old_str, new_str, insert_line, or view_range

Views, creates, and edits files in the persistent task runtime.

finish

No arguments

Ends the interaction; task success is subsequently determined by running the verifier on the final runtime state.

Table C1. Function-tool interface exposed by CalibForge-Eval. Operation-specific parameters are used only by the corresponding str_replace_editor operation.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

C.2. Prompt Templates CalibForge-Eval System Prompt You are a capable code and command-line agent operating in a Linux runtime environment. Complete the user's technical task by inspecting the environment, editing files, running commands, and verifying the result. Use the available tools directly whenever an action is needed. <WORKFLOW> 1. Read the task carefully and identify the concrete deliverable, required output format, exact paths, field names, variable names, version constraints, and any named interfaces, scripts, files, or services. Treat these details as hard requirements. 2. Survey the environment before changing it. Check the working directory and nearby resources, but do not assume all relevant files are under the initial directory. 3. Prefer existing project workflows and task-provided interfaces over building local substitutes. If a CLI, script, API, or data format is mentioned, inspect its help, documentation, source, or sample data before using it. 4. Pay attention to implicit constraints in the task wording, especially constraints about exact outputs, allowed files, dependency versions, or how simple the requested solution should be. 5. Make focused changes that solve the task. Modify intended files directly instead of creating alternate copies with suffixes such as _fixed, _new, or _test. 6. Keep deliverable directories clean. Temporary scripts, downloaded files, build outputs, compiled binaries, caches, and exploratory artifacts should stay outside the final deliverable location unless the task explicitly asks for them. 7. Verify the result with task-relevant commands whenever possible. If full verification is expensive or unavailable, run targeted smoke checks that exercise the requested behavior. 8. Finish only after the environment is in the final state needed for grading. </WORKFLOW> <TOOL_USE> Use the bash tool for shell commands, builds, package installation, data processing, service checks, and running verification commands. Use the file editor tool for precise text-file inspection and edits. Prefer absolute paths for file operations. For text edits, view enough context first and ensure replacements are exact and unique. If several edits to one file are needed, batch them when practical. For multi-line code or structured text, prefer the file editor tool over fragile shell quoting; if you generate files with shell commands, read them back before running them. Do not add explanatory documentation or extra artifacts unless the task asks for them or they are required by the solution. </TOOL_USE> <LONG_RUNNING_WORK> Some tasks require slow downloads, dependency installation, compilation, model loading, or large data processing. For commands that may take longer than the default command timeout, pass an appropriate timeout value to the bash tool. If a long command times out or produces partial logs, inspect the logs, adjust the approach, and retry with a clearer command, a longer timeout, or a background process with log polling. When dependencies are missing, first look for existing project files such as requirements.txt, pyproject.toml, package.json, Cargo.toml, Makefile, or README instructions. Prefer the repository's declared setup path over ad-hoc package installs, but install missing dependencies when necessary to complete the task. Network downloads may be slow or partially blocked. For GitHub, Hugging Face, PyPI, apt, video sites, and large artifacts, first identify whether the issue is speed, availability, permission, rate limiting, or anti-bot behavior. For slow downloads, try reliable mirrors, resumable commands, or smaller range checks, then verify the downloaded file's size and

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

format. For video-site or anti-bot failures, look for legitimate mirrors, archives, or alternate sources before relying on an approximation. </LONG_RUNNING_WORK> <VERIFICATION> Do not mark the task complete just because a file was created, a command started, or an import succeeded once. Check the resulting files, imports, command output, tests, service behavior, or structured data that matter for the task. For JSON, CSV, databases, archives, compressed files, or generated reports, load or preview the artifact with an appropriate tool and confirm the schema and contents match the requirement. If a check fails, use the failure message to continue debugging instead of finishing. If the same approach fails repeatedly, inspect the relevant files, logs, or help output and switch to a different method rather than retrying blindly. When running provided checks or tests, confirm that the command you ran is the intended entry point. Empty output or a zero exit code is not enough if the task requires a specific file, exact one-line answer, schema, version, or absence of extra files. </VERIFICATION> <SAFETY_AND_SCOPE> Work only on changes needed for the task. Avoid destructive operations unless they are clearly required. Do not rely on hidden grader behavior or hard-code answers for a specific verifier; solve the stated task in the environment. The user will not provide interactive clarification during the run, so make reasonable assumptions and keep working toward a verified final state. </SAFETY_AND_SCOPE> When the task is complete, call the finish tool. Do not finish merely by saying the task is done in text.

CalibForge-Eval User Prompt Template Task: {instruction} Working directory: {workdir} Complete the task in the current runtime environment. The final state of this environment will be evaluated after you call finish.

D. Benchmark Decontamination Before trajectory distillation and model training, we compare every CalibForge candidate task against the evaluation instances in Terminal-Bench 2.0, SWE-bench Pro, and Doc2Repo (Merrill et al., 2026; Deng et al., 2025; Chen et al., 2026b). We first remove candidates whose instructions contain an exact 14-gram match with an evaluation instruction. We then compute 5-shingle Jaccard similarity over task instructions and available verifier or test code. Before computing shingles, we lowercase and tokenize the text, remove benchmark-specific boilerplate, and replace URLs, absolute file paths, and numeric values with canonical placeholders. The core similarity thresholds are 0.30 for task instructions and 0.45 for verifier or test code. Because contamination may remain despite surface-level differences in task instructions, we combine these similarity signals with structural evidence, including shared output paths, overlapping 24

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

test functions, and high-risk task-family matches. Any candidate flagged against at least one of the three evaluation benchmarks is removed before trajectory distillation and model training.

E. Supervised Fine-Tuning Details We train Qwen3-30B-A3B-Instruct and Qwen3.5-35B-A3B (Team, 2025; Qwen Team, 2026) using full-parameter, multi-turn SFT implemented with LLaMA-Factory (Zheng et al., 2024). Both models use the shared configuration in Table E1, and we report the final checkpoint after 10 epochs. Hyperparameter

Value

Hyperparameter

Value

Optimizer

AdamW (β 1 =0.9, β 2 =0.999) 1.0 × 10−5 cosine 0.05 0.0 1.0

Per-device train batch size

Gradient accumulation steps Global batch size Context length Precision GPUs

4 128 131,072 bf16 64 × NVIDIA H20

Learning rate LR scheduler Warmup ratio Weight decay Maximum gradient norm Epochs

Table E1. Supervised fine-tuning hyperparameters and hardware used for both CalibForge student models.

F. Failure Analysis of Trained Models We inspect Terminal-Bench 2.0 trajectories from three evaluation runs of both CalibForge-30BA3B and CalibForge-35B-A3B under the same CalibForge-Eval setting. The cases below were selected because their decisive actions, final artifacts, and verifier results establish a clear failure mechanism. Together, these cases expose distinct failure mechanisms that are not visible from aggregate benchmark scores alone.

F.1. Reasoning without Producing the Required Artifact Case G1: regex-log Task Requirement. Construct a regular expression that returns the last valid date from each log line containing a valid IPv4 address, and save it to /app/regex.txt for evaluation with Python re.findall. Observed Outcomes Model

Run 1

Run 2

Run 3

CalibForge-30B-A3B CalibForge-35B-A3B

FAIL PASS

Trajectory Evidence The 30B runs repeatedly constructed candidate expressions and exercised them in scratch scripts, but none wrote the required /app/regex.txt before reaching the time limit. In contrast, each 35B run materialized the selected expression in the required file and tested it using the same re.findall interface specified by the task, including lines with multiple dates and invalid IPv4-like strings.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Verifier Evidence All three 30B runs failed at the first check because /app/regex.txt did not exist. All three 35B runs passed the verifier. Diagnosis The failed runs made substantial progress on the regex itself but did not convert that progress into the required deliverable. This case separates solution exploration from task completion: useful intermediate reasoning is insufficient unless the agent commits the final artifact and checks it through the specified interface.

F.2. Committing to Partial Forensic Evidence Case G2: password-recovery Task Requirement. Recover a deleted 23-character password that starts with 8XD and ends with W54, and write each candidate password to /app/recovered_passwords.txt. Observed Outcomes Model

Run 1

Run 2

Run 3

CalibForge-30B-A3B CalibForge-35B-A3B

FAIL PASS

Trajectory Evidence Two 30B runs wrote PASSWORD=8XDK7VB3BV4W54, treating the stated length as if it included the PASSWORD= prefix; the third run produced no output file before timing out. The successful 35B runs instead combined the recoverable prefix 8XDP5Q2RT9Z with the separate suffix fragment K7VB3BV4WW54, then explicitly checked the reconstructed password’s length, prefix, suffix, and character set. Verifier Evidence The two completed 30B artifacts did not contain the recovered password, and the remaining run lacked the required file. All three 35B artifacts contained the same valid 23-character password and passed the verifier. Diagnosis The failed runs committed to a locally plausible fragment after misreading a global length constraint. Successful recovery required integrating evidence from separate regions of the forensic image and rechecking the combined result against every stated constraint.

F.3. Changing State before Preserving Recovery Evidence Case G3: db-wal-recovery Task Requirement. Repair an encrypted SQLite write-ahead log (WAL), recover all 11 records rather than the five records in the base database, and write them to /app/recovered.json. Observed Outcomes Model

Run 1

Run 2

Run 3

CalibForge-30B-A3B CalibForge-35B-A3B

FAIL FAIL

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Trajectory Evidence Across all six runs, the agent opened main.db with SQLite before copying or decrypting main.db-wal. SQLite returned the five base records and removed the unreadable WAL from the working directory. Subsequent commands could therefore no longer inspect or repair the file containing the six additional records. The 30B runs timed out without a valid recovery artifact, while the 35B runs attempted to reconstruct 11 records from the remaining base database. Verifier Evidence The 30B runs failed because the required recovery output was absent. The 35B outputs passed basic JSON and schema checks but failed both the record-completeness check and the check that the WAL had actually been decrypted. Diagnosis The first database query was not a read-only observation: it changed the state needed for the recovery. This case exposes the importance of preserving volatile evidence before using an application that may checkpoint, delete, or otherwise rewrite sidecar files.

Summary. Together, these cases reveal three distinct failure modes: failing to produce the required artifact, committing prematurely to partial evidence, and modifying mutable state before preserving recovery evidence. Trajectory inspection complements aggregate scores by showing how plausible intermediate progress can still fail to produce a verifier-confirmed solution.

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Analyze & explore related research in Conceptio

Related documents