arXiv:2605.14084v1 [cs.SE] 13 May 2026
CRANE: Constrained Reasoning Injection for Code Agents via Nullspace Editing
Mingzhi Zhu Rensselaer Polytechnic Institute Troy, NY 12180 [email protected]
Michele Merler IBM Research Yorktown Heights, NY 10598 [email protected]
Raju Pavuluri IBM Research Yorktown Heights, NY 10598 [email protected]
Stacy Patterson Rensselaer Polytechnic Institute Troy, NY 12180 [email protected]
Abstract Code agents must both reason over long-horizon repository state and obey strict tool-use protocols. In paired Instruct/Thinking checkpoints, these capabilities are complementary but misaligned. The Instruct model is concise and tool-disciplined, whereas the Thinking model offers stronger planning and recovery behavior but often over-deliberates and degrades agent performance. We present CRANE (Constrained Reasoning Injection for Code Agents via Nullspace Editing), a training-free parameter-editing method that treats the Thinking–Instruct delta as a directional pool of candidate reasoning edits for the Instruct backbone. CRANE combines magnitude thresholding to denoise the delta, a Conservative Taylor Gate to retain edits that are jointly beneficial for reasoning transfer and tool-use preservation, and Graduated Sigmoidal Projection to suppress format-critical update directions. By merging paired Instruct and Thinking checkpoints, CRANE delivers strong gains over either individual model while preserving Instruct-level efficiency: on Roo-Eval it achieves pass@1 of 66.2% (+19.5%) for Qwen3-30BA3B and 81.5% (+8.7%) for Qwen3-Next-80B-A3B; on SWE-bench-Verified it resolves up to 14 additional instances at both scales (122/500 and 180/500); and on Terminal-Bench v2 it improves pass@1/pass@5 by up to 2.3%/7.8%, reaching 7.6%/17.9% and 14.8%/30.3%, respectively, consistently outperforming alternative merging strategies across all three benchmarks. Code is available at https://github.com/rpi-nsl/CRANE.
1
Introduction
Modern code agents solve software tasks through long, structured interactions with repositories, tools, and execution environments. Systems such as SWE-agent [Yang et al., 2024] and OpenHands [Wang et al., 2024] make this setting explicit: the model must inspect files, issue edits, execute tests, and react to tool outputs under a constrained agent–computer interface, so success depends on both reasoning quality and protocol fidelity. Yet recent work shows that large reasoning models can sometimes overthink at substantial token cost while actually reducing performance [Liu et al., 2024, Li et al., 2025, Zhou et al., 2026]. We confirm this on Roo-Eval RooCodeInc [2026], where Thinking checkpoints underperform their Instruct counterparts at two scales, achieving 34.9% versus 46.7% pass@1 at 30B (Qwen3-30B-A3B) and 35.4% versus 72.8% at 80B (Qwen3-Next-80B-A3B), while consuming substantially more tokens. Based on these observations, this paper studies how Preprint.
to selectively inject the richer planning, context integration, and recovery behavior of Thinking checkpoints into Instruct backbones while strictly preserving the deployed agent protocol: concise tool timing, schema fidelity, and compact outputs. Prior model-merging works [Ilharco et al., 2023, Yu et al., 2024] and reverse-direction methods such as RAIN-Merging [Huang et al., 2026] have shown that weight-space editing and task-vector composition can combine capabilities across fine-tuned models without retraining. However, these methods are not designed for the asymmetric code-agent setting, where it is paramount to preserve an Instruct model’s tool protocol while importing only those Thinking-side directions that improve agentic reasoning. The challenge is not generic fusion but behavior-conditioned directional editing. We address this with CRANE (Constrained Reasoning Injection for Code Agents via Nullspace Editing), a training-free parameter-editing method that treats the Thinking–Instruct difference vector ( δ = θthink − θinst ) as a pool of candidate reasoning edits for the Instruct backbone. CRANE has three stages: (1) a magnitude-thresholding operator that sparsifies the raw delta and removes lowconfidence coordinates; (2) a Conservative Taylor Gate that estimates blockwise injection strength from masked calibration losses, assigning positive salience only when moving along the Thinking-toInstruct direction is first-order helpful for both reasoning transfer and tool-use preservation; and (3) a Graduated Sigmoidal Projection that uses format-critical Instruct activations to suppress update components that would perturb format-control tokens, tool delimiters, or JSON/schema structure. In short, CRANE denoises the candidate delta, retains only tool-safe reasoning directions, and attenuates edits in the protected format subspace. We demonstrate empirically that CRANE yields consistent gains across three agentic coding benchmarks (Roo-Eval RooCodeInc [2026], SWE-bench-Verified (SWE-V) [Jimenez et al., 2023, OpenAI, 2024], Terminal-Bench v2 (TB-V2) [Merrill et al., 2026]) and two model scales (Qwen3-30B-A3B and Qwen3-Next-80B-A3B). On Roo-Eval, CRANE raises pass@1 to 66.2% at 30B scale, well above the Instruct endpoint (46.7%) and the best alternative merge (47.2%) – and to 81.5% at 80B. On SWE-V it resolves the most instances of any merging baseline at both scales (122/500 and 180/500, respectively), and on TB-V2 it achieves the strongest pass@1/pass@5 results (7.6%/17.9% at 30B; 14.8%/30.3% at 80B). These gains come with practical efficiency: CRANE consistently attains the lowest or near-lowest token budget on Roo-Eval and SWE-V and controls Terminal-Bench wall time rather than trading success for verbosity. Ablations confirm that each component (sparsifier, Taylor gate, and format-preserving projection) contributes meaningfully to the success–cost frontier. Contributions. • A directional formulation of model merging for paired Instruct/Thinking models, where the Thinking–Instruct delta is treated as a candidate edit pool rather than a symmetric target; • CRANE, a training-free three-stage merge recipe that combines sparse delta extraction, tool-useaware Conservative Taylor Gating, and format-preserving Graduated Sigmoidal Projection; • A six-setting empirical evaluation across Roo-Eval, SWE-bench-Verified, and Terminal-Bench v2 showing more consistent gains than endpoint substitution or standard global merge baselines; • Ablations and sensitivity analyses that characterize which modules matter and how the performance–efficiency trade-off behaves around the selected merge scale and projection threshold.
2
Related Work
Model merging and sparse delta editing. A broad class of weight-space methods motivates sparse editing, but most prior work targets symmetric endpoint fusion, compression, or generic interference. Task-vector and merge-interference methods such as Task Arithmetic [Ilharco et al., 2023], TIES [Yadav et al., 2023], DARE [Yu et al., 2024], SLERP [Shoemake, 1985], RegMean [Jin et al., 2023], AIM [Nobari et al., 2025], LEWIS [Chopra et al., 2025], and Fisher-weighted merging [Matena and Raffel, 2022] combine or weight endpoint deltas, while pruning methods such as magnitude pruning [Han et al., 2015, Frankle and Carbin, 2019], Wanda [Sun et al., 2024], and SparseGPT [Frantar and Alistarh, 2023] show that many weights can be suppressed with limited immediate degradation. These methods are natural baselines because they edit the same weight-space object, but they do not condition the edit on code-agent behavior. In contrast, our setting is directional and behaviorconditioned. A coordinate is useful only if moving along the actual Thinking–Instruct delta improves reasoning while remaining compatible with tool-use preservation. 2
Task: python/scale-generator. Implement Scale with chromatic() (12-note scale from a tonic, using sharps C#/D#/… for sharp keys, flats Db/Eb/ … for flat keys) and interval(pattern) (walk the chromatic circle by M/m/A whole/half/augmented steps). Tonic casing is mixed ("F", "bb", "Db"); flat tonics must keep the lowercase b, so blindly upper-casing the input is wrong. 17 unit tests, judged via pytest.
(a) Instruct θI (fails) list_files read_file
× 21
scale_generator.py self.tonic = tonic.upper() ← bug
run_tests
(7fail/17)
apply_diff
…
45 tools total; never reads test file
FAIL 905s, 28k tokens context-blind
no-self-reflect.
Test output (line 770):
ValueError: Invalid
tonic:
DB
from Scale("Db") → tonic.upper() → "DB"
(c) CRANE θM (passes) — magnitude threshold
.
eeps strong reasoning edits
K
think 3.7k chars
scale_generator.py
apply_diff
(b) Thinking θT (fails) list_files
.
think 9.7k chars
— Taylor Gate
Injects tool-safe reasoning
think 98.5k chars
2. Block-wise Taylor gating
(single block)
—graduated sigmoidal proj.
“Wait, let me reconsider how the chromatic scale should handle flats vs sharps…
Actually, perhaps I should structure…”
P
rotects format directions
over-terse
Inner long snippet: “…I should reconsider whether to use sharps or flats first. Actually, let me think about this differently — perhaps the chromatic scale needs…”
scale_generator.py
read_file
✓ reads spec first
scale_generator_test.py
apply_diff
scale_generator.py
self.tonic = tonic.capitalize() ←root fix
↻ recover
scale_generator.py
FAIL 905s, 30k tokens
.
read_file
run_tests
apply_diff
malformed payload, no tests run
list_files
(2 fail / 17) post-fail retry
apply_diff scale_generator.py addresses all failure classes
run_tests
all 17 pass
PASS 226s, 8.8k tokens
Failure Classes (30B Roo-Eval) Instruct Thinking CRANE
n = 303 n = 371 n = 100 ✓ −67%
First applied diff (line 510): self.tonic=tonic.capitalize() use_sharps=tonic.find('#') != -1 preserves casing ⇒ both branches addressable
Figure 1: Qualitative Roo-Eval trace illustrating the endpoint trade-off that motivates selective injection. On python-scale-generator task, the Instruct endpoint acts quickly but edits before reading the relevant test and then loops on failed tool calls, while the Thinking endpoint shows stronger deliberation but still fails through overlong reasoning without re-testing. CRANE preserves the tool workflow while importing useful planning behavior: it reads the specification first, applies a fix, recovers after a partial failure, and passes all tests. The inset summarizes failure classes over failed Qwen3-30B-A3B Roo-Eval trajectories; two additional trace triples are reported in Appendix A.6.
Preservation-aware merging. A closer line of work asks which endpoint behavior should be protected while another capability is imported. RAIN-Merging [Huang et al., 2026] studies the complementary direction. It injects instruction-following ability into a reasoning model while preserving the reasoning model’s thinking format. CRANE reverses both the transfer direction and the protected behavior: we inject Thinking-derived reasoning behavior into an Instruct code agent and protect the agent protocol rather than a public chain-of-thought (CoT) format. Other merge variants control the update family rather than explicitly protecting a code-agent protocol: AdaMerging [Yang et al., 2023] learns per-layer scalars, and LoRA-merging methods [Huang et al., 2023] act on low-rank adapters rather than full deltas. Unlike these methods, our preservation mechanism protects activation subspaces tied to code-agent protocol tokens. Reasoning transfer in code-agent settings. A separate route to importing reasoning behavior is to retrain or distill the target model, but code-agent deployment is more constrained than standalone CoT imitation. Distillation-from-reasoning approaches [Magister et al., 2023, Guo et al., 2025] teach instruction models to emit CoT, but they re-train the student and must rebuild tool-use formatting from scratch. Code-agent systems and benchmarks such as Roo-Code/Roo-Eval, SWE-bench, SWE-agent, Terminal-Bench, and OpenHands instantiate long-context interactions over repository state, tool observations, and structured tool calls [Roo-Code Contributors, 2025, RooCodeInc, 2026, Jimenez et al., 2023, Yang et al., 2024, Merrill et al., 2026, Wang et al., 2024]. In this setting, useful standalone reasoning can still shift the interaction policy away from tool use, schema fidelity, context-budget discipline, or recovery from tool observations. Our method instead uses Thinking outputs only as calibration targets while the Instruct model supplies the preservation targets.
3
Method
Starting from a base model with weights θbase ∈ RD , let θinst ∈ RD denote an instruction-tuned codeagent checkpoint and θthink ∈ RD a paired reasoning-tuned checkpoint. We write δ = θthink − θinst for the Thinking–Instruct delta and use θmerged for the edited model. The desired endpoint is not a symmetric average. It is an Instruct-style agent that preserves the deployed tool interface of θinst while selectively importing the problem-solving ability exposed by θthink . 3
Paired endpoints
Magnitude Thresholding Before: dense
• tool-call discipline • format fidelity • compact outputs
+
• planning • reflection • recovery
DA
agent
behavior
preserve
0
...
format
token
preserve
Preserve dominant reasoning signal while removing weak noisy edits
Calibration sets
DF
2
...
− • keep top 50% entries • rescale survivors × 2 • drop low-magnitude noise
...
reasoning
transfer
Identify protected subspace from DF Activation matrix
+
➤
Merged Model
1
−
harmful drift /
long CoT / format drift
Output
Apply GSP Protection
→
After: sparse
Candidate edit pool
DR
3 Stage 3
Conservative Taylor
Gate from DR from DA
0
Thinking model
useful
reasoning signal
2 Stage 2
1 Stage 1
Instruct model
SVD
Graduated sigmoidal weights
plan
1
➤ Target R
Reasoning Transfer
Target A Tool-use preservation
➤
0
higher alignment 3
optimize edits that improve reasoning while preserving tool-use behavior
read
recover
Apply GSP projector
→
✓ ✓ ✓
stronger protection
➤
edit
test
→
suppress protocol-drifting directions
compact reasoning under preserved tool-call protocol
Figure 2: CRANE implementation pipeline with three stages: (1) Magnitude thresholding to sparsify δ and discard low-confidence coordinates; (2) Conservative Taylor Gate that sets per-block injection strength so only directions first-order beneficial to both reasoning and tool-use are retained; (3) Graduated Sigmoidal Projection that attenuates updates along format-critical subspaces (tool control). This asymmetric goal leads to three objectives. Reasoning transfer (R) uses Thinking-generated continuations conditioned on code-reasoning prompts, capturing planning, context integration, and recovery behavior that we want to inject. Format preservation (F ) uses Instruct-generated continuations on format-critical prompts, focusing on chat-template tokens, tool-call delimiters, JSON/schema syntax, and other local protocol markers. Agent-behavior preservation (A) also uses Instructgenerated continuations, but keeps broader action spans that encode when to call tools, when to read context, and when to stop. The objectives are complementary because large components of δ can carry Thinking-side reasoning behavior while overlapping with Instruct-side directions needed for format control and tool-use behavior. A naive linear merge θinst + αδ may improve reasoning transfer, but it can also damage the Instruct-side agent interface. We instead define a three-stage approach that addresses all three objectives (see Figure 2): (l,c) (l,c) (l,c) θmerged = θinst + ΠGSP α · S (c, l) · T (δ ) CTG τ,q(l,c) | {z } | {z } | {z } stage 2
stage 3
(1)
stage 1
where l ∈ {0, . . . , L − 1} indexes the transformer layer and c ∈ C indexes the parameter component, such as Q/K/V/O attention projections, expert gate/up/down projections, layer norms, and routers. Stage 1 removes low-confidence coordinates from δ via a conservative sparsifier T . Stage 2 then addresses objectives R and A by scoring whether each remaining δ direction is both reasoning-helpful and tool-safe. We develop a Conservative Taylor Gate (CTG), denoted SCTG (c, l), to determine the scaling coefficient for each block. Finally, in Stage 3, we address objective F using a Graduated Sigmoidal Projection (GSP), denoted ΠGSP τ,q(l,c) , to project out format-critical activation directions, where the index q(l, c) identifies the input-side activation space whose format-critical directions are protected. We instantiate the three objectives through model evaluation on three small calibration sets. The sets DR and DA are used to define masked losses for the CTG; DF is a set of format traces used to collect the activations protected by GSP. Appendix B gives construction details. 3.1
Stage 1: Denoising the Delta by Magnitude Thresholding
Since θmerged is obtained by adding an edited delta to θinst , each active coordinate moves an Instruct parameter toward its Thinking counterpart. Small delta entries are less likely to contribute meaningfully to reasoning transfer and may perturb the agent interface. We therefore use a conservative sparsification rule that edits only large-magnitude delta coordinates. Following prior sparse-delta 4
merging methods [Yadav et al., 2023, Yu et al., 2024], we construct a sparse approximation of δ using a deterministic median-magnitude threshold with rescaling: T (δ)j = 2 δj · mj (δ),
mj (δ) = 1{|δj | > median(|δ|)} .
(2)
Because the sparsification is deterministic rather than randomized, the factor of two serves only to approximately preserve the overall update scale. For mixture-of-expert layers, T is applied independently to each expert tensor. 3.2
Stage 2: Tool-Use-Aware Conservative Taylor Gate
Stage 1 reduces element-level noise but still applies a uniform scale to every component and layer. However, reasoning gains and tool-use risks are unevenly distributed across layer-component blocks. A single scale can over-inject fragile blocks while under-utilizing blocks that carry useful reasoning behavior. Stage 2, therefore, determines block-wise importance coefficients for more fine-grained edit scaling. We first formalize loss functions for the two objectives for R and A. For K ∈ {R, A}, let DK contain K K K K triples (xK i , yi , mi ), where xi is the prompt, yi is the endpoint-generated target continuation, K SiK K and mi P∈ P {0, 1} selects the target tokens that contribute to the loss. With ziK = [xK i ; yi ] and K MK = i s mi,s , define LK (θ) = −
1 XX K K K mi,s log pθ zi,s | zi,<s , MK i s
K ∈ {R, A}.
(3)
The implementation value mK i,s = 0 corresponds to an ignored label, so prompt tokens and irrelevant continuation positions do not contribute to the loss gradient. Local first-order expansion. Let gK = ∇θ LK (θinst ) denote the gradient of (3) for K ∈ {R, A}. For a small coordinate-wise update along the Thinking–Instruct merge direction, θinst + ηδj ej
(4)
where ej is the unit coordinate vector for the j-th entry of the flattened parameter vector, Taylor expansion gives LK (θinst + ηδj ej ) = LK (θinst ) + ηgK,j δj + O(η 2 δj2 ). (5) Thus, the first-order change in loss is proportional to gK,j δj . We define the coordinate-wise score sK (j) = −gK,j δj
(6)
so that sK (j) > 0 indicates that moving along the merge direction decreases LK to first order. Unlike Fisher-style importance measures [Matena and Raffel, 2022], sK (j) is signed and direction-aware. Conservative Taylor Gate. Reasoning transfer and tool-use preservation are not redundant signals. We therefore assign positive weight only to coordinates where the same infinitesimal edit is first-order beneficial for both losses. CTG uses the positive part of the minimum directional improvement score: pj = [min {sR (j), sA (j)}]+ ,
[u]+ = max{u, 0}.
(7)
Thus, pj > 0 only when the Thinking delta is a common descent direction for the reasoning loss and the tool-use preservation loss at coordinate j. A coordinate with large reasoning gain but negative tool-use effect receives zero score. Aggregation by component and layer. Let Bc,l ⊆ {1, . . . , D} be the index set for component c in layer l. We aggregate the coordinate scores and define the relative block coefficient directly: (b,l)
P
j∈Bc,l pj
SCTG (c, l) = P
j∈Bb,l pj
·
θinst
(c,l) θinst
F
(8)
F
where b is the per-layer FFN/expert component, b ∈ C. Bb,l is the union of the gate, up, and down projection indices for dense FFN layers or the union of gate/up/down indices across all experts for MoE layers. We normalize each component relative to the layer FFN/expert block b, which serves as a common reference scale across components. Because both numerator and denominator aggregate 5
coordinate scores, the coefficient is insensitive to the absolute scale of the losses. Using summed coordinate scores rather than per-parameter averages also preserves the cumulative CTG-positive contribution of larger blocks. The pre-projection block update is ∆θ(l,c) = α SCTG (c, l) T (δ (l,c) ) where α is a global merge-scale hyperparameter shared by all edited tensors. It controls the overall amount of Thinking–Instruct delta injected after median denoising and CTG component scaling. Appendix B.3 reports a robustness analysis for calibration-set choice. Different calibration subsets preserve the same component ordering and maintain Spearman correlation above 0.990. 3.3
Stage 3: Format-Preserving Graduated Sigmoidal Projection
Even an importance-weighted delta can violate the Instruct-side protocol if it changes the local computation at tokens that control chat templates, tool-call delimiters, JSON/schema syntax, braces, or schema-critical keys. Stage 3 addresses objective F to preserve these format-critical aspects. Let W represent the weights of a tensor in θinst , and let h represent the input activation vector corresponding to a token we seek to protect. If the merge proposes an edit ∆, the local output becomes (W + ∆)h = W h + ∆h. Preserving the Instruct computation at format positions asks for ∆h ≈ 0 on the protected format activations. We achieve this by applying a GSP to the proposed tensor edits. Our formulation borrows from activation-null-space methods used in factual-association editing [Meng et al., 2022, 2023] and continual-learning gradient projection [Saha et al., 2021] but replaces hard subspace truncation with a smooth sigmoid mask in singular-value space. Let IF denote the support of the format mask and Nρ (IF ) its local token neighborhood; Appendix D.1 gives both definitions. We index the tensors in θinst by q. Let hq (ziF , t; θinst ) ∈ Rdq be the input activation vector for token t at tensor q. The masked activation matrix and its singular value decomposition are Hq = hq (ziF , s; θinst ) (i,s)∈Nρ (I ) ∈ RNq ×dq , Hq = Uq Σq Vq⊤ . (9) F
Write Vq = [vq,1 , . . . , vq,rq ] for the right singular vectors and σq,1 ≥ · · · ≥ σq,rq for the corresponding singular values. We then have 2 Hq ∆ ⊤ q F =
rq X 2 ⊤ ⊤ 2 2 Σq Vq ∆q F = σq,r ∥∆q vq,r ∥2 . r=1
(10)
Directions with large σq,r are the input directions along which an edit most changes the Instruct computation at format-critical positions. Attenuating ∆q vq,r for these directions keeps the outputs close to the Instruct endpoint on the masked format traces. The neighborhood Nρ (IF ) extends this protection from literal delimiter tokens to nearby hidden states that condition on those tokens. Define the normalized singular amplitude aq,r = σq,r /σq,1 and a smooth protection coefficient 1 wq,r = . (11) 1 + exp(−k(aq,r − τ )) The slope k controls the width of the transition around the threshold τ ; Appendix D gives the exact parameterization. For a merge delta tensor ∆q ∈ Rdout ×dq , GSP applies the soft spectral projector ⊤ ΠGSP τ,q (∆q ) = ∆q − ∆q Vq diag(wq )Vq .
(12)
After projection, the component of the edit along vq,r is scaled by 1 − wq,r . The sigmoid mask avoids a hard null-space cutoff. High-amplitude format directions are removed almost completely, lowamplitude directions are largely left unchanged, and boundary directions receive partial attenuation that varies continuously with τ . This soft attenuation is better matched to long-context agentic traces. For tensors without a matching activation matrix, ΠGSP τ,q(l,c) is the identity. Appendix D gives the tensor-layout details, router handling, and the full merge algorithm.
4
Experiments
This section organizes the experiments around three research questions: RQ1: Does CRANE improve code-agent task success over the Instruct endpoint and standard merge baselines across IDE, 6
Table 1: Roo-Eval pass rates and token usage aggregated across five languages. Detailed results are in Appendix E Method
pass@1
pass@3
pass_all
TTC
Input tok.
Output tok.
Cached input
Instruct (ref) Thinking (ref)
91/195 (46.7) 68/195 (34.9)
125/195 (64.1) 103/195 (52.8)
63/195 (32.3) 35/195 (17.9)
181.1M 146.9M
43,548,016 21,057,008
8,372,134 22,786,455
957,076,451 119,597,157
Task Arithmetic TIES SLERP AIM-TA AIM-TIES LEWIS RAIN
92/195 (47.2) 92/195 (47.2) 85/195 (43.6) 91/195 (46.7) 88/195 (45.1) 87/195 (44.6) 77/195 (39.5)
119/195 (61.0) 129/195 (66.2) 114/195 (58.5) 126/195 (64.6) 120/195 (61.5) 123/195 (63.1) 106/195 (54.4)
65/195 (33.3) 57/195 (29.2) 58/195 (29.7) 57/195 (29.2) 57/195 (29.2) 54/195 (27.7) 42/195 (21.5)
208.1M 208.9M 214.6M 212.6M 211.3M 194.3M 140.2M
50,345,389 49,128,311 51,323,145 51,338,605 50,606,755 48,090,553 20,409,513
8,011,542 7,644,147 8,418,811 7,914,166 8,090,525 7,657,204 21,681,930
1,177,364,978 1,215,445,711 1,211,975,312 1,216,900,832 1,202,205,511 1,079,258,386 113,698,415
CRANE
129/195 (66.2)
162/195 (83.1)
86/195 (44.1)
120.9M
34,678,861
8,759,443
424,474,281
Qwen3-30B-A3B
Qwen3-Next-80B-A3B Instruct (ref) Thinking (ref)
142/195 (72.8) 69/195 (35.4)
170/195 (87.2) 97/195 (49.7)
104/195 (53.3) 44/195 (22.6)
89.6M 109.5M
27,444,388 18,152,937
6,128,842 16,630,299
314,987,867 81,763,409
Task Arithmetic TIES SLERP AIM-TA AIM-TIES LEWIS RAIN
153/195 (78.5) 154/195 (79.0) 143/195 (73.3) 157/195 (80.5) 149/195 (76.4) 155/195 (79.5) 90/195 (46.2)
173/195 (88.7) 172/195 (88.2) 169/195 (86.7) 171/195 (87.7) 177/195 (90.8) 176/195 (90.3) 114/195 (58.5)
132/195 (67.7) 121/195 (62.1) 118/195 (60.5) 129/195 (66.2) 119/195 (61.0) 121/195 (62.1) 50/195 (25.6)
93.1M 89.0M 97.6M 100.0M 96.0M 95.9M 113.2M
27,492,207 26,783,953 28,915,441 28,687,721 28,855,031 28,113,529 17,933,387
6,284,994 6,346,889 6,283,713 6,703,140 6,689,030 6,631,916 17,375,213
341,909,682 305,139,154 372,314,291 377,874,779 337,415,124 345,905,209 83,718,010
CRANE
159/195 (81.5)
176/195 (90.3)
139/195 (71.3)
89.2M
26,567,238
6,072,681
322,364,655
repository, and terminal workflows? (Tables 1, 2, 3); RQ2: Do the success gains preserve a compact, Instruct-like rollout footprint, rather than relying on higher aggregate token cost, longer wall time, or Thinking-style output growth? (Tables 1, 2, 3, Figure 3); and RQ3: What is the contribution of each component of CRANE, namely sparse candidate extraction, CTG importance estimation, and format-preserving projection, to the final performance–cost trade-off? (Table 4, Figure 4). 4.1
Setup
Models and benchmarks. We evaluate three tool-using code-agent settings: Roo-Eval, a fivelanguage in-IDE suite; SWE-bench-Verified (SWE-V), a repository-level issue-resolution benchmark; and Terminal-Bench v2 (TB-v2), a long-horizon shell-workflow benchmark. SWE-V and TB-v2 use the OpenHands scaffold [Wang et al., 2024]; harness details are in Appendices A.2 and A.3. For all three datasets, we evaluate paired Instruct/Thinking checkpoints on two different architectures within the same family, at two scales: Qwen3-30B-A3B-Instruct/Thinking-2507 [Yang et al., 2025] and Qwen3-Next-80B-A3B-Instruct/Thinking [Cao et al., 2026]. Baselines and efficiency metrics. We compare the original checkpoints with Task Arithmetic, TIES, SLERP, AIM, LEWIS, and RAIN-Merging; hyperparameters and AIM details are in Appendices A.4 and A.4.1. All models are served locally with vLLM [Kwon et al., 2023]. We report TTC = Ni + 0.1Nc + 5No as an aggregate rollout-footprint proxy, using output tokens and TB-v2 wall time to distinguish compact gains from inflated traces; accounting details are in Appendices A.1. 4.2 Benchmarks Results Roo-Eval Results. For RQ1, CRANE improves over the Instruct endpoint by +19.5, +19.0, and +11.8 percentage points on 30B pass@1, pass@3, and pass_all, respectively; relative to the strongest non-CRANE row for each metric, the corresponding margins are +19.0, +16.9, and +10.8 points. At 80B, CRANE improves over Instruct by +8.7 points on pass@1 and +18.0 points on pass_all, beats the strongest non-CRANE pass@1/pass_all rows by +1.0 and +3.6 points, and is within 0.5 points of the best pass@3 row. For RQ2, Roo-Eval shows that these gains are not purchased by longer outputs or larger TTC. At 30B, CRANE reduces TTC by 60.2M tokens relative to Instruct and by 19.3M relative to the lowest-TTC non-CRANE row while improving all three success metrics. At 80B, CRANE stays within 0.2M TTC of the lowest-TTC alternative and slightly below the Instruct endpoint, while cutting more than 10M output tokens relative to the Thinking and RAIN rows. Figure 3 visualizes the same success–TTC trade-off across all three benchmarks. 7
Task Arithmetic
TIES
(c) 30B · TB v2
40
20 better 15 10
150
200
5
TTC (M)
10
15
7 better 6
5
TTC (B)
75
pass@1 (%)
better
50
pass@1 (%)
25
60
SLERP
AIM-TA
(d) 80B · Roo-Eval
100
80 better
60
40 90
TTC (M)
100
110
AIM-TIES
LEWIS
RAIN
(e) 80B · SWE-bench
(f) 80B · TB v2 15.0
35 better
30
pass@1 (%)
CRANE
(b) 30B · SWE-bench Resolved (%)
pass@1 (%)
70
Thinking
Resolved (%)
Instruct
(a) 30B · Roo-Eval
12.5 better 10.0
25 5
TTC (M)
10
15
7.5 100
TTC (B)
200
300
TTC (M)
Figure 3: TTC vs. pass-rate, three benchmarks × two scales. (a–c) Qwen3-30B-A3B on Roo-Eval, SWE-bench-Verified, Terminal-Bench v2; (d–f) Qwen3-Next-80B-A3B on the same three. Table 2: SWE-bench-Verified results. Resolved cells report count (resolved%). TTC is the same token-usage proxy as Table 1. Qwen3-30B-A3B
Qwen3-Next-80B-A3B
Method
Resolved
Input tok.
Output tok.
Cached input
TTC
Resolved
Input tok.
Output tok.
Cached input
TTC
Instruct (ref) Thinking (ref)
108 (21.6%) 47 (9.4%)
2.16B 479M
353M 2.15B
81.1B 31.0B
12.04B 14.33B
168 (33.6%) 125 (25.0%)
1.96B 1.21B
315M 2.10B
23.6B 25.1B
5.90B 14.22B
Task Arithmetic TIES SLERP AIM-TA AIM-TIES LEWIS RAIN
109 (21.8%) 110 (22.0%) 110 (22.0%) 113 (22.6%) 111 (22.2%) 110 (22.0%) 58 (11.6%)
1.59B 1.66B 1.49B 1.61B 1.66B 1.64B 0.50B
322M 299M 331M 313M 350M 303M 2.05B
50.0B 48.5B 46.5B 50.3B 54.6B 46.6B 29.3B
8.20B 8.01B 7.80B 8.21B 8.87B 7.82B 13.68B
169 (33.8%) 162 (32.4%) 169 (33.8%) 172 (34.4%) 169 (33.8%) 173 (34.6%) 120 (24.0%)
1.82B 1.91B 1.79B 1.81B 1.80B 1.90B 1.22B
318M 342M 326M 336M 311M 312M 2.00B
20.7B 22.4B 20.5B 20.4B 19.0B 19.9B 24.7B
5.48B 5.86B 5.47B 5.53B 5.26B 5.45B 13.69B
CRANE
122 (24.4%)
1.41B
373M
24.0B
5.68B
180 (36.0%)
1.81B
309M
18.6B
5.22B
SWE-bench-Verified Results. For RQ1, CRANE resolves 14 more instances than the Instruct reference, 9 more than the strongest merging baseline, and 75 more than Thinking at 30B. The corresponding 80B gains are +12 over Instruct, +7 over the strongest merging baseline, and +55 over Thinking. For RQ2, CRANE reaches those higher resolved counts with lower aggregate token cost. Its TTC is 6.36B lower than Instruct and 2.12B lower than the lowest-TTC baseline at 30B. At 80B, the savings are 0.68B relative to Instruct and 0.04B relative to the lowest-TTC non-CRANE row. Thus the repository-level gains are not an artifact of spending more total token budget. Terminal-Bench v2 Results. Terminal-Bench v2 evaluates shell-tool agents on long-horizon command-line workflows in cloud sandboxes. We run the 89-task public reporting subset of the tb2-zai dataset [Z.ai, 2026] at k = 5 attempts/task to match the public Terminal-Bench leaderboard. For RQ1, CRANE improves over the strongest non-CRANE rows by +1.5 points on pass@1 and +3.3 points on pass@5 at 30B, and by +0.6 and +3.3 points at 80B. For RQ2, Terminal-Bench provides the clearest wall-time evidence for a compact rollout footprint. At 30B, CRANE is 1h 56m faster than Instruct and 24m faster than the fastest non-CRANE row, while reducing output by 1.73M tokens relative to Instruct. At 80B, CRANE is 30m faster than Instruct and only 3m slower than the fastest row, while staying within 0.03M output tokens of the lowest-output row. The claim is therefore not that every raw token column is minimal, but that CRANE sits on a better success–footprint frontier with more compact successful rollouts. Cross-benchmark summary. Across Tables 1–3, plain merge baselines sometimes improve over a reference checkpoint, especially at 80B, but the gains are inconsistent and RAIN often retains Thinking-like over-deliberation. CRANE turns the endpoint complementarity into more reliable gains across benchmarks and scales while keeping the rollout footprint compact. 4.3
Ablations
We use ablations to answer RQ3: which parts of the recipe are needed for the observed performance– cost trade-off? One ablation study disables one module at a time (T (δ), CTG Taylor scaling, or GSP), while another evaluates the effect of varying the values of the global merge scale α and the GSP threshold τ within a range. Component-importance ablations. Table 4 shows that no single component can be removed without changing the trade-off. On Roo-Eval 30B, removing GSP causes the largest success drop: −14.9, −11.3, and −12.3 points on pass@1, pass@3, and pass_all. Removing Taylor or the sparsifier is less destructive on pass@1/pass@3 but still costs 8.8/3.6 and 5.7/3.6 points, respectively; the sparsifier removal is the only variant that improves pass_all, by 2.1 points. On Roo-Eval 80B, the full 8
Table 3: Terminal-Bench v2 main results. Test time is the end-to-end harness wall time. Tokens are in millions and Input counts non-cached prefill tokens. Other details are reported in Appendix F. Qwen3-30B-A3B
Qwen3-Next-80B-A3B
Method
pass@1
pass@5
Test time
Input
Output
pass@1
pass@5
Test time
Input
Output
Instruct (ref) Thinking (ref) Task Arithmetic TIES SLERP AIM-TA AIM-TIES LEWIS RAIN CRANE
4.8 (5.4%) 5.2 (5.9%) 4.8 (5.4%) 5.4 (6.1%) 4.8 (5.4%) 5.0 (5.6%) 5.0 (5.6%) 4.6 (5.2%) 5.0 (5.6%) 6.8 (7.6%)
9 (10.1%) 12 (13.5%) 13 (14.6%) 12 (13.5%) 13 (14.6%) 12 (13.5%) 12 (13.5%) 10 (11.2%) 9 (10.1%) 16 (17.9%)
4h 14m 4h 37m 2h 50m 2h 53m 2h 51m 2h 44m 2h 42m 2h 53m 4h 05m 2h 18m
16.96 4.34 8.54 9.97 7.13 7.18 9.47 7.00 4.01 7.68
5.43 18.41 3.77 4.40 3.80 3.85 4.33 3.70 16.76 3.70
12.0 (13.5%) 6.0 (6.7%) 11.6 (13.0%) 11.8 (13.3%) 12.0 (13.5%) 12.2 (13.7%) 12.6 (14.2%) 12.6 (14.2%) 7.0 (7.9%) 13.2 (14.8%)
20 (22.5%) 12 (13.5%) 22 (24.7%) 23 (25.8%) 24 (27.0%) 20 (22.5%) 22 (24.7%) 23 (25.8%) 14 (15.7%) 27 (30.3%)
2h 28m 5h 12m 2h 10m 1h 55m 2h 08m 2h 00m 2h 14m 2h 11m 4h 57m 1h 58m
10.84 4.45 266.39 11.71 12.96 10.10 301.41 10.59 4.36 10.42
3.85 20.39 3.65 3.86 3.55 3.72 3.62 3.74 19.35 3.58
Table 4: Component-removal ablations. Each row disables one module of CRANE. The upper block reports Roo-Eval; the lower block reports Terminal-Bench v2 and SWE-bench-Verified. Per-variant token breakdowns are in Appendix G, Tables 34–35. Qwen3-30B-A3B
Qwen3-Next-80B-A3B
Roo-Eval Method
Roo-Eval
pass@1
pass@3
pass_all
TTC
pass@1
pass@3
pass_all
TTC
CRANE w/o T (δ) CRANE w/o Taylor CRANE w/o GSP
118/195 (60.5) 112/195 (57.4) 100/195 (51.3)
155/195 (79.5) 155/195 (79.5) 140/195 (71.8)
90/195 (46.2) 68/195 (34.9) 62/195 (31.8)
142.3M 145.7M 100.8M
154/195 (79.0) 151/195 (77.4) 152/195 (77.9)
177/195 (90.8) 179/195 (91.8) 176/195 (90.3)
129/195 (66.2) 123/195 (63.1) 117/195 (60.0)
97.8M 106.2M 109.7M
CRANE (T (δ)+Taylor+GSP)
129/195 (66.2)
162/195 (83.1)
86/195 (44.1)
120.9M
159/195 (81.5)
176/195 (90.3)
139/195 (71.3)
Terminal-Bench v2
Terminal-Bench v2
89.2M SWE-V
pass@1
pass@5
TTC (M)
Resolved / TTC (B)
pass@1
pass@5
TTC (M)
Resolved / TTC (B)
CRANE w/o T (δ) CRANE w/o Taylor CRANE w/o GSP
6.80 (7.6%) 5.80 (6.5%) 4.80 (5.4%)
12 (13.5%) 14 (15.7%) 11 (12.4%)
94.1 85.1 42.5
120 (24.0%) / 8.43 106 (21.2%) / 7.34 94 (18.8%) / 5.35
12.20 (13.7%) 11.60 (13.0%) 11.40 (12.8%)
21 (23.6%) 22 (24.7%) 19 (21.3%)
52.8 50.4 57.3
164 (32.8%) / 5.51 162 (32.4%) / 5.50 175 (35.0%) / 5.35
CRANE (T (δ)+Taylor+GSP)
6.80 (7.6%)
16 (17.9%)
58.1
122 (24.4%) / 5.68
13.20 (14.8%)
27 (30.3%)
51.8
180 (36.0%) / 5.22
60 0.15 0.20 0.25 0.30 0.35 (a) merge scale α
80 75 70
3e-3 3e-2 3e-1 (b) GSP thresh. τ (log)
20 18 16 14 12 10 0.15 0.20 0.25 0.30 0.35 (c) merge scale α
20 18 16 14 3e-3 3e-2 3e-1 (d) GSP thresh. τ (log)
26 24 22 20 18 0.15 0.20 0.25 0.30 0.35 (e) merge scale α
SWE-V resolved (%)
70
85
SWE-V resolved (%)
80
22
TB v2 pass@5 (%)
90
Roo-Eval pass@3 (%)
Roo-Eval pass@3 (%)
90
TB v2 pass@5 (%)
Method
SWE-V
26
24
22
20
3e-3 3e-2 3e-1 (f) GSP thresh. τ (log)
Figure 4: Continuous-hyperparameter sensitivity analysis of the CRANE recipe on Qwen3-30B-A3B across three benchmarks, grouped by benchmark. All α sweep at τ =0.03 and τ sweep at α=0.25 on a log axis. (a)–(b) Roo-Eval pass@3. (c)–(d) TB-V2 pass@5. (e)–(f) SWE-V resolved. Stars mark the reported configuration; Roo-Eval sweep values are tabulated in Appendix G, Table 36. recipe improves pass@1 over all component removals by 2.5–4.1 points and pass_all by 5.1–11.3 points, while remaining within 1.5 points of the best pass@3 variant. The lower block gives the same module removals on Terminal-Bench v2 and SWE-bench-Verified. On Terminal-Bench v2, the full recipe gains +4.4 points in 30B pass@5 over the only variant that ties its pass@1, and improves 80B pass@5 by 5.6–9.0 points over all removals. On SWE-bench-Verified, the full recipe resolves 2–28 more 30B instances and 5–18 more 80B instances than the component-removal variants. These results support RQ3 as a trade-off statement: the full recipe is strongest on the primary success metrics, while individual removals can improve isolated secondary metrics or cost. Hyperparameter sensitivity analysis. The reported configuration was selected on Roo-Eval only, transfers to TB-v2 and SWE-V without per-benchmark tuning, and remains stable near the chosen point. The inner sweep neighborhood stays within ∼2.5 absolute points across all three benchmarks.
5
Limitations
First, CRANE assumes complementary paired endpoints: the Thinking checkpoint must provide useful reasoning behavior, and the Instruct checkpoint must define a useful deployment interface. If future Thinking models are already strong in task success, token efficiency, and tool discipline, a 9
simpler endpoint choice or global merge may be competitive. Second, the calibration sets must also cover the deployed tool surface; substantial drift in tools, formatting, or stopping behavior would require re-calibration. Third, the format-subspace SVD requires forward passes through the Instruct backbone on the 430 format traces, which can dominate wall-clock cost on very large models. Fourth, Java and Rust on Roo-Code remain weaker than Python/JS/Go for Qwen3-30B-A3B, suggesting asymmetric coverage in the underlying Thinking-model training rather than a pure merge artifact.
10
References Pierre Ablin, Gabriel Peyre, and Michael Sander. Do residual neural networks discretize neural ordinary differential equations? In Advances in Neural Information Processing Systems, 2022. BerriAI. LiteLLM: Open source ai gateway for 100+ llms. https://github.com/BerriAI/ litellm, 2026. Accessed: 2026-04-30. Ruisheng Cao, Mouxiang Chen, Jiawei Chen, Zeyu Cui, Yunlong Feng, Binyuan Hui, Yuheng Jing, Kaixin Li, Mingze Li, Junyang Lin, et al. Qwen3-coder-next technical report. arXiv preprint arXiv:2603.00729, 2026. Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018. Hetarth Chopra, Vidhi Rambhia, and Vikram S Adve. Lewis (layer wise sparsity)-a training free guided model merging approach. In Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference, 2025. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pages 10323–10337. PMLR, 2023. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023. Zhehao Huang, Yuhang Liu, Baijiong Lin, Yixin Lou, Zhengbao He, Hanling Tian, Tao Li, and Xiaolin Huang. Rain-merging: A gradient-free method to enhance instruction following in large reasoning models with preserved thinking format. In The Fourteenth International Conference on Learning Representations, 2026. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023. Zhiyuan Li, Yi Chang, and Yuan Wu. Think-bench: Evaluating thinking efficiency and chain-ofthought quality of large reasoning models, 2025. URL https://arxiv.org/abs/2505.22113. Ryan Liu, Jiayi Geng, Addison J. Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L. Griffiths. Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse, 2024. URL https://arxiv.org/abs/2410.21333. 11
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773–1781, 2023. Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703–17716, 2022. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022. Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. Amin Heyrani Nobari, Kaveh Alim, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, and Navid Azizan. Activation-informed merging of large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. OpenAI. Introducing swe-bench verified. introducing-swe-bench-verified/, 2024.
https://openai.com/index/
Podman contributors. Podman: A tool for managing OCI containers and pods. https://github. com/containers/podman, 2026. Accessed: 2026-04-29. Roo-Code Contributors. Roo-code: An open-source in-ide coding agent. https://github.com/ RooCodeInc/Roo-Code, 2025. GitHub repository. RooCodeInc. Roo Code Evals: Eval exercises for roo code. https://github.com/RooCodeInc/ Roo-Code-Evals, 2026. GitHub repository. Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021. Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pages 245–254, 1985. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In 12th International Conference on Learning Representations, ICLR 2024, 2024. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, et al. Livebench: A challenging, contamination-limited llm benchmark. In The Thirteenth International Conference on Learning Representations, 2025. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in neural information processing systems, 36:7093–7115, 2023. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02575, 2023. 12
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024. Z.ai. terminal-bench-2-verified: Z.ai-verified fork of terminal-bench 2.0 with environment and instruction fixes. https://huggingface.co/datasets/zai-org/terminal-bench-2-verified, 2026. Hugging Face dataset, accessed 2026-05-02. Shu Zhou, Rui Ling, Junan Chen, Xin Wang, Tao Fan, and Hao Wang. When more thinking hurts: Overthinking in llm test-time compute scaling, 2026. URL https://arxiv.org/abs/2604. 10739. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024.
13
A
Experimental Details
A.1
Roo-Eval Evaluation
Each checkpoint is evaluated on five programming languages with three independent rollouts per exercise. The exercise counts are Python 34, JavaScript 50, Go 36, Java 45, and Rust 30, for 195 exercises and 585 total rollouts per complete sweep. Table 5: Roo-Eval serving, judging, and reference-cost protocol used by the result logs.
A.2
Item
Setting
Languages Rollouts Sampling Context length Eval concurrency 80B serving Cost accounting Metrics
Python, JavaScript, Go, Java, Rust 3 independent iterations per exercise temperature 0.6, top_p 0.8, top_k 20 90000 64 vLLM 0.19.0, TP=4, expert parallel enabled, 4×H100 80GB Local vLLM serving; reported dollar values are token-usage reference proxies pass@1, pass@3, pass_all, iteration pass, reference cost proxy
SWE-bench-Verified Harness
SWE-bench-Verified runs use the OpenHands [Wang et al., 2024] agent scaffold over the 500-instance verified subset. All checkpoints are served locally by vLLM [Kwon et al., 2023] under the same TP/EP configuration as Roo-Eval; the harness drives OpenHands via litellm [BerriAI, 2026]. Table 6 records the scaffold and harness settings used for every row of Table 2. Table 6: SWE-bench-Verified scaffold, container, and harness configuration used for all rows of Table 2. Item
Setting
Subset Agent scaffold Max iterations Sampling Serving GPU Context length Container backend Image registry Per-instance deadline Agent / harness workers
SWE-bench-Verified, 500 instances OpenHands SDK [Wang et al., 2024] 100 per instance temperature 0.6, top_p 0.8, top_k 20 (Qwen3 defaults) vLLM [Kwon et al., 2023], bf16, TP= 4 4×H100 80GB 131072 rootless podman [Podman contributors, 2026] Epoch AI ghcr mirror 60 min wall-clock; main-thread join cap 61 min 24 / 24
Sampling. Without top_k the Qwen3 checkpoints occasionally drift into long hallucinated continuations that never emit a finish action. We adopt the Qwen3-recommended top_k = 20 for every row in Table 2, including endpoint references. This setting standardizes decoding across endpoints and reduces stalled-rollout effects in token-usage estimates. The litellm transport timeout is set to 90 s with 5 retries: the empirical p99 of per-call latency is ∼3 s, so 90 s gives ∼18× headroom on legitimate calls and bounds unresponsive calls at ∼8 min instead of the OpenHands default of ∼30 min. Token accounting. Input tokens are non-cached prefill tokens, computed as accumulated_token_usage.prompt_tokens − cache_read_tokens. Cache-read tokens are prompt tokens served by vLLM’s prefix cache (requires –enable-prompt-tokens-details). Completion tokens are model outputs. Across SWE-bench-Verified rollouts the agent-loop context is heavily redundant across iterations, and we observe a ∼97% prefix-cache hit rate; cached input is therefore a large term for concise Instruct/merge rows, while output tokens dominate the TTC of over-deliberative Thinking and RAIN rows. Since the cost of input, cached input and output tokens is 14
different for all major providers, we define the Total Token Count (TTC) as a weighted sum of the number of tokens as follows:
T T C = wi Ni + wc Nc + wo No = Ni + 0.1Nc + 5No
(13)
where Ni is the number of input tokens, Nc is the number of input cached tokens and No is the number of output tokens. Fixing the input tokens weight wi as 1, the weights wc , wo of the other token types were estimated as an industry average from the data reported in Table 7. Note: in all our experiments we run the models using local vLLM, therefore Total Token Count is used as a proxy to estimate the budget of running those models through providers, not actual incurred spending. Table 7: Token cost for major frontier lab providers used to estimate relative weights in total tokens count, and average cost ratios of token types relative to input tokens. Prices listed from official providers as of 05/04/2026. Input (per 1M tokens)
Cached Input (per 1M tokens)
Output (per 1M tokens)
Cached / Input
Output / Input
Claude Opus 4.7 Claude Sonnet 4.6 Claude Haiku 4.5
$5.00 $3.00 $1.00
$0.50 $0.30 $0.10
$25.00 $15.00 $5.00
0.10×
5×
GPT-5.5 GPT-5.4 GPT-5.4 Mini
$5.00 $2.50 $0.75
$1.25 $0.25 $0.075
$30.00 $15.00 $4.50
0.10–0.25×
4–6×
Gemini 3.1 Pro Gemini 3.1 Flash Gemini 2.5 Pro Gemini 2.5 Flash
$2.00 $0.25 $1.25 $0.30
$0.20 $0.025 $0.125 $0.03
$12.00 $1.50 $10.00 $2.50
0.10×
6–8×
DeepSeek
V4 Pro V4 Flash
$1.74 $0.14
$0.0145 $0.0014
$3.48 $0.28
∼0.01×
2×
Kimi
Kimi K2.6 Kimi K2.5
$0.74 $0.60
$0.185 $0.15
$3.49 $2.50
0.25×
4–5×
∼0.1×
∼5×
Provider
Model
Anthropic
OpenAI
Industry avg.
1×
Container backend: podman replacing Docker. Our cluster has no Docker daemon and no /etc/subuid entries for the user, so we run all SWE-bench eval images under rootless podman [Podman contributors, 2026]. Two consequences flow from the missing subuid range: (i) podman’s namespace is single-UID, so the host UID maps to container UID 0 and nothing else is valid; (ii) the upstream swebench harness’s copy_to_container tars files with the host UID and calls put_archive, which podman rejects with lchown ... invalid argument. We patch swebench.harness.docker_utils.copy_to_container to force uid=gid=0 in the tarinfo filter; the same patch is applied to every fresh swebench install in the eval venv. The harness reaches podman via DOCKER_HOST=unix:///.../podman.sock (podman system service –time=0); the OpenHands adapter shells out to podman run/exec directly and does not use the API socket. A.3
Terminal-Bench v2 Harness
Terminal-Bench v2 [Merrill et al., 2026] evaluates shell-tool agents on long-horizon command-line workflows. We run the official openhands reference agent against the tb2-zai dataset [Z.ai, 2026] on Daytona cloud sandboxes; Table 8 records the harness configuration used for every row of Table 3. Reporting denominator. The five excluded tasks fail to launch reliably under our default Daytona sandbox spec budget. Each excluded task is counted as failed for every model, preserving the 89-task denominator. This matches the Terminal-Bench leaderboard convention and keeps every method comparable. Daytona cost accounting. Daytona is the only component of Terminal-Bench v2 with real billable cash flow. We pull per-sandbox lifetimes from the audit-log API 15
Table 8: Terminal-Bench v2 scaffold, sandbox, and reporting configuration used for all rows of Table 3. Item
Setting
Dataset Excluded tasks
tb2-zai public reporting subset (89-task denominator) pytorch-model-cli, count-dataset-tokens, mcmc-sampling-stan, rstan-to-pystan, reshard-c4-data 89 (matches public Terminal-Bench leaderboard) openhands (standard online, in-sandbox) — official reference agent 5 temperature 0.6, top_p 0.8, top_k 20 longest-first 30B: 20 trials in parallel; 80B: 24 trials in parallel Daytona cloud sandboxes 300 s sweep interval, 75 min sandbox age cap vLLM, TP= 4, bf16, 4×H100 80 GB, 131,072 ctx, prefix caching on –tool-call-parser hermes; –reasoning-parser qwen3 on Thinking only GPT-5.4 nano: $0.20 / $0.02 / $1.25 per 1M input / cached / output tokens GPT-5.4 mini: $0.75 / $0.075 / $4.50 per 1M input / cached / output tokens 1 vCPU $0.0504/hr; mem $0.0162/hr/GiB; disk $0.000108/hr/GiB (5 GiB free) 1 vCPU / 2 GiB / 10 GiB (∼80% of trials) → $0.08334/hr per sandbox ∼80% 1c/2g/10d; ∼16% 1c/4g/10d; ∼4% 2c/4g/10d or 1c/8g/10d
Reporting denominator Agent scaffold Attempts per task (k) Sampling Schedule Concurrency Sandbox runtime Watchdog Serving Tool/reasoning parsers 30B reference schedule 80B reference schedule Daytona unit pricing Default sandbox spec Observed spec mix
(/api/audit/organizations/{orgId}) — every create (with cpu/mem/disk spec) and delete timestamp is recorded — and cost each sandbox at the per-spec rate in Table 8. Per-trial agent_execution sums under-count by ∼30% (they miss sandbox boot/teardown overhead and retries) and naive fleet-wall integration over-counts by ∼7%; the audit-log version is authoritative and matches the Daytona dashboard. The 30B sweep audit log contains 3,925 billable sandbox creations; we therefore cost actual create/delete lifetimes rather than infer cost from a nominal trial count. Reasoning-parser configuration on Thinking. Without –reasoning-parser qwen3, vLLM serves Thinking-checkpoint outputs with <think> blocks landing in the assistant content field, which then accumulates into next-turn prompts and inflates input-token traffic. Every Thinking row in Table 3 uses the parser-enabled setting. LLM cost. Same convention as Roo-Eval and SWE-bench-Verified: “LLM $” is a token-usage proxy under the GPT-5.4 nano (30B) or mini (80B) schedule; we serve self-hosted Qwen3 on local vLLM, so the dollar values are not incurred spending. We list this proxy in Appendix F.1 alongside the actual Daytona cost (which is incurred against our Daytona invoice, modulo the $200 free credit) and the total. Tunnels and quota separation. 30B and 80B sweeps run on separate alphagpu nodes with dedicated Cloudflare tunnels (qwen-30b.mzhi.men/v1, qwen-80b.mzhi.men/v1) and separate Daytona organizations so quota cascades on one scale do not corrupt the other. The 80B ties run was originally interrupted at 6 min by a 300 GB Daytona quota cascade and was rerun cleanly under the same harness; the rerun is the row reported in Table 3. A.4
Baseline Hyperparameters
Baseline rows use the method’s paper setting when it fixes the relevant value; otherwise we report the best completed Roo-Eval configuration available for that method at the corresponding scale. Table 9 lists the selected settings used in the main tables. Table 9: Selected baseline hyperparameters for the Roo-Eval results. Method
30B setting
80B setting
Selection note
Task Arithmetic TIES SLERP AIM-TA AIM-TIES LEWIS RAIN
α = 0.30 α = 0.30, density = 0.50 t = 0.30 α = 0.30, ω = 0.40 α = 0.30, density = 0.50, ω = 0.40 α = 0.30, γ = 0.30, ϵ = 0.80, density = 0.50 Plan-A qkvof reproduction, Thinking proxy base, scaling factor 0.50
α = 0.15 α = 0.15, density = 0.50 t = 0.15 α = 0.15, ω = 0.40 α = 0.15, density = 0.50, ω = 0.40 α = 0.15, γ = 0.30, ϵ = 0.80, density = 0.50 Plan-A qkvof reproduction, Thinking proxy base, scaling factor 0.30
Best completed Roo-Eval setting Best completed Roo-Eval setting Best completed Roo-Eval setting AIM weighting applied to Task Arithmetic AIM weighting applied to TIES Importance-weighted density schedule Reverse-direction diagnostic baseline
16
A.4.1
AIM variants.
AIM is implemented as a channel-wise relaxation on the update produced by another merge rule. For in a Linear weight Wq ∈ Rdout ×din , let mq ∈ Rd≥0 be the input-channel activation magnitude recorded on the Instruct checkpoint and let sq,j =
mq,j , maxj ′ mq,j ′
rq,j = 1 − (1 − ω)sq,j ,
ω = 0.40,
(14)
when maxj ′ mq,j ′ > 0; otherwise the AIM scaler leaves the update unchanged. The AIM-adjusted update is applied column-wise, e q,:,j = rq,j ∆q,:,j . ∆ (15) Thus channels that are highly activated by the Instruct model are protected by shrinking the merge update toward an ω fraction, while low-importance channels keep nearly the full update. AIM-TA sets ∆q = α(θthink,q − θinst,q ). AIM-TIES first computes the usual TIES update after trimming, sign election, and disjoint averaging at density 0.50, and then applies the same AIM relaxation to the final α-scaled update. Biases, embeddings, layer norms, rotary buffers, and Linear weights without a matching AIM importance vector are left unchanged by the AIM post-processing step. A.5
Failure-Mode Analysis
The failure-mode distribution panel in Figure 1 (lower bridge column) reports rule-based audits of failed Roo-Eval rollouts on 30B for three model variants. The Instruct-side 3-class taxonomy serves as the primary axis; Thinking and CRANE failures are mapped onto it (§below). 30B-Instruct audit (303 failed rollouts). One run per language: Python 52, JavaScript 64, Go 72, Java 57, Rust 58. Each failed rollout is bucketed by parsing its JSONL tool-use stream and applying: • over-terse: ≤ 6 finalized tool events or ≤ 1 test cycle. The agent converges prematurely without producing an implementation attempt. • context-blind: ≥ 2 edits with ≤ 1 read, or no read of the test file before editing. The agent fires edits before inspecting the specification scaffold. • no-self-reflection: ≥ 3 test runs with repeated failure signatures, or ≥ 3 commands + ≥ 3 edits. The agent repeats the same approach across multiple failed attempts. Counts: over-terse 88, context-blind 10, no-self-reflection 205. A 28-rollout human spot-check (10 over-terse, 8 context-blind, 10 no-self-reflection) agrees with the rule-based label on 23/28 cases (82%). The systematic skew is at the over-terse / no-self-reflection boundary: rollouts that fail at the first edit-test cycle and idle are sometimes labeled no-self-reflection by the rule but read as over-terse to a human. The relative ordering no-self-reflection ≫ over-terse ≫ context-blind is preserved. 30B-Thinking audit (371 failed rollouts). Canonical run dirs 20260413_205546 (Python), 20260414_052932 (JavaScript), 20260414_060714 (Go), 20260414_064105 (Java), 20260414_072117 (Rust). Thinking-native rule labels are mapped to the 3-class taxonomy: • over-terse: ≤ 1 test cycle (Thinking-native: premature-end; budget exhausts at the 900 s timeout without a productive edit→test cycle). • no-self-reflection: a single </think>-bounded inner-monologue block ≥ 20k chars, OR think text ≥ 50% of total assistant output and total think ≥ 30k chars (Thinking-native: monolithic-think; counts as no-self-reflection because the rollout never alternates between deliberation and tool feedback). • context-blind: n = 0 in Thinking — the model engages with the spec via <think> even when over-deliberating. Counts under the 3-class mapping: over-terse 131, context-blind 0, no-self-reflection 240. The no-self-reflection share decreases slightly from 67.7% (Instruct) to 64.7% (Thinking), but with a different mechanism: Instruct retries the same failing approach, Thinking deliberates without testing. 17
30B-CRANE audit (100 failed rollouts). Canonical run dirs 20260420_020103 (Python), 20260420_022201 (JavaScript), 20260420_025032 (Go), 20260420_031541 (Java), 20260420_035312 (Rust); model identifier crane-simple-v2-router-only-pl-nodh-a025-newgsp. Same 3-class scheme applied. Counts: over-terse 1, context-blind 0, no-self-reflection 99 — a 67% reduction in total reasoning failures vs Instruct and a 73% reduction vs Thinking, with Instruct-side over-terse and context-blind modes near-eliminated and Thinking-style monolithic deliberation suppressed (no <think> blocks appear in any CRANE log). Schema-error accounting. Tool-execution failures where the harness rejected an apply_diff payload as malformed or non-matching are tracked separately from the reasoning-failure taxonomy and are not included in the counts above. They affect both Thinking and CRANE traces and reflect a tool-protocol factor orthogonal to the planning/reflection/recovery axis the audit is designed to measure. Over-terse exemplar. python-transpose-iter3-attempt4.log. The agent reads the stub and the test file, then switches to architect mode and asks a clarifying question about trailing-space handling rather than implementing the function: listFilesRecursive docs → readFile transpose.py → readFile transpose_test.py → switchMode architect → ask_followup_question("Should the function handle trailing spaces ...")
The trace contains no edit or test execution. Although the test file specifies the expected behavior, the rollout terminates before implementation. Context-blind exemplar. javascript-forth-iter1-attempt3.log. The agent reads only the stub forth.js and never opens forth.spec.js; it then makes three edits guessing the API before running tests for the first time: readFile forth.js appliedDiff forth.js (constructor) appliedDiff forth.js (get stack) appliedDiff forth.js (evaluate) execute_command pnpm test # forth.spec.js never opened
This trace violates the read-before-edit criterion: the specification file defines the API, but the generated implementation is based only on the stub. No-self-reflection exemplar. python-zipper-iter3-attempt3.log. After an initial failing test run, the agent applies a near-identical edit to zipper.py’s to_tree method four consecutive times, each followed by an identical pytest signature: EDIT zipper.py (set_left) FAIL .....FFFFFF..F EDIT zipper.py (to_tree, identical) FAIL .....FF.FFF..F EDIT zipper.py (to_tree, identical) FAIL .....FF.FFF..F EDIT zipper.py (to_tree, identical) FAIL .....FF.FFF..F ... 12 test cycles, signature unchanged after the first
Across 12 test cycles, the failure signature remains unchanged; the trace contains no subsequent test reread, diagnostic instrumentation, or alternative implementation attempt.
18
A.6
Additional Qualitative Trace Triples
Figure 1 reports a single triple on python-scale-generator. The two additional triples below were chosen for the same property (Instruct fails, Thinking fails, CRANE succeeds on iter1) and exhibit different but consistent failure modes. javascript-parallel-letter-frequency. • Instruct (javascript-parallel-letter-frequency-iter1.log): 20 tool calls, zero edits. The trace contains 14 consecutive searchFiles calls with an empty regex and no edits before the harness emits Roo appears to be stuck in a loop. • Thinking (javascript-parallel-letter-frequency-iter1.log): 12 tool calls but with 47k characters of inner monologue between attempts; four separate appliedDiff revisions on the same Unicode-aware regex regress from 1 failing test to 8 failing tests, then time out. • CRANE (javascript-parallel-letter-frequency-iter1.log): single shot, 7 tools: list_files → list_files → read_file parallel-letter-frequency.js → read_file parallel-letter-frequency.spec.js → appliedDiff → pnpm install → pnpm test (PASS, all tests). 305 s, 4k output tokens. javascript-tournament. • Instruct (javascript-tournament-iter1.log): 21 tool calls, 6 edits, 4 test runs without convergence; 38k output tokens, 912 s timeout. • Thinking (javascript-tournament-iter1.log): 8 tool calls dominated by 112k characters of inner monologue, 2 edits, 2 test runs, no recovery; 40k output tokens. • CRANE (javascript-tournament-iter1.log): 9 tools, single attempt: list_files → read_file stub → read_file spec → short todo → appliedDiff → pnpm test (PASS). 79 s, 2.1k output tokens. The pattern in both triples mirrors Figure 1: Instruct either edits without reading the specification or repeatedly invokes search tools; Thinking allocates most output tokens to inner monologue; CRANE reads the test/spec file before the first edit and converges in one or two cycles.
19
B
Calibration and Signal Computation
This section separates method-internal calibration details from benchmark protocol. The reported recipe uses the paper calibration set below, while the public-source subsets in §B.3 are reserved for the calibration-set robustness analysis. B.1
Calibration Set Construction
The Taylor gate uses behavior targets, not hand-written output labels. The Thinking checkpoint supplies reasoning-transfer targets and the Instruct checkpoint supplies agent-behavior preservation targets. Table 10 summarizes the calibration inputs: DR and DA are the only masked-loss sets used by CTG, while DF is a format-trace set used only to build GSP activation projectors. Table 10: Calibration inputs used by the Taylor and GSP stages. The reported merge recipe uses DR and DA as masked-loss sets for CTG; DF provides format traces for GSP and does not define a loss. Public-source subsets are robustness checks only. Set
Size
Construction
Target generator
Role
DR DA DF format
36 16 430
Original code-agent reasoning prompts: 20 SWE-bench-style, 12 LiveBench-coding-style, 4 LiveCodeBench-style Original Roo-style tool-use repair prompts: 14 SWE-bench-style, 2 LiveBench-coding-style Instruct traces around format-critical tool tokens and local neighborhoods
Thinking Instruct Instruct
Reasoning-transfer loss Agent-behavior preservation loss Format activations for GSP; no loss
Reasoning-transfer set DR . The paper calibration set contains 36 DR prompts. They are original rewrites in code-agent reasoning styles inspired by SWE-bench [Jimenez et al., 2023], LiveBench coding [White et al., 2025], and LiveCodeBench. They cover debugging, concurrency, migrations, caching, pagination, parser edge cases, large backfills, rate limiting, pathfinding, and test-design tradeoffs. Each prompt is rendered as a user message; the Thinking checkpoint greedily generates the assistant target. The masked loss is then evaluated at the Instruct endpoint on the generated assistant span. Agent-behavior set DA . The paper calibration set contains 16 DA prompts. They are original Roo-style repository repair instructions. They ask the model to inspect relevant files, patch the smallest correct change, run focused tests, audit scripts or docs, and report intentional non-edits. The Instruct checkpoint generates the preservation target. This set activates the same tool-use and response-format behavior that must be preserved when injecting Thinking-derived deltas. Format-trace set DF . The 430 format traces are used only for GSP and do not define a masked loss. We locate format-token positions and local neighborhoods in Instruct traces, collect hidden states at the protected sites, and build per-component spectral projectors. The Taylor score itself does not use DF . B.2
Taylor Signal Computation
For each coordinate j, let δj = θthink,j − θinst,j . At the Instruct endpoint, we compute gradients of the masked reasoning and agent-behavior losses: gR = ∇θ LR (θinst ),
gA = ∇θ LA (θinst ).
(16)
The equations are written over the full parameter vector, but the implementation computes them shardwise: each shard stores its local entries of gR , gA , and δ, forms local coordinate scores, and contributes the relevant block sums. The signed first-order improvements along the actual merge direction are sR (j) = −gR,j δj , sA (j) = −gA,j δj . (17) The Conservative Taylor Gate (CTG) gives positive salience to a coordinate only when the same infinitesimal edit is beneficial for both objectives: pj = [min{sR (j), sA (j)}]+ .
(18)
Component/layer scores are obtained by summing pj within a block, normalizing by the Instruct parameter norm of that block, and then reporting all components in expert units. The normalization is not a cardinality correction: a block with more CTG-positive coordinates can receive a larger 20
aggregate score even after Frobenius normalization. This is a salience aggregation step rather than a per-coordinate Taylor mask: the final tensor update uses the thresholded delta T (δ (l,c) ) scaled by the scalar SCTG (c, l). The anchor is the per-layer FFN/expert pseudo-component b: dense FFN layers use the union of gate/up/down projections, while MoE layers use the union of gate/up/down projections across all expert replicas. The router is not part of this anchor. Figure 5 shows the resulting Qwen3-30B table. Sreason(c, l) on Qwen3-30B-A3B
2.5
Attention Expert (FFN)
1.5
LayerNorm
1.0
Sreason
2.0
0.5
Router
0.0 0
4
8
12
16
20
24
28
32
36
40
44
47
Layer index l
Figure 5: CTG Taylor importance SCTG (c, l) on Qwen3-30B-A3B, derived automatically from DR and DA in Table 10. Rows: components (Q, K, V, O, expert gate/up/down, norm, router, LM head); columns: layers 0–47. Late-layer attention, mid-depth experts, and the routing gate dominate; norm and LM head receive near-zero injection. B.3
Robustness to Calibration Set Choice
We assess the robustness of the CTG Taylor salience used by CRANE to calibration-set choice. On Qwen3-30B-A3B, we recompute the full layer-component salience table under five independently sampled public calibration subsets, while holding the model pair, target decoding protocol (TR =4096, TT =2048), layer chunking, and merge equations fixed. The analysis isolates calibration-set variation from the rest of the merge pipeline. Public mix construction. Each public_mix_seed{s} subset has the same 36+16 prompt budget as the paper calibration set. The frozen reasoning pool has 80 public code-reasoning prompts: 40 from LiveCodeBench code generation and 40 from BigCodeBench [Zhuo et al., 2024]. The frozen tool-use pool has 80 SWE-bench issue prompts [Jimenez et al., 2023], excluding SWE-bench Verified instance ids [OpenAI, 2024], wrapped as Roo-style repository repair prompts. For seed s, a seeded Python RNG samples 18 LiveCodeBench prompts, 18 BigCodeBench prompts, and 16 SWE-bench prompts without replacement. Items are sorted by source and id before writing the JSONL, making the prompt hash deterministic. Table 11: Robustness to calibration-set choice on Qwen3-30B-A3B. Public mix seeds use 18 LiveCodeBench prompts, 18 BigCodeBench prompts, and 16 SWE-bench issue/tool prompts. Pearson/Spearman are computed over flattened layer-component scores for attention/router/norm against the paper calibration set. Calibration paper calibration public_mix_seed0 public_mix_seed1 public_mix_seed2 public_mix_seed3 public_mix_seed4
|DR |/|DA |
Attention
Expert
Router
Norm
Pearson
Spearman
Top-10
Top-20
Top-30
Top-48
36/16 36/16 36/16 36/16 36/16 36/16
1.7912 1.7904 1.7706 1.7908 1.7720 1.8066
1.0000 1.0000 1.0000 1.0000 1.0000 1.0000
0.3225 0.3378 0.3399 0.3423 0.3447 0.3349
0.0151 0.0161 0.0161 0.0153 0.0159 0.0164
1.0000 0.9862 0.9868 0.9856 0.9853 0.9877
1.0000 0.9917 0.9911 0.9913 0.9906 0.9920
10/10 7/10 6/10 7/10 6/10 7/10
20/20 15/20 15/20 14/20 14/20 15/20
30/30 26/30 25/30 25/30 25/30 25/30
48/48 46/48 46/48 46/48 46/48 46/48
Findings. The five public mix seeds preserve the same component ordering as the paper calibration set, attention > expert baseline > router ≫ norm. Their Pearson correlations against the paper calibration set are 0.9853–0.9877 and Spearman correlations are 0.9906–0.9920; the top-48 overlap is 46/48 for every public seed. Per-component variation is small: attention CV is 0.84%, router CV is 1.12%, and norm remains near zero. Thus, the layer-component salience table used by the merge is insensitive to these calibration-set redraws at the level that determines component ordering and high-salience layer selection. 21
Table 12: Dispersion of the five public mix seeds. CV is the coefficient of variation across seeds; drift is relative to the paper 36/16 calibration value.
B.4
Component
Mean
Std.
CV
Drift vs. paper calibration
attention expert router norm
1.7861 1.0000 0.3399 0.0160
0.0150 0.0000 0.0038 0.0004
0.0084 0.0000 0.0112 0.0254
-0.28% +0.00% +5.39% +5.38%
Runtime and Artifacts
Table 13: Measured CRANE signal-computation and merge runtimes. Rows report wall-clock time on the listed hardware; for the 80B Taylor row, the parenthetical gives single-GPU-equivalent time. GSP projector construction is a one-time reusable cost. Stage
Wall time
30B instruct model load on 2×H100 30B Taylor signal on 2×H100 30B GSP projector build, 96 hidden-state components on 2×H100 30B final merge on one H100, 16 shards 30B end-to-end signal to merged model, reusing GSP projectors 30B end-to-end including GSP projector rebuild
∼28 s ∼6 min 179 s (∼3.0 min) ∼4 min ∼10 min ∼13 min
80B instruct model load on 4×H100 80B Taylor signal on 4×H100 80B GSP projector build, 96 hidden-state components on 4×H100 80B final merge on one H100, 41 shards 80B end-to-end signal to merged model, reusing GSP projectors 80B end-to-end including GSP projector rebuild
∼30 s ∼27 min ∼13 min 461.7 s (∼7.7 min) ∼35 min ∼48 min
These costs are one-time preprocessing and merge costs rather than fine-tuning. GSP projector construction can be reused across nearby merge-scale sweeps for the same Instruct endpoint and format-trace set, and the Taylor-signal and elementwise-merge steps are naturally shardable.
22
C
Architecture-Normalized Taylor
This section gives the derivation behind the hybrid-MoE normalization used for the Qwen3-Next-80B recipe. The main text defines CTG at the layer/component level. We keep that granularity here and use architecture families only to supply an exposure correction. Within this appendix only, let c̄ = ϕ(c) map a raw parameter component to an architecture-level family such as full-attention, linear-attention, experts, norms, or routers. The Qwen3-Next recipe replaces the main coefficient by arch SCTG (c, l) =
P (b,l) θinst 1 j∈Bc,l pj F · . ·P (c,l) κ(ϕ(c)) j∈Bb,l pj θinst
(19)
F
Here b is the per-layer FFN/expert pseudo-component defined in the main text: the union of gate/up/down projections, across all expert replicas for MoE layers, excluding the router. Eq. 19 does not sum salience across components in the same family; Q/K/V/O projections, routers, and expert projections keep their own CTG evidence and parameter-norm normalization. The family map only determines the residual-occupation multiplier κ. When κ(ϕ(c)) ≡ 1, Eq. 19 is exactly the main-text coefficient. The normalization is an exposure correction for a residual stack rather than a model of the relative output scale or expressivity of full- and linear-attention layers. C.1
Residual Occupation Measure
Consider a residual transformer block whose token mixer in layer l has family τl : hl+1 = hl + Mτl ,l (hl ) + El (hl ),
τl ∈ {full, linear},
(20)
where Mτl ,l is the attention or linear-state mixer and El denotes the remaining expert/MLP branch. This residual-stack view is consistent with the continuous-depth interpretation of residual networks as ODE discretizations [Chen et al., 2018, Ablin et al., 2022]. Let a merge induce a small mixer perturbation ∆Mτl ,l . If el is the hidden-state error between the original and merged networks at layer l, then first-order linearization gives el+1 = (I + Jl )el + ∆Mτl ,l (hl ) + O(∥el ∥2 + ∥el ∥ ∥∆Mτl ,l ∥),
(21)
where Jl = ∂(Mτl ,l + El )/∂hl . Dropping higher-order terms and unrolling, eL ≈
X
PL,l+1 ∆Mτl ,l (hl ),
PL,l+1 =
l
L−1 Y
(I + Jm ).
(22)
m=l+1
Thus the endpoint perturbation contributed by a mixer family is a sum over the layers in which that family appears. If the transported perturbations are bounded by a comparable layerwise scale ac for family c, then B(c) ≡
X
PL,l+1 ∆Mc,l (hl ) ≲ Λ µ(c) ac ,
l:τl =c
µ(c) =
X
1{τl = c},
(23)
l
for a transport bound ∥PL,l+1 ∥ ≤ Λ. The linear dependence on µ(c) is the conservative case for coherent parameter shifts. A square-root dependence would require treating per-layer perturbations as independent zero-mean noise; because the Instruct-to-Thinking delta is a directed model edit, coherent accumulation is the conservative modeling choice.
23
C.2
Full Attention Versus Linear Attention
A causal full-attention mixer has the form Mfull,l (h)t = WlO
X
softmax
s≤t
⊤ ql,t kl,s √ d
! vl,s .
(24)
s
A Gated DeltaNet-style linear mixer can be abstracted as a recurrent state-space operator, Sl,t = Γl,t Sl,t−1 + Ul (kl,t , vl,t , Sl,t−1 ), ⊤ Mlinear,l (h)t = WlO ql,t Sl,t ,
(25) (26)
with gates, normalization, local convolution, and state-update details absorbed into Γl,t and Ul . Equations 24–25 show that full and linear attention implement different token-mixing operators. They do not imply 1 ∥Mlinear,l (h)∥ ≈ ∥Mfull,l (h)∥. (27) 3 Layerwise output scale is learned and depends on projections, gates, normalization, recurrent decay, and sequence statistics. The factor used in the merge instead follows from matching family-level residual exposure. Let full attention be the reference family. To keep the integrated first-order update from family c comparable to the reference, Eq. 23 suggests µ(c)ac ≈ µ(full)afull ,
ac µ(full) . ≈ afull µ(c)
(28)
Qwen3-Next-80B has µ(linear) = 36 and µ(full) = 12, so the architecture coefficient is κ(linear) =
36 µ(linear) = = 3. µ(full) 12
(29)
Since Eq. 19 divides by κ, each linear-attention layer receives one third of the per-layer merge budget assigned to an otherwise comparable full-attention reference. This is an occupation correction: linear attention appears three times as often in the residual stack, so equal per-layer injection would give the linear family roughly three times the integrated first-order exposure. unit cell: L L L F
linear attn
layer 0
23
24
47
µ(linear) = 36, κ(linear) =
µ(full) = 12,
full attn
µ(full) is the reference
µ(linear) 36 = = 3 µ(full) 12
Figure 6: Qwen3-Next-80B residual stack laid out as 48 mixer slots: linear-attention layers (blue) repeat three times for every full-attention layer (orange), giving µ(linear) = 36 and µ(full) = 12. The 3:1 occupation is the geometric source of κ(linear) = 3 in Eq. 29. If activation-side measurements are available, the architecture-only coefficient can be generalized to κmeas (c) =
µ(c)ameas (c) , µ(cref )ameas (cref )
ameas (c) = El:τl =c, hl ∼Dcal [∥∆Mc,l (hl )∥] .
(30)
Here ameas (c) estimates the absolute layerwise perturbation scale ac in Eq. 23. We intentionally do not normalize by ∥hl ∥: the transport bound above controls absolute endpoint perturbations, while a relative output-to-state ratio would measure a different quantity. The experiments in this paper use the architecture-only version, ameas (c) ≈ ameas (cref ), because the merge statistics are intended to be computed once from masked losses and reused across model shards. 24
D
GSP Implementation Details
This section records the implementation-level details omitted from the main CRANE description. GSP does not optimize a format loss; the format traces provide only the mask support IF for protocolcontrol positions. GSP then expands IF to a local neighborhood before collecting activations. D.1
Token Neighborhood
For the format traces DF , the format-mask support is F F F IF = {(i, s) : (xF i , yi , mi ) ∈ DF , mi,s = 1}.
(31)
The experiments then use the symmetric token-window expansion Nρ (IF ) = {(i, t) : ∃(i, s) ∈ IF with |t − s| ≤ ρ, 1 ≤ t ≤ SiF }.
(32)
We set ρ = 2. The window is applied within each trace before collecting activations, clipped to valid token positions, and deduplicated. It is not a separate causal mask; causal dependence is already determined by the hidden states produced by the decoder at each selected token. D.2
SVD Derivation of the GSP Projector
This subsection expands the main-text derivation for Eq. 10 and Eq. 12. Fix an edited tensor and its protected activation space indexed by q, meaning the input-side activation space used to construct that tensor’s format-preserving projector. Orient the edited tensor as a linear map Wq ∈ Rdout ×dq , where dq is the dimension of the protected input activation. The notation q(l, c) in the main text maps a layer/component tensor to this input-activation space. For a selected format-neighborhood activation xn ∈ Rdq , the local output perturbation induced by an additive edit ∆q is (Wq + ∆q )xn − Wq xn = ∆q xn .
(33)
Stacking all selected activations row-wise gives ⊤ x1 Nq X .. Nq ×dq 2 Hq = . ∈ R , Eq (∆q ) = ∥∆q xn ∥22 = ∥Hq ∆⊤ q ∥F . x⊤ Nq
(34)
n=1
Thus GSP uses Eq (∆q ) as a local output-preservation surrogate: edits with small Eq leave the immediate module outputs nearly unchanged on the masked format traces. This is local to the selected module outputs and is not a global guarantee after downstream nonlinear layers. Let the compact SVD of Hq be Hq = Uq Σq Vq⊤ ,
Vq = [vq,1 , . . . , vq,rq ],
Σq = diag(σq,1 , . . . , σq,rq ),
(35)
with σq,1 ≥ · · · ≥ σq,rq > 0. By Frobenius-norm invariance under the left-orthogonal factor Uq , 2 ⊤ ⊤ 2 Eq (∆q ) = ∥Uq Σq Vq⊤ ∆⊤ q ∥F = ∥Σq Vq ∆q ∥F =
rq X
2 σq,r ∥∆q vq,r ∥22 .
(36)
r=1
The right singular vectors are the relevant directions because the weight edit acts on the input activation dimension: vq,r is an input-space direction, and ∆q vq,r is the output change caused by editing along that direction. Large σq,r therefore identifies an input direction that occurs strongly in format-critical traces, so preserving format behavior asks us to suppress the corresponding edit component. A hard activation-nullspace projection would choose a protected set Pq and remove those components: X ⊤ I − Πhard vq,r vq,r . (37) Pq (∆q ) = ∆q r∈Pq
CRANE instead uses a smooth mask over singular directions. Define normalized amplitudes σq,r aq,r = , σq,1 25
(38)
and protection weights wq,r = sigmoid(k(aq,r − τ )) ∈ [0, 1]. The resulting operator is ⊤ ⊤ ΠGSP . τ,q (∆q ) = ∆q − ∆q Vq diag(wq )Vq = ∆q I − Vq diag(wq )Vq
(39)
For each retained singular vector, ΠGSP τ,q (∆q )vq,r = (1 − wq,r )∆q vq,r .
(40)
Therefore high-amplitude format directions are nearly removed, low-amplitude directions are mostly unchanged, and boundary directions are partially attenuated. Substituting Eq. 40 into Eq. 36 gives the post-projection local surrogate Eq (ΠGSP τ,q (∆q )) =
rq X
2 σq,r (1 − wq,r )2 ∥∆q vq,r ∥22 .
(41)
r=1
Directions orthogonal to span(Vq ) are unconstrained by the observed activation matrix and pass through unchanged. If no activation matrix with matching input dimension is collected for a tensor, or if the collected matrix is numerically zero, the implementation uses the identity operator for that tensor. D.3
Sigmoid Weighting
The experiments use τ = 0.03 and set k = log(99)/τ ≈ 4.6/τ in Eq. 11; for the default τ = 0.03, this gives k ≈ 153.3. The constant 4.6 is the rounded logit log(0.99/0.01) = log(99), chosen so that the sigmoid protection coefficient is approximately 0.01 at aq,r = 0, 0.5 at aq,r = τ , and 0.99 at aq,r = 2τ . The transition from w ≈ 0.01 to w ≈ 0.99 therefore occurs over approximately [0, 2τ ] = [0, 0.06], so directions near the boundary receive partial attenuation rather than a discontinuous hard projection. Figure 7(a) plots wq,r for several τ values. The smooth transition makes ΠGSP τ,q vary continuously with τ , whereas a hard projector can switch a direction from fully removed to fully retained under a small numerical change in aq,r . Figure 7(b) visualizes the energy-weighted residual mask profile of the sigmoid mask against polynomial soft masks (w = a2 , a3 ) and a hard top-k mask across depth. (a) sigmoid weighting wq, r
0.8
τ = 0.003 τ = 0.03 τ = 0.3
0.6
dots: w(τ) = 0.5
(b) accumulated leakage vs.\ depth Energy-weighted leakage budget
protection weight wq, r
1.0
0.4
0.2
0.0 10−4
100 10−1 10−2 10−3 10−4 10−5 10−6
sigmoid GSP, τ = 0.03 (CRANE) polynomial w = a 2 polynomial w = a 3 hard top-k, k = 4
10−7 10−3
10−2
10−1
100
8
16
24
32
40
48
Depth L (residual layers traversed)
normalized singular amplitude aq, r = σq, r/σq, 1
Figure 7: GSP sigmoid-weighting diagnostics. (a) Sigmoid weighting wq,r = σ(k(aq,r − τ )) with k = log(99)/τ for τ ∈ {0.003, 0.03, 0.3}; the dot marks w(τ ) = 0.5 and the transition band [0, 2τ ] contains all partial attenuation. (b) Energy-weighted residual mask profile along format-protected directions across residual depth for the sigmoid mask, polynomial soft masks (w = ap ), and a hard top-k mask.
26
D.4
Tensor Orientation
Equation 12 is written for tensors whose protected input-activation dimension is on the right, ∆q ∈ Rdout ×dq . If a stored parameter tensor places that dimension on the left, the implementation applies the same operator after transposing the tensor and then transposes the result back. This changes only the array layout, not the mathematical projection. D.5
Protected Activation Map
The main-text notation q(l, c) maps each layer/component tensor to the input-side activation space used to build its GSP projector. For a linear map whose weight can be oriented as ∆q ∈ Rdout ×dq , q(l, c) indexes the activation vector multiplied by that weight in the forward pass. GSP is therefore an input-side projector for the edited weight matrix. For Q/K/V, routers, and FFN/expert gate/up projections, the protected input is the residual stream. For output projections and expert down projections, the protected input, when collected, is the attention/mixer or MLP intermediate activation rather than the residual stream. Tensors without a collected activation matrix of matching input dimension, such as scalar biases or unsupported buffers, use the identity projector. D.6
Complete Merge Algorithm
Algorithm 1 CRANE merge implementation Require: θinst , θthink , masked-loss sets DR , DA , format-trace set DF , GSP projectors {Vq , σq }q , scale α, threshold τ Ensure: θmerged 1: δ ← θthink − θinst 2: compute gR = ∇θ LR (θinst ) and gA = ∇θ LA (θinst ) 3: for each j: sR (j) ← −gR,j δj , sA (j) ← −gA,j δj , pj ← [min{sR (j), sA (j)}]+ 4: aggregate normalized CTG salience into SCTG (c, l) for each layer/component block 5: for each parameter tensor θ (l,c) do 6: δ̂ ← T (δ (l,c) ) 7: δ̂ ← α SCTG (c, l) δ̂ 8: δ̂ ← ΠGSP τ,q(l,c) (δ̂) (l,c)
(l,c)
θmerged ← θinst + δ̂ 10: end for 11: return θmerged 9:
27
E
Roo-Eval Detailed Results
This section collects the Roo-Eval results used in the main paper. Figure 8 gives a visual overview of pass@1 and pass_all across both scales. Sections E.1–E.2 report the per-language tables for the main 30B and 80B-Next comparisons. Unlike the headline totals in Table 1, these tables retain the full log metrics: pass@1, pass@3, pass-all, rollout-level pass count, reference-cost proxy, and input/cached/output token counts. Sections E.3–E.5 give compact pass@1, pass@3, and pass_all summaries by language. The α/τ sweep tables and component-removal ablations are collected separately in Appendix G. pass@1
pass_all
Qwen3-30B-A3B
100
Qwen3-Next-80B-A3B
% of 195 exercises
% of 195 exercises
80
60
40
20
0
ct ing tic IES RP -TA IES IS o T lor SP NE e T ru W / ay G A E T st nk m SL AIMIM- LE E w o T w/o CR In Thi rith / A A AN w NE R k s C NE RA A C Ta R C
ct ing tic IES RP -TA IES IS o T lor SP NE e T ru W / ay G A E T st nk m SL AIMIM- LE E w o T w/o CR In Thi rith / A A AN w NE R k s C NE RA A C Ta R C
Figure 8: Roo-Eval results across both scales. Per-method pass@1 (light) and pass_all (dark) on the 195 exercises. Plain merge baselines and CRANE component ablations are reported alongside the full CRANE recipe. E.1
30B Main Results by Language Table 14: 30B Roo-Eval full metrics for Python (34 exercises × 3 = 102 tasks).
Model
pass@1
pass@3
pass_all
iter pass
ref. cost
Input total
Cached total
Output total
Input avg
Cached avg
Output avg
qwen3-30b-instruct qwen3-30b-thinking baseline-ta baseline-slerp baseline-ties baseline-aim-ta baseline-aim-ties baseline-lewis baseline-rain CRANE
15 (44.1%) 12 (35.3%) 15 (44.1%) 17 (50.0%) 19 (55.9%) 17 (50.0%) 15 (44.1%) 18 (52.9%) 17 (50.0%) 27 (79.4%)
22 (64.7%) 21 (61.8%) 20 (58.8%) 20 (58.8%) 24 (70.6%) 21 (61.8%) 21 (61.8%) 23 (67.6%) 21 (61.8%) 31 (91.2%)
13 7 12 14 13 11 11 10 12 (35.3%) 19 (55.9%)
50/102 (49.0%) 43/102 (42.2%) 48/102 (47.1%) 52/102 (51.0%) 55/102 (53.9%) 50/102 (49.0%) 48/102 (47.1%) 51/102 (50.0%) 49/102 (48.0%) 74/102 (72.5%)
$6.21 $5.70 $6.81 $6.77 $6.65 $7.05 $7.44 $7.01 $5.41 $4.24
7,622,806 3,362,588 8,719,220 7,880,084 7,759,662 8,551,536 8,936,722 8,508,474 3,194,566 5,605,858
146,270,461 17,273,159 172,340,721 179,001,415 179,068,578 185,412,207 198,989,550 180,678,399 15,928,831 63,459,202
1,411,369 3,745,730 1,296,213 1,287,865 1,211,760 1,308,627 1,337,479 1,356,050 3,560,320 1,480,496
74,733 32,967 85,483 77,256 76,075 83,839 87,615 83,416 31,319 54,959
1,434,024 169,345 1,689,615 1,754,916 1,755,574 1,817,767 1,950,878 1,771,357 156,165 622,149
13,837 36,723 12,708 12,626 11,880 12,830 13,113 13,295 34,905 14,514
Table 15: 30B Roo-Eval full metrics for JavaScript (50 exercises × 3 = 150 tasks). Model
pass@1
pass@3
pass_all
iter pass
ref. cost
Input total
Cached total
Output total
Input avg
Cached avg
Output avg
qwen3-30b-instruct qwen3-30b-thinking baseline-ta baseline-slerp baseline-ties baseline-aim-ta baseline-aim-ties baseline-lewis baseline-rain CRANE
28 (56.0%) 20 (40.0%) 26 (52.0%) 26 (52.0%) 25 (50.0%) 25 (50.0%) 28 (56.0%) 21 (42.0%) 26 (52.0%) 39 (78.0%)
37 (74.0%) 27 (54.0%) 35 (70.0%) 33 (66.0%) 33 (66.0%) 35 (70.0%) 36 (72.0%) 33 (66.0%) 29 (58.0%) 42 (84.0%)
20 12 21 16 16 17 19 17 13 (26.0%) 30 (60.0%)
86/150 (57.3%) 60/150 (40.0%) 84/150 (56.0%) 75/150 (50.0%) 76/150 (50.7%) 76/150 (50.7%) 84/150 (56.0%) 76/150 (50.7%) 68/150 (45.3%) 111/150 (74.0%)
$9.63 $7.90 $10.63 $11.27 $11.76 $11.61 $10.80 $10.79 $7.56 $5.67
11,240,333 5,772,708 12,879,073 13,925,856 13,517,910 13,820,167 12,355,490 12,935,745 5,752,111 8,027,932
257,446,200 29,434,786 298,843,109 314,511,653 345,238,140 336,070,898 314,457,599 303,072,729 28,189,669 93,420,273
1,786,951 4,927,274 1,660,467 1,755,758 1,723,803 1,699,775 1,633,606 1,713,793 4,674,807 1,753,243
74,936 38,485 85,860 92,839 90,119 92,134 82,370 86,238 38,347 53,519
1,716,308 196,232 1,992,287 2,096,744 2,301,588 2,240,473 2,096,384 2,020,485 187,931 622,801
11,913 32,848 11,070 11,705 11,492 11,332 10,891 11,425 31,165 11,688
28
Table 16: 30B Roo-Eval full metrics for Go (36 exercises × 3 = 108 tasks). Model
pass@1
pass@3
pass_all
iter pass
ref. cost
Input total
Cached total
Output total
Input avg
Cached avg
Output avg
qwen3-30b-instruct qwen3-30b-thinking baseline-ta baseline-slerp baseline-ties baseline-aim-ta baseline-aim-ties baseline-lewis baseline-rain CRANE
12 (33.3%) 16 (44.4%) 19 (52.8%) 14 (38.9%) 17 (47.2%) 16 (44.4%) 13 (36.1%) 17 (47.2%) 14 (38.9%) 27 (75.0%)
19 (52.8%) 23 (63.9%) 22 (61.1%) 21 (58.3%) 26 (72.2%) 24 (66.7%) 21 (58.3%) 24 (66.7%) 20 (55.6%) 30 (83.3%)
6 8 11 10 9 10 9 8 9 (25.0%) 18 (50.0%)
36/108 (33.3%) 45/108 (41.7%) 48/108 (44.4%) 44/108 (40.7%) 53/108 (49.1%) 50/108 (46.3%) 44/108 (40.7%) 44/108 (40.7%) 47/108 (43.5%) 72/108 (66.7%)
$7.65 $6.53 $8.69 $8.80 $8.11 $8.35 $9.16 $7.48 $6.20 $4.78
8,091,205 3,505,650 9,503,998 9,358,943 8,657,592 9,172,381 10,082,733 8,539,543 3,443,171 6,025,226
179,249,487 20,108,265 225,774,679 229,965,935 214,344,134 219,968,657 238,993,851 187,248,082 19,262,428 73,353,048
1,955,963 4,341,897 1,816,555 1,866,615 1,676,051 1,694,560 1,891,978 1,625,549 4,100,136 1,684,501
74,919 32,460 88,000 86,657 80,163 84,929 93,359 79,070 31,881 55,789
1,659,717 186,188 2,090,506 2,129,314 1,984,668 2,036,747 2,212,906 1,733,779 178,355 679,194
18,111 40,203 16,820 17,283 15,519 15,690 17,518 15,051 37,964 15,597
Table 17: 30B Roo-Eval full metrics for Java (45 exercises × 3 = 135 tasks). Model
pass@1
pass@3
pass_all
iter pass
ref. cost
Input total
Cached total
Output total
Input avg
Cached avg
Output avg
qwen3-30b-instruct qwen3-30b-thinking baseline-ta baseline-slerp baseline-ties baseline-aim-ta baseline-aim-ties baseline-lewis baseline-rain CRANE
27 (60.0%) 13 (28.9%) 22 (48.9%) 21 (46.7%) 20 (44.4%) 20 (44.4%) 21 (46.7%) 20 (44.4%) 12 (26.7%) 24 (53.3%)
32 (71.1%) 21 (46.7%) 27 (60.0%) 25 (55.6%) 29 (64.4%) 28 (62.2%) 26 (57.8%) 29 (64.4%) 25 (55.6%) 37 (82.2%)
19 5 18 14 14 14 15 13 4 (8.9%) 10 (22.2%)
78/135 (57.8%) 35/135 (25.9%) 66/135 (48.9%) 61/135 (45.2%) 65/135 (48.1%) 61/135 (45.2%) 63/135 (46.7%) 62/135 (45.9%) 36/135 (26.7%) 70/135 (51.9%)
$8.63 $8.44 $8.46 $8.98 $8.65 $9.16 $8.57 $7.58 $8.30 $6.97
9,625,792 5,011,844 9,953,576 10,568,416 10,332,428 10,623,621 10,494,971 9,559,129 4,844,261 9,008,906
223,276,324 30,749,871 223,712,036 238,926,759 234,942,843 251,140,388 224,177,002 197,176,539 30,197,023 117,821,938
1,792,674 5,458,022 1,592,945 1,670,085 1,509,467 1,608,986 1,589,514 1,382,058 5,378,656 2,247,297
71,302 37,125 73,730 78,285 76,537 78,693 77,741 70,808 35,883 66,732
1,653,899 227,777 1,657,126 1,769,828 1,740,317 1,860,299 1,660,570 1,460,567 223,681 872,755
13,279 40,430 11,800 12,371 11,181 11,918 11,774 10,237 39,841 16,646
Table 18: 30B Roo-Eval full metrics for Rust (30 exercises × 3 = 90 tasks). Model
pass@1
pass@3
pass_all
iter pass
ref. cost
Input total
Cached total
Output total
Input avg
Cached avg
Output avg
qwen3-30b-instruct qwen3-30b-thinking baseline-ta baseline-slerp baseline-ties baseline-aim-ta baseline-aim-ties baseline-lewis baseline-rain CRANE
9 (30.0%) 7 (23.3%) 10 (33.3%) 7 (23.3%) 11 (36.7%) 13 (43.3%) 11 (36.7%) 11 (36.7%) 8 (26.7%) 12 (40.0%)
15 (50.0%) 11 (36.7%) 15 (50.0%) 15 (50.0%) 17 (56.7%) 18 (60.0%) 16 (53.3%) 14 (46.7%) 11 (36.7%) 22 (73.3%)
5 3 3 4 5 5 3 6 4 (13.3%) 9 (30.0%)
32/90 (35.6%) 18/90 (20.0%) 28/90 (31.1%) 28/90 (31.1%) 33/90 (36.7%) 33/90 (36.7%) 30/90 (33.3%) 29/90 (32.2%) 22/90 (24.4%) 41/90 (45.6%)
$6.19 $6.51 $9.05 $9.21 $8.51 $8.32 $8.31 $7.91 $6.00 $4.72
6,967,880 3,404,218 9,289,522 9,589,846 8,860,719 9,170,900 8,736,839 8,547,662 3,175,404 6,010,939
150,833,979 22,031,076 256,694,433 249,569,550 241,852,016 224,308,682 225,587,509 211,082,637 20,120,464 76,419,820
1,425,177 4,313,532 1,645,362 1,838,488 1,523,066 1,602,218 1,637,948 1,579,754 3,968,011 1,593,906
77,421 37,825 103,217 106,554 98,452 101,899 97,076 94,974 35,282 66,788
1,675,933 244,790 2,852,160 2,772,995 2,687,245 2,492,319 2,506,528 2,345,363 223,560 849,109
15,835 47,928 18,282 20,428 16,923 17,802 18,199 17,553 44,089 17,710
E.2
80B-Next Main Results by Language Table 19: 80B-Next Roo-Eval full metrics for Python (34 exercises × 3 = 102 tasks).
Model
pass@1
pass@3
pass_all
iter pass
ref. cost
Input total
Cached total
Output total
Input avg
Cached avg
Output avg
qwen3-next-80b-instruct qwen3-next-80b-thinking qwen3-next-80b-ta qwen3-next-80b-ties qwen3-next-80b-slerp qwen3-next-80b-aim-ta qwen3-next-80b-aim-ties qwen3-next-80b-lewis CRANE
29 (85.3%) 16 (47.1%) 28 (82.4%) 29 (85.3%) 28 (82.4%) 29 (85.3%) 27 (79.4%) 28 (82.4%) 30 (88.2%)
31 (91.2%) 21 (61.8%) 30 (88.2%) 30 (88.2%) 33 (97.1%) 31 (91.2%) 31 (91.2%) 31 (91.2%) 33 (97.1%)
22 (64.7%) 11 (32.4%) 24 (70.6%) 24 (70.6%) 24 (70.6%) 26 (76.5%) 21 (61.8%) 24 (70.6%) 27 (79.4%)
82/102 (80.4%) 46/102 (45.1%) 83/102 (81.4%) 83/102 (81.4%) 86/102 (84.3%) 85/102 (83.3%) 81/102 (79.4%) 83/102 (81.4%) 90/102 (88.2%)
$12.58 $15.37 $13.42 $11.73 $12.29 $13.66 $12.76 $12.67 $10.54
4,642,554 2,890,157 4,644,411 4,255,004 4,251,615 4,679,247 4,663,119 4,471,974 3,807,607
63,011,687 11,873,010 73,063,904 52,634,960 65,835,441 69,035,610 58,860,124 62,461,313 46,484,492
971,182 2,735,770 990,290 1,021,133 925,945 1,106,082 1,077,154 1,028,532 933,088
45,515 28,334 45,533 41,715 41,682 45,874 45,716 43,842 37,329
617,761 116,402 716,312 516,029 645,445 676,819 577,060 612,365 455,730
9,521 26,821 9,708 10,011 9,077 10,843 10,560 10,083 9,148
Table 20: 80B-Next Roo-Eval full metrics for JavaScript (50 exercises × 3 = 150 tasks). Model
pass@1
pass@3
pass_all
iter pass
ref. cost
Input total
Cached total
Output total
Input avg
Cached avg
Output avg
qwen3-next-80b-instruct qwen3-next-80b-thinking qwen3-next-80b-ta qwen3-next-80b-ties qwen3-next-80b-slerp qwen3-next-80b-aim-ta qwen3-next-80b-aim-ties qwen3-next-80b-lewis CRANE
42 (84.0%) 18 (36.0%) 44 (88.0%) 46 (92.0%) 45 (90.0%) 45 (90.0%) 44 (88.0%) 46 (92.0%) 46 (92.0%)
44 (88.0%) 30 (60.0%) 47 (94.0%) 49 (98.0%) 47 (94.0%) 46 (92.0%) 48 (96.0%) 48 (96.0%) 49 (98.0%)
38 (76.0%) 11 (22.0%) 39 (78.0%) 40 (80.0%) 42 (84.0%) 42 (84.0%) 42 (84.0%) 39 (78.0%) 42 (84.0%)
124/150 (82.7%) 60/150 (40.0%) 132/150 (88.0%) 137/150 (91.3%) 134/150 (89.3%) 132/150 (88.0%) 135/150 (90.0%) 132/150 (88.0%) 137/150 (91.3%)
$14.99 $23.69 $14.50 $13.50 $14.21 $15.34 $14.72 $14.87 $13.85
6,100,734 4,812,224 5,775,193 5,408,355 5,732,131 5,955,332 5,941,063 5,901,958 5,555,281
62,082,646 19,921,095 64,329,549 56,427,698 60,738,939 73,104,000 64,318,755 64,583,900 61,325,457
1,279,387 4,130,946 1,188,490 1,157,469 1,190,274 1,197,194 1,209,895 1,243,850 1,130,758
40,671 32,081 38,501 36,055 38,214 39,702 39,607 39,346 37,035
413,884 132,807 428,863 376,184 404,926 487,360 428,791 430,559 408,836
8,529 27,539 7,923 7,716 7,935 7,981 8,065 8,292 7,538
Table 21: 80B-Next Roo-Eval full metrics for Go (36 exercises × 3 = 108 tasks). Model
pass@1
pass@3
pass_all
iter pass
ref. cost
Input total
Cached total
Output total
Input avg
Cached avg
Output avg
qwen3-next-80b-instruct qwen3-next-80b-thinking qwen3-next-80b-ta qwen3-next-80b-ties qwen3-next-80b-slerp qwen3-next-80b-aim-ta qwen3-next-80b-aim-ties qwen3-next-80b-lewis CRANE
24 (66.7%) 19 (52.8%) 32 (88.9%) 28 (77.8%) 26 (72.2%) 29 (80.6%) 28 (77.8%) 31 (86.1%) 31 (86.1%)
30 (83.3%) 23 (63.9%) 33 (91.7%) 33 (91.7%) 30 (83.3%) 31 (86.1%) 34 (94.4%) 34 (94.4%) 33 (91.7%)
17 (47.2%) 14 (38.9%) 30 (83.3%) 23 (63.9%) 20 (55.6%) 24 (66.7%) 27 (75.0%) 26 (72.2%) 29 (80.6%)
71/108 (65.7%) 56/108 (51.9%) 95/108 (88.0%) 87/108 (80.6%) 78/108 (72.2%) 82/108 (75.9%) 92/108 (85.2%) 89/108 (82.4%) 92/108 (85.2%)
$10.34 $15.42 $12.14 $11.33 $13.13 $14.63 $11.26 $12.66 $13.11
4,241,044 3,009,313 4,410,040 4,384,786 5,075,282 4,995,851 4,296,087 4,452,145 4,654,524
41,041,332 13,574,043 50,264,599 41,693,441 62,531,130 71,376,393 43,593,567 55,444,704 55,659,080
906,858 2,699,233 1,124,352 1,093,254 1,030,326 1,229,321 1,059,110 1,147,981 1,209,340
39,268 27,864 40,833 40,599 46,993 46,258 39,778 41,223 43,097
380,012 125,685 465,412 386,050 578,991 660,893 403,644 513,376 515,362
8,396 24,992 10,410 10,122 9,540 11,383 9,806 10,629 11,198
29
Table 22: 80B-Next Roo-Eval full metrics for Java (45 exercises × 3 = 135 tasks). Model
pass@1
pass@3
pass_all
iter pass
ref. cost
Input total
Cached total
Output total
Input avg
Cached avg
Output avg
qwen3-next-80b-instruct qwen3-next-80b-thinking qwen3-next-80b-ta qwen3-next-80b-ties qwen3-next-80b-slerp qwen3-next-80b-aim-ta qwen3-next-80b-aim-ties qwen3-next-80b-lewis CRANE
26 (57.8%) 5 (11.1%) 26 (57.8%) 28 (62.2%) 25 (55.6%) 32 (71.1%) 30 (66.7%) 27 (60.0%) 28 (62.2%)
38 (84.4%) 8 (17.8%) 38 (84.4%) 35 (77.8%) 34 (75.6%) 38 (84.4%) 40 (88.9%) 36 (80.0%) 37 (82.2%)
12 (26.7%) 1 (2.2%) 18 (40.0%) 14 (31.1%) 17 (37.8%) 22 (48.9%) 15 (33.3%) 15 (33.3%) 20 (44.4%)
77/135 (57.0%) 14/135 (10.4%) 85/135 (63.0%) 76/135 (56.3%) 79/135 (58.5%) 91/135 (67.4%) 79/135 (58.5%) 79/135 (58.5%) 89/135 (65.9%)
$18.44 $24.83 $20.16 $21.60 $21.95 $20.96 $22.75 $22.02 $19.36
7,105,797 4,510,309 7,574,931 8,077,565 8,155,149 7,786,595 8,473,956 8,157,055 7,543,322
80,844,477 21,387,607 92,095,924 96,813,494 105,485,058 96,868,682 105,941,200 102,370,544 90,934,720
1,566,931 4,410,105 1,682,157 1,840,404 1,761,327 1,746,001 1,877,789 1,826,930 1,529,337
52,635 33,409 56,110 59,833 60,408 57,678 62,770 60,422 55,876
598,847 158,426 682,192 717,136 781,370 717,545 784,749 758,300 673,591
11,606 32,667 12,460 13,632 13,046 12,933 13,909 13,532 11,328
Table 23: 80B-Next Roo-Eval full metrics for Rust (30 exercises × 3 = 90 tasks). Model
pass@1
pass@3
pass_all
iter pass
ref. cost
Input total
Cached total
Output total
Input avg
Cached avg
Output avg
qwen3-next-80b-instruct qwen3-next-80b-thinking qwen3-next-80b-ta qwen3-next-80b-ties qwen3-next-80b-slerp qwen3-next-80b-aim-ta qwen3-next-80b-aim-ties qwen3-next-80b-lewis CRANE
21 (70.0%) 11 (36.7%) 23 (76.7%) 23 (76.7%) 19 (63.3%) 22 (73.3%) 20 (66.7%) 23 (76.7%) 24 (80.0%)
27 (90.0%) 15 (50.0%) 25 (83.3%) 25 (83.3%) 25 (83.3%) 25 (83.3%) 24 (80.0%) 27 (90.0%) 24 (80.0%)
15 (50.0%) 7 (23.3%) 21 (70.0%) 20 (66.7%) 15 (50.0%) 15 (50.0%) 14 (46.7%) 17 (56.7%) 21 (70.0%)
62/90 (68.9%) 32/90 (35.6%) 69/90 (76.7%) 68/90 (75.6%) 62/90 (68.9%) 62/90 (68.9%) 59/90 (65.6%) 66/90 (73.3%) 68/90 (75.6%)
$15.44 $15.27 $14.32 $13.37 $16.30 $15.43 $15.55 $14.66 $14.57
5,354,259 2,930,934 5,087,632 4,658,243 5,701,264 5,270,696 5,480,806 5,130,397 5,006,504
68,007,725 15,007,654 62,155,706 57,569,561 77,723,723 67,490,094 64,701,478 61,044,748 67,960,906
1,404,484 2,654,245 1,299,705 1,234,629 1,375,841 1,424,542 1,465,082 1,384,623 1,270,158
59,491 32,565 56,529 51,758 63,347 58,563 60,897 57,004 55,628
755,641 166,751 690,618 639,661 863,596 749,889 718,905 678,274 755,121
15,605 29,491 14,441 13,718 15,287 15,828 16,278 15,384 14,113
E.3
Pass@1 Language Summaries
Tables 24 and 25 summarize Roo-Eval pass@1 by language at the 30B and 80B-Next scales respectively. Means are unweighted over languages; exercise-weighted aggregate totals are reported in Table 1. Qwen3-Next-80B-A3B 90
Instruct
56
33
60
30
85
84
67
58
70
Thinking
35
40
44
29
23
47
36
53
11
37
80
Task Arithmetic
44
52
53
49
33
82
88
89
58
77
70
TIES
56
50
47
44
37
85
92
78
62
77
60
SLERP
50
52
39
47
23
82
90
72
56
63
AIM-TA
50
50
44
44
43
85
90
81
71
73
AIM-TIES
44
56
36
47
37
79
88
78
67
67
LEWIS
53
42
47
44
37
82
92
86
60
77
CRANE
79
78
75
53
40
88
92
86
62
80
50 40
pass@1 (%)
Qwen3-30B-A3B 44
30 20
Py
n
tho
Jav
aS
t
p cri
Go
a
Jav
st
n
tho
Ru
Py
pt
a
Go
cri
aS
Jav
Jav
st
Ru
Figure 9: Per-language Roo-Eval pass@1 across methods at both scales. Rows: methods (Instruct, Thinking, plain merges, CRANE); columns: Python, JavaScript, Go, Java, Rust. CRANE achieves the highest pass@1 on Python, JavaScript, and Go at 30B and remains among the top-performing methods at 80B-Next, with the residual Java/Rust gap on 30B discussed in §5. Table 24: 30B Roo-Eval pass@1 by language. Model
Python
JavaScript
Go
Java
Rust
Macro mean
Qwen3-30B Instruct Qwen3-30B Thinking Task Arithmetic SLERP TIES AIM-TA AIM-TIES LEWIS RAIN CRANE
44.1 35.3 44.1 50.0 55.9 50.0 44.1 52.9 50.0 79.4
56.0 40.0 52.0 52.0 50.0 50.0 56.0 42.0 52.0 78.0
33.3 44.4 52.8 38.9 47.2 44.4 36.1 47.2 38.9 75.0
60.0 28.9 48.9 46.7 44.4 44.4 46.7 44.4 26.7 53.3
30.0 23.3 33.3 23.3 36.7 43.3 36.7 36.7 26.7 40.0
44.7 34.4 46.2 42.2 46.8 46.4 43.9 44.6 39.5 65.1
30
Table 25: 80B-Next Roo-Eval pass@1 by language.
E.4
Model
Python
JavaScript
Go
Java
Rust
Macro mean
Qwen3-Next-80B Instruct Qwen3-Next-80B Thinking Task Arithmetic TIES SLERP AIM-TA AIM-TIES LEWIS RAIN CRANE
85.3 47.1 82.4 85.3 82.4 85.3 79.4 82.4 58.8 88.2
84.0 36.0 88.0 92.0 90.0 90.0 88.0 92.0 34.0 92.0
66.7 52.8 88.9 77.8 72.2 80.6 77.8 86.1 52.8 86.1
57.8 11.1 57.8 62.2 55.6 71.1 66.7 60.0 46.7 62.2
70.0 36.7 76.7 76.7 63.3 73.3 66.7 76.7 43.3 80.0
72.8 35.4 78.5 79.0 72.7 80.1 76.4 79.5 46.2 81.7
Pass@3 Language Summaries Table 26: 30B Roo-Eval pass@3 by language. Model
Python
JavaScript
Go
Java
Rust
Macro mean
CRANE Qwen3-30B Instruct Qwen3-30B Thinking Task Arithmetic SLERP TIES AIM-TA AIM-TIES LEWIS RAIN
91.2 64.7 61.8 58.8 58.8 70.6 61.8 61.8 67.6 61.8
84.0 74.0 54.0 70.0 66.0 66.0 70.0 72.0 66.0 58.0
83.3 52.8 63.9 61.1 58.3 72.2 66.7 58.3 66.7 55.6
82.2 71.1 46.7 60.0 55.6 64.4 62.2 57.8 64.4 55.6
73.3 50.0 36.7 50.0 50.0 56.7 60.0 53.3 46.7 36.7
82.8 62.5 52.6 60.0 57.7 66.0 64.1 60.6 62.3 54.4
Table 27: 80B-Next Roo-Eval pass@3 by language.
E.5
Model
Python
JavaScript
Go
Java
Rust
Macro mean
Qwen3-Next-80B Instruct Qwen3-Next-80B Thinking Task Arithmetic TIES SLERP AIM-TA AIM-TIES LEWIS RAIN CRANE
91.2 61.8 88.2 88.2 97.1 91.2 91.2 91.2 61.8 97.1
88.0 60.0 94.0 98.0 94.0 92.0 96.0 96.0 58.0 98.0
83.3 63.9 91.7 91.7 83.3 86.1 94.4 94.4 63.9 91.7
84.4 17.8 84.4 77.8 75.6 84.4 88.9 80.0 57.8 82.2
90.0 50.0 83.3 83.3 83.3 83.3 80.0 90.0 50.0 80.0
87.2 49.7 88.7 88.2 86.7 87.4 90.8 90.3 58.5 89.8
Pass-All Language Summaries Table 28: 30B Roo-Eval pass-all by language, i.e. exercises solved on all three iterations. Model
Python
JavaScript
Go
Java
Rust
Macro mean
Qwen3-30B Instruct Qwen3-30B Thinking Task Arithmetic SLERP TIES AIM-TA AIM-TIES LEWIS RAIN CRANE
38.2 20.6 35.3 41.2 38.2 32.4 32.4 29.4 35.3 55.9
40.0 24.0 42.0 32.0 32.0 34.0 38.0 34.0 26.0 60.0
16.7 22.2 30.6 27.8 25.0 27.8 25.0 22.2 25.0 50.0
42.2 11.1 40.0 31.1 31.1 31.1 33.3 28.9 8.9 22.2
16.7 10.0 10.0 13.3 16.7 16.7 10.0 20.0 13.3 30.0
30.8 17.6 31.6 29.1 28.6 28.4 27.7 26.9 21.5 43.6
31
Table 29: 80B-Next Roo-Eval pass-all by language, i.e. exercises solved on all three iterations. Model
Python
JavaScript
Go
Java
Rust
Macro mean
Qwen3-Next-80B Instruct Qwen3-Next-80B Thinking Task Arithmetic TIES SLERP AIM-TA AIM-TIES LEWIS RAIN CRANE
64.7 32.4 70.6 70.6 70.6 76.5 61.8 70.6 38.2 79.4
76.0 22.0 78.0 80.0 84.0 84.0 84.0 78.0 20.0 84.0
47.2 38.9 83.3 63.9 55.6 66.7 75.0 72.2 44.4 80.6
26.7 2.2 40.0 31.1 37.8 48.9 33.3 33.3 13.3 44.4
50.0 23.3 70.0 66.7 50.0 50.0 46.7 56.7 16.7 70.0
53.3 22.6 67.7 62.1 59.6 65.2 61.0 62.1 25.6 71.7
32
F
Terminal-Bench v2 Detailed Results
This appendix collects supplementary Terminal-Bench v2 tables omitted from the main text for space. Section F.1 reports the full per-method table at both scales, including pass@3, pass_majority, the LLM/Daytona/Total dollar split, and the four metric definitions. Sections F.2 and F.3 report per-task solve counts across ten variants at the 30B and 80B-Next scales, with the long tail of unsolvable tasks listed verbatim. Setup, sandbox specs, Daytona pricing, and parser configuration are documented in Appendix A.3. Metric definitions. pass@1 is the OpenAI-style mean reward = mean(c/5) × n_tasks, the expected single-shot pass count. pass@3 is the OpenAI pass@k estimator at k = 3, n = 5 attempts: per-task 1 − C(5 − c, 3)/C(5, 3), summed over the 89 tasks; this predicts what the same model would have scored with 3 attempts/task instead of 5. pass@5 is best-of-5: a task counts as a pass if any of 5 attempts passed. pass_majority requires ≥ 3/5 attempts to pass (per-task rate ≥ 0.60). pass_majority differs from pass@3: pass@3 weights by the probability of a 3-shot subsample landing a pass; pass_majority requires actual ≥ 3 successes. “Test time” is the end-to-end Terminal-Bench harness wall time; tokens are aggregated for the launched attempts, while excluded tasks contribute zero tokens and remain in the 89-task success denominator. F.1
Full Per-Method Table
Tables 30 and 31 report the full headline metrics. The bold cells in each table mark the best value in their column (lower is better for cost columns, higher is better for pass-rate columns). The CRANE row corresponds to the crane-simple-v2 30B and crane-next-80b runs. Table 30: 30B Terminal-Bench v2: full per-method metrics. Tokens are in millions; “Input” counts non-cached prefill tokens. “LLM $” is a token-usage reference proxy under the GPT-5.4 nano schedule; “Daytona $” is real cash that bills against the Daytona invoice; “Total $” is the sum. Method
pass@1
pass@3
pass@5
pass_maj.
Test time
Input
Cached
Output
LLM $
Daytona $
Total $
Instruct (ref) Thinking (ref) Task Arithmetic TIES SLERP AIM-TA AIM-TIES LEWIS RAIN CRANE
4.8 (5.4%) 5.2 (5.9%) 4.8 (5.4%) 5.4 (6.1%) 4.8 (5.4%) 5.0 (5.6%) 5.0 (5.6%) 4.6 (5.2%) 5.0 (5.6%) 6.8 (7.6%)
7.6 (8.5%) 9.4 (10.6%) 9.8 (11.0%) 9.6 (10.8%) 9.9 (11.1%) 9.4 (10.6%) 9.3 (10.4%) 8.2 (9.2%) 7.9 (8.9%) 12.4 (13.9%)
9 (10.1%) 12 (13.5%) 13 (14.6%) 12 (13.5%) 13 (14.6%) 12 (13.5%) 12 (13.5%) 10 (11.2%) 9 (10.1%) 16 (17.9%)
4 (4.5%) 4 (4.5%) 2 (2.2%) 3 (3.4%) 3 (3.4%) 4 (4.5%) 3 (3.4%) 4 (4.5%) 4 (4.5%) 7 (7.9%)
4h 14m 4h 37m 2h 50m 2h 53m 2h 51m 2h 44m 2h 42m 2h 53m 4h 05m 2h 18m
16.96 4.34 8.54 9.97 7.13 7.18 9.47 7.00 4.01 7.68
685.01 122.24 425.36 481.93 468.41 338.59 467.58 351.21 114.61 319.35
5.43 18.41 3.77 4.40 3.80 3.85 4.33 3.70 16.76 3.70
$23.88 $26.33 $14.93 $17.13 $15.54 $13.02 $16.66 $13.05 $24.04 $12.54
$7.34 $8.73 $4.95 $5.02 $4.99 $5.00 $4.67 $5.21 $9.28 $4.18
$31.22 $35.06 $19.88 $22.15 $20.53 $18.02 $21.33 $18.26 $33.32 $16.72
Table 31: 80B-Next Terminal-Bench v2: full per-method metrics. Tokens in millions; “LLM $” uses the GPT-5.4 mini schedule (mini chosen over nano because the 80B size is closer to mini’s tier; ∼3.7× nano price). The ta and aim-ties rows have elevated input-token totals due to lower prefix-cache hit rates in the audited sweep; the table reports and prices the recorded totals. Method Instruct (ref) Thinking (ref) Task Arithmetic TIES SLERP AIM-TA AIM-TIES LEWIS RAIN CRANE
pass@1
pass@3
pass@5
pass_maj.
Test time
Input
Cached
Output
LLM $
Daytona $
Total $
12.0 (13.5%) 6.0 (6.7%) 11.6 (13.0%) 11.8 (13.3%) 12.0 (13.5%) 12.2 (13.7%) 12.6 (14.2%) 12.6 (14.2%) 7.0 (7.9%) 13.2 (14.8%)
17.4 (19.6%) 9.6 (10.8%) 19.1 (21.5%) 20.5 (23.0%) 19.9 (22.4%) 18.0 (20.2%) 19.1 (21.5%) 19.6 (22.0%) 11.5 (12.9%) 22.1 (24.8%)
20 (22.5%) 12 (13.5%) 22 (24.7%) 23 (25.8%) 24 (27.0%) 20 (22.5%) 22 (24.7%) 23 (25.8%) 14 (15.7%) 27 (30.3%)
12 (13.5%) 6 (6.7%) 11 (12.4%) 13 (14.6%) 10 (11.2%) 12 (13.5%) 11 (12.4%) 13 (14.6%) 7 (7.9%) 11 (12.4%)
2h 28m 5h 12m 2h 10m 1h 55m 2h 08m 2h 00m 2h 14m 2h 11m 4h 57m 1h 58m
10.84 4.45 266.39 11.71 12.96 10.10 301.41 10.59 4.36 10.42
224.62 85.64 255.57 285.22 249.13 257.56 289.77 248.36 82.32 234.57
3.85 20.39 3.65 3.86 3.55 3.72 3.62 3.74 19.35 3.58
$42.28 $101.50 $235.39 $47.53 $44.37 $43.61 $264.08 $43.39 $96.52 $41.69
$4.27 $12.02 $5.01 $4.20 $4.85 $6.03 $4.76 $4.91 $11.69 $4.42
$46.55 $113.52 $240.40 $51.73 $49.22 $49.64 $268.84 $48.30 $108.21 $46.11
33
F.2
Per-Task Solve Counts at 30B
Table 32 reports per-task solve counts across the ten 30B variants. Each cell reports the count of pass attempts in 5 trials for that (task, method) pair; the right two columns report the row-sum out of 10 × 5 = 50 trials and the resulting solve rate. The 5 excluded tasks (pytorch-model-cli, count-dataset-tokens, mcmc-sampling-stan, rstan-to-pystan, reshard-c4-data) are treated as 5/5 failures across all methods (not listed). Tasks with Σ = 0 across all 10 variants are listed verbatim under the table. Table 32: 30B Terminal-Bench v2: per-task solve counts across ten variants (5 attempts each). Sorted by total passes (easiest first). Column order: Inst = Instruct, Think = Thinking (parser-fix), TA = Task Arithmetic, AIM-TA, AIM-TI = AIM-TIES, CRANE = CRANE, RAIN = RAINMerging [Huang et al., 2026]. Task
Inst
Think
TA
TIES
SLERP
AIM-TA
AIM-TI
LEWIS
CRANE
RAIN
modernize-scientific-stack fix-git prove-plus-comm constraints-scheduling log-summary-date-ranges git-leak-recovery build-pmars extract-elf nginx-request-logging multi-source-data-merger hf-model-inference portfolio-optimization cancel-async-tasks configure-git-webserver sqlite-with-gcov cobol-modernization git-multibranch openssl-selfsigned-cert model-extraction-relu-logits adaptive-rejection-sampler kv-store-grpc merge-diff-arc-agi-task pypi-server query-optimize
5 1 5 2 3 2 4 0 0 0 0 2 0 0 0 1 1 0 0 0 0 0 0 0
1 5 1 2 0 2 3 1 4 4 1 0 0 0 1 0 1 0 0 0 0 0 0 0
5 2 4 1 2 0 1 2 1 0 1 1 0 0 1 2 1 0 0 0 0 0 0 1
5 4 1 2 5 2 2 0 0 0 1 0 2 1 0 0 0 1 1 0 0 0 1 0
5 3 0 2 3 2 1 2 1 0 0 1 1 1 0 1 0 0 0 0 0 1 0 0
3 2 4 5 3 1 1 2 0 1 0 0 1 1 1 0 0 1 0 0 1 0 0 0
4 2 5 2 4 2 1 0 0 0 1 1 1 0 1 0 0 1 0 0 0 0 0 0
5 3 3 4 0 2 1 2 1 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0
4 3 5 3 3 4 1 1 3 0 1 1 0 1 1 0 1 0 1 1 0 0 0 0
2 5 1 4 0 4 4 2 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Σ/50
Rate
39 30 29 27 23 21 19 12 11 7 6 6 5 5 5 4 4 3 2 1 1 1 1 1
78% 60% 58% 54% 46% 42% 38% 24% 22% 14% 12% 12% 10% 10% 10% 8% 8% 6% 4% 2% 2% 2% 2% 2%
Tasks unsolved by every 30B variant (Σ = 0/50, 65 tasks). bn-fit-modify, break-filter-js-from-html, build-cython-ext, build-pov-ray, caffe-cifar-10, chess-best-move, circuit-fibsqrt, code-from-image, compile-compcert, count-dataset-tokens, crack-7z-hash, custom-memory-heap-crash, db-wal-recovery, distribution-search, dna-assembly, dna-insert, extract-moves-from-video, feal-differential-cryptanalysis, feal-linear-cryptanalysis, filter-js-from-html, financial-document-processor, fix-code-vulnerability, fix-ocaml-gc, gcode-to-text, gpt2-codegolf, headless-terminal, install-windows-3.11, large-scale-text-editing, largest-eigenval, llm-inference-batching-scheduler, mailman, make-doom-for-mips, make-mips-interpreter, mcmc-sampling-stan, mteb-leaderboard, mteb-retrieve, overfull-hbox, password-recovery, path-tracing, path-tracing-reverse, polyglot-c-py, polyglot-rust-c, protein-assembly, pytorch-model-cli, pytorch-model-recovery, qemu-alpine-ssh, qemu-startup, raman-fitting, regex-chess, regex-log, reshard-c4-data, rstan-to-pystan, sam-cell-seg, sanitize-git-repo, schemelike-metacircular-eval, sparql-university, sqlite-db-truncate, torch-pipeline-parallelism, torch-tensor-parallelism, train-fasttext, tune-mjcf, video-processing, vulnerable-secret, winning-avg-corewars, write-compressor.
34
F.3
Per-Task Solve Counts at 80B-Next
Table 33 reports per-task solve counts across the ten 80B-Next variants under the same conventions as Table 32. Compared with 30B, the 80B-Next class solves 13 additional tasks at least once, while 52 tasks remain unsolved by all variants; the long tail is listed verbatim under the table. Table 33: 80B-Next Terminal-Bench v2: per-task solve counts across ten variants (5 attempts each). Sorted by total passes (easiest first). Column order matches Table 32. Task
Inst
Think
TA
TIES
SLERP
AIM-TA
AIM-TI
LEWIS
CRANE
RAIN
modernize-scientific-stack log-summary-date-ranges prove-plus-comm cobol-modernization constraints-scheduling git-leak-recovery build-pmars fix-git multi-source-data-merger portfolio-optimization nginx-request-logging sqlite-with-gcov merge-diff-arc-agi-task git-multibranch openssl-selfsigned-cert query-optimize cancel-async-tasks extract-elf adaptive-rejection-sampler hf-model-inference vulnerable-secret crack-7z-hash fix-code-vulnerability fix-ocaml-gc pypi-server configure-git-webserver mteb-retrieve qemu-startup regex-log tune-mjcf distribution-search headless-terminal large-scale-text-editing largest-eigenval password-recovery path-tracing-reverse winning-avg-corewars
5 5 5 5 4 5 4 3 4 2 4 3 3 1 1 0 2 0 0 1 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0
5 0 0 0 3 1 4 5 4 3 1 1 0 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0
5 5 4 4 3 5 2 3 4 2 2 2 2 1 3 3 1 1 3 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
5 3 4 3 3 4 3 4 2 4 2 4 3 1 1 3 3 2 1 1 0 0 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0
4 5 5 5 5 1 5 1 3 3 3 3 2 2 0 1 1 2 1 1 1 2 2 1 0 0 1 0 0 0 0 0 0 0 0 0 0
5 5 5 4 4 5 4 4 2 4 3 2 2 0 3 3 0 2 0 1 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0
5 5 5 4 3 5 4 3 4 5 4 2 2 2 2 0 0 0 2 1 0 1 0 1 0 0 0 1 1 0 1 0 0 0 0 0 0
5 5 5 4 3 4 4 3 3 4 3 4 1 4 2 1 1 1 1 2 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0
5 5 5 4 4 5 4 4 3 2 3 2 3 2 1 2 2 1 1 1 1 0 0 0 0 1 0 1 0 0 0 1 1 0 1 0 1
5 1 0 3 4 1 1 5 4 3 3 2 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0
Σ/50
Rate
49 39 38 36 36 36 35 35 33 32 28 25 18 14 14 13 10 10 9 9 5 4 4 4 3 2 2 2 2 2 1 1 1 1 1 1 1
98% 78% 76% 72% 72% 72% 70% 70% 66% 64% 56% 50% 36% 28% 28% 26% 20% 20% 18% 18% 10% 8% 8% 8% 6% 4% 4% 4% 4% 4% 2% 2% 2% 2% 2% 2% 2%
Tasks unsolved by every 80B-Next variant (Σ = 0/50, 52 tasks). bn-fit-modify, break-filter-js-from-html, build-cython-ext, build-pov-ray, caffe-cifar-10, chess-best-move, circuit-fibsqrt, code-from-image, compile-compcert, count-dataset-tokens, custom-memory-heap-crash, db-wal-recovery, dna-assembly, dna-insert, extract-moves-from-video, feal-differential-cryptanalysis, feal-linear-cryptanalysis, filter-js-from-html, financial-document-processor, gcode-to-text, gpt2-codegolf, install-windows-3.11, kv-store-grpc, llm-inference-batching-scheduler, mailman, make-doom-for-mips, make-mips-interpreter, mcmc-sampling-stan, model-extraction-relu-logits, mteb-leaderboard, overfull-hbox, path-tracing, polyglot-c-py, polyglot-rust-c, protein-assembly, pytorch-model-cli, pytorch-model-recovery, qemu-alpine-ssh, raman-fitting, regex-chess, reshard-c4-data, rstan-to-pystan, sam-cell-seg, sanitize-git-repo, schemelike-metacircular-eval, sparql-university, sqlite-db-truncate, torch-pipeline-parallelism, torch-tensor-parallelism, train-fasttext, video-processing, write-compressor.
35
G
Ablations
Table 36 reports the Roo-Eval α and τ sweep values corresponding to the Roo panels in Figure 4 (§4.3), including the reference-cost proxy column omitted from the figure. Tables 34 and 35 report the full per-variant token breakdowns for the Terminal-Bench v2 and SWE-bench-Verified componentremoval ablations summarized in the lower block of Table 4. Table 34: Full per-variant Terminal-Bench v2 component-removal ablations. “Input” is non-cached prefill tokens (M); “Output” is generated tokens (M); “TTC” = Ni + 0.1Nc + 5No (M). Per-variant cached-prefix counts were not logged separately for the ablation runs, so the cached contribution to TTC is estimated using the same Nc /Ni ratio as the corresponding full CRANE run at the same scale. Qwen3-30B-A3B Method
Qwen3-Next-80B-A3B
pass@1
pass@5
Input
Output
TTC
pass@1
pass@5
Input
Output
TTC
CRANE w/o T (δ) CRANE w/o Taylor CRANE w/o GSP
6.80 (7.6%) 5.80 (6.5%) 4.80 (5.4%)
12 (13.5%) 14 (15.7%) 11 (12.4%)
13.47 12.02 4.78
4.92 4.61 3.56
94.1 85.1 42.5
12.20 (13.7%) 11.60 (13.0%) 11.40 (12.8%)
21 (23.6%) 22 (24.7%) 19 (21.3%)
10.86 9.95 11.80
3.49 3.61 3.79
52.8 50.4 57.3
CRANE (T (δ)+Taylor+GSP)
6.80 (7.6%)
16 (17.9%)
7.68
3.70
58.1
13.20 (14.8%)
27 (30.3%)
10.42
3.58
51.8
Table 35: Full per-variant SWE-bench-Verified component-removal ablations. “Compl.” counts patches that completed grading; “Empty” counts predictions filtered for empty patches before grading; “Output” is generated tokens (M); “TTC” = Ni + 0.1Nc + 5No (B). The full-recipe row’s “Empty” is omitted because the headline run did not log it separately. Qwen3-30B-A3B
Qwen3-Next-80B-A3B
Method
Resolved
Compl.
Empty
Output
TTC
Resolved
Compl.
Empty
Output
TTC
CRANE w/o T (δ) CRANE w/o Taylor CRANE w/o GSP
120 (24.0%) 106 (21.2%) 94 (18.8%)
439 454 374
60 43 116
316 308 476
8.43 7.34 5.35
164 (32.8%) 162 (32.4%) 175 (35.0%)
488 483 485
10 15 12
305 313 334
5.51 5.50 5.35
CRANE (T (δ)+Taylor+GSP)
122 (24.4%)
460
—
373
5.68
180 (36.0%)
487
—
309
5.22
Table 36: Continuous-hyperparameter sweeps of the CRANE recipe on Qwen3-30B-A3B Roo-Eval. The bold column is the reported configuration (α = 0.25, τ = 0.03); the α sweep varies α at fixed τ = 0.03, and the τ sweep varies τ at fixed α = 0.25. pass@1 / pass@3 / pass_all are exerciseweighted aggregates over the 195 Roo-Eval exercises; per-language splits follow below. α sweep (τ = 0.03)
reported Metric pass@1 (%) pass@3 (%) pass_all (%) Ref. cost
τ sweep (α = 0.25)
α = 0.25, τ = 0.03
α = 0.15
α = 0.20
α = 0.30
α = 0.35
τ = 0.003
τ = 0.3
66.2 83.1 44.1 26.37
47.2 63.1 33.3 31.93
63.1 78.5 47.7 28.15
54.4 74.9 31.8 20.55
39.5 61.0 16.9 17.53
63.1 80.5 43.1 26.38
52.3 76.4 29.7 22.79
This subsection contains two groups of tables. The first group is four pass@1 summary tables: Tables 37, 38, and 39 report 30B Roo-Eval pass@1 percentages by language for the α sweep, τ sweep, and component-removal ablations respectively, and Table 40 reports the corresponding 80B CRANE component ablations. The final column of each summary reports the five-language reference-cost proxy computed from recorded local-vLLM token usage. The second group is four detail tables (Tables 41, 42, 43, and 44) that group each ablation family by programming language and retain pass@1, pass@3, pass_all, iterative pass, reference cost, and recorded input/cached/output token totals and averages. Alpha sweep detailed per-language results. Table 41 reports per-language pass metrics, referencecost proxy, and recorded local-vLLM token usage for each row in this ablation family. Tau (GSP threshold) sweep detailed per-language results. Table 42 reports per-language pass metrics, reference-cost proxy, and recorded local-vLLM token usage for each row in this ablation family. 36
Table 37: Global merge-scale α sweep on the 30B CRANE recipe. Variant
Python
JavaScript
Go
Java
Rust
Macro mean
Ref. cost
α = 0.15 α = 0.20 α = 0.30 α = 0.35 CRANE
70.6 61.8 61.8 50.0 79.4
72.0 78.0 66.0 46.0 78.0
52.8 66.7 52.8 38.9 75.0
4.4 51.1 48.9 31.1 53.3
36.7 53.3 36.7 30.0 40.0
47.3 62.2 53.2 39.2 65.1
$31.93 $28.15 $20.55 $17.53 $26.37
Table 38: GSP threshold sweep on the 30B CRANE recipe. Variant CRANE (τ = 0.03) tau030 (τ = 0.3) tau0003 (τ = 0.003)
Python
JavaScript
Go
Java
Rust
Macro mean
Ref. cost
79.4 55.9 70.6
78.0 62.0 76.0
75.0 50.0 63.9
53.3 53.3 53.3
40.0 33.3 46.7
65.1 52.3 63.1
$26.37 $22.79 $26.37
Table 39: Component-removal ablations for the 30B CRANE recipe. Variant
Python
JavaScript
Go
Java
Rust
Macro mean
Ref. cost
58.8 73.5 58.8 79.4
70.0 70.0 58.0 78.0
61.1 58.3 30.6 75.0
48.9 57.8 51.1 53.3
43.3 36.7 56.7 40.0
56.4 59.3 51.0 65.1
$31.36 $30.78 $22.07 $26.37
unified (drop Taylor αc ) noT (drop T (δ)) noGSP (drop Πτ ) CRANE
Table 40: Component-removal ablations for the 80B CRANE recipe. The full recipe uses α = 0.15, τ = 0.03, arch-normalized Taylor scaling, and GSP for attention, linear-attention inner slots, and routers. Variant
Python
JavaScript
Go
Java
Rust
Macro mean
Ref. cost
85.3 88.2 88.2 88.2
90.0 92.0 72.0 92.0
86.1 83.3 86.1 86.1
66.7 51.1 73.3 62.2
63.3 73.3 73.3 80.0
78.3 77.6 78.6 81.7
$78.24 $84.69 $86.73 $71.43
noT (drop T (δ)) noTaylor (drop Taylor αc ) noGSP (drop Πτ ) CRANE (full)
Table 41: 30B alpha sweep detailed Roo-Eval metrics by language, including token usage. Model
pass@1
pass@3
pass_all
iter pass
ref. cost Input total
Cached total Output total Input avg Cached avg Output avg
$5.07 $4.62 $3.31 $2.95 $4.24
6,310,035 5,976,877 4,810,657 4,612,640 5,605,858
96,501,372 68,969,826 45,377,736 37,437,300 63,459,202
1,499,610 1,636,553 1,149,974 1,021,834 1,480,496
61,863 58,596 47,163 45,221 54,959
946,091 676,174 444,879 367,032 622,149
14,702 16,044 11,274 10,017 14,514
36 (72.0%) 42 (84.0%) 29 (58.0%) 109/150 (72.7%) $7.26 39 (78.0%) 42 (84.0%) 35 (70.0%) 115/150 (76.7%) $6.27 33 (66.0%) 40 (80.0%) 22 (44.0%) 97/150 (64.7%) $5.07 23 (46.0%) 34 (68.0%) 12 (24.0%) 68/150 (45.3%) $4.03 39 (78.0%) 42 (84.0%) 30 (60.0%) 111/150 (74.0%) $5.67
9,533,664 8,366,804 7,714,444 6,894,052 8,027,932
146,786,597 1,931,367 107,114,322 1,961,024 76,644,299 1,591,815 55,705,859 1,226,625 93,420,273 1,753,243
63,557 55,778 51,429 45,960 53,519
978,577 714,095 510,961 371,372 622,801
12,875 13,073 10,612 8,177 11,688
$6.31 $5.20 $3.60 $3.48 $4.78
7,730,307 6,213,547 5,133,709 5,179,880 6,025,226
114,301,139 1,981,231 84,881,875 1,809,862 49,002,330 1,274,416 46,117,197 1,214,226 73,353,048 1,684,501
71,576 57,532 47,534 47,961 55,789
1,058,343 785,943 453,725 427,011 679,194
18,344 16,757 11,800 11,242 15,597
$7.32 $6.25 $4.76 $4.08 $6.97
10,678,999 155,441,947 1,663,109 8,297,995 103,038,820 2,027,443 6,956,536 76,181,499 1,473,820 6,408,116 58,432,458 1,301,752 9,008,906 117,821,938 2,247,297
79,103 61,466 51,529 47,467 66,732
1,151,421 763,250 564,307 432,833 872,755
12,319 15,018 10,917 9,642 16,646
$5.97 $5.81 $3.82 $3.01 $4.72
7,018,272 6,752,178 5,310,968 4,542,968 6,010,939
77,980 75,024 59,010 50,477 66,788
1,302,429 1,105,387 613,551 463,355 849,109
19,751 21,943 14,716 11,231 17,710
Python (34 exercises × 3 = 102 tasks) α = 0.15 24 (70.6%) 29 (85.3%) 18 (52.9%) 70/102 (68.6%) α = 0.20 21 (61.8%) 26 (76.5%) 19 (55.9%) 67/102 (65.7%) α = 0.30 21 (61.8%) 26 (76.5%) 15 (44.1%) 63/102 (61.8%) α = 0.35 17 (50.0%) 25 (73.5%) 8 (23.5%) 50/102 (49.0%) crane α = 0.25 (ref) 27 (79.4%) 31 (91.2%) 19 (55.9%) 74/102 (72.5%) JavaScript (50 exercises × 3 = 150 tasks) α = 0.15 α = 0.20 α = 0.30 α = 0.35 crane α = 0.25(ref)
Go (36 exercises × 3 = 108 tasks) α = 0.15 19 (52.8%) 25 (69.4%) 12 (33.3%) 57/108 (52.8%) α = 0.20 24 (66.7%) 29 (80.6%) 19 (52.8%) 74/108 (68.5%) α = 0.30 19 (52.8%) 26 (72.2%) 13 (36.1%) 60/108 (55.6%) α = 0.35 14 (38.9%) 18 (50.0%) 4 (11.1%) 33/108 (30.6%) crane α = 0.25 (ref) 27 (75.0%) 30 (83.3%) 18 (50.0%) 72/108 (66.7%) Java (45 exercises × 3 = 135 tasks) α = 0.15 2 (4.4%) 7 (15.6%) 0 (0.0%) 11/135 (8.1%) α = 0.20 23 (51.1%) 34 (75.6%) 13 (28.9%) 74/135 (54.8%) α = 0.30 22 (48.9%) 33 (73.3%) 7 (15.6%) 63/135 (46.7%) α = 0.35 14 (31.1%) 26 (57.8%) 6 (13.3%) 50/135 (37.0%) crane α = 0.25 (ref) 24 (53.3%) 37 (82.2%) 10 (22.2%) 70/135 (51.9%) Rust (30 exercises × 3 = 90 tasks) α = 0.15 11 (36.7%) 20 (66.7%) 6 (20.0%) α = 0.20 16 (53.3%) 22 (73.3%) 7 (23.3%) α = 0.30 11 (36.7%) 21 (70.0%) 5 (16.7%) α = 0.35 9 (30.0%) 16 (53.3%) 3 (10.0%) crane α = 0.25 (ref) 12 (40.0%) 22 (73.3%) 9 (30.0%)
40/90 (44.4%) 43/90 (47.8%) 37/90 (41.1%) 31/90 (34.4%) 41/90 (45.6%)
37
117,218,617 1,777,625 99,484,904 1,974,889 55,219,647 1,324,501 41,701,994 1,010,861 76,419,820 1,593,906
Table 42: 30B GSP-threshold τ sweep detailed Roo-Eval metrics by language, including token usage. Model
pass@1
pass@3
pass_all
iter pass
ref. cost Input total Cached total Output total Input avg Cached avg Output avg
Python (34 exercises × 3 = 102 tasks) tau030 19 (55.9%) 26 (76.5%) 17 (50.0%) 64/102 (62.7%) tau0003 24 (70.6%) 27 (79.4%) 19 (55.9%) 71/102 (69.6%)
$3.36 $4.38
4,760,484 46,344,278 5,733,004 63,733,531
1,188,137 1,566,538
46,671 56,205
454,355 624,838
11,648 15,358
tau030 31 (62.0%) 40 (80.0%) 24 (48.0%) 97/150 (64.7%) $5.43 tau0003 38 (76.0%) 43 (86.0%) 34 (68.0%) 114/150 (76.0%) $5.60
7,887,087 85,462,251 7,970,896 90,562,897
1,714,622 1,753,504
52,580 53,139
569,748 603,752
11,430 11,690
$4.19 $4.94
5,541,504 60,447,267 6,062,990 74,891,532
1,500,727 1,782,621
51,310 56,138
559,696 693,440
13,895 16,505
$5.34 $6.18
7,497,429 82,818,956 1,745,685 8,258,313 100,277,698 2,014,566
55,536 61,172
613,473 742,797
12,931 14,922
$4.47 $5.29
5,793,208 73,444,760 6,419,556 88,119,661
64,368 71,328
816,052 979,107
16,349 19,942
JavaScript (50 exercises × 3 = 150 tasks)
Go (36 exercises × 3 = 108 tasks) tau030 18 (50.0%) 28 (77.8%) 8 (22.2%) 53/108 (49.1%) tau0003 23 (63.9%) 30 (83.3%) 15 (41.7%) 68/108 (63.0%) Java (45 exercises × 3 = 135 tasks) tau030 24 (53.3%) 35 (77.8%) 7 (15.6%) 66/135 (48.9%) tau0003 24 (53.3%) 35 (77.8%) 10 (22.2%) 70/135 (51.9%) Rust (30 exercises × 3 = 90 tasks) tau030 10 (33.3%) 20 (66.7%) 2 (6.7%) tau0003 14 (46.7%) 22 (73.3%) 6 (20.0%)
31/90 (34.4%) 45/90 (50.0%)
1,471,464 1,794,838
Component-ablation detailed per-language results. Table 43 reports per-language pass metrics, reference-cost proxy, and recorded local-vLLM token usage for each row in this ablation family. Table 43: 30B component ablation detailed Roo-Eval metrics by language, including token usage. Model
pass@1
pass@3
pass_all
iter pass
ref. cost Input total Cached total Output total Input avg Cached avg Output avg
Python (34 exercises × 3 = 102 tasks) noTaylor 20 (58.8%) 27 (79.4%) 15 (44.1%) 62/102 (60.8%) noT 25 (73.5%) 28 (82.4%) 19 (55.9%) 70/102 (68.6%) noGSP 20 (58.8%) 25 (73.5%) 15 (44.1%) 59/102 (57.8%)
$5.38 $5.00 $3.94
6,698,445 104,306,164 1,563,142 5,998,159 97,013,123 1,487,554 5,487,685 54,728,526 1,401,681
65,671 58,805 53,800
1,022,609 951,109 536,554
15,324 14,583 13,741
noTaylor 35 (70.0%) 45 (90.0%) 25 (50.0%) 105/150 (70.0%) $7.18 noT 35 (70.0%) 43 (86.0%) 31 (62.0%) 111/150 (74.0%) $6.85 noGSP 29 (58.0%) 41 (82.0%) 19 (38.0%) 93/150 (62.0%) $5.05
9,674,038 145,091,970 1,876,068 9,261,800 130,686,941 1,910,699 7,487,656 72,072,493 1,687,293
64,493 61,745 49,917
967,279 871,246 480,483
12,507 12,737 11,248
$5.51 $5.72 $4.37
6,598,698 104,522,691 1,678,456 6,660,774 99,054,445 1,923,562 5,751,413 60,167,905 1,609,440
61,099 61,673 53,253
967,802 917,170 557,110
15,541 17,810 14,902
$7.31 $7.39 $5.05
9,112,509 150,035,462 1,992,039 9,172,836 142,735,263 2,164,033 7,085,661 75,620,397 1,695,966
67,500 67,946 52,486
1,111,373 1,057,298 560,151
14,755 16,029 12,562
$5.98 $5.82 $3.66
7,107,117 118,415,728 1,752,833 6,916,585 108,020,668 1,817,406 4,971,062 55,546,738 1,244,441
78,967 76,850 55,234
1,315,730 1,200,229 617,185
19,475 20,193 13,827
JavaScript (50 exercises × 3 = 150 tasks)
Go (36 exercises × 3 = 108 tasks) noTaylor 22 (61.1%) 30 (83.3%) 9 (25.0%) 61/108 (56.5%) noT 21 (58.3%) 28 (77.8%) 17 (47.2%) 68/108 (63.0%) noGSP 11 (30.6%) 22 (61.1%) 8 (22.2%) 43/108 (39.8%) Java (45 exercises × 3 = 135 tasks) noTaylor 22 (48.9%) 33 (73.3%) 13 (28.9%) 70/135 (51.9%) noT 26 (57.8%) 33 (73.3%) 16 (35.6%) 76/135 (56.3%) noGSP 23 (51.1%) 31 (68.9%) 13 (28.9%) 69/135 (51.1%) Rust (30 exercises × 3 = 90 tasks) noTaylor 13 (43.3%) 20 (66.7%) 6 (20.0%) noT 11 (36.7%) 23 (76.7%) 7 (23.3%) noGSP 17 (56.7%) 21 (70.0%) 7 (23.3%)
38/90 (42.2%) 43/90 (47.8%) 45/90 (50.0%)
80B component-ablation detailed per-language results. Table 44 reports the 80B CRANE full recipe and its one-component removals by language. All rows use the same α = 0.15, τ = 0.03, Qwen3-Next-80B-A3B Instruct/Thinking pair, and Roo-Eval serving configuration; each ablation removes exactly one of Taylor scaling, median-magnitude denoising, or GSP protection.
38
Table 44: 80B CRANE component-ablation detailed Roo-Eval metrics by language, including token usage. Model
pass@1
pass@3
pass_all
iter pass
ref. cost Input total Cached total Output total Input avg Cached avg Output avg
Python (34 exercises × 3 = 102 tasks) CRANE 30 (88.2%) 33 (97.1%) 27 (79.4%) 90/102 (88.2%) noT 29 (85.3%) 33 (97.1%) 24 (70.6%) 85/102 (83.3%) noTaylor 30 (88.2%) 33 (97.1%) 25 (73.5%) 89/102 (87.3%) noGSP 30 (88.2%) 32 (94.1%) 23 (67.6%) 83/102 (81.4%)
$10.54 $11.10 $12.83 $15.86
3,807,607 4,035,791 4,499,559 4,847,925
46,484,492 933,088 52,833,706 912,765 67,386,007 977,139 102,658,102 1,004,775
37,329 39,567 44,113 47,529
455,730 517,978 660,647 1,006,452
9,148 8,949 9,580 9,851
CRANE 46 (92.0%) 49 (98.0%) 42 (84.0%) 137/150 (91.3%) $13.85 noT 45 (90.0%) 47 (94.0%) 44 (88.0%) 137/150 (91.3%) $14.80 noTaylor 46 (92.0%) 48 (96.0%) 42 (84.0%) 137/150 (91.3%) $15.40 noGSP 36 (72.0%) 46 (92.0%) 31 (62.0%) 117/150 (78.0%) $17.02
5,555,281 5,968,854 5,810,693 6,491,278
61,325,457 69,574,697 81,208,201 99,738,353
1,130,758 1,133,874 1,099,342 1,037,504
37,035 39,792 38,738 43,275
408,836 463,831 541,388 664,922
7,538 7,559 7,329 6,917
$13.11 $11.48 $18.01 $15.16
4,654,524 4,075,592 6,650,666 5,038,572
55,659,080 48,670,131 87,594,557 85,575,230
1,209,340 1,059,954 1,432,894 1,102,255
43,097 37,737 61,580 46,653
515,362 450,649 811,061 792,363
11,198 9,814 13,268 10,206
$19.36 $25.21 $22.61 $19.76
7,543,322 9,168,853 8,457,768 7,603,925
90,934,720 1,529,337 122,372,257 2,034,221 108,610,839 1,805,331 98,546,623 1,480,701
55,876 67,917 62,650 56,325
673,591 906,461 804,525 729,975
11,328 15,068 13,373 10,968
$14.57 $15.66 $15.85 $18.94
5,006,504 5,328,776 5,372,024 6,267,397
67,960,906 1,270,158 73,042,138 1,373,749 73,642,864 1,399,325 112,976,962 1,281,274
55,628 59,209 59,689 69,638
755,121 811,579 818,254 1,255,300
14,113 15,264 15,548 14,236
JavaScript (50 exercises × 3 = 150 tasks)
Go (36 exercises × 3 = 108 tasks) CRANE 31 (86.1%) 33 (91.7%) 29 (80.6%) 92/108 (85.2%) noT 31 (86.1%) 34 (94.4%) 25 (69.4%) 91/108 (84.3%) noTaylor 30 (83.3%) 34 (94.4%) 25 (69.4%) 87/108 (80.6%) noGSP 31 (86.1%) 32 (88.9%) 24 (66.7%) 84/108 (77.8%) Java (45 exercises × 3 = 135 tasks) CRANE 28 (62.2%) 37 (82.2%) 20 (44.4%) 89/135 (65.9%) noT 30 (66.7%) 38 (84.4%) 20 (44.4%) 91/135 (67.4%) noTaylor 23 (51.1%) 37 (82.2%) 13 (28.9%) 74/135 (54.8%) noGSP 33 (73.3%) 39 (86.7%) 22 (48.9%) 94/135 (69.6%) Rust (30 exercises × 3 = 90 tasks) CRANE 24 (80.0%) 24 (80.0%) 21 (70.0%) 68/90 (75.6%) noT 19 (63.3%) 25 (83.3%) 16 (53.3%) 63/90 (70.0%) noTaylor 22 (73.3%) 27 (90.0%) 18 (60.0%) 69/90 (76.7%) noGSP 22 (73.3%) 27 (90.0%) 17 (56.7%) 65/90 (72.2%)
39