Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First

arXiv:2608.04804v1 [cs.SE] 5 Aug 2026

Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First Ishaan Bhola

Adithyan Krishnan

Mukunda NS

SuperAGI Research

SuperAGI Research

SuperAGI Research

60% Solve rate

Abstract—Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone. We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandboxverified, with false claims stripped before delivery. The searcher’s hidden states, together with the task text, then feed a résumébased router that dispatches the task to one of four frontier fixers. Adding a new fixer requires no retraining. On the full Python slice of SWE-bench Pro (266 tasks) under the benchmark’s official capped budget tier, SuperScout matches the best single model’s solve rate (159 of 266 for SuperScout, 158 for the best model) at about a fifth of the total cost per solve, and the reported configuration sits above the random traffic-splitting baseline. A no-router ablation, always the cheapest fixer with the handoff, ties the routed system on this benchmark, so the handoff rather than the routing decision carries the result. A paired calibration study points to the mechanism: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, though at N =99 the per-fixer effects are directional only; the searcher’s hidden states improve cost routing on the calibration labels while the handoff’s own text does not. The searcher’s compute adds less than half a cent of GPU time per task.

SuperScout

Opus 4.6 solo

58% blind-mixing line

56%

Kimi K2.5 solo

54% GPT-5.2 solo

52% $0.20

$0.30

$0.60

$1.20

Cost per solve (log scale)

Fig. 1: Cost per solve versus solve rate on SWE-bench Pro (Python-266). The dashed curve is the blind-mixing line (random cost-blind mix of Kimi K2.5 and Claude Opus 4.6). SuperScout matches Claude Opus 4.6 at about a fifth of its cost per solve, well above the line. The system point is all-in; solo points are fixer API only. The x-axis is logarithmic.

SuperScout routes after scouting, a pattern we call scrouting. A 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff: a list of implicated files, diagnostic notes, and a candidate reproduction test. These claims pass through a sandbox verification step, and claims 1 Introduction that do not check out are stripped before a downstream model Frontier-model agents now resolve real repository issues, ever sees them. A résumé router then selects among four from locating bugs to generating patches [1], but each solve frontier fixers using the task text together with the searcher’s carries a real per-task price. Strong open-weights models cost own hidden states; adding a new fixer requires no retraining. a fraction yet close fewer tasks. Any team deploying an issueFigure 1 compares SuperScout to solo frontier models on solving system therefore faces a standing choice between the SWE-bench Pro’s full Python slice [4] (266 tasks), matching expensive best model and cheaper alternatives. Existing LLM the benchmark’s official capped budget tier exactly. SuperScout routers [2, 3] make that choice from the task text alone, before matches the best single frontier model’s solve rate, resolving anything has engaged with the actual repository. 159 of 266 tasks versus 158, at $0.230 total cost per solve Replaying learned routers over the public per-task results compared to $1.274: about a fifth. The configuration sits of three SWE benchmarks reveals that solve sets are largely above the blind-mixing line (the accuracy/cost segment any nested: models that solve more tasks nearly always subsume the solve sets of weaker ones, and no router reliably improves on random split of traffic between a cheap and a strong model always selecting the strongest model (§3). Routing for accuracy would achieve). The searcher’s contribution to system cost is therefore offers little headroom. Routing for cost, by contrast, negligible: SuperScout-7B’s entire GPU bill for the evaluation was $1.13. only requires predicting when a cheaper model will suffice. A paired calibration study on 100 fresh tasks points to why Model and data: https://huggingface.co/SuperAGI/SuperScout-7B (weights), this cost reduction holds: the handoff appears to redistribute https://huggingface.co/datasets/SuperAGI/superscout-sft-search (training rather than add solving ability, lifting the three cheaper fixers corpus), https://huggingface.co/datasets/SuperAGI/superscout-router-features (router features). Code and data: https://github.com/TransformerOptimus/ while slightly hurting the strongest, and the searcher’s hidden superscout. states improve cost routing where the handoff’s own text does

TABLE I: Capability comparison of SuperScout with related not (§8). routing and localization systems. Columns: Searcher, a model We make five contributions: trained to scout the repository before any fix is attempted; • An engage-then-route architecture with a trained searcher Handoff, the searcher’s work product is consumed by the whose verified handoff is consumed by the chosen fixer. downstream fixer rather than discarded; N -way, selection • A zero-cost replay audit of published per-task outcomes on over a pool of more than two fixers; Onboard, adding a new three SWE benchmarks showing that solve sets are largely fixer requires no retraining of any learned component; States, nested and that no learned router reliably beats always routing features include the searcher’s internal hidden states. calling the strongest model, which motivates routing for Concurrent code-routing work escalates between a fixed pair cost rather than accuracy. and discards the cheap model’s evidence on escalation; profile• Routing features drawn from the searcher’s hidden states, based routers select from the task prompt alone, before any with a résumé-based N -way pool where adding a new exploration; trained small localizers produce evidence a router fixer requires no retraining. could use but perform no routing. ✓† : SWE-Fixer’s retriever • A matched-protocol evaluation on SWE-bench Pro’s Python census showing frontier-matching accuracy at feeds one fixed larger editor, while LocAgent and SweRank stop at localization. Rows: SWE-Router [17], CodeRescue [18], about a fifth of the total cost per solve. TRACE-Router [19], profile routers [14–16], small localiz• A paired calibration study measuring the handoff’s reers [10–12]. distribution pattern and the router’s feature design space, plus a verification gate that strips the searcher’s false System Searcher Handoff N -way Onboard States reproduction claims. SWE-Router × × × × ×

2 Related Work

CodeRescue TRACE-Router Profile routers Small localizers

× × × ✓

× × × ✓†

× ✓ ✓ ×

× × ✓ ×

× × × ×

SWE-agent [5] introduced the agent-computer-interface paradigm for autonomous issue resolution in real repositories. SuperScout ✓ ✓ ✓ ✓ ✓ Agentless [6] showed that a fixed pipeline, with no agent autonomy at all, can achieve competitive resolve rates. SWEsmith [7] addresses data scarcity by synthesizing large-scale we pursue. CodeRescue [18] routes among recovery actions training corpora for such agents, while SWE-Gym [8] and R2E- (reflect, replan, or escalate) inside a single agent’s trajectory Gym [9] supply executable training environments built from rather than among fixer models. TRACE-Router [19] formulates real repositories. The community evaluates these systems on model selection as a contextual bandit over an N-way pool, SWE-bench [1] and its harder successor SWE-bench Pro [4]. though it is evaluated outside software engineering. Self-play A parallel line of work trains small models specifically SWE-RL [20] trains solvers via self-play and evaluates on for code localization. SWE-Fixer [10] trains a 7B retriever SWE-bench Pro but does not route. As Table I summarizes, whose output feeds one fixed larger editor. LocAgent [11] SuperScout is distinct in combining a trained searcher, a handoff and SweRank [12] similarly train compact models to identify the fixer actually consumes (after verification), an N-way fixer fault locations within a repository. These systems produce pool where adding a new fixer requires no retraining, and exactly the kind of evidence a router could consume, yet none routing features drawn from the searcher’s hidden states. of them routes: the localizer’s output terminates in a single, 3 Problem Setup and Metrics predetermined consumer. LLM routing has been studied along two axes. Cost-quality 3.1 Task and Cost Model routers such as RouteLLM [2], FrugalGPT [3], and Hybrid We study repo-level issue resolution: given the text of an LLM [13] learn to dispatch queries to cheaper or stronger models based on predicted difficulty. Profile-based selectors issue and a snapshot of the repository, the system must produce take a complementary approach, building per-model signatures a source-code patch whose correctness is judged by held-out from benchmark outcomes or learned embeddings and matching tests [1, 4]. Frontier language models already solve these tasks incoming queries against them [14–16]. Both families share at high rates, but their per-task cost is substantial. Strong a structural property: they decide from the task prompt, and open-weights models are far cheaper yet less reliable. In a optionally from model profiles, before any system has engaged deployment that selects from a pool of candidate fixer models, the natural question is whether one can retain the strongest with the concrete problem instance. Several concurrent efforts address routing in code-generation model’s effectiveness while spending less. settings. SWE-Router [17] probes each issue with a weak To answer that question we adopt a single yardstick, defined model and then escalates between exactly two models; on before any experiment: total cost per solve at a matched escalation the strong model restarts from scratch, so the probe’s solve rate. “Total” means every system component’s spend, work is not consumed. Its theoretical analysis of trajectory- including searcher GPU time, verification sandboxes, and fixer conditioned routing nonetheless supports the design direction API calls, not just the final model invocation. A system claims

a cost improvement only if its solve rate matches that of the best individual model. This metric exposes a trivial lower bound. Any point on the line segment between a cheap model’s (cost, accuracy) and a strong model’s (cost, accuracy) is achievable by randomly splitting traffic between the two, a strategy we call blind mixing. A routing system is interesting only if it operates above that segment. We describe the operational details of the blind-mixing line in §6. 3.2 Why Accuracy-Only Routing Is Insufficient Before building SuperScout we replayed a set of learned routers over the publicly available per-task results of three SWE-coding benchmarks: SWE-bench Verified [21], SWEbench Multilingual [22], and SWE-bench Pro [4]. The replay incurs zero model cost, and the routers span four families: embedding retrieval (k-NN), trained heads (logistic and MLP), clustering, and a language-rule baseline. Two findings shaped the design that followed. First, solve sets are strongly nested: the tasks a weaker model solves are largely a subset of those the strongest model solves. Measured as set containment, the overlap is 0.941, 0.912, and 0.773 across the three benchmarks. Nor is this an artifact of comparing cheap models with frontier ones: replaying pools built from frontier models alone, one per lab, leaves containment at 0.90 to 0.93. These models are generally strong rather than complementary specialists, so an accuracyoriented router has almost no room to combine their strengths. Figure 2 sketches the consequence: between any two models the accuracy prize is only a thin sliver, while the large shared region is where a cheaper model would have sufficed all along. Second, no learned router we tested exceeded the solve rate of always picking the strongest model; every observed gap fell within noise. Taken together, these results suggest a plain conclusion: routing for accuracy has little headroom on current issue-resolution benchmarks. Routing for cost, by contrast, does not require complementary skills at all. It requires only the ability to predict when a cheaper model will suffice. Full audit tables appear in Appendix A.

(a) ANY TWO FRONTIER PEERS solved by only one: rare

all tasks

solved by both

(b) A CHEAPER MODEL VS . THE STRONGEST

Cheaper model’s solve set Strongest model’s solve set

a cheaper model suffices here: route for cost

solved only by the cheaper model: rare

solved by neither

Fig. 2: Solve sets are nearly nested (schematic). (a) Even two frontier peers solve almost the same tasks, so accuracy routing has little to win. (b) A cheaper model’s set sits mostly inside the strongest model’s; that shared region is where routing for cost pays.

4 The SuperScout System Figure 3 traces a single task through the pipeline. The task enters a search phase, where SuperScout-7B explores the repository and writes a structured handoff. A sandbox gate then verifies the handoff’s reproduction claims before a résumé router selects one fixer from a pool of frontier models. That fixer receives the task and the surviving handoff, produces a patch, and submits it to official scoring. The entire trajectory is a single pass: no parallel sampling, no cross-model escalation mid-task. When the searcher produces no handoff at all, the chosen fixer proceeds from the issue text alone; this fallback defines the floor as fixer-solo performance. 4.1 The Searcher and Its Handoff

SuperScout-7B is a 7B model built on Qwen2.5-Coder [23], trained exclusively for the search phase (§5). Given a task, it 3.3 Goal explores the repository, localizes the implicated files, attempts Our objective inverts the premise of prior routing work. to write a failing reproduction test, and then produces a Rather than seeking accuracy gains by combining models, we handoff document before stopping. The handoff is a structured aim to match the strongest available model’s solve rate at a artifact kept short by design, typically about a page of text (4 KB). It contains implicated files with line regions, ranked by materially lower total cost per solve. The thesis is that routing should happen after engaging with confidence; a reproduction attempt specifying a file, command, the problem, not from the task text alone. In SuperScout a and observed output; dead ends the searcher already tried; small searcher model first explores the repository; both its and free-form repository notes. A verified example appears in explicit work product (a structured handoff) and its internal Appendix M. representation of the task (hidden states) then inform the When the search exhausts its turn budget without committing routing decision. The evaluation partly supports this thesis: to a handoff, a single extra generation step demands one. the engagement’s payoff arrives through the handoff more than Such handoffs are tagged forced and are never pooled with through the routing decision (§7). §4 details the architecture. spontaneous ones anywhere in this paper.

S EARCH

H ANDOFF

SuperScout-7B Task issue text

explores the repo sampled decoding bounded turns

Structured handoff

G ATE

ROUTE

Verify-then-strip

Résumé router

replays the repro claim

task text + SuperScout-7B hidden states

implicated files repo notes repro test

✓ replays: kept × fails: stripped

vs. one résumé per fixer

F IX

Fixer pool

▶

GPT-5.2 Claude Opus 4.6 Gemini 3 Flash Kimi K2.5 + new fixer: résumé only

Patch Official scoring

no handoff: the chosen fixer proceeds from the issue text alone

the chosen fixer gets the task + surviving handoff

Fig. 3: The SuperScout pipeline. SuperScout-7B explores the repository and emits a structured handoff, which a sandbox gate verifies before a résumé router selects one of four frontier fixers. The dashed path is the fallback: when no handoff is produced, the chosen fixer proceeds from the issue text alone. Adding a new fixer requires only a résumé, not retraining. TABLE II: The résumé router, end to end. Each fixer’s résumé stores two embedding centroids and a base rate, computed from public per-task outcomes. A logistic head per feature space scores P (solve); the router walks the pool in cheap-first order, stopping at the first fixer clearing θ. Adding a new fixer requires only 25–50 public outcomes and no retraining. Component

Specification

Embedder

Qwen3-Embedding-0.6B frozen 4096-token cap, L2 -normalized

Résumé

per fixer: mean embedding of solved, mean of failed, base rate from 25–50 public outcomes

Features

cos(x, s), cos(x, f ), cos(x, s)− cos(x, f ), p, cos(x, s)·p, (cos(x, s)− cos(x, f ))·p uncentered, per feature space

Hidden state

SuperScout-7B layer −4, pre-decode final-position state (3584-d)

Scorer

logistic regression per space, uniform probability average, θ=0.30, cheapest-adequate

New fixer

25–50 public outcomes; no retraining of SuperScout-7B, the embedder, or the scorer

4.2 Verify-then-Strip

The searcher’s reproduction claims are not trusted. Before any fixer sees a handoff, a sandbox replays the claimed reproduction command against the unpatched repository. A claim that does not genuinely fail is stripped: both the test file and the claim itself are removed from the handoff. A verified claim, by contrast, is materialized so the fixer can use it directly. Calibration revealed that most reproduction claims are in fact false (§8), and §7 quantifies the guard’s effect at benchmark scale. Injection is blanket: every routed fixer receives the surviving handoff identically.

4.3 The Résumé Router Each fixer in the pool is summarized by a résumé built from 25–50 public per-task outcomes. A résumé stores three quantities: the mean embedding of tasks the fixer solved, the mean embedding of tasks it failed, and a base solve rate. Table II lists the full specification. At routing time the task is embedded in two complementary feature spaces. Its text is encoded by a frozen off-the-shelf embedder [24]. Its content as SuperScout-7B experienced it is captured through the searcher’s 3,584-dimensional pre-decode hidden state, drawn from the fourth-from-last layer at the final token position. In each feature space the router computes uncentered cosine similarities between the task and every résumé’s solved and failed centroids, their difference, and the base rate; a single logistic regression per feature space, shared across fixers, then scores P (solve) for each fixer from its résumé-relative features. The routing head blends the tasktext and hidden-state feature sets by uniformly averaging their predicted probabilities. Routing walks the pool in cheap-first order and assigns the task to the first fixer whose predicted probability clears a caution threshold θ; if none clears it, the task falls to a designated anchor model (§6). Logistic regression was chosen over a multi-layer perceptron during calibration, where the simpler scorer won repeatedly (§8). 4.4 Extensibility Adding a new fixer requires no retraining. A new model’s résumé consists of two averaged embeddings and a base rate, computed from its public outcomes. The embedder is frozen and SuperScout-7B is untouched; §7 presents a measured discussion.

5 Training the Searcher 5.1 Data and Supervised Training SuperScout-7B is trained on search-phase demonstrations sliced from openly licensed agent trajectories produced by other systems. Each demonstration captures one complete search

Value

Corpus Examples trained on Built / frozen set Held-back shelf (untrained) Tokens per epoch Packed 32k blocks (fill)

19,905 19,911 9,478 382.4M 12,723 (91.7%)

Language mix Go Python TypeScript JavaScript

7,304 (36.7%) 7,417 (37.3%) 3,932 (19.7%) 1,258 (6.3%)

Trace sources Open-SWE-Traces SWE-rebench-openhands SWE-Hero-openhands

Go/TS/JS + Py Python Python

Filters Traces kept per issue Verified reproduction Excluded repositories Excluded held-out issues

top 2 97.7% 23 450

0.4 0.306

0.3 0.2

2.65× timeoutmatched 0.110

0.1

Ratio, temp 0.9 / greedy

Component

Decomposition (dial-100)

Find rate (vault-450)

Mean gold-file recall

TABLE III: Composition of SuperScout-7B’s supervised search corpus. Each example is a full search trajectory with a synthesized handoff-emission turn as the final supervised step; loss is on assistant turns only. Traces are success-filtered and deduplicated to two per issue. The 23 evaluation repositories and 450 held-out vault issues are excluded; both exclusions were verified by a programmatic gate.

3.03×

3 2 parity

1

0.96×

0

0 greedy temp 0.9

commit rate

handoff quality

Fig. 4: Sampled decoding is what makes the searcher useful. Left: switching from greedy to temperature 0.9 raises the find rate from 0.110 to 0.306 (2.65×, n=426 of 450). Right: on a separate 100-task dial set, commitment rises 3.03× while per-handoff quality holds at 0.96×; on the exam itself emission rises 3.4× against an 18% per-handoff recall cost (Appendix K). The two panels use different task sets; the right panel is a mechanism check, not a replication. the demonstrations above. Hyperparameters are reported in Appendix L. 5.2 Decoding Matters More Than Expected The most consequential training-era finding was not about the training itself but about inference. On a 450-task held-out exam, greedy decoding finds the right files at a rate of 0.1104. A single sampled draw at temperature 0.9 reaches 0.3058, a 2.65× gain after matching for infrastructure timeouts. The exam’s own decomposition shows what sampling buys and what it spends: the searcher’s emission rate rises from 0.213 to 0.718 (3.4×), while recall per emitted handoff falls from 0.517 to 0.426, an 18% quality cost (Appendix K). Sampling trades a slice of per-handoff quality for a much larger gain in willingness to commit, a strongly net-positive exchange. A separate 100-task dial set shows the same mechanism with commitment up 3.03× and per-handoff quality flat at 0.96× (Figure 4); the quality cost visible on the exam does not appear there, so the dial set is a mechanism check, not a replication. Greedy decoding, it turns out, makes the searcher reluctant to declare a result; sampling at moderate temperature recovers ability the model already has. The shipped configuration is therefore a single sampled draw at temperature 0.9 (§6).

episode: the agent explores a repository, localizes the fault, and writes a failing reproduction, with a synthesized handoffemission turn appended as the final supervised target. Within that synthesized turn, the file list is extracted deterministically from the trace and the reproduction record is copied verbatim; only the free-form notes are written by a model, the openweights gpt-oss-120b [25]. Traces are drawn from three public sources (O PEN -SWE-T RACES [26], SWE- REBENCH OPENHANDS [27, 28], and SWE-H ERO [29]; the latter two are trajectories of the OpenHands scaffold [30]) and then successfiltered against the gold patch, keeping only trajectories that actually found the right files. Deduplication retains the two highest quality-ranked traces per issue, so the corpus counts distinct bugs rather than retellings of the same fix. The resulting dataset contains 19,911 examples. Six rows carrying malformed tool-call wrappers are dropped at load time, giving the 19,905 examples actually trained on. The realized language mix is Python 37.3%, Go 36.7%, TypeScript 19.7%, and JavaScript 6.3%. Table III summarizes the corpus 5.3 Language Transfer composition. SuperScout-7B is trained on four languages (Go, Python, Contamination control was decided on day one: a 23repository blocklist covering all 11 SWE-bench Pro repositories TypeScript, JavaScript), yet SuperScout is deployed against and 12 SWE-bench Verified [21] repositories is excluded from repositories in languages the searcher has never seen. On every training and calibration set. A programmatic gate verified the nine-language SWE-bench Multilingual evaluation [22], which contains no Python and so covers three of the four zero blocklist hits in the frozen file. We fine-tune Qwen2.5-Coder-7B [23] with LoRA [31] on training languages (Figure 5), spontaneous-handoff localization a single GPU. Before training, the base model’s localization quality is higher on the six never-trained languages (file-level rate is near zero; the search behavior is entirely learned from F1 = 0.630) than on the three trained ones (F1 = 0.455). The

Go JavaScript TypeScript*

0.546 0.389

6 Experimental Setup 6.1 Benchmark and Protocol

0.343

We evaluate on SWE-bench Pro [4], using the full Python slice: all 266 tasks, a census rather than a sample. These span 0.701 three repositories: 96 from ansible, 91 from openlibrary, and 79 0.667 Rust from qutebrowser, out of 731 tasks in the public set. We chose 0.630 PHP Pro because its tasks are harder and less saturated than earlier 0.435 C SWE-bench variants, and because all 11 of its repositories 0.708 C++* were excluded from SuperScout-7B’s training data from day 0 0.2 0.4 0.6 0.8 one via the contamination blocklist of §5. Spontaneous handoff file F1 Our protocol matches the benchmark’s official capped budget trained (Go/JS/TS) never trained (6 langs) tier exactly: a 50-LLM-call turn cap and a $2.00 per-attempt trained pool 0.455 never-trained pool 0.630 cost limit. The cap is silent; the agent is never told a budget Fig. 5: Localization transfers to languages SuperScout- exists. Source-code inspection of the official scaffold found no 7B was never trained on. Per-language file-level F1 of budget sentence in any prompt; when the cap is exhausted, the spontaneous handoffs (searcher alone, single draw, SWE-bench current working diff is auto-submitted. Task input follows the Multilingual). Solid bars: three trained languages; hatched: scaffold’s exact three-field concatenation of problem statement, six never-trained. Dashed lines mark pooled aggregates (0.455 requirements, and interface. For 105/266 tasks a byte-lossless trained, 0.630 never-trained). The TypeScript and C++ cells (12 JSON decode of double-serialized fields was applied; this assigned tasks each; 7 and 8 spontaneous handoffs analyzed) deviation is logged. Fixer prompts are byte-identical to the are indicative only. SWE-bench Multilingual leaderboard templates [22], and a single task-text definition is used throughout, for the searcher, every fixer, and the router embedding alike. inversion survives a difficulty-matched control restricted to single-gold-file tasks, where recall on never-trained languages 6.2 Frozen System Configuration reaches 0.791 versus 0.524 for the trained pool. JavaScript The searcher, gate, résumés, embedder, and router weights and TypeScript account for much of the trained pool’s lower were frozen before first contact with the benchmark. average; the TypeScript and C++ cells each rest on 12 assigned SuperScout-7B serves at temperature 0.9, single draw, with tasks, of which 7 and 8 respectively produced the spontaneous pinned sampling parameters, per-episode cache isolation, a handoffs analyzed, and are indicative only (full per-language serving-health canary gate, and a 40-turn search budget. The table in Appendix J). handoff policy is blanket injection: every routed fixer receives The contrast carries a confound: JavaScript and TypeScript the surviving handoff identically. Verify-then-strip runs first are at once the trained pool’s weakest cells and its smallest on every reproduction claim (mechanism in §4). The router’s training slices, and both may be structurally harder to localize logistic head was fit on all calibration data (recipe in Table II) in (dynamic imports, build artifacts) independently of train- and operates at caution threshold θ=0.30. This threshold ing exposure, so the trained-versus-never-trained comparison was initially calibrated against solo-fixer outcomes; because should not be read as causal. Our hypothesis for the pattern, the handoff systematically lifts inexpensive fixers, it was and it is a hypothesis, is that SuperScout-7B learned a search recalibrated on the 99-task lab set using with-handoff outcomes method, not a language-specific vocabulary. The exploration via the pre-existing matched-point rule. The router’s benchmark strategies it relies on (reading directory trees, tracing imports, result is evaluated by routing each task and scoring it against scanning test files) are structural operations that generalize that fixer’s measured episode under the official capped protocol; across languages; the specific tokens involved matter less than every one of the 266 routed picks has a real measured episode behind it. the strategy of following them. Java Ruby

0.612

6.3 Arms, Baselines, and Cost Accounting Reinforcement learning. We built and validated a Four measured arms run fully paired on the same 266 tasks: GRPO [32] training rig and ran 50 clean steps. Learning was three solo fixers (Claude Opus 4.6 [33], GPT-5.2 [34], and flat. A trace-level autopsy showed that the reward signal was Kimi K2.5 [35]) and the SuperScout fixer runs with injected real but aimed at a behavior already near its ceiling after handoffs. Both frontier models were run in full as solo baselines supervised training: the model could localize when it chose to because our calibration study and a small paid probe disagreed commit, and sampling had already recovered that commitment. about which was stronger; the resolution is itself a result (§7). We shelved RL in favor of the decoding fix described above; At the calibrated operating point (θ=0.30) the router’s fallback rig details are in Appendix L. never fires, so anchor choice is moot.

TABLE IV: Main results on SWE-bench Pro (Python-266). TABLE V: SuperScout-7B’s file localization on SWE-bench SuperScout matches the pool’s best single model (Claude Opus Pro (Python-266). Per-task mean recall and precision against 4.6) at about a fifth of its cost per solve; 159 versus 158 is a the gold patch’s file set. Spontaneous and forced handoffs are one-task gap at n=266, a match, not a win. The last row is reported separately. All numbers are from a single sampled the no-router ablation, which ties the routed system on this draw at temperature 0.9 (no majority vote). benchmark (§7.1). Handoff type

Recall

Precision

All-gold

Rate $/task $/solve

vs. mix

Overall (n=266)

0.566

0.821

24.8%

SuperScout (router, θ=0.30)

159 59.77% $0.137 $0.230

+3.60

Spontaneous (n=206) Forced (n=60)

0.586 0.499

0.833 0.780

28.2% 13.3%

Claude Opus 4.6 solo Kimi K2.5 solo GPT-5.2 solo

158 59.40% $0.757 $1.274 149 56.02% $0.106 $0.190 139 52.26% $0.570 $1.091

— — —

System

Solves/266

one cheap fixer reaches parity is a property of this pool and benchmark, not a general rule: in deployment that answer is not No-router ablation (Kimi 159 59.77% $0.136 $0.227 +3.60 known in advance, and the router is the component that predicts K2.5 + handoff) it, from free public data, before any spend. The ablation is Solves are out of n=266, top to bottom: 159, 158, 149, 139, 159. The ex-ante specifiable too, but only hindsight shows it reaches SuperScout row and the ablation row share one all-in cost convention: fixer API spend plus the SuperScout-7B searcher’s GPU bill plus verify-then-strip parity; the router turns that gamble into a calibrated decision. sandbox infrastructure, amortized over 266 tasks. Solo rows are fixer API Both system rows sit above the blind-mixing line defined in spend only. “vs. mix” is percentage points above the Kimi K2.5–Claude §6 by +3.60 percentage points at their respective cost points Opus 4.6 blind-mixing line at the row’s own $/task. (visible in Figure 1). SuperScout-7B’s entire GPU bill for all Two baselines anchor interpretation. The strongest-solo 266 search episodes was $1.13, under half a cent per task. Why were both frontier models run in full as solo baselines? baseline is the single fixer with the highest solve rate. The blindmixing line traces the accuracy-versus-cost segment produced Our calibration study and a small paid probe disagreed about by randomly assigning each task to Kimi K2.5 or Claude Opus which was stronger. On the fresh calibration tasks (§8), GPT4.6 in varying proportions; a system point above this line does 5.2 was the strongest solo fixer at 60.6% solve rate. A small paid probe of 15 tasks per fixer leaned the other way: Claude something smarter than chance mixing. Cost accounting is deliberately conservative against Super- Opus 4.6 solved two more tasks, but p=0.625, well below Scout. System costs are all-in: routed-fixer API spend plus the pre-declared decision bar, while inverting the calibration SuperScout-7B’s entire GPU bill plus verify-then-strip sandbox prior. Both were therefore run in full. The complete paired infrastructure, all amortized per task. Solo arms count only their comparison resolved the question: Claude Opus 4.6 beats GPTpure fixer API spend. We report total cost per solve at a matched 5.2 by +7.14 percentage points on the same 266 tasks, with solve rate; measured spend is disclosed in Appendix N. Scoring 44 tasks solved only by Claude Opus 4.6 versus 25 solved uses the benchmark’s official harness; a gold-patch control only by GPT-5.2 (exact McNemar test [37], p=0.029). One further observation from the solo baselines deserves passes 265/266 tasks. The single failure stems from a dataset artifact (a truncated test-spec identifier) that is unwinnable for note. Kimi K2.5 solo resolves 149 of 266 tasks (56.02%) at $0.106 per task, dominating GPT-5.2 solo (139/266, 52.26%, every arm equally and is kept in all denominators. $0.570 per task) on both axes, gaining +3.76 percentage points 7 Results of accuracy at roughly one fifth of the per-task cost. 7.1 Headline

7.2 Component Analysis

Table IV summarizes the primary evaluation. SuperScout a) Localization quality. solves 159 of 266 tasks (59.77%) at a total cost per solve Table V reports the quality of SuperScout-7B’s file localizaof $0.230. The pool’s best single model, Claude Opus 4.6 tion. Against a mean of 3.44 gold files per task, the searcher run solo, solves 158 of 266 (59.40%) at $1.274 per solve. names a median of 2.0 files, achieving per-task mean recall of SuperScout matches this ceiling at about a fifth of the cost per 0.566 and precision of 0.821; in 24.8% of tasks every gold file solve. The one-task gap between 159 and 158 at n=266 is not appears in the handoff. Spontaneous handoffs (n=206) localize a meaningful difference; it is a match. substantially better than forced ones (n=60) across the board: The no-router ablation sends every task to Kimi K2.5 with recall 0.586 versus 0.499, precision 0.833 versus 0.780, and allthe handoff and no routing decision at all, also solving 159 gold rate 28.2% versus 13.3%. This gap is unsurprising: tasks of 266 at $0.227 per solve. The calibrated router concentrates that exhaust the turn budget are typically harder. It underscores 263 of 266 tasks on the cheapest fixer and adds no solves why the two handoff types are never pooled when reporting over this ablation; its three diversions to Gemini 3 Flash [36] component behavior (per-repository detail in Appendix D). cost $0.003 per solve, insurance rather than accuracy. On this b) Verify-then-strip guard. benchmark, then, the handoff carries the result and routing Of 266 handoffs, 249 included a claimed verified reprocollapses to cost allocation. Whether blanket assignment to duction of the bug. Replaying every claim inside the task

S TEP 1

S TEP 2

S TEP 3

Collect outcomes

Average into a résumé

Routable at once

θ=0.30 100

✓ × ✓ ×

tasks

75 50

existing fixer existing fixer existing fixer new fixer

25–50 public per-task outcomes

25 0

solved failed

two centroids + a base rate

the router reads it

searcher, embedder, scorer: all frozen, nothing retrains

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 P (solve) for the cheapest fixer (Kimi K2.5) task-text features hidden-state features

routing head (blend)

Fig. 6: Two feature spaces, two very different confidence distributions at the routing gate (θ=0.30). Histograms of P (solve) for Kimi K2.5 (cheapest fixer) across all 266 tasks. The task-text head spans nearly the full unit interval (sd 0.20), while the hidden-state head concentrates tightly (sd 0.07, support entirely above θ). Their blend, the routing head, gates 263 of 266 tasks to the cheapest fixer. sandbox revealed that only 50 of them (20%) were genuine, while 174 (70%) were demonstrably false. The guard stripped every false claim before any fixer saw it. Forced handoffs overclaimed more aggressively: just 9% of their reproduction claims were genuine, compared with 22% for spontaneous handoffs. These benchmark rates are worse than those observed during calibration (32% genuine, 56% false; §8), consistent with harder tasks producing more overclaiming. The guard neutralizes this overclaiming before any fixer sees it: 174 fixer prompts were stripped of confident misinformation that would otherwise have been treated as ground truth (full outcome classes in Appendix C); we did not run a pass-through arm, so the guard’s contribution to the headline is an inference from this claim census rather than a measurement (§9). c) Protocol findings. The auto-submit-at-cap mechanism rescued 24 of Claude Opus 4.6’s 158 solves and 33 of GPT-5.2’s 139. Without it, both frontier models’ solve rates would read roughly 9–13 percentage points lower. The 50-call cap binds often: it was reached in 36.5% of GPT-5.2 attempts, 17.3% of Claude Opus 4.6 attempts, and 48% of Kimi K2.5 attempts. The $2 cost cap bound only 5 times (all Claude Opus 4.6, maximum $2.09). The gold-control census confirmed 265 of 266 tasks (§6) (full cap statistics in Appendix E). d) Router behaviour. The router sends 263 of 266 tasks to Kimi K2.5: the first fixer in cheap-to-expensive order whose blended solve probability clears θ=0.30 receives the task. Figure 6 reveals how the two underlying feature spaces distribute that probability. The tasktext head spreads wide (sd 0.20, support 0.08–0.97), while the hidden-state head concentrates tightly (sd 0.07, support 0.40–0.74) and sits entirely above the threshold. Because that

Fig. 7: Onboarding a fixer costs a résumé, not a training run. A short public outcome record averages into two centroids and a base rate, and the router reads that résumé at inference time. No component is retrained. support lies entirely above θ, part of the hidden-state head’s effect at the gate is mechanical: blending shrinks the text head’s predictions toward the state head’s near-constant mean, which by itself admits more tasks to the cheapest fixer. We did not run a matched shrinkage control, so the feature’s informational contribution cannot be fully separated from this thresholdshifting effect; the held-out separation gap (AUC 0.600 versus 0.561 for task text at N =99, Appendix F) is suggestive rather than decisive. The hidden state’s contribution is not boosting accuracy per se but shifting when the cheap model is trusted, consistent with its cost-side rather than accuracy-side value observed during calibration (§8). 7.3 Onboarding a New Fixer Adding a new fixer requires no retraining. A fixer’s résumé consists of two averaged embeddings (one over its solved tasks, one over its failed tasks) plus a base solve rate, all buildable from 25–50 public per-task outcomes (Table II). The embedder is frozen infrastructure and SuperScout-7B never retrains; the router simply reads each new résumé at inference time (Figure 7). This design keeps the pool extensible: a fixer released tomorrow can be routed to as soon as a short publicbenchmark run produces its outcome record. Appendix H bounds what such public outcomes can carry: on our fresh calibration tasks every fixer’s public rate drops and the public ordering does not survive. A résumé built from public outcomes is therefore a cold-start device, adequate for admitting a new fixer to the pool but not for fine-grained ranking, and the thresholds used in this paper were calibrated on freshly measured labels for exactly this reason.

8 Calibration on Fresh Tasks 8.1 The Label Run SuperScout’s router assigns each incoming task to a fixer on the basis of per-task résumés and cost thresholds (§4). Both mechanisms require outcome labels: which fixer solved which task, and at what price. Public leaderboard data supplied these labels for several languages but contained zero usable Python

8.2 The Handoff Redistributes Figure 8 shows the paired comparison; exact rates and intervals appear in Appendix G. Confidence intervals here and in the appendix are 95% percentile intervals from a bootstrap resampling tasks [38]. The handoff lifted the three weaker fixers: Claude Opus 4.6 rose from 48.5% to 53.5% (+5.1 pp, p = 0.125), Kimi K2.5 from 52.5% to 56.6% (+4.0 pp, p = 0.481), and Gemini 3 Flash from 55.6% to 57.6% (+2.0 pp, p = 0.774). The strongest fixer, GPT-5.2, moved in the opposite direction, dropping from 60.6% to 56.6% (−4.0 pp, p = 0.424). Pooled across all four, the effect is +1.8 percentage points with a 95% confidence interval of [−1.0, +4.5] and p = 0.401, not statistically significant. At N =99 every per-fixer test is underpowered; the confidence intervals, not the point estimates, carry the information. What the pattern does establish is that fixer rankings survive handoff conditioning, with only one adjacent flip falling within the confidence interval. That stability is what the router’s résumé assumption requires. The pattern is consistent with redistribution rather than addition: solving ability shifts toward the cheaper models, which is exactly the shape cost-aware routing needs, though no single contrast reaches significance. The redistribution pattern recurs on the benchmark’s router-selected subset, though that comparison is uncontrolled and serves only as corroboration; exact numbers appear in Appendix I.

Claude Opus 4.6

+5.1

Kimi K2.5

+4.0

Gemini 3 Flash

+2.0 −4.0

GPT-5.2 pooled

+1.8

45%

50%

55%

60%

65%

Solve rate (paired, N =99) solo

with handoff

Fig. 8: The handoff pattern is redistributive, not additive. Paired comparison (N =99 tasks, four fixers). Open circles: solo rates; filled markers: with-handoff. Horizontal bars span the 95% CI of each paired delta, anchored at the solo rate. The three weaker fixers gain; the strongest loses. All deltas are directional only (p > 0.05 at this sample size).

Held-out cost saving

outcomes for any fixer in the pool. Benchmark contamination policies lock those results away, and Appendix H documents why leaderboard-derived labels mislead when applied outside their original distribution. We therefore collected fresh labels: 100 Python bugs drawn from 2026 repositories, each attempted by all four fixers twice, once cold and once with SuperScout7B’s handoff injected (99 of the 100 yielded a valid paired comparison). This paired design let every task serve as its own control, isolating the handoff’s effect from task difficulty. The same 100 tasks exposed a problem with SuperScout-7B’s reproduction claims. Replaying those claims against ground truth revealed that 32% were genuine and 56% demonstrably false. The verify-then-strip gate described in §4 was built as a direct consequence, before any benchmark contact.

deployed router 34.3%

40% 30.5%

30% 20%

handofftext 9.4%

handofftext 8.0%

10% 0% A

B

C

D

task text

+handoff

+state

+both

Fig. 9: Which router features buy cost savings. Held-out cost saving against an always-best-model anchor, all variants pinned to the same solve rate (.606). Adding SuperScout-7B’s hidden states (variant C, the routing blend) lifts savings from 30.5% to 34.3%. Variants with handoff-text features (B, D) collapse to 8–9%. The C-vs-A gap is fold-fragile at N =99.

Supporting signals reinforce this split. As an outcome predictor, the hidden state reaches an AUC of 0.600 (fold-stable in four of five folds), while handoff text manages only 0.510. A simple logistic regression beat a multi-layer perceptron in 8.3 The Router Design Space every comparison, so the simpler scorer shipped. The full grid We compared four input-feature variants under an offline appears in Appendix F. The C-versus-A cost gap, a different simulation of the capped cost protocol, pinning each to the quantity from the AUC separation, holds in only three of five same matched solve rate so that the bars in Figure 9 differ only folds, and the gate-level caveat of §7 applies: at deployment in spend. Variant A, using task text alone, saves 30.5% on total the hidden state’s informational contribution cannot be fully cost per solve at a matched solve rate. Adding SuperScout-7B’s separated from its threshold-shifting effect. On the calibration hidden states produces variant C, the routing blend, raising the labels, the cost-aware router matched the strongest fixer’s solve saving to 34.3%. The two variants that incorporate handoff- rate at 29% lower total cost per solve ($0.49 versus $0.69). text features tell a sharply different story: variant B collapses Calibration settled four design choices: the routing blend to 8.0% and variant D to 9.4%. The lesson is concrete. The (variant C), the matched-point threshold (θ=0.30), logistic searcher’s internal state helps the router, while the handoff’s regression as the scoring function, and blanket handoff injection. text helps the fixers; feeding the handoff memo to the router It left one question open. GPT-5.2 was the strongest fixer on actively hurts. these fresh tasks, a ranking the benchmark later inverted (§7).

9 Limitations

shows that this recall suffices for effective routing; it does not establish SuperScout-7B as a state-of-the-art localizer. All absolute solve rates reported here are specific to the capped, silent-budget, auto-submit evaluation tier defined in §6. Under other tiers these numbers would change. Our protocol comparisons (§7) show that such tier choices are worth percentage points of solve rate, underscoring the relativity of any single absolute figure. Finally, the hidden-state features that drive routing are tied to the exact SuperScout-7B checkpoint. Retraining the searcher would require re-extracting hidden states and refitting the router heads. The zero-retrain onboarding property applies to adding new fixers, not to changing the searcher itself.

Our headline result rests on a single benchmark’s Python slice, comprising 266 tasks. While the searcher’s localization ability transfers across languages (§5), that evidence covers the searcher alone, not the full routed system. Cross-benchmark generality remains untested, and the ablation finding, that blanket assignment to the cheapest fixer ties the routed system, is a property of this task pool; it need not transfer to other pools or benchmarks. The 266 tasks themselves span three repositories, with per-repository recall ranging from 0.472 to 0.671 (Appendix D); the effective diversity is closer to three codebases than to 266 independent draws. At n=266 the headline is a one-task edge, which is precisely why we claim a match and nothing stronger. The robustness 10 Conclusion evidence is structural: both the routed system and the no-router SuperScout shows that engaging with the repository before ablation sit +3.60 percentage points above the blind-mixing dispatch pays, though not where we expected: the verified line, so on this pool the margin over random traffic splitting handoff, not the routing decision, carries the result. A small is carried by the handoff rather than by the routing decision. trained searcher, SuperScout-7B, explores the repository and Like the ablation finding itself, this attribution is specific to writes a structured handoff whose reproduction claims are this pool and benchmark and can change with either. verified in a sandbox (false claims are stripped), and whose The calibration label run uses n=100 tasks. No handoff hidden states feed a résumé router that selects the downstream contrast in this paper reaches statistical significance, on the fixer. On the benchmark’s full Python slice, this pipeline solves calibration labels or on the benchmark subset; the mechanism 159 of 266 problems, matching the best single frontier model rests on the consistency of the pattern across two independent (158 of 266) at about a fifth of the total cost per solve, with settings, not on any single test. the searcher’s own compute contributing a rounding error to The benchmark-side handoff comparisons themselves come the budget. from router-selected subsets rather than random assignment, Two mechanism-level findings underpin this result. The so they carry observational caveats. The controlled reference handoff appears to redistribute rather than add ability, lifting point is the paired calibration study on held-out tasks (§8). the three cheaper fixers while slightly hurting the strongest, Contamination control is one-sided. The searcher’s training a directional pattern at N =99. The searcher’s hidden states, corpus excludes a 23-repository blocklist covering every SWE- meanwhile, change which tasks the router trusts to the cheap bench Pro and SWE-bench Verified repository, enforced by a model, yet feeding the handoff’s text directly to the router programmatic gate on the frozen training file. No such control hurts it. Because adding a new fixer requires no retraining, the is possible on the fixer side: the three evaluation repositories system is built for a model market that changes quarter by are public, we have no visibility into the four fixers’ training quarter. data, and differential fixer-side contamination would distort the Natural next steps include wider benchmark coverage across relative solve rates that the résumés and the calibration are fit languages and problem domains. Equally valuable would be to. measuring how quickly a newly released fixer can be onboarded We also did not run a pass-through arm in which fixers into the live pipeline without retraining the searcher or the receive the handoff with its unverified reproduction claims router. intact, so the guard’s contribution to the headline solve rate is asserted from the claim census (§7) rather than measured; a pass-through ablation is future work. All costs are provider list prices recorded at measurement time. Prices drift, and one fixer was served as a quantized build through a pinned provider, so cost conclusions are snapshots rather than stable constants. The cost-accounting asymmetry described in §6 (all-in for the system versus API-only for solo baselines) is conservative in our disfavor, but it is still an accounting choice that readers should weigh. SuperScout-7B achieves an overall localization recall of 0.566, a moderate figure drawn from a single sampled inference pass whose variance is uncharacterized. The system-level result

References [1] C. E. Jimenez et al., “SWE-bench: Can language models resolve real-world GitHub issues?” in International Conference on Learning Representations (ICLR), 2024, arXiv:2310.06770. [2] I. Ong et al., “RouteLLM: Learning to route LLMs with preference data,” 2024, arXiv:2406.18665. [3] L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to use large language models while reducing cost and improving performance,” 2023, arXiv:2305.05176. [4] X. Deng et al., “SWE-bench Pro: Can AI agents solve long-horizon software engineering tasks?” 2025, arXiv:2509.16941. [5] J. Yang et al., “SWE-agent: Agent-computer interfaces enable automated software engineering,” in Advances in Neural Information Processing Systems (NeurIPS), 2024, arXiv:2405.15793. [6] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying LLM-based software engineering agents,” 2024, arXiv:2407.01489. [7] J. Yang et al., “SWE-smith: Scaling data for software engineering agents,” 2025, arXiv:2504.21798. [8] J. Pan et al., “Training software engineering agents and verifiers with SWE-Gym,” in International Conference on Machine Learning (ICML), 2025. [9] N. Jain, J. Singh, M. Shetty, L. Zheng, K. Sen, and I. Stoica, “R2E-Gym: Procedural environments and hybrid verifiers for scaling open-weights SWE agents,” in Conference on Language Modeling (COLM), 2025, arXiv:2504.07164. [10] C. Xie et al., “SWE-Fixer: Training open-source LLMs for effective and efficient GitHub issue resolution,” 2025, arXiv:2501.05040. [11] Z. Chen et al., “LocAgent: Graph-guided LLM agents for code localization,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics, 2025, pp. 8697–8727, arXiv:2503.09089. [12] R. G. Reddy et al., “SweRank: Software issue localization with code ranking,” 2025, arXiv:2505.07849. [13] D. Ding et al., “Hybrid LLM: Cost-efficient and quality-aware query routing,” in International Conference on Learning Representations (ICLR), 2024, arXiv:2404.14618. [14] T. Shnitzer et al., “Large language model routing with benchmark datasets,” 2023, arXiv:2309.15789. [15] R. Zhuang, T. Wu, Z. Wen, A. Li, J. Jiao, and K. Ramchandran, “EmbedLLM: Learning compact representations of large language models,” in International Conference on Learning Representations (ICLR), 2025, arXiv:2410.02223. [16] C. Wang et al., “ICL-router: In-context learned model

representations for LLM routing,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 39, 2026, pp. 33 413–33 421, arXiv:2510.09719. [17] S. Son, S. Yoon, J. Tang, S. Wang, L. Wolf, and I. Bogunovic, “SWE-Router: Routing in multi-turn agentic software engineering tasks,” 2026, presented at the 5th Deep Learning for Code Workshop, ICML 2026; arXiv:2607.00053. [18] Q. He et al., “CodeRescue: Budget-calibrated recovery routing for coding agents,” 2026, arXiv:2607.19338. [19] R. Raj, S. Kundu, S. Banerjee, D. Joshi, I. Vohra, and T. Krishna, “TRACE-Router: Task-consistent and adaptive online routing for agentic AI,” 2026, arXiv:2607.22465. [20] Y. Wei et al., “Toward training superintelligent software agents through self-play SWE-RL,” in International Conference on Machine Learning (ICML), 2026, arXiv:2512.18552. [21] OpenAI, “Introducing SWE-bench Verified,” https:// openai.com/index/introducing-swe-bench-verified/, 2024, accessed 2026-08-03. [22] J. Yang et al., “SWE-bench multilingual,” https://www. swebench.com/multilingual.html, 2025, dataset released alongside SWE-smith. [23] B. Hui et al., “Qwen2.5-Coder technical report,” 2024, arXiv:2409.12186. [24] Y. Zhang et al., “Qwen3 Embedding: Advancing text embedding and reranking through foundation models,” 2025, arXiv:2506.05176. [25] OpenAI, “gpt-oss-120b & gpt-oss-20b model card,” 2025, arXiv:2508.10925. [26] W. U. Ahmad, N. Ludwig, S. Majumdar, and B. Ginsburg, “Open-SWE-Traces: Advancing dual-mode multilingual distillation for software engineering agents,” 2026, arXiv:2606.16038; dataset: https://huggingface.co/ datasets/nvidia/Open-SWE-Traces. [27] M. Trofimova, A. Shevtsov, I. Badertdinov, K. Pyaev, S. Karasik, and A. Golubev, “OpenHands Trajectories with Qwen3-Coder-480B-A35B-Instruct,” Nebius blog / HuggingFace dataset, 2025, dataset: https://huggingface. co/datasets/nebius/SWE-rebench-openhands-trajectories. [28] I. Badertdinov et al., “SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025. [29] N. Ludwig, W. U. Ahmad, S. Majumdar, and B. Ginsburg, “From SWE-ZERO to SWE-HERO: Execution-free to execution-based fine-tuning for software engineering agents,” 2026, arXiv:2604.01496; dataset: https://huggingface.co/datasets/nvidia/ SWE-Hero-openhands-trajectories. [30] X. Wang et al., “OpenHands: An open platform for AI software developers as generalist agents,” in International Conference on Learning Representations (ICLR), 2025,

arXiv:2407.16741. [31] E. J. Hu et al., “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations (ICLR), 2022, arXiv:2106.09685. [32] Z. Shao et al., “DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,” 2024, arXiv:2402.03300. [33] Anthropic, “Introducing Claude Opus 4.6,” https://www. anthropic.com/news/claude-opus-4-6, 2026, accessed 2026-08-03. [34] OpenAI, “Introducing GPT-5.2,” https://openai.com/index/ introducing-gpt-5-2/, 2025, accessed 2026-08-03.

[35] Kimi Team, “Kimi K2.5: Visual agentic intelligence,” 2026, arXiv:2602.02276. [36] Google DeepMind, “Gemini 3 Flash model card,” https: //deepmind.google/models/model-cards/gemini-3-flash/, 2025, accessed 2026-08-03. [37] Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947. [38] B. Efron, “Bootstrap methods: Another look at the jackknife,” The Annals of Statistics, vol. 7, no. 1, pp. 1–26, 1979.

Appendix A The Prior Routing Audit

Appendix C Verify-then-Strip Outcome Classes

The cost-routing premise of §3 rests on a zero-cost replay of published per-task outcome matrices. Table VI records the structural statistics behind that replay: set containment, the unique-solver shell, and the gap between the best learned router and always picking the strongest model, across three benchmarks and matched four-model pools.

§7 reports the headline guard statistics; the full failure-class census behind those numbers appears in Table VIII, broken down by outcome category and by handoff type.

TABLE VI: Why we do not route for accuracy. Zero-cost audit of published per-task outcome matrices on matched four-model pools. Containment measures solve-set nesting; the unique-solver shell is the share of routable tasks only one model solves. On all three benchmarks, no learned router beat always calling the strongest model. This is background motivation, not a system result. Verified

Multiling.

Pro

Best single model Mean containment Unique-solver shell

0.768 0.941 21.5%

0.726 0.912 20.8%

0.449 0.773 30.8%

Best router − best model p

−0.006 0.66

+0.010 0.24

−0.004 0.90

Appendix B Router Threshold Sweep Table VII sweeps the routing threshold θ that the calibration’s matched-point rule selects. At θ=0.25 every task clears the cheapest fixer’s gate and the router degenerates into the no-router ablation: 159 solves at $0.227 per solve, identical to always calling Kimi K2.5. Raising the threshold to the operating point θ=0.30 diverts three tasks to Gemini 3 Flash at a marginal cost of $0.003 per solve, without changing the solve count. Above the operating point, the router’s growing caution pushes progressively more tasks onto expensive fixers: cost per task rises monotonically while the solve count ceases to be exactly measurable, because the diverted tasks lack a with-handoff outcome on their new fixer. We report those rows as bracketing intervals rather than point estimates (the table note details the convention). On this benchmark, routing collapses to a cost-allocation decision: no threshold examined buys additional solves beyond the 159 that the cheapest fixer already delivers. TABLE VII: The gate threshold sweep. Raising θ pushes tasks off the cheapest fixer onto more expensive ones: cost per task rises monotonically while the solve count ceases to be exactly measurable, since diverted tasks lack a with-handoff outcome on their new fixer. The operating point θ=0.30 routes all but three tasks to Kimi K2.5 and is the last threshold at which every number is exact. Routing split (tasks) θ

Kimi Gemini 3 Claude K2.5 Flash GPT-5.2 Opus 4.6

0.25 0.30† 0.35 0.40 0.50

266 263 236 203 124

0 3 19 34 34

0 0 10 28 72

Solves $/task $/solve

0 159 $0.136 $0.227 0 159 $0.137 $0.230 1 [149, 169] $0.154 $0.272∗ 1 [132, 179] $0.178 $0.337∗ 36 [94, 196] $0.334 $0.672∗

† Operating point: the threshold used for every SuperScout number in the

paper. ∗ $/solve from a solo-filled point estimate of the solve count, since tasks routed to a fixer with no measured with-handoff outcome take that fixer’s solo outcome; rows whose solve count is not measured exactly report the bracketing interval instead, and no midpoint. Rows above the operating point send tasks that pass no gate to the most capable fixer (Claude Opus 4.6); at the operating point every task passes at least one gate, so the fallthrough never fires. Solves are out of n=266. All rows use the board’s cost convention: measured fixer API spend plus a pinned $5.13 of searcher GPU and sandbox infrastructure, amortized over 266 tasks.

TABLE VIII: What the verify-then-strip sandbox found in 266 handoffs. Top: outcome class of each reproduction claim, replayed before any fixer saw it (50 genuine, 174 false and stripped). Bottom: split by handoff type; spontaneous handoffs are more than twice as likely to be genuine as forced ones (never pooled). Forced-handoff percentages use the 43 of 60 that claim a reproduction as their denominator. The label-run row shows the same measurement on an earlier, easier task set. n

Outcome of the replayed reproduction Passed at base (claim stripped) Genuinely failed (claim kept) Claim not true Import error Non-zero exit, other Missing file No reproduction block No command

174 50 16 12 9 3 1 1

Total

266

Handoff set

Claiming

Genuine

Stripped

Spontaneous (n=206) Forced (n=60)

206 43

22% 9%

68% 79%

All Pro handoffs (n=266) Label-run baseline

249 91

20% 32%

70% 56%

Appendix D Localization by Repository The per-task mean localization quality reported in Table V pools three repositories that differ substantially in how many files their gold patches touch. Table IX disaggregates by repository and provides the solved-versus-unsolved cross-tab behind the main-text numbers. TABLE IX: Localization by repository, and against outcome. Top: SuperScout-7B’s file-localization quality per repository; spread tracks the number of gold files per task. Bottom: quality split by whether the system solved the task. Better localization correlates weakly with solving (+0.024 recall), so localization is not a gate on the outcome. Repository

Recall

Prec.

All-gold

Gold files

ansible (n=96) openlibrary (n=91) qutebrowser (n=79)

0.472 0.574 0.671

0.846 0.766 0.853

10.4% 28.6% 38.0%

3.80 3.58 2.84

All (n=266)

0.566

0.821

24.8%

3.44

Outcome

Recall

Prec.

All-gold

Solved (n=159) Unsolved (n=107)

0.575 0.552

0.830 0.808

26.4% 22.4%

Appendix E Capped-Tier Statistics The protocol findings in §7 quote selected cap-binding rates and auto-submit rescue counts. Table X gives the complete picture for all three fixer arms.

TABLE X: How often the evaluation caps bound, and what auto-submission was worth. All arms ran the official capped tier (50 turns, $2.00 per attempt). The turn cap is the binding constraint; the cost cap bound only five times. Auto-submission rescued 24 of Claude Opus 4.6’s solves and 33 of GPT-5.2’s; without it, both anchors’ rates would fall roughly 9–13 points. The gold-patch control passes on 265 of 266 tasks.

Hit 50-turn cap Hit $2 cost cap Max attempt cost Auto-submitted at cap Solves rescued Solves Rate without rescue

GPT-5.2

Claude Opus 4.6

Kimi K2.5

36.5% 0 $1.59 94 33 139 39.8%

17.3% 5 $2.09 48 24 158 50.4%

48% 0 — 126 — 149 —

The calibration analysis in §8 summarizes the four feature variants under the logistic-regression scorer. Table XI extends that comparison to include the MLP head and provides the outcome-separation (AUC) measurements the cost numbers rest on. TABLE XI: The complete router design space. Top: held-out cost saving for every feature-set × scorer combination, all pinned to solve rate .606. Adding hidden states helps under both scorers; handoff-text features collapse savings to 8–9%. The MLP halves every variant’s saving. At N =99 the best variant is only 3/5 fold-stable. Bottom: the hidden state separates solved from failed tasks at AUC .600, while handoff-text embedding sits at chance. Saving

A text only B + handoff text C + hidden state (dep.) D + both

LR

MLP

30.5% 8.0% 34.3% 9.4%

15.4% 5.3% 18.7% 4.2%

Routing representation

Folds

Brier

3/5 1/5 3/5 4/5

0.270 0.254 0.249 0.251

Pooled AUC

Stable

0.600 0.561 0.510

4/5 — —

Hidden state (pre-decode, layer −4) Task text Handoff text (prereg. baseline)

The router’s résumés are built from public per-task outcomes, so the router inherits whatever biases those outcomes carry. Buying fresh labels was not a design preference but a measured necessity. Table XIII records the calibration receipts that motivated that decision: every fixer’s public rate compared with its freshly measured rate under a single controlled harness. TABLE XIII: Why we bought fresh labels: public rates do not transfer to unseen tasks. Each fixer’s public multilingual rate versus its freshly measured rate on 100 post-cutoff Python tasks under one harness. Every fixer drops; the ordering does not survive. Rates use the full 100-task solo denominator; the paired analysis in Appendix G uses the 99 tasks with both arms. These are calibration receipts, not a benchmark claim.

Appendix F Router Design Space, Full Grid

Router features

Appendix H Benchmark-Versus-Fresh Calibration Receipts

Fixer

Public

Fresh (95% CI)

Gap

Rank

Gemini 3 Flash Claude Opus 4.6 Kimi K2.5 GPT-5.2

0.727 0.720 0.673 0.667

0.550 [.450, .650] 0.480 [.380, .570] 0.520 [.420, .610] 0.610 [.510, .700]

−17.7 −24.0 −15.3 −5.7

#1 → #2 #1 → #4 #2 → #3 #3 → #1

Appendix I Handoff Effect on Router-Selected Benchmark Subset The redistribution pattern observed under controlled conditions (§8.2) also appears in the benchmark evaluation, where the tasks the router sends to Kimi K2.5 were available both with and without a handoff. Table XIV reports that comparison. The subset was selected by the router rather than randomly assigned; it remains measured, uncontrolled corroboration rather than a controlled experiment. TABLE XIV: The handoff effect on the deployment path. The tasks routed to Kimi K2.5 were also run without a handoff, giving a same-task comparison: the workhorse gains 3.8 pp on 263 tasks. b/c counts tasks solved only with the handoff versus only solo; the exact McNemar test on those discordant pairs gives p = 0.245. The router selected which tasks enter the subset, not the within-task contrast, so the pairing is intact, but the subset is not randomly assigned and this table remains corroboration, not a controlled measurement. The three tasks routed elsewhere have no solo arm for their fixer and are omitted.

Appendix G Redistribution, Exact Numbers

Fixer Kimi K2.5

Figure 8 in the main text visualizes the paired handoff ablation; Table XII gives the exact per-fixer rates, deltas, confidence intervals, discordant-pair counts, and p-values behind that figure. TABLE XII: The handoff pattern is redistributive, not additive. Exact per-fixer rates, deltas, and p-values behind Figure 8 (99 paired tasks, 396 attempts). The three weaker fixers gain and the strongest loses; pooled effect is +1.8 pp with CI including zero. No fixer reaches p < 0.05; every row is directional only. b counts rescued tasks, c counts broken ones. Fixer

Solo

Handoff

∆ (95% CI)

b/c

p

Claude Opus 4.6 Kimi K2.5 Gemini 3 Flash GPT-5.2

48.5 52.5 55.6 60.6

53.5 56.6 57.6 56.6

+5.1 [0.0, 10.1] +4.0 [−4.0, 12.1] +2.0 [−5.1, 9.1] −4.0 [−11.1, 3.0]

6/1 11/7 7/5 5/9

0.125 0.481 0.774 0.424

pooled

54.3

56.1

+1.8 [−1.0, 4.5]

29/22

0.401

Routed n

Handoff

Solo

∆

b/c

p

263

60.1%

56.3%

+3.8

35/25

0.245

Appendix J Nine-Language Localization Detail Figure 5 in the main text shows the per-language F1 bars; Table XV gives the full precision, recall, F1 , and all-gold counts behind that figure, split by trained and never-trained language pools.

TABLE XV: Localization transfers to languages SuperScout-7B never saw in training. Per-language precision, recall, and F1 of spontaneous handoffs against gold patch files. The six never-trained languages outperform the three trained ones (F1 0.630 vs. 0.455). JavaScript and TypeScript are genuine soft spots; the TypeScript and C++ cells rest on 12 assigned tasks each, with 7 and 8 spontaneous handoffs analyzed (†). Multi-file recall is weak across all languages (0.26–0.34). Forced handoffs excluded; searcher alone, one sampled draw. Language

n

P

R

F1

All-gold

Go JavaScript TypeScript†

27 26 7

0.605 0.500 0.333

0.542 0.365 0.357

0.546 0.389 0.343

13/27 7/26 2/7

trained pool

60

0.528

0.444

0.455

22/60

Java Ruby Rust PHP C C++†

27 36 20 33 18 8

0.722 0.833 0.800 0.697 0.546 0.688

0.574 0.661 0.626 0.632 0.435 0.813

0.612 0.701 0.667 0.630 0.435 0.708

11/27 18/36 10/20 17/33 6/18 6/8

never-trained

142

0.731

0.613

0.630

68/142

Appendix K Vault Decoding Comparison The decoding finding reported in §5.2 is demonstrated on a 100task dial subset; Table XVI gives the full A/B comparison on the 450-task held-out vault that established the finding. TABLE XVI: Sampled decoding is what makes the searcher commit. The same checkpoint on 450 held-out issues, greedy versus temperature 0.9. Find rate nearly triples, from higher commitment (72% vs. 21%) against an 18% drop in per-handoff recall. The greedy arm lost 24 episodes to infrastructure timeouts; the 2.65× figure adjusts for this. The vault is excluded from all training data. Metric (n=450) Find rate 95% CI timeout-matched ratio

Sampled

Greedy

0.306 0.110 [.273, .337] — 2.65× (n=426)

Emission rate Handoffs emitted

0.718 323

0.213 96

Recall per handoff Precision Files per handoff All-gold handoffs

0.426 0.643 1.93 71

0.517 0.726 2.05 27

0

0

Malformed

Appendix L Training Hyperparameters §5 describes the supervised recipe at the level relevant to the contribution; Table XVII records the full configuration as realized in the actual training run.

TABLE XVII: Supervised fine-tuning configuration, as run. Rank-64 LoRA on every linear projection, trained in bf16 for two epochs on packed 32ktoken blocks at 91.7% fill; loss is on assistant turns only. All values are the realized run, not the planned recipe. The complete run cost $86 of GPU time and peaked at 24.2 GB. Setting

Value

Base model Adaptation Target modules Dropout Precision

Qwen2.5-Coder-7B-Instruct LoRA, r=64, α=128 all linear projections (no embedding, no output head) 0 bf16

Learning rate Warmup Epochs Context Micro-batch Gradient accum. Gradient ckpt. Seed

1×10−4 , cosine to 10% 3% of steps 2 (3,182 steps) 32k, packed blocks 1×32k 8 on 3407

Tokens per epoch Wall clock Peak memory Final loss Cost

382.4M (91.7% fill) 31.8 h 24.2 GB 0.114 $86

a) Reinforcement-learning rig. We built and validated a GRPO training rig before concluding that RL was unnecessary (§5). The rig used a two-GPU topology: one GPU trained while the other generated rollouts, with kill-andresume checkpointing so that preempted runs lost no more than a single episode. Exact backpropagation through long episodes was verified against ground truth before any paid run began. The reward function was validated separately, by replaying candidate reward assignments against thousands of previously saved episodes and confirming agreement before committing GPU spend. 50 clean steps then ran flat, as reported in §5, and the rig was shelved in favor of the decoding fix.

Appendix M A Verified Handoff, Before and After the Gate Figure 10 reproduces one real spontaneous handoff from the SWEbench Pro evaluation, shown in the exact post-strip form the fixer received. This handoff’s reproduction claim was replayed in the task’s own sandbox and genuinely failed, so the claim was forwarded intact rather than deleted.

<files> qutebrowser/utils/utils.py qutebrowser/utils/qtutils.py qutebrowser/browser/downloads.py qutebrowser/mainwindow/tabbedbrowser.py </files> <repro> command: cd /app && python test_issue.py reproduced: true observed: OBSERVATION: SUCCESS: interpolate_color exists in utils Interpolation at 50%: 128:0:128 Traceback (most recent call last): File "/app/test_issue.py", line 26, in <module> interp = qtutils.interpolate_color AttributeError: module ’qutebrowser.utils.qtutils’ has no attribute ’interpolate_color’ </repro> <notes> Search surfaced definitions of interpolate_color in qutebrowser/utils/utils.py (line 260) and references to utils.interpolate_color in qutebrowser/browser/downloads.py (line 563) and qutebrowser/mainwindow/tabbedbrowser.py (lines 866, 883). No occurrences of interpolate_color were found in qutebrowser/utils/qtutils.py. Grep for _get_color_percentage returned nothing. Attempted import of qutebrowser.utils.qtutils showed only top-level symbols, confirming the function is absent there. Running a custom script test_issue.py via cd /app && python test_issue.py raised AttributeError: module ’qutebrowser.utils.qtutils’ has no attribute ’interpolate_color’. No build or test tooling issues observed; commands executed directly with python. No false leads beyond missing function in qtutils. </notes> STOP

Fig. 10: One real handoff, exactly as the fixer received it. A spontaneous handoff from the SWE-bench Pro evaluation, reproduced verbatim after the verify-then-strip stage. This reproduction claim was genuine and forwarded intact; 174 of 249 claims were false and stripped. Line wrapping in the notes is ours; no content is added or redacted.

Appendix N Compute and Cost Disclosure Table XVIII itemizes the measured spend across the three eras of this project: evaluation, label collection, and training. The one-off cost of building the system, comprising the label run (≈$370), the fine-tuning run (≈$86), and corpus note generation (≈$9), totals roughly $465. Against the $0.62 per-task saving over the best solo fixer ($0.757 API-only versus $0.137 all-in), that sunk cost amortizes in roughly 750 tasks. TABLE XVIII: Measured spend. Every figure is a measured value from the run ledgers. The evaluation era covers all five arms plus infrastructure; the label-run era covers fresh per-task outcomes; the training era is the single SFT run. SuperScout-7B’s entire evaluation contribution was $1.13 of GPU time (∼0.4 cents per task). Era totals are not summed. Item Evaluation era Probe (40 attempts) GPT-5.2 solo arm Claude Opus 4.6 solo arm Kimi K2.5 solo arm System arm Smoke Unrecorded partials GPU (all pods) Sandbox compute era total Label-run era Fixer API GPU pod Sandbox compute era total Training era Fine-tuning run Corpus note generation

$ 19.24 151.64 201.25 28.23 86.49 1.32 ≈11 ≈5.9 40–55 ≈545–560 340.03 ≈4 ≈25 ≈370 ≈86 ≈9

Related documents

• Probe-and-Refine Tuning of Repository Guidance for Coding Agents#290565
• SWE-Explore: Benchmarking How Coding Agents Explore Repositories#266239
• SWE-Shepherd: Advancing PRMs for Reinforcing Code Agents#10441
• CURATE: Leveraging LLM Agents to Compose, Catalog, and Deploy Reproducible Workflows#1131633
• LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems#1132344