PREPRINT, JULY 2026
1
Resample or Reroute? Budget-Aware Test-Time Model Selection for Large Language Models
arXiv:2607.08665v1 [cs.LG] 9 Jul 2026
Teng-Ruei Chen
Abstract—Routing among large language models (LLMs) trades response quality against serving cost, motivated by the reported gap between deployed routers and a per-instance oracle. Recent analysis shows that test-time resampling can recover per-instance selection headroom that no single-commit router captures; however, that guarantee holds only under an idealized oracle equipped with correctness labels and an unconstrained budget, neither of which a deployed system has. To the best of our knowledge, no previous work treats resampling the committed model and rerouting to an alternative model as competing uses of a single per-query cost budget. Therefore, this work formulates budget-aware test-time model selection: given a per-query budget and an imperfect verifier, allocate each unit of budget between resampling and rerouting so that expected correctness is maximized. An online resample-or-reroute (RoR) allocation policy driven by estimated marginal correctness per unit cost is proposed, and its behavior is grounded in the recoverability asymmetry between selection and sampling. Replay experiments on newly regenerated multi-draw correctness tensors from an eleven-model open-weight pool over four benchmarks of differing difficulty show that the proposed RoR policy attains a favorable cost–quality Pareto front relative to single-route, one-commitrouter, budget-aware best-of-K, cascade, and random-allocation baselines for the tested pools, with the largest gains on the most heterogeneous benchmark; an ablation further shows the gains are verifier-gated, shrinking as verifier quality degrades, and robustness replays under a provider price vector and a label-free agreement verifier delineate where the conclusions carry over. Index Terms—Best-of-K sampling, cost-aware inference, inference efficiency, large language models, model routing, model selection, test-time compute.
I. I NTRODUCTION ERVING a query with large language models (LLMs) increasingly means choosing how to spend inference compute, not just which model to call: inference-time cost and energy are first-class deployment constraints [1], costand QoS-aware resource management for ML services is an established concern of serving infrastructures [2], and the space of candidate models keeps broadening [3]. Model routing promises to cut cost by sending each query to the cheapest model that can answer it, motivated by the large reported gap between a deployed router and a per-instance oracle that, in hindsight, always picks a correct model [4]– [7]. Two facts complicate this picture. First, under stochastic decoding the per-instance oracle is not a reproducible property: it is built from single draws, so part of the reported gap is single-draw label noise that no single-commit router
S
T.-R. Chen is with the Institute of Bioinformatics and Systems Biology, National Yang Ming Chiao Tung University, Hsinchu 300, Taiwan, and also with Krixvon, Taipei 100, Taiwan (e-mail: [email protected]).
can capture, while the rest is genuine, recoverable specialist advantage [8]. Second, that recoverable component can also be reached without any router at all: test-time resampling (best-of-K on one committed model) provably recovers the selection floor at the oracle’s own budget [8]. The catch is that this guarantee assumes access to correctness labels (a perfect verifier) and spends the oracle’s budget; a real serving system has an imperfect verifier and a fixed cost budget per query. This work addresses the operational question these results leave open: given a per-query cost budget and an imperfect verifier, should a system resample the model it already committed to, or reroute to a different (possibly more expensive) model? The question is cast as a budgeted correctnessmaximization problem, and an online policy is proposed that allocates each unit of budget to whichever action—one more sample of a candidate, or the first sample of a new candidate— has the highest estimated marginal correctness per unit cost. The main contributions of this work are summarized as follows. To the best of our knowledge, this is the first work to treat resampling the committed model and rerouting to an alternative as competing uses of a single per-query budget. This budget-aware test-time model selection problem is formalized as maximizing expected correctness under a per-query cost budget with an imperfect verifier (Section III). • An online resample-or-reroute (RoR) allocation policy (a marginal-gain greedy rule and a UCB variant) is proposed, together with an oracle allocation that serves as a reference ceiling (Section IV). • The policy’s behavior is connected to the recoverability asymmetry between selection and sampling established in the companion analysis [8] (Section V). • A system-level replay evaluation on four regenerated open-model pools reports the cost–quality Pareto front against single-route, one-commit-router, budget-aware best-of-K, cascade, and random-allocation baselines, together with a verifier-quality ablation (Sections VI–VII). •
Relation to the companion paper. This work is a methodoriented follow-up to the theoretical study of the router-tooracle gap [8]; the differences are threefold: 1) for the type of contribution, that work proves a decomposition and a recoverability asymmetry, whereas this work builds and evaluates a deployable policy; 2) for the information assumptions, that work grants the recovery procedure oracle correctness labels at the oracle’s own budget, whereas this work assumes only an imperfect verifier under a fixed per-query budget; and 3) for
PREPRINT, JULY 2026
the role of routing, that work adds no router by design, whereas this work actively decides between resampling and rerouting. The rest of this paper is organized as follows. Section II reviews related work. Section III formulates the problem, and Section IV presents the resample-or-reroute policy. Section V grounds the policy theoretically. Sections VI and VII report the experimental setup and results, Section VIII discusses limitations, and Section IX concludes this work. II. R ELATED W ORK Cost-aware routing and cascades. FrugalGPT [6] and model cascades escalate from cheaper to more expensive models under a confidence gate; learned routers such as RouteLLM [7] predict, per query, a single model to commit to; and Sheokand et al. [9] escalate unresolved reasoning traces from a base model to a brokered layer of expert LLMs, a reflection-style variant of cascading. Budgeted cascades predate LLMs: Xu et al. [10] learn classifier cascades that trade test-time evaluation cost against accuracy, and the rejectoption literature [11], [12] formalizes paying a fixed cost to abstain rather than accept a likely-wrong prediction — the primitive behind escalation. The same accept-or-escalate structure predates LLMs in two adjacent literatures: QoSaware service selection chooses, per request, which functionally equivalent service to invoke under quality and cost constraints [13], and adaptive edge offloading uses a calibrated early-exit confidence to decide whether to accept a cheap local prediction or escalate to a larger remote model [14], a decision studied broadly in the computation-offloading literature [15], [16]. These approaches optimize which model to call but treat each call as a single draw. The proposed policy adds an orthogonal axis: spending budget on repeated draws of the committed model, and choosing between that and rerouting. Test-time sampling. Best-of-N , self-consistency [17], and test-time compute scaling [18] improve quality by drawing multiple samples and selecting among them; Farr et al. [19] aggregate ensemble disagreement into uncertainty estimates that flag unreliable LLM outputs, a verifier-style signal of the kind the proposed policy consumes. These works fix the model and scale samples; this work integrates sample scaling and model choice into one budgeted decision, and uses the recoverability asymmetry [8] to explain when resampling beats rerouting. Routing benchmarks. RouterBench [4], RouterEval [20], and LLMRouterBench [5] provide per-(query, model) correctness matrices we build on; we reuse and extend multi-draw regenerations of these pools. Reliability pitfalls of singlerun evaluation are increasingly documented in the evaluation literature [21], including large score variance across prompt variants [22] — the same concern the multi-draw protocol addresses on the decoding axis. Efficient LLM inference. An orthogonal family of levers reduces the cost of each call rather than the allocation of calls: quantization, compression, and related techniques are surveyed in [23], and inference-time energy profiling guides deployment choices [1]. A second orthogonal family operates at the serving-system layer: SLO-aware scheduling and provisioning of inference workloads [24], [25] decides how to
2
run the chosen model, whereas the proposed policy decides which model to sample next. Both families compose with the proposed policy, which reallocates a fixed per-query budget across models and draws. Table I positions this work against the related families: prior approaches either choose among models or spend repeated draws, but none treats the two as competing uses of one explicit per-query budget under an imperfect verifier. Best-arm identification. Allocating draws to candidates connects to pure-exploration bandits [26], [27]; this work borrows their marginal-value intuition but targets a budgeted correctness objective rather than identifying a single best arm. III. P ROBLEM F ORMULATION Consider a query i, a pool of models M = {1, . . . , M } with per-draw costs c1 , . . . , cM , and a per-query cost budget B. Drawing a sample from model m yields a candidate answer that is correct with (unknown) probability pim . A (possibly imperfect) verifier v scores candidates and is used both to select a final answer and to guide allocation; v is not assumed to reveal ground-truth correctness. Table II summarizes the notation. A policy π observes the history of drawn candidates and their verifier scores and, at each step, chooses an action a ∈ {RESAMPLE(m), REROUTE(m′ )} (one more draw from an already-used model m, or the first draw from a new model m′ ) until the spent budget reaches B; it then returns a final answer â. The objective is P max E 1{â correct} s.t. cost ≤ B. (1) π
We report performance as a cost–quality curve: expected correctness as a function of the average budget B, and compare policies by Pareto dominance. Note that the constraint binds per query, not in expectation over queries; with a verifier that stops a query early, the spent cost can therefore be far below B, which is exactly the effect the cost axis of the Pareto curves measures. IV. M ETHOD : R ESAMPLE - OR -R EROUTE The proposed resample-or-reroute (RoR) policy maintains, for each candidate model, a running estimate of its success probability on the current query and its per-draw cost, and repeatedly spends the next unit of budget where the estimated return per unit cost is highest. This section presents the belief update, the allocation rule, the two variants, and the computational cost. A. Per-Query Belief Update Let p̄m denote model m’s offline accuracy, calibrated once on the train split. During a query i, after nim draws of model m of which wim were verified correct, the policy scores m by the posterior mean s p̄m + wim , (2) p̂im = s + nim where the pseudo-count s controls how strongly the offline prior anchors the estimate. The choice of s implements the
PREPRINT, JULY 2026
3
TABLE I P OSITIONING AGAINST RELATED APPROACH FAMILIES .
Approach family
Chooses among models
Repeated draws
Per-query budget
Imperfect verifier
Learned routers [7] Cascades [6], [10] Best-of-N / self-consistency [17], [18] LLM ensembles [19] Reject option [11], [12] RoR (this work)
✓ ✓(sequential) – ✓(parallel) abstain/accept ✓
– – ✓ – – ✓
– cost gate fixed N – reject cost ✓
– confidence gate selector needed UQ signal confidence ✓(parametric)
TABLE II M AIN NOTATION . Symbol
Meaning
i M, M cm B pim k v, q p̄m nim , wim p̂im s â
query index model pool and its size cost of one draw from model m per-query cost budget probability that one draw of m answers i correctly recorded draws per (query, model) cell in the replay verifier and its quality parameter offline (train-split) accuracy of model m draws of m spent on query i, and how many verified correct posterior mean of pim during a query prior pseudo-count in the belief update final answer returned by the policy
resample-or-reroute trade-off: with a small s, one or two failed draws pull p̂im below a fresh model’s p̄m′ and thereby trigger a reroute, whereas a run of successes keeps the policy resampling the committed model. For a model not yet tried on this query, nim = 0 and p̂im = p̄m , so rerouting competes on the offline prior alone. Section VII shows the policy is insensitive to s over an order of magnitude. B. Marginal-Gain Allocation Rule
a
\ ∆corr(a) , cost(a)
D. Computational Cost Each step scores at most M actions with O(1) arithmetic per action, and a query takes at most ⌈B/cmin ⌉ steps, so the perquery overhead is O(M B/cmin ) time and O(M ) memory — negligible relative to a single LLM call. The policy is stateless across queries except for the M offline priors p̄m . V. T HEORETICAL G ROUNDING
At each step the policy takes the action with the highest estimated marginal correctness per unit cost: a⋆ = arg max
Algorithm 1 Resample-or-Reroute (greedy) 1: Input: pool M, costs cm , budget B, verifier v 2: spent ← 0; draw one sample from the cheapest model; init p̂ 3: while spent < B do 4: for each action a (resample used m, or reroute to new m′ ) do \ and cost 5: estimate marginal gain ∆corr(a) 6: end for \ 7: take a⋆ = arg maxa ∆corr(a)/cost(a) 8: update p̂; spent ← spent + cost(a⋆ ) 9: end while 10: return answer maximizing verifier score across all drawn candidates
(3)
\ estimates the increase in the probability that the where ∆corr finally selected answer is correct; under early stopping this is the probability p̂im that the next draw of m succeeds, so the rule reduces to a⋆ = arg maxm affordable p̂im /cm . Resampling model m increases the chance that a verifier-selected answer from m is correct (a best-of-K-style lift); rerouting to m′ opens a new, possibly higher-p candidate at a different cost. Both actions are scored on the same scale, which is what makes them competing uses of one budget rather than separate mechanisms. C. Variants and Reference Ceiling We consider a greedy pvariant and a UCB variant that adds an exploration bonus 2 ln(t + 1)/(nim + 1)/cm at step t to under-sampled models; an oracle allocation (with access to true pim ) provides a non-deployable upper bound. Algorithm 1 summarizes the greedy variant.
The recoverability asymmetry [8] states that the per-instance selection floor is closed by no single-commit router but is recovered by resampling at the oracle’s budget. The proposed policy inherits two consequences. First, when a committed model’s reproducible success probability is high, resampling has larger marginal correctness-per-cost than rerouting, so the greedy rule prefers resampling—matching the asymmetry. Second, with an imperfect verifier the reachable correctness interpolates between the verifier-free aggregate and the labeled oracle bound (Orepro ≤ Oagg ≤ Oexp in the companion paper’s notation); we characterize this dependence empirically in Section VII. VI. E XPERIMENTAL S ETUP Data. We evaluate on newly regenerated multi-draw correctness tensors built with the companion protocol [8]: for each of four benchmarks — GSM8K (arithmetic, near-saturated), MATH-500 (competition math, intermediate), GPQA-Diamond (graduate science, hard and heterogeneous), and HumanEval+ (code generation, execution-scored) — every (query, model) cell holds k=30 seed-aligned draws at T =0.2 (top-p 1.0) from a pool of 11 open-weight models spanning
PREPRINT, JULY 2026
4
TABLE III E XPERIMENTAL SETTINGS . Setting
Value
Benchmarks (queries) Model pool Draws per cell Decoding Train/test split Draw orderings Budget grid Prior pseudo-count Implementation
GSM8K (500), MATH-500 (500), GPQA-Diamond (198), HumanEval+ (164) M = 11 open-weight models, eight lineages k = 30, seed-aligned T = 0.2, top-p 1.0 50/50 (priors on train, results on test) 20 independent permutations per policy 12 points in [cmin , 6 cmax ] s = 2 (sensitivity in Table VI) Python/NumPy replay, CPU only
eight pretraining lineages (Mistral; Qwen 2.5 at 7B/14B/32B plus a Qwen-based DeepSeek-R1 distill; Phi-4; OLMo-2; Yi-1.5; Granite-3.3; Gemma-2; Llama-3.1). The first three benchmarks are exact-match scored; on HumanEval+ a draw is correct iff it passes the full EvalPlus test suite (per-draw pass@1). Sizes: 500/500/198/164 queries. Protocol. Policies are replayed offline on the recorded draws: a “sample from model m” consumes one of its k pre-computed draws (without replacement, order randomized per trial). Queries are split 50/50; per-model priors and the best single model are calibrated on the train half; we report test-half accuracy averaged over 20 random draw orderings, sweeping the per-query budget B. The replay is implemented in Python/NumPy and runs on CPU only, so every reported number is reproducible from the released tensors without model inference. Table III lists the experimental settings; parameters follow the companion protocol [8], and the budget grid spans from one draw of the cheapest model to six draws of the most expensive one. Generation environment. The correctness tensors were produced by serving the eleven-model pool under vLLM at T =0.2 on the hardware and NVIDIA CUDA software in Table IV — the vLLM/PyTorch build rests on NVIDIA’s CUDA math and communication libraries (cuBLAS, cuDNN, NCCL) alongside CUTLASS and FlashInfer attention kernels; following the companion protocol’s small-disk design, one model is resident at a time and its weights are evicted before the next is loaded. This concerns only how the tensors were generated: the allocation replay analyzed in this paper is CPU-only (Table III) and reproduces every reported number from the released tensors without any model inference or accelerator. Cost. Per-draw cost is proxied by the model’s parameter count in billions — a monotone stand-in for $-per-token serving cost. Replacing it with a measured price/latency vector is a deployment-specific calibration that does not change the mechanism (Section VIII). Verifier. The main experiment assumes a reliable verifier: a policy stops as soon as a drawn answer is verified correct (early stopping), which is the setting where the recoverability asymmetry is fully available. Because raw language-model confidence is known to be poorly calibrated on question answering [28], a deployed verifier is imperfect; we therefore degrade the verifier parametrically: with quality q, final selection succeeds with probability q · 1{any drawn sample correct} + (1 − q) · (fraction of drawn samples correct), interpolating be-
TABLE IV G ENERATION ENVIRONMENT ( TENSOR PRODUCTION ONLY; THE ALLOCATION REPLAY OF THIS PAPER IS CPU- ONLY ). VALUES ARE THE AUDITED RUNTIME RECORDED BY AN AUTOMATIC ENVIRONMENT PROBE . Component
Value
CPU RAM GPU OS NVIDIA driver CUDA toolkit / runtime NVIDIA math libraries NVIDIA communication CUDA attention kernels Framework Serving
AMD EPYC 7J13, 24 vCPUs 64 GB 2× NVIDIA GeForce RTX 4090, 24 GB each (Ada Lovelace, cc 8.9) Ubuntu 24.04 LTS (kernel 6.8) 580.159.03 13.0 (NVCC 13.2, NVRTC 13.0) cuBLAS 13.1, cuDNN 9.19, cuSPARSELt 0.8 NCCL 2.28.9, NVSHMEM 3.4.5 CUTLASS 4.5, FlashInfer 0.6.12 (vLLM backend) PyTorch 2.11.0, vLLM 0.23.0, Transformers 5.12.1 one model resident at a time (weights evicted between models)
Also installed as CUDA 13 build dependencies (pulled via pip by the PyTorch/vLLM build, beyond those above): cuFFT, cuRAND, cuSOLVER, cuSPARSE, cuPTI, nvJitLink, NVVM, NVTX, cuFile, and nvidia-ml-py; the complete environment probe ships with the released code.
tween a perfect verifier (q=1) and a random pick over drawn samples (q=0), with no early stopping for q < 1. Baselines. (i) single-route: one draw of the most accurate model; (ii) router (one commit): a learned per-query router’s committed model, one draw; (iii) budget-aware bestof-K: the strongest single-model baseline — it picks, per budget, the model maximizing expected best-of-K accuracy and only resamples it; (iv) FrugalGPT-style cascade: escalate cheap→expensive, one draw each; (v) random allocation; and (vi) oracle allocation, a non-deployable ceiling that routes each query to the cheapest model with a correct draw. VII. R ESULTS Regimes across benchmarks. From Table V and Fig. 1, the same mechanism pays off differently by regime. On saturated GSM8K every budget-using policy approaches the ceiling (cascade and random reach ≥0.992; best-of-K plateaus at 0.983), so the margin is mostly cost: the proposed policy reaches 0.993 at mean cost 9.2, i.e. 24–34% cheaper than the cascade (12.1) and budget-aware best-of-K (13.9), and 3.5× cheaper than single-routing the best model (9.2 vs 32.0) while also being 2.7 points more accurate (0.993 vs 0.966). This is reasonable because on a near-saturated benchmark almost every model can eventually produce a correct draw, so the advantage comes from stopping early on cheap models rather than from finding a specialist. On intermediate MATH500 the policy is 2.1 points above the strongest baseline at matched cost (0.887 vs 0.867) and 10 points above the onecommit router at 18% lower cost. On hard, heterogeneous GPQA-Diamond — where the pool’s specialists genuinely differ — rerouting matters most: 0.968 vs 0.861 for best-of-K at matched cost (+10.7 points), +4.1 over the cascade, and roughly +40 points over either single-commit policy. Note one exception: at the very lowest GPQA budgets (∼8, below one draw of most strong models) the single-commit points (0.55– 0.57 at cost 8) beat the policy’s cheapest-first exploration (0.35 at cost 7); the policy overtakes them once the budget affords a second draw (∼11). The overall ordering matches the companion analysis: the more heterogeneous the pool, the more of the gap is recoverable by moving budget between models rather than only resampling one.
PREPRINT, JULY 2026
5
TABLE V ACCURACY AT A MATCHED MID BUDGET ( POINT NEAREST MEAN COST 26; COST = PARAMETER - SIZE PROXY ). S INGLE - COMMIT BASELINES ARE BUDGET- INDEPENDENT AND SHOWN AT THEIR FIXED OPERATING POINT. O RACLE ALLOCATION IS A CEILING , NOT A DEPLOYABLE POLICY.
Policy
GSM8K cost acc
MATH-500 cost acc
GPQA cost acc
HumanEval+ cost acc
Resample-or-Reroute (ours) ours (UCB variant) budget-aware best-of-K cascade (FrugalGPT-style) random allocation
9.2 9.3 13.9 12.1 14.3
0.993 0.993 0.983 0.992 0.992
26.4 26.2 25.5 25.8 26.6
0.887 0.876 0.867 0.847 0.846
25.9 26.2 27.0 26.2 24.3
0.968 0.934 0.861 0.926 0.706
20.5 22.6 26.5 22.8 25.8
0.952 0.954 0.852 0.952 0.959
router (one commit) single-route (best model) oracle allocation (ceiling)
32.0 32.0 7.0
0.976 0.966 1.000
32.0 32.0 6.7
0.784 0.776 0.944
8.0 8.0 7.0
0.566 0.551 1.000
32.0 32.0 7.1
0.817 0.858 0.988
On code (HumanEval+) the picture resembles saturated GSM8K: at matched mid budget RoR reaches 0.952, against 0.852 for budget-aware best-of-K (+10.0 points) and 0.858 for single-routing the best model at lower cost. Here, though, the honesty cuts the other way. With a near-perfect execution verifier and a high union ceiling (0.988 oracle allocation), the two undirected budget-scalable baselines catch up at this budget — the cascade ties RoR (0.952) and random allocation slightly exceeds it (0.959) — because when almost every query is solvable by some draw and verification is reliable, spreading draws widely eventually succeeds regardless of which model is chosen. RoR’s advantage on code is therefore concentrated at low budget (it matches single-routing at ∼3.2× lower cost) rather than in a high-budget accuracy ceiling; crucially, Table IX shows this low-budget advantage is preserved under a realistic, imperfect code verifier, whereas the undirected baselines are not evaluated there because they lack a per-query stopping signal. Distance to the ceiling. From Table V, oracle allocation attains 1.0/0.944/1.0 at mean cost ∼7. For the tested pools, this suggests that knowing which cheap model will succeed is worth more than additional blind budget; the gap between the blue curve and the amber line can accordingly be read as the value of a better router signal rather than of more sampling — the two axes the framework separates. The gains are verifier-gated. From Fig. 2, degrading the verifier shrinks the recovered advantage most where it was largest: at matched mid budget, GPQA falls 0.968 → 0.710 → 0.675 as q : 1.0 → 0.8 → 0.6 (MATH-500: 0.887 → 0.811 → 0.794; GSM8K: 0.993 → 0.979 → 0.970). At q=0.6 the policy still leads all deployable baselines on GPQA (0.675 vs 0.648 budget-aware best-of-K, 0.509 cascade), but on MATH500 its margin inverts slightly against budget-aware best-ofK (0.794 vs 0.803) — it is speculated that, when the verifier cannot be trusted, exploring alternative models buys less than concentrating draws on one strong model. This mirrors the companion paper’s Orepro ≤ Oagg ≤ Oexp ordering: test-time recovery appears available roughly to the extent a deploy-time verifier is. Degraded verifier, full front. Fig. 3 repeats the whole budget sweep at q=0.6 rather than a single matched point. Two observations follow. First, the policies that spread budget across many models (cascade, random) lose the most — at matched mid budget they fall to 0.509/0.507 on GPQA and
0.625/0.637 on MATH-500 — which is reasonable because a weak verifier turns every extra candidate into a chance of selecting a wrong answer. Second, the concentrated policies degrade far less, and the ordering between RoR and budgetaware best-of-K becomes benchmark-dependent (RoR ahead on GPQA, 0.675 vs 0.648; slightly behind on MATH-500, 0.794 vs 0.803), consistent with the interpretation that rerouting pays only when the verifier can be trusted to recognize a specialist’s correct answer. Sensitivity and stability. Table VI varies the two knobs the policy actually has, at a fixed per-query budget B=26. The prior pseudo-count s changes accuracy by at most 1.7 points over an order of magnitude (s=0.5 to 8; GSM8K and MATH500 are essentially flat), so the default s=2 is not tuned to the test set. Calibrating the offline priors on 30% instead of 70% of the queries costs at most 1.3 points (GSM8K 0.983 → 0.996 over the sweep; GPQA 0.735 → 0.744), indicating the policy needs only a coarse ranking of the models, not precise accuracies. Across the 20 draw orderings the standard deviation is 0.003/0.009/0.023 on GSM8K/MATH-500/GPQA respectively — small relative to the reported margins, though largest on GPQA, whose test half has only 99 queries. Real-price replay. The parameter-count proxy preserves size ordering but not real serving economics, so we replay the whole q=1 sweep under a provider-calibrated price vector: the exact OpenRouter output price for the three pool models listed there, and the median output price of same-size-tier text models on the same catalog otherwise (7–9B $0.15/M, n=13; 14–16B $0.24/M; 30–34B $0.458/M; accessed July 2026).1 Real prices decouple cost from size: Phi-4 (14.7B, $0.14/M) is cheaper than most 7–9B models, Llama-3.1-8B ($0.03/M) is 5× cheaper than same-size peers, and the 32B model costs only ∼3× the 7–9B median. From Table VII, the conclusions carry over under this snapshot: RoR reaches 0.992 on GSM8K at $0.06 (tying the cascade), leads MATH-500 with 0.903 at $0.42 against 0.800 for best-of-K and 0.870 for the cascade, and leads GPQA with 0.970 at $0.31 against 0.919/0.929. Notably, budget-aware best-of-K weakens under real prices — committing to one model is harder when price ordering no longer tracks ability — while RoR, which reallocates per query, is less sensitive to the reshuffle. These numbers are 1 openrouter.ai/api/v1/models; prices in $ per million output tokens.
PREPRINT, JULY 2026
6
GSM8K (saturated)
accuracy
1.0 0.9
0.8
0.8
0.6
0.7
0.4
0.6
0.2 10
accuracy
MATH-500 (intermediate)
15
20
25
30
mean cost per query (size proxy)
10
GPQA-Diamond (hard, heterogeneous)
1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3
15
20
25
30
35
40
30
35
mean cost per query (size proxy)
HumanEval+ (code)
1.0 0.9 0.8 0.7 0.6 0.5
10
15
20
25
30
35
mean cost per query (size proxy)
oracle allocation (ceiling) random allocation
40
45
cascade (FrugalGPT-style) budget-aware best-of-K
0.4
10
15
20
25
mean cost per query (size proxy)
ours (UCB variant) Resample-or-Reroute (ours)
single-route (best model) router (one commit)
accuracy @ matched budget
Fig. 1. Cost–quality Pareto fronts on the four regenerated pools (verifier q=1; mean over 20 draw orderings). The policy (blue) dominates or matches every budget-scalable baseline across budgets on GSM8K, MATH-500, and GPQA; on the near-saturated HumanEval+ it leads at low budget but converges with cascade and random allocation at high budget (see text); the amber dashed line is the non-deployable oracle-allocation ceiling. Single-commit policies (star/plus) are budget-independent points — Pareto-incomparable at the lowest GPQA budgets, overtaken beyond cost ∼11.
1.00 0.95 0.90 0.85 0.80 0.75 0.70
GSM8K MATH-500 GPQA-Diamond HumanEval+
0.6
0.8 verifier quality q
1.0
Fig. 2. Verifier-quality ablation: policy accuracy at matched mid budget as the verifier degrades from perfect (q=1) toward a random pick over drawn samples. The largest gains (GPQA) degrade fastest as verifier quality drops.
specific to one provider snapshot; the replay itself re-runs in minutes on CPU for any other price vector. A real verifier: agreement-based verification. Beyond the parametric q, we instantiate a verifier a deployed system can actually run: agreement (self-consistency) — a drawn
sample counts as verified when its extracted answer matches another drawn sample, the query stops once any answer reaches a consensus of A=2 draws, and the final answer is the plurality cluster. No labels or reward model are involved. From Table VIII, the outcome splits by answer space. On GSM8K, whose free-form numeric answers rarely collide by chance, agreement is informative: RoR attains 0.958 against 0.942 for agreement-gated best-of-K, retaining a small lead. On MATH-500 the two are within a point at the larger budget (0.789 vs 0.787). On GPQA, a four-option multiple-choice benchmark, agreement is nearly uninformative — two wrong draws easily agree on the same letter — and the ordering reverses (0.552 vs 0.572 at B=26), with RoR degrading as budget grows (0.536 at B=58) because exploring more models manufactures spurious consensus faster. This matches the parametric prediction that RoR’s edge shrinks and can invert in the low-q regime, and yields a concrete deployment rule: agreement-based verification suffices for open-ended answer spaces, while multiple-choice tasks need a verifier stronger than consensus. A real verifier for code: partial test suites. For HumanEval+ the natural deployable verifier is not agreement
PREPRINT, JULY 2026
7
GSM8K
1.0
MATH-500 0.8
accuracy
0.9
0.6
0.8 0.7
0.4
0.6
0.2 0
25
50 75 100 125 150 175 mean cost per query (size proxy)
0
25
accuracy
GPQA-Diamond
50 75 100 125 150 175 200 mean cost per query (size proxy)
HumanEval+ 0.9
0.8 0.7 0.6 0.5 0.4 0.3
0.8 0.7 0.6 0.5 0
25
0.4
50 75 100 125 150 175 200 mean cost per query (size proxy) RoR (ours)
budget-aware best-of-K
0
25
cascade
50 75 100 125 150 175 mean cost per query (size proxy) random allocation
Fig. 3. Cost–quality fronts of the budget-scalable policies under a degraded verifier (q=0.6; mean over 20 draw orderings). The cascade and random allocation degrade the most; budget-aware best-of-K becomes the only competitive baseline, slightly overtaking RoR on MATH-500 while RoR keeps the lead on GPQA-Diamond.
but a partial test suite: the original HumanEval base tests are visible in the prompt (weak and gameable), while the held-out EvalPlus tests are the truth. Using base-test pass as the early-stopping signal and full-suite pass as ground truth (Table IX), the measured false-accept rate is only 1.0% — a draw that clears the base tests almost always clears the full suite. RoR under this real verifier therefore nearly matches its perfect-verifier ceiling (0.897 vs 0.897 at B=26; 0.918 vs 0.921 at B=58) and stays ahead of base-verified bestof-K (0.870/0.838). This is the code counterpart to the rule above: where multiple-choice agreement is too weak to gate recovery, code ships with a reliable partial verifier, so the testtime recovery RoR exploits is available in practice, not just under an idealized oracle. Latency. Cost is not the only deployment axis. RoR and the cascade are adaptive — each draw’s outcome informs the next choice, so they incur a round-trip per draw — whereas best-of-K and single-route issue their draws in a single parallel batch. Table X reports the mean sequential rounds per query at matched budget. RoR is sequential (2.5 rounds on HumanEval+, 3.3 on GPQA), i.e. higher latency than the one-round baselines, but it consistently uses fewer rounds than the cascade while reaching at least its accuracy (HumanEval+
2.46 vs 2.87; GPQA 3.32 vs 3.59). The adaptivity that buys accuracy is paid for in round-trips, and among the adaptive policies RoR is Pareto-efficient on the accuracy–latency tradeoff. UCB variant. From Table V, the exploration bonus is at best neutral: it ties the greedy rule on GSM8K, trails it slightly on MATH-500 (0.876 vs 0.887 at matched cost), and loses on GPQA at low budgets (0.603 vs 0.744 at mean cost ∼14). It is speculated that exploratory draws on a hard benchmark burn budget that greedy exploitation of the offline prior spends better. Hence, the greedy rule is recommended as the default. VIII. D ISCUSSION Verifier dependence. The policy assumes a usable verifier signal; where no better-than-random verifier exists, resampling cannot be steered and the advantage over the budget-aware best-of-K baseline narrows or inverts (Fig. 2); the agreement instantiation of Table VIII exhibits both regimes on real data. The verifier model is parametric by design — it isolates how much verifier quality the mechanism needs; plugging in a concrete process-reward or execution verifier (e.g. unit tests for code, where q ≈ 1 holds naturally) is a direct instantiation rather than a new method.
PREPRINT, JULY 2026
8
TABLE VI S ENSITIVITY OF THE RO R POLICY AT A FIXED PER - QUERY BUDGET B = 26 ( EARLY STOPPING SPENDS ONLY PART OF IT, SO THIS IS THE LOW- BUDGET OPERATING POINT OF TABLE V): PRIOR PSEUDO - COUNT s, TRAIN FRACTION USED TO CALIBRATE OFFLINE PRIORS , AND STABILITY ( MEAN ± STD OVER THE 20 DRAW ORDERINGS ). GSM8K
MATH-500
GPQA
HumanEval+
prior s = 0.5 prior s = 2 (default) prior s = 8
0.989 0.989 0.989
0.828 0.828 0.833
0.744 0.744 0.761
0.897 0.897 0.888
train frac. 0.3 train frac. 0.5 (default) train frac. 0.7
0.983 0.989 0.996
0.827 0.828 0.843
0.735 0.744 0.744
0.914 0.897 0.906
mean ± std (orderings)
0.989 ± 0.003
0.828 ± 0.009
0.744 ± 0.023
0.897 ± 0.011
TABLE VII R EAL - PRICE REPLAY ( PROVIDER - CALIBRATED $ PER M OUTPUT TOKENS ; EXACT O PEN ROUTER PRICES FOR THREE LISTED MODELS , SIZE - TIER MEDIANS OTHERWISE , ACCESSED J ULY 2026): ACCURACY AT THE POINT NEAREST MEAN COST $0.6 PER 1 K QUERIES SCALE . C OST ORDERING DIFFERS FROM THE PARAMETER PROXY ( E . G . P HI -4 IS CHEAPER THAN MOST 7–9B MODELS ; L LAMA -3.1-8B IS 5× CHEAPER THAN SAME - SIZE PEERS ).
Policy
GSM8K cost acc
MATH-500 cost acc
GPQA cost acc
HumanEval+ cost acc
RoR (ours) budget-aware best-of-K cascade random allocation router (one commit) single-route (best model)
0.06 0.18 0.06 0.23 0.46 0.46
0.42 0.26 0.37 0.61 0.46 0.46
0.31 0.22 0.37 0.59 0.14 0.14
0.25 0.52 0.22 0.41 0.46 0.46
0.992 0.982 0.992 0.992 0.976 0.966
0.903 0.800 0.870 0.880 0.784 0.776
0.970 0.919 0.929 0.887 0.566 0.551
0.953 0.863 0.952 0.960 0.817 0.858
TABLE VIII AGREEMENT- VERIFIED REPLAY ( CONSENSUS THRESHOLD A=2; LABEL - FREE SELF - CONSISTENCY VERIFIER ): ACCURACY ( MEAN SPENT COST ) AT TWO PER - QUERY BUDGETS , MEAN OVER 20 DRAW ORDERINGS .
Benchmark
Policy
B = 26 acc cost
B = 58 acc cost
GSM8K
RoR (agreement) best-of-K (agreement)
0.948 0.941
15.3 16.2
0.958 0.942
15.7 16.3
MATH-500
RoR (agreement) best-of-K (agreement)
0.757 0.779
17.3 17.4
0.789 0.787
20.7 18.9
GPQA
RoR (agreement) best-of-K (agreement)
0.552 0.572
19.8 18.5
0.536 0.569
23.1 19.0
Cost model. Costs are proxied by parameter count, which preserves the pool’s cost ordering but not exact $-ratios; rerunning the replay with a deployment’s own price/latency vector requires no new generation, only re-weighting — Table VII demonstrates this with a provider price snapshot. Batched and latency-constrained serving, where draws of one query can be parallel, is left to future work. Deployment considerations. The policy’s overhead is a few floating-point operations per step (Section IV), so the practical integration cost lies elsewhere: engineering a verifier signal (a reward model, execution tests, or agreement across draws), measuring the deployment’s own price/latency vector, and deciding how much per-query latency the sequential draws may add. When the pool is homogeneous or the benchmark is near-saturated, the simpler budget-aware best-of-K captures most of the benefit (Table V); the case for full RoR is strongest when models genuinely specialize and a trustworthy verifier exists. Offline replay. Replaying pre-computed draws is exact for correctness-based rewards under the companion protocol’s
seed-aligned generation, and makes every number reproducible from the released tensors on CPU; it does not capture promptadaptive behaviors (e.g. revising the prompt between draws). IX. C ONCLUSION This work formulated the choice between resampling a committed LLM and rerouting to an alternative model as a single budgeted decision, proposed an online allocation policy driven by estimated marginal correctness per unit cost, and grounded its behavior in the recoverability asymmetry of the companion analysis. Replay experiments on four regenerated open-model pools showed that, across these benchmarks and for the tested pools and baselines, the proposed policy attains a favorable cost–quality Pareto front: it dominates or matches every budget-scalable baseline and overtakes the single-commit baselines once the budget affords a second draw, winning chiefly on cost where the benchmark is saturated and on accuracy where the pool is heterogeneous; the gains are verifier-gated, as the companion theory predicts.
PREPRINT, JULY 2026
9
TABLE IX H UMAN E VAL + WITH A deployable CODE VERIFIER : THE IN - PROMPT BASE TESTS GATE EARLY- STOPPING ( WEAK , GAMEABLE ), THE FULL E VAL P LUS SUITE IS TRUTH . M EASURED BASE - VERIFIER FALSE - ACCEPT RATE 1.0%; RO R UNDER THIS REAL VERIFIER NEARLY MATCHES ITS PERFECT- VERIFIER CEILING ( MEAN OVER 20 ORDERINGS ).
Policy
B = 26 acc cost
B = 58 acc cost
RoR (base-verifier) RoR (oracle ceiling) best-of-K (base-verifier)
0.897 0.897 0.870
0.918 0.921 0.838
10.2 10.1 10.4
13.2 13.1 16.7
TABLE X L ATENCY PROXY: MEAN SEQUENTIAL ROUNDS ( ROUND - TRIPS ) PER QUERY AT MATCHED BUDGET. RO R AND CASCADE ARE ADAPTIVE ( ONE DRAW PER ROUND ); BEST- OF -K AND SINGLE - ROUTE ISSUE THEIR DRAWS IN A SINGLE PARALLEL ROUND .
single-route best-of-K (parallel) cascade RoR (ours)
GSM8K
MATH-500
GPQA
HumanEval+
1.0 1.0 1.7 1.2
1.0 1.0 3.8 3.6
1.0 1.0 3.6 3.3
1.0 1.0 2.9 2.5
Robustness checks with a provider-price replay and a labelfree agreement verifier show where these conclusions carry over (real prices; open-ended answer spaces) and where they weaken (consensus on multiple-choice tasks). In the future, a line of research is to instantiate stronger learned verifiers (e.g. process-reward models) and to extend the replay protocol to batched and latency-constrained serving. R EFERENCES [1] M. F. Argerich and M. Patiño-Martı́nez, “Measuring and improving the energy efficiency of large language models inference,” IEEE Access, vol. 12, pp. 80 194–80 207, 2024. [2] T. Khan, W. Tian, G. Zhou, S. Ilager, M. Gong, and R. Buyya, “Machine learning (ML)-centric resource management in cloud computing: A review and future directions,” Journal of Network and Computer Applications, vol. 204, p. 103405, 2022. [3] M. Shao, A. Basit, R. Karri, and M. Shafique, “Survey of different large language model architectures: Trends, benchmarks, and challenges,” IEEE Access, vol. 12, pp. 188 664–188 706, 2024. [4] Q. J. Hu et al., “RouterBench: A benchmark for multi-LLM routing system,” arXiv preprint arXiv:2403.12031, 2024. [5] H. Li, Y. Zhang, Z. Guo, C. Wang, S. Tang, Q. Zhang, Y. Chen, B. Qi, P. Ye, L. Bai, Z. Wang, and S. Hu, “LLMRouterBench: A massive benchmark and unified framework for LLM routing,” in Findings of ACL, 2026, arXiv:2601.07206. [6] L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to use large language models while reducing cost and improving performance,” Transactions on Machine Learning Research (TMLR), 2024. [7] I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica, “RouteLLM: Learning to route LLMs from preference data,” in Proc. ICLR, 2025. [8] T.-R. Chen, “How much of the routing gap is real? Decomposing the router-to-oracle gap into reproducible specialist advantage and singledraw label noise,” 2026, arXiv:2607.03436 (companion paper). [9] T. Sheokand, G. Jain, A. Bahga, and V. K. Madisetti, “Enhancing LLM reasoning capabilities through brokered multi-expert reflection,” IEEE Access, vol. 13, pp. 67 993–68 019, 2025. [10] Z. Xu, M. J. Kusner, K. Q. Weinberger, M. Chen, and O. Chapelle, “Classifier cascades and trees for minimizing feature evaluation cost,” Journal of Machine Learning Research, vol. 15, pp. 2113–2144, 2014. [11] P. L. Bartlett and M. H. Wegkamp, “Classification with a reject option using a hinge loss,” Journal of Machine Learning Research, vol. 9, pp. 1823–1840, 2008. [12] K. Hendrickx, L. Perini, D. Van der Plas, W. Meert, and J. Davis, “Machine learning with a reject option: A survey,” Machine Learning, vol. 113, pp. 3073–3110, 2024.
[13] Z. Zheng, H. Ma, M. R. Lyu, and I. King, “QoS-aware web service recommendation by collaborative filtering,” IEEE Transactions on Services Computing, vol. 4, no. 2, pp. 140–152, 2011. [14] R. G. Pacheco, R. S. Couto, and O. Simeone, “On the impact of deep neural network calibration on adaptive edge offloading for image classification,” Journal of Network and Computer Applications, vol. 217, p. 103679, 2023. [15] E. Mustafa, J. Shuja, F. Rehman, A. Riaz, M. Maray, M. Bilal, and M. K. Khan, “Deep neural networks meet computation offloading in mobile edge networks: Applications, taxonomy, and open issues,” Journal of Network and Computer Applications, vol. 226, p. 103886, 2024. [16] H. Gauttam, G. Nain, K. K. Pattanaik, and P. Mendes, “Edge-AI: A systematic review on architectures, applications, and challenges,” Journal of Network and Computer Applications, vol. 245, p. 104375, 2026. [17] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” in Proc. ICLR, 2023, arXiv:2203.11171. [18] C. Snell, J. Lee, K. Xu, and A. Kumar, “Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning,” in Proc. ICLR, 2025, arXiv:2408.03314. [19] D. T. Farr, L. H. X. Ng, I. J. Cruickshank, N. Manzonelli, N. Clark, K. Starbird, N. D. Bastian, and J. West, “Ensemble-based uncertainty quantification for reliable large language model classification in social data applications,” IEEE Access, vol. 13, pp. 116 419–116 429, 2025. [20] Z. Huang, G. Ling, Y. Lin, Y. Chen, S. Zhong, H. Wu, and L. Lin, “RouterEval: A comprehensive benchmark for routing LLMs to explore model-level scaling up in LLMs,” arXiv preprint arXiv:2503.10657, 2025. [21] S. Sheikhi, L. Lovén, and P. Kostakos, “Beyond the leaderboard: A survey of the science of evaluation, benchmarking, and methodologies for large language models,” IEEE Access, vol. 14, pp. 66 493–66 515, 2026. [22] M. Mizrahi, G. Kaplan, D. Malkin, R. Dror, D. Shahaf, and G. Stanovsky, “State of what art? A call for multi-prompt LLM evaluation,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 933–949, 2024. [23] A. Mussa, Z. Tuimebayev, and M. Mansurova, “Make large language models efficient: A review,” IEEE Access, vol. 13, pp. 154 466–154 490, 2025. [24] J. Hu, M. Xu, K. Ye, and C. Xu, “BrownoutServe: SLO-aware inference serving under bursty workloads for MoE-based LLMs,” IEEE Transactions on Computers, 2026, arXiv:2507.17133. [25] F. Xu, J. Xu, J. Chen, L. Chen, R. Shang, Z. Zhou, and F. Liu, “iGniter: Interference-aware GPU resource provisioning for predictable DNN inference in the cloud,” IEEE Transactions on Parallel and Distributed Systems, vol. 34, no. 3, pp. 812–827, 2023. [26] J.-Y. Audibert, S. Bubeck, and R. Munos, “Best arm identification in multi-armed bandits,” in Proc. Conf. on Learning Theory (COLT), 2010, pp. 41–53.
PREPRINT, JULY 2026
[27] E. Kaufmann, O. Cappé, and A. Garivier, “On the complexity of bestarm identification in multi-armed bandit models,” Journal of Machine Learning Research, vol. 17, no. 1, pp. 1–42, 2016. [28] Z. Jiang, J. Araki, H. Ding, and G. Neubig, “How can we know when language models know? On the calibration of language models for question answering,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 962–977, 2021.
10