arXiv CS open access

Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

2026 · arxiv_cs

arXiv CS · Papers · 2026

artificial-intelligence knowledge-representation reasoning

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

arXiv:2608.12150v1 [cs.AI] 12 Aug 2026

Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation Rodrigo Guedes de Souza Alison R. Panisson Federal University of Santa Catarina (UFSC) Brazil guedes.rodrigo@grad.ufsc.br, alison.panisson@ufsc.br August 13, 2026 Abstract Standard evaluation of large language models assumes stable model rankings across inference conditions. We challenge this assumption by varying the token generation budget, i.e., the maximum tokens a model may produce, across seven levels (64–4,096), evaluating four models on three reasoning benchmarks (56,476 inferences). We report four findings: (i) 3– 19% of items exhibit non-monotone behavior (accuracy decreasing with more budget), even after controlling for truncation, and this phenomenon is model-specific (cross-model overlap: 6–14%). (ii) Model rankings reverse across budgets on all benchmarks (p<0.01, McNemar). (iii) Oracle analysis reveals model complementarity up to +27.8 pp, most pronounced at constrained budgets. (iv) A budget-aware router captures 14.1% of the oracle gap cross-domain; budget features help within-domain (+1.6 to +5.7 pp) but are domain-specific and hurt transfer (−1.2 pp). These results argue for budget-conditioned evaluation protocols.

Introduction

Leaderboards and benchmark comparisons form the primary evidence base for selecting Large Language Models (LLMs) in practice [1, 2]. A model declared “state of the art” on GSM8K or GPQA is presumed superior across deployment scenarios. Yet this conclusion rests on an implicit assumption: that model rankings are invariant to the inference-time token generation budget, i.e., the maximum number of tokens the model is allowed to produce before its output is truncated or terminated. This assumption is increasingly untenable. Recent work on test-time compute scaling [3, 4] has shown that allowing models more tokens to “think” can 1

substantially improve performance, but the effect varies across models and tasks. Meanwhile, the “overthinking” literature [7, 8] documents cases where additional reasoning tokens harm accuracy. These observations hint at a deeper phenomenon: the evaluation landscape may be fundamentally budget-dependent, with model rankings, item difficulty, and model complementarity all varying as a function of the token budget. We provide the first systematic investigation of this phenomenon. We evaluate four models spanning 8B to 70B parameters across three reasoning benchmarks at seven token budgets from 64 to 4,096 tokens, a total of 56,476 individual inferences at temperature T =0 for full determinism. Our analysis yields four contributions: 1. Item-level behavioral taxonomy. We classify each model–item pair into four behavioral categories under budget variation (always-correct, monotoneincreasing, non-monotone, always-wrong). We find that non-monotone behavior, representing performance degradation with increased budget, is not rare (up to 25.8% of items) and, crucially, is model-specific: the same item rarely triggers overthinking across different models (cross-model overlap as low as 9%). 2. Statistically significant ranking reversals. The best-performing model changes across budget levels on all three benchmarks. On GSM8K, LLaMA3.3 70B leads at b=256 (62.4%) while GPT-OSS 20B dominates at b=4096 (94.8%, p < 0.001); on GPQA, LLaMA-3 8B ranks first at b=512 (21.2%) before GPT-OSS 20B leads nominally at b=4096 (51.0% vs. 50.0%, not significant; n=198). Multiple intermediate reversals are significant (McNemar’s χ2 , p < 0.01). 3. Oracle gap dynamics. A per-item oracle ensemble reveals that model complementarity is most valuable under constrained budgets. On GPQA, the oracle exceeds the single best model by 27.8 pp at b=4096 and exhibits even larger relative gains at lower budgets. On GSM8K, the oracle gap is non-monotonic, peaking at b=256 (+16.9 pp) before declining as models converge (Jaccard similarity: 0.048 → 0.741). 4. Budget-aware routing proof-of-concept. We train per-model XGBoost classifiers on text features and budget to predict item-level correctness, then route each item to the model with the highest predicted probability. In a cross-domain evaluation (train on GSM8K+MATH-500, test on GPQA), this approach achieves +2.67 pp over the best-per-budget baseline (95% CI [0.94, 4.40]), capturing 14.1% of the oracle gap. A within-domain ablation reveals that budget features provide +1.6 to +5.7 pp, yet these patterns are domain-specific and hurt cross-domain transfer by −1.2 pp. Figure 1 illustrates our core finding: model rankings are not stable across token budgets. These results have direct implications for evaluation practice, model deployment, and the design of routing systems that must operate under heterogeneous compute constraints.

LLaMA-3 8B

Qwen-3 32B

LLaMA-3.3 70B

GPT-OSS 20B

6 25

96 40

Token Budget

48 20

96 40

48 20

2 51

6 25

8 12

Token Budget

Accuracy (%)

LLaMA-3 8B Qwen-3 32B LLaMA-3.3 70B GPT-OSS 20B

64 12 8 25 6 51 2 10 24 20 48 40 96

Qwen-3 32B

64 12 8 25 6 51 2 10 24 20 48 40 96

LLaMA-3 8B

64 12 8 25 6 51 2 10 24 20 48 40 96

Accuracy (%)

Budget-Dependent Model Landscapes: Rankings Reverse Under Different Token Budgets GSM8K MATH-500 GPQA

Token Budget

Figure 1: Model rankings depend on token budget. Top: Accuracy heatmap across four models and seven budgets on three benchmarks; black borders indicate the best model at each budget. Bottom: Corresponding scaling curves showing crossover points where the identity of the top-performing model changes. Rankings reverse on all three benchmarks.

Related Work

Test-time compute scaling. A growing body of work studies how allocating additional computation at inference time affects model performance. Chain-ofthought prompting [6] implicitly increases the token budget by eliciting stepby-step reasoning. More recent approaches explicitly scale test-time compute through search [3], extended thinking [4], and budget-forcing mechanisms [5]. These works generally study a single model under varying compute; we study multiple models simultaneously, revealing that scaling curves cross and rankings reverse. Overthinking and reasoning efficiency. Several studies document cases where allowing LLMs more reasoning steps degrades performance. In [7], the authors identify overthinking as a failure mode of o1-like models, and [8] propose early termination strategies. Our work quantifies this at the item level across multiple models, revealing that overthinking is predominantly a model-specific phenomenon rather than an item-inherent property. Model routing and selection. LLM routing systems [9, 10, 11] aim to select the best model per query based on input features. Most approaches treat routing as budget-agnostic, optimizing for a single inference condition. Our budgetaware routing formulation is, to our knowledge, the first to incorporate the token budget as an explicit routing signal, and our SHAP analysis reveals it dominates text-based features.

Evaluation methodology. The reliability of LLM benchmarks has been questioned on multiple fronts: contamination [12], saturation [13], and sensitivity to prompt formatting [14]. Our work identifies a new axis of fragility: sensitivity to the token generation budget. We show that a benchmark’s model ranking is not a single stable ordering but a family of orderings parameterized by budget.

Experimental Framework

Models. We evaluate four open-weight reasoning models spanning an order of magnitude in parameter count: LLaMA-3 8B [15], Qwen-3 32B [16], LLaMA-3.3 70B [15], and GPT-OSS 20B (model ID: openai/gpt-oss-20b, served via Together.ai; see Appendix A for full model details). All models are evaluated with greedy decoding (T =0) to ensure deterministic outputs. Benchmarks. We use three reasoning benchmarks of increasing difficulty: GSM8K [17] (1,319 grade-school math problems), MATH-500 [18] (500 competition-level math problems), and GPQA-Diamond [19] (198 graduatelevel science questions). These span a wide difficulty range: IRT-inspired analysis yields mean easiness scores of 0.507, 0.253, and 0.172, respectively. Token budgets. We evaluate each model–item pair at seven token budgets b ∈ {64, 128, 256, 512, 1024, 2048, 4096}, where b is the max_tokens parameter. This yields 4×3×7 = 84 model–dataset–budget configurations and 4×2,017×7 = 56,476 individual inferences. Evaluation protocol. We extract final answers using regex-based parsing and evaluate via exact match against ground truth. Correctness is binary: a model scores 1 on item i at budget b if and only if it produces the correct final answer within b tokens. Truncation and three-tier analysis. When a model’s generation is cut off at b tokens before producing a final answer, we mark it as truncated (finish_reason = “length”) and score it as incorrect. Truncation rates vary dramatically across models: Qwen-3 32B, which uses internal <think> tokens, remains truncated on 59.7% of GPQA items even at b=4,096, while LLaMA-3.3 70B drops to 0.5% at the same budget (see Appendix, Figure 8). To disentangle genuine reasoning effects from truncation artifacts, we report a three-tier analysis throughout: (a) all items (standard scoring), (b) stop-only (restricting per model to items where that model completed its generation), and (c) common non-truncated (items where all four models completed, enabling paired comparisons on identical item sets). We note that stop-only accuracy is upward-biased (completed items tend to be easier for that model), while common non-truncated sets can be small at low budgets; we mark results as unreliable when N < 30.

Item Behavioral Taxonomy by Model GSM8K

Percentage of Items (%)

100 19%

12%

MATH-500

20% 36%

17%

76%

60 40

84% 64%

25%

GPQA 24%

11%

50%

64%

69%

26%

24%

30%

60%

LLaMA-3 8B

Qwen-3 32B

Always Correct Monotone Improve Non-Monotone Always Wrong

23%

18%

LLaMA-3 8B

61%

45%

93% 74%

20 0

32%

LLaMA-3.3 GPT-OSS 70B 20B

LLaMA-3 8B

Qwen-3 32B

LLaMA-3.3 GPT-OSS 70B 20B

44%

51%

LLaMA-3.3 GPT-OSS 70B 20B

Figure 2: Behavioral taxonomy across models and benchmarks. Each bar decomposes items into four categories based on how correctness evolves with budget. Non-monotone items (red), indicating “overthinking,” are non-trivial across all settings and most pronounced on GPQA.

Budget-Dependent Model Behavior

4.1

Item-Level Behavioral Taxonomy

For each model m and item i, we observe a binary correctness trajectory across 1 7 seven budgets: cm,i = (cbm,i , . . . , cbm,i ) ∈ {0, 1}7 . We classify each trajectory into one of four behavioral categories: • Always-correct: cbm,i = 1 for all b. The model solves this item regardless of budget. • Monotone-increasing: the sequence transitions from 0 to 1 and never reverts. More budget always helps. • Non-monotone: the sequence contains at least one 1 → 0 transition at increasing budget. The model loses a previously correct answer when given more tokens—an “overthinking” failure. • Always-wrong: cbm,i = 0 for all b. The model fails regardless of budget. Figure 2 shows the distribution of these categories across all model–dataset combinations. The monotone-increasing category dominates across all settings, confirming that more budget generally helps. However, non-monotone items are far from negligible: they represent 3.6% (GPT-OSS 20B on GSM8K) to 25.8% (LLaMA-3 8B on GPQA) of items. GPQA exhibits the highest non-monotone rates across all models, up to 25.8% for LLaMA-3 8B and 23.2% for LLaMA3.3 70B, consistent with the intuition that harder problems are more susceptible to overthinking. Controlling for truncation in the taxonomy. Since truncation at lower budgets mechanically creates 1 → 0 transitions when a model answers correctly at some budget but is truncated at a later one, we recompute the taxonomy using only the non-truncated portion of each trajectory (budgets where finish_reason = “stop”). Non-monotone rates decrease but remain substantial (Figure 3): LLaMA-3 8B on GPQA drops from 25.8% to 19.1%, LLaMA-3.3 70B on MATH500 barely changes (10.6% to 10.3%), and even Qwen-3 32B on GSM8K retains 5

Non-Monotone (Overthinking) Rates: All vs Stop-Only Trajectories GSM8K Non-Monotone Rate (%)

MATH-500

GPQA-D.

All budgets Stop-only budgets

25 20 15 10 5 0

LLaMA-3 8B

Qwen-3 32B

LLaMA-3.3 70B

GPT-OSS 20B

LLaMA-3 8B

Qwen-3 32B

LLaMA-3.3 70B

GPT-OSS 20B

LLaMA-3 8B

Qwen-3 32B

LLaMA-3.3 70B

GPT-OSS 20B

Figure 3: Non-monotone (overthinking) rates: all budgets vs. stop-only trajectories. Rates decrease after excluding truncated budgets but remain substantial, particularly on GPQA, confirming that overthinking is a genuine reasoning failure. a 3.3% non-monotone rate after filtering (down from 11.7%, where most were truncation artifacts from its verbose <think> tokens). These results confirm that overthinking is a genuine phenomenon, not a truncation artifact. Overthinking is model-specific, not item-inherent. A natural question is whether certain items inherently induce overthinking, or whether this is a model-specific phenomenon. We compute the cross-model overlap: the fraction of non-monotone items that are flagged as non-monotone by at least two of the four models. On stop-only trajectories, the overlap rates are remarkably low: 10.1% on GSM8K, 6.4% on MATH-500, and 13.8% on GPQA (compared to 16.2%, 9.7%, and 27.7% on all-item trajectories). Filtering strengthens the claim: 86–94% of items exhibiting genuine overthinking do so for only one model. The implication is important: overthinking cannot be mitigated by removing “problematic” items from benchmarks, because the set of problematic items is model-dependent. Where do non-monotone transitions occur? Analysis of the budget level at which non-monotone drops happen reveals they are concentrated at high budgets (b ≥ 1024), suggesting that the additional tokens generated under generous budgets can derail an initially correct reasoning chain (see Appendix, Figure 9).

4.2

Ranking Reversals

We now examine whether budget variation leads to changes in which model ranks first, not just changes in absolute performance. Table 1 reports the best model at selected budget levels along with McNemar test statistics for the significance of the difference with the second-ranked model. Several patterns emerge. On GSM8K, LLaMA-3.3 70B leads at b=256 (χ2 = 59.57, p < 0.001), but GPT-OSS 20B overtakes it by b=512 and maintains the lead through b=4096 with increasing significance (χ2 = 15.86 at b=4096). On MATH-500, LLaMA-3.3 70B dominates across mid-range budgets (b=256 through b=1024, all p < 0.001), but GPT-OSS 20B closes the gap and leads at 6

Table 1: Ranking reversals across budgets. Best model and McNemar’s test comparing it to the second-ranked model. The identity of the best model shifts with budget on all three benchmarks. Significance: ∗ p < 0.05, ∗∗ p < 0.01, ∗∗∗ p < 0.001. Best Model

Acc

2nd Model

Acc

χ2

Sig

256 512 1024 4096

LLaMA-3.3 70B GPT-OSS 20B GPT-OSS 20B GPT-OSS 20B

62.4% 79.2% 92.3% 94.9%

GPT-OSS 20B LLaMA-3.3 70B Qwen-3 32B Qwen-3 32B

48.6% 76.6% 80.2% 91.7%

59.57 2.78 86.12 15.86

∗∗∗

LLaMA-3.3 70B LLaMA-3.3 70B LLaMA-3.3 70B GPT-OSS 20B

21.3% 49.4% 63.3% 70.8%

LLaMA-3 8B GPT-OSS 20B GPT-OSS 20B LLaMA-3.3 70B

13.5% 30.0% 48.2% 67.1%

16.41 70.51 43.81 2.54

∗∗∗

MATH-500

256 512 1024 4096

GPQA

256 512 1024 4096

LLaMA-3 8B LLaMA-3 8B LLaMA-3.3 70B GPT-OSS 20B

7.1% 21.2% 40.9% 51.0%

LLaMA-3.3 70B LLaMA-3.3 70B GPT-OSS 20B LLaMA-3.3 70B

4.5% 13.6% 27.8% 50.0%

1.07 4.56 9.47 0.02

Dataset

Budget

GSM8K

b=4096—though this final reversal is not significant (p = 0.11). On GPQA, the most dramatic trajectory unfolds: the smallest model (LLaMA-3 8B) ranks first at b=256–512, is overtaken by LLaMA-3.3 70B at b=1024 (p < 0.01), which is in turn matched by GPT-OSS 20B at b=4096—though this last difference is not statistically significant (χ2 =0.02, p=0.89, n=198). Results on GPQA-Diamond should be interpreted with caution due to the small sample size. Controlling for truncation. Since truncation rates differ across models (e.g., Qwen-3 32B is 59.7% truncated on GPQA at b=4096 vs. 0.5% for LLaMA3.3 70B), ranking reversals could reflect differential truncation rather than genuine reasoning differences. We repeat the analysis on common non-truncated items, i.e., those where all four models completed their generation, and report paired McNemar tests. At budgets where common non-truncated sets are sufficiently large (N ≥ 30), the key findings survive: on GSM8K at b=4096 (N =1,264): GPT-OSS 20B leads (95.9%) over Qwen-3 32B (93.2%, χ2 =12.66, p<0.001); on GPQA at b=4096 (N =71): GPT-OSS 20B leads (85.9%) over Qwen-3 32B (64.8%, χ2 =9.33, p<0.01). We also observe that some all-items rankings are genuine truncation artifacts: on MATH-500 at b=4096, GPT-OSS 20B leads on all items (71.0%) but Qwen3 32B leads on common non-truncated items (86.7%), reflecting GPT-OSS’s higher completion rate rather than superior reasoning. This nuance strengthens rather than weakens our thesis: the landscape depends on whether one measures “ability to answer given enough tokens” or “ability to answer within the budget,” and both are legitimate evaluation criteria. Figure 4 provides a side-by-side comparison of all-items vs. stop-only accuracy.

∗∗∗ ∗∗∗

∗ ∗∗

7.1%

21.2%

GPQA-D.: Stop-Only Items

26.3%

28.6%

100

28.8%

0.5%

0.0%

1.0%

2.5%

9.6%

18.7%

33.7%

0.5%

3.0%

4.5%

13.6%

40.9%

51.0%

50.0%

60 40 20

0.0%

0.5%

1.5%

128

256

12.1%

27.8%

43.9%

51.0%

512

1024

2048

4096

Token Budget

(0)

(16)

35.6%

30.3%

32.4%

28.9%

(0)

(1)

(11)

71.1%

62.0%

(0)

(26)

49.7%

51.3%

50.3%

(0)

(1)

(3)

(24)

85.9%

86.1%

84.2%

128

256

512

1024

2048

4096

100 80

Accuracy (%)

2.5%

60 40 20

Token Budget

Accuracy (%)

GPQA-D.: All Items 1.0%

Figure 4: All-items vs. stop-only accuracy on GPQA. Left: standard scoring (truncated = incorrect). Right: accuracy restricted to items where each model completed its generation. Parenthesized values indicate small sample sizes (N < 30). Black borders mark the best model per budget. Rankings persist at high budgets even under stop-only scoring. Cross-Model Agreement (Jaccard) vs. Token Budget

MATH-500

GPQA

GSM8K

14.6pp 12.8pp 10.8pp

MATH-500

GPQA

6.2pp

4.5pp 2.0pp

0.6 0.4 0.2

(a) Oracle gap vs. token budget.

96 40

Token Budget

24 10

Token Budget

24 10

Token Budget

24 10

40 96

Token Budget

20 48

51 2

0.0

10 24

40 96

51 2

Token Budget

20 48

25 6

1.0pp

25 6

1.2pp

10 24

40 96

20 48

51 2

10 24

25 6

0.0pp

Token Budget

5.8pp

3.5pp

12 8

6.6pp 4.0pp

12 8

5.5pp

LLaMA QwenLLaMA LLaMA LLaMA GPT-O Qwen- LLaMA Qwen- GPT-O LLaMA GPT-O Mean

0.8

14.9pp

Jaccard Similarity

7.5pp

12 8

Oracle Gap (percentage points)

Oracle Ensemble Gap vs. Token Budget GSM8K 16.9pp

18 15 12 10 8 5 2.0pp 2 0

(b) Pairwise Jaccard similarity vs. budget.

Figure 5: Model complementarity varies with budget. (a) The oracle gap (oracle ensemble minus best single model) peaks at low-to-moderate budgets for GSM8K but grows monotonically for harder benchmarks. (b) Mean pairwise Jaccard similarity between models’ correct-answer sets starts near zero and increases with budget, but never reaches unity, i.e., models remain complementary even at b=4096.

4.3

Model Complementarity and Oracle Gap

If different models excel on different items at different budgets, then an oracle that selects the best model per item could substantially outperform any single model. We quantify this via the oracle gap: the difference between a per-item oracle ensemble and the single best model at each budget. Figure 5a plots the oracle gap across budgets for all three benchmarks. The dynamics differ markedly by dataset difficulty: • GSM8K exhibits a non-monotonic oracle gap that peaks at +16.9 pp around b=256 before declining to +3.5 pp at b=4096. At moderate budgets, models solve different items, creating maximum complementarity. At high budgets, they converge on the same (easier) items, reducing the benefit of selection. • MATH-500 shows a monotonically increasing gap, reaching +12.8 pp at b=4096 (CI [+10.0, +15.8]). On this harder benchmark, models remain complementary even at generous budgets. • GPQA displays the largest gap: +27.8 pp at b=4096 (CI [+21.7, +33.8]), indicating that the four models solve highly non-overlapping subsets of these graduate-level questions.

Convergence without consensus. Figure 5b shows the mean pairwise Jaccard similarity between models’ correct-answer sets. At b=64, models agree on almost no items (J¯ = 0.048), consistent with the near-random performance at this budget. As budgets increase, agreement grows substantially: J¯ = 0.285 at b=256, 0.528 at b=512, and 0.741 at b=4096. However, the asymptotic value of 0.741 indicates that even at generous budgets, models solve meaningfully different item subsets, consistent with the persistent oracle gap of 3.5–27.8 pp at b=4096 across benchmarks. Item-level difficulty. Across all items and budgets, 1.1% of GSM8K items, 13.6% of MATH-500 items, and 10.1% of GPQA items are answered incorrectly by all four models at all seven budgets, representing a hard core of items beyond current model capabilities. Conversely, only 16.5% of item–budget pairs see all four models correct, reinforcing that model selection matters.

Budget-Aware Model Routing

The oracle gap analysis reveals that an ideal router could achieve 14–28 pp above the best single model. Can a practical router capture some of this gap? We design a budget-aware routing mechanism that selects the best model for each item, given the token budget as an explicit feature. Method. For each model m, we train an XGBoost binary classifier fm (x, b) → [0, 1] that predicts the probability that model m answers item x correctly at budget b. Features include: log2 (b) (budget), surface-level text statistics (character count, word count, number of special characters, presence of LaTeX, word entropy, maximum number magnitude), and 20 PCA-reduced dimensions from sentence embeddings (all-MiniLM-L6-v2). At inference time, for each item–budget pair, the router selects m∗ = arg maxm fm (x, b). Baselines. We compare against five baselines: Random (uniform selection), Largest-Always (always select GPT-OSS 20B, the largest model), Best-Overall (select the model with highest aggregate accuracy across all budgets), Best-PerBudget (select the model with highest accuracy at each budget, applied uniformly to all items at that budget), and Oracle (per-item optimal selection with ground truth). Cross-domain evaluation. To test generalization, we train on GSM8K and MATH-500 and evaluate on GPQA, a domain-transfer setting. Table 2 shows the results. The Router-Scoring strategy achieves 22.9% accuracy, a +2.67 pp improvement over Best-Per-Budget (95% bootstrap CI [0.94, 4.40], significant since the interval excludes zero). This captures 14.1% of the oracle gap. On the discriminative subset (520 items where models disagree), the gain is +7.12 pp. The per-budget breakdown (Figure 6) reveals that the router’s advantage is concentrated at moderate budgets: at b=1024, it achieves 40.9% versus 27.8% for Best-Per-Budget (+13.1 pp). At extreme budgets (b=64 and b=4096), the 9

Table 2: Cross-domain routing results (train: GSM8K+MATH-500, test: GPQA). The per-model scoring router captures 14.1% of the oracle gap over Best-Per-Budget with a statistically significant improvement. “Disc. subset” restricts to the 520 out of 1,386 item–budget pairs where models disagree. Accuracy

∆ vs BPB

Oracle Gap

Disc. Subset

Random Largest-Always Best-Overall Best-Per-Budget Router-Scoring

17.1% 23.4% 19.6% 20.3% 22.9%

−3.2 pp +3.1 pp −0.7 pp — +2.67 pp

— — — 0% 14.1%

41.0% 57.7% 47.5% 49.4% 56.5%

Oracle

39.2%

+18.9 pp

100%

100.0%

Strategy

router offers minimal improvement, consistent with the low model differentiation at these extremes. Feature importance. SHAP analysis (Figure 10) reveals that the budget feature (log2 b) dominates all other features by a wide margin: its mean absolute SHAP value (2.21) is 6.1× larger than the next feature (presence of LaTeX: 0.36). Embedding-derived features contribute modestly (0.10–0.24 each), while text statistics contribute even less. This confirms that budget is the primary axis of variation in model performance, the core message of this paper. Ablation study. Table 3 reports ablations removing key feature groups in the cross-domain setting. A notable finding is that removing budget features (“No budget”) yields higher cross-domain accuracy (24.2%) than the full router (22.9%, ∆=−1.2 pp). To determine whether this reflects a global failure of budget features or a domain-transfer artifact, we run a complementary withindomain ablation: 5-fold cross-validation on each benchmark separately (Table 4). Within-domain, budget features provide a consistent advantage: +5.7 pp on GSM8K, +3.0 pp on MATH-500, and +1.6 pp on GPQA. The reversal in the cross-domain setting reveals that budget-accuracy mappings are domain-specific: the relationship between budget and correctness learned on math problems does not transfer to graduate-level science. Text features, being domain-agnostic, generalize better across domains—but within a domain, budget remains the dominant routing signal, consistent with the SHAP analysis above. Figure 7 visualizes this contrast.

Discussion

Implications for evaluation. Our findings challenge the common practice of reporting a single accuracy number per benchmark. Since model rankings depend on the token budget, a benchmark’s ranking is not a fixed ordering but a family of orderings parameterized by b. We advocate for budget-conditioned evaluation: reporting accuracy at multiple budget levels and explicitly stating 10

Routing Accuracy by Budget (Cross-Dataset GPQA) 80

Random Best-Per-Budget Router-Scoring Oracle

Routing Accuracy (%)

70 60 50 40 30 20 10 0 64

128

256

512

Token Budget

1024

2048

4096

Figure 6: Router accuracy by budget level (cross-domain, GPQA). The router’s advantage is concentrated at moderate budgets (b=512–2048), where model complementarity is highest and the oracle gap is largest. the budget used. This is especially critical for constrained deployment scenarios (mobile, edge, real-time) where generous budgets are infeasible. Implications for routing. The large oracle gaps (+3.5 to +27.8 pp) indicate that substantial gains are available from intelligent model selection. Our routing experiments reveal a nuanced picture: budget is the dominant within-domain feature (+1.6 to +5.7 pp), yet budget–accuracy patterns are domain-specific and hurt cross-domain transfer (−1.2 pp). This suggests that practical routing systems should treat budget as a powerful but non-transferable signal, complementing it with domain-agnostic features or domain adaptation techniques. Future routing systems may additionally benefit from model-internal signals (e.g., logit entropy, hidden-state representations) rather than text features alone. The truncation confound. Truncation is a significant confound when varying token budgets: at low budgets, most models are truncated on most items, and truncation is near-perfectly correlated with incorrect answers. We have addressed this through a three-tier analysis (Section 4.2): all items, stop-only per model, and common non-truncated items with paired McNemar tests. The key findings, non-monotone behavior (Section 4.1), ranking reversals, and oracle gaps, all persist under truncation control, though some specific rankings are revealed to be truncation artifacts (e.g., MATH-500 at b=4096; see Section 4.2). Notably, non-monotone rates decrease by only 1–8 pp after filtering truncated budgets, confirming that overthinking is predominantly a genuine reasoning failure. We further validate this with a direct search for items where a model is correct at a low budget but wrong at a higher budget, both with finish_reason=stop: across all models and datasets, we identify 1,193 such pairs (Appendix G). Future

Table 3: Cross-domain ablation (train: GSM8K+MATH-500, test: GPQA). Removing budget features improves cross-domain accuracy, suggesting that budget–accuracy patterns are domain-specific. Configuration

Accuracy

Random Best-Per-Budget Text stats only (no budget, no embeddings) No budget No embeddings Full Router Oracle

17.1% 20.3% 21.8% 24.2% 21.3% 22.9% 39.2%

Table 4: Budget feature impact: within-domain vs. cross-domain. Budget features consistently help within-domain (5-fold CV) but hurt cross-domain transfer, confirming that budget–accuracy patterns are domain-specific rather than universally transferable. Full Router

No Budget

∆ Budget

GSM8K (within-domain) MATH-500 (within-domain) GPQA (within-domain)

64.9% 37.5% 24.0%

59.2% 34.5% 22.4%

+5.7 pp +3.0 pp +1.6 pp

Cross→GPQA

22.9%

24.2%

−1.2 pp

Setting

work should explore “budget-forcing” techniques [5] that encourage models to produce complete answers within the budget, rather than simply truncating. Limitations. Our study has a few limitations. (i) We evaluate only four models; expanding to a broader set would strengthen the generality of our findings. (ii) Our token budgets are logarithmically spaced; finer-grained budgets might reveal additional structure. (iii) Our routing features are limited to surface-level text properties and static embeddings; richer features could capture more of the oracle gap. (iv) We focus on reasoning benchmarks with clear correct answers; generalization to open-ended tasks remains untested. (v) The 19 pp gap between our router and the oracle on GPQA suggests that most model complementarity remains unexploited, which is a compelling direction for future work. (vi) We deliberately exclude dedicated reasoning models (o1, DeepSeek-R1, QwQ) because their dual-stream architecture, allocating tokens between internal “thinking” and visible output, changes the semantics of max_tokens. Controlling max_tokens restricts visible output but may not constrain internal deliberation, making budget comparisons non-equivalent. Extending our framework to reasoning models is an important direction for future work.

Accuracy Change: Full Router No Budget (pp)

+5.71pp

Effect of Budget Features on Routing Accuracy: Within-Domain (blue) vs. Cross-Domain (pink)

5 4

+3.00pp

+1.59pp

2 1 0 1 GSM8K

MATH-500

GPQA

-1.23pp

Cross-Domain ( GPQA)

Figure 7: Budget feature impact across evaluation settings. Withindomain, budget is the most valuable feature (+1.6 to +5.7 pp). Cross-domain, budget features overfit to the training domain and hurt transfer (−1.2 pp). This contrast reveals that budget–accuracy patterns are domain-specific.

Conclusion

We have shown that the evaluation landscape of LLMs is fundamentally budgetdependent. Model rankings, item difficulty, and model complementarity all vary as a function of the token generation budget. These findings have three actionable implications: (1) benchmarks should adopt budget-conditioned evaluation protocols, reporting rankings at multiple budget levels; (2) model selection and routing systems should incorporate budget as a first-class signal; and (3) the substantial oracle gaps we document (+3.5 to +27.8 pp) represent a concrete opportunity for ensemble and routing methods to exploit model complementarity. The question “which model is best?” has no single answer, it depends on how long you let them think.

References [1] Open LLM Leaderboard. https://huggingface.co/spaces/ open-llm-leaderboard/open_llm_leaderboard, 2024. [2] L. Zheng, W.-L. Chiang, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS, 2023. [3] C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Model Parameters. arXiv:2408.03314, 2024. [4] N. Muennighoff, Z. Yang, et al. arXiv:2501.19393, 2025.

s1:

Simple Test-Time Scaling.

[5] S. Aggarwal, Y. Arora, and A. Goyal. L1: Controlling How Long A

Reasoning Model Thinks With Reinforcement Learning. arXiv:2503.04697, 2025. [6] J. Wei, X. Wang, D. Schuurmans, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS, 2022. [7] X. Chen, Z. Xu, et al. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. arXiv:2412.21187, 2024. [8] Y. Sui, H. Yu, et al. Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models. arXiv:2503.16419, 2025. [9] D. Jiang, X. Ren, and B. Y. Lin. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. In ACL, 2023. [10] K. Lu, H. Yuan, et al. Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models. In NAACL, 2024. [11] T. Shnitzer, A. Ou, et al. Large Language Model Routing with Benchmark Datasets. arXiv:2309.15789, 2023. [12] O. Sainz, J. Campos, et al. NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for each Benchmark. In EMNLP Findings, 2023. [13] D. Kiela, M. Bartolo, et al. Dynabench: Rethinking Benchmarking in NLP. In NAACL, 2021. [14] P. Liang, R. Bommasani, et al. Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences, 2023. [15] Meta AI. The LLaMA 3 Herd of Models. arXiv:2407.21783, 2024. [16] Qwen Team. Qwen3 Technical Report. arXiv:2505.09388, 2025. [17] K. Cobbe, V. Kosaraju, et al. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168, 2021. [18] D. Hendrycks, C. Burns, et al. Measuring Mathematical Problem Solving With the MATH Dataset. In NeurIPS, 2021. [19] D. Rein, B. L. Hou, et al. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In ICLR, 2024.

Additional Experimental Details

Model details. All models are evaluated via their standard chat/instruct variants through a unified API. Table 5 provides full identification for all models. The models span a deliberate range of architectures and parameter counts to test whether budget-dependence is a universal phenomenon or model-specific. Temperature is set to 0.0 for all models to ensure deterministic outputs and exact reproducibility. Table 5: Full model identification. Paper Name

Model ID

Provider(s)

Params

LLaMA-3 8B Qwen-3 32B LLaMA-3.3 70B GPT-OSS 20B

meta-llama/Llama-3.1-8B-Instruct Qwen3-32B meta-llama/Llama-3.3-70B-Instruct-Turbo openai/gpt-oss-20b

Groq, Cerebras, SambaNova Groq, SambaNova Together.ai Together.ai

8B 32B 70B 20B

Compute resources. Inference was performed using cloud API endpoints. Total inference count: 56,476 individual API calls (4 models × 2,017 items × 7 budgets). Routing experiments used scikit-learn and XGBoost on a single CPU. Sentence embeddings were computed with all-MiniLM-L6-v2 (22M parameters).

Full Accuracy Tables

Table 6 shows accuracy and 95% bootstrap confidence intervals for all model– budget combinations on GSM8K. Table 6: Full accuracy with 95% bootstrap CIs on GSM8K. Model

b=64

b=256

b=512

b=1024

b=2048

b=4096

LLaMA-3 8B Qwen-3 32B LLaMA-3.3 70B GPT-OSS 20B

1.5±0.7 1.6±0.7 1.2±0.6 0.1±0.1

41.7±2.6 7.7±1.5 62.4±2.6 48.4±2.7

67.9±2.5 35.3±2.6 76.4±2.3 79.3±2.2

68.7±2.8 79.2±2.2 76.3±2.3 91.6±1.5

68.1 90.4 77.6 93.9

67.0±2.6 91.5±1.5 77.6±2.3 94.8±1.2

Truncation Analysis Truncation Rate (%) by Model and Token Budget

Token Budget

100 80 60 40 20 0

GPT-OSS 20B 100 99 98 88 68 49 39

Token Budget

LLaMA-3.3 70B 100 100 100 87 21

Qwen-3 32B 100 100 100 99 94 81 60

GPT-OSS 20B 100 96 87 62 40 22 10

LLaMA-3 8B 100 100 92 47 17 13

LLaMA-3.3 70B 100 98 77 42 18

100 80 60 40 20 0

Qwen-3 32B 100 100 100 100 90 58 31

Token Budget

GPQA

LLaMA-3 8B 100 98 68 25 11

GPT-OSS 20B 100 83 52 19

LLaMA-3.3 70B 100 93 22

100 80 60 40 20 0

MATH-500

Qwen-3 32B 100 100 100 74 24

LLaMA-3 8B 100 99 49

GSM8K

Figure 8: Truncation rate by model, budget, and dataset. Qwen-3 32B experiences the highest truncation rates across all budgets, while LLaMA-3.3 70B shows the fastest decline. At b=4096, truncation rates are near zero for all models except Qwen-3 32B (15% on GSM8K, higher on MATH-500 and GPQA).

Non-Monotone Transition Analysis Non-Monotone Drop Location (% of non-monotone items)

23 51

Budget Transition

24 22 39

GPT-OSS 20B

22 11 67

GPT-OSS 20B

10 24 10 24 20 48 20 48 40 96

LLaMA-3.3 70B

17 28 49

11 22 28 33

51 2

Qwen-3 32B

10 10 24 22 31

12 8

LLaMA-3.3 70B

44 56

LLaMA-3 8B

25 6

GPQA 50 40 30 20 10 0

12 8

51 2

25 6

12 8

Budget Transition

10 24 10 24 20 48 20 48 40 96

15 21 35 23

Qwen-3 32B

43 32 11 14

51 2

GPT-OSS 20B

24 10 24 20 48 20 48 40 96

17 28 12 17

12 8

LLaMA-3.3 70B

12 10 22 16 16 24

12 8

Qwen-3 32B

LLaMA-3 8B

39 37

25 6

12 8

MATH-500

LLaMA-3 8B

GSM8K

Budget Transition

Figure 9: Distribution of non-monotone transitions by budget level. The heatmap shows at which budget level non-monotone items first lose a previously correct answer. Most transitions are concentrated at high budgets (b ≥ 1024), especially for GSM8K.

SHAP Feature Importance Feature Importance for Budget-Aware Routing Log (Budget) Has Latex Emb PC0 Emb PC1 # Special Chars Emb PC7 Emb PC13 Emb PC2 Emb PC8 Emb PC12 # Numbers Emb PC9 Max Number Log Emb PC4 Emb PC15 Emb PC3 Emb PC6 Word Entropy Emb PC17 Emb PC5

Budget Embedding Text stats

0.0

0.5

1.0

1.5

2.0

Mean |SHAP value| (averaged over 4 per-model classifiers)

Figure 10: SHAP feature importance for the per-model scoring router. Budget (log2 b) dominates with a mean absolute SHAP value of 2.21, approximately 6× larger than the next feature (presence of LaTeX: 0.36). Text features and embeddings contribute modestly.

IRT Difficulty Analysis Item Difficulty Spectrum (IRT-Inspired) GSM8K

Number of Items

140

MATH-500

Mean=0.51 Median=0.56

120

GPQA Mean=0.25 Median=0.25

Mean=0.17 Median=0.14

100 80 60 40 20 0

0.0

0.1

0.2

0.3

0.4

0.5

Easiness Score

0.6

0.7

0.8

0.0

0.1

0.2

0.3

0.4

0.5

Easiness Score

0.6

0.7

0.8

0.0

0.1

0.2

0.3

0.4

Easiness Score

0.5

0.6

Figure 11: IRT-inspired difficulty distribution across benchmarks. Easiness is computed as the fraction of model–budget combinations that answer each item correctly. GSM8K items cluster around easiness 0.5 (mean: 0.507), MATH-500 around 0.25 (mean: 0.253), and GPQA around 0.17 (mean: 0.172), confirming the intended difficulty gradient.

Genuine Overthinking Examples

To confirm that non-monotone behavior is not merely a truncation artifact, we identify items where a model answers correctly at a low budget bl but incorrectly at a higher budget bh > bl , with finish_reason=stop at both budgets (i.e., 17

neither response was truncated). Table 7 reports the number of unique items exhibiting this pattern per model and dataset. Across the four models and three benchmarks, we find 1,193 such (low-budget, high-budget) pairs—concrete evidence that overthinking is a genuine reasoning failure, not an artifact of output truncation. LLaMA-3 8B exhibits the most overthinking instances (550 on GSM8K alone), consistent with its high non-monotone rate in the behavioral taxonomy (Section 4.1). Qwen-3 32B shows the fewest (65 on GSM8K, 4 on GPQA), despite its high truncation rate, suggesting that its non-monotone behavior is more often truncation-driven. Table 7: Genuine overthinking instances (correct→wrong, both with finish_reason=stop). Counts show the number of unique items where a model answers correctly at a lower budget but incorrectly at a higher budget, with no truncation at either level. Model LLaMA-3 8B Qwen-3 32B LLaMA-3.3 70B GPT-OSS 20B

GSM8K

MATH-500

GPQA

550 65 159 46

88 9 93 52

74 4 45 8

Prompt Templates

We use the following prompt templates, adapted per benchmark. For GSM8K and MATH-500 (numerical answer): Solve the following problem step by step. Problem:

Show your complete reasoning.

{question}

After your reasoning, provide your final answer on a new line in the exact format: #### [your answer]

For GPQA-Diamond (multiple choice): Answer the following question by reasoning step by step. Question:

{question}

Options: (A) {option_a} (B) {option_b} (C) {option_c} (D) {option_d} After your reasoning, provide your final answer on a new line in the exact format: #### [A/B/C/D]

Answer parsing. For numerical benchmarks (GSM8K, MATH-500), we parse the #### [answer] pattern and compare numerically with tolerance 10−6 . Format variations (e.g., 12 vs. 0.5) are handled via float conversion with LaTeX-aware parsing. For GPQA, we extract a single letter (A–D) using a cascade of regex patterns applied to the model output.

Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

Analyze & explore related research in Conceptio

Related documents