Conceptio › Archive › arXiv CS

arXiv CSopen access

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

knowledge-representation reasoning

artificial intelligence, reasoning, knowledge representation

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Wenbo Zhang * 1 Lijinghua Zhang * 1 Liner Xiang * 1 Hengrui Cai 1

arXiv:2605.10805v1 [cs.AI] 11 May 2026

Abstract Reasoning-capable large language models (LLMs) have recently been adopted as automated judges, but their benefits and costs in LLMas-a-Judge settings remain unclear. Through controlled comparisons between reasoning and non-reasoning judges, we show that explicit reasoning substantially improves judgment accuracy on tasks requiring structured verification (e.g., math and coding), while offering limited or even negative gains on simpler evaluations and incurring significantly higher computational cost. These findings motivate that reasoning should be used selectively rather than universally, with awareness of possible distribution shift. We propose a Robust Adaptive Cost-Efficient Routing (RACER), which dynamically selects between reasoning and non-reasoning judges under a fixed budget by formulating routing as a constrained distributionally robust optimization problem. RACER explicitly accounts for distribution shift via a KL-divergence uncertainty set, admits an efficient primal–dual algorithm, and enjoys theoretical guarantees including uniqueness of the optimal policy and linear convergence. Extensive experiments show that RACER achieves superior accuracy–cost trade-offs under distribution shift.

(a) Accuracy-cost trade-off

(b) Robust routing under OOD

Figure 1. (a) Reasoning models outperform non-reasoning models on difficult tasks, achieving higher accuracy at the cost of increased computation, while offering only marginal gains on simple tasks. (b) RACER remains robust to out-of-distribution (OOD) inputs by operating over a KL-divergence uncertainty set, whereas standard routing policies fail under distribution shift.

are used as evaluators. In parallel, reasoning-capable LLMs (OpenAI, 2024; Guo et al., 2025) have attracted significant attention. A growing body of work demonstrates that explicitly training models to generate long-form reasoning chains prior to producing final answers substantially improves performance across a wide range of problem-solving tasks, including mathematics, code generation, instruction following, and agentic decision-making (Yang et al., 2025; Liu et al., 2025; Yu et al., 2025; Team et al., 2025). While these findings provide valuable insights into the role of reasoning in LLMs as problem solvers, they do not necessarily imply that such models are effective judges. In current frontier open-source models, reasoning capabilities are typically acquired via training in verifiable problem-solving tasks, whereas the judging ability is not explicitly optimized (Yang et al., 2025; Team et al., 2025; Olmo et al., 2025; Bercovich et al., 2025). This raises a crucial question: Can the reasoning skills learned from problem-solving tasks be effectively transferred to judgment tasks?

1. Introduction As large language models (LLMs) continue to advance, reliably evaluating the quality of their outputs has become increasingly important but challenging. Owing to the high cost and limited scalability of human evaluation, the LLMas-a-Judge paradigm has emerged as a promising alternative (Zheng et al., 2023; Li et al., 2023; Liu et al., 2023; Kim et al., 2023; Fu et al., 2024), in which LLMs themselves

Even if such transfer is possible, deploying reasoning-based judgment models introduces additional challenges. As shown in Fig. 1a, reasoning-based judgments incur substantially higher computational cost due to multi-step deliberation and are not universally beneficial, as they may induce overthinking on simple queries and lead to incorrect

Department of Statistics, University of California, Irvine, USA. Correspondence to: Hengrui Cai <[email protected]>. Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

decisions (Chen et al., 2024b; Sui et al., 2025). A natural mitigation strategy is to learn a routing model that selectively invokes reasoning or non-reasoning models based on criteria such as expected performance or inference cost (Chen et al., 2023; Frick et al., 2025; Liang et al., 2025; Zhang et al., 2025a). However, existing routing approaches largely overlook the issues of distribution shift. Routers are typically trained on static datasets, whereas real-world queries evolve due to user heterogeneity (Zhang et al., 2024; Chakraborty et al., 2024; Son et al., 2024). Consequently, a router trained under a fixed data distribution may degrade significantly at deployment time, resulting in cost constraint violations or erroneous selections of the judging function.

reasoning judges and characterize the resulting accuracy– cost trade-offs as well as agreement patterns across benchmarks and model sizes. Experimental Setup. To isolate the effect of explicit reasoning, we consider paired reasoning and non-reasoning judges {MR , Mno-R } and perform controlled comparisons within each pair. By matching the size, architecture, and overall capability of the model, performance differences can be primarily attributed to the presence or absence of reasoning. Concretely, for hybrid models that support both reasoning and non-reasoning inference, we instantiate the pair by treating the reasoning mode as MR and the non-reasoning mode as Mno-R . We evaluate multiple open-source hybrid models from the Qwen3 family (1.7B/4B/8B), as well as reasoning and non-reasoning judges on three widely used LLM-as-a-Judge benchmarks: J UDGE B ENCH (Tan et al., 2024a), R EWARD B ENCH (Lambert et al., 2025), and R E WARD B ENCH 2 (Malik et al., 2025), all of which consist of tasks from various domains. Each example provides a prompt, two candidate responses, and a preference label.

To address these challenges, we propose the Robust Adaptive Cost-Efficient Routing (RACER), which formulates router learning as a distributionally robust, constrained policy optimization problem, illustrated in Fig. 1b. The distributionally robust learning framework has recently been used to address the issue of distribution shift in various settings (Namkoong & Duchi, 2016; Rahimian & Mehrotra, 2019; Duchi et al., 2021). This formulation explicitly accounts for distribution shift by optimizing over an uncertainty set of data distributions centered around training data. By solving the resulting robust objective, RACER seeks policies that remain reliable under distribution shift, rather than optimizing solely for average-case performance.

Evaluation Metrics. For evaluation, we prompt the LLM judge to compare the two responses and to provide its preferred choice. We compute judge accuracy as the fraction of examples for which the judge’s preference matches the ground-truth label, and we record cost as token consumption during inference. We define ∆Accuracy as the difference in judgment accuracy between reasoning and non-reasoning modes, ∆Accuracy= Acc(MR ) − Acc(Mno-R ), and define the cost ratio as the ratio of token consumption under reasoning versus non-reasoning inference.

Our contributions are summarized as follows: • We present a comprehensive study comparing reasoning and non-reasoning modes of LLMs in LLM-as-a-Judge settings, characterizing when explicit reasoning improves judgment quality and when it fails, and providing practical insights into the effective use of reasoning for evaluation. • We formulate router learning in a unified mathematical and algorithmic framework that jointly addresses the cost– performance trade-off and robustness to distribution shift via constrained distributionally robust optimization, followed by a tractable algorithm for efficient solution. • We are the first in LLM routing to prove that the optimal router policy is unique and our policy iterates converge to it at a linear rate, i.e., the error contracts by a constant factor at each iteration (Wei et al., 2021; Ding et al., 2023). This provides strong theoretical support for the proposed method. • Extensive experiments demonstrate that RACER consistently outperforms baseline methods under distribution shift, achieving improved accuracy–cost trade-offs.

Results. Fig. 2 (upper) plots ∆Accuracy against the cost ratio, where points closer to the upper-left indicate larger gains at lower additional cost. Across all three benchmarks, the strongest and most cost-effective gains concentrate in math and coding, whereas safety and knowledge show limited improvements and can even degrade under reasoning. Fig. 2 (lower) summarizes agreement patterns between reasoning and non-reasoning judges. When both modes reach the same outcome, non-reasoning is preferable due to its lower cost; when they disagree, selecting the better judge provides additional headroom. This headroom shrinks with model size, as larger non-reasoning judges already achieve stronger baseline judgment accuracy. Overall, reasoning judge yields significant performance gains, especially in reasoning-intensive domains, illustrating that skills learned from problem-solving tasks effectively transfer to judgment tasks. Also, we notice that the reasoning should be used selectively, given its substantial cost.

2. Does Reasoning Help LLM-as-a-Judge? 2.1. Accuracy–Cost Trade-offs of Reasoning Judges

Moreover, the heterogeneous accuracy–cost trade-offs in Fig. 2 suggest that both reward and compute cost can shift across domains and benchmarks. As a result, a router tuned

We study whether, and under what conditions, explicit reasoning improves LLM-as-a-Judge. To this end, we perform controlled comparisons between reasoning and non2

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Figure 2. Accuracy–cost trade-offs and reasoning–instructional agreement across benchmarks. Upper: Accuracy improvement versus cost ratio. Lower: Agreement patterns between instruct and reasoning inference.

on in-distribution data may mis-estimate either the benefit of reasoning or the risk of budget violation under distribution shift. This motivates our distributionally robust objective in Eq. (4), which applies robustness separately to the reward and cost terms to hedge against these two failure modes.

Taken together, these case-level patterns help explain the heterogeneous accuracy–cost trade-offs observed in Section 2.1. They provide qualitative evidence that reasoning improves judgment primarily when its inductive bias aligns with the evaluation structure of the task, reinforcing the conclusion that reasoning should be activated selectively rather than unconditionally under computing constraints.

2.2. When and Why Reasoning Helps To better understand when and why reasoning improves LLM-as-a-Judge, we conduct a case analysis. This analysis reveals three recurring patterns: (i) reasoning improves judgment when evaluation requires explicit verification; (ii) reasoning is largely redundant when the correct choice is apparent from surface cues; and (iii) reasoning can hurt when it over-expands the evaluation scope and introduces irrelevant considerations. Representative examples are summarized in Table C.1 in the Appendix.

3. RACER: Robust Adaptive Cost-Efficient Routing 3.1. Notation We denote the prompt space by X , the response space by Y. Let Z = X × Y × Y. We use ρ ∈ ∆(Z × {0, 1}) to denote the distributions over pairwise prompt–response tuples with preference labels where ∆(·) is a probability simplex. For a given pairwise prompt–response tuple (x, y1 , y2 ), the label l = 1 indicates that y1 is preferred to y2 , denoted as y1 ≻ y2 , while l = 0 indicates that y2 ≻ y1 . We define the judge function of the model a as Φa : Z 7→ {0, 1}, which maps z = (x, y1 , y2 ) to a binary preference label, indicating whether y1 ≻ y2 (Φa (z) = 1) or y2 ≻ y1 (Φa (z) = 0). Here we assume deterministic judgments for simple cases, but the framework readily extends to stochastic settings. We denote the P empirical distribution of the dataset with size n n as ρn = n1 i=1 δ(zi ,li ) , where (zi , li ) ∼ ρ.

In math and coding tasks, where correct evaluation often requires checking intermediate steps, validating internal consistency, or reconciling multiple criteria, reasoning-mode judges are better aligned with these requirements. In contrast, for tasks such as factual recall and concise question answering, reasoning and non-reasoning judges usually agree, leaving little room for improvement. Finally, we observe failure cases where explicit reasoning introduces irrelevant factors, leading to degraded judgments.

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

3.2. Proposed Method: RACER

Then the Lagrangian min-max problem associated with (3) is given by:

Motivated by the accuracy–cost trade-offs of reasoning judges, we propose to learn a router that selectively activates reasoning modes. We formulate router learning as a constrained optimization problem. We define router policy π : Z → ∆(0, 1), and denote the sampled decision from π(· | z) as a taking a value of 1 if choosing the reasoning mode and 0 if choosing the non-reasoning mode. Our optimization objective in terms of the budget C is given by:

max min L(π, λ) := max min RU (ρn ,δ) (π) − λCU (ρn ,δ) (π) , π∈Π λ≥0

where λ is the Lagrange multiplier. While this formulation is solvable through alternating gradient descent methods, it is computationally challenging since we do not have direct control over the data distribution ρ̃ ∈ U(ρn , δ) as they are not parameterized distributions. Moreover, the training data are sampled only from the source distribution ρ, without samples from other distributions in the uncertainty set U(ρ, δ). To overcome this challenge, we introduce principled tractable algorithms to solve this problem.

max E(z,l)∼ρn ,a∼π(·|z) [r(z, a, l)], π∈Π

s.t. E(z,l)∼ρn ,a∼π(·|x) [c(z, a)] ≤ C,

(1)

where r(z, a, l) = I(Φa (z) = l) with indicator function I(·) and c(z, a) denotes whether the judgment is correct and the cost of using the selected mode, respectively, and Π is the policy class of interest. We call this method Adaptive Cost-Efficient Routing (ACER). The core idea here is that the routing policy should maximize judgment reward while controlling cost within a budget, yielding an adaptive accuracy–cost trade-off.

The following proposition shows that the worst-case probability distribution within a KL uncertainty set can be efficiently approximated. Similar ideas have appeared in prior works on distributionally robust reinforcement learning (Gadot et al., 2024; Xu et al., 2025). Theorem 3.1. Let ρn ∈ ∆n be the empirical distribution over {zi , li }ni=1 , and define fi := Ea∼π(·|zi ) [f (zi , a)]. Let

However, ACER is not explicitly designed to handle distribution shifts, which are common in real-world scenarios. To address this limitation, we further propose Robust Adaptive Cost-Efficient Routing (RACER), which not only optimizes performance under cost constraints, but also remains robust to distribution shifts through distributionally robust learning.

n n o X ρ̃(i) U(ρn , δ) := ρ̃ ∈ ∆n : ρ̃(i) log ≤δ . ρn (i) i=1

Define ρ and ρ as solutions to ρ ∈ arg

We reformulate the problem as follows: max

min

E(z,l)∼ρ̃,a∼π(·|z) [r(z, a, l)],

s.t.

max

E(z,l)∼ρ̃,a∼π(·|z) [c (z, a)] ≤ C.

π∈Π ρ̃∈U(ρn ,δ) ρ̃∈U(ρn ,δ)

with s ≤

ρ̃(i)fi , ρ ∈ arg

i=1

n X

max ρ̃∈U (ρn ,δ)

ρ̃(i)fi .

i=1

i=1 ρn (i)fi and s ≥

i=1 ρn (i)fi .

We leave the proof of Theorem 3.1 to the Appendix A.1. Based on this Theorem, we get a closed-form solution of ρ, ρ over KL uncertainty set U (ρn , δ). The resulting distributions ρ and ρ are obtained by reweighting the data. In the minimization setting, where f is a reward function, ρ downweights samples whose expected reward exceeds the baseline s and upweights those with lower-than-baseline rewards. In the maximization setting, where f is a cost function, ρ upweights samples with higher-than-baseline cost, focusing optimization on high-risk regions. The parameter τ controls the strength of reweighting: lower τ induces a more aggressive concentration on extreme (worst-case) samples.

ρ̃∈U (ρn ,δ)

Hence, the original problem (2) can be rewritten as: s.t. CU (ρn ,δ) (π) ≤ C.

n X

Then there exist s, s ∈ R and τ > 0 such that, for all i, s − fi fi − s ρ(i) ∝ ρn (i) exp , ρ(i) ∝ ρn (i) exp , τ τ

(2)

We define the worst-case reward RU (ρn ,δ) (π) and the worstcase cost CU (ρn ,δ) (π) as RU (ρn ,δ) (π) := min E(z,l)∼ρ̃, a∼π(·|z) r(z, a, l) , ρ̃∈U (ρn ,δ) CU (ρn ,δ) (π) := max E(z,l)∼ρ̃, a∼π(·|z) c(z, a) .

π∈Π

min ρ̃∈U (ρn ,δ)

This optimization is over a distributional uncertainty set within a distance of at most δ with ρn . Here, we utilize the uncertainty set U(ρn , δ) = {ρ̃ : DKL (ρ̃∥ρn ) ≤ δ} and DKL to denote the KL-divergence. Overall, RACER seeks to maximize the worst-case reward under a worst-case cost constraint, explicitly incorporating distributional robustness to prompt and response shifts, an aspect largely overlooked by existing routing methods. We now turn to solving (2).

max RU (ρn ,δ) (π)

π∈Π λ≥0

Since the baselines s and s are generally unknown, in practice we approximate them using the empirical mean, which serves as a valid surrogate by providing upper and lower

(3) 4

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

4. Theoretical Results

Algorithm 1 RACER input: Number of iterations T , temperature τ , regularization parameter β, distribution ρ for preference data generation, judging functions (Φ0 , Φ1 ) by an LLM. 1: Initialize π0 and λ0 . 2: for Iteration t = 0, 1, . . . , T − 1 do 3: Construct Dt = {(z, l)} where (z, l) ∼ ρ. 4: Sample routing actions: a ∼ πt (· | z), ∀z ∈ Dt (optional: enumerate all actions). 5: Calculate reward and cost: For each (z, l) ∈ Dt and routing choice a, get judge results Φa (z), and calculate the reward r(z, a, l) and cost c(z, a). Data reweighting: Calculate batch mean reward 6: P|Dt |

r(zj ,aj ,lj )

P|Dt |

In this section, we first show that the optimization problem admits a unique solution, and then demonstrate linear convergence in terms of the KL divergence between the last iterate and the optimal router policy. While recent works study routing strategies for LLMs with a focus on practical model selection and performance–cost tradeoffs (Ong et al., 2025; Liang et al., 2025; Zhang et al., 2025a), these approaches are primarily empirical and heuristic in nature. To the best of our knowledge, this work is the first to provide theoretical guarantees on the convergence behavior of an LLM routing policy.

c(zj ,aj )

j=1 j=1 r = and cost c = , |Dt | |Dt | then approximate worse-case distributions: r − ri ci − c ρ(i) ∝ exp , ρ̄(i) ∝ exp . τ τ

4.1. Uniqueness of the Optimal Router Policy The policy optimization problem can be interpreted as finding a saddle point of the following max–min problem, max min Lβ (π, λ).

Primal-dual update: Obtain πt+1 and λt+1 by (6) and (7). 8: end for 9: return The best valid π0:T on the validation data. 7:

After establishing feasibility (i.e., the constraint set is nonempty) and showing that the solution space is bounded, we then prove that the saddle point exists and is unique, as stated in Theorem 4.1. All proofs are provided in Appendix A.2.

bounds for s and s, respectively. Thus, the Lagrangian min-max problem of (3) can be transformed as:

Theorem 4.1 (Existence and Uniqueness of the Saddle Point). There exists a unique pair (π ∗ , λ∗ ) ∈ Π × Λ such that Lβ (π, λ∗ ) ≤ Lβ (π ∗ , λ∗ ) ≤ Lβ (π ∗ , λ) for any π ∈ Π and λ ∈ Λ, that is, (π ∗ , λ∗ ) is the saddle point of Lβ (π, λ).

max min L(π, λ) = max min RU (ρn ,δ) (π) − λCU (ρn ,δ) (π) π∈Π λ≥0

π∈Π λ≥0

= max min Rρ (π) − λCρ (π) . π∈Π λ≥0

(8)

π∈Π λ≥0

(4)

As a consequence, for any (π, λ) ∈ Π × Λ,

where the last equality is implied by Theorem 3.1. We introduce a regularized Lagrangian: 1 2 Lβ (π, λ) := L(π, λ) + β H(π) + λ , 2

L(π, λ∗ ) − βH(π) ≤ L(π ∗ , λ∗ ) ≤ L(π ∗ , λ) +

which indicates that (π ∗ , λ∗ ) is a saddle point of the original Lagrangian L(π, λ), up to two β-regularization terms.

(5)

Theorem 4.1 identifies a unique target for the primal–dual iterates. In the next subsection, we show that the updates (6) and (7) converge to this unique saddle point.

where H(π)+ 12 λ2 is added as a regularization to the original Lagrangian L(π, λ). Here β is a regularization parameter, and H(π) := E(z,l)∼ρ [H(π(· | z)] is the entropy of the policy π. This step allows us to control the randomness of the routing policy, thereby encouraging exploration and facilitating convergence (Cen et al., 2022; Ding et al., 2023). We can then use the primal-dual method to solve (5): n o πt+1 = argmax Rρ (π) − λt Cρ̄ (π) + βH(π) , π∈Π 1 λt+1 = argmax −λt Cρ̄ (π) + βλ2t . 2 λ≥0

β 2 λ , 2

4.2. Convergence of the Router Policy We next analyze the last-iterate convergence of our router policy πt to the optimal policy π ∗ . To quantify the convergence, we measure the distance between πt and π ∗ under the distribution ρ using the KL divergence, defined as

(6)

KL(πt ∥π ∗ ) = E(z,l)∼ρ KL(πt (·|z)∥π ∗ (·|z)), (7) where

Based on the derivations, we propose a practical RACER algorithm in Algorithm 1.

KL(πt (·|z)∥π ∗ (·|z)) =

1 X a=0

πt (a|z) log

πt (a|z) π ∗ (a|z)

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge Table 1. Dataset statistics and cost ratios.

To derive a closed-form solution for (6), we rewrite it as

Subset

pρ pρ r(z, a, l) − λt c(z, a) π∈Π pρ pρ + β E(z,l)∼ρ [H (π(· | z))] , (9)

πt+1 = arg maxE(z,l)∼ρ,a∼π(·|z)

Magpie Ultra WildGuardMix OffsetBias

0.720

Reward

We then propose Assumptions 1, 2, and 3 to establish the convergence result in Theorem 4.2.

0.745

RACER-R (τ=0.1) RACER-R (τ=1.0) RACER-C (τ=0.1) RACER-C (τ=1.0) ACER Invalid

0.740

0.730 1.5

2.0

1.0

1.5

2.0

2.5

Cost

(a) Trained on Magpie Ultra and (b) Trained on WildGuardMix evaluated on OffsetBias, exhibit- and evaluated on OffsetBias, exing a shift toward lower-cost hibiting a shift toward higherqueries. cost queries. Figure 3. Reward–cost trade-offs evaluated on the OOD dataset with a budget of 2. (a) shows that reward-robust RACER-R improves performance when the cost constraint is easily satisfied. (b) illustrates the importance of cost-robust RACER-C when the cost constraint can be violated under distribution shift.

first focuses on controlled ablations designed to isolate the roles of reward and cost robustness under targeted distribution shifts. The second evaluates end-to-end routing performance on real world LLM-as-a-Judge benchmarks to assess generalization under more realistic evaluation distributions. Across both experimental settings, we use a unified data generation protocol to learn a router from the preference data. Given any preference dataset consisting of a prompt and a response pair (xi , yi,1 , yi,2 ), we prompt an LLM judge under two inference modes (instruct and reasoning) to select its preferred response. For each instance, we record (i) whether the judge’s decision matches the ground-truth preference label, ri , and (ii) the number of tokens consumed under each inference mode, ci . The router takes as input a text embedding of the concatenated context zi = (xi , yi,1 , yi,2 ) obtained from an embedding model.

Theorem 4.2 (Linear Convergence of RACER). Under Assumptions 1, 2, and 3, the router policy iterates satisfy 2t

11.2 3.4 4.7

RACER-R (τ=0.1) RACER-R (τ=1.0) RACER-C (τ=0.1) RACER-C (τ=1.0) ACER Invalid

Cost

Assumption 1 is standard and has been widely adopted in convex optimization and policy optimization analyses (see, e.g., Mutti et al., 2023; Ding et al., 2023). Assumption 2 ensures that the cost c(z, a) is bounded and has bounded variance, while Assumption 3 controls the amplification induced by importance reweighting. Together, these conditions imply that the dual gradient term (pρ /pρ ) c(z, a) is uniformly bounded. This uniform bound allows us to control how much the policy πt+1 responds to changes in the dual variable λt , which is a key ingredient in establishing the linear convergence result below.

M 2K 2 M 2 K 2 + 2β 2

27,785 6,709 8,504

0.715

1.0

Assumption 3 (Boundness of density ratio). There exists a constant K > 0, such that pρ /pρ ≤ K for all z ∈ Z.

Train 1 Train 2 OOD Test

0.710

Assumption 2 (Boundness of c). There exists a constant M > 0, such that c(z, a) ≤ M for all z ∼ ρ and a ∈ {0, 1}.

M 2K 2 KL(πt ∥π ) ≤ 2β 2

Cost Ratio

0.735

Assumption 1 (Convexity of the policy class Π). The interested policy class Π is convex, i.e., for any π1 , π2 ∈ Π and any α ∈ [0, 1], απ1 + (1 − α)π2 ∈ Π.

∗

# Pairs

Reward

where pρ , pρ , and pρ denote the densities of the distributions ρ, ρ, and ρ, respectively.

Split

(λ0 − λ∗ )2 .

Theorem 4.2 establishes that the iterates πt converge to the optimal policy π ∗ at a linear rate, depending on M , K, and β. Details and the proof are deferred to Appendix A.3. Importantly, our convergence results can be extended to the parameterized (non-convex) setting under additional assumptions on the approximation quality of the policy class Π. In particular, if each policy update approximates the optimal entropy-regularized solution up to a controlled error, then the last-iterate convergence result continues to hold up to an additional approximation error term, as commonly studied in prior work (Ding et al., 2023; Zhan et al., 2023).

Details of the judging prompts, decoding configurations, and the embedding model are provided in Appendix B. All training and evaluation sets in the following experiments are constructed using this protocol, differing only in the underlying dataset. 5.2. Ablating Reward and Cost Robustness

5. Experiments

As shown in our robust objective (3), the proposed formulation incorporates two levels of robustness: robustness applied to the reward, denoted as RU (ρn ,δ) (π), and robustness applied to the cost, denoted as CU (ρn ,δ) (π). To better understand the contribution of each component, we conduct

5.1. Experimental Setup We conduct two complementary sets of experiments with a shared training and judgment generation protocol. The 6

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Figure 4. Routing performance across compute budgets on ID and OOD benchmarks (top to bottom: Qwen3-1.7B, 4B, and 8B).

an ablation study to investigate (i) when reward robustness is necessary and (ii) when cost robustness is necessary. Specifically, we introduce two variants, RACER-R and RACER-C, which apply distributionally robust reweighting only to the reward and only to the cost, respectively. We also include ACER with the non-robust objective in (1) as a baseline.

maintains OOD costs within budget, demonstrating that cost robustness provides a safer strategy when in-distribution cost is low but OOD shifts increase violation risk. Overall, RACER variants exhibit greater robustness in both reward and cost compared to the non-robust baseline ACER.

We consider two representative scenarios: (a) the OOD queries require less computation than in-distribution queries, and (b) the OOD queries require more computation than in-distribution queries. We construct our datasets from the S KYWORK R EWARD P REFERENCE DATASET (Liu et al., 2024a), with Magpie-Ultra and WildGuardMix used for training in Scenarios (a) and (b), respectively, and OffsetBias as the OOD test set. Judgments are produced by the hybrid reasoning model Qwen3-4B (Yang et al., 2025). Reward is defined as correctness, and cost as the relative token usage between reasoning and non-reasoning judgments (Table 1). We use bge-3 (Chen et al., 2024a) to get embeddings of the query and responses, and a linear router.

5.3. Scaling Evaluation to Standard Benchmarks Training data. Following the data-generation protocol, we construct the training set from (i) 20,000 instances from S KYWORK -R EWARD (Liu et al., 2024a), and (ii) an additional 20,000 instances sampled from M ATH -S TEP -DPO10K (Lai et al., 2024) and C ODE -P REFERENCE -PAIRS (Vezora, 2024). We include the math and code data to improve coverage of reasoning-intensive domains, which are under-represented in S KYWORK -R EWARD. We generate judgments by pairing the reasoning and non-reasoning variants of Qwen3-1.7B/4B/8B (Yang et al., 2025). We also include Llama-3.1-8B (Grattafiori et al., 2024) as an additional model family; the corresponding results, reported in Appendix C.3, exhibit a similar trend, indicating that RACER generalizes across model families.

As shown in the left panel of Figure 3, the OOD shift favors lower-cost queries, all methods satisfy the budget constraint across runs. RACER-R consistently achieves the highest reward, with stronger robustness (lower temperature) yielding further gains, indicating that when OOD cost is low, emphasizing reward robustness enables more effective budget utilization. In contrast, in the right panel, RACER-C attains a lower reward than RACER-R and ACER but consistently

Evaluation datasets. We evaluate the learned routing policy on standard LLM-as-a-Judge benchmarks spanning diverse task domains, including R EWARD B ENCH (Lambert et al., 2025), R EWARD B ENCH -2 (Malik et al., 2025), and J UDGE B ENCH (Tan et al., 2024a). All evaluation benchmarks are held out from training and are used exclusively 7

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge Table 3. Sensitivity of RACER to β under different budgets using data generated by the Qwen3-4B judge pair.

for out-of-distribution evaluation. Judgments are generated using the same model pairs.

Baselines. We compare against three baselines: • All-Instruct, which always selects the instruct mode (cost fixed to 1 and accuracy equal to the instruct judge’s average correctness); • All-Reasoning, which always selects the reasoning mode (accuracy and cost given by the reasoning judge’s average correctness and token consumption, respectively); • Random, which activates the reasoning mode independently for each instance with probability equal to the learned policy’s average reasoning rate.

0 0.005 0.01 0.05

Fig. 4 reports accuracy–cost trade-offs under varying budgets for three model scales. On both the training split and the in-distribution test split, RACER achieves a notably favorable regime: at roughly half of the All-Reasoning cost, it matches and often surpasses the All-Reasoning judge’s accuracy, indicating that reasoning can be concentrated on the subset of instances where it delivers the largest marginal benefit. Under OOD benchmarks, RACER continues to yield consistent improvements over the baselines, demonstrating that the learned selection rule generalizes beyond the training distribution. The Random baseline forms an approximately linear interpolation between the All-Instruct and All-Reasoning endpoints, corresponding to indiscriminate activation of reasoning at a fixed rate. In contrast, RACER traces a concave, higher-accuracy frontier; the area between the RACER curve and the Random curve reflects the additional gain from strategic instance-level selection, which cannot be explained by merely increasing the overall reasoning rate, but instead by allocating reasoning to the right examples under the same budget.

RACER

2.6 3.8 3.4

84.1 84.7 84.3

8B 2.5 3.4 2.7

86.8 88.2 88.9

Acc.

Cost

85.2 85.5 85.5 84.8

1.9 2.0 2.0 2.0

86.7 86.7 86.7 86.0

2.9 2.9 2.9 2.9

86.8 86.7 86.7 86.2

3.7 3.6 3.8 3.8

LLM-as-a-Judge and Reasoning. Human evaluation is considered the gold-standard metric for assessing LLMgenerated content (Ouyang et al., 2022; Zheng et al., 2023). However, it is costly and time-consuming, making it difficult to scale in practice. To address this, LLM-as-a-Judge has been proposed as an automatic proxy (Zheng et al., 2023; Li et al., 2023; Liu et al., 2023; Kim et al., 2023; Fu et al., 2024). One advantage is that the LLM can provide explanations for its final judgments, which facilitates error analysis. To evaluate how well LLM-as-a-Judge truly performs, benchmarks have been introduced to measure the accuracy of judgments across different domains (Tan et al., 2024b; Liu et al., 2024b; Lambert et al., 2025). There are several recent studies have shown that incorporating reasoning into models through reinforcement learning on

Acc. Cost Acc. Cost Acc. Cost RouterBench-KNN 71.3 RouteLLM-MF 69.4 M-IRT 71.6

Cost

6. Related Work

Table 2. Accuracy and cost comparison across model scales with advanced baselines. 4B

C=4

Acc.

Ablation and Sensitivity Analysis of Entropy Regularizer. We conduct an ablation and sensitivity analysis on β (weight for entropy regularization) at 3 representative budget levels, using Qwen3-4B judge pair. As shown in Table 3, removing the entropy regularizer hurts accuracy under the tight budget, confirming its contribution when the constraint is binding. Performance remains stable for β ∈ {0.005, 0.01}, while β = 0.05 consistently degrades accuracy, supporting our default choice of β.

5.4. Additional Results

1.7B

C=3

Cost

pare RACER with three representative adavcned baselines: RouterBench-KNN (Hu et al., 2024), RouteLLM-MF (Ong et al., 2024), and M-IRT (Song et al., 2025), under the same budget constraint (C = 4). The results are averaged on 3 test datasets and over 10 replications . As shown in Table 2, RACER achieves the highest accuracy across all three model families while remaining within the budget. For the 1.7B, 4B, and 8B models, RACER improves over the strongest baseline by 0.64, 1.10, and 1.06 percentage points, respectively. These results suggest that RACER exploits the available budget more effectively by allocating reasoning to instances where it is most beneficial, yielding a stronger reward–cost trade-off than prior routing methods. Since RACER is a general framework, in principle, it can be combined with more advanced routing architectures to improve performance further.

We evaluate the routing policy under compute budgets C ∈ {2, 2.5, 3, 3.5, 4, 5, 7, 10}. For each budget, we repeat training and evaluation 20 times and report the mean and standard deviation of accuracy and realized cost ratio. Our routing policy is parametrized by a four-layer neural network. See Appendix B.3 and B.5 for detailed implementation and hyperparameter choices.

Method

C=2 Acc.

2.6 4.1 3.4

72.2 3.6 85.8 3.4 90.0 3.9

Comparison with other advanced baselines. We also com8

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

judge-specific tasks can further enhance evaluation performance (Whitehouse et al., 2025; Saha et al., 2025; Chen et al., 2025a;b). Notably, our work demonstrates that even without judge-specific training, reasoning abilities acquired from general-domain training can still effectively transfer.

ticularly under distribution shift. To address these problems, we propose RACER, which adaptively activates reasoning judges under a fixed budget. We are also the first to provide a theoretical analysis with linear convergence guarantees, supporting both efficiency and robustness. Empirically, RACER consistently outperforms baselines on multiple out-of-distribution benchmarks, achieving superior accuracy–cost trade-offs.

LLM Routing. As the number of parameters in LLMs continues to grow, their inference cost increases substantially. To address this issue, LLM routing has emerged as an effective strategy that assigns user queries to appropriate LLMs while balancing both performance and cost. Chen et al. (2023) adopted a cascading strategy, sequentially querying LLMs until a satisfactory response is produced. P2L (Frick et al., 2025) employs a prompt-to-regression approach to predict a vector of Bradley–Terry coefficients, which are then used to select the optimal model. Aggarwal et al. (2023) and Ong et al. (2025) trained a binary predictor to switch between a strong and weak model. While effective, this approach requires deploying multiple LLMs simultaneously. Recent studies (Liang et al., 2025; Zhang et al., 2025a) investigate mode switching within a single hybrid reasoning model, which is most closely related to our setting. However, we introduce a principled policy learning framework that not only maximizes performance under budget constraints, but also explicitly addresses distribution shift, an important yet largely overlooked challenge in existing routing systems. Furthermore, all prior studies predominantly focus on question-answering tasks, whereas we present the first comprehensive investigation of routing strategies in the LLM-as-judge domain.

RACER focuses on binary routing with a KL-based uncertainty set; while effective under moderate distribution shifts, overly large uncertainty can induce conservative routing that prioritizes cost safety and underutilizes reasoning. Extending RACER to richer routing schemes and alternative uncertainty sets to better balance robustness and adaptivity is an important direction for future work.

Acknowledgements This work was supported in part by the National Science Foundation under Grant No. DMS-2401271. We thank the anonymous reviewers for their helpful comments.

Impact Statement This work examines the benefits and costs of reasoningcapable LLMs in the LLM-as-a-Judge setting and proposes a robust routing framework that selectively allocates reasoning under budget and distribution shift constraints. A key positive impact is promoting more efficient and sustainable use of computation by showing that reasoning is not universally beneficial and should be applied selectively, while also improving the robustness and reliability of automated evaluation under realistic, non-stationary data distributions. These advances can support more trustworthy benchmarking and model development practices. Potential risks include increased reliance on automated judges, which may propagate biases or systematic errors inherited from the underlying models, and the possibility that adaptive routing could be misused to unevenly apply stronger evaluation. We mitigate these concerns by explicitly analyzing failure cases, prioritizing robustness over maximal performance, and positioning the method as a complement to human evaluation rather than a replacement.

Distributionally Robust Learning. Distributionally Robust Optimization (DRO) has been extensively studied in machine learning (Shafieezadeh Abadeh et al., 2015; Deng et al., 2020), statistics (Belloni et al., 2011; Duchi & Namkoong, 2021), and operations research (Goh & Sim, 2010; Kuhn et al., 2019). It is typically formulated as a minimax problem, in which an adversary perturbs the datagenerating distribution within a prescribed uncertainty set to maximize the expected loss, while the learner optimizes model parameters to minimize this worst-case risk. Among various constructions of uncertainty sets, f -divergence balls are a commonly used choice (Hu & Hong, 2013; Namkoong & Duchi, 2016; Levy et al., 2020; Duchi & Namkoong, 2021), due to their strong connections with classical divergence measures and well-studied statistical properties that facilitate analysis and implementation in DRO frameworks.

References Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappaganthu, K., Yang, Y., et al. Automix: Automatically mixing language models. arXiv preprint arXiv:2310.12963, 2023.

7. Conclusion and Limitation In this work, we study whether reasoning capabilities acquired by LLMs from problem-solving tasks transfer to judgment tasks. We find that reasoning-based judges can substantially improve accuracy, but the gains are task-dependent and often incur significantly higher computational cost, par-

Belloni, A., Chernozhukov, V., and Wang, L. Square-root lasso: pivotal recovery of sparse signals via conic pro9

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

gramming. Biometrika, 98(4):791–806, 2011.

Duchi, J. C., Glynn, P. W., and Namkoong, H. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 46(3): 946–969, 2021.

Bercovich, A., Levy, I., Golan, I., Dabbah, M., El-Yaniv, R., Puny, O., Galil, I., Moshe, Z., Ronen, T., Nabwani, N., et al. Llama-nemotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949, 2025.

Frick, E., Chen, C., Tennyson, J., Li, T., Chiang, W.-L., Angelopoulos, A. N., and Stoica, I. Prompt-to-leaderboard. arXiv preprint arXiv:2502.14855, 2025.

Cen, S., Cheng, C., Chen, Y., Wei, Y., and Chi, Y. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4): 2563–2578, 2022.

Fu, J., Ng, S. K., Jiang, Z., and Liu, P. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6556–6576, 2024.

Chakraborty, S., Qiu, J., Yuan, H., Koppel, A., Huang, F., Manocha, D., Bedi, A., and Wang, M. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024.

Gadot, U., Wang, K., Kumar, N., Levy, K. Y., and Mannor, S. Bring your own (non-robust) algorithm to solve robust mdps by estimating the worst kernel. In Forty-first International Conference on Machine Learning, 2024.

Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 4(5), 2024a.

Goh, J. and Sim, M. Distributionally robust optimization and its tractable approximations. Operations research, 58 (4-part-1):902–917, 2010. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

Chen, L., Zaharia, M., and Zou, J. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023. Chen, N., Hu, Z., Zou, Q., Wu, J., Wang, Q., Hooi, B., and He, B. Judgelrm: Large reasoning models as a judge. arXiv preprint arXiv:2504.00050, 2025a.

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.

Chen, X., Xu, J., Liang, T., He, Z., Pang, J., Yu, D., Song, L., Liu, Q., Zhou, M., Zhang, Z., et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187, 2024b.

Hu, Q. J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K., and Upadhyay, S. K. Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv:2403.12031, 2024.

Chen, X., Li, G., Wang, Z., Jin, B., Qian, C., Wang, Y., Wang, H., Zhang, Y., Zhang, D., Zhang, T., et al. Rm-r1: Reward modeling as reasoning. arXiv preprint arXiv:2505.02387, 2025b.

Hu, Z. and Hong, L. J. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online, 1(2):9, 2013.

Deng, Y., Kamani, M. M., and Mahdavi, M. Distributionally robust federated averaging. Advances in neural information processing systems, 33:15111–15122, 2020.

Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., et al. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2023.

Ding, D., Zhang, K., Duan, J., Başar, T., and Jovanović, M. R. Convergence and sample complexity of natural policy gradient primal-dual methods for constrained mdps. arXiv preprint arXiv:2206.02346, 2022.

Kuhn, D., Esfahani, P. M., Nguyen, V. A., and ShafieezadehAbadeh, S. Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations research & management science in the age of analytics, pp. 130–166. Informs, 2019.

Ding, D., Wei, C.-Y., Zhang, K., and Ribeiro, A. Lastiterate convergent policy gradient primal-dual methods for constrained mdps. Advances in Neural Information Processing Systems, 36:66138–66200, 2023.

Lai, X., Tian, Z., Chen, Y., Yang, S., Peng, X., and Jia, J. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629, 2024.

Duchi, J. C. and Namkoong, H. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378–1406, 2021. 10

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Lambert, N., Pyatkin, V., Morrison, J., Miranda, L. J. V., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 1755– 1797, 2025.

Olmo, T., Ettinger, A., Bertsch, A., Kuehl, B., Graham, D., Heineman, D., Groeneveld, D., Brahman, F., Timbers, F., Ivison, H., et al. Olmo 3. arXiv preprint arXiv:2512.13961, 2025. Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., and Stoica, I. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665, 2024.

Levy, D., Carmon, Y., Duchi, J. C., and Sidford, A. Largescale methods for distributionally robust optimization. Advances in neural information processing systems, 33: 8847–8860, 2020.

Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., and Stoica, I. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025.

Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/ alpaca_eval, 5 2023.

OpenAI. Learning to reason with LLMs. https://openai.com/index/ 2024. learning-to-reason-with-llms/, Accessed: 2025-12-28.

Liang, G., Zhong, L., Yang, Z., and Quan, X. Thinkswitcher: When to think hard, when to think fast. arXiv preprint arXiv:2505.14183, 2025.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

Liu, C. Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451, 2024a. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023.

Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 53728–53741, 2023.

Liu, Y., Yao, Z., Min, R., Cao, Y., Hou, L., and Li, J. Rmbench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.16184, 2024b.

Rahimian, H. and Mehrotra, S. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659, 2019.

Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., and Lin, M. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.

Saha, S., Li, X., Ghazvininejad, M., Weston, J., and Wang, T. Learning to plan & reason for evaluation with thinkingllm-as-a-judge. arXiv preprint arXiv:2501.18099, 2025.

Malik, S., Pyatkin, V., Land, S., Morrison, J., Smith, N. A., Hajishirzi, H., and Lambert, N. Rewardbench 2: Advancing reward model evaluation, 2025. URL https://arxiv.org/abs/2506.01937.

Shafieezadeh Abadeh, S., Mohajerin Esfahani, P. M., and Kuhn, D. Distributionally robust logistic regression. Advances in neural information processing systems, 28, 2015.

Mutti, M., De Santi, R., De Bartolomeis, P., and Restelli, M. Convex reinforcement learning in finite trials. Journal of Machine Learning Research, 24(250):1–42, 2023.

Sion, M. On general minimax theorems. Pacific Journal of Mathematics, 8:171–176, 1958.

Namkoong, H. and Duchi, J. C. Stochastic gradient methods for distributionally robust optimization with fdivergences. Advances in neural information processing systems, 29, 2016.

Son, S., Bankes, W., Chowdhury, S. R., Paige, B., and Bogunovic, I. Right now, wrong then: Non-stationary direct preference optimization under preference drift. arXiv preprint arXiv:2407.18676, 2024.

Nesterov, Y. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.

Song, W., Huang, Z., Cheng, C., Gao, W., Xu, B., Zhao, G., Wang, F., and Wu, R. Irt-router: Effective and interpretable multi-llm routing via item response theory. 11

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15629–15644, 2025.

Zhang, W., Qiao, S., Luo, L., Li, Y., Zheng, C., Xu, Q., Li, M., Gui, Y., He, Y., Qiu, J., et al. Synapseroute: An autoroute switching framework on dual-state large language model. arXiv preprint arXiv:2507.02822, 2025a.

Sui, Y., Chuang, Y.-N., Wang, G., Zhang, J., Zhang, T., Yuan, J., Liu, H., Wen, A., Zhong, S., Zou, N., et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025.

Zhang, Y., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025b.

Tan, S., Zhuang, S., Montgomery, K., Tang, W. Y., Cuadron, A., Wang, C., Popa, R. A., and Stoica, I. Judgebench: A benchmark for evaluating llm-based judges, 2024a. URL https://arxiv.org/abs/2410.12784.

Zhang, Z., Rossi, R. A., Kveton, B., Shao, Y., Yang, D., Zamani, H., Dernoncourt, F., Barrow, J., Yu, T., Kim, S., et al. Personalization of large language models: A survey. arXiv preprint arXiv:2411.00027, 2024.

Tan, S., Zhuang, S., Montgomery, K., Tang, W. Y., Cuadron, A., Wang, C., Popa, R. A., and Stoica, I. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784, 2024b.

Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36: 46595–46623, 2023.

Team, K., Bai, Y., Bao, Y., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y., Chen, Y., Chen, Y., et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. Vezora. Code-preference-pairs dpo mix. https: //huggingface.co/datasets/Vezora/ Code-Preference-Pairs, 2024. Hugging Face dataset. Wei, C.-Y., Lee, C.-W., Zhang, M., and Luo, H. Linear last-iterate convergence in constrained saddle-point optimization. In International Conference on Learning Representations, 2021. Whitehouse, C., Wang, T., Yu, P., Li, X., Weston, J., Kulikov, I., and Saha, S. J1: Incentivizing thinking in llmas-a-judge via reinforcement learning. arXiv preprint arXiv:2505.10320, 2025. Xu, Z., Vemuri, S., Panaganti, K., Kalathil, D., Jain, R., and Ramachandran, D. Robust llm alignment via distributionally robust direct preference optimization. arXiv preprint arXiv:2502.01930, 2025. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L., et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Zhan, W., Cen, S., Huang, B., Chen, Y., Lee, J. D., and Chi, Y. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. SIAM Journal on Optimization, 33(2):1061–1091, 2023. 12

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Appendix The appendix provides additional theoretical analysis, experimental details, and supplementary experimental results. In Section A, we establish the existence and uniqueness of the saddle point and prove last-iterate convergence, providing theoretical guarantees for the convergence and optimality of the proposed algorithm, RACER. In Section B, we describe in detail the data generation process, the training procedure, and the evaluation protocol. In Section C, we report supplementary results, including training curves and representative case studies that further motivate our proposed algorithm.

A. Theoretical Analysis A.1. Deriving Data Distribution from KL Uncertainty Set Lemma A.1. Let ρn ∈ ∆n be the empirical distribution over {zi }ni=1 , and define fi := Ea∼π(·|zi ) [f (zi , a)]. Let n n o X ρ̃(i) U(ρn , δ) := ρ̃ ∈ ∆n : ρ̃(i) log ≤δ . ρn (i) i=1

Define ρ and ρ as the solutions to ρ ∈ arg

n X

min ρ̃∈U (ρn ,δ)

ρ ∈ arg

ρ̃(i)fi ,

max ρ̃∈U (ρn ,δ)

i=1

Then there exist s, s ∈ R and τ > 0 such that, for all i, s − fi ρ(i) ∝ ρn (i) exp , τ Pn Pn with s ≤ i=1 ρn (i)f¯i and s ≥ i=1 ρn (i)f¯i .

n X

ρ̃(i)fi .

i=1

fi − s ρ(i) ∝ ρn (i) exp , τ

Proof. We prove the claim for ρ; the proof for ρ is analogous. Consider the convex optimization problem minimize n ρ̃∈R

subject to

n X i=1 n X i=1 n X

ρ̃(i)fi ρ̃(i) log

ρ̃(i) ≤ δ, ρn (i)

(A.1)

ρ̃(i) = 1,

i=1

ρ̃(i) ≥ 0, ∀i. The objective is linear and the KL constraint is convex. Moreover, Slater’s condition holds (e.g., ρ̃ = ρn is strictly feasible when δ > 0). Hence, the Karush–Kuhn–Tucker (KKT) conditions are necessary and sufficient for optimality. Introducing Lagrange multipliers λ ≥ 0 for the KL constraint, µ ∈ R for normalization, and νi ≥ 0 for non-negativity, the Lagrangian of (A.1) is ! ! n n n n X X X X ρ̃(i) L(ρ̃, λ, µ, ν) = ρ̃(i)fi + λ ρ̃(i) log −δ +µ ρ̃(i) − 1 − νi ρ̃(i). (A.2) ρn (i) i=1 i=1 i=1 i=1 The stationarity condition of the KKT implies that, for all i, ρ̃(i) fi + λ log + 1 + µ − νi = 0. ρn (i) 13

(A.3)

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Solving (A.3) for ρ̃(i) yields fi + µ + λ − νi ρ̃(i) = ρn (i) exp − . λ

(A.4)

In the non-degenerate case (e.g., δ > 0 and {fi } not all equal), the KL constraint is active at the optimum, and thus λ > 0. Since ρn (i) > 0 and the exponential is strictly positive, Eq. (A.4) implies ρ̃(i) > 0 for all i. By complementary slackness, we therefore have νi = 0 for all i, and Eq. (A.4) simplifies to fi + µ + λ . (A.5) ρ̃(i) = ρn (i) exp − λ Let τ := λ and define s := −(µ + λ). Then Eq. (A.5) can be written as s − fi ρ̃(i) ∝ ρn (i) exp , τ

(A.6)

which establishes the stated form of ρ. P P It remains to show that s ≤ i ρn (i)fi . Using the normalization condition i ρ̃(i) = 1 and Eq. (A.6), we obtain s 1 . exp =P n fi τ i=1 ρn (i) exp − τ Applying Jensen’s inequality to the convex function exp(·) yields Pn X n ρn (i)fi fi exp − i=1 ρn (i) exp − ≤ . τ τ i=1 Combining (A.7) and (A.8) gives exp which implies s ≤

s τ

(A.7)

(A.8)

Pn i=1 ρn (i)fi ≤ exp , τ

i=1 ρn (i)fi .

The Pproof for ρ follows identically by applying the same argument P to maxρ̃∈U (ρn ,δ) − i ρ̃(i)fi ), yielding ρ(i) ∝ ρn (i) exp((fi − s)/τ ) and s ≥ i ρn (i)fi .

i ρ̃(i)fi (equivalently, minimizing

A.2. Existence and Uniqueness of the Saddle Point (Theorem 4.1) We first present Proposition A.2, which states the Slater condition (feasibility) and is commonly required in the duality analysis of constrained optimization (see, e.g., Ding et al., 2022; 2023). A.2.1. F EASIBILITY C HECK Proposition A.2 (Feasibility). There exist a constant ξ > 0 and a policy π0 ∈ Π such that E(z,l)∼ρ,a∼π0 (·|z) [c(z, a)] − C ≤ −ξ, where π0 (0 | z) = 1 and π0 (1 | z) = 0 for all z ∈ Z. Proposition A.2 is straightforward to verify. Since E(z,l)∼ρ,a∼π0 (·|z) [c(z, a)] = 1, the inequality holds whenever ξ ≤ C − 1 with C > 1. To analyze the saddle point of Lβ (π, λ), we define π ∗ as the optimal solution to (8), that is, π ∗ ∈ arg max min Lβ (π, λ), π∈Π λ≥0

and let λ∗ denote the corresponding optimal dual variable, λ∗ ∈ arg min max Lβ (π, λ), λ≥0 π∈Π

with Λ∗ denoting the set of all optimal dual variables. We denote Lλβ := maxπ∈Π Lβ (π, λ). We can then show that Λ∗ is bounded as established in Lemma A.3. 14

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

A.2.2. B OUNDNESS OF S OLUTION S PACE Lemma A.3 (Boundness of λ∗ ). The optimal dual variable λ∗ satisfies ∗

∗

0≤λ ≤

Lλβ − Pρ (ϕ0 (z) = l) ξ

and hence Λ∗ is bounded. Proof. Let Λa := {λ ≥ 0 | Lλβ ≤ a} be a sublevel set of the dual objective for a ∈ R. Recall that Lλβ = maxπ∈Π Lβ (π, λ). For any λ ∈ Λa , we have a ≥ Lλβ ≥ Lβ (π0 , λ) 1 ≥ E(z,l)∼ρ,a∼π0 (·|z) [r(z, a, l)] − λ E(z,l)∼ρ,a∼π0 (·|z) [c(z, a)] − C + β E(z,l)∼ρ [H (π0 (· | z))] + λ2 2 1 2 ≥ E(z,l)∼ρ,a∼π0 (·|z) [r(z, a, l)] + λξ + β E(z,l)∼ρ [H (π0 (· | z))] + λ , 2 where the last inequality comes from Proposition A.2. Because H (π0 (· | z)) = 0 and E(z,l)∼ρ,a∼π0 (·|z) [r(z, a, l)] = Pρ (ϕ0 (z) = l), it follows that a ≥ Lλβ ≥ Pρ (ϕ0 (z) = l) + λξ. Thus we get λ ≤

∗ Lλ β −Pρ (ϕ0 (z)=l) . Finally, setting a = Lλβ with Λa = Λ∗ , we derive ξ ∗

0 ≤ λ∗ ≤

Lλβ − Pρ (ϕ0 (z) = l) ξ

Based on Lemma A.3, we define the domain for λ as ∗

Λ = [0,

Lλβ − Pρ (ϕ0 (z) = l) ξ

which ensures Λ∗ ⊆ Λ. With this setup, all the required conditions for applying Sion’s minimax theorem (Sion, 1958) are satisfied, which ensures the existence and uniqueness of the regularized saddle point. A.2.3. P ROOF OF T HEOREM 4.1 Proof. Based on Assumption 1, Π is a convex set. Moreover, since Λ is convex and bounded, i.e., compact, Sion’s minimax theorem (Sion, 1958) guarantees the existence of a saddle point of Lβ (π, λ) in Π × Λ. We next prove the uniqueness of this saddle point. Recall that 1 Lβ (π, λ) = E(z,l)∼ρ,a∼π(·|z) [r(z, a, l)] − λ E(z,l)∼ρ,a∼π(·|z) [c(z, a)] − C + β E(z,l)∼ρ [H (π(· | z))] + λ2 . 2 ∂ 2 L (π,λ)

β Since = β > 0, Lβ (π, λ) is strictly convex in λ. Then we will show Lβ (π, λ) is strictly concave in π. We first ∂λ2 examine the expectation terms: ! 1 X E(z,l)∼ρ,a∼π(·|z) [r(z, a, l)] = E(z,l)∼ρ r(z, a, l)π(a|z) ,

a=0

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

and

E(z,l)∼ρ,a∼π(·|z) c(z, a) = E(z,l)∼ρ

1 X

! c(z, a)π(a|z) .

a=0

Thus, both E(z,l)∼ρ,a∼π(·|z) [r(z, a, l)] and E(z,l)∼ρ,a∼π(·|z) c(z, a) are linear in π. Because the entropy term H (π(· | z)) is strongly concave in π, it follows that its expectation E(z,l)∼ρ [H (π(· | z))] remains strongly concave in π. Consequently, we have Lβ (π, λ) is also strongly concave in π for any fixed λ. Now we have shown Lβ (π, λ) is strictly concave in π and strictly convex in λ, which implies saddle point (π ∗ , λ∗ ) is unique. The existence of a saddle point as guaranteed by Lemma A.3, further implies that for any (π, λ) ∈ Π × Λ, Lβ (π, λ∗ ) ≤ Lβ (π ∗ , λ∗ ) ≤ Lβ (π ∗ , λ). The first inequality indicates that for any π ∈ Π, 1 1 L(π ∗ , λ∗ ) + β E(z,l)∼ρ [H (π ∗ (· | z))] + (λ∗ )2 ≥ L(π, λ∗ ) + β E(z,l)∼ρ [H (π(· | z))] + (λ∗ )2 2 2 β ≥ L(π, λ∗ ) + (λ∗ )2 , 2 while the second inequality implies that for any λ ∈ Λ, 1 1 L(π ∗ , λ) + β E(z,l)∼ρ [H (π ∗ (· | z))] + λ2 ≥ L(π ∗ , λ∗ ) + β E(z,l)∼ρ [H (π ∗ (· | z))] + (λ∗ )2 2 2 ≥ L(π ∗ , λ∗ ) + β E(z,l)∼ρ [H (π ∗ (· | z))] . Combining the two inequalities above shows that (π ∗ , λ∗ ) is a saddle point of the original Lagrangian L(π, λ), up to two β-regularization terms. A.3. Proof of Last-iterate Convergence (Theorem 4.2) Proof. In practice, we will update (πt , λt ) as the following steps. πt+1 = arg max Ez∼ρ,a∼π(·|z) [r(z, a)] − λt Ez∼ρ,a∼π(·|z) [g(z, a)] + β Ez∼ρ [H (π(· | z))] , π∈Π λt+1 = λt + η Ez∼ρ,a∼πt+1 (·|z) [g(z, a)] − C − βλt + ,

(A.9) (A.10)

where η denotes the step size. For notation simplicity, we denote w1 (z) = pρ (z)/pρ (z) and w2 (z) = pρ (z)/pρ (z) for any z ∈ Z, which are well defined under Assumption 3. Step 1.

We first derive the closed-form solution to (6).

According to (6), we optimize the following objective max E(z,l)∼ρ,a∼π(·|z) [r(z, a, l)] − λt E(z,l)∼ρ,a∼π(·|z) [c(z, a)] + β E(z,l)∼ρ [H (π(· | z))] . π∈Π

(A.11)

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Analogously to arguments used in the proof of DPO (Rafailov et al., 2023), (A.11) can be rewritten as max E(z,l)∼ρ,a∼π(·|z) [r(z, a, l)] − λt E(z,l)∼ρ,a∼π(·|z) [c(z, a)] + β E(z,l)∼ρ [H (π(· | z))] π∈Π

(A.12)

= max E(z,l)∼ρ,a∼π(·|z) [w1 (z)r(z, a, l) − λt w2 (z)c(z, a)] + β E(z,l)∼ρ [H (π(· | z))] π∈Π

(A.13)

where we have partition function: Q(z, λt ) =

1 X

exp

a=0

1 λt w1 (z)r(z, a, l) − w2 (z)c(z, a) . β β

Note that the partition function depends only on z, λt and is independent of the policy π. We now define 1 1 λt πt+1 (a|z) = exp w1 (z)r(z, a, l) − w2 (z)c(z, a) , Q(z, λt ) β β which forms a valid probability distribution. Substituting this definition into (A.13), we can rewrite the objective as −β min E(z,l)∼ρ [KL(π(· | z)∥πt+1 (· | z)) − log(Q(z, λt ))] . π∈Π

Since Q(z, λt ) does not depend on π, the minimum is achieved by the policy that minimizes the KL term. By Gibbs’ inequality, the KL divergence attains its minimum value of zero if and only if the two distributions are identical. Therefore, the updated policy is given by 1 1 λt (A.14) πt+1 (a|z) = exp w1 (z)r(z, a, l) − w2 (z)c(z, a) . Q(z, λt ) β β Step 2.

We next establish the convergence of λt .

We view πt+1 as a function of λt , and denote it by π eλt := πt+1 . For notational convenience, we denote d(λ) := Lλβ = maxπ∈Π Lβ (π, λ) = Lβ (e πλ , λ). Then d(λ) can be explicitly written as, β d(λ) =β E(z,l)∼ρ [log(Q(z, λ))] + λC + λ2 2 " # 1 X λ β 1 w1 (z)r(z, a, l) − w2 (z)c(z, a) + λC + λ2 . =β E(z,l)∼ρ log exp β β 2 a=0 Consequently, the update rule in (7) can be expressed as λt+1 = [λt − η d′ (λt )]+ . We then compute the first and second derivatives of d(λ). Notice that ′ d′ (λ) = β E(z,l)∼ρ [log(Q(z, λ))] + C + βλ, (A.15) 17

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

′ so it suffices to compute E(z,l)∼ρ [log(Q(z, λ))] . We have 1 ∂Q(z, λ) X λ 1 1 = w1 (z)r(z, a, l) − w2 (z)c(z, a) − w2 (z)c(z, a) , exp ∂λ β β β a=0 and therefore,

1 − w (z)c(z, a) 2 β ∂ log Q(z, λ) = P1 λ 1 ∂λ w (z)r(z, a, l) − w (z)c(z, a) exp 1 2 a=0 β β 1 X 1 1 = π eλ (a|z) − w2 (z)c(z, a) = Ea∼eπλ (·|z) − w2 (z)c(z, a) . β β a=0 P1

a=0 exp

1 λ β w1 (z)r(z, a, l) − β w2 (z)c(z, a)

Based on Assumption 2 and 3, we have | − βλ w2 (z)c(z, a)| ≤ MβK , thus ∂ log Q(z, λ) MK ≤ , ∂λ β which ensures the boundedness of the derivative. Hence, the order of differentiation and expectation can be interchanged. Thus, (A.15) comes to be d′ (λ) = −E(z,l)∼ρ,a∼eπλ (·|z) w2 (z)c(z, a) + C + βλ. For the second derivative, we have ′ d′′ (λ) = − E(z,l)∼ρ,a∼eπλ (·|z) w2 (z)c(z, a) + β, where Ea∼eπλ (·|z) w2 (z)c(z, a)

′

= w2 (z)

1 X ∂π eλ (a|z) a=0

Let sλ (a) = exp have

1 λ β w1 (z)r(z, a, l) − β w2 (z)c(z, a)

∂λ

c(z, a).

(A.16)

(A.17)

, then π eλ (a|z) = sλ (a)/((sλ (0) + sλ (1)). By the chain rule, we 1

∂π eλ (a|z) X ∂ π eλ (a|z) ∂sλ (b) = . ∂λ ∂sλ (b) ∂λ

(A.18)

∂sλ (b) 1 = sλ (b) − w2 (z)c(z, b) , ∂λ β

(A.19)

∂π eλ (a|z) I(a = b)sλ (1 − a) − I(a ̸= b)sλ (a) I(a = b)e πλ (1 − a|z) − I(a ̸= b)e πλ (a|z) = = . 2 ∂sλ (b) s (0) + s (1) λ λ ((sλ (0) + sλ (1))

(A.20)

b=0

We can derive that

and

Combining (A.19) and (A.20), (A.18) comes to be 1 X 1 ∂π eλ (a|z) =π eλ (a|z) (δab − π eλ (b|z)) − w2 (z)c(z, b) ∂λ β b=0

w2 (z) = π eλ (a|z) (Ea′ ∼eπλ (·|z) c(z, a′ )) − c(z, a) . β

(A.21)

Similarly, since (A.21) is bounded according to Assumptions 2 and 3, we can interchange the order of differentiation and expectation. Combining (A.17) and (A.21), (A.16) turns to be 1 X 1 d′′ (λ) = β − E(z,l)∼ρ w2 (z)2 π eλ (a|z) (Ea′ ∼eπλ (·|z) c(z, a′ )) − c(z, a) c(z, a) β a=0

=β+

1 E(z,l)∼ρ w2 (z)2 Vara∼eπλ (·|z) c(z, a). β 18

(A.22)

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Again, due to Assumption 2 and 3, we have c(z, a) ≤ M and w2 (z) ≤ K. Hence, β ≤ d′′ (λ) ≤ β +

M 2K 2 . β

We therefore conclude that d(λ) is strongly convex with parameter β, and its gradient is Lipschitz continuous with constant 2 2 β + M βK . Then, by Theorem 2.1.5 in (Nesterov, 2013), choosing the step size as η = 2/(β + β + M 2 K 2 /β) = 2β/(M 2 K 2 + 2β 2 ) in (7), we have !t t M 2K2 β + − β M 2K 2 β ∗ ∗ |λ0 − λ∗ |. (A.23) |λt+1 − λ | ≤ |λ0 − λ | = 2 2 M 2 K 2 + 2β 2 β + M βK + β In our setting, although the update involves a projection onto the feasible set, the projection operator is non-expansive. Consequently, standard results for projected gradient descent apply, and the projected updates achieve the same convergence rate as the corresponding unconstrained gradient descent. Step 3.

Finally, we show that πt converges to π ∗ using the KL divergence.

Given z, the quantity KL(πt (·|z)∥π ∗ (·|z)) is a Bregman divergence between the corresponding natural parameters of the exponential family: Dψ (θt , θ∗ ) = KL(πt (·|z)∥π ∗ (·|z)), where the convex potential ψ is the log-partition function ψ(θ) = log Q(z, −θβ) = log

1 X

exp

a=0

1 w1 (z)r(z, a, l) − θw2 (z)c(z, a) . β

Here we let θt = −λt /β and θ∗ = −λ∗ /β, treating π eλ (·|z) as an exponential-family distribution with natural parameter θ. Then we have exp ψ ′ (θ) = P 1

1 β w1 (z)r(z, a, l) − θw2 (z)c(z, a)

c(z, a) = Ea∼eπ−θβ (·|z) w2 (z)c(z, a). 1 ′ ′ a′ =0 exp β w1 (z)r(z, a ) − θw2 (z)c(z, a )

By (A.22), we further obtain, w2 (z)2 ψ ′′ (θ) = (−β) − Vara∼eπλ (·|z) c(z, a) = w2 (z)2 Vara∼eπλ (·|z) c(z, a) ≤ M 2 K 2 . β By applying Taylor’s expansion to ψ, there is 1 e t − θ ∗ )2 , ψ(θt ) = ψ(θ∗ ) + ψ ′ (θ∗ )(θt − θ∗ ) + ψ ′′ (θ)(θ 2 where θe lies between θt and θ∗ . Consequently, we get Dψ (θt , θ∗ ) =

1 ′′ e M 2K 2 M 2K 2 ψ (θ)(θt − θ∗ )2 ≤ (θt − θ∗ )2 = (λt − λ∗ )2 , 2 2 2β 2

which is equivalent to KL(πt (·|z)∥π ∗ (·|z)) ≤

M 2K 2 (λt − λ∗ )2 . 2β 2

Averaging over (z, l) ∼ ρ, we get KL(πt ∥π) = E(z,l)∼ρ KL(πt (·|z)∥π ∗ (·|z)) ≤

M 2K 2 (λt − λ∗ )2 . 2β 2

Combining this with (A.23), we finally derive M 2K 2 KL(πt ∥π) ≤ 2β 2

M 2K 2 M 2 K 2 + 2β 2

(λ0 − λ∗ )2 .

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

B. Experimental Details B.1. Data Construction and Evaluation Protocol We construct all judge data using a unified generation protocol, and explicitly separate datasets used for training and in-distribution analysis from those reserved for out-of-distribution (OOD) evaluation.

Judge Generation Protocol. Across all datasets, we generate judge responses using the same base model from the Q WEN 3 family (1.7B/4B/8B), under two inference modes: non-reasoning and reasoning. All generations use an identical prompting format (Table C.3) and a fixed sampling temperature of 0.6. This ensures that any observed performance or cost differences are attributable solely to the inference mode and data distribution.

Training and In-Distribution Data. We construct the training corpus from three sources: S KYWORK -R EWARD (Liu et al., 2024a), M ATH -S TEP -DPO-10K (Lai et al., 2024), and C ODE -P REFERENCE -PAIRS (Vezora, 2024). Specifically, we sample 20,000 instances from S KYWORK -R EWARD, and an additional 20,000 instances from a mixture of M ATH -S TEP -DPO-10K and C ODE -P REFERENCE -PAIRS. Together, these datasets cover general instruction-following, mathematical reasoning, and code-related preference judgments. For each instance, we generate paired judge outputs under both inference modes. We record two signals for each mode: (i) a binary correctness indicator, obtained by comparing the judge’s preference with the reference label provided by the dataset, and (ii) the number of tokens consumed during generation. These paired correctness and token-cost measurements define the empirical accuracy–cost trade-offs used both for the analysis in Section 2 and for training the routing policy. All training and in-distribution data are used exclusively for learning the routing policy.

Out-of-Distribution Evaluation Data. To evaluate generalization, we assess routing performance on multiple held-out benchmarks that are disjoint from the training data, including J UDGE B ENCH (Tan et al., 2024a), R EWARD B ENCH (Lambert et al., 2025), and R EWARD B ENCH -2 (Malik et al., 2025). These benchmarks span diverse evaluation domains and exhibit distributional shifts relative to the training corpus. We briefly summarize the domains they cover and the distribution in Table B.1. For all OOD benchmarks, we apply the same judge generation protocol as used for training data, including the same base models, inference modes, prompts, and sampling temperature. Evaluation datasets are used solely for OOD testing and do not influence model selection or hyperparameter tuning. B.2. Text Representations Each input instance is represented by a fixed-dimensional text embedding extracted via Qwen3-embedding-4B (Zhang et al., 2025b). Specifically, we encode the prompt and the two candidate responses separately, and concatenate their embeddings to form the policy input vector. The same representation procedure is applied consistently across all training and evaluation datasets. B.3. Routing Policy Architecture and Optimization Our routing policy is a 4-layer MLP with ReLU activations and hidden widths {256, 128, 64}, mapping the input embedding to a single scalar logit. Models are trained for 60 epochs using the AdamW optimizer with learning rate 10−4 and batch size 64. The dual variable associated with the budget constraint is updated using a step size of 10−3 and projected onto the non-negative reals after each update. An entropy regularization term with coefficient 0.005 is applied to encourage exploration and stabilize training. For model selection, we evaluate checkpoints on a held-out validation set. Among checkpoints that satisfy the target budget constraint, we select the one achieving the highest validation accuracy. If no checkpoint satisfies the constraint, we select the checkpoint whose validation cost is closest to the target budget. 20

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge Table B.1. Domain distribution of OOD evaluation datasets. Benchmark

Domain

Count

Proportion

JudgeBench (Tan et al., 2024a)

Coding

11.8%

Math

14.5%

Reasoning Knowledge

149 308

24.0% 50.0%

Math

447

15.0%

Coding

984

33.0%

Chat

814

27.3%

Safety

740

24.8%

Factuality

475

26.9%

Precise IF

160

9.1%

Math

183

10.4%

Safety

450

25.5%

Focus

495

28.1%

RewardBench (Lambert et al., 2025)

RewardBench 2 (Malik et al., 2025)

Description Challenging programming questions from contest-style platforms (e.g., LeetCode, AtCoder, Codeforces). Problems drawn from math competitions (e.g., AMC12, USAMO). Reasoning-focused evaluation set. College-level multiple-choice questions across 14 disciplines (e.g., Physics, Chemistry, Law), with up to 10 answer options. Aggregated from RewardBench original subsets. Subsets: math-prm. Subsets: hep-cpp, hep-go, hep-java, hep-js, hep-python, hep-rust. General assistant-style. Subsets: alpacaeval-easy, alpacaeval-length, alpacaeval-hard; mt-bench-easy, mtbench-medium, mt-bench-hard; llmbar-natural, llmbaradver-neighbor, llmbar-adver-GPTInst, llmbar-adverGPTOut, llmbar-adver-manual. Refusal/safety-oriented. Subsets: refusals-dangerous, refusals-offensive, xstest-should-refuse, xstest-shouldrespond, do-not-answer. Evaluates detection of hallucinations and other basic factual errors in completions. Precise instruction following: judges whether outputs satisfy detailed, constraint-heavy instructions. Assesses math ability on open-ended prompts spanning middle-school topics through college-level chemistry, calculus, and combinatorics. Measures appropriate compliance vs. refusal under harmful-use prompts and general safety-related behaviors. Tests whether models prefer high-quality, on-topic answers for general user queries.

B.4. Budget Settings and Repeated Trials We evaluate the routing policy under a range of target compute budgets: C ∈ {2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 6.0, 7.0, 10.0}. For each budget level, we repeat the entire training and evaluation procedure 10 times with independent random seeds. We report the mean of both accuracy and realized cost ratio across these runs. B.5. Hyperparameter Tuning Optimization hyperparameters. We select standard optimization hyperparameters via a grid search on an in-distribution validation set, and then fix them for all experiments. This includes the regularization parameter β, the learning rate (10−4 ), batch size (64), the dual update step size (10−3 ), and the entropy regularization coefficient (0.005). Temperature for robustness. Since we assume no access to supervision for validation for OOD evaluation, we adopt the following rule-of-thumb. Let τR and τC denote the robustness strengths for reward and cost, respectively, where smaller values impose more aggressive (more robust) reweighting, while larger values indicate a more non-robust behavior. If the anticipated an OOD dataset requires less computation than in-distribution, we recommend using a smaller τR (stronger reward robustness) and a larger τC (weaker cost robustness) to better utilize the budget. If the anticipated an OOD dataset require more computation, we recommend using a smaller τC (stronger cost robustness) to reduce budget-violation risk, 21

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge Table B.2. Judge prompting templates for instruct vs. reasoning modes. Differences are highlighted in red. Field

Instruct mode

Reasoning mode

System

Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user’s instructions and answers the user’s question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: ‘‘[[A]]’’ if assistant A is better, ‘‘[[B]]’’ if assistant B is better.

Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user’s instructions and answers the user’s question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide an explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: ‘‘[[A]]’’ if assistant A is better, ‘‘[[B]]’’ if assistant B is better.

User template

[User Question]{question} [The Start of Assistant A’s Answer]{answer a}[The End of Assistant A’s Answer] [The Start of Assistant B’s Answer]{answer b}[The End of Assistant B’s Answer]

while keeping τR moderate to avoid over-conservatism. When the shift direction is uncertain, a conservative default is to use moderate robustness on both terms. In the OOD evaluation in Section 5.3, we fix τR = 1 and disable robust reweighting on the cost term.

C. Supplementary Experimental Results C.1. Training Curves In Figure C.1, we report the training curves for reward, realized cost, and lambda under different budgets. These training curves indicate that the Lagrange multiplier dynamically adapts to the tightness of the compute budget: it grows under stringent budgets to actively enforce cost control, but collapses toward zero as the budget loosens, signaling that the constraint becomes non-binding. C.2. Case Studies In Table C.1, we report some representative cases to illustrate when and why reasoning helps. 22

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Figure C.1. Training dynamics of RACER under different cost budgets. Training reward (left), realized cost (middle), and the learned dual variable λ (right) across epochs for budgets B ∈ {2, 4, 6}. The dashed line denotes the target cost budget. Reward increases monotonically while λ adapts to the tightness of the constraint, driving the training cost toward the specified budget.

C.3. Evaluation on Additional Model Family To show that RACER is not specific to the Qwen3 family, we additionally train and evaluate RACER using DeepSeek-R1Distill-Llama-8B (Guo et al., 2025) as the reasoning judge, and the corresponding Llama-3.1-8B-Instruct (Grattafiori et al., 2024) as the instruct mode judge. We follow the same data generation protocol defined in Section 5.3 and construct the training set from the mixture of S KYWORK -R EWARD (Liu et al., 2024a), M ATH - STEP -DPO-10K (Lai et al., 2024) and CODE-PREFERENCE-PAIRS (Vezora, 2024). As shown in the table C.2, RACER consistently outperforms random routing at matched budgets and often surpasses all-reasoning accuracy at lower cost, showing that RACER has generalization with respect to the model family. C.4. Analysis of Self-Routing Behavior Our approach learns the routing policy from data. As a comparison, we consider a simple self-routing baseline, where the LLM itself decides whether reasoning is needed. The training and evaluation data is the same as we used in Section 5.3, generated by Qwen3-4B pair. The self-routing baseline is implemented as a two-step procedure: we first prompt (see table C.3) the instruct mode judge to decide whether to enable reasoning, and then evaluate the response generated under the chosen mode, measuring the corresponding accuracy and cost. As shown in Table C.4, this strategy routes nearly all inputs to the reasoning mode, resulting in behavior and cost that are effectively equivalent to the all-reasoning judge. This suggests that self-routing fails to learn meaningful selectivity. In contrast, RACER learns to activate reasoning only when beneficial under a budget, achieving a better accuracy–cost trade-off.

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Table C.1. Empirical case studies across benchmarks (excerpts only). Each row contrasts non-reasoning vs. reasoning judge outputs on the same instance; the label indicates whether the judge decision matches the ground-truth winner. Dataset (ID)

Non-Reasoning Judge

J UDGE B ENCH -M ATH (515)

I NCORRECT (✗)

Reasoning Judge C ORRECT (✓)

(excerpt) Both Assistant A and Assistant B provide step-by-step solutions. The judge prefers Assistant B due to its more detailed explanations and justifications , which are considered more convincing and thorough, despite an error in counting the contribution of 49 = 72 . . . .

(excerpt) The judge explicitly verifies the computation using the Legendre formula:

Final verdict:

It identifies that Assistant B undercounts the contribution of 49, and correctly determines that Assistant A provides the correct answer. . . .

[[B]]

50 7

50 49

=7+1=8

Final verdict: [[A]]

J UDGE B ENCH -K NOWLEDGE (38)

RM-B ENCH -C HAT (chat/404/0 0)

C ORRECT (✓)

I NCORRECT (✗)

(excerpt) Both Assistant A and Assistant B provided thorough and accurate responses to the user’s question. . . . Assistant B’s response is more detailed and provides a step-by-step analysis of each option. They correctly identify the correct answers as (D), (G), and (I) but choose to focus on (D) as the main correct answer. . . Additionally, Assistant B’s response is more aligned with the user’s request for a single correct answer. Therefore, based on the evaluation, I would say that Assistant B’s response is slightly better than Assistant A’s response. [[B]]

(excerpt) Okay, so I need to evaluate which AI assistant, A or B, provided a better answer . . . Assistant A’s response . . . They provided explanations for each, pointing out which were correct and incorrect. For example, they correctly identified that option (I) is the right answer for vibration sense with a 128 Hz tuning fork. . . . Assistant B’s response . . . However, their reasoning had some issues. For instance, they mentioned that the assessment should focus on the dominant arm . . . and they didn’t fully explain why 128 Hz is better than 256 Hz for vibration testing. They also made a mistake by suggesting that the correct answer is (D) when the user’s main question was about vibration sense, which should point to (I) instead. . . . Assistant B, while correct on some points, didn’t address the primary question accurately and had some inaccuracies in their reasoning. Therefore, Assistant A’s response is more comprehensive and accurate. [[A]]

C ORRECT (✓)

(excerpt) Both Assistant A and Assistant B provide a concise and relevant answer. Assistant A correctly states that Mars appears reddish due to iron oxide . . . On the other hand, Assistant B

(excerpt) Okay, let’s see. I need to evaluate the responses . . . Assistant A’s response . . . It’s concise and accurate. Assistant B’s response . . . Both responses mention reddish and rust, but Assistant B says copper oxide instead of iron oxide . That’s a mistake. . . . So, in terms of accuracy, Assistant A is correct, whereas Assistant B is wrong. . . . However, the accuracy is key here. Since Assistant A provided the right information, it’s better. . . . [[A]]

incorrectly states that Mars appears reddish due to copper oxide . . . Considering the accuracy of the responses, Assistant A’s answer is more reliable and trustworthy. . . . Therefore, based on the evaluation, the final verdict is: [[A]]

Table C.2. Routing performance and cost comparison across budgets using data generated by the DeepSeek-R1-Distill-Llama-8B and Llama-3.1-8B-Instruct pair. Budget All-Instruct 2.0 2.5 3.0 3.5 All-Reasoning

RACER Acc (%)

RACER Cost

Random Acc (%)

Random Cost

67.32 79.37 80.41 80.87 80.83 79.93

1.00 2.05 2.45 2.89 3.20 4.12

– 73.01 74.99 76.06 76.98 –

– 2.44 2.88 3.15 3.39 –

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Table C.3. User prompt template used for self-routing (instruct mode). Template

Content

User Prompt

You are about to judge which of two AI responses better answers a user question. [User Question]{question} [The Start of Response A]{answer a}[The End of Response A] [The Start of Response B]{answer b}[The End of Response B] Do you need to enable your reasoning mode (i.e., think step-by-step carefully) to judge which response is better for the above question and responses? Answer with a single word: yes or no.

Table C.4. Self-routing behavior across model scales in Qwen3 family. Model 1.7B 4B 8B

Reasoning Fraction 99.91 99.51 99.88

Accuracy (%)

Cost Ratio

All-instruct

All-reasoning

Self-routing

70.04 81.96 82.20

78.35 86.50 87.86

78.34 86.50 87.86

19.07 6.25 6.36

Record · ID 175300 · SHA-256 b77f2e306b3ad80c
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.