How to Train a Critic Stably and Efficiently

How to Train a Critic Stably and Efficiently

H OW TO T RAIN A C RITIC S TABLY AND E FFICIENTLY Penghui Qi1 , Xiangxin Zhou2 , Wee Sun Lee1 National University of Singapore 2 Tencent Hunyuan {penghuiq,leews}@comp.nus.edu.sg 1

arXiv:2608.23566v1 [cs.LG] 24 Aug 2026

A BSTRACT Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop Best-Practice Critic Optimization (BPCO), a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.

1

I NTRODUCTION

Reinforcement learning (RL) has become a standard approach for improving the reasoning and instruction-following abilities of large language models (LLMs) (Ouyang et al., 2022; Guo et al., 2025; Qi et al., 2026a). Effective RL depends on assigning credit to the sampled tokens (Sutton & Barto, 2018). Group-based methods such as GRPO estimate this signal by sampling several responses for each prompt and comparing their rewards (Shao et al., 2024; Liu et al., 2025). This approach avoids training a value function, but it uses multiple rollouts per prompt and assigns the same outcome-based advantage to every token in a response. A learned critic offers a direct alternative (Schulman et al., 2017). By estimating the expected return of each response prefix, a critic can construct token-level advantages from one rollout (Schulman et al., 2015; Hou et al., 2026). In practice, however, critic-based LLM training remains fragile. PPO’s ratio clipping treats low- and high-probability tokens unevenly (Qi et al., 2026b). Bootstrapped value targets can inherit critic error (Yuan et al., 2025), and a fixed GAE parameter gives the terminal reward very different weights in short and long responses (Yue et al., 2025). We identify two additional mismatches in common implementations. First, a linear value head can predict outside the known range of the return. Second, batch-wise advantage normalization forces every batch to have unit-scale advantages, even when the residual policy signal has become small. Our controlled study shows that both choices can destabilize training. A critic also creates an opportunity that group-relative estimators do not directly exploit. Because the critic is discarded after training, it may receive reward-defining information that is unavailable to the policy. Examples include a reference answer or official solution in mathematical reasoning and a prompt-specific rubric in open-ended evaluation. Such information is determined by the prompt and therefore does not change the ideal value function. Presenting it explicitly can nevertheless make that function easier to approximate, without changing the policy’s inputs or deployment requirements. We combine these choices into Best-Practice Critic Optimization (BPCO), a single-rollout actor– critic recipe. BPCO uses DPPO to define clipping in terms of the sampled token’s probability 1

How to Train a Critic Stably and Efficiently

change. It bounds value predictions to the reward range and trains the critic directly on observed outcomes. For the policy update, it preserves the scale of the raw advantages and adapts the GAE parameter to response length. BPCO can additionally use reward-defining information as privileged critic input when such information is available. Together, these choices align the critic’s output, target, and inputs with the policy signal it produces. We develop the recipe incrementally in a controlled sanity test (Section 3), where failure to fit a small, solvable dataset reveals optimization problems. We then evaluate BPCO on a 40.3K-problem mathematical dataset (Section 4.1), two 30B-A3B mixture-of-experts models (Section 4.2), and a rubric-reward task (Section 4.3). The experiments support three findings. First, BPCO improves the critic-based baseline across model and dataset scales. Second, privileged information can accelerate critic learning, but its policy benefit depends on the task and the degree of overfitting. Third, BPCO matches or exceeds a group-based baseline while using one response per prompt. These results establish a practical recipe for single-rollout critic-based LLM RL.

2

BACKGROUND

2.1

P ROXIMAL P OLICY O PTIMIZATION

Given a prompt x, a language model with parameters θ generates a response y = (y1 , . . . , yT ) autoregressively. At step t, the state is the prefix st = (x, y<t ), the action is the next token yt , and the policy is πθ (yt | st ). We consider outcome rewards: a completed response receives a scalar reward R(x, y), and all intermediate rewards are zero. Proximal Policy Optimization (PPO) uses a clipped surrogate objective (Schulman et al., 2017). Let µ be the behavior policy that generated the rollouts, and define sampled-token probability ratio as ρt (θ) =

πθ (yt | st ) . µ(yt | st )

bt , PPO maximizes Given an advantage estimate A h  i bt , clip(ρt (θ), 1 − ϵ, 1 + ϵ)A bt . LPPO (θ) = Et min ρt (θ)A

(1)

The clipped term removes the incentive to move the sampled-token ratio farther beyond the clipping bt , forming a trust region to stabilize training. boundary in the direction favored by A 2.2

D IVERGENCE P ROXIMAL P OLICY O PTIMIZATION

PPO applies the same ratio threshold to every token. In a large vocabulary, this rule clips small absolute changes to low-probability tokens while allowing much larger absolute changes to highprobability tokens (Qi et al., 2026b). Divergence Proximal Policy Optimization (DPPO) instead defines the clipping boundary in terms of the sampled token’s probability change. The binary totalvariation variant used in this work replaces ϵ in Equation (1) with ϵ/µ(yt | st ):      ϵ ϵ b bt . (2) LDPPO (θ) = Et min ρt (θ)At , clip ρt (θ), 1 − ,1 + A µ(yt | st ) µ(yt | st ) Equivalently, DPPO constrains the probability shift of the sampled token under the policy update, i.e., |πθ (yt | st ) − µ(yt | st )| ≤ ϵ. This gives sampled tokens a common absolute-probability threshold rather than a common ratio threshold. 2.3

C RITIC -BASED M ETHODS

Critic-based methods estimate the expected return of each prefix. For rollouts from µ, the value function is V µ (st ) = Eµ [R(x, y) | st ] , and the critic Vϕ (st ) approximates this quantity. Let ϕold denote the frozen critic parameters used to construct targets. Generalized advantage estimation (GAE) (Schulman et al., 2015) first computes 2

How to Train a Critic Stably and Efficiently

temporal-difference residuals and then forms an exponentially weighted sum: δt = rt + γVϕold (st+1 ) − Vϕold (st ), bGAE(λ) A = t

T −t X

(γλ)l δt+l .

(3) (4)

l=0

Here rt = 0 for t < T , rT = R(x, y), and Vϕold (sT +1 ) = 0. The discount factor is γ, and λ controls the degree of bootstrapping. Smaller λ can reduce variance but makes the estimate more sensitive to critic error. With γ = 1 and λ = 1, the sum telescopes to R(x, y) − Vϕold (st ) and contains no bootstrapped value target. Many implementations construct the critic target as bGAE(λ) Vbt (λ) = A + Vϕold (st ) t and minimize LV (ϕ) = Et GAE(λ)

bt The policy update uses A γ = 1 is commonly used. 2.4



Vϕ (st ) − Vbt (λ)

2 

(5)

.

(6)

in Equation (1) or Equation (2). In outcome-reward LLM training,

G ROUP -BASED M ETHODS

Group-based methods avoid a critic by sampling G responses {y (i) }G i=1 for each prompt (Shao et al., 2024). Let Ri = R(x, y (i) ), and let µR and σR be the mean and standard deviation of the G rewards. GRPO assigns every token in response i the advantage Ri − µ R bGRPO . A = t,i σR

(7)

Dr. GRPO removes the standard-deviation normalization, which can otherwise reweight prompts according to their within-group reward variance (Liu et al., 2025). Its advantage is bDr.GRPO = Ri − µR . A t,i

3

(8)

B UILDING BPCO: A C ONTROLLED S TUDY

We begin from a verl commit from June 16, 2026 and study critic stability in a controlled sanity test (Qi et al., 2025; 2026b). We fine-tune DeepSeek-R1-Distill-Qwen-1.5B (Guo et al., 2025) on 1,460 mathematical problems that the initial model can solve. A suitable training recipe should fit this deliberately small dataset to nearly 100% reward. Failure to do so exposes an optimization problem rather than a lack of model capacity or reward signal. Each iteration contains 1,024 trajectories. We use a minibatch size of 256 and one optimization epoch, giving four optimizer minibatches per iteration. Following the verl defaults (Sheng et al., 2025), the policy and critic learning rates are 10−6 and 10−5 , respectively. We observed no benefit from critic warm-up in this small-data setting and therefore update the policy and critic from the first iteration. Each run lasts 1,500 iterations. Because fitting this small dataset can harm generalization, we monitor AIME 2025 avg@32, the mean accuracy over 32 sampled responses per problem, as a held-out metric. We modify the default recipe one component at a time. The starting point uses PPO (Section 2.1), standard GAE and critic targets (Section 2.3), and λ = 1. Unless stated otherwise, each step retains all preceding changes. 3.1

S TEP 1: R EPLACING PPO WITH DPPO

With the PPO objective in Equation (1), the training reward collapses. Replacing it with the DPPO objective in Equation (2) yields stable optimization when λ = 1, as shown in Figure 1. 3

1.0

0.30

0.8

0.25

PPO with = 1 DPPO with = 1 DPPO with = 0.99

0.6 0.4 0.2 0.0

0

250

500

750

AIME 2025 Avg@32

Rewards

How to Train a Critic Stably and Efficiently

0.20 0.15 0.10 0.05 0.00

1000 1250 1500

Training Step

0

250

500

750

1000 1250 1500

Training Step

Figure 1: PPO and DPPO in the sanity test. With λ = 1, PPO’s training reward collapses after an initial increase, whereas DPPO remains stable. Reducing the GAE parameter to λ = 0.99 makes DPPO unstable and exposes sensitivity to critic error.

Reducing the GAE parameter to λ = 0.99 makes DPPO unstable again. When λ < 1, the policy advantage contains bootstrapped critic predictions. Unless Vϕ (st ) = V µ (st ) for every visited state, approximation error biases the advantage estimate relative to the Monte Carlo estimator obtained with λ = 1. We therefore use λ = 0.99 as a stress test in the next steps: stabilizing this setting requires the recipe to control how critic error enters the policy update. S TEP 2: B OUNDING VALUES TO THE R EWARD R ANGE

1.0

Rewards

0.8 0.6 0.4 0.2 0.0

Unbounded value Bounded value 0

250 500 750 1000 1250 1500

Training Step

AIME 2025 Avg@32

0.25 0.20

Value Max & Min

3.2

0.15 0.10 0.05 0.00

0

250 500 750 1000 1250 1500

Training Step

8 6 4 2 0 2 4 6 8

0

250 500 750 1000 1250 1500

Training Step

Figure 2: Effect of bounding critic predictions to the reward range. The unbounded linear head predicts values outside the binary-reward range [0, 1] (right), leading to unstable training reward (left) and AIME 2025 avg@32 (middle). The bounded value keeps predictions within the reward range and yields stable training. Most existing recipes use a linear head to predict the value directly, which is unbounded even when the return is known to lie in a finite interval. Let [Rmin , Rmax ] be the reward range and let zϕ (st ) ∈ R be the linear head output. Because the expectation of a bounded random variable lies in the same interval, a valid value prediction must satisfy Vϕ (st ) ∈ [Rmin , Rmax ]. We enforce this property with a scaled arctangent:  Vϕ (st ) = Rmin + (Rmax − Rmin )


1 1 + arctan zϕ (st ) . 2 π

(9)

This parameterization maps every finite head output to the open interval (Rmin , Rmax ) and approaches either endpoint asymptotically. The sanity test uses binary rewards, so Rmin = 0 and Rmax = 1. Empirically, it removes the extreme values produced by the linear head and allows the training reward to approach one (Figure 2). 4

Rewards

0.9 0.8 0.7 0.6 0.5

Biased critic Unbiased critic

0.4 0

AIME 2025 Avg@32

1.0

250 500 750 1000 1250 1500

0.25 0.23 0.20 0.17 0.15 0.12 0.10 0.08

Training Step

Explained Variance

How to Train a Critic Stably and Efficiently

0

250 500 750 1000 1250 1500

1.0 0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0

Training Step

0

250 500 750 1000 1250 1500

Training Step

Figure 3: Effect of the unbiased value target. Regressing to the final outcome gives more stable training reward (left) and AIME 2025 avg@32 (middle) than a target bootstrapped with λ = 0.99. Explained variance against the bootstrapped target rapidly approaches one (right) because that target is biased; this unexpectedly high value does not imply accurate prediction of the observed return. 3.3

S TEP 3: U SING U NBIASED M ONTE C ARLO VALUE TARGET

We track how much target variance the critic explains using   Var Vbt − Vϕ (st )   EV(Vϕ , Vb ) = 1 − . Var Vbt

(10)

An explained variance near one normally indicates a close fit to the chosen target. For the standard target in Equation (5), however, Vbt (λ) contains Vϕold whenever λ < 1. This self-referential target can be easy for the updated critic to predict even when it is inaccurate with respect to the observed return. In Figure 3, explained variance against the bootstrapped target rapidly approaches one while policy training remains unstable. Following decoupled GAE in VC-PPO (Yuan et al., 2025), we use separate parameters for the policy advantage and the critic target. We retain λπ = 0.99 for the policy, but set λV = 1 for critic training. With γ = 1 and outcome-only rewards, the target telescopes to bGAE(1) Vbt = A + Vϕold (st ) = R(x, y). t

(11)

For a continuation sampled from µ, the final outcome is an unbiased Monte Carlo sample of V µ (st ). Decoupling the estimators retains the variance reduction of λπ < 1 for the policy while removing bootstrapping from the critic target. This change improves reward stability and convergence speed in Figure 3; the reported explained variance is now measured against the observed outcome, and becomes reasonable. S TEP 4: R EMOVING A DVANTAGE N ORMALIZATION

Rewards

0.9 0.8 0.7

w. adv normalization wo. adv normalization

0.6 0

AIME 2025 Avg@32

1.0

250 500 750 1000 1250 1500

Training Step

0.28 0.26 0.24 0.22 0.20 0.18 0.16 0.14

60

Advantage Max & Min

3.4

0

250 500 750 1000 1250 1500

Training Step

40 20 0 20 40 60 0

250 500 750 1000 1250 1500

Training Step

Figure 4: Effect of removing batch-wise advantage normalization. Removing normalization achieves comparable reward (left) while mitigating the overfitting risk on AIME 2025 (middle). Normalization expands the range of the advantages when the policy approaches to optimal (right). 5

How to Train a Critic Stably and Efficiently

Many PPO implementations normalize advantages within each batch before the policy update. If Ā and σA denote the batch mean and standard deviation of the estimated advantages, respectively, this bt in Equation (1) and Equation (2) with replaces A b et = At − Ā . A σA

(12)

We find this transformation fundamentally problematic. As the model approaches optimal policy, the advantages approach zero with a small variance. The policy update should then naturally diminish to maintain its optimality. However, dividing by σA removes this behavior. A small standard deviation rescales estimation noise into a large training signal, so the policy update will not shrink when the policy is already close to optimal. In addition, subtracting Ā may change the sign of examples whose positive advantage is smaller than the batch mean, thus hamper exploration. We therefore remove this batch advantage normalization, using the raw GAE advantages to update policy. As shown in Figure 4, removing normalization keeps the advantage range small and stable, while using normalization causes an increasing advantage magnitude. Notably, removing advantage normalization also improves the validation performance, consistent with less aggressive updates after the training set has nearly been fit. S TEP 5: P ROVIDING P RIVILEGED I NFORMATION TO THE C RITIC

Rewards

0.9 0.8 0.7

wo. privileged information w. privileged information

0.6 0

250 500 750 1000 1250 1500

Training Step

AIME 2025 Avg@32

1.0

0.28 0.26 0.24 0.22 0.20 0.18 0.16 0.14

Explained Variance

3.5

0

0.4 0.2 0.0 0.2

250 500 750 1000 1250 1500

Training Step

0

250 500 750 1000 1250 1500

Training Step

Figure 5: Effect of giving the reference answer only to the critic. Privileged input accelerates training-reward improvement (left) and increases explained variance (right). Its AIME 2025 avg@32 rises faster but peaks earlier (middle), revealing a greater risk of overfitting in this small-data setting. The critic is needed only during training, so its inputs need not be identical to the policy’s inputs. This observation parallels centralized training with decentralized execution in multi-agent RL (Amato, 2024), including systems that expose hidden game state to a training-time critic (Vinyals et al., 2019; Wang et al., 2021). We apply the same principle to reward-defining information in LLM RL. Let q(x) denote information used to evaluate responses to prompt x. For mathematical reasoning, q(x) can be the reference answer. A privileged critic estimates Vϕµ (st , q(x)) ≈ Eµ [R(x, y; q(x)) | st , q(x)] .

(13)

Because q(x) is fixed by x, exposing it explicitly does not change the optimal value associated with a prompt. It can nevertheless reduce the approximation burden on a finite model. The rollout policy still receives only x and the generated prefix. In Figure 5, the reference answer improves critic fit and accelerates optimization. The validation curve also declines earlier after reaching its peak. Thus, a more informative critic can accelerate both learning and overfitting; privileged input is useful but not uniformly beneficial. 3.6

S TEP 6: A DOPTING L ENGTH -A DAPTIVE GAE

A fixed λπ < 1 gives the terminal reward exponentially less weight in longer responses. For a token at position t, the coefficient on the terminal TD residual is proportional to λTπ −t . When T −t is large, early-token advantages depend primarily on bootstrapped critic residuals, and systematic critic error can dominate the policy signal. 6

How to Train a Critic Stably and Efficiently

0.8 0.7

GAE = 0.99 GAE = 1 LA-GAE = 0.4

0.6 0

250 500 750 1000 1250 1500

Training Step

0.30 0.28 0.26 0.24 0.22 0.20 0.18 0.16 0.14

6000

Response Length

Rewards

0.9

AIME 2025 Avg@32

1.0

5000 4000 3000 2000

0

250 500 750 1000 1250 1500

Training Step

0

250 500 750 1000 1250 1500

Training Step

Figure 6: Effect of length-adaptive GAE. A fixed λπ = 0.99 attains rapid reward improvement (left), but its AIME 2025 avg@32 declines after the initial peak (middle). LA-GAE with α = 0.4 retains better training efficiency than λ = 1 while mitigating the overfitting risk.

Following VAPO and SAO (Yue et al., 2025; Hou et al., 2026), we use length-adaptive GAE. For a response of length L = |y|, the policy parameter is λπ (L) = 1 −

1 , αL

(14)

where α > 0 controls the bias–variance trade-off.
The coefficient of the terminal residual in the 1 L earliest-token advantage is approximately 1 − αL ≈ exp(−1/α), which is nearly invariant to response length. We continue to use λV = 1 for critic and do not use privileged inputs in this step. As shown in Figure 6, fixed λπ = 0.99 fits the training set fastest but exhibits a pronounced validation decline. Setting λπ = 1 avoids this decline at the cost of slower optimization. LA-GAE with α = 0.4 provides the best trade-off in this study.

4

B ROADER E VALUATION

We compare BPCO with two baselines. The group-based baseline uses Dr. GRPO (Liu et al., 2025) to estimate advantages, matching the total batch size with a group size of 16. The critic-based baseline adopts existing techniques including decoupled GAE (Yuan et al., 2025) for unbiased value target (Section 3.3) and length-adaptive GAE (Yue et al., 2025) for stable reward signal (Section 3.6), but retains an unbounded value head and batch-wise advantage normalization. All methods use DPPO (Qi et al., 2026b) for policy optimization, as defined in Sections 2.2 and 3.1. In all criticbased experiments, we adopt the single-rollout setting without a group for the same prompt. For the experiments below, BPCO differs from the critic-based baseline only by bounding the value prediction and removing batch-wise advantage normalization. We explicitly denote variants with privileged input for critic by BPCO+Ans or BPCO+Sol, where the privileged input is respectively the ground-truth answer or the official solution. The policy has the same inputs as every baseline. 0.35 0.33 0.30 0.28 0.25 0.23 0.20 0.18

AIME 2025 Avg@32

Group Baseline Critic Baseline BPCO + Ans BPCO + Ans + Sol 0

500

1000 1500 2000

Training Step

Rewards

0.80 0.75 0.70 0.65 0.60 0.55 0.50 0.45 0

500

1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00

1000 1500 2000

Training Step

Explained Variance

8000 7000 6000 5000 4000 0

500

1000 1500 2000

3000

Training Step

Figure 7: Results with a larger DeepScaleR Dataset.

7

Response Length

9000

0

500

1000 1500 2000

Training Step

How to Train a Critic Stably and Efficiently

4.1

S CALING TO L ARGER DATASET

We evaluate whether our BPCO recipe scales beyond the deliberately small sanity-test dataset. We fine-tune DeepSeek-R1-Distill-Qwen-1.5B (Guo et al., 2025) on DeepScaleR (Tan et al., 2026), which contains 40.3K math problem–answer pairs. Official solutions are available for approximately 7.3K of these problems. The maximum response length is limited at 24k. Figure 7 shows that our BPCO recipe consistently outperforms both group-based and critic-based baselines, improving both training and validation performance clearly. Compared to critic-based baseline, it demonstrates a consistently higher explained variance during the training process, which is a strong evidence that our BPCO recipe learns a better critic. These results show that with our BPCO recipe, a single-rollout critic can remain effective at a large-scale dataset.

AIME 2025 Avg@32 0.33

Rewards

0.75 0.70

0.30 0.28

0.65

0.25

0.60

0.23 0.20 0

500

BPCO + Ans 0.55 bounded value + adv normalization 0.50 1000

Training Step

1500

Advantage Max & Min

10 5 0 5

0

500

1000

Training Step

1500

0

500

1000

Training Step

1500

Figure 8: Ablation of bounded value prediction and batch-wise advantage normalization. We then ablate our proposed tricks based on a well-performed BPCO+Ans run, by removing the bounded value prediction and adding the batch-wise advantage normalization. Bounded Value Prediction. As shown in Figure 8, removing the value bound slows down training efficiency in rewards, and lowers AIME 2025 performance. This gap indicates that matching the critic output range to the return target remains useful when the training dataset is substantially larger. Removing Advantage Normalization. As shown in Figure 8, batch-wise advantage normalization produces a growing advantage magnitude, though it is not as obvious as in the sanity test. Removing advantage normalization brings only slight performance improvement in this setting because the training is not fully converged, however, we still recommend it as a universal solution. Privileged Information. Figure 9 ablates the effects of privileged information based on our BPCO recipe. With ground-truth answer as the privileged information, it produces significantly faster training speed, higher explained variance, and better AIME 2025 performance. Using official solution also performs slightly better, even only 7.3k out of 40.3k problems include this information. These results show that the privileged information can greatly help the critic training, when the dataset is large enough and overfitting doesn’t appear.

AIME 2025 Avg@32

0.33

Rewards 0.70

0.6

0.28

0.65

0.4

0.25

0.60

0.2

0.55

0.0

0.50

0.2

0.30

0.23

BPCO BPCO + Sol BPCO + Ans

0.20 0.18

0

250 500 750 1000 1250 1500

Training Step

0

250 500 750 1000 1250 1500

Training Step

Explained Variance

0.8

0

250 500 750 1000 1250 1500

Training Step

Figure 9: Ablation of privileged information for critic training.

8

How to Train a Critic Stably and Efficiently

Qwen3-30B-A3B

Qwen3-30B-A3B-Base

4.2

S CALING TO L ARGER M ODELS

AIME 2025 Avg@32

0.40

Rewards 0.50

0.30

Group Baseline Critic Baseline BPCO + Ans

0.10 0.00

0

200

400

600

Training Step

800

1000

0.60 0.50 0.40 0.30 0.20

5000

0.25

0.20

0

200

400

600

Training Step

800

1000

0.00

4000

0.25

3000

0.50

2000

0.75 1.00

1000 0

200

400

600

Training Step

800

1000

0.8

8000

0.6

7000

0.4

6000

0.2

5000

0.0

4000

0.2

3000

0.4

Response Length

6000

0

200

400

600

Training Step

800

1000

2000

0

200

0

200

400

600

800

1000

400

600

800

1000

Training Step

Training Step

Figure 10: Results with larger MoE models on DAPO-Math-17k dataset. We further evaluate BPCO with two larger models: Qwen3-30B-A3B-Base and Qwen3-30BA3B (Yang et al., 2025), using DAPO-Math-17k dataset (Yu et al., 2026) for training. Figure 10 shows that our BPCO recipe continues to improve the critic baseline at this scale. On Qwen3-30B-A3B, the critic baseline failed to further improve AIME 2025 after the first 100 training steps, suffering from an unstable optimization. In both settings, BPCO achieves a substantially higher accuracy on AIME 2025, indicating it learns a much better critic than previous recipe. Comparing to the group baseline, BPCO also performs better on Qwen3-30B-A3B, and comparable on Qwen3-30B-A3B-Base. This indicates that our BPCO is strong alternative to the widely adopted group-based method, without relying on a group sampling (Xu & Ding, 2026; Hou et al., 2026). 4.3

RUBRICS AS R EWARDS

Rubrics as Rewards

Validation Rewards 0.90 0.80 0.70 0.60 0.50 0.40 0.30

Training Rewards 0.8 0.6

Group Baseline Critic Baseline BPCO + Rubrics BPCO 0

50

100

150

Training Step

200

0.4 0.2

0

50

100

150

Training Step

200

0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 1.0

Explained Variance

0

50

100

150

Training Step

200

Figure 11: Results under rubric-based rewards. We finally consider a setting in which rewards are produced by a rubric-based judge (Gunjal et al., 2026). We train Qwen3-4B-Base (Yang et al., 2025) as both policy and critic on OpenRubrics (Liu et al., 2026), and use Qwen3-4B-Instruct-2507 as the judge. The judge assigns a reward from generated response and per-prompt golden rubrics. The golden rubrics are invisible to the policy, and we use these golden rubrics as privileged information for critic. 9

How to Train a Critic Stably and Efficiently

As shown in Figure 11, both BPCO variants improve faster than both group and critic baselines, while group baseline eventually converges to similar performance. The critic baseline converges to a slightly lower reward, due to its problematic recipe and low explained variance. The privileged information brings no benefit in performance despite its higher explained variance, likely because the task is relatively trivial. Nevertheless, the superior performance of BPCO without privileged information shows that bounded value prediction and unnormalized advantages remain useful when rewards come from a rubric-based judge.

5

C ONCLUSION

We studied why critic-based RL for LLMs can become unstable and assembled the resulting fixes into Best-Practice Critic Optimization. BPCO combines DPPO with reward-range-bounded value predictions, unbiased Monte Carlo critic targets, unnormalized policy advantages, and lengthadaptive GAE. It can additionally provide the training-only critic with privileged information, such as a ground-truth answer, solution, or grading rubric, while leaving the policy’s inputs unchanged. Across controlled sanity tests, larger datasets, 1.5B and 30B-A3B models, and rubric-based rewards, BPCO consistently improves the standard critic-based recipe. It also matches or exceeds groupbased optimization while using a single response per prompt. Privileged information further improve critic learning when they provide useful reward context, but their gains are task dependent and can be offset by overfitting in small-data regimes. Overall, these results show that the critic itself is not an inherent weakness of LLM RL. When its output range, target, inputs, and induced policy signal are designed coherently, it provides a stable and efficient alternative to group-based estimation. Limitations. Evidence is limited to mathematical and rubric rewards. BPCO assumes a known reward range, privileged variants require evaluator information, and critic training adds computation and memory not captured by trajectory-matched comparisons.

10

How to Train a Critic Stably and Efficiently

R EFERENCES Christopher Amato. An introduction to centralized training for decentralized execution in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2409.03052, 2024. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. In International Conference on Learning Representations, volume 2026, pp. 127924–127945, 2026. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Zhenyu Hou, Yujiang Li, Jie Tang, and Yuxiao Dong. Single-rollout asynchronous optimization for agentic reinforcement learning. arXiv preprint arXiv:2607.07508, 2026. Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 17417–17437, 2026. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730–27744, 2022. Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Defeating the training-inference mismatch via fp16. arXiv preprint arXiv:2510.26788, 2025. Penghui Qi, Zichen Liu, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Optimizing anytime reasoning via budget relative policy optimization. Advances in Neural Information Processing Systems, 38:23429–23451, 2026a. Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. Rethinking the trust region in llm reinforcement learning. arXiv preprint arXiv:2602.04879, 2026b. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279–1297, 2025. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. Sijun Tan, Michael Luo, Justin Wong, Colin Cai, Xiaoxiang Shi, William Yuan Tang, Manan Roongta, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Effective RL scaling of reasoning models via iterative context lengthening, 2026. URL https: //openreview.net/forum?id=I6GzDCne7U. 11

How to Train a Critic Stably and Efficiently

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350–354, 2019. Xiangjun Wang, Junxiao Song, Penghui Qi, Peng Peng, Zhenkun Tang, Wei Zhang, Weimin Li, Xiongjun Pi, Jujie He, Chao Gao, et al. Scc: An efficient deep reinforcement learning agent mastering the game of starcraft ii. In International conference on machine learning, pp. 10905– 10915. PMLR, 2021. Zhongwen Xu and Zihan Ding. Single-stream policy optimization. In International Conference on Learning Representations, volume 2026, pp. 140925–140944, 2026. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026. Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What’s behind ppo’s collapse in long-cot? value optimization holds the secret. arXiv preprint arXiv:2503.01491, 2025. Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025.

12

Related documents

• The roving critic#65456
• Algorithm Output Stability in Reinforcement Learning#929302
• Model-Free Output Feedback Stabilization via Policy Gradient Methods#1128550
• DynaKRAG: A Unified Framework for Learnable Evidence Control in Multi-Hop Retrieval-Augmented Generation#665839
• DynaKRAG: A Unified Framework for Learnable Evidence Control in Multi-Hop Retrieval-Augmented Generation#664339