Rethinking the Divergence Regularization in LLM RL

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Rethinking the Divergence Regularization in LLM RL

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

Rethinking the Divergence Regularization in LLM RL Jiarui Yao1,2,∗

Xiangxin Zhou1,∗ ¶

Wee Sun Lee3

Liefeng Bo1

arXiv:2606.09821v1 [cs.LG] 8 Jun 2026

1 Tencent Hunyuan ∗

Equal contribution

Tianyu Pang1,‡

2 UIUC

Project Lead

Penghui Qi3,∗ ¶

‡

3 NUS

Corresponding author

Abstract. Reinforcement learning (RL) has become a key component of post-training large language models (LLMs). In practice, LLM RL is often off-policy because of training-inference mismatch and policy staleness, making trust-region control essential for stable optimization. Mainstream methods such as PPO and GRPO approximate this control with a ratio-clipping mechanism, but the importance ratio can be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such as DPPO addresses this mismatch by replacing ratio-based clipping with a divergence-based mask, yielding a trust region defined by the sampled token’s absolute probability shift. However, DPPO still relies on a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded rather than corrected. To address this, we propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift. DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights that attenuate diverging updates and provide corrective signals beyond the boundary. Experiments across model scales, architectures, and precision settings show that DRPO improves the stability and efficiency of LLM RL training. Date: June 8, 2026 Code: https://github.com/Tencent-Hunyuan/UniRL/tree/main/DRPO

1 Introduction Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), enabling models to better align with human preferences and improve performance on complex reasoning tasks (Ouyang et al., 2022; Rafailov et al., 2023; Guo et al., 2025; Liu et al., 2025c). During training, an LLM is optimized as an autoregressive token-level policy that generates a response and receives a scalar reward from either a learned reward model (Ouyang et al., 2022) or a rule-based verifier (Guo et al., 2025; Yu et al., 2025). In practice, modern LLM RL is typically off-policy: rollouts are generated by inference engines whose numerical behavior differs from training engines (Qi et al., 2025; Yao et al., 2025), and collected trajectories are commonly split into multiple mini-batches or gradient steps (Liu et al., 2025a). As a result, the policy being updated is not identical to the behavior policy that generated the data. 1

PPO, A > 0

1.0

PPO, A < 0

SPO, A > 0

1.0

SPO, A < 0

0.8

0.6

0.4

0.2

0.0

0.2 0.4 0.6 0.8 1.0 0.0

0.2 0.4 0.6 0.8 1.0

DPPO, A > 0

DPPO, A < 0

DRPO, A > 0

DRPO, A < 0

1.0

0.8

0.6

0.4

0.2

0.0

0.2 0.4 0.6 0.8 1.0 0.0

2.0 1.5 1.0 0.5 0.0

normalized weight w

1.0

0.5 1.0 1.5 2.0

0.2 0.4 0.6 0.8 1.0

Figure 1: Per-token gradient weights of different algorithms as a function of the current probability π(yt |st ) and behavior probability µ(yt |st ). For SPO, ϵ = 1; for DRPO, δ = 1; for PPO, εlow = 0.2 and εhigh = 0.28; for DPPO, δ = 0.2. SPO’s weight grows without bound as µ(yt |st ) → 0, while the weight of DRPO remains bounded for all tokens. In such off-policy settings, Trust Region Policy Optimization (TRPO) provides a principled solution by maximizing a surrogate objective under an explicit divergence constraint between the current and behavior policy (Schulman et al., 2015; Achiam et al., 2017). However, its second-order optimization makes TRPO impractical to scale. Proximal Policy Optimization (PPO) (Schulman et al., 2017) replaces the constrained optimization with a simple ratio-clipping heuristic and has become the dominant recipe in modern LLM RL training. Building on PPO, GRPO improves practicality by replacing a learned critic with group-relative reward normalization (Shao et al., 2024; Ahmadian et al., 2024; Liu et al., 2025c). More recently, Simple Policy Optimization (SPO) (Xie et al., 2024) replaces hard clipping with a smooth quadratic regularizer that preserves the same ratio boundary while avoiding the zero-gradient issue outside the clipping range. These methods differ in implementation details, but they share the same trust-region geometry: the per-token update is controlled through its importance ratio. The importance ratio, however, is a poor proxy for distributional shift for LLMs due to large and long-tailed vocabularies (Qi et al., 2026). A small increase on a low-probability token can produce a very large ratio while changing little probability mass. Conversely, a moderate ratio change on a high-probability token can move substantial mass and meaningfully alter the policy. A fixed ratio window therefore tends to over-constrain low-probability tokens and under-constrain high-probability tokens (Qi et al., 2026; Yu et al., 2025; Chen et al., 2025). DPPO addresses this issue by replacing ratio-based clipping with a divergence-based mask (Qi et al., 2026). When the policy divergence exceeds a prescribed threshold and the current update would increase it further, DPPO disables the corresponding token gradient. Its Binary-TV variant, which we refer to as DPPO unless otherwise stated, measures the absolute probability shift of the sampled token. This quantity aligns more closely with total variation (TV) geometry than the importance ratio in long-tailed vocabularies. However, DPPO still enforces the trust region with a binary mask. Once a token moves outside the trust region in a harmful direction, its gradient is set to zero. This 2

prevents further movement away from the behavior policy, but it provides no corrective signal to move the policy back toward the boundary and can introduce abrupt changes near the threshold. We propose DRPO, a divergence-regularized policy optimization method that replaces the hard mask while preserving the Binary-TV trust region in DPPO. Our method is motivated by SPO, which places the per-token optimum exactly at PPO’s trust-region boundary through an advantage-weighted χ2 regularizer. We rewrite the Binary-TV constraint as a token-adaptive ratio bound and apply the same construction as SPO, which yields an advantage-weighted ℓ22 regularizer. The resulting regularizer changes the trust-region geometry from a fixed ratio constraint to an absolute probability-shift constraint, combining the smoothness of SPO with the divergence-based geometry of DPPO. Our method also gives a simple and stable gradient form. Each token’s policy-gradient contribution is multiplied by a continuous weight determined by its Binary-TV shift and by whether the current update moves away from or toward the behavior policy. When the update moves away from the behavior policy, the weight decays to zero at the trust-region boundary and becomes corrective beyond it. When the update moves back toward the behavior policy, the weight is amplified. Because this weight depends on an absolute probability shift rather than an importance ratio, it better captures the geometry of policy change and remains bounded even in the low-probability tail where SPO’s ratio-based weight can grow without bound. Beyond the specific algorithm, our results motivate a gradient-centered view of regularizer design for LLM RL. Our ablations show that standard KL or TV penalties can underperform because their gradients reintroduce ratio-based geometry. They also show that the per-token penalty should be weighted by the absolute-advantage because it keeps the trust-region boundary independent of reward scale. These findings suggest three practical criteria for an effective regularizer: it should induce a stable boundary aligned with distributional shift, keep per-token gradient weights bounded in the long-tailed vocabulary, and provide a smooth corrective signal when the policy moves away. DRPO satisfies these criteria with a simple Binary-TV-aligned regularizer, offering an empirical lens for designing stable policy-optimization objectives for LLMs.

2 Background The generation process of LLMs can be formulated as a token-level MDP (Bellman, 1957) M = (S, A, R, pX ). Given a prompt x ∼ pX , a response y = (y1 , . . . , yT ) is autoregressively sampled by a conditional stochastic policy π(yt |st ) over the vocabulary A, where the state st = (x, y1 , . . . , yt−1 ) ∈ S is the concatenation of the prompt and the generated tokens so far. The generation terminates upon producing the [eos] token or reaching the token limit. A scalar reward R(x, y) is then provided, either from a reward model (Ouyang et al., 2022) or a rule-based verifier (Guo et al., 2025). The policy objective is to maximize the expected reward: J (π) = Ex∼pX [J (x, π)] = Ex∼pX Ey∼π(·|x) [R(x, y)] . Modern RL frameworks for LLM fine-tuning rely on highly optimized training and inference engines to maximize throughput, which inevitably introduces subtle but non-negligible numerical discrepancies (Qi et al., 2025; Yao et al., 2025). A further common practice is to collect a large batch of rollouts and split it into multiple mini-batches for multiple gradient updates (Liu et al., 2025a). Both cases bring RL training into an off-policy paradigm, where the data is sampled from a behavior policy µ

(1)

t=1

2.1 Trust Region Policy Optimization Directly optimizing Equation (1) often suffers from high variance due to the product of importance sampling ratios. TRPO (Schulman et al., 2015) handles this with a token-level surrogate objective:   |y| X π(yt |st ) L(x, π) = Ey∼µ(·|x)  · Ât  , (2) µ(yt |st ) t=1

where Ât = R(x, y) − V (st ) is the advantage estimate, and V (st ) is a variance-reduction baseline that does not change the expected policy gradient. Typically, V (st ) is set to the expected reward conditioned on state st . TRPO and later work (Achiam et al., 2017; Qi et al., 2026) have shown that this surrogate is a first-order approximation1 of Equation (1), and a monotonic performance improvement can be guaranteed within a trust region defined by the KL divergence or TV distance. Formally, TRPO solves the following constrained optimization problem:   |y| X max L(x, π) s.t. Ey∼µ(·|x)  DTV (µ(·|st )∥π(·|st )) ≤ δ. (3) π

t=1

2.2 Proximal Policy Optimization TRPO requires second-order methods that are computationally prohibitive at scale. PPO (Schulman et al., 2017) was introduced as a simple alternative that approximates the trust region via a ratiot |st ) clipping mechanism. Letting rt ≜ π(y µ(yt |st ) denote the per-token importance ratio, PPO optimizes:   |y| X LPPO (x, π) = Ey∼µ(·|x)  min rt · Ât , clip(rt , 1 − ϵ, 1 + ϵ) · Ât  . (4) t=1

The clipping mechanism deactivates the gradient whenever rt leaves the interval [1 − ϵ, 1 + ϵ] and further increase the loss, thereby enforcing a per-token, ratio-based trust region, i.e., |rt − 1| ≤ ϵ. Group Relative Policy Optimization. In traditional RL settings, V (st ) is typically estimated by a critic model. Learning such a critic is, however, expensive and noisy for LLMs. To address this, Shao et al. (2024); Ahmadian et al. (2024); Liu et al. (2025c) propose sampling a group of 1 PG G responses {yi }i=1 per prompt and estimating the advantage as Ât,i = R(x, yi ) − G j=1 R(x, yj ). This critic-free approach is widely known as Group Relative Policy Optimization (GRPO). 2.3 Simple Policy Optimization While effective in practice, PPO enforces its trust region through a hard clipping rule. This mechanism is brittle near the clipping boundary: a small change in rt can abruptly switch a token’s gradient 1. We adapt TRPO to the LLM setting and ignore a constant term; see Qi et al. (2026) for a rigorous derivation.

from active to zero. Moreover, once a token has moved outside the clip range in a harmful direction, PPO removes its gradient entirely and provides no corrective signal back toward the trust region. SPO (Xie et al., 2024) addresses these issues by replacing the hard clip with a smooth quadratic regularizer:   |y| X | Â | t LSPO (x, π) = Ey∼µ(·|x)  rt · Ât − (rt − 1)2  . (5) 2ϵ t=1

For each token, the integrand is a concave quadratic in rt . Setting its derivative Ât − |Âϵt | (rt − 1) to zero gives the unique maximizer rt⋆ = 1 + sign(Ât )ϵ, which exactly matches PPO’s clipping boundary in Equation (4). SPO therefore preserves the same ratio-based trust region as PPO, but enforces it through a continuous gradient weight. 2.4 Divergence Proximal Policy Optimization PPO, GRPO, and SPO all derive their trust region from the per-token ratio rt . Qi et al. (2026) argues that this estimator is poorly behaved over LLMs’ long-tailed vocabulary: a low-probability token can produce an enormous ratio (e.g., 10−5 → 10−3 ) while contributing negligibly to the actual distributional shift, whereas a high-probability token may exhibit a modest ratio (e.g., 0.99 → 0.80) that nevertheless induces a substantial change in policy. Ratio-based trust regions thus over-penalize low-probability tokens, which are often exploratory, and under-penalize high-probability ones, harming both efficiency and stability. DPPO conditioned DPPO (Qi et al., 2026) replaces the ratio-based clip with a divergence-based mask Mt on the policy divergence Dt ≜ D µ(·|st ) ∥ π(·|st ) , where D is either the TV or KL divergence over the full per-state token distributions. The DPPO objective and mask are   |y| X LDPPO (x, π) = Ey∼µ(·|x)  MtDPPO · rt · Ât  , t=1

(6)

( 0, sign Ât · (rt − 1) > 0 and Dt > δ, DPPO Mt = 1, otherwise, with divergence threshold δ. The mask zeros the gradient only when the policy has already moved outside the trust region in a direction that would push it further away. For tractability over large vocabularies, DPPO approximates Dt with binary or top-k surrogates. Most relevant to our method is the Binary-TV approximation, which collapses the per-state distribution into a Bernoulli over the sampled token versus the rest, yielding DtBin-TV ≜ π(yt |st ) − µ(yt |st ) .

(7)

The corresponding trust region π(yt |st ) − µ(yt |st ) ≤ δ constrains the absolute probability shift on the sampled token, in contrast to the relative ratio constraint |rt − 1| ≤ ϵ shared by PPO and SPO.

Compared with PPO in Equation (4), DPPO replaces the fixed ratio interval with an adaptive one whose width is inversely proportional to the behavior probability of the sampled token. Lowprobability tokens therefore receive a looser ratio tolerance, while high-probability tokens receive a tighter one. This constraint avoids the main failure mode of ratio-based trust regions, which can over-penalize rare tokens and under-penalize common ones (Qi et al., 2026). However, DPPO still enforces this divergence-based trust region through a binary mask, which makes the update brittle near the boundary: a small change in the estimated divergence can abruptly switch a token’s gradient from full strength to zero. The key lesson from SPO is that the same trust-region boundary can be enforced by a smooth regularizer instead of a discontinuous cutoff. Such a regularizer induces a continuous gradient weight that varies with both the magnitude and direction of the probability shift. Inside the boundary, it smoothly reweights the policy gradient; outside the boundary, it provides a corrective mechanism that can pull the policy back toward the trust region. We apply this principle to the Binary-TV trust region by replacing DPPO’s mask with a quadratic regularizer on the sampled token’s absolute probability shift. The resulting objective, Divergence Regularized Policy Optimization (DRPO), is   |y| X | Â | t LDRPO (x, π) = Ey∼µ(·|x)  rt Ât − µ(yt |st ) (rt − 1)2  . (8) 2δ t=1

The first term is the token-level surrogate in Equation (2). The second term is a quadratic regularizer whose curvature is scaled by the behavior probability of the sampled token. This single factor changes the equilibrium from a fixed ratio shift, as in PPO and SPO, to a fixed absolute probability shift, as required by DPPO. Taking the gradient of Equation (8) gives (see Appendix B for a full derivation)   |y| Bin-TV X D ∇LDRPO (x, π) = Ey∼µ(·|x)  1 − sign(Ât (rt − 1)) t rt Ât ∇ log π(yt |st ) . (9) δ t=1

Relative to the unregularized gradient of Equation (2), DRPO multiplies each token’s policy-gradient contribution by a continuous weight wt = 1 − sign(Ât (rt − 1))

DtBin-TV . δ

(10)

The sign term indicates whether the current update moves the sampled probability away from or toward the behavior policy. The magnitude term measures the Binary-TV shift that should be controlled. Together, these terms make the weight vary smoothly with both the size and direction of the sampled token’s probability shift. 3.1 Trust Region Analysis We now examine how the smooth gradient weight in Equation (10) encodes the trust-region boundary. 6

Table 1: Comparison of trust-region mechanisms. DPPO and DRPO enforce a Binary-TV constraint on the sampled token’s absolute probability shift. Because TV is bounded in [0, 1], DRPO produces bounded gradient weights, whereas the ratio-based constraint in SPO does not. Method

Mechanism

Trust-region constraint

Gradient weight wt

Range of wt

PPO SPO DPPO DRPO

hard clip smooth regularizer hard mask smooth regularizer

|rt − 1| ≤ ϵ |rt − 1| ≤ ϵ |π(yt |st ) − µ(yt |st )| ≤ δ |π(yt |st ) − µ(yt |st )| ≤ δ

0 or 1 1 − sign(Ât (rt −1)) |rt − 1|/ϵ 0 or 1 1 − sign(Ât (rt −1)) |π(yt |st ) − µ(yt |st )|/δ

{0, 1} (−∞, +∞) {0, 1} [1 − 1δ , 1 + 1δ ]

Diverging update (sign(Ât (rt − 1)) > 0). When the update moves π(yt |st ) away from µ(yt |st ), the weight becomes wt = 1 − DtBin-TV /δ. Thus the gradient is gradually attenuated as the Binary-TV shift approaches the boundary. Inside the trust region, where DtBin-TV < δ, the weight remains positive and the update still follows the reward-improving direction. Outside the trust region, where DtBin-TV > δ, the weight is negative, so the gradient reverses and provides a corrective signal back toward the trust region. Since the per-token objective in Equation (8) is a concave quadratic in rt , the zero-weight condition gives the stationary point π(yt |st )⋆ = µ(yt |st ) + sign(Ât ) δ,

(11)

which matches DPPO’s trust region boundary when the same threshold δ is used. Converging update (sign(Ât (rt − 1)) < 0). When the update moves π(yt |st ) toward µ(yt |st ), the weight becomes wt = 1 + DtBin-TV /δ. The gradient is therefore amplified rather than suppressed, encouraging the policy to move smoothly back toward the behavior policy. Takeaway. The two cases show that DRPO preserves the same trust-region boundary as DPPO when the same threshold δ is used, but replaces the brittle hard mask with continuous gradient reweighting. Inside the boundary, tokens continue moving in the reward-improving direction with smoothly attenuated gradients. Outside the boundary, the gradient reverses and provides a corrective signal back toward the trust region. 3.2 Comparison with SPO To justify why the probability factor in Equation (8) is essential, we compare DRPO and SPO from two perspectives: the divergence each method implicitly regularizes, and the stability of the resulting per-token gradient weight. Table 1 summarizes the key design differences across the four objectives. Implicit regularizer: ℓ22 versus χ2 . For a fixed state st , write Ât (a) for the advantage that would be assigned when the sampled token is a. The regularization term in DRPO has expectation h i X 2 Eyt ∼µ(·|st ) |Ât (yt )| µ(yt |st )(rt − 1)2 = |Ât (a)| π(a|st ) − µ(a|st ) . a∈A

This is an advantage-weighted Pearson-χ2 penalty. The advantage weights modulate which tokens matter more for learning, but the key geometric difference comes from the denominator µ(a|st ). SPO scales each squared probability shift by 1/µ(a|st ), making the penalty highly sensitive to deviations on low-probability tokens. DRPO instead penalizes the absolute probability shift directly: at a fixed advantage value, the same shift |π(a|st ) − µ(a|st )| receives the same cost regardless of the token’s behavior probability. In this sense, the ℓ22 -type penalty is symmetric in π and µ, whereas the χ2 -type penalty is tied to the behavior policy and can be dominated by the low-probability tail of µ.

Figure 2: Histogram, cumulative distribution, and absolute probability shift |π − µ| of rollout probabilities µ(yt |st ) for tokens sampled from Qwen3-30B-A3B-Base (Yang et al., 2025). The shift |π − µ| reflects training-inference mismatch. Tokens with µ(yt |st ) ≤ 0.01 account for 7.8% of all sampled tokens, showing that the low-probability tail is sampled non-negligibly often. Gradient stability in the long tail. A similar distinction appears in the gradient weights. From Table 1, SPO weights each token by a term involving |rt − 1|. Under yt ∼ µ(·|st ), this quantity is an unbiased single-sample Monte Carlo estimator of the unnormalized TV distance (Qi et al., 2026): X Eyt ∼µ(·|st ) [|rt − 1|] = |π(a|st ) − µ(a|st )| = 2 DTV µ(·|st ) ∥ π(·|st ) . a∈A

Its variance, however, is Varyt ∼µ(·|st ) (|rt − 1|) = χ2 (π(·|st ) ∥ µ(·|st )) − 2DTV µ(·|st ) ∥ π(·|st )

The χ2 term contains the factor 1/µ(a|st ), so the variance can become arbitrarily large when probability mass shifts on tokens with very small behavior probability. This is the typical long-tail regime of LLM sampling. As Figure 2 shows, tokens with µ(yt |st ) ≤ 0.01 account for 7.8% of all sampled tokens. For these tokens, even a modest absolute probability shift can induce a large ratio change, causing the SPO weight 1 ± |rt − 1|/ϵ to dominate the gradient despite a small contribution to the actual distributional shift. DRPO avoids this instability by replacing |rt − 1| with |π(yt |st ) − µ(yt |st )|, which directly measures absolute probability shift and more faithfully reflects the geometry of TV divergence (compare the right panel of Figure 2 with Figure 1 of Qi et al. (2026)). Since it is bounded in [0, 1] for every token, its variance is bounded by 1/4, and the gradient weight of DRPO is confined to 1 − 1δ ≤ wt ≤ 1 + 1δ . Figure 1 illustrates this contrast. SPO’s weight grows without bound along the low-µ axis, whereas DRPO remains bounded everywhere. Thus DRPO realizes a smooth version of DPPO’s divergence-based trust region while avoiding the high-variance weighting induced by ratio-based regularization.

4 Experiments and Results Models, Data, and Benchmarks. We perform RL fine-tuning on Qwen3-4B-Base, Qwen3-30BA3B-Base, and Qwen3.5-35B-A3B-Base (Yang et al., 2025), using a filtered subset of the original DAPO dataset (Yu et al., 2025) that contains approximately 13K math problems with rule-based verification. In addition, we fine-tune DeepSeek-R1-Distill-Qwen-1.5B (R1D) (Guo et al., 2025) on a small sanity test dataset of 1,460 solvable questions (Qi et al., 2025). During training, we evaluate on AIME 2024 and AIME 2025 (MAA, 2025). For each problem, we sample 16 responses and report the average score. Experimental Settings. We use the VeRL framework (Sheng et al., 2024) for RL training, with BF16 precision by default. For Qwen3-30B-A3B-Base, we additionally consider two low-precision settings: FP8 for rollout only, and FP8 for both training and rollout (FP8-E2E). These settings make optimization more challenging because FP8 precision, together with the MoE architecture, can increase the numerical mismatch between training and inference. Across all settings, we evaluate the unregularized trust-region-free surrogate (Equation (2)), GRPO (Equation (4)), SPO (Equation (5)), DPPO (Equation (6)), and our proposed DRPO (Equation (8)). For GRPO, we adopt the cliphigher trick with ϵlow = 0.2 and ϵhigh = 0.28, following Yu et al. (2025). For DPPO, we use the recommended value δ = 0.15. For SPO and DRPO, we set the regularization threshold to 12.5. For other hyperparameters and hardware requirements, please refer to Appendix D and Table 2.

Qwen3-30B-A3B-Base

FP8 Rollout

R1D

0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05

0.325

100

150

200

250

Training Step

300

Qwen3.5-35B-A3B-Base 0.70

0.250 0.225 0.200 0.175

0 0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00

100

150

200

Training Step

250

300

0.60 0.55 0.50 0

Training Step

GRPO Clip-Higher

100

FP8 E2E

500

1000 1500 2000 2500 3000

Training Step

Qwen3-4B-Base

0.25 0.20

Accuracy

0.65

0.275

Accuracy

0.300

Accuracy

0.45 0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05

0.15 0.10 0.05

100

150

200

Training Step

GRPO No-Clip

250

DPPO

200

400

Training Step

SPO

600

DRPO

Figure 3: Average accuracy across all main experiment settings on AIME24 and AIME25.

800

4.1 Main Results We present the main results in Figure 3 (see Appendix D.1 for comparing with KL regularization). Across all six settings, our DRPO consistently enables stable and efficient training, matching or exceeding the best evaluation accuracy achieved by the baselines. Instability of ratio-based methods. We find that ratio-based methods, namely GRPO and SPO, generally suffer from unstable training. This issue is especially severe in the low-precision settings, where they often collapse before reaching reasonable performance. Even in their strongest settings, their training efficiency and final accuracy lag behind their divergence-based counterparts. This observation is consistent with Qi et al. (2026), which shows that |rt − 1| is a poor proxy for the true divergence and that ratio-based trust regions can lead to unstable and inefficient optimization. Limitations of a hard mask. Another observation is that hard-mask methods, such as GRPO and DPPO, often underperform their counterparts with smooth regularization. For example, although DPPO trains stably on Qwen3-30B-A3B-Base, it often converges more slowly and reaches lower final accuracy than DRPO. This supports our main claim that a smooth gradient signal is more effective in practice than a brittle hard mask. The need for a proper trust region. In some cases, the unregularized trust-region-free surrogate in Equation (2) already achieves strong performance, while a hard mask or ratio-based trust region can degrade performance. However, this unregularized surrogate is not reliable across settings, suffering a performance drop in three of the six settings. The most notable example is in the Qwen3-4B-Base experiment, where the accuracy decreases from 0.25 to 0.17. These results support the claim of Qi et al. (2026) that a trust region remains necessary, but suggest that its form is crucial. Overall, DRPO combines the stability of divergence-based trust regions with the flexibility of a smooth regularizer, yielding the best overall performance across our experiments. 4.2 Ablation Studies 0.350 0.325 0.300 0.275 0.250 0.225 0.200 0.175 0.150 0

R1D SPO SPO wo. |At| DRPO DRPO wo. |At|

Accuracy

Qwen3-30B-A3B-Base

Accuracy

0.45 0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05

100 150 200 250 300

Training Step

500 1000 1500 2000 2500 3000

Training Step

Figure 4: Ablation on |Ât |. Removing this term degrades performance and destabilizes training. To further evaluate the effectiveness of our proposed method, we conduct a series of ablation studies on the design considerations of the regularizer. Advantage weight. In both SPO and DRPO, the regularization term is weighted by the absolute advantage |Ât |. This weighting ensures that the per-token optimum lies on a stable trust-region boundary that does not depend on the magnitude of the advantage. Without this weighting, the trust-region boundary in Equation (11) would be coupled with |Ât |, making it sensitive to token-level 10

advantage noise and group-level advantage variance. However, this choice also makes the regularizer advantage-weighted rather than a pure divergence, as used in many prior works (Luo et al., 2026; Becker et al., 2025). To examine whether |Ât | is necessary, we conduct ablations on Qwen3-30B-A3B-Base FP8-E2E and R1D by removing this factor from SPO and DRPO (see Appendix D.2 for this ablation on other alternative regularizations). As shown in Figure 4, removing |Ât | consistently causes a performance drop and leads to training instability. These results suggest that maintaining a stable trust-region boundary is more important than enforcing a pure divergence form for the regularizer. This behavior is reasonable because |Ât | also determines the scale of the per-token policy gradient. Scaling the regularizer by |Ât | preserves the same relative corrective strength across tokens with different advantage magnitudes. Without this scaling, tokens with small advantages can be over-regularized, while tokens with large advantages can move too far before receiving sufficient correction. Other alternative regularizations. As shown in Section 3.2, the regularizer in DRPO can be interpreted as an advantage-weighted ℓ22 penalty, whereas the regularizer in SPO corresponds to an advantage-weighted χ2 divergence. This raises a natural question: can other divergence measures yield better performance? To answer this question, we compare DRPO with several alternatives, including commonly used forward KL and TV penalties (Equation (12) and 14). As shown in Figure 5, all of these alternatives underperform DRPO. We argue that this result is expected because their per-token gradients induce either binary or ratio-based optima rather than a smooth Binary-TV boundary, with the detailed analysis deferred to Appendix C. In contrast, DRPO induces a Binary-TV trust region, which provides more stable gradients and better captures the true distributional shift, as detailed in Section 3.2. 0.350 0.325 0.300 0.275 0.250 0.225 0.200 0.175 0.150 0

R1D SPO TV DRPO KL

Accuracy

FP8 E2E

Accuracy

0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0

100

150

200

Training Step

250

500 1000 1500 2000 2500 3000

Training Step

Figure 5: Ablation on alternative divergence metrics. DRPO achieves the best performance. Applying the regularizer only outside DPPO’s trust region. To examine where the performance gain of DRPO primarily comes from, we conduct an experiment in which the regularizer is applied only outside the DPPO trust region. We refer to this variant as Mask-DRPO. Within the DPPO trust-region boundary, Mask-DRPO has the same gradient as DPPO; outside this boundary, it has the same gradient as DRPO. As shown in Figure 11, Mask-DRPO achieves performance comparable to DRPO, suggesting that the main performance gain comes from the corrective regularization outside the trust region. In addition, other regularizer alternatives still do not match DRPO’s performance, further supporting the effectiveness of our design. See Appendix D.6 for more details.

5 Closing Remarks Many prior works design regularizers from the objective perspective, typically by adopting standard divergence measures such as KL, JS, or related variants. Our empirical results suggest that the induced gradient form is more critical than the nominal divergence in the objective. For example, although the absolute-advantage term in Equation (5) and Equation (8) prevents the regularizer from being a pure divergence, we find that it is essential for maintaining a stable trust-region boundary and enabling stable training. The choice of regularizer therefore requires careful consideration. A regularization term that appears reasonable at the objective level can perform poorly if its gradient induces undesirable geometry. In particular, we identify a common failure mode in which the gradient induces a ratio-based trust region, whose weights can have high variance and become unbounded under the long-tailed vocabularies of LLMs. In contrast, the absolute probability shift, namely Binary-TV, provides a better alternative: it is bounded and better captures the geometry of TV divergence. This observation is consistent with DPPO (Qi et al., 2026), which replaces ratio-based clipping in PPO with a divergence-based mask. However, DPPO still relies on a hard mask, whose effective gradient changes abruptly near the mask boundary and provides no corrective signal outside the trust region. To address this, we propose DRPO, which replaces the hard mask with a smooth quadratic regularizer while preserving the same trust-region geometry. Across dense and MoE architectures, reasoning and non-reasoning models, and BF16 and FP8 precision settings, DRPO improves training stability and achieves stronger performance than a diverse set of baselines.

References Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22–31. PMLR, 2017. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248–12267, 2024. Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167–175, 2003. Philipp Becker, Niklas Freymuth, Serge Thilges, Fabian Otto, and Gerhard Neumann. Troll: Trust regions improve reinforcement learning for large language models. arXiv preprint arXiv:2510.03817, 2025. Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, 1957. Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025a. Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Yu Shen. When speed kills stability: Demystifying rl collapse from the inference-training mismatch, 2025b. https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-fromthe-Inference-Training-Mismatch-271211a558b7808d8b12d403fd15edda. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025c. Yu Luo, Shuo Han, Yihan Hu, Dong Li, and Jianye Hao. Ratio-variance regularized policy optimization for efficient llm fine-tuning. arXiv preprint arXiv:2601.03320, 2026. MAA. American invitational mathematics examination - aime. https://maa.org/, 2025. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– 27744, 2022. Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Defeating the training-inference mismatch via fp16. arXiv preprint arXiv:2510.26788, 2025. Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. Rethinking the trust region in llm reinforcement learning. arXiv preprint arXiv:2602.04879, 2026. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025a. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. Ling Team, Anqi Shen, Baihui Li, Bin Hu, Bin Jing, Cai Chen, Chao Huang, Chao Zhang, Chaokun Yang, Cheng Lin, et al. Every step evolves: Scaling reinforcement learning for trillion-scale thinking model. arXiv preprint arXiv:2510.18855, 2025b. Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=aBO5SvgSt1. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939, 2025. Yuhui Wang, Hao He, Xiaoyang Tan, and Yaozhong Gan. Trust region-guided proximal policy optimization. Advances in Neural Information Processing Systems, 32, 2019. Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In Uncertainty in artificial intelligence, pages 113–122. PMLR, 2020. Zhengpeng Xie, Qiang Zhang, Fan Yang, Marco Hutter, and Renjing Xu. Simple policy optimization. arXiv preprint arXiv:2401.16025, 2024. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, August 2025. https://fengyao.notion.site/offpolicy-rl. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025.

Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892, 2025. Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, et al. Stabilizing reinforcement learning with llms: Formulation and practices. arXiv preprint arXiv:2512.01374, 2025a. Haizhong Zheng, Jiawei Zhao, and Beidi Chen. Prosperity before collapse: How far can off-policy rl reach with stale data on llms? arXiv preprint arXiv:2510.01161, 2025b.

Appendix A. Related Work A.1 Traditional RL based on Trust Region Methods Trust region methods ensure stable policy optimization by limiting how much the policy can change in each update. TRPO (Schulman et al., 2015) derives a policy improvement bound penalized by TV divergence and solves the resulting KL-constrained optimization via conjugate gradient, guaranteeing monotonic improvement. CPO (Achiam et al., 2017) extends this to constrained MDPs. However, both require second-order optimization that is prohibitive at scale. PPO (Schulman et al., 2017) replaces the explicit KL constraint with a ratio-clipping heuristic, enabling first-order optimization. Despite its success, the clipping mechanism neither strictly bounds the likelihood ratio nor enforces a well-defined divergence constraint (Wang et al., 2020). Truly PPO (Wang et al., 2020) addresses this by introducing a rollback clipping function with a KLbased triggering condition. Trust Region-Guided PPO (Wang et al., 2019) proposes adaptive clipping thresholds guided by KL divergence, providing stronger guarantees than fixed-width clipping. MDPO (Tomar et al., 2022) connects trust-region policy optimization with mirror descent (Beck and Teboulle, 2003), approximately solving the trust-region subproblem via multiple gradient steps on a Bregman divergence objective rather than enforcing a hard constraint. Most relevant to our work, SPO (Xie et al., 2024) replaces PPO’s hard clipping with a smooth quadratic regularizer on the importance ratio. The per-token optimum of the resulting concave quadratic exactly matches PPO’s clipping boundary, while providing non-zero corrective gradients outside the trust region. Our method adopts SPO’s smooth regularization principle but changes the trust-region geometry from ratio-based to divergence-based. Specifically, we weight SPO’s quadratic penalty by the behavior probability µ(yt |st ), which transforms the implicit regularization from a χ2 type penalty to an ℓ22 -type penalty on probability shifts. This single modification changes the per-token optimum from the ratio boundary |rt − 1| = ϵ to the Binary-TV boundary |π(yt |st ) − µ(yt |st )| = δ, inheriting the smooth gradient structure of SPO while aligning the trust region with the TV geometry of DPPO. A.2 RL for LLM Reasoning Reinforcement learning has become a key technique for improving reasoning in LLMs (Guo et al., 2025; Team et al., 2025a). In practice, LLM RL is inherently off-policy due to training-inference mismatch (Yao et al., 2025; Qi et al., 2025) and mini-batch policy staleness (Liu et al., 2025a), making trust-region optimization essential for stable training. The dominant approach uses PPO-style hard clipping to impose ratio-based trust regions. GRPO (Shao et al., 2024; Liu et al., 2025c) retains this objective while replacing critic-based advantages with group-relative advantages (Liu et al., 2025c; Zeng et al., 2025). DAPO (Yu et al., 2025) asymmetrically widens the upper clipping bound, CISPO (Chen et al., 2025) removes clipping through truncated importance sampling, and M2PO (Zheng et al., 2025b) constrains the second moment of importance weights. To reduce variance under off-policy data, prior work has also proposed truncated (Yao et al., 2025; Zheng et al., 2025a) and masked (Liu et al., 2025b; Team et al., 2025b) importance sampling. Another line of work uses regularization to enforce trust-region behavior instead of relying on hard clipping or masking. Kimi k1.5 (Team et al., 2025a) and Kimi k2.5 (Team et al., 2026) adopt online policy mirror descent. R2 VPO (Luo et al., 2026) replaces hard clipping with a smooth Lagrangian

penalty on ratio variance, but it remains ratio-based and can induce unbounded gradient weights for low-probability tokens. TROLL (Becker et al., 2025) enforces per-token KL constraints through differentiable projections, but requires solving an optimization problem for each token. These mask-based and regularizer-based methods either remain tied to the importance ratio or adjust the trust region heuristically, without directly resolving the mismatch between ratio change and distributional shift. DPPO (Qi et al., 2026) identifies this flaw in long-tailed vocabularies (Wang et al., 2025) and replaces ratio clipping with a divergence-based binary mask on TV or KL divergence. However, DPPO still changes gradients abruptly at the boundary and provides no corrective signal once a token moves outside the trust region. DRPO combines the divergence-based geometry of DPPO with the smooth enforcement principle used by regularizer-based methods such as R2 VPO, while avoiding their main limitations. DRPO preserves the directional structure of PPO and DPPO: it attenuates updates that move the policy away from the behavior policy and amplifies updates that move it back. Through a lightweight advantage-weighted ℓ22 regularizer, DRPO aligns the update with Binary-TV geometry, provides smooth corrective gradients, and keeps per-token gradient weights bounded.

Appendix B. Detailed Derivation of the Gradient of DRPO The gradient of the objective in Equation (8) can be derived as follows: ∇LDRPO (x, π)   |y| X | Â | t = Ey∼µ(·|x)  ∇rt Ât − µ(yt |st )(rt − 1)∇rt  δ t=1   |y| X µ(yt |st )(rt − 1) 1 − sign(Ât ) = Ey∼µ(·|x)  ∇rt Ât  δ t=1   |y| X |π(yt |st ) − µ(yt |st )| = Ey∼µ(·|x)  1 − sign(Ât (rt − 1)) rt Ât ∇ log π(yt |st ) δ t=1   |y| Bin-TV X D = Ey∼µ(·|x)  1 − sign(Ât (rt − 1)) t rt Ât ∇ log π(yt |st ) . δ t=1

Appendix C. Induced Trust Regions of Alternative Regularizers We analyze the trust region induced by each alternative regularizer through its per-token gradient. Fix a state st and a sampled token yt . For compactness, denote µt ≜ µ(yt |st ),

πt ≜ π(yt |st ),

rt ≜

πt . µt

This appendix is intended to clarify a subtle point in regularizer design. Two objectives can look similar at the loss level but induce very different gradient geometries after importance sampling. For LLM RL, this distinction is important because the optimization update is driven by sampled tokens from a highly long-tailed vocabulary. A useful trust-region regularizer should therefore be judged 17

not only by the name of the divergence it resembles, but also by the scalar weight it applies to the token-level policy gradient. We consider the following alternative regularizers:   |y| X |Ât | LKL (x, π) = Ey∼µ(·|x)  rt · Ât + · log rt  , 2δ t=1   |y| X |Ât | rt · Ât − · (rt − 1 − log rt ) , LK3 (x, π) = Ey∼µ(·|x)  2δ t=1   |y| X | Â | t rt · Ât − · |rt − 1| , LTV (x, π) = Ey∼µ(·|x)  2δ

(12)

(13)

(14)

t=1

Since µt is fixed during the policy update, ∇rt = rt ∇ log πt . We therefore write each gradient as the original policy-gradient term rt Ât ∇ log πt multiplied by an induced weight. The zero of this weight gives the boundary at which the regularizer cancels the reward-improving gradient. When this boundary is expressed as a fixed value of rt , the regularizer inherits the same ratio-based geometry as PPO and SPO. When the boundary is expressed as a fixed value of |πt − µt |, it matches the Binary-TV geometry used by DRPO and DPPO. Advantage-weighted KL regularizer. Consider the per-token KL-regularized objective ℓKL (rt ) = rt Ât +

|Ât | log rt . 2δ

(15)

This is the sampled contribution of the forward KL penalty DKL (µ∥π) under the behavior-policy expectation, up to the sign convention induced by maximizing the objective. Taking the gradient gives ! |Ât | ∇ℓKL (rt ) = rt Ât + ∇ log πt 2δ ! sign(Ât ) rt Ât ∇ log πt . (16) = 1+ 2δrt Thus the KL-induced gradient weight is wKL (rt ) = 1 +

sign(Ât ) . 2δrt

A key observation is that the gradient weight only depends on rt , which leads to a ratio-based geometry. Setting wKL (rt ) = 0 yields rt⋆ = −

sign(Ât ) . 2δ

(17)

For Ât > 0, this equation has no feasible solution because rt > 0 and the gradient weight is always positive. For Ât < 0, the zero-gradient point is rt⋆ =

1 , 2δ

πt⋆ = 18

µt . 2δ

Therefore, whenever the KL penalty induces a finite cancellation boundary, that boundary is ratiobased. The stopping condition depends on πt /µt , not on the absolute probability shift. This also explains why directly adding a KL penalty is not a drop-in replacement for DRPO. For positive-advantage tokens, the sampled forward-KL term does not create a finite rollback point in this one-sample gradient form; for negative-advantage tokens, the rollback point scales with µt . Consequently, a rare token and a frequent token can receive very different absolute probability tolerances even when their semantic effect on the next-token distribution should be judged by probability mass rather than by relative ratio. Advantage-weighted KL regularizer with the K3 estimator. The previous objective uses the K1 estimator − log rt for DKL (µ∥π), which can have high variance. A common lower-variance alternative is the K3 estimator k3 (rt ) = rt − 1 − log rt , which has the same expectation under yt ∼ µ(·|st ) because Eyt ∼µ [rt − 1] = 0. The corresponding per-token objective is |Ât | rt − 1 − log rt . (18) ℓKL3 (rt ) = rt Ât − 2δ Taking the gradient gives ! |Ât | (rt − 1) ∇ log πt ∇ℓKL3 (rt ) = rt Ât − 2δ ! sign(Ât )(rt − 1) = 1− rt Ât ∇ log πt . (19) 2δrt Thus the K3-induced gradient weight is wKL3 (rt ) = 1 −

sign(Ât )(rt − 1) , 2δrt

which also gives a ratio-based geometry because it only depends on rt . Setting wKL3 (rt ) = 0 yields  1  1−2δ , Ât > 0 and δ < 12 , ⋆ rt =  1 , Â < 0. t 1+2δ For Ât > 0 and δ ≥ 12 , the gradient weight remains positive for all feasible rt > 0, so no finite cancellation boundary exists. When a finite boundary does exist, it is again expressed as a fixed value of the importance ratio rt = πt /µt . The K3 estimator reduces the variance of the KL estimate, but it still induces a ratio-based trust region. In other words, K3 changes the estimator but not the relevant geometry. It can make the KL estimate numerically better behaved, yet the corrective force is still calibrated in ratio space. This is the key mismatch for long-tailed language-model distributions: a small absolute movement on a low-probability token can dominate the gradient through the ratio factor rt as rt grows large, while a much larger movement on a high-probability token may appear modest in ratio terms. Advantage-weighted TV regularizer. Now consider the per-token TV-regularized objective ℓTV (rt ) = rt Ât − 19

|Ât | |rt − 1|. 2δ

(20)

For rt ̸= 1, its gradient is ∇ℓTV (rt ) = =

! |Ât | rt Ât − rt sign(rt − 1) ∇ log πt 2δ ! sign(Ât ) sign(rt − 1) rt Ât ∇ log πt . 1− 2δ

(21)

Thus the TV-induced gradient weight is wTV (rt ) = 1 −

sign(Ât ) sign(rt − 1) . 2δ

(22)

This weight takes only two values: ( 1 , sign Ât (rt − 1) > 0, 1 − 2δ wTV (rt ) = 1 1 + 2δ , sign Ât (rt − 1) < 0. It depends only on whether the current ratio shift has the same sign as the advantage. It does not depend on the magnitude of |rt − 1|. The advantage-weighted TV penalty therefore induces a binary gradient weight, not a smooth trust-region boundary. This behavior is undesirable for a different reason from KL. The TV penalty removes the unbounded ratio magnitude, but the sampled absolute-value form has a nondifferentiable kink at rt = 1 and a piecewise-constant gradient weight away from that point. As a result, it distinguishes only whether the update is moving away from or toward the behavior policy, not how far the token has moved. It therefore cannot reproduce the gradual attenuation inside the trust region or the strength-calibrated correction outside the boundary that DRPO provides. Summary. The above derivations show that the nominal divergence in the objective is not sufficient to determine whether a method has the desired trust-region behavior. KL and K3 penalties induce ratio-based boundaries; the sampled TV penalty induces a two-level gradient weight; and none of them yields a smooth Binary-TV boundary. By contrast, the DRPO regularizer in Equation (8) produces the weight |πt − µt | 1 − sign(Ât (rt − 1)) , δ which depends continuously on the absolute probability shift. This is the property that lets DRPO preserve DPPO’s divergence-based trust-region geometry while replacing the hard mask with a corrective smooth update. The empirical comparisons in Appendix D and Appendix D.6 are consistent with this analysis: penalties whose gradients remain ratio-based or binary are less stable than the Binary-TV quadratic penalty.

Appendix D. More Experimental Details We provide the detailed experiment configurations, more ablation studies, and results in this section as a complementary part of Section 4. For the computation resources, we use 4 × 8 NVIDIA H20 to conduct most of the experiments. We build our codebase on VeRL (Sheng et al., 2024) and use Megatron (Shoeybi et al., 2019) as the 20

Table 2: Hyperparameters. Hyperparameters

Qwen3-4B-Base

Qwen3-30B-A3B-Base

Qwen3.5-35B-A3B-Base

R1D

Learning Rate PPO Epochs Max Prompt Length Max Response Length Train Batch Size PPO Mini Batch Size Rollout Temperature Group Size

1e-6 1 2048 8192 64 32 1.0 8

1e-6 1 2048 8192 256 32 1.0 16

1e-6 1 2048 8192 64 16 1.0 8

training backend and vLLM (Kwon et al., 2023) as the inference backend to speed up rollout. To verify the correctness of the solutions in math reasoning tasks, we utilize the third-party library math-verify2 . Besides, we have tried various kinds of objective functions, revealing the effects of the advantage scaling |Ât |, different divergences, binary approximation, etc. Typically, we train Qwen3-4B-Base with 800 steps, Qwen3-30B-A3B-Base with 300 steps, Qwen3.5-35B-A3B-Base with 110 steps, and R1D with 3000 steps. Since we used Megatron (Shoeybi et al., 2019) as the training backend, and at the time we conducted experiments, it did not have sufficient support for efficiently training Qwen3.5, we chose to train fewer steps compared to Qwen3-30B-A3B-Base. D.1 Comparing with KL Regularization

Qwen3-30B-A3B-Base 0.45 0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05

0.70

0.60 0.55 0.50 0

Training Step

100

Qwen3-30B-A3B-Base (FP8 E2E)

0.40 0.35

100

150

200

Training Step

250

300

0.15

0.20

0.10

0.15 0.10 50

100

150

200

Training Step

250

100

150

200

Training Step

250

300

R1D

0.05 0

0.325 0.300 0.275 0.250 0.225 0.200 0.175 0.150 0

Accuracy

0.25

Qwen3-4B-Base

0.25 0.20

0.30

0.05

0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05

Accuracy

0.65

Qwen3-30B-A3B-Base (FP8 Rollout) Accuracy

Qwen3.5-35B-A3B-Base

200

400

Training Step

600

800

500 1000 1500 2000 2500 3000

Training Step

KL wo. |At|

DRPO

Figure 6: Training dynamics for DRPO and directly applying a KL penalty term without introducing the advantage weight |Ât |. 2. https://github.com/huggingface/Math-Verify

In addition to the baselines in Section 4.1, another common method is to use a pure KL regularizer (without the advantage weight) as in the Algorithm 1 of Schulman et al. (2015) and the Equation (8) of Schulman et al. (2017). We conduct an experiment to compare with this method. Specifically, we instantiate the below objective   |y| X 1 LKL_wo_A (x, π) = Ey∼µ(·|x)  rt · Ât + · log rt  2δ t=1

with the same hyperparameter δ = 12.5 (see the hyperparameter tuning results in Figure 8). As shown in Figure 6, DRPO consistently outperforms this KL regularizer across all six experiments. This gap can be explained from two complementary perspectives. First, the regularizer should adapt to the per-token advantage scale: as shown in Appendix D.2, removing the factor |Ât | degrades performance because the token-wise optimum depends on the current advantage magnitude. Second, even after setting the advantage-weight issue aside, the KL penalty still induces a ratio-based trustregion geometry as analyzed in Appendix C, which is less aligned with the desired constraint than DRPO. D.2 Extended Ablations on Advantage Weighting

Qwen3-30B-A3B-Base (FP8 E2E)

Qwen3-4B-Base

0.25

R1D

0.15 0.10 0.05

100

150

Training Step

200

250

KL w. |At|

0.30 0.25 0.20 0.15 0.10 0.05 0.00

Accuracy

0.20

Accuracy

0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00

200

400

Training Step

KL wo. |At|

600

800

TV w. |At|

500 1000 1500 2000 2500 3000

Training Step

TV wo. |At|

Figure 7: Comparison among experiments applying a KL penalty term or a TV penalty term, with or without the advantage weight |Ât |. To further isolate the role of the advantage weight |Ât |, Figure 7 compares two penalty types, KL and TV, each evaluated both with and without this factor. The pattern is consistent across all settings: adding the advantage weight leads to clearly better performance, supporting the importance of weighting the regularizer by |Ât |. This behavior matches the analysis in Section 3.1 and the ablations in Section 4.2. Without |Ât |, the effective trust-region boundary becomes entangled with the advantage magnitude rather than remaining stable. As a result, the update is overly restrictive for small-advantage tokens and too loose for large-advantage ones. Since token-level advantage estimates are also noisy in practice, this mismatch further hurts both training stability and final accuracy. D.3 Hyperparameter Tuning of Advantage-Weighted KL Regularizer To rule out the concern that the KL baseline may simply be under-tuned, we sweep a range of hyperparameters for the advantage-weighted KL regularizer (Equation (13)) under the R1D setting. 22

AIME 2024

0.40

AIME 2025 0.30

0.35 0.30

0.25

0.20

0.20 0.15 0

500 1000 1500 2000 2500 3000

Training Step

0.15 0

Response Length 6000 5000 4000 500 1000 1500 2000 2500 3000

Training Step

DRPO

Training Step

Training Reward

7000

500 1000 1500 2000 2500 3000

Entropy 1.25 1.00 0.75 0.50 0.25 0.00

1.0 0.9 0.8 0.7 0.6 0.5 0

500 1000 1500 2000 2500 3000

Training Step

PPO-KL

0.0015 0.0010 0.0005 500 1000 1500 2000 2500 3000

Training Step

KL w. |At| (12.5)

KL w. |At| (5.0)

500 1000 1500 2000 2500 3000

Training Step

KL w. |At| (2.5)

Figure 8: The hyperparameter tuning for KL with advantage weight |Ât | under the R1D setting. Figure 8 shows that DRPO remains stronger across the full sweep, even when the KL baseline is equipped with the same advantage weight |Ât |. This robustness gap is consistent with our theoretical analysis. As discussed in Appendix C, the KL penalty fundamentally imposes a ratio-based trust-region geometry. By contrast, DRPO yields a Binary-TV geometry, which more faithfully reflects the intended divergence constraint and therefore produces more reliable optimization behavior (see Section 3.1). D.4 Hyperparameter Tuning of DPPO Baseline To compare DRPO against a carefully tuned DPPO baseline, we sweep several DPPO thresholds. Unlike DPPO, DRPO provides corrective gradients for tokens outside the trust region. Figure 9 shows that DPPO needs a more fine-grained parameter tuning, and ε = 0.15 works best on the Qwen3-30B-A3B-Base setting, which still performs worse than DRPO, while ε = 0.6 works best on the FP8-E2E setting, which achieves similar performance with DRPO. So DRPO has a relatively universal hyperparameter δ = 12.5 compared to DPPO. D.5 Hyperparameter Tuning of DRPO To assess the hyperparameter sensitivity of DRPO, we evaluate two choices of the threshold parameter δ. As shown in Figure 10, reducing δ substantially from 12.5 to 2.5 leads to only a minor drop in performance. This result suggests that DRPO is relatively robust to the choice of threshold and performs well across a broad hyperparameter range. D.6 Mask Ablation with Alternative Divergence Penalties We further repeat the mask ablation with several choices of divergence penalty. Let Mtout = Id[DtBin-TV > δ] denote the indicator that the sampled token is outside the DPPO trust region. For 23

AIME 2024

0.4 0.3 0.2 0.1 0

100

200

Training Step

0.3

0.08

4000

0.2

0.06

3000

0.04

0.1

0.3 0.2 0.1 0.0

100

200

300

Training Step

100

200

Training Step

300

100

200

300

Training Step

0.08

100

200

300

DPPO-TV (0.6)

100

200

300

Training Step

100

200

300

2000

0.02

Training Step

4000

0.04 0

6000

0.06

0.1

1000

8000

0.10

0.2

0.0

2000

0.02 0

Accuracy

FP8-E2E Accuracy

5000

0.3

0.4

Response Length

0.10

300

0.5

Entropy

AIME 2025

0.4

Accuracy

Base Accuracy

0.5

100

200

300

Training Step

DPPO-TV (0.2)

DPPO-TV (0.15)

DRPO

Figure 9: Training dynamics for different parameters under DPPO, compared to DRPO. Top: Qwen3-30B-A3B-Base; Bottom: Qwen3-30B-A3B-Base using FP8 precision for end-to-end training.

AIME 2024

0.5

AIME 2025

0.4

0.06 0.05

Entropy

Accuracy

0.3

0.04

0.2

0.03

0.1

0.1 0

100

200

Training Step

300

0.02 0

Response Length

100

200

Training Step

0.8 0.7 0.6 0.5 0.4 0.3 0.2

100

200

300

100

200

Training Step

300

PPO-KL

2000

Training Step

0.0025

Reward

Tokens

3000

300

Training Reward

4000

1000

Entropy

0.0020 0.0015 0.0010

100

200

Training Step

DRPO (2.5)

300

100

200

Training Step

300

DRPO (12.5)

Figure 10: Hyperparameter tuning of the coefficient on DRPO. a generic penalty Ωt , the masked variant uses   |y| X |Ât |  LMask-Ω (x, π) = Ey∼µ(·|x)  rt Ât − Mtout Ωt . 2δ t=1

Inside the trust region, this objective reduces to the unregularized token surrogate, matching DPPO’s active gradient. Outside the trust region, it restores the corrective penalty gradient that DPPO’s 24

hard mask discards. Under this framework, we instantiate three objectives as follows   |y| X |Ât | rt Ât − Mtout LMask-DRPO (x, π) = Ey∼µ(·|x)  µ(yt |st )(rt − 1)2  , 2δ t=1   |y| X |Ât | rt Ât − Mtout LMask-SPO (x, π) = Ey∼µ(·|x)  (rt − 1)2  , 2δ t=1   |y| 2 X | Â | (log r ) t  t LMask-KL (x, π) = Ey∼µ(·|x)  . rt Ât − Mtout 2δ 2

(23)

(24)

(25)

t=1

Notably, (log rt )2 in Equation (25) is exactly the penalty term in online policy mirror descent that Kimi series (Team et al., 2025a, 2026) utilized.

FP8 E2E

0.40 0.35

Mask-DRPO DRPO Mask-KL DPPO Mask-SPO SPO

Accuracy

0.30 0.25 0.20 0.15 0.10 0.05 0.00

100

150

Training Step

200

250

Figure 11: Ablation on applying the DRPO regularizer only outside DPPO’s trust region. As shown in Figure 11, applying the penalty only outside the trust region achieves performance close to applying it everywhere for the DRPO regularizer, confirming that the main gain comes from correcting tokens that have crossed the boundary. At the same time, the penalty choice still matters: ratio-space and KL-type penalties are harder to calibrate, whereas the Binary-TV quadratic penalty used by DRPO gives the best and most stable behavior because its gradient directly follows absolute probability displacement.

Record · ID 267634 · SHA-256 6129f463e70dc52c
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.