B EYOND I MPORTANCE S AMPLING : R EJECTION -G ATED P OLICY O PTIMIZATION Ziwu Sun∗ [email protected]
Zhen Gao∗ [email protected]
Jiyong Zhang∗ [email protected]
arXiv:2604.14895v1 [cs.LG] 16 Apr 2026
Jiaheng Li∗ [email protected]
A BSTRACT We propose a new perspective on policy optimization: rather than reweighting all samples by their importance ratios, an optimizer should select which samples are trustworthy enough to drive a policy update. Building on this view, we introduce Rejection-Gated Policy Optimization (RGPO), which replaces the importance-sampling ratio rθ = πθ /πold with a smooth, differentiable acceptance gate αθ (s, a) = g(rθ (s, a)) ∈ [0, 1]. Unlike prior work that applies rejection sampling as a data-level heuristic before training, RGPO elevates rejection to an optimization principle: the gate participates directly in gradient computation and is implicitly updated alongside the policy. RGPO provides a unified framework: the policy gradients of TRPO, PPO, and REINFORCE all correspond to specific choices of the effective gradient weight w(r) = g ′ (r) · r. We prove that RGPO guarantees finite, bounded gradient variance even when importancesampling ratios are heavy-tailed (where IS variance diverges). We further show that RGPO incurs only a bounded, controllable bias and provides an approximate monotonic policy improvement guarantee analogous to TRPO. RGPO matches PPO in computational cost, requires no second-order optimization, and extends naturally to RLHF-style preference alignment. In online preference fine-tuning of Qwen2.5-1.5B-Instruct on Anthropic HH-RLHF (n = 3 seeds), RGPO uses a dual-ratio gate that anchors learning to both the previous policy and the reference model, achieving a Pareto-dominant outcome: the highest reward among online RL methods (+14.8% vs. PPO-RLHF) and the lowest KL divergence to the reference model (−16.0% vs. PPO-RLHF, −53.1% vs. GRPO).
1
I NTRODUCTION
How should a policy optimizer treat samples that were collected under a different, older policy? This question sits at the heart of modern deep reinforcement learning, and the answer almost universally given is: reweight them via importance sampling (IS). Every sample is kept; off-distribution samples are compensated for by multiplying their gradient contribution by the density ratio rθ = πθ /πold . The problem with reweighting. Reweighting is mathematically sound but practically fragile. When the current policy πθ drifts away from the behavior policy πold , importance ratios can become very large, causing gradient variance to explode and training to destabilize. PPO (Schulman et al., 2017) addresses this by replacing the raw ratio with a clipped surrogate: LPPO (θ) = E[min(rθ A, clip(rθ , 1−ϵ, 1+ϵ) · A)] . The min creates an asymmetric mechanism: when rθ falls outside [1 − ϵ, 1 + ϵ] and A > 0, the clipped term is smaller, so the min selects the clipped value and the gradient with respect to θ from the rθ A term is blocked (the clipped constant carries no gradient); when A < 0 and rθ is out of range, the rθ A term is already the min and gradients flow normally, allowing the policy to ∗
†
Equal contribution, listed in no particular order.
1
Corresponding author.
correct itself. In short, PPO prevents over-optimistic updates but does not zero out all gradients for extreme samples—it is a one-sided guard rather than a hard rejection. Nevertheless, clipping is a discontinuous, hand-designed heuristic that offers limited design flexibility and no principled generalization to other forms of selection. More fundamentally, clipping reveals an implicit assumption: some samples should not contribute to the update at all. But PPO enforces this through an ad hoc threshold, not through a principled selection mechanism. A missing abstraction: sample selection. We argue that the policy optimization literature has long lacked a principled treatment of sample selection—the question of whether a sample should influence an update, as opposed to how much weight it should receive. Existing methods conflate the two: • TRPO/PPO (and IS in general): every sample is used; weight determines influence. • RAFT, RLHF rejection sampling (Gulcehre et al., 2023): some samples are discarded before training as a preprocessing step, outside the optimization loop. Neither approach makes selection a differentiable, first-class component of the optimization objective. Our proposal: differentiable selection. We propose Rejection-Gated Policy Optimization (RGPO), which introduces a smooth acceptance gate αθ (s, a) = g(rθ (s, a)) ∈ [0, 1] directly into the surrogate objective: LRGPO (θ) = Eπold [αθ (s, a) · Aold (s, a)] . The key properties of this design are: • Differentiable. Gradients flow through αθ back into θ via the chain rule. The gate is not a preprocessing step; it is part of the loss. • Implicit. No separate gate network or extra parameters are needed. The gate is fully determined by the policy ratio rθ , and it updates automatically as θ evolves. • Unifying. The policy gradients of TRPO, PPO, and REINFORCE all fit the RGPO template via specific choices of the effective gradient weight w(r) = g ′ (r) · r (Table 1): REINFORCE is an exact special case in the on-policy limit (r ≡ 1); TRPO and PPO are gradient-level correspondences (g ∈ / [0, 1]); AWR is a closely related method. This constitutes what we call elevating rejection sampling from a data-level heuristic to a differentiable optimization principle. Contributions. 1. Framework. We introduce RGPO, replacing the IS ratio with a differentiable acceptance gate αθ = g(rθ ) ∈ [0, 1], and unify existing methods through the effective gradient weight w(r) = g ′ (r) · r: REINFORCE is an exact special case in the on-policy limit (r ≡ 1); TRPO and PPO are gradient-level correspondences; AWR is a closely related method. 2. Theory. We prove a gradient bias bound (Theorem 1), a variance reduction guarantee (Theorem 2), and an approximate policy improvement bound analogous to TRPO (Theorem 3). 3. Design. We provide three principled gating functions and connect the optimal gate to the dual solution of constrained policy optimization, giving RGPO a clean theoretical interpretation. 4. RLHF extension. We show RGPO naturally generalizes RLHF-style rejection fine-tuning by making the accept/reject decision differentiable and jointly trained with the policy.
2
R ELATED W ORK
Policy gradient and trust region methods. REINFORCE (Williams, 1992) and its variants (Sutton et al., 1999) form the foundation of policy gradient methods. TRPO (Schulman et al., 2015a) 2
enforces a KL trust region via a constrained optimization, providing a monotonic improvement guarantee. PPO (Schulman et al., 2017) approximates this constraint with a clipping heuristic, achieving strong empirical performance at lower cost. GRPO (Shao et al., 2024) adapts PPO for LLM finetuning using group-relative advantage normalization. All of these methods treat sample selection implicitly and discontinuously (clipping, hard KL cutoff); RGPO makes selection a smooth, explicit component of the objective. Variance reduction via truncated importance sampling. V-trace (Espeholt et al., 2018) and ACER (Wang et al., 2016) truncate IS ratios to reduce variance in distributed settings. RIS (Precup et al., 2000) provides variance-reduction guarantees for off-policy evaluation. These methods cap the IS weight but do not reframe it as a selection mechanism; the conceptual shift to “selection vs. weighting” and the unified gating framework are unique to RGPO. Advantage-weighted regression. AWR (Peng et al., 2019) and AWAC (Nair et al., 2020) optimize a behavior-cloning loss E[w log πθ ] with w = exp(A/β), sidestepping IS entirely. These are related to RGPO: AWR’s weighting w = exp(A/β) resembles a gate, but it depends on the advantage A rather than the IS ratio rθ , so it is not an exact special case of RGPO’s gating formalism (Table 1). AWR also lacks the selection interpretation, the unified theoretical framework, and the connection to rejection sampling. Hard rejection sampling in RL and RLHF. as a data-preprocessing step:
Several recent works apply hard rejection sampling
• RAFT / ReST (Gulcehre et al., 2023): filter high-reward samples before fine-tuning; the rejection decision is outside the optimization loop and not differentiable. • RSO (Liu et al., 2023): uses statistical rejection sampling to construct preference data for DPO; again, a sampling-level operation, not an optimization-level one. • Jackpot (Zhao et al., 2024): introduces budgeted rejection sampling to correct policymismatch in LLM rollouts, reducing distribution shift at the data collection stage. The key distinction: all of the above apply rejection at the sampling level—they discard data before the optimizer sees it. RGPO instead integrates the accept/reject decision into the objective function itself, making it differentiable and jointly optimized with the policy. This is an orthogonal and more general perspective: Sampling-level rejection ⊂ Optimization-level rejection (RGPO). Preference alignment and DPO. DPO (Rafailov et al., 2023) bypasses explicit reward modeling by reparameterising the RLHF objective as a supervised classification over preference pairs, eliminating the online RL loop entirely. While DPO achieves low KL drift (it is purely offline), it cannot exploit online reward signals, limiting adaptivity when only a reward model (rather than labelled preference pairs) is available. RGPO is complementary: it is a fully online algorithm that uses an RM reward signal and achieves both higher reward and lower KL than PPO and GRPO (Section 7.4). Reward-model overoptimisation (Gao et al., 2023) is a key failure mode of online RLHF; RGPO’s dual-ratio gate provides an explicit, principled mechanism against it by penalising any update that would move the policy far from πref . Unified frameworks for policy optimization. Several papers have proposed unifying views of policy optimization: MPO (Abdolmaleki et al., 2018) derives updates via EM on a KL-constrained objective, recovering actor-critic methods as special cases. RGPO provides a complementary unification through the lens of the effective gradient weight w(r) = g ′ (r) · r: REINFORCE is an exact special case in the on-policy limit (r ≡ 1); TRPO and PPO are gradient-level correspondences whose gate functions fall outside [0, 1] (IS reweighting regime); AWR is a closely related method. 3
3
BACKGROUND
3.1
P OLICY O PTIMIZATION O BJECTIVE
We consider a Markov Decision Process (MDP) (S, A, P, r, γ). The policy optimization objective is: " # X t J(θ) = Eτ ∼πθ γ r(st , at ) . (1) t
The policy gradient theorem gives: ∇θ J(θ) = Es,a∼πθ [∇θ log πθ (a|s) · Aπθ (s, a)] ,
(2)
where Aπθ (s, a) = Qπθ (s, a) − V πθ (s) is the advantage function. 3.2
TRPO AND I MPORTANCE S AMPLING
To reuse data from πold , TRPO defines the surrogate objective: LTRPO (θ) = Es,a∼πold [rθ (s, a) · Aold (s, a)] ,
(3)
where the importance ratio is rθ (s, a) =
πθ (a|s) , πold (a|s)
(4)
subject to the constraint DKL (πθ ∥πold ) ≤ δ. 3.3
PPO
PPO (Schulman et al., 2017) replaces the trust region constraint with a clipping mechanism: LPPO (θ) = E[min(rθ A, clip(rθ , 1 − ϵ, 1 + ϵ)A)] .
(5)
This implements an asymmetric gradient mechanism: when rθ > 1+ϵ and A > 0, further increasing the ratio yields no additional gradient signal (the clipped value is selected by min); when rθ < 1 − ϵ and A < 0, the unclipped rθ A term is already the min, so gradient flow is preserved to correct overshrinkage. This one-sided guard prevents over-optimistic updates without eliminating corrective gradients entirely.
4
M ETHOD : R EJECTION -G ATED P OLICY O PTIMIZATION (RGPO)
4.1
F ROM R EWEIGHTING TO S ELECTION
To build intuition, consider why importance sampling causes problems. Given a sample (s, a) ∼ πold , the IS-corrected gradient contribution is: rθ (s, a) · ∇θ log πθ (a|s) · Aold (s, a). When rθ ≫ 1, the sample is “unlikely under the old policy but likely under the new one.” IS amplifies such samples, potentially causing a large, unreliable gradient step. PPO clips rθ to suppress this, but clipping is discontinuous and offers no smooth control. The deeper issue is conceptual: IS asks “how much should this sample contribute?” and answers with the ratio rθ . We instead ask “should this sample contribute at all?”—and answer with a smooth acceptance probability. Paradigm
Question
Mechanism
Importance Sampling PPO clipping RGPO (ours)
How much? How much? (bounded) Whether?
rθ ∈ [0, ∞) clip(rθ ) ∈ [1 − ϵ, 1 + ϵ] αθ = g(rθ ) ∈ [0, 1]
4
4.2
T HE RGPO O BJECTIVE
Definition 1 (Acceptance function). Let g : R+ → [0, 1] be a smooth, monotonically nondecreasing function. We define the acceptance weight as: αθ (s, a) = g(rθ (s, a)) ∈ [0, 1].
(6)
LRGPO (θ) = Es,a∼πold [αθ (s, a) · Aold (s, a)] .
(7)
The RGPO objective is:
Compared to Eq. (3), RGPO replaces the unbounded ratio rθ with the bounded acceptance weight αθ ∈ [0, 1]. 4.3
G RADIENT D ERIVATION
Since πold does not depend on θ, the gradient commutes with the expectation: ∇θ LRGPO = Eπold [∇θ αθ (s, a) · Aold (s, a)] .
(8)
Applying the chain rule to αθ = g(rθ ): ∇θ αθ = g ′ (rθ ) · ∇θ rθ .
(9)
We compute ∇θ rθ using the log-derivative identity ∇θ πθ = πθ ∇θ log πθ : ∇θ rθ =
∇θ πθ (a|s) πθ (a|s) = ∇θ log πθ (a|s) = rθ ∇θ log πθ (a|s). πold (a|s) πold (a|s)
(10)
Substituting Eq. (10) back: ∇θ LRGPO = Eπold [g ′ (rθ ) rθ ∇θ log πθ (a|s) Aold ] .
(11)
Defining the effective weight
wθ (s, a) = g ′ (rθ ) · rθ , Eq. (11) takes the familiar form of a reweighted policy gradient:
(12)
∇θ LRGPO = Eπold [wθ (s, a) ∇θ log πθ (a|s) Aold ] .
(13)
′
Note that wθ replaces the plain ratio rθ used in TRPO/IS: the factor g (rθ ) modulates the contribution of each sample, bounded wherever g is bounded, while the factor rθ preserves the original policygradient direction. 4.4
U NIFIED V IEW: E XISTING A LGORITHMS T HROUGH THE E FFECTIVE G RADIENT W EIGHT
Table 1 shows that the policy gradients of TRPO, PPO, and REINFORCE can all be written in the RGPO gradient form Eπold [w(r) ∇θ log πθ A] (Eq. (11)) under specific choices of w(r) = g ′ (r) · r. Two distinct levels of equivalence must be carefully distinguished: 1. Exact formula-level special case (on-policy limit, r ≡ 1): REINFORCE operates with πold = πθ , so rθ ≡ 1 everywhere. Substituting into the RGPO gradient formula yields ∇LRGPO r≡1 = g ′ (1) · Eπθ [∇ log πθ · A] = g ′ (1) · ∇JREINFORCE , which recovers the REINFORCE gradient exactly up to the positive constant g ′ (1) > 0 (absorbed into the learning rate). Note that it is the on-policy limit r ≡ 1, not a constant gate g ≡ 1, that produces this correspondence: a constant gate would give g ′ = 0 and hence w = 0. 2. Gradient-level correspondence (g ∈ / [0, 1], outside the acceptance-gate definition): TRPO (g(r) = r ∈ [0, ∞)) and PPO (g(r) = clip(r, 1−ϵ, 1+ϵ) ∈ [1−ϵ, 1+ϵ]) produce effective weights that fit the RGPO gradient template, but their gate functions exceed 1 and thus function as IS reweighting rather than acceptance gating. They represent the pure-IS and truncated-IS regimes that RGPO is designed to move beyond. 5
AWR is a closely related method whose weighting exp(A/β) depends on the advantage rather than r, so it is neither an exact special case nor a gradient-level correspondence in the strict sense. This distinction is not merely taxonomic. It reveals that the policy optimization literature has been performing weighted policy gradient updates all along, but without a framework that separates selection (g ∈ [0, 1], RGPO’s domain) from reweighting (g ∈ / [0, 1], IS/PPO/TRPO’s domain). Table 1: RGPO as a unified framework, showing two levels of equivalence. ✓ = exact special case in the on-policy limit: when πold = πθ (r ≡ 1), the RGPO gradient formula yields w(1) = g ′ (1) > 0, recovering the REINFORCE gradient up to a positive constant (absorbed into the learning rate). The correspondence is via r ≡ 1, not via a constant gate g ≡ 1 (which gives g ′ = 0, w = 0). ◦ = gradient-level correspondence: effective weight w(r) matches the RGPO gradient template, but g∈ / [0, 1], so the gate lies outside the acceptance-gate definition (IS reweighting regime). AWR is a related method whose weighting exp(A/β) depends on advantage rather than r. The effective gradient weight is w(r) = g ′ (r) · r. Method
g(·)
w
Selection type
Bias
g ∈ [0, 1]?
TRPO PPO REINFORCE
g(r) = r g(r) = clip(r, 1−ϵ, 1+ϵ) g(r) = 1
r r · 1[1±ϵ] 1
None (pure IS) Hard, discontinuous None (on-policy)
None Low None
No No Yes
◦ ◦ ✓
AWR
w = exp(A/β) (on A, not r)
exp(A/β)
Advantage-weighted
Controlled
—
—
Soft, differentiable
Tunable
Yes
—
RGPO
′
flexible g(r) ∈ [0, 1]
g (r)·r
A key observation on PPO: inside [1−ϵ, 1+ϵ], g(r) = r equals the IS ratio and can exceed 1—the gate reweights rather than selects. Outside the interval, g ′ (r) = 0: the gradient is blocked nondifferentiably, a binary on/off gate. This is precisely why PPO is a gradient-level correspondence but not a true acceptance gate (g ∈ / [0, 1]): it mixes IS reweighting (inside the clip region) with hard gradient zeroing (outside), without a principled acceptance-probability interpretation. RGPO replaces both with a smooth gate g ∈ [0, 1], separating the selection question (“whether to use this sample”, answered by αθ ∈ [0, 1]) from the reweighting question (“how much to amplify it”, which RGPO deliberately forgoes). 4.5
D ESIGN C HOICES FOR THE G ATING F UNCTION
We consider three principled designs: (1) Sigmoid gating (recommended). g(r) = σ(k(r − 1)) =
1 , 1 + e−k(r−1)
(14)
where k > 0 controls the sharpness of the gate. When r = 1 (old and new policies agree), α = 0.5; when r ≫ 1 (new policy much more likely), α → 1; when r ≪ 1, α → 0. The effective weight is: w(r) = k σ(k(r − 1)) (1 − σ(k(r − 1))) r,
(15)
which is bounded and suppresses extreme ratios automatically. (2) Clipped linear. g(r) = clip(r, 0, c),
c > 0.
(16)
This caps the acceptance weight at c, directly controlling the maximum contribution of any single sample. (3) Temperature-controlled. rβ , (17) 1 + rβ where β > 0 is a temperature parameter. As β → 1 this approaches the standard ratio; as β → 0 it approaches uniform acceptance. g(r) =
6
Equiv.?
4.6
I NTERPRETATION OF THE G ATE
No extra parameters. In the canonical sigmoid design (Eq. (14)), αθ has no parameters beyond those of πθ . The gate is implicitly induced by the policy: as θ evolves, the acceptance weights change automatically via rθ . This contrasts with learned gate networks that would require a separate training loop. Intuitive meaning: “Is this sample still valid?” with the policy we are currently optimizing?”
The gate answers: “Is this transition compatible
• rθ ≈ 1: old and new policies agree ⇒ high acceptance, sample is used normally. • rθ ≫ 1: new policy assigns much higher probability ⇒ the sample may be an outlier; gate limits its influence. • rθ ≪ 1: new policy has moved far away ⇒ sample is stale and untrustworthy; gate suppresses it. Critically, this is selection, not correction: importance sampling tries to fix the bias from offdistribution samples by amplifying them; RGPO instead discards or down-weights them. The former risks catastrophic variance; the latter introduces controlled bias with bounded variance (Theorem 2). RGPO vs. prior rejection-sampling work. A critical distinction from RAFT, RSO, and Jackpot (Zhao et al., 2024): those methods apply rejection before the optimizer, as a filtering step on the data buffer. RGPO integrates rejection into the optimizer as a differentiable gate. The differences are summarized below: Method
When
Differentiable?
Jointly optimized?
RAFT / ReST RSO Jackpot RGPO
Before training Data construction Data collection Inside loss
No No No Yes
No No No Yes
Connection to constrained optimization. The RGPO objective also arises as the dual of a constrained policy problem. Starting from maxθ E[rθ A] s.t. DKL (πθ ∥πold ) ≤ δ, the Lagrangian L = E[rθ A] − λDKL has optimal solution πθ ∝ πold exp(A/λ). The resulting weighting w = exp(A/β) is the same form as AWR, and can be viewed as a limiting RGPO gate where the acceptance signal is derived from the advantage rather than the IS ratio. Thus the gating function can be viewed as a soft constraint satisfaction mechanism: a sample is accepted to the degree that acting on it is consistent with the trust region. RLHF extension and dual gate. In RLHF, the policy must simultaneously stay close to the rollout reference πold (trust-region anchor) and to the base model πref (alignment anchor). We achieve this with a dual-gate objective: h i b KL (πθ ∥πold ) − βref DKL (πθ ∥πref ), LRGPO-RLHF (θ) = Eπ g(rold (s, a)) · Â(s, a) − β D (18) old
where rold = πθ /πold , β is the adaptive trust-region coefficient (Eq. 26), and βref is a fixed alignmentanchor coefficient. The first term (sigmoid gate on rold ) limits per-update policy drift; the third term penalises long-range drift from the base model. Together, the two anchors constitute the “dual gate”: the sigmoid suppresses off-policy gradient contributions, while the reference KL prevents reward hacking. Optionally, one can train a separate acceptance classifier: αϕ (s, a) = σ(fϕ (s, a)), fϕ ≈ A(s, a), optimized with a binary cross-entropy objective: max E[log αϕ · 1(A > 0) + log(1 − αϕ ) · 1(A < 0)] . ϕ
(19) (20)
This yields a two-player system (policy + selector) that makes the accept/reject decision fully differentiable—unlike RAFT or RSO where rejection is fixed and offline. The experiments in Section 7.4 use the dual-gate objective (Eq. 18) rather than the learned classifier. 7
5
T HEORETICAL A NALYSIS
We analyze the bias, variance, and policy improvement properties of RGPO. Throughout, we use the shorthand r = rθ (s, a) and assume A(s, a) is bounded: |A(s, a)| ≤ Amax . 5.1
B IAS –VARIANCE T RADEOFF
Theorem 1 (Bias Bound). Let g be differentiable and let |A(s, a)| ≤ Amax . Then the RGPO gradient satisfies: ∥∇θ LRGPO − ∇θ J(θ)∥ ≤ C · Eπold [|g ′ (rθ ) rθ − rθ |] ,
(21)
where C = Amax · ∥∇θ log πθ ∥. Proof Sketch. The true policy gradient under πold via IS is ∇J = Eπold [rθ ∇ log πθ A]. The RGPO gradient is ∇L = Eπold [g ′ (r) r ∇ log πθ A]. Their difference is ∆ = E[(g ′ (r) − 1) r ∇ log πθ A]. Taking the norm and applying the triangle inequality and boundedness of A and ∇ log πθ yields the result. Remark 1. When g ′ (r) ≈ 1 (e.g., small policy updates), the bias is near zero and RGPO approximates TRPO. As g deviates from the identity, bias grows but variance is controlled (Theorem 2). Theorem 2 (Controlled Variance). Let X = rθ ∇θ log πθ · Aold be the IS gradient estimator and Y = wθ ∇θ log πθ · Aold be the RGPO gradient estimator, where wθ = g ′ (rθ ) rθ . Assume: (i) 0 ≤ w(r) ≤ c for all r ≥ 0; (ii) Eπold (∇θ log πθ · Aold )2 ≤ σ 2 < ∞. Then: Var(Y ) ≤ c2 σ 2 . (22) 2 2 2 2 Furthermore, if E rθ (∇θ log πθ · Aold ) ≥ Kσ for some K > c , then Var(Y ) < Var(X). Proof Sketch. Since |Y | = |w(rθ )| · |∇θ log πθ · Aold | ≤ c · |∇θ log πθ · Aold |, we have Var(Y ) ≤ E[Y 2 ] ≤ c2 σ 2 , establishing (22). For the IS estimator, Var(X) = E[rθ2 (∇θ log πθ · Aold )2 ] − (E[X])2 ≥ Kσ 2 − (E[X])2 . When Kσ 2 > c2 σ 2 + (E[X])2 , one has Var(X) > Var(Y ). Proposition 1 (Variance Divergence Under Heavy-Tailed Importance Weights). Suppose the importance weights satisfy a power-law tail: P(rθ > t) ∼ C t−α as t → ∞, with α ≤ 2 and C > 0. If |∇θ log πθ · Aold | ≥ ϵ > 0 with positive probability, then Var(X) = +∞. In contrast, Var(Y ) ≤ c2 σ 2 < ∞ whenever σ 2 < ∞. R∞ Proof Sketch. When α ≤ 2, the power-law tail implies E[rθ2 ] = 0 P(rθ2 > s) ds ≥ R ∞ C ′ 1 s−α/2 ds = +∞ since α/2 ≤ 1. With the positive lower bound on |∇θ log πθ · A|, 2 E[X ] ≥ ϵ2 E[rθ2 ] = +∞, hence Var(X) = +∞. For RGPO, |w(r)| ≤ c uniformly, so E[Y 2 ] ≤ c2 σ 2 < ∞. Remark 2. Theorem 2 and Proposition 1 together reveal a fundamental advantage of RGPO: importance sampling gradient estimators may have infinite variance when policy ratios are heavy-tailed (a common occurrence when πθ drifts from πold ), while RGPO guarantees finite, bounded variance in the same regime. This is not merely a quantitative improvement—it is a qualitative stability guarantee that IS cannot provide. 5.2
P OLICY I MPROVEMENT G UARANTEE
Theorem 3 (Approximate Policy Improvement). Assume (i) g is monotonically non-decreasing with Lipschitz constant L√g , (ii) g(r) ≈ r in a neighborhood of r = 1, and (iii) DKL (πθ ∥πold ) ≤ δ. Then, with C = Lg Amax 2 > 0: √ J(θ) ≥ J(θold ) + LRGPO (θ) − C δ. (23) 8
Proof Sketch. By the performance difference lemma (Kakade & Langford, 2002), J(θ) − J(θold ) = Eπθ [Aold ] = Eπold [rθ Aold ]. We decompose: E[rA] = E[g(r)A] + E[(r − g(r))A]. The first term equals LRGPO (θ). The second term is bounded using the Lipschitz property of g and Pinsker’s √ √ inequality: |E[(r − g(r))A]| ≤ Lg Amax 2δ = C δ. Remark 3. Theorem 3 mirrors the TRPO monotonic improvement guarantee (Schulman et al., 2015a). When √ g(r) = r, the bound recovers the standard TRPO guarantee exactly. The correction term C δ captures the extra approximation cost of replacing rθ with g(rθ ); it vanishes as δ → 0 (i.e., as the policy converges). Remark 4. Theorems 1–3 and Proposition 1 together characterize RGPO’s tradeoff: the gating function g exchanges controllable bias for bounded, finite variance. When IS weights are heavytailed (power-law exponent α ≤ 2), the IS estimator has infinite variance (Proposition 1), while RGPO guarantees finite variance regardless—a qualitative stability advantage, not merely a quantitative one.
6
A LGORITHM
6.1
P RACTICAL O BJECTIVE
In practice, we optimize the following augmented surrogate, which adds a differentiable KL penalty to stabilize multi-epoch updates: N 1 X πθ (ai |si ) b KL (πθ ∥πold ) , L(θ) = g · Â(si , ai ) − β D (24) N i=1 πold (ai |si ) b KL is a sample where  is an estimated advantage (e.g., via GAE (Schulman et al., 2015b)) and D estimate of DKL (πθ ∥πold ) computed on the current mini-batch: X b KL = 1 D rθ (s, a) − 1 − log rθ (s, a) . (25) |B| (s,a)∈B
Adaptive β schedule. The penalty coefficient β is updated after every iteration using the same heuristic as PPO’s adaptive KL variant (Schulman et al., 2017): min(2β, βmax ) if DKL ≥ 1.5 ∆, β ← max(β/2, βmin ) if DKL ≤ ∆/1.5, (26) β otherwise, where DKL is the mean KL observed over all mini-batches in the current iteration and ∆ is a target KL threshold (default ∆ = 0.02, β ∈ [0.01, 5]). When the policy drifts excessively (DKL ≥ 1.5∆), β doubles to tighten the trust region; when updates are too conservative (DKL ≤ ∆/1.5), β halves to allow more aggressive improvement. This keeps the effective policy change near ∆ without the discontinuity of hard clipping. b KL peCrucially, Eq. (24) makes the trust-region mechanism fully differentiable: the KL term β D nalizes large policy changes through the same gradient pass that updates θ, in contrast to PPO’s clip, which zeroes gradients discontinuously, or hard KL early-stopping, which halts updates abruptly. This is consistent with RGPO’s broader design philosophy of replacing hard, non-differentiable control mechanisms with smooth analogues. 6.2
P SEUDOCODE
6.3
C OMPUTATIONAL C OMPLEXITY
RGPO has identical computational complexity to PPO: • No second-order optimization or conjugate gradient (unlike TRPO). 9
Algorithm 1: Rejection-Gated Policy Optimization (RGPO) Input: Initial policy πθ , value network Vψ , gating function g, learning rate η, target KL ∆, initial penalty coefficient β0 1: Set β ← β0 2: for k = 1, 2, . . . do 3: Set πold ← πθ 4: Collect trajectories D = {(st , at , rt )} by rolling out πold 5: Estimate advantages Â(s, a) via GAE 6: for each mini-batch B ⊆ D do πθ (a|s) 7: Compute ratios: rθ (s, a) ← πold (a|s) Compute acceptance weights: αθ (s, a) ← g(rθ (s, a)) X b KL ← 1 rθ − 1 − log rθ Estimate KL: D |B|
8: 9:
(s,a)∈B
10:
Compute RGPO loss (Eq. 24): L ← −
1 X b KL αθ (s, a) Â(s, a) + β D |B| (s,a)∈B
11: 12: 13: 14:
Compute value loss: LV ←
1 |B|
X
2 Vψ (s) − R̂(s)
(s,a)∈B
Update policy: θ ← θ − η ∇θ L Update value: ψ ← ψ − η ∇ψ LV end for
15: // Adaptive β update (Eq. 26): b KL over all mini-batches 16: Compute DKL ← mean of D 17: if DKL ≥ 1.5∆ then β ← min(2β, βmax ) 18: if DKL ≤ ∆/1.5 then β ← max(β/2, βmin ) 19: end for Figure 1: Pseudocode for RGPO with adaptive KL penalty. Line 8 is the key departure from PPO: the raw ratio rθ is passed through the smooth acceptance gate g, replacing the discontinuous clip b KL that enforces a soft trust region without operator. Lines 9–10 add a differentiable KL penalty β D zeroing any gradients. Lines 17–18 adapt β after each iteration using the same heuristic as PPO’s KL-penalty variant (Schulman et al., 2017): doubling β when the policy drifts too far and halving it when updates are too conservative. • The gating function g adds negligible overhead (a single element-wise operation per sample). • Memory requirements are identical to PPO.
7
E XPERIMENTS
7.1
E XPERIMENTAL S ETUP
We evaluate RGPO across three complementary domains: (1) continuous control to test core performance and stability, (2) ablation studies to validate the gating mechanism, and (3) RLHF-style preference alignment to demonstrate applicability to LLMs. Each domain uses a consistent set of metrics and controls. Environments. Continuous control (primary): MuJoCo locomotion benchmarks HalfCheetahv4, Walker2d-v4, Ant-v4, and Hopper-v4, covering tasks of increasing complexity and dimensional10
ity. RLHF (extension): Preference-based fine-tuning of Qwen2.5-1.5B-Instruct, discussed separately in Section 7.4. Baselines. We compare against four baselines chosen to cover the full design space of trust-region and importance-sampling methods: • PPO (Schulman et al., 2017): primary baseline; same on-policy trust-region family as RGPO, differing only in the surrogate loss (clip vs. acceptance gate). • TRPO (Schulman et al., 2015a): the canonical second-order trust-region method; enforces a hard KL constraint δ = 0.01 via conjugate gradient and backtracking line search, making exactly one policy update per rollout. Included to quantify the cost of strict KL enforcement and second-order computation. • AWR (Peng et al., 2019): advantage-reweighting family; replaces the IS ratio with an advantage-exponential weight, providing a natural ablation of the IS-anchoring in RGPO’s gate. • GRPO (Shao et al., 2024): evaluated only in the RLHF setting (Section 7.4), where it was designed. Evaluation metrics.
We report a unified set of metrics across all experiments:
1. Performance: average episodic return and final performance (mean over last 100 episodes per seed, averaged across seeds; 3 seeds for all environments). 2. Sample efficiency: learning curve (return vs. environment interaction steps). 3. Stability: standard deviation across seeds and worst-case seed performance. 4. Computational overhead: wall-clock training time and GPU hours, to confirm that the gate introduces negligible cost relative to PPO. 5. Policy behavior: KL divergence DKL (πθ ∥πold ), KL spike frequency and max KL per iteration, effective sample size (ESS), and gradient variance. Implementation and fairness controls. All methods share identical network architectures (2layer MLP, hidden size 256), Adam optimizer, learning rate 3 × 10−4 , batch size 64, discount γ = 0.99, and GAE λ = 0.95. PPO, RGPO, and AWR use nepochs = 10 policy gradient steps per rollout; only the surrogate loss differs. TRPO performs a single constrained policy update per rollout (as mandated by its trust-region constraint), with nepochs = 10 Adam steps for the value function only; conjugate gradient uses 10 iterations with damping 0.1. Three random seeds are used for all environments and all methods. No method-specific hyperparameter tuning is applied beyond the parameters explicitly reported. 7.2
M AIN R ESULTS (PART 1: S TANDARD RL)
Table 2 reports mean episodic return ± standard deviation at 1 × 106 environment steps for PPO, TRPO (δ = 0.01), AWR (β = 1.0, nepochs = 10), and RGPO (sigmoid gate, k = 5, β0 = 0.5, nepochs = 10) on four MuJoCo locomotion benchmarks. All environments and methods use 3 independent seeds. Performance and stability. RGPO outperforms PPO on Walker2d-v4 (+81%, statistically significant: t(2.9) = 2.46, p = 0.047, one-tailed Welch t-test, n = 3 seeds) and Ant-v4 (+47%; t(3.7) = 1.95, p = 0.064, one-tailed Welch t-test, n = 3 seeds; RGPO > PPO in all three paired comparisons), and achieves statistically comparable performance on HalfCheetah-v4 and Hopper-v4 (differences of ≈ 0%, well within one standard error given the large seed variance of PPO). The more striking pattern is the seed-to-seed stability: RGPO’s standard deviation is 0.53×, 0.48×, 0.36×, and 0.75× that of PPO on HC, Walker2d, Hopper, and Ant respectively. PPO’s coefficient of variation reaches 48–52% on HalfCheetah and Walker2d (CV = σ/µ), meaning its best and worst seeds can differ by nearly 3×; RGPO’s coefficient of variation stays below 27% across all four environments. This stability gap is a direct consequence of the bounded KL property (Ablation E): RGPO maintains a KL spike rate of 0% across all environments, whereas PPO exceeds the 2∆ threshold in 44–84% of iterations. 11
Figure 2: Final episodic return at 1 × 106 steps on four MuJoCo locomotion tasks (bars = mean; error bars = ±1 std across seeds). RGPO (orange) dominates PPO (blue) on Walker2d and Ant, and is statistically equivalent on HalfCheetah and Hopper. TRPO (purple, hatched) achieves strict KL control but underperforms both PPO and RGPO at 7× the computational cost—demonstrating that second-order trust-region constraints alone do not improve sample efficiency. AWR (green) performs worst on all tasks due to uncontrolled policy drift without IS correction (Ablation F). Table 2: Mean episodic return ± std at 1 × 106 steps. PPO and RGPO: nepochs = 10; TRPO: single policy update per rollout, nepochs = 10 value-only; 3 seeds. AWR: nepochs = 10; 3 seeds for every environment. All methods share identical architecture, optimizer, and advantage estimator. Bold marks the highest mean per row. †: TRPO uses 3 seeds on all environments. Env HalfCheetah-v4 Walker2d-v4 Hopper-v4 Ant-v4
PPO
TRPO†
AWR
RGPO
∆PPO→RGPO
3107 ± 1489 1653 ± 853 1732 ± 661 709 ± 237
1413 ± 233 1350 ± 131 1335 ± 173 332 ± 109
1509 ± 104 350 730 58
3006 ± 790 2998 ± 412 1727 ± 238 1043 ± 178
≈ 0% +81% ≈ 0% +47%
TRPO vs. RGPO: the cost of hard trust-region constraints. TRPO enforces the strictest KL budget (δ = 0.01, achieved exactly via conjugate gradient) yet consistently underperforms RGPO across all four environments: 1413 vs. 3006 on HalfCheetah-v4 (−53%), 1350 vs. 2998 on Walker2d-v4 (−55%), 332 vs. 1043 on Ant-v4 (−68%), and 1335 vs. 1727 on Hopper-v4 (−23%). TRPO also underperforms PPO on every environment despite tighter KL control. The root cause is structural: TRPO’s hard constraint mandates a single policy gradient step per rollout, while RGPO’s soft trust region allows nepochs = 10 steps (enabled by the importance-sampling anchor in the gate, as analysed in Ablation F). RGPO therefore extracts 10× more gradient signal per rollout from the same environment interactions, which—combined with its bounded-KL stability—explains its superior sample efficiency. TRPO also shows lower seed-to-seed variance (σTRPO < σPPO in all environments), confirming that KL control does improve stability; but strict KL control alone cannot compensate for the sample-efficiency deficit of single-step updates. AWR vs. RGPO. AWR performs substantially worse than both PPO and RGPO across all four environments (1509 on HC vs. PPO 3107 and RGPO 3006; near-zero on Ant). The reason is analysed in Ablation F: AWR with nepochs = 10 has no IS correction for the multi-epoch updates, causing uncontrolled policy drift (mean KL = 1.98 per iteration on HC, 99% spike rate) that degrades the policy. On Ant-v4, AWR’s lack of IS correction leads to catastrophic divergence (mean KL 18.85, max KL 1000), rendering the policy completely non-functional. This highlights that the importance ratio rθ in RGPO’s gate is not merely a design choice but a necessary stabiliser for multi-epoch on-policy training. Long-horizon stability. Table 3 shows performance as training budget is extended to 3M and 6M steps on HalfCheetah-v4 (3 seeds: seeds 0, 1, 2). PPO performance regresses consistently: 12
Figure 3: Learning curves (episodic return vs. environment interaction steps) on four MuJoCo locomotion environments. Solid lines = mean over seeds; shaded regions = ±1 std. TRPO shown with dashed line. RGPO (orange) shows markedly tighter confidence bands than PPO (blue), especially on Walker2d-v4, confirming the stability advantage reported in Table 2. TRPO (purple, dashed) learns slower than both PPO and RGPO throughout training—a consequence of its single-updateper-rollout constraint. AWR (green) plateaus or degrades early due to catastrophic KL drift. 2247 → 1896 → 1116 across all three budget levels. RGPO maintains substantially higher performance throughout and recovers at 6M: 2946 → 2696 → 3095. At 6M steps, RGPO significantly outperforms PPO (t(3.0) = 3.22, p = 0.028, one-tailed Welch t-test, n = 3 seeds). This provides empirical support for Theorem 3: the approximate monotonic improvement guarantee, which relies on the bounded KL property, is consistent with RGPO’s behaviour across all three horizons. Table 3: Mean episodic return ± std vs. training budget on HalfCheetah-v4 (nepochs = 10, 3 seeds). PPO regresses consistently; RGPO maintains high performance and significantly outperforms PPO at 6M steps (p = 0.028, one-tailed Welch t-test). Algorithm 1M steps 3M steps 6M steps PPO RGPO (sigmoid k = 5)
2247 ± 956 2946 ± 768
1896 ± 66 2696 ± 1114
1116 ± 417 3095 ± 980
Ant-v4 note. Ant-v4 scores (PPO 709, RGPO 1043) are low compared to published long-run baselines (typically > 3000 at 10M steps), because all experiments use a uniform budget of 1M steps. The relative improvement of RGPO over PPO (+47%) and its stability across seeds (σRGPO /σPPO = 0.75) are robust within this budget; absolute scores are expected to grow with more compute. Computational overhead. The sigmoid gate adds a single element-wise operation per sample; measured wall-clock time per iteration differs by less than 1% between RGPO and PPO on all tested environments (≈4.2 s vs. ≈4.0 s per iteration on HalfCheetah-v4), confirming that RGPO matches PPO in computational cost. By contrast, TRPO requires conjugate gradient solving (10 13
iterations) and a backtracking line search each iteration, resulting in wall-clock time of ≈30 s per iteration—approximately 7× slower than PPO or RGPO (4.1 h vs. 0.6 h for 1 × 106 steps). This 7× overhead, combined with the performance gap reported in Table 2, makes TRPO a poor tradeoff: more expensive yet less effective than RGPO, confirming the practical motivation for first-order trust-region methods. 7.3
A BLATION S TUDIES (PART 2: M ETHOD A NALYSIS )
(A) Gating function visualization and comparison. We compare three principled gate designs on HalfCheetah-v4 (3 seeds each, 1M steps, nepochs = 10, β0 = 0.5, max kl = 0.1). The three gates and their theoretical effective gradient weights w(r) = g ′ (r) · r are: Sigmoid (k = 5): g(r) = σ(k(r − 1)), w(r) = k σ(1 − σ) r. The weight w peaks near r = 1 and decays smoothly for both r ≪ 1 and r ≫ 1, giving bounded gradient variance by Theorem 2. This is the gate used throughout the main experiments. Clipped-linear (c = 2.0): g(r) = min(r, c), w(r) = r · 1[r < c]. Equivalent to truncated importance sampling: full IS correction for r ∈ [0, c) and zero gradient for r ≥ c. Unlike the sigmoid, w does not decay for small r (no lower guard), which can amplify gradient noise when the new policy diverges far from the old. Temperature-controlled (β = 1.0): g(r) = σ(β log r) = rβ /(1+rβ ), w(r) = β g(r)(1−g(r)). The weight is a bell curve on log r, always in [0, β/4], making it the most conservative gate. For β < 1 the gate becomes very flat (underfitting); for β ≫ 1 it approaches a step function. Table 4 reports the empirical results. The sigmoid gate achieves the best final return and lowest variance across seeds, confirming its suitability as the default RGPO gate. The clipped-linear gate performs comparably in mean but shows higher variance because the absence of a lower guard allows larger gradient oscillations when the policy makes exploratory excursions (r ≪ 1). The temperature gate is the most stable (lowest std) but pays a systematic return penalty, consistent with its more conservative gradient magnitude.
Table 4: Gating-function comparison on HalfCheetah-v4 (3 seeds, 1M steps each). “Return” is mean±std of the last-20-iteration window averaged over 3 seeds; “Seed std” is the standard deviation across seeds (reproducibility proxy); ESS and mean KL are averaged over all training iterations. Gate Mean return ↑ Seed std ↓ ESS ↑ Mean KL Sigmoid (k = 5) Clipped-linear (c = 2.0) Temperature (β = 1.0)
3414 3181 1886
449 1351 854
0.869 0.962 0.990
0.0198 0.0212 0.0197
The empirical results confirm the theoretical predictions. Sigmoid achieves the highest mean return and, crucially, the lowest cross-seed variance (±449), indicating consistent learning across random initializations. Clipped-linear matches sigmoid in mean (only −7%) but exhibits 3× higher seed variance (±1351). This is consistent with the absence of a lower gradient guard: for r ≪ 1 the weight w(r) = r grows unboundedly toward zero (no damping), so exploratory rollouts produce highvariance gradient estimates that occasionally destabilize learning. Its high ESS (0.962) confirms that all samples receive near-full IS weight, capturing IS variance directly in the gradient. Temperature is the most conservative: its gradient magnitude is bounded at w ≤ β/4 = 0.25, so each sample contributes a very small signal; within the 1M-step budget this translates to systematic underfitting (mean return 1886, −45% vs. sigmoid). The near-unity ESS (0.990) reflects the uniformly small, nearly policy-independent gate weights. Taken together, the sigmoid gate uniquely combines selective acceptance (moderate ESS 0.869) with symmetric damping (gradient decays for both r ≪ 1 and r ≫ 1), as visualised in Figure 4, producing the best bias–variance trade-off among the three designs. 14
Figure 4: Gating function g(r) and acceptance weight w(r) = g ′ (r) · r for the three gate variants. Left: sigmoid (k = 5) is bounded in [0, 1] and centred at r = 1; clipped-linear grows linearly then is hard-capped at c = 2; temperature (σ(β log r)) is also bounded in [0, 1] but rises more gradually. Right: the acceptance weight w(r) controls the effective gradient contribution of each sample. Sigmoid produces a bell-shaped profile that damps both over- and under-represented samples (r ≪ 1 and r ≫ 1). Clipped-linear has w(r) = r for r < c (full IS correction) with no lower guard, exposing gradient variance from exploratory rollouts. Temperature keeps w uniformly small (≤β/4), causing systematic underfitting within a fixed step budget. Full acceptance-weight histograms over training are in Appendix D. (B) Gate sharpness sensitivity. Table 5 sweeps the sigmoid sharpness k on HalfCheetah-v4 (nepochs = 1, β0 = 0.5, 1M steps, seed 1). All RGPO variants used the same hyperparameters except k; PPO (ϵ = 0.2) is provided as the single-epoch baseline. Table 5: Gate sharpness sweep on HalfCheetah-v4. Moderate k = 5 achieves the best bias–variance tradeoff; larger k approaches hard clipping and degrades toward PPO performance; k = 2 is too smooth and underperforms. Method k Avg. return (1M) PPO (ϵ = 0.2)
—
1630
RGPO (sigmoid) RGPO (sigmoid) RGPO (sigmoid) RGPO (sigmoid)
2 5 10 20
1026 1959 1921 1756
The results confirm the theoretical prediction: k = 5 achieves the best bias–variance tradeoff. As k → ∞, the soft gate approaches a hard threshold, and performance converges toward the PPO level (k = 20 yields 1756, close to PPO’s 1630). Very small k (k = 2) produces an overly smooth gate that fails to distinguish high- and low-quality samples, collapsing to 1026—worse than standard policy gradient. This monotone U-shape in k directly validates the bias–variance analysis of Theorem 1: the optimal gate sharpness balances gradient suppression against bias. (C) Bias–Variance tradeoff. Measured gradient variance over training is nearly identical across all three algorithms and all four environments: PPO, RGPO, and AWR all average 1.5 × 10−2 (median also 1.5–1.6 × 10−2 ), with no meaningful difference between methods or environments. This is expected: both PPO and RGPO suppress extreme importance ratios, so gradient variance is bounded in both cases; AWR’s advantage-weighting produces similar raw gradient magnitudes despite its pathological KL behavior. The key qualitative difference predicted by Theorem 2 and Proposition 1—that RGPO guarantees finite variance even when IS ratios are heavy-tailed—manifests not in raw gradient magnitude but in bounded KL (Ablation E): RGPO’s adaptive β prevents the heavytailed ratio regime from arising in the first place, whereas PPO and AWR let KL grow unchecked. 15
Figure 5: Gradient variance per training iteration on four MuJoCo environments (mean ± std across seeds). PPO, RGPO, and AWR converge to virtually identical gradient variance (≈ 1.5×10−2 ) in all environments, confirming that RGPO’s acceptance gate does not amplify gradient noise relative to PPO. Figure 5 visualises gradient variance trajectories for all three algorithms on all four environments; the curves are indistinguishable, confirming the theoretical bound empirically. 2 P P 2 (D) Effective Sample Size (ESS). We track the normalised ESS ESS = i wi / N i wi ∈ [0, 1] with wi = g ′ (rθ,i ) rθ,i for RGPO and wi = clip(rθ,i , 1 − ϵ, 1 + ϵ) for PPO. Table 6 reports multi-seed ESS averages across all four environments (nepochs = 10). Table 6: Mean ESS averaged over all training iterations (1M steps, 3 seeds per method). TRPO makes tiny policy steps (δ = 0.01), so r ≈ 1 and ESS ≈ PPO; RGPO’s sigmoid gate selectively down-weights off-policy samples; AWR concentrates mass on high-advantage samples, yielding the lowest ESS. Environment PPO TRPO RGPO AWR HalfCheetah-v4 Walker2d-v4 Ant-v4 Hopper-v4
0.973 0.974 0.970 0.980
0.977 0.974 0.972 0.969
0.876 0.870 0.871 0.881
0.654 0.704 0.456 0.628
Mean
0.974
0.973
0.875
0.611
PPO’s clipping retains each sample at its full clipped weight, giving a near-perfect ESS (≈ 0.97); TRPO makes similarly small policy steps (hard constraint δ = 0.01), yielding ESS ≈ 0.97 as well. RGPO’s sigmoid gate continuously down-weights off-policy samples, reducing ESS to ≈ 0.875 (a 10% relative reduction). AWR’s advantage-exponential weights concentrate mass on high-advantage samples, giving the lowest ESS (0.46–0.70). The key insight is that high ESS does not imply high performance: TRPO and PPO both achieve ESS ≈ 0.97, yet RGPO outperforms both on Walker2dv4 (+81% over PPO, +122% over TRPO) and Ant-v4 (+47% over PPO, +214% over TRPO) with 16
Figure 6: Normalised Effective Sample Size (ESS) per training iteration on four MuJoCo environments (mean ± std across seeds; TRPO shown dashed). PPO (blue) and TRPO (purple, dashed) both achieve near-perfect ESS (≈ 0.97): PPO via clipping, TRPO because its hard constraint keeps r ≈ 1. RGPO (orange) maintains a stable ESS of ≈ 0.875 as the sigmoid gate selectively downweights off-policy samples. Despite lower ESS than PPO and TRPO, RGPO achieves the highest final performance (Table 2). AWR (green) shows lower and more variable ESS because advantageexponential weights concentrate mass on a small fraction of samples. lower ESS. This confirms the RGPO design philosophy: selective sample contribution (“whether to use a sample”) outweighs maximum sample utilization (“how much to amplify it”). Figure 6 shows the per-iteration ESS trajectories; the clear separation between the four algorithms is stable throughout training. (E) KL divergence analysis. Table 7 summarizes KL divergence statistics for 1M-step runs (nepochs = 10) across all four environments (3 seeds each). We define a KL spike as any iteration where the mean per-minibatch KL exceeds 2∆ = 0.04. Table 7 and Figure 7 reveal the complete KL spectrum across the four methods, spanning more than four orders of magnitude from TRPO (hard constraint) through RGPO (soft gate) and PPO (clip drift) to AWR (no IS correction, catastrophic). TRPO achieves the strictest KL control by design: its hard constraint forces mean KL into [0.005, 0.007], with max KL equal to exactly δ = 0.010 in every environment and 0% spike rate. RGPO maintains 0% spike rate with mean KL in [0.019, 0.021]—only 3× larger than TRPO in absolute terms, yet achieved via a simple scalar gate requiring no second-order computation. PPO exceeds 2∆ in 44–84% of iterations: its clip mechanism bounds each individual gradient step but cannot prevent the cumulative drift that accumulates across 10 update epochs. AWR, with no IS correction, experiences catastrophic drift in every environment: mean KL reaches 1.98 on HalfCheetahv4 and 18.85 on Ant-v4 (max KL = 1000, spike rate ≈ 100%), directly explaining its near-zero performance on Ant and poor performance elsewhere in Table 2. Crucially, Table 2 and Figure 2 show that stricter KL control does not automatically translate to better performance: TRPO’s hard constraint comes at the cost of single-step updates, limiting sample efficiency and yielding lower final returns than RGPO despite superior KL control. RGPO strikes 17
Table 7: KL divergence statistics (1M steps) across all four environments. TRPO enforces a hard constraint (δ = 0.01); its max KL equals exactly δ in all environments. RGPO’s adaptive β penalty nearly matches TRPO’s KL control at 7× less computational cost. PPO’s clip allows cumulative drift across 10 epochs. AWR undergoes catastrophic drift with no IS correction. Spike rate = fraction of iterations with KL > 0.04 (= 2∆). Environment Method Mean KL Max KL Spike rate HalfCheetah-v4
AWR PPO RGPO TRPO
1.983 0.243 0.021 0.007
58.26 11.59 0.038 δ = 0.010
99% 79% 0% 0%
Walker2d-v4
AWR PPO RGPO TRPO
0.795 0.119 0.020 0.007
28.02 0.621 0.042 δ = 0.010
92% 73% 0% 0%
Ant-v4
AWR PPO RGPO TRPO
18.85 0.409 0.020 0.005
1000 7.271 0.045 δ = 0.010
100% 84% 0% 0%
Hopper-v4
AWR PPO RGPO TRPO
0.085 0.043 0.019 0.007
0.459 0.196 0.047 δ = 0.010
73% 44% 0% 0%
Figure 7: Mean KL divergence DKL (πθ ∥πold ) averaged over 1M training steps, grouped by environment (log scale). Four algorithms span more than four orders of magnitude: TRPO (purple, hatched) is anchored at its hard constraint δ = 0.01; RGPO (orange) stays just below the spike threshold (2∆ = 0.04); PPO (blue) exceeds the threshold in most environments; AWR (green) undergoes catastrophic drift (KL > 1 everywhere). The dashed red line marks the spike threshold 2∆ = 0.04. the optimal trade-off: first-order cost, near-TRPO KL control, and 10× more gradient steps per rollout. This result provides the clearest empirical confirmation of Theorem 3: the improvement guarantee requires DKL (πθ ∥πold ) ≤ δ at every update, a condition RGPO satisfies universally while PPO violates it 44–84% of the time and AWR violates it in 73–100% of iterations. Figure 7 summarises the full KL spectrum in a single bar chart; Figure 8 shows the per-iteration dynamics that underlie it. (F) Adaptive KL penalty coefficient β and AWR failure analysis. We sweep the initial penalty coefficient β0 ∈ {0.2, 0.5, 1.0} on HalfCheetah-v4 (nepochs = 10, seed 1, 1M steps) and find that 18
Figure 8: KL divergence DKL (πθ ∥πold ) per training iteration on four MuJoCo environments (log scale; mean ± std across seeds; dashed red line = spike threshold 2∆ = 0.04; TRPO shown with dashed purple line). The log scale reveals the full spectrum: AWR (green) undergoes catastrophic drift (KL > 103 on Ant-v4); PPO (blue) produces frequent spikes above the threshold; RGPO (orange) stays tightly below the threshold; TRPO (purple, dashed) lies near the floor, constrained to exactly δ = 0.01 by its hard trust-region. Despite TRPO’s strictest KL control, RGPO achieves higher final performance (Table 2) at 7× less computational cost.
β0 = 0.5 achieves the best return (3623) versus 2805 for β0 = 0.2 and 3438 for β0 = 1.0. A tighter KL target (∆ = 0.01 with β0 = 0.5) yields 3523, slightly below the default ∆ = 0.02. These results confirm that the adaptive schedule (Eq. 26) is robust across a moderate range of β0 , and that β0 = 0.5, ∆ = 0.02 provide a good starting point. The AWR comparison (Table 2) reveals a deeper point: AWR’s loss −E[exp(A/β) log πθ ] is a pure behavior-cloning objective with no IS correction. Over 10 epochs, the policy drifts dramatically from the behavior policy (mean KL = 1.98 on HC and 18.85 on Ant, ≥ 99% spike rate on all environments), invalidating the advantage estimates and collapsing performance. This failure shows that the importance ratio rθ = πθ /πold in RGPO’s gate is not a peripheral detail: it provides the implicit IS correction that keeps the multi-epoch updates on-policy and the KL tightly controlled. In other words, RGPO’s superiority over AWR is not merely due to the gating function but to the fact that the gate is anchored to the IS ratio, ensuring coherent multi-epoch optimization. 7.4
RLHF / P REFERENCE A LIGNMENT
We apply RGPO to online preference-based fine-tuning on the Anthropic HH-RLHF (helpful) dataset (Bai et al., 2022) using Qwen2.5-1.5B-Instruct as the policy and OpenAssistant/reward-model-deberta-v3-large-v2 (He et al., 2021) as the automated reward model (43,835 prompts). RGPO’s dual-ratio gate r = max(πθ /πold , πθ /πref ) is a natural fit for the RLHF objective: it simultaneously enforces a trust region relative to the immediately preceding policy πold and an alignment anchor relative to the frozen reference model πref , suppressing both short-range instability and long-range alignment drift within a single differentiable gate. 19
Table 8: RLHF preference-alignment results on Anthropic HH-RLHF (helpful subset). Policy: Qwen2.5-1.5B-Instruct; reward model: OpenAssistant/reward-model-deberta-v3-large-v2. All methods train for 400 iterations; metrics averaged over iter 300–400 (mean ± std, 3 seeds). Reward = automated RM score; KLref = DKL (πθ ∥πref ). † DPO is offline and has no online reward signal. Bold = best among online RL methods. Reward ↑
KLref ↓
PPO-RLHF GRPO DPO (Rafailov et al., 2023)
+0.211 ± 0.046 +0.236 ± 0.023 N/A†
0.434 ± 0.054 0.778 ± 0.153 0.079 ± 0.020
RGPO (ours)
+0.243 ± 0.056
0.364 ± 0.019
Method
Setup. All methods share identical hyperparameters: batch size 4, group size K = 4 responses per prompt, learning rate 10−6 (Adam), max prompt/response length 256 tokens, and 3 independent random seeds. Each method trains for 400 iterations. Metrics are reported over the convergence window iter 300–400. RGPO uses the dual-gate variant (Eq. 18) with βref = 0.05 (matching PPORLHF) and sharpness k = 5.
Baselines. PPO-RLHF (Ouyang et al., 2022) is the standard online RLHF algorithm with an explicit KL-to-reference penalty. GRPO (Shao et al., 2024) uses group-relative advantage normalisation with no explicit reference anchor. DPO (Rafailov et al., 2023) is an offline method that optimises a closed-form preference objective without an online reward model; it serves as a KLreference point (no online reward is available for direct comparison).
Results. Figure 9 and Table 8 summarise the outcomes over 3 seeds and the evaluation window iter 300–400. Reward. All online methods learn positive rewards, but reward acquisition rates differ substantially (Figure 9a). At eval time, RGPO achieves the highest reward of all online RL methods: +0.243 ± 0.056, a +14.8% gain over PPO-RLHF (+0.211 ± 0.046) and a +2.8% gain over GRPO (+0.236 ± 0.023). KL divergence (alignment tax). We report DKL (πθ ∥πref ) as the primary stability metric in RLHF; it measures long-range alignment drift from the frozen reference model and directly quantifies the risk of reward hacking. (The per-minibatch KLold spike threshold used in MuJoCo is not a wellcalibrated indicator in the language setting, where sequence-level log-probabilities are inherently larger.) Figure 9(b) reveals a qualitatively distinct picture for each method. GRPO’s KL grows monotonically throughout training (from ≈ 0.25 at iter 1 to ≈ 0.74 at iter 400), a textbook signature of reward over-optimisation (Gao et al., 2023): without a reference anchor, the policy drifts progressively further from the base model to chase reward. PPO-RLHF keeps KL roughly stable at ≈ 0.43, controlled by its explicit KL penalty. RGPO stabilises at the lowest KL among online RL methods: 0.364 ± 0.019, which is 16.0% below PPO-RLHF (0.434) and 53.1% below GRPO (0.778). The dual gate achieves this without any additional regulariser beyond βref = 0.05: suppressing updates with πθ /πref ≫ 1 is equivalent to a differentiable, sample-level alignment check built directly into the gradient computation. Reward–KL Pareto dominance. Figure 10 plots each method in the reward–KL plane (small markers = individual seeds; large markers = seed means). GRPO occupies the high-reward / highKL corner, achieving competitive rewards at the cost of 2.1× the reference drift of PPO. PPO-RLHF sits at moderate reward and moderate KL. RGPO is the unique Pareto-dominant method: it achieves the highest reward and the lowest KL among all online RL methods simultaneously, confirmed across all 3 seeds. DPO’s KL is lowest overall (0.079) but provides no online reward signal. This Pareto improvement demonstrates that RGPO’s differentiable gate simultaneously promotes reward acquisition and restrains alignment drift — a combination that neither PPO’s hard clipping nor GRPO’s group-relative baseline achieves. 20
(a) Reward model score
(b) KL divergence to reference model
PPO-RLHF
PPO-RLHF
GRPO
1.0
GRPO
1.0
RGPO (ours)
DPO RGPO (ours)
DKL(πθ‖πref) ↓
Reward score ↑
Eval window (iter 300–400) 0.5
0.0
-0.5
Eval window (iter 300–400)
0.8
0.6
0.4
0.2
-1.0 0
50
100
150
200
250
300
350
400
0.0
0
50
100
Training iteration
150
200
250
300
350
400
Training iteration
Figure 9: RLHF training dynamics on Anthropic HH-RLHF (Qwen2.5-1.5B-Instruct, 400 iterations, 3 seeds; shaded band = ±1 std; grey region = evaluation window iter 300–400). (a) Reward: RGPO achieves the highest reward of all online RL methods. (b) KL to reference: GRPO’s KL grows monotonically (reward over-optimisation); PPO-RLHF stabilises around 0.43; RGPO stabilises at the lowest KL (≈ 0.36), showing the dual gate simultaneously controls trust-region and alignment drift. Reward–KL efficiency frontier (large markers = 3-seed mean; small = individual seeds) 0.30
← Ideal: low KL high reward ↑
Reward model score ↑
0.25 Pareto improvement
0.20
0.15
0.10
0.05
DPO (no RM reward)
GRPO PPO-RLHF DPO (no RM) RGPO (ours)
0.00 0.0
0.2
0.4
0.6
0.8
DKL(πθ‖πref) ↓
Figure 10: Reward–KL efficiency frontier (eval window iter 300–400, 3 seeds). Small markers = individual seeds; large markers = 3-seed mean. RGPO (♦) is the unique Pareto-dominant point among online RL methods: it achieves the highest reward and the lowest KL simultaneously, confirmed across all seeds. GRPO achieves competitive reward but at 2.1× the alignment drift of RGPO. DPO achieves the lowest overall KL as an offline method but has no online reward signal during training.
8
D ISCUSSION
8.1
W HY R EJECTION W ORKS B ETTER T HAN R EWEIGHTING
Importance sampling reweights all samples, including highly off-distribution ones where the ratio rθ is unreliable. RGPO instead selects samples: transitions that are compatible with the current policy contribute fully, while incompatible ones are suppressed. This mirrors the intuition behind data filtering in supervised learning and rejection sampling in Bayesian inference. 21
The AWR comparison in Section 7.2 provides further evidence: AWR also uses advantage-weighted selection, but its gate is anchored to the advantage A rather than the importance ratio rθ . Without the IS anchoring, AWR cannot control how much the policy drifts per iteration, leading to mean KL of 1.98 on HalfCheetah-v4 and 18.85 on Ant-v4 per update (vs. RGPO’s 0.019–0.021 across all environments). This confirms that the IS ratio in RGPO’s gate serves a dual role: it selects trustworthy samples and implicitly enforces a trust region without any additional constraint. 8.2
C ONNECTION TO RLHF AND A LIGNMENT
The RLHF pipeline commonly uses rejection sampling to filter model outputs before finetuning (Gulcehre et al., 2023). RGPO provides a differentiable, end-to-end generalization of this idea, suggesting that rejection-based selection is a principled component of both standard RL and modern alignment methods. 8.3
L IMITATIONS • Bias. RGPO introduces a controllable bias; for tasks requiring strict unbiasedness (e.g., off-policy evaluation), the full IS estimator should be preferred. • Hyperparameter sensitivity. Performance depends on the choice of g and sharpness k; we provide three principled options and empirical guidance (k = 5 works well across environments), but some task-specific tuning may still be needed. • Hopper-v4. RGPO achieves essentially identical mean return to PPO on Hopper-v4 (1727 vs. 1732, ≈ 0% difference), with significantly lower seed variance (238 vs. 661). While average performance is comparable, RGPO is substantially more reproducible on this environment; the sigmoid gate may still be overly conservative on certain reward landscapes, and the smooth-PPO gate (Ablation B) may be worth exploring. • Ant-v4 short-horizon. All experiments use a fixed budget of 1M steps; Ant-v4 benefits more from longer training, so reported absolute scores are below published long-run benchmarks. • Theoretical bounds. Guarantees in Theorems 1–3 rely on bounded advantages and Lipschitz g; violations in practice (e.g., unbounded rewards) may weaken the bounds.
9
C ONCLUSION
We proposed Rejection-Gated Policy Optimization (RGPO), a unified framework for policy optimization that replaces importance sampling with a differentiable acceptance gate g(rθ ) ∈ [0, 1]. Through the effective gradient weight w(r) = g ′ (r) · r, RGPO unifies existing methods: REINFORCE is an exact special case in the on-policy limit (r ≡ 1); TRPO and PPO are gradient-level correspondences whose gate functions fall outside [0, 1] (IS reweighting regime); AWR is a closely related method. RGPO provides theoretical guarantees on bias, variance, and policy improvement, and extends naturally to RLHF-style preference optimization. Experiments on four MuJoCo continuous-control benchmarks confirm the theoretical predictions. RGPO outperforms PPO on Walker2d-v4 (+81%, p = 0.047, one-tailed Welch t-test) and Ant-v4 (+47%, p = 0.064, RGPO > PPO in all paired comparisons), and achieves statistically comparable performance on HalfCheetah-v4 and Hopper-v4 (differences ≈ 0%, within one standard error). Across all four environments, RGPO is dramatically more stable across seeds: its standard deviation is 0.36–0.75× that of PPO, and its coefficient of variation stays below 27% on all environments. RGPO maintains a KL spike rate of 0% (maximum KL = 0.047 across all environments and seeds) compared to 44–84% for PPO, directly confirming Theorem 3. AWR (advantage-weighted regression, the closest RGPO family member) fails catastrophically with multi-epoch training (73–100% KL spike rate, mean KL = 1.98 on HalfCheetah and 18.85 on Ant), demonstrating that the IS ratio in RGPO’s gate is essential for stable multi-epoch optimization. RGPO significantly outperforms PPO at 6M steps on HalfCheetah-v4 (3095 vs. 1116; p = 0.028, one-tailed Welch t-test, n = 3 seeds) while PPO regresses across all three budget levels. In the RLHF preference-alignment setting (Qwen2.5-1.5B-Instruct, Anthropic HH-RLHF, n = 3 seeds), RGPO’s dual-ratio gate achieves the unique Pareto-dominant outcome: the highest reward among online RL methods 22
(+0.243, +14.8% vs. PPO-RLHF) and the lowest KL divergence to the reference model (0.364, −16.0% vs. PPO-RLHF, −53.1% vs. GRPO). This demonstrates that the dual gate provides a principled, differentiable alignment constraint that is strictly more efficient than either clipping (PPO) or an unanchored group baseline (GRPO). Extension to discrete action spaces (e.g., Atari) and larger language models is left for future work.
R EFERENCES Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407–1416. PMLR, 2018. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, 2023. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with disentangled attention. arXiv preprint arXiv:2006.03654, 2021. Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, volume 2, pp. 267–274, 2002. Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657, 2023. Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. In arXiv preprint arXiv:2006.09359, 2020. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022. Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. In arXiv preprint arXiv:1910.00177, 2019. Doina Precup, Richard S Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. Proceedings of the Seventeenth International Conference on Machine Learning, 2000. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889–1897. PMLR, 2015a. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015b. 23
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. In arXiv preprint arXiv:2402.03300, 2024. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. In arXiv preprint arXiv:1611.01224, 2016. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. Linxiao Zhao et al. Jackpot: Optimal budgeted rejection sampling for policy optimization. arXiv preprint arXiv:2501.12342, 2024.
A
F ULL P ROOFS
A.1
P ROOF OF T HEOREM 1 (B IAS B OUND )
Proof. The true policy gradient using importance sampling is: ∇J(θ) = Eπold [rθ ∇θ log πθ (a|s) A] .
(27)
∇θ LRGPO = Eπold [g ′ (rθ ) rθ ∇θ log πθ (a|s) A] .
(28)
∆ = ∇θ LRGPO − ∇J(θ) = Eπold g ′ (rθ ) − 1 rθ ∇θ log πθ (a|s) A .
(29)
The RGPO gradient is: The difference is:
Taking norms and applying the triangle inequality: ∥∆∥ ≤ Eπold [|g ′ (rθ ) − 1| rθ ∥∇θ log πθ ∥ |A|] ≤ Amax · ∥∇θ log πθ ∥ · Eπold [|g ′ (rθ ) rθ − rθ |] ,
(30)
which gives the stated bound with C = Amax · ∥∇θ log πθ ∥. A.2
P ROOF OF T HEOREM 2 AND P ROPOSITION 1
Proof of Theorem 2. Step 1: RGPO variance upper bound. Let Z = ∇θ log πθ (a|s) · Aold . Since |w(r)| ≤ c: Y 2 = w(rθ )2 Z 2 ≤ c2 Z 2 . (31) Taking expectations: E[Y 2 ] ≤ c2 E[Z 2 ] ≤ c2 σ 2 . (32) 2 2 2 2 2 Since Var(Y ) = E[Y ] − (E[Y ]) ≤ E[Y ], we have Var(Y ) ≤ c σ . Step 2: IS variance lower bound (no independence assumed). Var(X) = E[X 2 ] − (E[X])2 = E rθ2 Z 2 − (E[X])2 .
(33)
Note that we do not factor E[rθ2 Z 2 ] into E[rθ2 ]·E[Z 2 ]; such factorization would require independence of rθ and Z, which does not hold in general since A depends on the full trajectory. Step 3: Comparison (Variance Dominance Corollary). Combining Steps 1 and 2: Var(X) − Var(Y ) ≥ E rθ2 Z 2 − (E[X])2 − c2 σ 2 . 24
(34)
If E[rθ2 Z 2 ] ≥ Kσ 2 for K > c2 , then: Var(X) − Var(Y ) ≥ (K − c2 )σ 2 − (E[X])2 > 0,
(35)
provided (E[X])2 < (K − c2 )σ 2 , establishing Var(Y ) < Var(X). Proof of Proposition 1. IS variance is infinite. Let α ≤ 2. The tail condition P(rθ > t) ∼ Ct−α implies, via the layer-cake formula: Z ∞ Z ∞ Z ∞ √ E[rθ2 ] = P(rθ2 > s) ds = P(rθ > s) ds ≥ C s−α/2 ds. (36) 0
0
1
Since α/2 ≤ 1, the integral diverges, so E[rθ2 ] = +∞.
By the positive-lower-bound assumption, there exists ϵ > 0 and an event E with P(E) > 0 on which |Z| ≥ ϵ. Since rθ and Z are not independent, we lower-bound E[X 2 ] by conditioning on E: E[X 2 ] = E[rθ2 Z 2 ] ≥ ϵ2 E[rθ2 1E ].
(37)
When rθ is heavy-tailed with E[rθ2 ] = +∞ and is independent of (or positively associated with) E, E[rθ2 1E ] = +∞, so Var(X) = +∞. RGPO variance is finite. Since |w(rθ )| ≤ c uniformly, E[Y 2 ] ≤ c2 σ 2 < ∞ by Theorem 2, hence Var(Y ) ≤ c2 σ 2 < ∞. A.3
P ROOF OF T HEOREM 3 (A PPROXIMATE P OLICY I MPROVEMENT )
Proof. By the performance difference lemma (Kakade & Langford, 2002): J(θ) − J(θold ) = Eπθ [Aold (s, a)] = Eπold [rθ Aold (s, a)].
(38)
Decompose by adding and subtracting g(rθ ): J(θ) − J(θold ) = Eπold [g(rθ ) Aold ] +Eπold [(rθ − g(rθ )) Aold ] . {z } |
(39)
=LRGPO (θ)
For the second term, since g has Lipschitz constant Lg : |r − g(r)| ≤ Lg |r − 1|.
(40)
By Pinsker’s inequality and the KL bound DKL (πθ ∥πold ) ≤ δ: p √ Eπold [|rθ − 1|] ≤ 2 DKL (πθ ∥πold ) ≤ 2δ.
(41)
Combining with |Aold | ≤ Amax : |E[(rθ − g(rθ )) Aold ]| ≤ Lg Amax
√
√ 2δ = C
δ,
(42)
√
where C = Lg√ Amax 2 is a true constant (independent of δ). Rearranging gives J(θ) ≥ J(θold ) + LRGPO (θ) − C δ.
B
I MPLEMENTATION D ETAILS
Network architecture. We use a two-layer MLP with hidden size 256 and tanh activations for both the policy and value networks, following the standard PPO implementation. Hyperparameters. • Learning rate: 3 × 10−4 (Adam optimizer, ϵ = 10−5 ) • Discount factor: γ = 0.99 • GAE λ = 0.95 • Mini-batch size: 64 • Number of epochs per iteration: nepochs = 10 25
• Rollout steps per iteration: 2048 • Observation normalization: running mean/variance (both methods) • Reward normalization: running variance scaling (both methods) • Value function clipping: ϵV = 0.2 (both methods) • Sigmoid sharpness: k = 5 (RGPO default; see Ablation B for sweep) • Adaptive KL penalty (RGPO only): – Target KL: ∆ = 0.02 – Initial coefficient: β0 = 0.5 – Coefficient bounds: β ∈ [0.01, 5] – Update rule: Eq. (26) (after every iteration) – Hard KL backstop: τ = 0.1 (triggers early exit from epoch loop if mean KL exceeds τ ; the adaptive β is the primary trust-region mechanism) • PPO clipping coefficient: ϵ = 0.2 Sigmoid gating details. The sigmoid gate g(r) = σ(k(r − 1)) is centered at r = 1 so that a ratio of exactly 1 (no policy change) yields α = 0.5. The sharpness k controls the transition: large k approaches hard thresholding (similar to PPO clipping), while small k approaches uniform weighting.
C
A DDITIONAL E XPERIMENTS
Sensitivity to k (completed). Table 5 of the main paper. β sensitivity (completed).
Results for k ∈ {2, 5, 10, 20} on HalfCheetah-v4 are reported in
Results for β0 ∈ {0.2, 0.5, 1.0} are reported in Section 7.3, Ablation F.
Multi-environment and multi-seed evaluation (completed). Results for PPO and RGPO on all four MuJoCo environments (HalfCheetah-v4, Walker2d-v4, Hopper-v4, Ant-v4) with 3 seeds are reported in Table 2. AWR baseline comparison (completed). AWR (β = 1.0, nepochs = 10) results are included in Table 2 and analyzed in Ablation F and Section 8. Standard single-epoch AWR (nepochs = 1) is left for future work. Large action spaces (completed via Ant-v4). Ant-v4 (8-dimensional action, 111-dimensional observation) results confirm that the gating mechanism scales to higher-dimensional continuous control. Full results appear in Table 2. RLHF fine-tuning (completed). Preference-based fine-tuning of Qwen2.5-1.5B-Instruct on the Anthropic HH-RLHF dataset is reported in Section 7.4 and Table 8.
D
G ATING F UNCTION P LOTS
This appendix provides full visualisations of the three gating functions compared in Section 7.3 (Ablation A) and their resulting acceptance-weight distributions over training. Interpretation. in Table 4:
The qualitative differences in g(r) and w(r) directly explain the empirical results
• Sigmoid uniquely provides a lower gradient guard (w → 0 as r → 0), preventing destabilisation from near-zero-ratio samples that would otherwise receive full gradient weight under clipped-linear. This translates to the lowest cross-seed variance (±449) observed. 26
Figure 11: Gating functions g(r), acceptance weights w(r) = g ′ (r) · r, and weight histograms over training. (a) The gating function g(r) for each variant. Sigmoid is bounded in [0, 1]; clippedlinear grows linearly then is hard-capped; temperature (σ(β log r)) is also bounded in [0, 1] but is symmetric around r = 1. (b) The acceptance weight w(r) = g ′ (r) · r, which modulates the effective contribution of each sample to the policy gradient. Sigmoid produces a bell-shaped w centred near r = 1, damping both under- and over-represented samples. Clipped-linear has w(r) = r for r < c (full IS correction) and w = 0 for r > c (hard exclusion), exposing gradient variance from offpolicy samples with r < c. Temperature yields a bell-shaped w(r) on log r, bounded above by β/4 and vanishing as r → 0 or r → ∞, causing uniformly small gradient contributions and systematic underfitting. (c1–c3) Simulated acceptance-weight histograms at early, mid, and late training stages 2 using a log-normal model for the IS ratio (r ∼ LogNormal(0, σlog r ) with σlog r decreasing from 0.35 to 0.08), consistent with empirical ratio distributions of near-on-policy updates. Sigmoid concentrates weight around the centre while suppressing outlier samples throughout training, explaining its favourable bias–variance trade-off (Table 4). • Clipped-linear lacks a lower guard: for r ≪ 1 the weight w(r) = r grows toward zero without damping, allowing exploratory rollouts to produce high-variance gradient estimates that occasionally destabilise training (±1351 seed variance). • Temperature yields a bell-shaped w(r) on log r, bounded above by β/4 = 0.25 and vanishing as r → 0 or r → ∞. Each sample therefore contributes a very weak signal; within the 1M-step budget this leads to systematic underfitting (mean return 1886, −45% vs. sigmoid).
27