Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
arXiv:2605.12380v1 [cs.LG] 12 May 2026
Rasool Fakoor∗ Murdock Aubry
Nicholas Stranges Boson AI
Alexander J. Smola
Abstract Reinforcement learning (RL) is structurally harder than supervised learning because the policy changes the data distribution it learns from. The resulting fragility is especially visible in large-model training, where the training and rollout systems differ in numerical precision, sampling, and other implementation details. Existing methods manage this fragility by adding more hyper-parameters to the training objective. Each may help in its tuned regime but makes the resulting algorithm more sensitive to its configuration, requiring retuning whenever the task, model scale, or distribution mismatch changes. This fragility traces to two concerns that current objectives entangle through hyper-parameters set before training begins. The first is a trust-region concern, in that each update should not move the policy too far from its current value. The second is an off-policy concern, in that data collected by older or different behavior policies should influence the current update only to the extent that the update remains reliable. Mishandling off-policy data is consequential, yet such data can still carry useful signal that must be weighted adaptively as training proceeds. Neither concern is a constant to set before training, and their severity is reflected in the policy-ratio distribution of the current batch. We present a simple yet effective batch-adaptive objective that replaces fixed clipping with the normalized effective sample size of the policy ratios. The same statistic caps the score-function weight and sets the strength of an off-policy regularizer. When the ratios are nearly uniform, the update stays close to the usual on-policy score-function update. When stale or mismatched data cause ratio concentration, the update tightens automatically while retaining a nonzero learning signal on highratio tokens. Experiments across a wide range of settings show that our method matches or exceeds tuned baselines, introducing no new objective hyper-parameters and removing several existing ones.2
1
Introduction
Post-training large (language) models often means optimizing a policy by sampling completions and scoring them with a reward model, a human-preference-derived objective, or a task verifier [39, 27, 21, 4]. Because these scores are assigned to discrete sampled text and are usually not differentiable through the policy, policy-gradient reinforcement learning (RL) is a natural tool for increasing expected score. RL is a fragile optimization process whose outcomes depend heavily on hyperparameters, implementation details, and random seeds [12, 5, 2, 8, 14], and this fragility sharpens at large-model scale, where runs are expensive, rollouts are costly to regenerate, and small choices (clip range, allowed rollout staleness, etc.) determine whether training improves, stalls, or becomes unstable. These choices are often retuned for each task, model scale, and rollout infrastructure. ∗ Corresponding author: [email protected] 2 The code is available at https://github.com/FeynRL-project/FeynRL.
Preprint.
Two concerns underlie this fragility. The first is a trust-region concern, that each gradient update should not move the policy too far from its current value, regardless of where the data come from [24, 25]. The second is an off-policy concern, that data collected by older or different behavior policies should influence the current update only to the extent that the update remains reliable [9, 10, 6]. The trust region controls the size of an update, while off-policy correction weights data by how reliably it can be used. Off-policy data are common in practice and arise from several sources, including optimizer steps reused across iterations, demonstrations or rollouts from older or heterogeneous policies (the batchRL setting [16, 11, 17]), and practical mismatches between the rollout and training engines such as numerical precision, sampling strategy, or architectural differences [22, 19]. Mishandling such data is consequential, since stale or mismatched samples destabilize training, yet discarding it forfeits sample efficiency because each rollout is then consumed by a small number of updates [7]. Current large-model RL methods commit to one regime, either generating fresh on-policy rollouts after every update or applying fixed off-policy corrections tuned for a particular staleness, which forces a choice between sample efficiency and stability before training begins. These methods conflate the two concerns inside a single fixed-clip mechanism [25, 26], where a fixed range bounds the per-step policy change and also caps the off-policy variance. This is simple, but the range is a strong algorithmic choice made before training starts, determining which tokens receive gradient, how much stale data can be reused, and how quickly the policy is allowed to move. Decoupled-loss methods separate the proximal policy used for clipping from the behavior policy used for importance correction, improving learning from stale data [13], and recent variants add asymmetric clipping, dynamic sampling, and related modifications [34]. These methods recognize the issue but still leave the amount of trust as an external choice, and the burden of tuning remains. The premise of this paper is that fixed clipping itself, not the value of the clip range, is what causes the fragility, and iterating on the clip (symmetric, asymmetric, sequence-level) leaves the underlying problem in place. Our contribution is to step back from the fixed-clip framework rather than propose another variant, and measure how on-policy the current batch is and use that measurement to drive the update. The behavior-policy mismatch, defined as how different the current policy is from the policy that produced each completion, can be summarized by an effective sample size (ESS) [15, 7] statistic over the per-token importance ratios. ESS is close to one when the ratios are nearly uniform, and falls when a few tokens dominate. We use only ESS to drive both the score-function cap and the regularizer in the surrogate objective, removing the fixed clip range entirely. Concretely, we adopt the P3O objective of [7] for large-model post-training, the first such application, and show that it removes the fixed clip without introducing any new hyper-parameter. The same objective handles on-policy and off-policy data automatically, with the cap loose on fresh data and tight on stale data, and unifies what prior methods split into specialized losses. Across a range of regimes, this objective compares favorably with tuned baselines despite carrying fewer hyperparameters, demonstrating that adaptivity, not careful clip tuning, is what drives stability at scale. Removing the clip rather than re-parameterizing it also opens space for downstream work. Because P3O makes off-policy data a feature rather than a burden, future methods can combine off-policy reuse (for sample efficiency) with on-policy collection (for exploration) inside one objective, without retuning a clip range, behavior-weight cap, or staleness budget for each setting. We view this paper as a foundation for that direction.
2
Background
Given a pre-trained model, our objective is to fine-tune it using reinforcement learning so that completions sampled from it receive high scores. For a prompt x, the model assigns a probability to a completion y = (y1 , . . . , yT ) autoregressively, πθ (y | x) =
T Y
πθ (yt | x, y<t ),
(1)
t=1
where each yt is a token from a finite vocabulary V, y<t = (y1 , . . . , yt−1 ) is the prefix at position t, and T is the completion length. To simplify notation, we write c<t = (x, y<t ) for the conditioning context. 2
Token-level MDP. We cast this autoregressive generation as a finite-horizon MDP in which the state at position t is st = (x, y<t ), the action is the next token yt ∈ V, and a scalar terminal reward r(x, y) ∈ R is assigned to the full completion.3 The agent’s objective is to maximize the expected return J(πθ ) = Ex, y∼πθ (·|x) [ r(x, y) ], where x is a prompt from the training set and y is a completion sampled from πθ . Using the likelihood-ratio trick [32], the policy gradient of J(πθ ) is " T # X ∇θ J(πθ ) = Ex, y∼πθ (·|x) r(x, y) ∇θ log πθ (yt | x, y<t ) . (2) t=1
This is known as the REINFORCE policy-gradient estimator [32]. It is an on-policy estimator: it optimizes the same policy that was used to collect the data. Despite its simplicity, the policy-gradient estimator has high variance because the update uses sampled trajectories and their corresponding rewards as noisy estimates of how good each action was, so the same policy can produce very different rewards across rollouts, especially with long horizons and delayed rewards [28]. The standard remedy is to subtract a control variate (baseline) b that does not depend on the sampled action yt , replacing r by an advantage A = r −b. For any such baseline the gradient remains unbiased while its variance is typically reduced substantially. Actor-critic methods learn a state-dependent value function Vϕ (st ) = Vϕ (x, y<t ) and use it as the baseline at each step. Group-relative advantage. Learning a value function alongside the policy is usually expensive when training large models, because the value function can itself be a network of comparable size to the policy and inherits all the difficulties of training a large model, from training instability to challenges at scale. GRPO [26] sidesteps this by replacing the learned baseline with a within-batch reward statistic. For each prompt xp , it samples a group of G completions {yp,j }G j=1 from the current policy and uses the group’s reward statistics as the baseline. Writing rp,j := r(xp , yp,j ) for the reward of completion j in group p, the group-relative advantage is PG 1 rp,j − G k=1 rp,k Ap,j = r , (3) 2 PG PG 1 1 +ϵ k=1 rp,k − G l=1 rp,l G where ϵ > 0 avoids division by zero, and the variance in the denominator is computed in the population form to match the original GRPO definition. The group mean acts as a per-prompt baseline that replaces the value function, and the group standard deviation rescales the advantage so it is comparable across prompts whose reward magnitudes differ. Trust Region Policy Optimization. A separate optimization concern, beyond the variance and baseline issues above, is that a single gradient update can move πθ too far from its current value and destabilize training, even when the data are on-policy. Trust-region methods address this by constraining each update to stay close to the current policy. Where TRPO solves this problem with a complex second-order method [24], Proximal Policy Optimization (PPO) [25] takes a simpler approach by using first-order methods to keep new policies close to old. Concretely, PPO samples data from a fresh snapshot πθold taken at the start of the current optimizer epoch, and clips the per-token policy ratio ρt (θ) before applying it, LPPO (θ) = −Eπθold min ρt (θ) A, clip(ρt (θ), 1 − ϵℓ , 1 + ϵh ) A ,
ρt (θ) =
πθ (yt | x, y<t ) πθold (yt | x, y<t ) (4)
with the clip range [1 − ϵℓ , 1 + ϵh ] fixed before training. Because πθold is close to πθ in PPO’s intended setting, ρt stays close to one for most tokens and the clip serves a clean trust-region role: bounding the per-step policy change. From on-policy to off-policy data. The on-policy estimator in Eq. (2) is unbiased but sampleinefficient, since each rollout is consumed by the gradient step that produced it. In practice the data we train on is rarely drawn from πθ exactly. This may be because we take several optimizer steps on each batch of completions and reuse them across training iterations, because we want to use data collected by an earlier or entirely different policy (for instance demonstrations or rollouts from a prior run), or because of a more general discrepancy between the policy we sample from and the 3 The formulation extends to per-token rewards r (x, y t ≤t ) by replacing r(x, y) with rt (x, y≤t ) inside the sum in Eq. (2).
3
policy we optimize. In all of these cases the data was sampled from a behavior policy πb but the loss is evaluated under the current πθ , so the empirical gradient computed on samples from πb is a biased estimate of ∇θ J(πθ ). The standard fix is importance sampling [9, 10, 23], which restores unbiasedness by changing the measure from πθ to πb . For any function f , Z Z πθ (x) πθ (x) Ex∼πθ [f (x)] = πθ (x) f (x) dx = πb (x) f (x) dx = Ex∼πb f (x) , (5) πb (x) πb (x) provided that the Radon-Nikodym derivative dπθ /dπb is well defined. Applied to the RL objective, this rewrites the expected return as an expectation under the behavior policy, h i πθ (y | x) J(πθ ) = Ex, y∼πθ (·|x) r(x, y) = Ex, y∼πb (·|x) r(x, y) , (6) πb (y | x) so J(πθ ) can be evaluated on data drawn from πb . In our token-level setting we work with the per-token importance ratio ρt (θ) =
πθ (yt | x, y<t ) , πb (yt | x, y<t )
(7)
which generalizes the PPO-snapshot ratio in Eq. (4) to any behavior policy πb , and apply this correction directly inside the per-token gradient. Remark (Importance sampling is unbiased but high variance). Importance sampling restores unbiasedness, but the per-token ratios ρt (θ) are bounded below by zero and unbounded above, so a few tokens that were unlikely under πb can produce ratios many orders of magnitude larger than one and dominate the gradient estimate. The standard remedy is to clip ρt (θ) before applying it, but as we discuss in Sec. 3, fixed clipping is itself fragile. RL in language modeling. In practice we do not maximize J(πθ ) alone. When fine-tuning language models with RL, the objective is regularized by a KL penalty toward a frozen reference policy πθ0 , the same policy used to initialize πθ (typically a pre-trained or supervised-fine-tuned model, the latter trained on ground-truth tokens via teacher forcing [3]),4 " # T X JLM (πθ ) = Ex, y∼πθ (·|x) r(x, y) − η KL πθ (· | x, y<t ) πθ0 (· | x, y<t ) , (8) t=1
Here η > 0 controls how far πθ is allowed to drift from πθ0 [39, 27, 21]. The KL anchor is an effective regularizer, intended to keep πθ from drifting too far from πθ0 and potentially from forgetting its language-modeling capabilities. The per-token KL is combined with the policy-gradient term under the same batch-token average (Sec. 3).
3
Approach
The PPO clipped surrogate in Eq. (4) contains two hyper-parameters, ϵℓ and ϵh , that specify the clip range [1 − ϵℓ , 1 + ϵh ] around the per-token ratio ρt = π(yt | c<t )/πold (yt | c<t )5 . Their effect is most easily seen by considering the per-token contribution to Eq. (4) under the two signs of A. For tokens with A > 0, this contribution simplifies to π(yt | c<t ) − min , 1 + ϵh A. (9) πold (yt | c<t ) The objective is improved by increasing π(yt | c<t ), which raises ρt . Once ρt > 1 + ϵh , the contribution hits the ceiling −(1 + ϵh ) A, and the new policy gains no further credit by increasing the probability of the token. For tokens with A < 0, the contribution becomes π(yt | c<t ) − max , 1 − ϵℓ A, (10) πold (yt | c<t ) 4 More generally, π θ0 can be any reference policy that shares the token vocabulary V with πθ . When the vocabularies differ, the per-token KL is no longer well defined and the regularization must be applied at the sequence level. 5We drop the subscript θ and use the shorthand c <t from Sec. 2.
4
where the max replaces the min because A is negative. The objective is improved by decreasing π(yt | c<t ) and hence ρt . Once ρt < 1 − ϵℓ , the contribution hits −(1 − ϵℓ ) A, and the new policy gains no further credit by decreasing the probability of the token. In both cases, (ϵℓ , ϵh ) specify how far the new policy can move from πold while continuing to improve the surrogate; clipping acts as a regularizer that removes the incentive to change the policy dramatically from one update to the next. The effect of (ϵℓ , ϵh ) depends on the regime. When the batch is fresh and on-policy, ρt ≈ 1 for most tokens and the clip is rarely active. When the batch is stale or generated by a different policy, ρt can be far from one for many tokens, and (ϵℓ , ϵh ) then determines what fraction of the batch contributes to the update. If (ϵℓ , ϵh ) is too small, many tokens are clipped before they can contribute; if it is too large, high-variance importance-weighted gradients destabilize training. There is no single value appropriate across both regimes, and (ϵℓ , ϵh ) in practice has to be tuned for the task, model scale, and degree of off-policy mismatch. Prior work re-parameterizes the clip but does not remove it. Most previous works retain these two hyper-parameters and differ only in how they re-parameterize them. GRPO [26] uses Eq. (4) with a symmetric range ϵℓ = ϵh . DAPO [34] relaxes the symmetry, allowing ϵℓ ̸= ϵh to handle positiveand negative-advantage updates separately. GSPO [38] moves the clipped ratio from the token level seq to the sequence level via a geometric mean, but introduces sequence-level analogues ϵseq ℓ , ϵh that still must be chosen in advance. Across these objectives, (ϵℓ , ϵh ) survives in some form (see Table 1). Effective Sample Size to the Rescue. The value of (ϵℓ , ϵh ) is therefore consequential, and tuning it for large models is expensive. The effective sample size (ESS) of the policy ratios in the current batch [7] provides a way to determine this cap automatically, removing the need to fix (ϵℓ , ϵh ) in advance. Concretely, the ESS is given by 2 π(yt |c<t ) b EB πold (yt |c<t ) b B [ρt ]2 1 E ,1 , (11) ESS(B; θ) = 2 = b 2 ∈ |B| EB [ρt ] π(yt |c<t ) b EB πold (yt |c<t )
b B denotes where B is the set of valid response tokens in the current training batch, |B| is its size, and E the empirical average over B. We use eB = sg(ESS(B; θ)) to denote its detached value (treated as a constant for backpropagation), where sg(·) is the stop-gradient operator. The ESS is close to one when the batch is on-policy and falls when a few large ratios dominate, as happens with stale or mismatched data. An objective driven by eB therefore behaves like an on-policy update on fresh data and tightens automatically when the data drifts. Policy-on Policy-off Policy Optimization (P3O) for large-model. We adopt the P3O objective of [7], applied here to large-model post-training for the first time. P3O replaces the fixed clip in Eq. (4) with two terms whose strength is set by the batch ESS, X T LP3O (θ) = Ex, y∼πθold (·|x) − sg(min{ρt , eB }) log πθ (yt | c<t ) A t=1
+ (1 − eB ) KL πθ (· | c<t ) πθold (· | c<t ) ,
(12)
where ρt is the per-token policy ratio defined in the section opening, and the per-token KL, written in the same form as Eq. (8), vanishes when the batch is on-policy and grows as it drifts. P3O makes two structural changes to Eq. (4). First, it eliminates the fixed clip range (ϵℓ , ϵh ) entirely, replacing it with the data-driven cap eB that adapts to the current batch. Second, it adds a regularizer with coefficient (1 − eB ) that is large precisely when the batch is most off-policy and vanishes on fresh data. Crucially, while the clip in Eq. (4) discards the gradient on every token whose ratio falls outside (1 − ϵℓ , 1 + ϵh ), the cap min{ρt , eB } in Eq. (12) only scales the score-function gradient by a positive factor: every token in the batch contributes to the update and no data is wasted. Both adaptations are driven by a single statistic of the current batch, so Eq. (12) introduces no clipping range, no trust-region coefficient, and no staleness budget. While the form of Eq. (12) matches the original P3O, previous works for large-model RL have pursued a different path, proposing narrow variants of the GRPO clip with new hyper-parameters that succeed only in limited regimes. Adopting P3O for large-model post-training instead addresses the fragility at its source and yields a simpler and more effective solution. 5
Table 1: Comparison of policy objectives, with ρt = πθ (yt | x, y<t )/πb (yt | x, y<t ). Red/green: fixed hyper-parameters (high/low). Purple: auxiliary choices. Blue: batch-adaptive quantities. Auxiliary entropy and reference-policy KL terms are omitted. Method
Policy objective
GRPO / DAPO
− min (ρt A, clip (ρt , 1 − ϵℓ , 1 + ϵh ) A)
GSPO Decoupled
P − min (Sθ (y)A, clip (Sθ (y), 1 − ϵseq , 1 + ϵseq ) A) , Sθ (y) = exp T1 Tt=1 log ρt π πθ πθ −sg clip prox , 0, cw min πprox A, clip πprox , 1 − ϵℓ , 1 + ϵh A πb
P3O
−sg(min{ρt , eB }) log πθ (yt | x, y<t ) A + (1 − eB ) KL(πθ ∥ πb )
Remark (ESS adapts the clip without hyper-parameters). P3O sets both the cap on the scorefunction weight and the regularizer coefficient from a single batch statistic, the ESS, and updates them at every gradient step from the data the optimizer is currently seeing. This one adaptive mechanism replaces the clip range (ϵℓ , ϵh ) and the auxiliary trust-region or staleness parameters that fixed-clip methods rely on, and introduces no new hyper-parameter in their place. As the off-policy degree of the batch changes during training, the cap and the regularizer track it without retuning. Off-policy data in large-model training. At large-model scale, the data the optimizer sees is rarely strictly on-policy. Training and rollout engines differ in numerical precision, sampling, and other implementation details (Sec. 2), and reusing rollouts across optimizer epochs widens the gap further. The standard fix is a decoupled-loss objective [13] that clips against a proximal snapshot πprox rather than against πθold . This stabilizes training but adds new hyper-parameters, including the construction of πprox , its lifetime, and a cap cw on an outer behavior weight, all of which must be tuned for each off-policy regime. P3O sidesteps this. The same eB that drives the cap on fresh data falls when the batch becomes off-policy, so Eq. (12) handles both regimes in a single objective without any additional hyper-parameter or special code path. Our experiments (Sec. 4) confirm that this single objective performs well across off-policy regimes that previously required specialized losses. This makes off-policy data a feature rather than a burden, since P3O can reuse rollouts from older or different policies and demonstrations to improve sample efficiency and reduce training time without a specialized loss (see Table 1 for the parameter contrast across methods). 3.1
A potential issue with P3O
One possible issue with P3O is that the single ESS eB in Eq. (12) conflates two distinct drifts in the batch: the difference between πθ and the data-generating policy πb , and the within-epoch difference between πθ and its pre-update snapshot πprox . When both drifts are large but in different directions, a single anchor cannot distinguish them. A natural extension introduces a second anchor at πprox and pulls πθ toward an ESS-weighted mixture πmix of the two anchors: Lext (θ) = −Eπb sg(min{rb , emix }) log πθ (yt | c<t ) A + (1 − emix ) KL πθ πmix , (13) where rb = πθ (yt | c<t )/πb (yt | c<t ) is the per-token behavior ratio, emix is a joint ESS computed from both anchors, and πmix is a per-token mixture of πb and πprox weighted by their respective (1 − eb ) and (1 − eprox ) so that the more on-policy anchor drops out (full formulation in Sec. B). This variant introduces no new hyper-parameter and reduces to P3O when either anchor is uninformative. Empirically, however, the single-anchor P3O matches or slightly outperforms this variant across the off-policy regimes considered in Sec. 4, suggesting that the behavior-axis ESS already captures most of the relevant drift signal.
4
Experiments
We evaluate P3O across three axes: (i) sensitivity to the clipping hyper-parameters (ϵℓ , ϵh ) that P3O eliminates entirely; (ii) robustness to off-policy data arising from practical mismatches in numerical precision and sampling temperature; and (iii) downstream benchmark performance on held-out mathematical reasoning tasks. In all settings we compare against GRPO, which shares the 6
Average Reward
0.6
0.3
0.4
0.25
0.2
0.2 Ours GRPO
0
0.15 10
20
30
10
Training Steps
20
30
40
50
Training Steps
(a) Qwen3-4B-Thinking-2507
(b) Qwen2.5-1.5B
Figure 1: Sensitivity of GRPO’s reward to the clip range ϵ ∈ {0.2, 0.4, 0.6} (shaded region: ±1 std over clip values) versus P3O run once with no clip hyperparameter. P3O’s ESS-driven cap (Eq. (11)) removes this tuning burden while matching or exceeding the best GRPO variant across both model families.
same base objective (Sec. 2) but relies on a fixed clip range; both methods otherwise use identical hyper-parameters within each experiment family (Table 3). 4.1
Experiment Setup
Experiments use the open-source models Qwen3-4B-Thinking-2507 [30] and Qwen2.5-1.5B [29], trained on the DeepScaleR-Preview dataset of 40,000 mathematics problem-answer pairs compiled from AIME (1984-2023), AMC (prior to 2023), Omni-MATH, and Still [18]. Binary rewards are assigned by matching the model’s output against the DeepScaleR-Preview reference; to isolate algorithmic differences between P3O and GRPO, we use no reward shaping or auxiliary bonuses. All runs use 8 NVIDIA H100 GPUs in a distributed stack that separates optimizer workers from rollout engines and synchronizes policy weights under a shared scheduler. Training, evaluation, hardware, and benchmark details are in Tables 3 to 6 and Sec. C and E. 4.2
Effects of Hyperparameters
Clipping Factor. As argued in Sec. 3, the fixed clip range (ϵℓ , ϵh ) is a pre-committed choice that cannot adapt to the batch, and a value suited to on-policy data may over- or under-clip when rollouts become stale or mismatched (Eqs. (9) and (10)). We verify this sensitivity by sweeping the symmetric clip ϵℓ = ϵh = ϵ ∈ {0.2, 0.4, 0.6} for GRPO and contrasting with a single P3O run. As shown in Fig. 1, GRPO’s reward trajectory varies substantially with ϵ (shaded region), while P3O remains stable by adjusting the score-function cap and regularizer from the batch ESS (Eq. (11)). Other clipbased baselines such as DAPO [34] and GSPO [38] retain a fixed clip range and exhibit equivalent hyper-parameter sensitivity under this ablation, so the key variable we isolate is the presence or absence of a fixed clip. 4.3
Off-Policy Data
Off-policy mismatch arises in large-model RL post-training whenever rollouts are not drawn from the current policy, including from optimizer steps reused across iterations, mixed-precision inference engines, and non-standard sampling strategies (Sec. 1). The two experiments below isolate two of these sources and test whether P3O’s batch ESS (Eq. (11)) handles each organically. Temperature of Rollouts. Sampling rollouts at temperature T ̸= 1.0 uniformly rescales token log-probabilities, shifting the per-token ratio ρt (Eq. (7)) by a constant factor across the entire batch. For GRPO, this offset falls either inside or outside the fixed clip interval independent of how onpolicy the batch otherwise is, a bias that cannot be corrected without retuning ϵ. The ESS (Eq. (11)) directly measures this shift as ratio concentration and tightens the score-function cap and regularizer accordingly. 7
0.8 Average Reward
Average Reward
0.8 0.6 0.4 0.2
0.6 0.4 0.2
Ours GRPO
0
0 5
10
15
20
10
25
20
30
(b) Qwen3-4B-Thinking-2507 (Temperature 1.2)
(a) Qwen3-4B-Thinking-2507 (Temperature 0.6)
Figure 2: P3O is robust to off-policy data introduced through the varied sampling temperature of rollouts. Sampling rollouts at a temperature other than 1.0 introduces a distribution shift in the token-level log probabilities, creating off-policy data for Qwen3-4B-Thinking-2507. Corresponding Qwen2.5-1.5B results are deferred to Fig. 7.
Average Reward
0.6
0.3 0.25
0.4
0.2 0.2
0.15
0
Ours GRPO
0.1 10
20
30
10
Training Steps
20
30
40
Training Steps
(a) Qwen3-4B-Thinking-2507
(b) Qwen2.5-1.5B
Figure 3: P3O is robust to off-policy data introduced through the BF16 Train + FP8 Rollout training scheme. As accuracy collapse is observed in longer rollout lengths [33], a rollout length of 16,384 tokens was used in this experiment. The demonstrated robustness of P3O to off-policy data allows for the use of faster rollout generation strategies. In contrast, GRPO’s performance degrades significantly under the same conditions, highlighting its sensitivity to off-policy data. As shown in Fig. 2, GRPO’s performance degrades at both T = 0.6 and T = 1.2 relative to the on-policy baseline on Qwen3-4B, while P3O matches or exceeds standard-temperature performance without retuning. The same qualitative trend appears for Qwen2.5-1.5B in Fig. 7, confirming that the batch ESS adapts to the induced ratio shift across both model families. BF16 Train + FP8 Rollout. The practical off-policy mismatch described in Sec. 1 is directly instantiated by mixed-precision pipelines: rollouts generated by an FP8-quantized policy carry different token-level log-probabilities than the BF16 training model, shifting the per-token ratio ρt (Eq. (7)) away from one. We test whether P3O’s adaptive regularizer (Eq. (12)) handles this mismatch without any change to the training configuration. Off-policy data is created when rollouts are generated by a model quantized to a different numerical precision than the training model. Rollouts generated in lower precision can be much faster to generate, leading to a faster training run overall. A common training strategy is to use BF16 precision for training and FP8 while creating rollouts, but this strategy can lead to collapse in reward performance, particularly for large max_tokens, because the rollout policy and training policy no longer induce the same token-level probabilities [33]. In practice, dynamic quantization methods are used where the latest trained policy is moved to the rollout engine with BF16 precision, then dynamically quantized to FP8 for rollout generation [31]. As shown in Fig. 3, P3O remains stable when FP8 quantization pushes importance ratios away from one, maintaining training quality without any change to the training configuration, whereas GRPO collapses later in training under the same mismatch. 8
Baseline
Ours
GRPO
iter 15
iter 30
0.4 0.4 Pass@k
Pass@k
0.3 0.2
0.2
0.1 0
0 1
4
8
12
16
1
k
4
8
12
16
k
(a) Clip variants (4K-token eval)
(b) FP8 variants (16K-token eval)
Figure 4: Pass@k averaged over all five held-out benchmarks (AIME24/25/26, AMO-Bench, AMC). Left: clip-ratio variants at 4K-token evaluation; GRPO is averaged over ϵ ∈ {0.2, 0.4, 0.6}. Ours matches or exceeds the averaged GRPO sweep without requiring a clip-ratio choice. Right: BF16-train + FP8-rollout variants at 16K-token evaluation. GRPO collapses by iter 30 (near-zero pass@k) while Ours retains strong performance. 4.4
Benchmark Results
To confirm that the reward-curve advantages of P3O translate to held-out task performance, we evaluate checkpoints of Qwen3-4B-Thinking-2507 trained with each method on five mathematical reasoning benchmarks (Table 7). These include AIME24 [35], AIME25 [36], AIME26 [37], AMOBench [1], and AMC [20], all of which are standard held-out benchmarks for mathematical reasoning. Notably, the DeepScaleR-Preview training set contains no samples from these benchmarks, so the table measures generalization. Fig. 4 shows that the training-time stability differences seen in the reward curves also matter at evaluation time. In the clip-sensitivity study (Fig. 4a), P3O is competitive with or better than the averaged GRPO sweep at every k while avoiding the clip-selection burden entirely. In the FP8 rollout setting (Fig. 4b), P3O retains benchmark performance much later into training, while GRPO degrades sharply—by iter 30 its pass@k is near zero across all benchmarks. Full per-benchmark results are reported in Table 7 in the appendix.
5
Discussion
RL is structurally and algorithmically fragile, and large-model post-training sharpens that fragility because rollouts are expensive, hyper-parameters are consequential, and tuning costs compound with model scale. Recent approaches respond by adding or re-parameterizing fixed choices (asymmetric clip ranges, staleness budgets, etc.) tuned per task and model, and each addition makes the algorithm more sensitive to its configuration rather than less. We take the opposite path. The amount of trust placed in an update should be adaptive rather than pre-committed before training begins. Using the normalized effective sample size of the current policy ratios, P3O replaces fixed clipping with a batch-adaptive score-function cap and matching regularizer, behaving like an on-policy update on fresh data and tightening automatically on stale or mismatched data. This does not make arbitrary off-policy data reliable, since meaningful token-level ratios and adequate support in the behavior data are still required, but it turns off-policy mismatch into a measured batch property and removes the need for clip ranges, behavior-weight caps, and staleness budgets. The same mechanism also makes off-policy data directly usable inside one objective, whereas prior methods reuse it only through specialized losses or per-regime retuning. Across the regimes we test, including clip sweeps, temperature shifts, and BF16/FP8 mixed precision, P3O matches or exceeds tuned GRPO baselines with no objective hyper-parameters to set. We view this paper as a starting point for batch-adaptive methods that improve large-model post-training by reducing, rather than expanding, the surface of pre-committed knobs.
9
References [1] Shengnan An, Xunliang Cai, Xuezhi Cao, Xiaoyu Li, Yehao Lin, Junlin Liu, Xinxuan Lv, Dan Ma, Xuanlin Wang, Ziwen Wang, and Shuang Zhou. Amo-bench: Large language models still struggle in high school math competitions, 2025. 9 [2] Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? A large-scale study. In International Conference on Learning Representations (ICLR), 2021. 1 [3] Zhepeng Cen, Yao Liu, Siliang Zeng, Pratik Chaudhari, Huzefa Rangwala, George Karypis, and Rasool Fakoor. Bridging the training-inference gap in LLMs by leveraging self-generated tokens. Transactions on Machine Learning Research, 2025. 4 [4] DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Nature, 645:633–638, 2025. 1 [5] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on PPO and TRPO. In International Conference on Learning Representations (ICLR), 2020. 1 [6] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407–1416, 2018. 2 [7] Rasool Fakoor, Pratik Chaudhari, and Alexander J. Smola. P3O: policy-on policy-off policy optimization. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, page 371, 2019. 2, 5 [8] Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. Ddpg++: Striving for simplicity in continuous-control off-policy reinforcement learning. arXiv:2006.15199, 2020. 1 [9] Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-learning. In ICLR, 2020. 2, 4 [10] Rasool Fakoor, Jonas Mueller, Zachary C. Lipton, Pratik Chaudhari, and Alexander J. Smola. Time-varying propensity score to bridge the gap between the past and present. In ICLR, 2024. 2, 4 [11] Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. Advances in Neural Information Processing Systems, 34:11260–11273, 2021. 2 [12] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence, 2018. 1 [13] Jacob Hilton, Karl Cobbe, and John Schulman. Batch size-invariance for policy optimization. In Advances in Neural Information Processing Systems, volume 35, pages 17086–17098, 2022. 2, 6 [14] Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022. 1 [15] Augustine Kong. A note on importance sampling using standardized weights. Technical Report 348, 1992. 2 [16] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv:2005.01643, May 2020. 2 10
[17] Yao Liu, Pratik Chaudhari, and Rasool Fakoor. Budgeting counterfactual for offline rl. In Advances in Neural Information Processing Systems, volume 36, pages 5729–5751, 2023. 2 [18] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog. 7 [19] Wenhan Ma, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Zhifang Sui, and Fuli Luo. Stabilizing MoE reinforcement learning by aligning training and inference routers. arXiv preprint arXiv:2510.11370, 2025. 2 [20] Mathematical Association of America. 2023 american mathematics competitions (amc 10 and amc 12). https://huggingface.co/datasets/math-ai/amc23, 2023. Dataset curated by the Math-AI community. 9 [21] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. 1, 4 [22] Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Defeating the training-inference mismatch via FP16. arXiv preprint arXiv:2510.26788, 2025. 2 [23] Sidney I Resnick. A probability path. Springer Science & Business Media, 2013. 4 [24] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), pages 1889–1897, 2015. 2, 3 [25] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 2, 3 [26] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 2, 3, 5 [27] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc. 1, 4 [28] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018. 3 [29] Qwen Team. Qwen2.5: A party of foundation models, September 2024. 7 [30] Qwen Team. Qwen3 technical report, 2025. 7, 16 [31] vLLM Team. Fp8 quantization — vllm documentation. https://docs.vllm.ai/en/v0.5. 0.post1/quantization/fp8.html, 2024. Accessed: 2026-05-06. 8 [32] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229–256, 1992. 3 [33] Haocheng Xi, Charlie Ruan, Peiyuan Liao, Yujun Lin, Han Cai, Yilong Zhao, Shuo Yang, Kurt Keutzer, Song Han, and Ligeng Zhu. Jet-rl: Enabling on-policy fp8 reinforcement learning with unified training and rollout precision flow, 2026. 8 [34] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, 11
Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. 2, 5, 7 [35] Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2024, 2024. 9 [36] Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2025, 2025. 9 [37] Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2026, 2026. 9 [38] Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. 5, 7 [39] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 1, 4
12
Algorithm 1 GRPO Require: Policy πθ ; group size G; clip (ϵℓ , ϵh ); entropy coefficient βent Rollout: for each prompt xp , sample G completions from πθ , record log πb = log πθ , and compute P 1 rp,j − G k rp,k Ap,j = q P 2 1 k (rp,k − r̄p ) + ϵ G Update: for each micro-batch B with mask M :
1: ρt ← exp log πθ (yt | c<t ) − log πb (yt | c<t )
2: L ← − min ρt At , clip(ρt , 1−ϵℓ , 1+ϵh ) At 3: Backward pass; optimizer step
M
− βent ⟨Hθ ⟩M
Algorithm 2 P3O Require: Policy πθ ; group size G; entropy coefficient βent
(no clip range)
Rollout: identical to Algorithm 1 Update: for each micro-batch B with mask M : 1: ρt ← exp log πθ (yt | c<t ) − log πb (yt | c<t ) 2 P P 2: eB ← |M | · M ρ2t (ESS, all-reduced across workers) M ρt 3: L ← −⟨sg(min{ρt , eB }) log πθ At ⟩M + (1−eB )⟨KL(πθ ∥πb )⟩M − βent ⟨Hθ ⟩M 4: Backward pass; optimizer step
A
Algorithm Pseudocode
We present pseudocode for GRPO and P3O as implemented in our FeynRL framework. Both algorithms share the same rollout phase and use group-relative advantages (Eq. (3)); they differ only in how the policy update is computed. GRPO uses a fixed clip range (ϵℓ , ϵh ) to bound the per-token policy ratio, while P3O replaces this fixed clip with a batch-adaptive ESS cap and an adaptive KL regularizer, introducing no new hyper-parameters. Throughout both algorithms, ⟨·⟩M denotes the mean over the valid (non-padded) tokens indicated by mask M , sg(·) is the stop-gradient operator, and c<t = (x, y<t ) is the conditioning context at position t.
B
Two-anchor extension of P3O: full formulation
The extension discussed in Sec. 3.1 replaces the single-anchor regularizer of Eq. (12) with a KL toward an ESS-weighted mixture of the behavior policy πb and a proximal snapshot πprox of πθ taken at the start of each optimizer epoch. The full loss is Lext (θ) = −Eπb sg(min{rb , emix }) log πθ (yt | c<t ) A + (1 − emix ) KL πθ πmix , (14) with the following per-batch quantities: πθ (yt | c<t ) πθ (yt | c<t ) , rprox = , πb (yt | c<t ) πprox (yt | c<t ) 2 2 P P B rb B rprox P P 2 , eb = , eprox = |B| B rb2 |B| B rprox rb =
emix = min(eb , eprox ), (1 − eb ) πb (· | c<t ) + (1 − eprox ) πprox (· | c<t ) πmix (· | c<t ) = . (1 − eb ) + (1 − eprox )
(15)
Each anchor is weighted by its (1−ESS), so a fully on-policy axis (whose ESS approaches one) drops out of the mixture and the regularizer pulls only toward the mismatched anchor. The construction reduces to the single-anchor P3O regularizer when either anchor is uninformative: if eb → 1 the 13
behavior weight vanishes and πmix → πprox ; if eprox → 1 the proximal weight vanishes and πmix → πb , recovering Eq. (12). The construction adds no hyper-parameter beyond what P3O already uses.
C
Additional experimental details
This appendix summarizes the settings for the main runs and a small set of supplementary off-policy ablations. Hyper-parameter Value Optimizer AdamW Learning rate 1e-5 Betas (β1 , β2 ) (0.9, 0.95) Weight decay 0.01 Gradient clipping 1.0 LR scheduler WarmupCosineLR (10% warmup ratio)
Table 2: Shared optimizer settings used across all experiments.
Hyper-parameter Clip Runs Temperature Runs FP8 Runs Model Qwen3-4B-Thinking-2507 Qwen3-4B-Thinking-2507 Qwen3-8B-Base Training / rollout GPU split 6 train + 2 rollout 6 train + 2 rollout 5 train + 3 rollout Global train batch size 48 48 5 Train micro-batch / GPU 8 4 1 Gradient accumulation 1 2 1 Rollout batch size / GPU 64 64 16 Rollout samples / epoch 512 512 256 Max tokens 1024 4096 16384 Sampling temperature 1.0 T ∈ {0.6, 1.2} 1.0 GRPO clip setting ϵ ∈ {0.2, 0.4, 0.6} ϵ = 0.4 ϵ = 0.4
Table 3: Per-experiment training configuration. Clip and temperature runs use the 4B model with a 6/2 GPU split; the FP8 ablation uses the 8B model with a 5/3 split and a 16K-token rollout budget.
Hyper-parameter
Value Shared benchmark-evaluation settings Temperature 1.0 Top-p 0.95 Group Size (nsamples ) 16 Rollout Batch Size / GPU 16 Rollout GPUs 8 Data Workers 8 4K benchmark eval family (clip + temperature) Benchmarks AIME24, AIME25, AIME26, AMO-Bench, AMC Max Response Tokens 4096 16K benchmark eval family (baseline + FP8) Benchmarks AIME24, AIME25, AIME26, AMO-Bench, AMC Max Response Tokens 16384 Model & system configuration Precision bfloat16 Tensor Parallel Size 1
Table 4: Benchmark-evaluation settings grouped by rollout-length family. Separate 4K-token and 16K-token evaluation regimes are used in the paper, but the sampling policy and GPU allocation are otherwise held fixed across benchmarks and checkpoints. The main distinction across training families is the rollout budget and GPU partition. Table 3 summarizes all three experiment families: clip and temperature runs use the 4B model with a 6/2 14
Hardware Attribute
Value Accelerator Configuration GPU Model NVIDIA H100 (as reported in Sec. 4) Total GPUs per run 8 Evaluation partition 8 rollout GPUs Tensor Parallel Size 1 Runtime Configuration Training Precision bfloat16 Rollout Precision bfloat16; FP8 only in the mixed-precision ablation Distributed Training DeepSpeed ZeRO-3 Attention Backend Flash Attention 2 Workload Summary Training Dataset DeepScaleR-Preview Training Models Qwen3-4B-Thinking-2507 and Qwen3-8B-Base Evaluation Rollout Batch Size / GPU 16
Table 5: Compute layout for the reported runs, together with the accelerator SKU reported in the main text. The eight GPUs are partitioned differently across ablation families depending on whether the workload is optimizer-heavy or rollout-heavy.
Environment Attribute Python version NVIDIA driver CUDA toolkit PyTorch build PyTorch / TorchVision DeepSpeed vLLM Transformers Ray FlashAttention
Value Base environment 3.13.1 (recommended/tested) CUDA 12.x-compatible; driver version ≥ 525.85 recommended 12.2 (tested); toolkit versions ≥ 12.2 supported CUDA 12.6 wheels Core training and rollout stack CUDA-enabled install 0.18.9 0.19.0 4.57.6 2.54.1 2.8.3 (built from source)
Table 6: Software environment used for the reported experiments. The table records only the main CUDA and library versions needed to contextualize the reported pipeline.
train-rollout split, while the FP8 ablation uses the 8B model with a 5/3 split and a 16K-token rollout budget. Benchmark evaluation is likewise split into separate 4K-token and 16K-token regimes, which is why Table 7 mixes both context budgets. Under asynchronous optimization, both the one-step and two-step pipeline variants preserve the same qualitative ordering: P3O remains more stable than GRPO as rollout staleness increases. We show the one-step setting in Fig. 5 because it is visually cleaner; the omitted two-step variant follows the same trend.
D
Two-anchor extension training stability
Fig. 8 reports training curves for the two-anchor extension of P3O described in Sec. B—alongside P3O and GRPO baselines. All three algorithms are trained on Qwen3-4B-Thinking using the DeepSeek dataset with temperature 1.2 and a rollout length of 4,096 tokens. The two-anchor extension matches the peak reward of P3O and GRPO (≈0.67) during the first 16 steps but then collapses abruptly, falling to near-zero reward before the run terminates at step 24. P3O remains stable throughout the full training run, and GRPO degrades more gradually in the later steps. The collapse suggests that the dual-anchor regularizer amplifies gradient variance once the proximal snapshot drifts far from the behavior policy, destabilizing optimization under the current 15
0.8
Average Reward
0.6
0.4
Ours GRPO
0.2
0 5
10
15
20
25
30
35
40
45
50
Training Steps
Figure 5: Asynchronous-training comparison between P3O and GRPO under one optimizer step per rollout epoch. Rollouts are generated by a stale policy while the learner continues updating, creating the off-policy lag discussed in the main text. In this one-step pipeline setting, P3O maintains a higher and more stable reward trajectory than GRPO across training. The corresponding twostep pipeline produces the same qualitative ordering and is omitted to avoid duplicating the same comparison with only a modest increase in late-training noise.
0.2
Average Reward
0.15
0.1
Ours GRPO
5 · 10−2
0 2
4
6
8
10
12
Training Steps
Figure 6: Comparison of GRPO and P3O with respect to the mixing of off-policy data. A rollout length of 4,096 tokens was used in this experiment. This experiment uses Qwen3-8B [30] to generate rollouts for training Qwen3-4B-Thinking-2507, as it is from a different model family. Data was mixed at a 50% ratio, meaning half of the rollouts were generated by the training model and half were generated by the separate policy.
16
0.3
0.3
0.2
0.2
0.1
0.1 Ours GRPO
0
0 5
10
15
20
25
10
Training Steps
20
30
Training Steps
(a) Qwen2.5-1.5B (Temperature 0.6)
(b) Qwen2.5-1.5B (Temperature 1.2)
Figure 7: Temperature-robustness results for Qwen2.5-1.5B, corresponding to Fig. 2(c,d). As in the main-text Qwen3-4B experiments, changing rollout temperature introduces a token-level distribution shift that degrades GRPO while P3O remains comparatively stable. 0.8
Average Reward
0.6
0.4
Two-anchor extension P3O (Ours) GRPO
0.2
0 5
10
15
20
25
30
35
Training Steps
Figure 8: Training curves for the two-anchor extension of P3O, P3O, and GRPO on Qwen3-4BThinking. All runs use the DeepSeek dataset with temperature 1.2 and a rollout length of 4,096 tokens. The two-anchor extension peaks at a reward comparable to P3O and GRPO but undergoes a sharp collapse after step 16, dropping to near-zero reward by step 24 before the run terminates. P3O maintains stable, high reward throughout training, while GRPO shows partial instability in later steps. These results indicate that the two-anchor regularizer, though theoretically motivated, introduces training instability under the current hyperparameter regime. hyperparameter setting. Addressing this instability—through tighter proximal resets, adaptive KL weighting, or learning-rate schedules—is left as future work. Fig. 9 presents a complementary experiment at default rollout temperature, where the same three algorithm families are compared but GRPO is run with a larger clip ratio (ϵ=0.4).
E
Benchmark results
Table 7 reports pass@1 for each checkpoint on all five held-out benchmarks.
17
0.6
Average Reward
0.5 0.4 0.3 0.2
Two-anchor extension P3O (Ours) GRPO (ϵ=0.4)
0.1 0 5
10
15
20
25
30
35
Training Steps
Figure 9: Training curves for the two-anchor extension of P3O, P3O, and GRPO (ϵ=0.4) on Qwen3-4B-Thinking at default temperature. All runs use the DeepSeek dataset with a rollout length of 4,096 tokens. Unlike the temperature-1.2 regime (Fig. 8), the two-anchor extension remains stable throughout training and achieves the highest final reward (≈0.50), slightly outpacing both P3O and GRPO with ϵ=0.4. The standard GRPO baseline (default ϵ) reached only ≈0.19 reward in this setting and is excluded from the plot for clarity. Together with Fig. 8, these results suggest that the two-anchor extension’s instability is sensitive to temperature: at higher rollout temperatures the dual-anchor regularizer can destabilize training, whereas at the default temperature it performs comparably to or better than P3O.
Training Method Baseline Model (4K tokens) Baseline Model (16K tokens)
AIME24
AIME25
AIME26
AMO-Bench
AMC
0.029 0.371
0.033 0.471
0.006 0.396
0.007 0.019
0.217 0.618
0.126±0.087 0.160
0.012±0.007 0.010
0.381±0.195 0.493
0.160 0.002 0.179 0.175
0.021 0.019 0.024 0.026
0.499 0.029 0.478 0.529
Clip Variants (ϵ ∈ {0.2, 0.4, 0.6}, 4K tokens) GRPO (clip avg) 0.176±0.039 0.160±0.123 P3O 0.165 0.183 FP8 Variants (BF16 train + FP8 rollout, 16K tokens) FP8 Rollout GRPO Iter 15 0.154 0.250 FP8 Rollout GRPO Iter 30 0.002 0.000 FP8 Rollout P3O Iter 15 0.158 0.237 FP8 Rollout P3O Iter 30 0.173 0.254
Table 7: Benchmark results (pass@1) across trained checkpoints. Baseline rows report untrained Qwen3-4B-Thinking-2507 at two rollout lengths. Clip-variant GRPO shows mean ± std across ϵ ∈ {0.2, 0.4, 0.6}. FP8 rows report individual checkpoints. Bold: best within each group per column.
18