1
arXiv:2605.04960v1 [cs.LG] 6 May 2026
EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance Song Yu1 , Li Li1* , Wenwen Zhao1 , Zhisheng Yang1 1 School of Computer and Information Science, Southwest University, Chongqing 400715, China [email protected] [email protected] (Corresponding author.) [email protected] [email protected]
advantage contribution across the entire sequence [28, 6]. This coarse granularity ignores the inherent heterogeneity of reasoning processes. At logical branch points, token distributions exhibit high entropy that reflects active exploration, while at deterministic derivation steps, low entropy tokens often encode stylized redundancy rather than genuine reasoning progress [34]. Recent efforts such as Process Reward Models (PRMs) [17, 33] provide steplevel supervision but require costly human annotations and generalize poorly across domains. Entropy-based signals [11] and length-aware penalties [29] operate at the trajectory or position level, failing to differentiate individual tokens by their actual information content. Second, GRPO assigns the same sign of advantage to all tokens within a sequence. Since the sequencelevel reward determines whether advantage is positive or negative, every token in a successful response receives positive reinforcement, while every token in a failed response receives negative reinforcement. This uniform polarity ignores a fundamental reality: a reasoning chain Index Terms—Reinforcement Learning, Large Language invariably contains a mixture of correct and incorrect steps. Under GRPO, every one of these correct steps Model, Policy Optimization, Credit Assignment. is indiscriminately penalized alongside genuine errors, suppressing valid reasoning patterns. Recent works such I. I NTRODUCTION as PRPO [5] and Step-GRPO [16] employ a pretrained ITH the advancement of large language models PRM to score individual steps and aligns process reward (LLMs), reinforcement learning with verifiable distributions with sequence-level outcome advantages. rewards (RLVR) has become the cornerstone for enHowever, these approach still depends on an external hancing complex reasoning. The release of DeepSeek-R1 reward model. [6] marks the significant effectiveness of RLVR [13] in Third, GRPO’s advantage normalization collapses enhancing model reasoning abilities, powered by Group when all sampled responses within a group receive Relative Policy Optimization (GRPO) [28]. By employing identical rewards. In this case, the normalized advantage intra-group relative advantages, GRPO improves training becomes identically zero for all tokens, resulting in a efficiency without a critic network. Despite its success, GRPO still has limitations, as complete loss of the outcome-driven gradient signal. shown in Figure 1. First, it assumes uniform token-level While increasing the group size mitigates this issue, it proportionally escalates computational cost, making * Corresponding author. it impractical for resource-constrained settings. Recent
Abstract—Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has advanced LLM reasoning. However, GRPO suffers from three credit assignment failures: uniform tokenlevel granularity that ignores heterogeneous informational value, uniform polarity that penalizes correct steps and rewards incorrect ones, and zero-variance collapse that erases outcome-driven gradients. We systematically quantify these failures, revealing highly non-uniform token informativeness, widespread step-level polarity misalignment, and substantial training waste. To address these limitations, we propose Entropy-Progress Aligned GRPO (EP-GRPO), a framework that mines the model’s intrinsic information flow for dense, self-supervised guidance. EP-GRPO integrates entropygated modulation to prioritize high entropy decision pivots, implicit process signals from policy divergence anchored to outcome advantages for directional token-level feedback without external reward models, and cumulative entropy mapping that enables progress-aligned advantage normalization, naturally maintaining gradient flow under zero reward variance. Extensive experiments on mathematical reasoning benchmarks demonstrate that EP-GRPO achieves superior accuracy and efficiency compared to GRPO and its variants. The code will be available.
W
2
Limitation 1: Uniform Granularity – Token Information is Heterogeneous Generated Reasoning (Example)
Input (Problem)
Question: Positive real numbers 𝑥 and 𝑦 satisfy 𝑦 ! = 𝑥 " and (𝑦 − 𝑥)" = 4𝑦 " . What is 𝑥 + 𝑦?
LLM Generation High
To
find
the
value
of
(
x
+
y
)
.
We
will
solve
the
given
equations
:
1
.
y
^
3
=
x
3
…
1.39
0.73
1.11
0.49
0.00
0.72
0.63
0.03
0.00
0.00
0.67
0.54
1.66
1.73
0.01
0.20
0.54
0.28
0.24
0.02
0.00
0.00
0.00
0.00
0.00
0.00
…
Token Entropy (H)
Low
Logical Branch / Decision Pivot (High Entropy)
Deterministic Derivation (Low Entropy)
Limitation 2: Uniform Polarity – Mixed Correct and Incorrect Steps Share the Same Advantage Sign Case A: Final Answer is Wrong -> All Tokens Get Negative Advantage
Case B: Final Answer is Correct -> All Tokens Get Positive Advantage
Generated Reasoning (Example)
Input (Problem)
Question: Positive real numbers 𝑥 and 𝑦 satisfy 𝑦 ! = 𝑥 " and (𝑦 − 𝑥)" = 4𝑦 " . What is 𝑥 + 𝑦?
Generated Reasoning (Example)
Token Advantage (All Negative)
1. 𝑦 − 𝑥 = 2𝑦. ✅ 2. 𝑦 − 𝑥 = −2𝑦 ⟹ 𝑥 = 3𝑦. ✅ LLM Generation (Wrong Answer)
3. 𝑦 ! = (3𝑦)" ⟹ 𝑦 ! = 9𝑦 " ⟹ 𝑦 = 9. ✅ 4. 𝑥 = 27, 𝑥 + 𝑦 = 12.
❌
Final Answer: 12 ❌
Input (Problem)
Question: Positive real numbers 𝑥 and 𝑦 satisfy 𝑦 ! = 𝑥 " and (𝑦 − 𝑥)" = 4𝑦 " . What is 𝑥 + 𝑦?
1. (𝑦 − 𝑥)" = 4𝑦 " ⟹ 𝑦 − 𝑥 = 2𝑦. ❌ 2. −𝑥 = 𝑦 impossible, must use 𝑦 − 𝑥 = −2𝑦. ✅ LLM Generation (Correct Answer)
Token Advantage (All Positive)
3. 𝑥 = 3𝑦. ✅ 4. 𝑦 = 9, 𝑥 = 27, 𝑥 + 𝑦 = 36. ✅ Final Answer: 36 ✅
Limitation 3: Zero-Variance Collapse – Gradient Vanishes When All Rewards Are Identical Group Sampling (G responses) Input (Problem)
… Final Answer: 12
❌
0
Response 2
… Final Answer: 0
❌
0
Group Statistics Mean Reward: 0 …
…
…
…
Question: Positive real numbers 𝑥 and 𝑦 satisfy 𝑦 ! = 𝑥 " and (𝑦 − 𝑥)" = 4𝑦 " . What is 𝑥 + 𝑦?
Reward (Sequence-level)
Response 1
Variance: 0 (Zero Variance)
Response G
… Final Answer: -15
❌
GRPO Advantage
All tokens' advantages are zero
𝑹𝒊 − 𝝁 =𝟎 𝝈 (for all response)
→ Gradient signal vanishes
𝑨𝒊 =
→ No learning occurs
0
Fig. 1: Conceptual illustration of the fundamental limitations in standard GRPO. The top panel demonstrates Uniform Granularity, where the model fails to distinguish between critical high entropy decision pivots and deterministic low entropy derivations. The middle panel shows Uniform Polarity, where sequence-level rewards lead to the indiscriminate reinforcement or penalization of both correct and incorrect intermediate steps. The bottom panel illustrates Zero-Variance Collapse, where identical rewards within a group cause the learning signal to vanish.
efforts have addressed this from complementary angles: data-level methods such as DAPO [35] detect and filter zero-variance groups, while signal-level approaches such as RL-ZVP [14] and Dr. GRPO [21] reshape advantages via entropy shaping or learned bias terms. Orthogonal work including RE-GRPO [19] employs LLM-guided reflection to repair failed trajectories, at the cost of additional inference overhead. These methods either rely on extra sampling, auxiliary model calls, or heuristic advantage reshaping that does not explicitly align with logical reasoning progress. To solve these problems, we propose Entropy-Progress Aligned GRPO (EP-GRPO), a progress-aware optimization algorithm that transforms traditionally sparse, outcome-only feedback into dense, token-level learning signals. EP-GRPO addresses three orthogonal limitations of GRPO through complementary mechanisms. First, to resolve the granularity issue, it applies entropy-gated modulation to the original sequence-level advantage. Second, to resolve the polarity issue, it leverages policy divergence as a token-level signal and anchors its polarity to the outcome advantage, yielding directional process rewards without external models. Third, to resolve the zero-variance issue, when group rewards are identical and the outcome advantage vanishes, the anchoring signal degrades to the raw reward polarity. Since the implicit signal depends on policy divergence rather than reward
variance, it remains fully active and sustains gradient flow even when outcome comparisons fail. The implicit advantages are then normalized within cumulative entropy buckets to enable progress-aligned, dense rewards, and combined to form the final token-level learning signal. Our main contributions are as follows:
We introduce an entropy-gated modulation mechanism that transforms the coarse-grained sequencelevel advantage into token-level weights, prioritizing gradient updates at critical decision pivots. • We propose a method to convert policy divergence into directional process rewards by anchoring its polarity with the outcome advantage, providing finegrained token-level feedback without external reward models. • We design a logical-progress alignment mechanism based on cumulative entropy mapping, which projects asynchronous reasoning trajectories into a unified coordinate system. It not only enables progress-aligned normalization of process rewards, but also ensures that the implicit process advantage provides meaningful gradient signals even when outcome-based advantages vanish due to zero reward variance.
•
3
II. R ELATED W ORK A. Token-Level Granularity in Credit Assignment
and self-supervised process reward frameworks have emerged. PRIME [4] enables online PRM updates using only policy rollouts and outcome labels, mining implicit process signals directly from the policy’s predictive dynamics. TEPO [18] uses Markov likelihood to link group-level rewards to token-level optimization via tokenmean aggregation, incorporating entropy control to avoid collapse. However, these implicit mechanisms still operate under trajectory-level or physical-position alignment, leaving the heterogeneous timing of logical progress unaddressed. Work on length bias and verbosity [29] further highlights that sequence-level or position-based penalties remain insufficient because they do not capture intrinsic information content. This motivates a self-supervised process signal that provides token-level directional guidance aligned with logical progress rather than physical token position.
The coarse granularity of sequence-level rewards has been a longstanding limitation in RLVR. Recent work has begun to address this by moving toward token-level or step-level credit assignment. GTPO [30] proposes a true token-level reward by dynamically weighting each token with its predictive entropy, providing finegrained credit to high entropy decision points. Similarly, research [20] explicitly tailor policy optimization to each token’s nature based on entropy, treating heterogeneous tokens with differentiated optimization weight rather than uniform updates. From a causal perspective, [12] introduce counterfactual importance weighting, which masks reasoning spans and measures the resulting drop in answer probability to identify tokens with genuine causal impact, effectively upweighting critical tokens while suppressing filler tokens. Exploration enhancing strategies provide another angle C. Zero-Variance Collapse and Gradient Continuity on token-level differentiation. SIREN [10] selectively A critical failure mode of GRPO occurs when all applies entropy regularization to specific logical tokens, responses in a group receive identical rewards, causing the attempting to focus exploration where it matters most. normalized advantage to vanish and the outcome-driven Research [3] directly augment the GRPO advantage with gradient signal to collapse. This issue is particularly a clipped, gradient-detached per token entropy term, prevalent in early stage training or on highly challenging encouraging exploration at pivotal tokens. In a complereasoning tasks, where policies tend to produce uniformly mentary line, SEED-GRPO [2] employs semantic entropy incorrect outputs, or conversely in saturated regimes rather than token-level entropy to quantify promptwhere all outputs are correct. Existing solutions fall into level uncertainty, enabling uncertainty-aware conservative three categories. updates for difficult prompts. Data-level interventions detect and filter uninformative Despite these advances, existing token-level methods groups before training. DAPO [35] introduces Dynamic primarily focus on exploration or importance weighting. Sampling with decoupled clipping to identify and resamThey do not address the polarity of token-level signals, ple zero-variance groups, ensuring informative gradient whether a token’s contribution is positive or negative updates. GRESO [39] proposes selective rollout that relative to the final outcome. Moreover, they rely on predicts and skips uninformative prompts based on reward heuristic entropy shaping rather than progress-aligned training dynamics, reducing wasted computation. signal extraction from the policy’s own information Task-level rebalancing addresses the uneven distribudynamics. tion of zero-variance frequency across different tasks. MTGRPO [24] observes that zero-gradient rates vary drastiB. Process-Level Polarity and Directional Feedback cally across tasks in multi-task settings, and combines a Beyond granularity, the uniform polarity of sequence- Ratio-Preserving Sampler with Improvement-Aware Task level advantages has motivated research on process- Reweighting to dynamically allocate optimization effort, level supervision. Explicit PRMs [17, 33] train separate substantially improving worst-case task performance. verifiers on human-annotated or synthetically generated Signal-level methods reshape the advantage computareasoning steps, providing dense step-wise directional tion itself to avoid collapse. RL-ZVP [14] constructs nonfeedback. Process Advantage Verifiers [27] extend this zero advantages from per-token policy entropy even under by measuring the change in future correctness probability zero reward variance, enabling continued learning from induced by each step. More recent methods such as otherwise wasted prompts. Dr. GRPO [21] introduces a PRPO [5] and Step-GRPO [16] employ pretrained PRMs learned bias term that perturbs the advantage calculation to score individual steps and align process reward to prevent collapse. EDGE-GRPO [36] combines guided distributions with sequence-level outcome advantages. error correction with entropy-driven advantage shaping Despite these advances, explicit PRMs incur high an- to inject diversity into the advantage distribution without notation costs and frequently suffer from reward hacking additional sampling. Complementary to these, RE-GRPO or poor cross-domain generalization. In response, implicit [19] maintains a hard negative case pool and applies
4
LLM-guided reflection with dual validation to repair failed trajectories, which are then distilled back into the policy, effectively converting collapsed groups into useful supervision at the cost of added inference. Despite these advances, a principled signal source that remains informative under zero reward variance without external computation, and that aligns naturally with logical reasoning progress, remains an open challenge. III. P RELIMINARIES We begin by formalizing the probabilistic framework of LLMs [1], followed by an overview of RLVR. Then, we delineate the GRPO algorithm, providing the necessary background for our proposed method. LLMs. Specifically, given an input prompt x, an LLM πθ sequentially generates a T -token response y = (y1 , ..., yT ): πθ (y|x) =
T Y
πθ (yt |x, y<t ).
where r = [r1 , r2 , ..., rG ] is the vector of rewards for all responses in the group, mean(·) and std(·) denote the mean and standard deviation operations respectively, and δ is a small constant for numerical stability. This normalization provides a relative comparison of responses within the same group, effectively estimating advantages. This assignment assumes an equal contribution of all tokens to the final outcome reward, bypassing the need for per-token value estimation. To prevent the policy from diverging too far from the reference policy, GRPO incorporates KL divergence regularization. Specifically, for each token position, the KL divergence is estimated using a low-variance approximation [25]:
DKL [πθ ∥πref ] =
(1)
t=1
RLVR. RLVR [13] is a family of reinforcement learning methods that utilize verifiable reward signals rather than learned reward models. Unlike RLHF [22], RLVR employs rule-based objective rewards, such as the correctness of a programming output, the correctness of the final answer to a mathematical problem, or compliance with formatting. These rewards originate from tasks with explicit ground truth verification. Consider a dataset D = {(x, y)} where x is the prompt and y is the ground truth. The optimization objective in RLVR is to maximize the expected reward: J (θ) = E(x,y)∼D Eŷ∼πθ (·|x) [R(ŷ, y)] ,
(2)
πref (oi,t | x, oi,<t ) πθ (oi,t | x, oi,<t ) πref (oi,t | x, oi,<t ) − log − 1. πθ (oi,t | x, oi,<t )
(4)
This estimator provides a low-variance approximation of the KL divergence while maintaining computational efficiency. The GRPO optimization objective combines a clipped surrogate objective with a KL divergence penalty. After each generation, multiple updates can be performed using the following loss function:
LGRPO (θ) = − PG
1
|oi | G X X
i=1 |oi | i=1 t=1
min ρi,t (θ)Âi,t ,
clip(ρi,t (θ), 1 − ϵ, 1 + ϵ)Âi,t
− βDKL [πθ ∥πref ]
(5) where R(ŷ, y) is a verifiable reward function that comπθ (oi,t |x,oi,<t ) where ρ (θ) = is the probability ratio i,t πθold (oi,t |x,oi,<t ) pares the generated output ŷ against the ground truth between the current policy and the old policy, clip(·, 1 − y. GRPO. GRPO [28] is a more efficient policy op- ϵ, 1 + ϵ) constrains the probability ratio to the interval timization algorithm compared with Proximal Policy [1−ϵ, 1+ϵ] to prevent excessively large policy updates, ϵ Optimization (PPO) [26], as it estimates advantages is the clipping hyperparameter, and β controls the strength through group-based response sampling, eliminating the of KL regularization. This loss function aims to maximize expected rewards need for a separate value network. For each prompt x, while constraining the magnitude of policy updates, GRPO samples G responses {o1 , o2 , . . . , oG } from the ensuring training stability. Through the combination of policy model πθ , with each response consisting of |oi | group-based advantage estimation and KL regularization, tokens. These responses are evaluated using a reward GRPO achieves stable policy optimization without requirmodel or function R(x, oi ), yielding a reward ri for each ing a value network, making it particularly suitable for response. reinforcement learning tasks with verifiable rewards. Token-level advantages Âi,t are computed through within-group normalization. Specifically, for all tokens in response oi , the advantage is set to the normalized IV. M OTIVATION reward of that response: In this section, we perform more empirical and theoretical analysis to reveal the underlying mechanisms and ri − mean(r) Âi,t = , ∀t ∈ {1, ..., |oi |}, (3) provide new insights. std(r) + δ
5
(a)
(b)
(c)
(d)
Fig. 2: Empirical analysis using standard GRPO. (a) Perturbing high entropy tokens degrades sequence accuracy substantially more than perturbing low entropy tokens, highlighting their disproportionate importance. (b) Incorrect tokens in otherwise correct sequences frequently receive positive advantages, while correct tokens in failed sequences receive negative ones, indicating poor credit assignment. (c) Severe zero-variance issues persist throughout training. (d) During the middle-to-late stages, over 50% of tokens belong to such zero-variance groups, rendering them ineffective for policy updates.
A. Contribution by Token Entropy To quantify the impact of token-level heterogeneity, we conducted perturbation experiments on a GRPO-trained policy. From the MATH [8] dataset, we selected 2,460 problems that the model answered correctly. For each problem, we identified the 5% of tokens with the highest predictive entropy and the 5% with the lowest entropy in the generated reasoning sequence, randomly replaced them with alternative tokens, and measured the resulting change in final answer accuracy. As shown in Figure 2(a), perturbing high entropy tokens caused a more than 3.5× larger accuracy drop compared to perturbing low entropy tokens. We hypothesize that this phenomenon occurs because high entropy tokens typically coincide with strategic decision points where the model chooses among competing reasoning paths, so perturbing them derails the entire subsequent trajectory; low entropy tokens, in contrast, largely appear in deterministic derivations where the model can often self-correct using surrounding logical context to preserve coherence.
incorrectly answered problems, 51.5% of steps were locally correct, yet these were all indiscriminately assigned negative advantages. These results demonstrate that GRPO inadvertently rewards errors in successful trajectories while suppressing valid reasoning in failed ones.
C. Vanishing gradients during training To quantify the prevalence of zero-variance collapse, we analyzed training logs from a standard GRPO run on the MATH dataset with G=8. Across 16,882 training steps, we recorded whether the reward standard deviation within each group fell to zero. As shown in Figure 2(c), this failure mode is not a rare edge case: 58.77% of all training steps (9,922 out of 16,882) produced zerovariance groups, rendering over 46 million tokens computationally wasted, as shown in Figure 2(d). Furthermore, the problem intensifies as training progresses. Figure 2(c) reveals that the zero-variance ratio rises from 53.6% in the early stage, to 61.0% in the middle stage, and reaches 61.8% in the late stage. This upward trend indicates that the model does not naturally outgrow this failure mode; B. Polarity misalignment in the reasoning process instead, outcome-driven gradients become progressively To further quantify the polarity misalignment, we sparser, leaving an increasing fraction of training steps sampled 500 problems from the MATH dataset and guided only by the KL regularization term rather than generated responses using a GRPO-trained policy. To by reward signals. These empirical findings raise a obtain unbiased step-level correctness annotations, each natural question: why does GRPO suffer from these response was automatically segmented into atomic rea- limitations? We now turn to a theoretical analysis of soning steps by an external LLM. Every atomic step was GRPO’s gradient dynamics. independently evaluated for local correctness by the LLM, crucially without access to the final answer outcome, D. Entropy-Driven Gradient Contribution thereby preventing hindsight bias. As shown in 2(b), the Policy optimization fundamentally relies on gradient results reveal a striking misalignment. Among correctly updates. We therefore begin by analyzing the origins of answered problems, 6.6% of intermediate reasoning gradient signals in sequence generation tasks. Consider steps were locally erroneous yet received uniformly policy: positive advantages under GRPO. More critically, among πθ (yt | x, y<t ) (6)
6
TABLE I: Confusion Matrix of Sign Combinations. Âi > 0 (Winner Group)
Âi < 0 (Loser Group)
si,t > 0 (More Certain)
True Positive (TP) Model is more confident in a correct path. → Strongest Reward
False Positive (FP) Model is more confident in a wrong path. → Strongest Penalty
si,t < 0 (Deviated)
False Negative (FN) Model deviated from reference on a correct path. → Mild Penalty
True Negative (TN) Model deviated from reference on a wrong path. → Implicit Reward
and cross-entropy loss: Lt = − log πθ (yt | x, y<t )
(7)
Taking the partial derivative of Lt with respect to the model parameters θ via the chain rule: ∂Lt ∂Lt ∂πθ 1 ∂πθ = = − · (8) · ∂θ ∂πθ ∂θ πθ ∂θ The gradient intensity is defined as the magnitude of the contribution from the loss derivative with respect to πθ : ∂Lt ∂πθ
=
1 πθ
(9)
This derivation reveals a fundamental fact: the magnitude of the gradient signal at each token position is determined solely by the model’s prediction probability πθ , independent of any specific reinforcement learning algorithm. When the model is uncertain about a token, which means a high entropy token, the loss function automatically assigns a larger gradient to drive learning. Having established the source of gradient intensity, we now examine how GRPO interacts with this inherent gradient signal. The loss function can be simplified to: G
LGRPO = −
T
1 XX Âi · log πθ (yi,t | xi , yi,<t ) (10) G i=1 t=1
where Âi is the advantage value for sequence i, computed via group-based reward normalization. Critically, Âi is a sequence-level constant, it takes the same value for all tokens t within the same sequence. Taking the partial derivative with respect to θ via the chain rule: 1 ∂πθ ∂LGRPO = −Âi · · ∂θ πθ ∂θ
(11)
GRPO Gradient
1 ∂πθ Âi × − · |{z} πθ ∂θ | {z } GRPO’s Contribution
Natural Gradient
E. Dense Rewards and Gradient Reversal In GRPO, all tokens within a sequence share the identical sign of Âi . Correct tokens in a failed trajectory are uniformly penalized, while incorrect tokens in a successful trajectory are inadvertently rewarded. The ideal solution is token-level directional feedback—dense rewards that assign positive advantage to correct tokens and negative advantage to incorrect ones. Existing PRMs provide step-level supervision but incur substantial annotation costs and generalize poorly across domains. A free process signal is needed. Consider parameterizing an outcome reward as the log-likelihood ratio between the current policy and a reference model: rθ (y) = β log ππrefθ (y) (y) . As shown in prior work [4], training with this reward implicitly learns a Q-function of the form: qθt (y<t , yt ) =
t X
β log
i=1
πθ (yi | y<i ) πref (yi | y<i )
(13)
= β log Eπref (y|y<t ) [erθ (y) ]. From this, the per-step process reward emerges by taking the difference of adjacent Q-functions: πθ (yt | y<t ) πref (yt | y<t ) πθ (yt−1 | y<t−1 ) + β log . πref (yt−1 | y<t−1 )
rtp = qθt − qθt−1 = β log
We can decompose this gradient as: ∂LGRPO = | ∂θ {z }
We can see that GRPO does not actively create or amplify the gradient disparity between high entropy and low entropy tokens. Instead, it passively inherits the natural gradient 1/πθ from the cross-entropy loss, and simply scales the entire sequence uniformly by Âi . It makes no distinction between tokens based on their information content, treating critical reasoning pivots and routine deterministic tokens identically. This passive inheritance, while computationally efficient, leaves substantial room for improvement by actively modulating token-level contributions based on their informational value.
(12)
(14)
Thus, the token-level log-probability divergence si,t = log πθ (oi,t | x, oi,<t ) − log πref (oi,t | x, oi,<t ) constitutes
7
a principled, self-supervised process signal requiring no Each token’s entropy is then standardized and mapped external annotations. Intuitively: nosep through a sigmoid gate to produce a weight Wi,t : • si,t > 0: The model is more confident than the Hi,t − µH reference model. Wi,t = σ γ · , (18) σH + ϵ • si,t < 0: The model is less certain than the reference model. where σ(·) is the sigmoid function, γ > 0 controls the However, si,t lacks intrinsic directionality. Increased steepness of the gate, ϵ ensures numerical stability. confidence is beneficial only when converging toward The entropy-gated outcome advantage is then: a correct solution, but harmful when reinforcing an Âoutcome = Wi,t · Âi . (19) error. The sequence-level outcome advantage Âi provides i,t the missing directional context. Table I summarizes the four interpretable regimes arising from the interaction of This modulation preserves the original sign of Âi while sign(Âi ) and sign(si,t ), yielding a symbolically anchored adjusting magnitude to reflect token-level informational dense reward without external supervision. Taken together, content, amplifying gradients at critical decision points these analyses point to a clear direction: amplify token- and suppressing redundant derivations. level advantages at high entropy decision pivots while suppressing redundant low entropy derivations; leverage the B. Implicit Progress Advantage self-supervised process signal si,t for directional tokenThe entropy-gated outcome advantage still inherits the level feedback; and exploit its independence from reward variance to naturally resolve zero-variance collapse. These uniform polarity of Âi . To provide token-level directional three principles form the foundation of our proposed feedback, we leverage the policy divergence against a frozen reference model πref : method. si,t = λ (log πθ (oi,t | x, oi,<t ) − log πref (oi,t | x, oi,<t )) , (20) Building on the design principles established in Sec- where λ > 0 scales the sensitivity. As derived in tion IV, we now introduce EP-GRPO, as shown in Figure Section IV-E, si,t constitutes a self-supervised process 3. Our method involves three components: entropy-gated signal requiring no external annotations. outcome advantage modulation, an implicit progress Since si,t intrinsically lacks directionality, it must be advantage derived from policy divergence, and their anchored to an external sign. We define the anchoring combination under a unified token-level objective. signal di as: ( sign(Âi ), if std(r) > 0, A. Entropy-Gated Outcome Advantage di = (21) sign(ri − θreward ), if std(r) = 0, Given an input prompt x, the policy πθ generates a response oi of length |oi |. For each token oi,t , we compute where θ reward is a reward threshold (typically 0.5 for binary the predictive entropy: rewards). When group reward variance is non-zero, the X Hi,t = − πθ (v | x, oi,<t ) log πθ (v | x, oi,<t ), (15) outcome advantage provides directional context as in Table I. When variance collapses, Âi vanishes and the v∈V anchor gracefully degrades to the raw reward, ensuring where V denotes the vocabulary. High Hi,t indicates s always receives meaningful polarity. The anchored i,t uncertainty at logical branch points, while low Hi,t signal is: corresponds to deterministic derivations. s̃i,t = di · si,t . (22) To differentiate high entropy decision pivots from V. M ETHOD
low entropy redundancy, we introduce an entropy-gated modulation. For a group of G responses, we first compute the group-level mean and standard deviation of entropy across all tokens: µH = PG
1
|oi | G X X
i=1 |oi | i=1 t=1
σH
v u u =t
1 PG
Hi,t .
|oi | G X X (Hi,t − µH )2 .
i=1 |oi | i=1 t=1
(16)
(17)
A direct global normalization of s̃i,t would be statistically unsound, as the distribution of si,t varies systematically with reasoning phase. We introduce a progress-aligend normalization strategy named logical progress buckets defined by cumulative entropy. For each response, we compute the cumulative entropy and map it to a relative information progress τi,t ∈ [0, 1]: Si,t =
t X k=1
Hi,k ,
τi,t = Si,t / Si,|oi | .
(23)
8
1
Group Sampling & Reward
3
Sample 𝐺 responses Prompt 𝒙
𝑂" : … 𝑥 + 𝑦 = 27.
𝑟" = 0
…
Policy Model 𝝅𝜽
𝑟& = 1
…
Positive real numbers 𝑥 and 𝑦 satisfy 𝑦 ! = 𝑥 " and (𝑦 − 𝑥)" = 4𝑦 " . What is 𝑥 + 𝑦?
𝑂& : … 𝑥 + 𝑦 = 36.
𝑂' : … 𝑥 + 𝑦 = 36.
𝑟' = 1
Implicit Progress Advantage (Resolve Polarity) Policy Model 𝝅𝜽
Ref Model 𝝅𝒓𝒆𝒇
(Current)
(Frozen)
…
-
Strong positive (encourage)
Near zero
Token 𝑻
…
Bucket 1 (0% → 10%)
Sequence-level Advantage (same for all tokens) +0.58
…
Bucket 𝐾
Bucket 2 (10% → 20%)
(90% → 100%)
…
+0.58
Strong negative (discourage) Circle size means magnitude strength
Update Policy Parameters EP-GRPO Loss (final advantage + KL penalty)
Entropy Axis (0% → 100%)
High Weight (amplify)
+0.58
)
Cumulative Entropy Alignment
…
+0.58
Moderate negative
Blue (+): move toward better solution Red (-): move toward worse solution
High Entropy (decision pivot)
Low Weight (suppress)
+0.58
Narrow negative
…
Entropy Weight (gate) …
Narrow positive
Implicit Progress Advantage (direction varies, strength varies)
( Low Entropy (deterministic)
Moderate positive
…
Per-token Entropy (uncertainty)
…
…
…
Entropy-Gated Outcome Advantage (Resolve Granularity)
Token 3
Implicit Progress Advantage
Final Token-Level Advantage (dense, directional, progress-aware)
Anchor with 𝒅𝒊 (from Step 1)
Token 2
From Step 3
…
Else: use reward (0/1)
Token 1
From Step 2 Entropy-Gated Outcome Advantage …
Policy Divergence per token log 𝜋! − log 𝜋"#$
+
If 𝑠𝑡𝑑 𝑟 > 0: use sign(𝐴% )
2
Combine to Final Token-Level Advantage
Compare per token
(Current) Anchoring Signal 𝒅𝒊
4
Policy Model 𝝅𝜽 (Current) Gradient Update 𝜋+ (updated)
Entropy-Gated Outcome Advantage (direction consistent, strength varies)
(
…
)
Normalize implicit progress advantage with each bucket
Fig. 3: Overview of EP-GRPO. EP-GRPO transforms sparse outcome rewards into dense, token-level learning signals. It first applies entropy-gated modulation to assign higher weights to important tokens while keeping a consistent direction. Then, an implicit progress signal is computed by comparing the policy with a reference model, providing token-level feedback with varying magnitude and direction, further aligned via cumulative entropy-based bucketing. Finally, both signals are combined into a unified token-level advantage for stable policy optimization.
Tokens from all group responses are partitioned into K equal-width buckets based on τi,t . Within each bucket Bk , we compute the Z-score of s̃i,t : s̃i,t − µk s̃norm , i,t = σk + ϵ
C. Final Token-Level Advantage The complete token-level advantage combines both components: outcome Âfinal + Âprogress . i,t = Âi,t i,t
∀(i, t) ∈ Bk ,
(24)
where µk and σk are the bucket mean and standard deviation. This aligns tokens by logical advancement rather than physical position: tokens at similar reasoning stages are compared against each other regardless of their absolute sequence index. The implicit progress advantage is then: Âprogress = η · s̃norm i,t , i,t
(25)
where η > 0 controls its contribution relative to the outcome advantage.
(26)
When std(r) > 0, both components are active: entropygated modulation resolves the granularity issue, and the anchored implicit signal provides token-level polarity. When std(r) = 0, the outcome advantage Âoutcome i,t vanishes, but Âprogress remains fully operative. Its anchor i,t degrades to the raw reward ri , and its magnitude depends solely on policy divergence. The zero-variance collapse is thus resolved without additional sampling, auxiliary models, or heuristic advantage shaping. VI. T HEORETICAL A NALYSIS We establish a theoretical properties of EP-GRPO. We prove that its gradient is equivalent to that of a regularized GRPO objective, providing a principled interpretation of the optimization dynamics.
9
A. Gradient Equivalence and Implicit Regularization We begin by establishing the relationship between the EP-GRPO gradient and the original GRPO objective. For analytical simplicity, we consider the typical case where std(r) > 0.
Completing the square, this is exactly the gradient of: " 2 # β πθ (oi,t ) , (33) F (πθ ) = Ei,t Λi,t log 2 πref (oi,t )
which completes the proof. Theorem VI.1. The EP-GRPO gradient is equivalent to the gradient of the GRPO objective augmented with an entropy-weighted squared log-ratio regularization term. Remark VI.1. Theorem VI.1 reveals that EP-GRPO Formally, there exists a potential function F (πθ ) such implicitly penalizes large deviations from the reference policy, with penalty strength modulated by Λi,t . Crucially, that: Λi,t inherits the entropy-gated weight Wi,t , meaning ∇θ JEP-GRPO (θ) = ∇θ JGRPO (θ) + η · ∇θ F (πθ ), (27) tokens at critical decision points (high entropy) receive 2 stronger regularization than those in deterministic derivaπθ (oi,t ) β where F (πθ ) = 2 Ei,t Λi,t log πref (oi,t ) , and Λi,t tions (low entropy). This transforms the passive, uniform captures the entropy-gated modulation and sign anchor- KL constraint of standard GRPO into an active, precisioning. guided mechanism that stabilizes exploration precisely where it matters most. Proof. Recall from Section V that the EP-GRPO tokenlevel advantage decomposes as: outcome Âfinal + Âprogress = Wi,t Âi + η · s̃norm i,t . (28) i,t = Âi,t i,t
The policy gradient for EP-GRPO is therefore: X . (29) ∇θ JEP-GRPO = E ∇θ log πθ (oi,t ) · Âfinal i,t
VII. E XPERIMENTS A. Experimental Setup
Datasets and Evaluation Benchmarks. For reinforcement learning, we used the Skywork-OR1-RL-Data [7], i,t a curated dataset of 105K math problems with verified Substituting the decomposition and isolating the answers. From this dataset, we randomly sampled 8,000 progress advantage term yields: problems for training. To evaluate generalization and reasoning capabilities, we tested on five competitive X gated mathematics benchmarks: MATH500 [8], AMC23 [31], ∇θ JEP-GRPO = ∇θ JGRPO +η·E ∇θ log πθ (·) · s̃norm , i,t Minerva [15], AIME24 [37] and AIME25 [38]. All i,t (30) benchmarks were evaluated using accuracy, format rate gated where JGRPO denotes the GRPO objective with entropy- and pass@k metrics. gated weighting. Model Configurations. We evaluated EP-GRPO on Under the assumption std(r) > 0, the anchored signal two scales of the Qwen2.5 series [23]: 3B and 7B. is s̃i,t = sign(Âi ) · β(log πθ − log πref ). During gradient To ensure training efficiency, we employed Low-Rank computation, bucket-level normalization statistics µk , σk Adaptation (LoRA) [9] with rank r = 32 and αlora = 64, are treated as constants computed from the old policy πθold . targeting all linear layers in both the attention and feedLet Λi,t denote the combined scaling factor absorbing forward modules. For EP-GRPO, we set the implicit sign(Âi ), the normalization constant, and η. The progress signal scale λ = 0.1, progress advantage weight η = 0.2, gradient term becomes: number of cumulative entropy buckets K = 10, entropy X gate steepness γ = 5.0 and reward threshold θreward = ∆∇θ = β · E Λi,t ·∇θ log πθ (oi,t ) log πθ (oi,t ) 0.5. i,t Training Specifications. All models were trained for 1,000 steps using the TRL framework [32] with a − log πref (oi,t ) . maximum sequence length of 2,048 tokens. We used a (31) group size of G = 8 rollouts per prompt with temperature 1 2 Using the identity ∇θ log π · log π = 2 ∇θ (log π) , we T = 1.0 and top-p = 0.95 for sampling. The learning obtain: rate was set to 5 × 10−6 with a 0.1 warmup ratio and " 1 X linear decay, optimized using AdamW with weight decay Λi,t · ∇θ (log πθ (oi,t ))2 ∆∇θ = β · E 0.001. The β of KL divergence is set to 0.001. Training 2 i,t (32) employed a global batch size of 16. All experiments were # conducted on NVIDIA RTX PRO 6000 Blackwell 96GB − log πref (oi,t ) log πθ (oi,t ) . GPUs with a fixed random seed 42.
10
TABLE II: Performance comparison across reasoning benchmarks. Acc denotes sample accuracy over all responses and Fmt denotes format correctness (boxed rate). Sampling temperature is set to 1.0, and the maximum output length is limited to 2048 tokens. Each problem is sampled 16 times to reduce variance. Best results are in bold and second-best are underlined.
Model
Math 500
AMC 23
Minerva
AIME 24
AIME 25
Acc
Acc
Acc
Acc
Acc
Fmt
Fmt
Fmt
Fmt
Avg
Fmt
Acc
Fmt
Commercial Models DeepSeek-R1-671B-0528 47.03 58.28 33.91 33.91 11.41 34.38 13.54 13.54 11.04 11.04 Qwen3-235B-A22b-Instruct 72.03 87.19 47.81 55.00 17.66 92.34 24.58 29.58 16.88 20.21
23.39 35.79
30.23 38.40
Qwen2.5-3B Base GRPO - Higher Temp - More Rollouts EP-GRPO
31.56 50.94 51.41 55.16 56.88
79.38 91.88 92.71 92.71 90.00
0.83 0.63 1.04 1.46 2.92
10.71 80.00 95.21 18.14 97.08 18.92 97.50 19.13 97.50 22.93 (+26.4%)
Qwen2.5-7B Base GRPO EP-GRPO
29.06 79.53 23.13 82.34 3.13 77.03 3.96 80.42 61.25 99.22 47.19 96.88 13.13 98.59 8.75 94.17 64.53 99.53 49.38 97.03 15.47 99.06 12.92 94.38
1.88 5.21 9.38
84.58 12.23 80.78 96.04 27.11 96.98 96.67 30.34 (+11.9%) 97.33
B. Baselines
73.12 98.28 97.50 99.22 99.06
15.94 28.28 32.19 30.31 39.53
74.38 96.56 96.09 96.25 95.31
3.13 6.25 5.78 5.78 9.06
65.94 99.22 98.12 98.28 96.88
2.08 4.58 4.17 2.92 6.25
74.56 96.23 96.30 96.79 95.75
C. Main Results
To comprehensively evaluate EP-GRPO, we compare To validate the effectiveness of EP-GRPO across against the following configurations: different model scales, we compare EP-GRPO against • Base Model. The pre-trained Qwen2.5 model with- all baselines on Qwen2.5-3B and Qwen2.5-7B. out any fine-tuning, serving as the performance lower Table II reports the sample accuracy (Acc) and format bound. correctness (Fmt) of all methods across five mathematical • Commerical Model. A high-parameter model that reasoning benchmarks. EP-GRPO consistently outpercan be accessed through a user interface has been forms standard GRPO at both model scales. At the 3B launched. scale, EP-GRPO achieves the highest accuracy on all five • GRPO. The standard Group Relative Policy Opdatasets, improving the average accuracy from GRPO’s timization [28] under identical rollout and reward 18.14% to 22.93%. At the 7B scale, EP-GRPO also settings. achieves the highest accuracy on five datasets, with the • Higer Temp. Increases the sampling temperature average accuracy increasing from 27.11% to 30.34%. from 1.0 to 1.2 while keeping G = 8. Notably, at the 3B scale, EP-GRPO even surpasses • More Rollouts. Increases the rollout from G = 8 DeepSeek-R1-671B on AMC23 (39.53% vs. 33.91%) to G = 10 while keeping T = 1.0. and approaches it on Minerva (9.06% vs. 11.41%), • Ablation Configurations. To assess the contribudemonstrating the substantial potential of improved credit tion of each EP-GRPO component, we evaluate assignment in RLVR under limited parameter budgets. four stripped-down variants on Qwen2.5-3B, each Comparisons with two GRPO variants further rule isolating a specific mechanism: out alternative explanations. Higher Temp and More – +EG: GRPO with entropy gating only, scaling Rollouts fail to consistently outperform standard GRPO sequence advantage to token-level advantage. across most datasets and are significantly inferior to EP– +IPS: GRPO with implicit progress signals and GRPO in all cases. This confirms that the performance cumulative entropy bucketing. gains of EP-GRPO stem not from increased exploration – +EG+IPS: GRPO with entropy gating and or more extensive sampling, but from the fundamental implicit progress signals (including bucketing), improvements in credit assignment enabled by entropybut without zero-variance degradation. gated modulation and implicit process rewards. – +IPS+ZVD: GRPO with implicit progress sigFor commercial models, we observe that under the nals (including bucketing) and zero-variance 2,048 token generation limit, a fraction of their outputs degradation, without entropy gating. are truncated before producing a final answer, resulting
11
TABLE III: Comprehensive pass@k evaluation (k ∈ {4, 8, 16}). Model
@4
Math 500 @8 @16
@4
AMC 23 @8 @16
@4
Minerva @8 @16
@4
AIME 24 @8 @16
@4
AIME 25 @8 @16
Commercial Models DeepSeek-R1-671B-0528 66.63 69.33 70.00 58.72 66.19 72.50 20.94 23.78 25.00 28.07 32.14 33.33 23.68 28.82 33.33 Qwen3-235B-A22B-Instruct 76.06 78.07 80.00 59.66 63.00 65.00 21.44 23.40 25.00 28.32 30.00 33.33 22.21 24.22 26.67 Qwen2.5-3B Base GRPO - Higher Temp - More Rollouts EP-GRPO
78.16 95.28 99.79 50.15 75.27 94.26 99.68 100.00 73.64 93.12 94.48 99.70 100.00 78.95 95.62 96.00 99.85 100.00 76.51 94.54 96.58 99.89 100.00 86.71 98.26
Qwen2.5-7B Base GRPO EP-GRPO
74.78 93.70 99.62 65.17 87.96 98.60 11.95 22.54 40.20 14.96 27.79 48.15 7.31 14.15 26.49 97.78 99.95 100.00 92.29 99.42 100.00 43.12 67.77 89.77 30.75 52.20 77.45 19.32 35.02 58.10 98.44 99.98 100.00 93.49 99.59 100.00 49.03 74.14 93.44 42.60 67.21 89.47 32.64 54.78 79.84
94.00 99.55 99.82 99.72 99.97
11.95 22.54 40.20 8.10 15.61 28.99 3.30 6.52 12.72 22.80 40.50 64.84 17.16 31.49 53.37 2.48 4.93 9.69 21.24 38.07 61.88 15.70 29.04 49.95 4.11 8.09 15.65 21.24 38.07 61.88 11.20 21.23 38.20 5.72 11.17 21.25 31.68 53.44 78.54 22.82 40.56 64.99 11.20 21.23 38.20
in lower than expected pass rates on complex reasoning This functional specialization across datasets aligns tasks. EP-GRPO achieves correct answers using substan- with the nature of the problems each component addresses. tially shorter reasoning paths. The implicit progress signal primarily resolves polarity Table III reports the pass@k (k ∈ {4, 8, 16}) scores for misalignment, and its benefits are most pronounced on all methods. EP-GRPO achieves the best results across high difficulty, long chain reasoning datasets such as all datasets and model scales. We find that on medium- AIME24 and Minerva. Zero-variance degradation targets difficulty datasets such as MATH500 and AMC23, GRPO gradient collapse, playing a critical role in later training and its variants exhibit rapid accuracy growth as the stages and on prompts where all rollouts receive identical number of samples increases, narrowing the gap with rewards. Entropy gating provides magnitude modulation EP-GRPO at high k values. However, on high-difficulty that filters noisy signals at low entropy tokens, compledatasets such as AIME24 and AIME25, the advantage menting both of the above mechanisms. The fact that no of EP-GRPO widens substantially with larger k (3B: single ablated variant dominates across all benchmarks AIME25 pass@16 38.20% vs. 9.69%; 7B: AIME25 confirms that the three components are non-redundant pass@16 79.84% vs. 58.10%). This indicates that EP- and collectively necessary for robust performance. GRPO’s implicit process rewards and progress-aligned normalization enable the model to discover correct reasoning paths that standard GRPO fails to identify, E. Training Dynamics Figure 4 visualizes the accuracy trends of GRPO and particularly on challenging problems. EP-GRPO over training steps at both model scales. At the 7B scale, EP-GRPO exhibits a stable advantage over D. Ablation Study GRPO from the earliest training steps. At the 3B scale, Table IV reports the ablation results on Qwen2.5-3B, EP-GRPO begins to consistently outperform GRPO after with each variant isolating one or two core components of approximately 200 steps and progressively widens the EP-GRPO. Overall, every component contributes positive gap. gains in average accuracy, but their contributions exhibit This discrepancy may be attributed to the initial notable specialization across datasets. reasoning capability of the base model. The 7B model On datasets with abundant non-zero-variance training already possesses a certain level of mathematical reasteps, such as AMC23 and Minerva, the full EP-GRPO soning ability at initialization, allowing EP-GRPO’s with all three components working in concert achieves the implicit process rewards to provide effective token-level absolute best performance (AMC23: 39.53% vs. the best directional feedback from the early stages. The 3B model, ablation 36.09%; Minerva: 9.06% vs. the best ablation by contrast, requires a warm-up period of approximately 7.97%). On AIME24, +IPS achieves 6.04%, close to the 200 steps to establish reliable entropy estimates and full EP-GRPO’s 6.25%. On AIME25, +EG and +IPS and implicit signals before these mechanisms can take full ZVD both achieve 2.08%, within one percentage point effect. This interpretation is corroborated by the reward of the full EP-GRPO’s 2.92%. curves in Figure 5, where the reward advantage of EP-
12
Fig. 4: Comparison of training trends over all datasets, smoothed with EMA (α = 0.2). EP-GRPO (Ours)
Accuracy (%)
Math500
AMC23
0.56 0.48
0.08
0.30
0.06
0.24
0.32
0.18 300
600
Minerva
0.36
0.40
0
GRPO (Baseline)
900
AIME24 0.024 0.045
Steps
600
900
0.012 0.006
0.015
0.02 300
0.018
0.030
0.04
0
AIME25 0.030
0.060
0
300
Steps
600
900
0
300
Steps
600
900
0
300
600
Steps
Steps
AIME24
AIME25
900
(a) Training dynamics of 3B models.
Accuracy (%)
Math500
AMC23
0.64
Minerva 0.15
0.48
0.56
0.12
0.40
0.48 0.40
0.32
0.32
0.24 0
300
600
900
0.09 0.06
Steps
300
600
900
0.08
0.100
0.06
0.075
0.04
0.050
0.03 0
0.125
0
300
Steps
600
900
0.025
0.02 0
Steps
300
600
Steps
900
0
300
600
900
Steps
(b) Training dynamics of 7B models.
GRPO emerges and widens following the same temporal pattern. F. Training Monitoring Metrics Figure 5 further presents the trends of key monitoring metrics throughout training. EP-GRPO achieves higher average rewards than GRPO at both model scales, with the reward advantage progressively widening during the middle and late stages of training. In terms of KL divergence, EP-GRPO maintains slightly higher values than GRPO with occasional spikes, particularly in the early stages of training. This behavior is consistent with the design of the implicit progress advantage: by providing token-level directional feedback derived from policy divergence si,t , EP-GRPO actively encourages the policy to deviate from the reference model at high entropy decision points where the implicit signal is most informative. The occasional KL spikes coincide with training steps where the model encounters unusual or particularly challenging reasoning patterns, triggering stronger token-level corrections. Importantly, despite these transient increases, the KL divergence does not exhibit unbounded growth, confirming that the entropy-gated modulation effectively constrains the regularization to critical tokens rather than allowing uncontrolled deviation. Regarding gradient norms, EP-GRPO exhibits larger values than GRPO with occasional sharp spikes. This is a direct consequence of the zero-variance degradation mechanism: on training steps where std(r) = 0, standard
GRPO produces near-zero gradient norms because the outcome advantage vanishes, whereas EP-GRPO sustains non-zero gradients through the implicit process advantage . These sustained gradients manifest as larger Âprogress i,t overall gradient norms and, when accumulated across multiple consecutive zero-variance steps, can produce the observed spikes. Rather than indicating training instability, these larger gradient norms reflect EP-GRPO’s ability to continue learning from batches that would otherwise be wasted. G. Computational Efficiency Figure 6 compares the training time and average output length of each method on Qwen2.5-3B. The total training wall-clock time of EP-GRPO is on par with that of standard GRPO. The additional computational overhead introduced by EP-GRPO is negligible, as all required quantities are either already computed during the standard GRPO training pipeline or can be derived with lightweight operations. More Rollouts incurs substantially longer training time due to the increased number of responses generated during the sampling phase. In terms of output length, EP-GRPO produces reasoning chains that are approximately 20% longer on average than those of GRPO, indicating that the model engages in deeper and more thorough reasoning. Taken together with the comparable training time, EP-GRPO achieves a superior accuracy efficiency trade-off, it obtains significantly higher accuracy through longer reasoning chains, without incurring additional temporal cost.
13
TABLE IV: Ablation study on Qwen2.5-3B. Acc denotes sample accuracy and Fmt denotes format correctness (boxed rate). Best results are in bold. Math 500
AMC 23
Minerva
AIME 24
AIME 25
Model
Acc
Acc
Acc
Acc
Acc
Fmt
Acc
Fmt
Base GRPO
31.56 73.12 15.94 74.38 3.13 65.94 2.08 79.38 0.83 80.00 50.94 98.28 28.28 96.56 6.25 99.22 4.58 91.88 0.63 95.21
10.71 18.14
74.56 96.23
Fmt
Fmt
Fmt
Fmt
Avg
Single Component +EG 52.66 98.59 32.03 96.56 6.72 98.12 4.58 92.29 2.08 95.42 19.61 (-14.5%) 96.20 +IPS 54.06 98.75 31.72 96.09 6.88 96.72 6.04 90.83 1.25 95.62 19.99 (-12.8%) 95.60 Two Components +EG+IPS +IPS+ZVD
52.81 98.44 36.09 97.19 7.97 98.44 4.79 92.71 0.83 97.50 20.50 (-10.6%) 96.86 54.84 99.53 35.63 95.62 7.97 96.56 4.17 88.75 2.08 96.67 20.94 (-9.5%) 95.43
Full Model EP-GRPO (All)
56.88 99.06 39.53 95.31 9.06 96.88 6.25 90.00 2.92 97.50
22.93
95.75
Fig. 5: Comparison of training dynamics between EP-GRPO and GRPO. EP-GRPO (Ours)
Reward
GRPO (Baseline)
Entropy
Grad Norm
KL Divergence
Value
0.6 200
0.4
500
0.3
0.0
0
0 0
300
600
900
0
300
Steps
600
900
0
300
Steps
600
900
0
300
Steps
600
900
Steps
(a) Training dynamics on Qwen2.5-3B. Reward
Entropy
Grad Norm
KL Divergence
Value
0.50 0.5
8
4 0.25
0.0
0
0 0
300
600
900
0
Steps
300
600
Steps
900
0
300
600
Steps
900
0
300
600
900
Steps
(b) Training dynamics on Qwen2.5-7B.
VIII. C ONCLUSION
tent and substantial improvements over standard GRPO, with ablation studies confirming the non-redundant contriThis paper identified three fundamental credit as- bution of each component and control experiments ruling signment failures in GRPO for RLVR, uniform token out alternative explanations. In the future, we will explore granularity, uniform polarity, and zero-variance collapse. more possibilities for process rewards and the laws of We proposed EP-GRPO, a framework that addresses entropy-driven exploration. them through entropy-gated modulation, implicit process R EFERENCES signals anchored to outcome advantages, and cumulative entropy mapping for progress-aligned normalization. [1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Extensive experiments on two model scales across five Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind mathematical reasoning benchmarks demonstrated consisNeelakantan, Pranav Shyam, Girish Sastry, Amanda
14
GRPO
Higher Temp
More Rollouts
Training Efficiency
15
Length Dynamics
17.4h 14.2h
14.5h
900
15.0h
Tokens
Runtime (Hours)
20
EP-GRPO
10
800
700
5 600 0
GRPO
Higher Temp More Rollouts
EP-GRPO
Methods
0
300
600
900
Training Steps
Fig. 6: Training efficiency and output length comparison on Qwen2.5-3B.
Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020. [2] Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization, 2025. [3] Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 30377–30385, 2026. [4] Guoxin Cui et al. Process reinforcement through implicit rewards, 2025. [5] Rui Ding, Yifan Lv, Xiang Meng, Jiaming Song, Chi Wang, Chengrui Jiang, and Yong Cheng. Prpo: Aligning process reward with outcome reward in policy optimization, 2026. [6] Daya Guo, Dejian Yang, Haowei Zhang, Junmei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseekr1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. [7] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312, 2025. [8] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Advances in Neural Information Processing Systems, 2021. [9] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large
language models. In International Conference on Learning Representations, 2022. [10] Yuxian Jiang, Yafu Li, Guanxu Chen, Dongrui Liu, Yu Cheng, and Jing Shao. Rethinking entropy regularization in large reasoning models, 2025. [11] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know, 2022. [12] Mykola Khandoga, Rui Yuan, and Vinay Kumar Sankarapu. Beyond uniform credit: Causal credit assignment for policy optimization, 2026. [13] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training, 2024. COLM 2024. [14] Thanh-Long V. Le, Myeongho Jeon, Kim Vu, Viet Lai, and Eunho Yang. No prompt left behind: Exploiting zero-variance prompts in llm reinforcement learning via entropy-guided advantage shaping, 2025. Also accepted at ICLR 2026. [15] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843– 3857, 2022. [16] Wen Li, Jinpeng Wang, Liang-Chi Yu, and Xiaozhen Zhang. Step-grpo: Enhancing reasoning quality and efficiency via structured prm-based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026. [17] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike,
15
John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. [18] Xingyu Lin, Yilin Wen, En Wang, Du Su, Wenbin Liu, Chenfu Bao, and Zhonghou Lv. Token-level policy optimization: Linking group-level rewards to token-level aggregation via markov likelihood, 2025. [19] Haoyu Liu and Le Xiao. Re-grpo: Leveraging hard negative cases through large language model guided self training. Neurocomputing, page 132543, 2025. [20] Zheng Liu, Mengjie Liu, Siwei Wen, Mengzhang Cai, Bin Cui, Conghui He, and Wentao Zhang. From uniform to heterogeneous: Tailoring policy optimization to every token’s nature, 2025. [21] Omar El Mansouri, Mohamed El Amine Seddik, and Salem Lahlou. Noise-corrected grpo: From noisy rewards to unbiased gradients, 2025. [22] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730– 27744, 2022. [23] Qwen Team. Qwen2.5 technical report, 2025. [24] Shyam Sundhar Ramesh, Xiaotong Ji, Matthieu Zimmer, Sangwoong Yoon, Zhiyong Wang, Haitham Bou Ammar, Aurelien Lucchi, and Ilija Bogunovic. Multi-task grpo: Reliable llm reasoning across tasks, 2026. [25] John Schulman. Approximating kl divergence. http: //joschu.net/blog/kl-approx.html, March 2020. [26] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [27] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning, 2024. [28] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. [29] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf, 2023. [30] Hongze Tan, Zihan Wang, Jianfei Pan, Jinghao Lin, Hao Wang, Yifan Wu, Tao Chen, Zhihang Zheng, Zhihao Tang, and Haihua Yang. Gtpo and grpos: Token and sequence-level reward shaping with policy entropy, 2025.
[31] Math-AI Team. Amc 2023 dataset. https:// huggingface.co/datasets/math-ai/amc23, 2025. Problems from 2023 American Mathematics Competitions (AMC 12A/12B). [32] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformers Reinforcement Learning, 2020. [33] Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-bystep without human annotations, 2024. [34] Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [35] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An opensource llm reinforcement learning system at scale, 2025. v2: May 2025. [36] Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. Edge-grpo: Entropy-driven grpo with guided error correction for advantage diversity, 2025. [37] Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2024. https: //huggingface.co/datasets/math-ai/aime24, 2024. [38] Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2025. https: //huggingface.co/datasets/math-ai/aime25, 2025. [39] Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for llm reasoning via selective rollouts. In Advances in Neural Information Processing Systems (NeurIPS), 2025. Also known as GRESO (GRPO with Efficient Selective Rollout).