STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability Qingfeng Sun Songli Wu Can Xu† Wenfeng Deng Han Hu Yansong Tang† Shenzhen International Graduate School, Tsinghua University Tencent Hunyuan {luohp24@mails., tang.yansong@sz.}tsinghua.edu.cn {victorqsun,leocaxu}@tencent.com

Haipeng Luo

Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage-surprisal four-quadrant structure and a near-criticality property. Motivated by it, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), which identifies entropy-critical token subsets via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation. Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%–8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration–exploitation balance that further unlocks RL training potential. Code is available at https://github.com/hp-luo/STARE 0.4

0.5 0.4 0.3

0.2 0.1 0

GRPO-ds STARE

400

0.2

800

1200 1600 2000

Training Steps

(a) Training Entropy

0.1

AIME25 Acc

0.3

AIME24 Acc

0.3

Entropy

arXiv:2606.19236v1 [cs.LG] 17 Jun 2026

Abstract

400

800

GRPO-ds STARE 1200 1600 2000

Training Steps

(b) AIME24 Acc

0.1 0

400

800

GRPO-ds STARE 1200 1600 2000

Training Steps

Figure 1: Training dynamics of STARE vs. GRPO-ds on Qwen2.5-7B-Base (cold-start SFT from Retool 2K) in the tool-use agent scenario: policy entropy, AIME24 accuracy, and AIME25 accuracy.

Introduction

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as the dominant post-training paradigm for eliciting complex reasoning in LLMs, as exemplified by DeepSeek-R1, Qwen3, and Kimi K1.5(Guo et al., 2025; Achiam et al., 2023; Team et al., 2025a; Team, 2025; Anthropic, 2025). Among RLVR algorithms, Group Relative Policy Optimization (GRPO) dispenses with the value network and instead employs group-normalized rewards as the baseline for advantage estimation; and it has been widely adopted in mathematical reasoning, code generation, and is effective at inducing emergent behaviors like long-cot reasoning and self-reflection(Shao et al., 2024; Schulman et al., 2017). As RL training extends over more optimization steps, however, GRPO-style algorithms commonly suffer from policy entropy collapse: entropy decays rapidly, output diversity vanishes, the policy converges prematurely, and within-group rollouts homogenize, degrading relative advantage estimation and † Corresponding authors. This work was done during Luo’s internship at Tencent and was supported by the CIE-Tencent Ph.D. Student Research Incentive Program (Tencent Hunyuan Special Fund).

ultimately capping trainable steps, a critical bottleneck for long-horizon post-training(Yue et al., 2025a; Farquhar et al., 2024). Existing mitigations fall into three directions. (i) Adjusting the clipping thresholds for the importance-sampling ratio (i.e., DAPO’s clip-higher) protects low-probability exploratory tokens, but exerts an asymmetric and uncontrollable effect on entropy and is largely inactive in the on-policy regime where the ratios stay near one(Yu et al., 2025; Chen et al., 2026; Xi et al., 2025; Fu et al., 2026). (ii) Asymmetric trajectory-level weighting between positive and negative rollouts: Upweighting rare correct rollouts or biasing updates toward negative samples, provides coarse-grained control(Zhu et al., 2025; Tang et al., 2025; Wang et al., 2025b; 2026; Yang et al., 2025b; He et al., 2025a; 2026). (iii) Entropy-aware advantage reshaping or entropy regularization couples token-level entropy into the advantage, but tends to overamplify high-entropy tokens, induce oscillations, and remain hyperparameter-sensitive(Cheng et al., 2025; Cui et al., 2025; Xu et al., 2025; He et al., 2025c; Yang et al., 2025a; Huang et al., 2025). These approaches slow entropy decay to varying degrees, yet operate at trajectory or sample granularity, lacking a principled account of the collapse mechanism. Two questions remain open: which tokens drive entropy decay under GRPO, and how strong an intervention suffices to reverse it. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: although the trajectory-level advantage Âi is shared across all tokens within a rollout, the per-token entropy contribution decomposes into the product of Âi and a local entropy sensitivity function Φ determined by the next-token distribution (Section 3). This decomposition yields an advantage–surprisal four-quadrant view: within positive-advantage trajectories, low-surprisal tokens dominate the sampling frequency and drive most entropy-decreasing updates, whereas the rare high-surprisal tokens that could raise entropy are diluted (a mirror-image asymmetry holds for negativeadvantage trajectories). We establish a near-criticality property: a mild token-level weight perturbation suffices to flip the sign of entropy evolution and is robust to the specific weight value and beyond the critical threshold, the weight modulates the magnitude rather than the direction of the entropy shift. Motivated by these, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), a minimally invasive token-level credit-rebalancing mechanism that operates inside the clipped surrogate of GRPO. STARE selects an entropy-critical token subset via batch-internal surprisal quantiles, thereby selectively amplifying the effective advantage of positive-advantage high-surprisal tokens and attenuating that of negative-advantage high-surprisal tokens. A target-entropy closed-loop gate further governs the intervention: the reweighting is activated to restore exploration when the batch-averaged entropy H̄k drops below a target level Htgt , and reverts to GRPO once entropy recovers, yielding closed-loop, stable, and low-intrusion entropy regulation. We validate STARE across multiple model scales and task regimes. On 7B models, STARE stably sustains over 5k RL training steps; on 14B and 32B models, it sustains over 1.5k steps; throughout training, the policy entropy is held within the target band. Across three task families spanning Short CoT, Long CoT, and multi-turn tool-use agents, STARE consistently outperforms DAPO on AIME24 and AIME25 by 4%-8% in average accuracy, with reflection-related tokens and response length growing in tandem, indicating an improved exploration–exploitation balance. The main contributions of this work are as follows: (i) From a first-order entropy-dynamics analysis, we expose the token-level credit assignment mismatch in GRPO and establish a near-criticality property: a mild weight perturbation suffices to flip the direction of entropy evolution while remaining robust to the weight value. (ii) We propose a surprisal-based advantage reweighting mechanism coupled with a target-entropy closed-loop constraint, achieving stable policy-entropy regulation through a minimal modification to the GRPO objective and sustaining RL training over thousands of steps. (iii) We validate STARE across model scales from 1.5B to 32B and across Short CoT, Long CoT, or multi-turn tool-use regimes, where it maintains stable policy entropy and substantially outperforms DAPO and other baselines by a consistent margin. We defer the discussion of related work to Appendix A.

Preliminaries

GRPO. Given prompt x, the old policy πθold samples G responses with rewards {ri }iG=1 ; the groupnormalized advantage is Âi = (ri − mean({r j }))/ std({r j }).The clipped surrogate objective is:

JGRPO (θ ) =

1 B Ti min ρi,t (θ ) Âi , clip(ρi,t (θ ), 1−ϵ, 1+ϵ) Âi , ∑ ∑ N i =1 t =1

where ρi,t (θ ) ≜ πθ (oi,t | xi , oi,<t )/πθold (oi,t | xi , oi,<t ) is the per-token importance ratio, B is the number of responses, N = ∑iB=1 Ti is the total token count, and β = 0 throughout (no KL penalty). Let c = ( x, o<t ) denote context and V the vocabulary. The next-token distribution is parameterized as πv ≜ πθ (v | c) = exp(zv )/ ∑v′ exp(zv′ ), with softmax Jacobian ∂πv′ /∂zv = πv′ (δv′ v − πv ). Token Surprisal, 2

Entropy, and Logit updates. The token surprisal is sv ≜ − ln πv and the position-level policy entropy is H ≜ − ∑v πv ln πv = Eπ [s]Shannon (1948); Oh et al. (2024); Zeng et al. (2026); Oh & Schuler (2023); Smith & Levy (2013); the batch mean H̄ ≜ N −1 ∑i,t Hi,t typically decreases during RL fine-tuning (entropy collapse). In the unclipped regime, the GRPO gradient at a position where token a was sampled yields the logit update ∆zv = η Â (δva − πv ) for all v ∈ V , where η > 0 is an infinitesimal step size (Appendix D.2). ∂H = πv(sv − H ). Lemma 2.1 (Entropy gradient w.r.t. logits: surprisal-deviation form). For any v ∈ V , ∂z v Appendix D.3 provides the derivation.

Raising zv increases entropy when token v is rarer than average (sv > H) and decreases it otherwise. It relates entropy to individual logits and underlies the token-level entropy dynamics derived below.

Theoretical Analysis

We develop a theoretical framework for analyzing entropy evolution during GRPO training, proceeding from token-level gradient analysis (Section 3.1) through an advantage–surprisal decomposition (Section 3.2) and a batch-level near-criticality result (Section 3.3) to cross-step feedback dynamics (Section 3.4). Complete proofs are in Appendices D–F. 3.1

First-Order Gradient Analysis of Token-Level Policy Entropy

Consider the next-token distribution π (· | c) with entropy H. Let a denote the sampled token, with probability p = π ( a | c) and surprisal sa = − ln p. Define S2 ≜ ∑v∈V πv2( ln πv + H ) and Φ( p) ≜ p( ln p + H ) − S2 . We call Φ the entropy sensitivity function: it measures the signed excess of the sampled token’s probability-weighted surprisal deviation over the distributional baseline S2 . Theorem 3.1 (Token-level entropy variation). In the unclipped regime of GRPO, let Â denote the advantage at this position, and let η be the step size along the GRPO policy-gradient direction. Then dH dη

η =0

= − Â Φ( p).

The proof (Appendix D.4) follows by taking the inner product of ∂H/∂zv (Lemma 2.1) with the GRPO logit update. The result decomposes the instantaneous entropy effect into the advantage Â and Φ( p), governing whether probability redistribution concentrates or disperses the distribution. And their product determines the sign and magnitude of entropy variation at each position. 3.2

Advantage–Surprisal Four-Quadrant Decomposition

Determining sign(dH/dη ) reduces to analyzing sign(Φ( p)). Two properties, proved in Appendix E, underpin the analysis: S2 > 0 for any non-uniform distribution (Lemma E.1), and H > S2 for any non-degenerate distribution (Lemma E.2). These imply Φ(0+ ) = −S2 < 0 and Φ(1) = H − S2 > 0. Since Φ′ ( p) = ln p + H + 1, the function is strictly increasing on (e−( H +1) , 1], yielding: Proposition 3.2 (Uniqueness of the Critical Surprisal Threshold). For any non-uniform, non-degenerate π, there exists a unique p∗ ∈ (e− H , 1) with s∗ ≜ − ln p∗ ∈ (0, H ) such that Φ( p∗ ) = 0 and when Φ( p) > 0 ⇐⇒ p > p∗ ⇐⇒ sa < s∗ . The proof is in Appendix E.3. The threshold s∗ partitions the vocabulary into a low-surprisal region (sa < s∗ ) and a high-surprisal region (sa > s∗ ). Substituting into Theorem 3.1 yields the four-quadrant structure. Corollary 3.3 (Four-Quadrant Decomposition). The sign of dH/dη is determined by (sign Â, 1[sa < s∗ ]): (i) reinforcing low-surprisal tokens ( Â > 0, sa < s∗ ) reduces entropy; (ii) reinforcing high-surprisal tokens ( Â > 0, sa > s∗ ) increases it; (iii) suppressing low-surprisal tokens ( Â < 0, sa < s∗ ) increases it; (iv) suppressing high-surprisal tokens ( Â < 0, sa > s∗ ) reduces it. Each GRPO step thus propagates four token-level entropy signals with opposing signs. The proof is in Appendix E.4. Asymmetric entropy contributions. Since rollouts are sampled from πθ , low-surprisal tokens are drawn more frequently than high-surprisal ones at each decoding position. Within positive-advantage trajectories, entropy-decreasing tokens (low surprisal, Φ > 0) therefore constitute the statistical majority. Because GRPO assigns a single trajectory-level Âi to all tokens, it cannot distinguish these opposing entropy effects: the reinforced low-surprisal majority systematically drives the distribution toward concentration, while the high-surprisal minority that could preserve diversity contributes limited entropy-increasing effects. An analogous asymmetry governs the negative-advantage subset, revealing a fundamental gradient-level mechanism underlying entropy collapse in GRPO. 3

Token-Level Entropy Analysis H↓(majority)

Â >0

Surprisal-Guided Partitioning Sort tokens by surprisal s (descending), then take top-P%.

H↑(minority)

top-P%

L (entropy-critical)

Â >0

positiveadvantage okens

Advantage Â

H↑(majority)

Â <0

low

H↓(minority)

L- (entropy-critical)

s* Surprisal s

top-P%

Â <0

high

negativeadvantage tokens

Four-Quadrant Decomposition

high s 4

low s 6

high s 4

…

H K

{Oi,t}~πθold

group-normalized Â batch entropy H

attenuate

Entropy Dynamics

Yes

<Htgt

L+ ω=W

W if(i,t)∈L+ ωi,t=

M if(i,t)∈L 1

E. Policy Update

amplify attenuate advantage-reweighted gradient step

otherwise

Policy Entropy Hk

C. Decision

Batch-internal Surprisal Quantile Proxy

D. Token Weight Assignment

B. Compute

amplify

low s

Closed-Loop Token-Level Reweighting A. Rollouts

L+ ω=W

STARE (stable) Htgt

GRPO (collapse)

ω = 1 (bypass reweighting)

training step k

Figure 2: Overview of STARE. Guided by a four-quadrant decomposition of token-level entropy dynamics (top-left) and a batch-internal surprisal-quantile proxy that identifies entropy-critical tokens (top-right), STARE applies target-entropy-gated advantage reweighting in GRPO (bottom-left), stabilizing policy entropy where vanilla GRPO collapses (bottom-right). 3.3

Batch-Level Entropy Decomposition and Near-Criticality

The preceding analysis establishes that the entropy-contribution asymmetry persists within both advantage subsets. A natural quantitative question arises: what reweighting of the entropy-increasing minority suffices to reverse the batch-level net entropy gradient? Theorem 3.4 (Entropy Neutrality Identity). For any conditional distribution π, Ea∼π[Φ( a)] = ∑v∈V πv Φ(πv ) = 0. The proof is in Appendix E.6. ∗ }. Scaling the effective Token-level advantage reweighting. Define L+ = {(i, t) : Âi > 0, si,t > si,t advantage of each token in L+ by a multiplicative factor W ≥ 1, while retaining unit weight at all remaining positions, yields:

Proposition 3.5 (Entropy Gradient under Token-Level Reweighting).

d H̄ 1 dη W = − N [ Λ − (W −1) Γ ], where

Λ ≜ ∑i,t Âi Φi,t , Γ ≜ ∑(i,t)∈L+ Âi |Φi,t | > 0, and the critical weight is W ∗ = 1 + Λ/Γ.

The proof is in Appendix E.7. Two structural properties jointly ensure that W ∗ remains near unity (nearcritical regime). First, the entropy neutrality identity guarantees that Λ is a weak residual arising solely from the statistical dependence between trajectory-level advantages and token-level entropy sensitivities; under the assumptions formalized in Appendix E.9, |Λ|/Σabs = O( T −1 ) where Σabs = ∑i,t | Âi | |Φi,t |. Second, high-surprisal tokens carry amplified entropy sensitivity (Appendix E.8): s ≥ H ⇒ |Φ( p)| ≥ S2 , whereas s < s∗ ⇒ |Φ( p)| ≤ H − S2 . Although L+ is a sampling minority, the amplified per-token |Φ| values ensure that Γ remains appreciable. Corollary 3.6 (Near-Criticality). When the sequence length T and the batch size are both sufficiently large, W ∗ − 1 = Λ/Γ = O( T −1 ) (Appendix E.11). So beyond the critical threshold, the specific value of W principally controls the magnitude rather than the sign of the per-step entropy shift. 3.4

Cross-Step Entropy Dynamics

Let ∆ H̄k = − N −1 [Λk − (W −1) Γk ] denote the batch-level entropy variation at step k. Under W = 1, the sampling asymmetry implies Λk > 0 in expectation, so ∆ H̄k < 0. The resulting entropy reduction further concentrates πθ , lowering the sampling frequency of high-surprisal tokens in subsequent batches, shrinking Γk+1 , and raising Wk∗+1 , forming a self-reinforcing loop of entropy collapse. Conversely, when 4

W > Wk∗ , entropy increases disperse the distribution, enlarge Γk+1 , and lower Wk∗+1 , forming a symmetric loop of entropy recovery (formalized in Appendix F). The near-criticality result (corollary 3.6) establishes that a modest token-level weight adjustment suffices to alter the macroscopic entropy trajectory. This theoretical insight directly motivates the algorithmic design proposed in the next section: by selectively modulating the effective advantage weights of a targeted subset of tokens within the GRPO policy gradient, one can restore a sustainable dynamic equilibrium between the entropy-increasing and entropydecreasing gradient forces.

Method

The theoretical analysis in Section 3 reveals that GRPO’s shared trajectory-level advantages induce a tokenlevel credit assignment mismatch: high-frequency low-surprisal tokens dominate gradient aggregation while sparse high-surprisal tokens with critical entropy effects are under-represented. Motivated by it, we propose Surprisal-Guided Token-Level Advantage Reweighting For Policy Entropy Stability (STARE): a surprisal-based credit rebalancing mechanism that operates within the clipped surrogate of GRPO, assigns differentiated weights to entropy-critical tokens, and incorporates target-entropy closed-loop feedback for stable entropy regulation on training, as shown in Figure 2. 4.1

Entropy-Critical Token Partitioning via High-Surprisal Quantiles

Let T + = {(i, t) : Âi > 0} and T − = {(i, t) : Âi < 0} denote the positive- and negative-advantage ∗ (Proposition 3.2) at every position requires the token sets. Computing the exact critical threshold si,t full conditional distribution, incurring prohibitive overhead. Instead, STARE employs a simple, stable, and theoretically motivated batch-internal surprisal-quantile proxy. Concretely, within T + and T − separately, tokens are ranked in descending order of surprisal si,t = − ln πθ (oi,t | xi , oi,<t ), and the top P% tokens are selected to form the entropy-critical sets below:

L± = {(i, t) ∈ T ± : si,t ≥ Q P({s j,s }( j,s)∈T ± )}. By Corollary 3.3, L+ approximately denotes the entropy-increasing tokens among positive-advantage responses, and L− indicates the entropy-decreasing tokens in negative-advantage responses. The fixed proportion P directly controls the intervention scale, obviating per-position threshold computation. 4.2

Advantage-Conditioned Token-Level Credit Rebalancing

We augment the GRPO objective with positive token-level weights ωi,t > 0: 1 JSTARE (θ ) = ω min ρ ( θ ) Â , clip ρ ( θ ) , 1 − ϵ, 1 + ϵ Â ( ) i,t i,t i i,t i . N∑ i,t Setting ωi,t ≡ 1 recovers STARE to standard GRPO. Because ωi,t > 0, STARE preserves all tokenlevel gradient directions: tokens with positive advantage remain reinforced, while those with negative advantage remain suppressed. STARE therefore acts as an advantage-conditioned credit-rebalancing mechanism, selectively rescaling relative magnitudes along the surprisal dimension. Variant I: One-Sided Entropy Amplification ( Âi > 0, High-Surprisal tokens, denoted as O1). W, (i, t) ∈ L+ , (V1) ωi,t = W > 1. 1, otherwise, Tokens in L+ simultaneously carry positive advantage and entropy-increasing effect; amplifying their weights directly strengthens the minority that GRPO systematically underweights. The resulting batchlevel net entropy shift is ΛV1 = Λ − (W −1) Γ+ , where Γ+ = ∑(i,t)∈L+ | Âi Φi,t | > 0. And under the near-criticality condition (Corollary 3.6), a moderate W > 1 usually suffices to reverse the sign of the batch net entropy shift. We provide the Algorithmic 1 pseudocode in the Appendix C. Variant II: Dual-Sided Entropy Regulation. Extending Variant I, this variant additionally attenuates the weights of tokens in L− ( Âi < 0, High-Surprisal tokens, denoted as C2):  + W, (i, t) ∈ L , (V2) ωi,t = M, (i, t) ∈ L− , W > 1, 0 < M < 1.  1, otherwise,

Tokens in L− reside in the high-surprisal tail of negative-advantage responses; and large negativeadvantage updates on these tokens redistribute mass from the tail toward the peak, exacerbating concentration. Attenuating their weights alleviates this entropy-decreasing pressure. Letting Γ− = ∑(i,t)∈L− | Âi Φi,t |, the batch net entropy shift becomes ΛV2 = Λ − (W −1) Γ+ − (1− M) Γ− . Two-sided regulation simultaneously amplifies the entropy-increasing signal and attenuates the entropydecreasing signal to adjust policy entropy. Definitions and analysis for all four single-polarity (O1, O2, O3, O4) and four combined operations (C1, C2, C3, C4) are deferred to Appendices G and H. 4.3

Closed-Loop Regulation via Target-Entropy Gating

A purely open-loop reweighting strategy risks overshooting from entropy collapse into uncontrolled divergence. To achieve stable regulation, we introduce a batch-level target entropy Htgt and employ the current batch mean entropy H̄k as a closed-loop feedback signal via a binary gate gk = 1[ H̄k < Htgt ] and express the weights in unified form below. When H̄k < Htgt , the gate activates and  + 1 + gk (W −1), (i, t) ∈ L , ωi,t = 1 − gk (1− M), (i, t) ∈ L− (two-sided only),  1, otherwise. STARE strengthens entropy-increasing signals; when H̄k ≥ Htgt , all weights revert to unity, automatically recovering standard GRPO. This drives entropy toward Htgt via bounded oscillation. Finer-grained sample-level and token-level closed-loop variants are presented in Appendix G.6. 4.4

Static and Adaptive Weighting Schedules

Fixed weights. Near-criticality (Corollary 3.6) implies that the required reweight perturbation is typically modest: beyond the critical point, the specific value principally controls the magnitude rather than the direction of the per-step entropy shift, reducing sensitivity to hyperparameter choices. Fixed W and M therefore suffice in most settings. Adaptive weights. As training progresses, intervention strength may vary across phases with distinct distributional. STARE also supports adaptive weight updates driven by the target-entropy signal: Wk+1 = clip Wk + α sgn Htgt − H̄k , [1, Wmax ] , Mk+1 = clip Mk − α sgn Htgt − H̄k , [ Mmin , 1] . When H̄k < Htgt , W increases and M decreases, intensifying the intervention; otherwise both relax toward GRPO. The constraints W ≥ 1 and M ≤ 1 ensure graceful degradation. Setting α = 0 recovers the fixed-weight. Default values α=0.01, Wmax =1.5, Mmin =0.5 yield robust performance. Default configuration. All main experiments adopt Variant I (O1) with batch-level target-entropy gating and fixed weights. This minimal configuration suffices to stabilize entropy and improve performance. Ablations on two-sided regulation and adaptive weights are provided in Appendix H.

Experiments

5.1

Experimental Setup

Models and scenarios. We systematically evaluate STARE across three scenarios. In the Short CoT scenario, we use Qwen2.5-Math-7B-Base with a maximum decoding length of 4k, Qwen2.5-14B-Instruct with 8k, and Qwen2.5-32B-Base with 8k(Yang et al., 2024; Qwen et al., 2025). In the Long CoT scenario, we employ DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base with 16k to elicit deep reasoning and self-reflection(Guo et al., 2025; Team, 2025). In the tool-use scenario, we first perform cold-start SFT on Qwen2.5-7B-Base using Retool 2K data(Feng et al., 2025), then conduct RL training with 8k length. Training. We use a learning rate of 1 × 10−6 and a batch size of 64 samples with 8 rollouts per sample, and on-policy updates with a single gradient step per batch. Decoding adopts Top-p = 1.0 and temperature T = 1.0. The STARE hyperparameters are set to W = 1.1, M = 0.9, Htgt = 0.3, and P% = 10%, with batch-level gating and fixed weights as the default configuration. The training corpus consists of 100k samples deduplicated and sampled from open-source RL datasets including DeepScaler, Skywork-o1, Polaris, and DAPO(Tan et al., 2026; He et al., 2025c; An et al., 2025; Yu et al., 2025). We construct an 6

enhanced GRPO baseline, denoted GRPOds, which removes the KL penalty and incorporates dynamic sampling with token-level loss(Yu et al., 2025). We report two variants: STARE-O1 amplifies only L+ q , whereas STARE-C2 additionally attenuates L− q over O1. Ablation studies adopt STARE-O1 as the default configuration. We append the instruction “Please reason step by step, and put your final answer within \boxed{}” to each question, and then extract the content enclosed in \boxed{} as the final answer for correctness evaluation. Evaluation. We evaluate on six mathematical benchmarks: AIME24, AIME25, AMC23, MATH-500, Minerva Math, and OlympiadBench(Yang et al., 2023; Hendrycks et al., 2021; Lewkowycz et al., 2022; He et al., 2024), with Top-p = 0.95 and T = 0.7 at inference. AIME24/25 and AMC23 are evaluated 32 times and the other benchmarks 4 times, and all results report average accuracy.

Table 1: Performance comparison of STARE and competitive RL algorithms on six math benchmarks across 1.5B–32B scales and three scenarios (Acc avg@N); † denotes results from prior works. Method

AIME24

AIME25

AMC

MATH

Minerva

Olympiad

Avg.

In the Short CoT Scenario W-REINF† (Zhu et al., 2025) GRPO† (Shao et al., 2024) EntroReg† (Cheng et al., 2025) DAPO† (Yu et al., 2025) 80/20 Rule† (Wang et al., 2025b) EntroAdv† (Cheng et al., 2025) STEER† (Hao et al., 2025) GRPO-ds KL-Cov† (Cui et al., 2025) EAPO† (He et al., 2026) STARE-O1 STARE-C2

RL from the Qwen2.5-Math-7B-Base 31.2 10.1 58.1 76.2 34.0 9.3 58.6 79.9 34.6 12.6 61.1 82.7 35.7 17.1 61.0 82.6 35.8 15.1 63.7 84.1 36.8 16.3 63.8 83.5 36.9 16.2 72.2 82.4 37.1 17.7 75.3 82.7 38.9 13.8 59.2 81.5 39.8 17.2 62.1 83.7 44.2 23.8 83.4 86.1 42.9 24.2 84.1 85.8

34.2 38.1 43.1 42.8 43.0 42.7 41.7 39.4 40.9 43.9 44.2 44.7

38.9 42.8 42.8 43.9 45.5 44.5 43.3 42.6 44.2 45.1 44.7 45.3

41.6 43.8 46.2 47.2 47.9 47.9 49.1 49.1 46.4 48.6 54.4 54.5

Base-14B† (Qwen et al., 2025) GRPO† (Shao et al., 2024) GRPO-Clipν † (Wang et al., 2026) GRPO-ds STARE-O1 STARE-C2

RL from the Qwen2.5-14B-Instruct 12.1 11.7 22.5 17.6 23.4 21.4 24.2 21.9 69.1 80.6 30.8 27.1 77.5 85.4 31.5 28.3 76.3 86.1

37.2 40.5 40.2

43.4 50.9 51.4

46.1 52.0 52.3

GRPO† (Shao et al., 2024) 80/20 Rule† (Wang et al., 2025b) GSPO† (Zheng et al., 2025) Lp-Reg† (Huang et al., 2025) DAPO† (Yu et al., 2025) GRPO-ds STARE-O1 STARE-C2

RL from the Qwen2.5-32B-Base 28.5 22.5 86.6 32.5 28.5 89.4 33.3 22.3 87.6 38.1 27.1 90.0 38.3 29.8 87.6 38.5 28.8 85.3 85.6 43.3 34.1 87.3 90.4 42.9 35.7 88.8 90.6

44.9 45.6 48.5 46.32 48.5 44.6 48.8 49.3

60.3 57.6 55.6 61.2 55.6 54.0 60.1 60.9

56.1 60.7 61.4

26.6 37.5 37.6 26.5 37.8 27.8 26.0 39.4 39.1 28.2 40.5 40.4 39.7 49.2 51.5 52.6 51.8

43.9 55.2 56.2 45.8 55.9 45.6 44.9 58.7 58.4 48.0 59.9 59.1 60.8 64.1 68.0 67.8 68.8

45.6 50.7 51.9 48.4 51.8 49.2 48.9 53.9 55.1 52.7 56.5 56.3 57.0 62.5 65.6 65.9 66.3

55.9 54.8 57.2 52.3 55.6 56.9

61.5 62.5 63.0 59.0 61.8 62.3

56.2 57.6 58.8 58.5 62.0 62.2

6.2 49.9 48.8 54.8 52.3 53.6

53.9 59.4 60.4

In the Long CoT Scenario

5.2

Main Results

Table 1 presents the full performance comparison, where results marked with † are cited from prior work such as EAPO, STEER, GRPOClipν , Lp-Reg, DGPO, JustRL, STAPO, A3PO, and SimpleTIR(He et al., 2026; Hao et al., 2025; Wang et al., 2026; Huang et al., 2025; Fu et al., 2026; He et al., 2025b; Liu et al., 2026; Tang et al., 2025; Xue et al., 2025). STARE consistently delivers substantial gains across the six math benchmarks at scales ranging from 1.5B to 32B and three scenarios. Short-CoT Scenario. At the 7B scale, STAREO1 attains an average accuracy of 54.4%, outperforming STEER (49.1%, +5.3%) and GRPOds (49.1%, +5.3%), while reaching 44.2% and 23.8% on AIME24 and AIME25, corresponding to improvements of roughly 10% and 7% over DAPO. At the 14B scale, STARE-O1 (52.0%) surpasses GRPO-ds (46.1%) by 5.9%. At the 32B scale, STARE-O1 (60.7%) exceeds GRPO-ds (56.1%) by 4.6%.

RL from the DeepSeek-R1-Distill-Qwen-1.5B 29.6 23.5 73.5 76.3 Base-1.5B† (Guo et al., 2025) 32.5 24.3 69.4 85.7 EntroReg† (Cheng et al., 2025) 34.6 26.4 70.2 86.8 CISPO† (Chen et al., 2025a) 34.8 25.8 76.9 76.8 W-REINF† (Zhu et al., 2025) 35.0 25.0 69.8 87.5 CE-GPPO† (Su et al., 2026) 35.1 27.7 82.5 76.7 ASPO† (Wang et al., 2025a) 36.4 28.3 83.1 74.6 GRPO† (Shao et al., 2024) 38.1 27.1 71.4 88.9 DAPO† (Yu et al., 2025) 40.9 28.6 73.6 89.9 DGPO† (Fu et al., 2026) 43.3 32.8 86.0 77.9 KL-Cov† (Cui et al., 2025) 43.9 30.1 75.0 90.0 EntroAdv† (Cheng et al., 2025) 44.0 30.4 73.9 90.2 EAPO† (He et al., 2026) 45.1 30.1 75.5 91.1 GRPO-ds 50.4 37.4 85.9 88.3 JustRL† (He et al., 2025b) 52.6 38.8 91.0 91.7 STARE-O1 53.8 41.5 89.5 90.4 STARE-C2 53.1 40.5 91.3 92.1 CE-GPPO† (Su et al., 2026)

GRPO† (Shao et al., 2024) 80/20 Rule† (Wang et al., 2025b) STAPO† (Liu et al., 2026) DAPO† (Yu et al., 2025) Lp-Reg† (Huang et al., 2025) A3PO† (Tang et al., 2025) GRPO-ds STARE-O1 STARE-C2

RL from the Qwen3-8B-Base 31.3 24.7 75.2 88.9 31.3 27.5 79.9 89.9 33.4 28.7 79.9 90.4 34.2 26.1 84.5 35.9 25.8 87.4 37.8 30.4 91.3 39.5 30.8 80.8 88.6 43.9 34.7 85.3 90.6 44.3 32.6 86.1 91.2 In the Tool-Use Agent Scenario

Base-7B-TIR† (Qwen et al., 2025) ToRL† (Wang et al., 2023) Effective TIR† (Bai et al., 2025) ZeroTIR† (Mai et al., 2025) GRPO-ds SimpleTIR† (Xue et al., 2025) STARE-O1 STARE-C2

RL from the Qwen2.5-7B-Base 1.7 0.6 10.8 18.0 40.2 27.9 75.0 82.2 42.3 29.2 74.2 86.4 46.7 30.0 85.2 46.8 32.4 75.9 81.4 50.5 30.9 79.1 88.4 53.2 37.5 84.9 86.8 52.8 38.1 86.9 87.2

38.3 41.9 43.7

Long-CoT Scenario. At the 1.5B scale, STARE-O1 reaches an average of 65.9%, substantially exceeding EAPO (57.0%, +8.9%) and DAPO (55.1%, +10.8%), while also outperforming JustRL on AIME24 and AIME25. At the 8B scale, STARE-O1 (62.0%) surpasses STAPO (58.8%) and GRPO-ds (59.0%), and STARE-C2 further raises the average to 62.2%. Tool-Use Scenario. STARE-O1 achieves an average of 59.4%, improving over GRPO-ds (53.9%) by 5.5% and outperforms SimpleTIR, while reaching 53.2% and 37.5% on AIME24 and AIME25. STARE-C2 further lifts the average to 60.4%. Key findings. Three principal observations emerge. (i) On the challenging AIME24 and AIME25 benchmarks, STARE improves over GRPO-ds by 4%–8% in average accuracy, with gains of 3%–6% across all six benchmarks. (ii) Across different thinking scenarios and model scales from 1.5B to 32B, STARE consistently outperforms the majority of competitive RL improvement methods, confirming its effectiveness and robustness. (iii) The additional gains of STARE-C2 further indicate that dualsided regulation, which simultaneously strengthens entropy-increasing signals and attenuates entropydecreasing ones, can yield an even more favorable exploration-exploitation balance.

0.4

0.2

0.0 0

1000 2000 3000 4000 5000

Training Steps

(a) Training Entropy

0.0

AIME25 Acc

AIME24 Acc

0.6

0.15

0.5

0.10

GRPO-ds STARE 1000 2000 3000 4000 5000

0.05 0

Training Steps

(e) AIME24 Acc

Training Steps

0.7

0.20

0.15

GRPO-ds STARE 1000 2000 3000 4000 5000

0.25

0.30

Training Steps

(b) Training Reward

0.45

0.00

0.2

GRPO-ds STARE 1000 2000 3000 4000 5000

AIME24 Pass@32

0.0

0.4 0

Training Steps

(f) AIME25 Acc

GRPO-ds STARE 1000 2000 3000 4000 5000

Training Response

0.6

2000 1800 1600 1400 1200 1000 800 0

GRPO-ds STARE

1000 2000 3000 4000 5000

Training Steps

(d) Train Response Length 0.45 0.40 0.35 0.30 0.25 0.20

AIME25 Pass@32

Training Reward

0.8

Entropy

0.4

0.8

Training Full-Solve Ratio

GRPO-ds STARE

Training Steps

(g) AIME24 Pass@32

GRPO-ds STARE 1000 2000 3000 4000 5000

Training Steps

(h) AIME25 Pass@32

Figure 3: Comparison of key training metrics between STARE and GRPO-ds on Qwen2.5-Math-7B-Base model in the Short CoT scenario over 5k RL steps. GRPO-ds STARE: W=1.01 STARE: W=1.05

STARE: W=1.1 STARE: W=1.2 STARE: W=1.5

STARE: W=2.0 STARE: W=3.0 STARE: W=4.0

STARE: W=1.1 STARE: W=1.2 STARE: W=1.5

STARE: W=2.0 STARE: W=3.0 STARE: W=4.0

2 0

200

400

600

Training Steps

800

1000

0.6 0.5 0.4 0.3 0.2 0.1 0.0

GRPO-ds STARE: Top 5% STARE: Top 10%

STARE: Top 15% STARE: Top 20% STARE: Top 25%

STARE: Top 30% STARE: Top 35% STARE: Top 40%

Entropy

GRPO-ds STARE: W=1.01 STARE: W=1.05

Entropy

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

200

400

600

Training Steps

800

1000

200

400

600

Training Steps

800

1000

(a) Entropy evolution under varying (b) Entropy evolution under varying (c) Entropy evolution under varying W without target-entropy gating W with target-entropy gating Top-P ratio (W : 1.1, Htgt : 0.3)

Figure 4: STARE policy entropy evolution under ablation on the varying reweighting factor W, targetentropy gating, and the high-surprisal selection ratio P on Qwen2.5-Math-7B-Base. 5.3

Cognitive Analysis

STARE vs. GRPO-ds: Training Dynamics across Scales and Scenarios. To validate STARE in longhorizon RL, we run 5000 training steps on Qwen2.5-Math-7B-Base under the Short CoT scenario. Figure 3 compares STARE with GRPO-ds on key metrics, and Figures 5–1 further verify its effectiveness across 1.5B–32B scales and diverse task scenarios. Entropy stability and performance evolution. GRPO-ds exhibits entropy collapse over steps 0–1000, with policy entropy approaching zero (Figure 3(a)), consistent with Section 3; correspondingly, its AIME24/25 accuracy peaks around step 1000 and saturates thereafter (Figure 3(e)-(f)), indicating premature convergence. In contrast, STARE stabilizes entropy near Htgt = 0.3 via token-level reweighting and closed-loop gating, with accuracy continuing to rise beyond step 1000 and peaking at 5000, thereby unlocking long-horizon RL potential. Exploration-exploitation balance. STARE’s Pass@32 consistently exceeds GRPO-ds throughout training (Figure 3(g)-(h)), preserving output diversity and mitigating mode-seeking; whereas GRPO-ds’s reward and Full-Solve Ratio plateau early (Figure 3(b)-(c)), STARE keeps growing, and its sustained response-length increase (Figure 3(d)) reflects deeper reasoning. Cross-scale and cross-scenario generalization. Figures 5-1 reveal a consistent pattern across Short CoT (14B, 32B), Long CoT (R1-Distill-Qwen-1.5B, Qwen3-8B-Base), and Tool-Use (7B): GRPOds suffers collapse with performance saturation, while STARE maintains stable entropy and continuous accuracy gains. Notably, on DeepSeek-R1-Distill-Qwen-1.5B, GRPO-ds’s entropy decays below 0.2 by step 5000, whereas STARE rapidly restores entropy to the target band starting around step 3500, with AIME24/25 accuracy improving in tandem (Figure 7), showing its intervention-recovery capability and robustness across scales and scenarios. Further details are provided in Appendix B.1 Ablation on Key Hyperparameters and Target-Entropy Gating. We ablate W, P, and the target-entropy gate on Qwen2.5-Math-7B-Base (Figure 4). Under open-loop reweighting ( without the target-entropy gate; Figure 4(a)), W = 1.01 already mitigates the entropy decay of GRPO-ds, W ≥ 1.05 yields steady growth, and W ≥ 2.0 triggers divergenc, corroborating the near-criticality property (Corollary 3.6) that beyond the critical threshold W controls magnitude rather than direction. Open-loop reweighting, however, stabilizes entropy at an excessively high level, inducing over-exploration that hampers overall training(Appendix B.5). With the closed-loop gate (Htgt = 0.3), Figure 4(b) shows that all W ∈ [1.05, 1.5] steer entropy into the target band with bounded oscillation, confirming closed-loop stability and

0.30

AIME24 Acc

0.24

0.25

0.20

0.16

0.15

500

1000

1500

Training Steps

0.10 0

(a) Training Entropy

AIME25 Acc

GRPO-ds STARE

Entropy

0.5 0.4 0.3 0.2 0.1 0.0

500

1000

Training Steps

GRPO-ds STARE 1500

0.12 0.08

(b) AIME24 Acc

500

1000

Training Steps

GRPO-ds STARE 1500

Figure 5: Training dynamics of STARE vs. GRPO-ds on Qwen2.5-14B-Instruct in the Short CoT scenario: policy entropy, AIME24 accuracy, and AIME25 accuracy.

Entropy

0.3

0.4

0.2

0.2 0.0

0.3

0.1

500

1000

1500

Training Steps

AIME25 Acc

0.6

0.4

AIME24 Acc

GRPO-ds STARE

0.8

(a) Training Entropy

500

1000

Training Steps

GRPO-ds STARE 1500

0.1 0

(b) AIME24 Acc

500

1000

Training Steps

GRPO-ds STARE 1500

Figure 6: Training dynamics of STARE vs. GRPO-ds on Qwen2.5-32B-Base in the Short CoT scenario: policy entropy, AIME24 accuracy, and AIME25 accuracy.

Entropy

0.8 0.6

0.40

0.5

0.35

0.4

0.30

0.3

0.2 0

1000

2000

3000

Training Steps

4000

5000

AIME25 Acc

GRPO-ds STARE

AIME24 Acc

1.0

(a) Training Entropy

1000

2000

3000

Training Steps

GRPO-ds STARE 4000 5000

0.25 0.20 0

(b) AIME24 Acc

1000

2000

3000

Training Steps

GRPO-ds STARE 4000 5000

Figure 7: Training dynamics of STARE vs. GRPO-ds on DeepSeek-R1-Distill-Qwen-1.5B in the Long CoT scenario: policy entropy, AIME24 accuracy, and AIME25 accuracy.

0.3 0.2

200

400

600

Training Steps

800

(a) Training Entropy

1000

0.1 0

200

400

600

Training Steps

(b) AIME24 Acc

GRPO-ds STARE 800 1000

0.35 0.30 0.25 0.20 0.15 0.10

AIME25 Acc

0.4

AIME24 Acc

GRPO-ds STARE

Entropy

1.0 0.8 0.6 0.4 0.2 0.0

200

400

600

Training Steps

GRPO-ds STARE 800 1000

Figure 8: Training dynamics of STARE vs. GRPO-ds on Qwen3-8B-Base in the Long-CoT scenario: policy entropy, AIME24 accuracy, and AIME25 accuracy. substantially reducing sensitivity to W. Fixing W = 1.1 and Htgt = 0.3, Figure 4 (c) further shows that entropy stays within the target band for P ∈ [5%, 20%] and remains confined to [0.1, 0.2] even at P = 40%, effectively preventing collapse and indicating a broad operating range. Overall, STARE is robust to W and P, and the target-entropy gate stably constrains the policy entropy. More detailed entropy evolution and performance comparisons are reported in Appendix B.5 ( Table 3, Figure 12), with additional analysis provided in Appendix B.4.

0.6

GRPO-ds STARE-O1

STARE-O2 STARE-O3

0.6

STARE-O4

STARE-C2 STARE-C3

1000

2000

STARE-C4

0.4

Entropy

0.4

0.2

0.2 0.0

GRPO-ds STARE-C1

1000

2000

3000

Training Steps

0.0

4000

(a) Entropy evolution under single-polarity operations O1-O4, each targeting one entropy-critical quadrant.

3000

Training Steps

4000

(b) Entropy curves under combined operations C1-C4, each jointly regulating two entropy-critical quadrants.

Figure 9: Policy entropy trajectories of all eight STARE variants versus GRPO-ds on Qwen2.5-Math-7BBase over 4000 RL training steps, covering four single-polarity operations (O1-O4) and four combined operations (C1-C4) derived from the advantage-surprisal four-quadrant decomposition. Validation of the High-Surprisal Quantile Proxy. Figure 11 validates the batch-internal top-P% surprisalquantile proxy. Panel (a) shows that the fraction of selected tokens falling within the theoretical entropy∗ ) rises steadily from ∼60% to ∼95% during training, indicating strong increasing region (si,t > si,t consistency with the critical threshold s∗ (Proposition 3.2); panel (b) further confirms that the cumulative net entropy contribution of L+ q remains positive and monotonically increasing, consistent with Corollary 3.3. Thus the proxy reliably identifies entropy-critical tokens while obviating per-position solutions of Φ( p∗ ) = 0; see Appendix B.2 for further details. Effects of Single-Polarity and Combined Operations in Table 2: The results of single-polarity STARE. To validate the token-level reweighting mechanism and combined STARE operations on under the four-quadrant decomposition, we ablate four single- AIME24 and AIME25. polarity (O1–O4) and four combined (C1–C4) operations, where O1/O3 amplify entropy-increasing signals, O2/O4 attenModel AIME24 AIME25 uate entropy-decreasing ones, and C1–C4 jointly intervene on GRPO-ds 37.1 17.7 two quadrants. As shown in Table 2 and Figure 9, while GRPOSTARE-O1 44.2 23.8 ds suffers rapid entropy decay, all eight variants effectively STARE-O2 40.5 20.3 mitigate this collapse and substantially outperform GRPO-ds STARE-O3 39.6 21.6 AIME24/25 across all four entropy-critical quadrants. Among STARE-O4 42.1 19.9 + + them, O1 (amplifying Lq ) and C2 (amplifying Lq while attenSTARE-C1 43.1 23.5 uating L− q ) perform best at 44.2%/23.8% and 42.5%/24.2%, so STARE-C2 42.5 24.2 we adopt STARE-O1 as the default configuration and STARESTARE-C3 39.9 20.8 STARE-C4 41.7 22.6 C2 as the dual-sided variant. Further details are provided in Appendix B.3. Emergent Reflection Behaviors. To examine how STARE elicits deep reasoning, we inspect the tokens selected for advantage amplification during RL training. As exemplified on Qwen2.5-32B-Base, the word cloud in Figure 10 shows reweighted tokens concentrating on uncertainty and selfcorrection vocabulary, such as should be, but, instead, and verification, confirming that the batch-internal surprisal-quantile proxy effectively identifies rare forking tokens (Wang et al., 2025b) with exploratory semantics. A complementary count across six reflection categories (Figure 13) further shows STARE markedly surpasses GRPO-ds with the largest margins on reflection and self-correction, jointly demonstrating Figure 10: Word cloud of tokens selected that STARE activates deep exploration and delivers consis- by STARE for advantage reweighting. tent gains through token-level credit rebalancing. The complete analysis is deferred to Appendix B.6. Further ablations are deferred to the appendices: fixed vs. adaptive weighting( B.7), target-entropy threshold( B.8), gating granularity( B.9), off-policy training( B.10), and fixed-threshold reweighting( B.11).

Conclusion

We present STARE, a surprisal-guided token-level advantage reweighting mechanism that mitigates policy entropy collapse in GRPO-style RLVR. A first-order analysis of entropy dynamics reveals an advantage–surprisal four-quadrant structure and a near-criticality property. Guided by this insight, STARE reweights entropy-critical tokens via batch-internal surprisal quantiles, coupled with a targetentropy closed-loop gate for stable, minimally intrusive regulation. Across 1.5B–32B models and three task scenarios, STARE sustains thousands of stable RL steps and outperforms DAPO and other competitive baselines by 4%–8% on AIME24/25, offering a principled foundation for entropy-aware credit assignment in long-horizon RL post-training of LLMs.

References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, et al. Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. URL https://hkunlp. github. io/blog/2025/Polaris, 2025. AI Anthropic. System card: Claude opus 4 & claude sonnet 4. Claude-4 Model Card, 2025. Fei Bai, Yingqian Min, Beichen Zhang, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Towards effective code-integrated reasoning, 2025. URL https://arxiv.org/ abs/2505.24480. Edward Y. Chang, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms. CoRR, abs/2502.03373, 2025. doi: 10.48550/ARXIV.2502.03373. URL https://doi.org/10.48550/arXiv.2502.03373. Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025a. Kun Chen, Peng Shi, Fanfan Liu, Haibo Qiu, Zhixiong Zeng, Siqi Yang, and Wenji Mao. Flexible entropy control in rlvr with gradient-preserving perspective. arXiv preprint arXiv:2602.09782, 2026. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do NOT think that much for 2+3=? on the overthinking of long reasoning models. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net, 2025b. URL https://openreview.net/forum?id=MSbU3L7V00. Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. CoRR, abs/2506.14758, 2025. doi: 10.48550/ ARXIV.2506.14758. URL https://doi.org/10.48550/arXiv.2506.14758. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models. CoRR, abs/2505.22617, 2025. doi: 10.48550/ARXIV.2505.22617. URL https://doi.org/10.48550/arXiv. 2505.22617. Wenlong Deng, Yi Ren, Muchen Li, Danica J. Sutherland, Xiaoxiao Li, and Christos Thrampoulidis. On the effect of negative gradient in group relative deep reinforcement optimization. CoRR, abs/2505.18830, 2025. doi: 10.48550/ARXIV.2505.18830. URL https://doi.org/10.48550/arXiv.2505.18830. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nat., 630(8017):625–630, 2024. doi: 10.1038/S41586-024-07421-0. URL https://doi.org/10.1038/s41586-024-07421-0. Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms, 2025. URL https://arxiv.org/abs/2504.11536.

Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Chaowen Hu, Cong Qin, Zekai Shao, Binbin Zheng, Lu Pan, and Ke Zeng. From log π to π: Taming divergence in soft clipping via bilateral decoupled decay of probability gradient weight, 2026. URL https://arxiv.org/abs/2603.14389. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pp. 10835–10866, 2023. URL https://proceedings.mlr.press/v202/gao23h.html. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pp. 1856–1865, 2018. URL http://proceedings.mlr.press/v80/haarnoja18b.html. Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, and Jiawei Chen. Rethinking entropy interventions in rlvr: An entropy change perspective. arXiv preprint arXiv:2510.10150, 2025. Andre Wang He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 25559–25571, 2025a. Bingxiang He, Zekai Qu, Zeyuan Liu, Yinghao Chen, Yuxin Zuo, Cheng Qian, Kaiyan Zhang, Weize Chen, Chaojun Xiao, Ganqu Cui, Ning Ding, and Zhiyuan Liu. Justrl: Scaling a 1.5b llm with a simple rl recipe, 2025b. URL https://arxiv.org/abs/2512.16649. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 3828–3850, 2024. doi: 10.18653/V1/2024.ACL-LONG.211. URL https://doi.org/10.18653/v1/2024.acl-long.211. Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312, 2025c. Yuhang He, Haodong Wu, Siyi Liu, Hongyu Ge, Hange Zhou, Keyi Wu, Zhuo Zheng, Qihong Lin, Zixin Zhong, and Yongqi Zhang. Rethinking token-level credit assignment in rlvr: A polarity-entropy analysis. arXiv preprint arXiv:2604.11056, 2026. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. Guanhua Huang, Tingqiang Xu, Mingze Wang, Qi Yi, Xue Gong, Siheng Li, Ruibin Xiong, Kejiao Li, Yuhao Jiang, and Bo Zhou. Low-probability tokens sustain exploration in reinforcement learning with verifiable reward. arXiv preprint arXiv:2510.03222, 2025. Renren Jin, Pengzhi Gao, Yuqi Ren, Zhuowen Han, Tongxuan Zhang, Wuwei Huang, Wei Liu, Jian Luan, and Deyi Xiong. Revisiting entropy in reinforcement learning for large reasoning models. arXiv preprint arXiv:2511.05993, 2025. Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron C. Courville, and Nicolas Le Roux. Vineppo: Refining credit assignment in RL training of llms. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, 2025. URL https://openreview.net/forum?id=Myx2kJFzAn. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 18abbeef8cfe9203fdf9053c9c4fe191-Abstract-Conference.html. 12

Shiqi Liu, Zeyu He, Guojian Zhan, Letian Tao, Zhilong Zheng, Jiang Wu, Yinuo Wang, Yang Guan, Kehua Sheng, Bo Zhang, et al. Stapo: Stabilizing reinforcement learning for llms by silencing rare spurious tokens. arXiv preprint arXiv:2602.15620, 2026. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jianguang Lou, Shifeng Chen, Yansong Tang, and Weizhu Chen. Arena learning: Build data flywheel for llms post-training via simulated chatbot arena. arXiv preprint arXiv:2407.10627, 2024a. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jianguang Lou, Shifeng Chen, Yansong Tang, and Weizhu Chen. Wizardarena: Post-training large language models via simulated offline chatbot arena. Advances in Neural Information Processing Systems, 37:111544–111570, 2024b. Haipeng Luo, Huawen Feng, Qingfeng Sun, Can Xu, Kai Zheng, Yufei Wang, Tao Yang, Han Hu, and Yansong Tang. Agentmath: Empowering mathematical reasoning for large language models via tool-augmented agent. arXiv preprint arXiv:2512.20745, 2025. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evolinstruct. In International Conference on Learning Representations, volume 2024, pp. 27168–27188, 2024c. Xinji Mai, Haotian Xu, Zhong-Zhi Li, Weinong Wang, Jian Hu, Yingying Zhang, Wenqiang Zhang, et al. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving. arXiv preprint arXiv:2505.07773, 2025. Byung-Doh Oh and William Schuler. Transformer-based language model surprisal predicts human reading times best with about two billion training tokens. In Findings of the association for computational linguistics: EMNLP 2023, pp. 1915–1921, 2023. Byung-Doh Oh, Shisen Yue, and William Schuler. Frequency explains the inverse correlation of large language models’ size, training data amount, and surprisal’s fit to reading times, 2024. URL https: //arxiv.org/abs/2402.02255. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27 (3):379–423, 1948. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. Qiannan Shen and Jing Zhang. Ai-enhanced disaster risk prediction with explainable shap analysis: A multi-class classification approach using xgboost. In 2026 5th International Symposium on Computer Applications and Information Technology (ISCAIT), pp. 692–698. IEEE, 2026. Nathaniel J Smith and Roger Levy. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319, 2013. Zhenpeng Su, Leiyu Pan, Minxuan Lv, Yuntao Li, Wenping Hu, Fuzheng Zhang, Kun Gai, and Guorui Zhou. Ce-gppo: Coordinating entropy via gradient-preserving clipping policy optimization in reinforcement learning, 2026. URL https://arxiv.org/abs/2509.20712. Sijun Tan, Michael Luo, Justin Wong, Colin Cai, Xiaoxiang Shi, William Yuan Tang, Manan Roongta, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Effective RL scaling of reasoning models via iterative context lengthening, 2026. URL https://openreview.net/forum?id= I6GzDCne7U. 13

Xinyu Tang, Yuliang Zhan, Zhixun Li, Wayne Xin Zhao, Zhenduo Zhang, Zujie Wen, Zhiqiang Zhang, and Jun Zhou. Rethinking sample polarity in reinforcement learning with verifiable rewards. arXiv preprint arXiv:2512.21625, 2025. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with llms, 2025a. URL https://arxiv.org/abs/2501.12599. Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Tencent Hunyuan Team, Ao Liu, Botong Zhou, Can Xu, Chayse Zhou, ChenChen Zhang, Chengcheng Xu, Chenhao Wang, Decheng Wu, Dengpeng Wu, et al. Hunyuan-turbos: Advancing large language models through mamba-transformer synergy and adaptive chain-of-thought. arXiv preprint arXiv:2505.15431, 2025b. Jiakang Wang, Runze Liu, Lei Lin, Wenping Hu, Xiu Li, Fuzheng Zhang, Guorui Zhou, and Kun Gai. Aspo: Asymmetric importance sampling policy optimization, 2025a. URL https://arxiv.org/abs/ 2510.06062. Kezhou Wang, Ruijie Wu, Qinlin Zeng, Huao Lu, Hanye Wu, Qingfeng Cui, Haichao Lin, Yujia Liu, Xiaoyan Huang, Qingpeng Guo, Songtao Jian, Kaiyuan Lu, Shiyu Li, Hao Tian, Yongqin Sun, Xue Yang, Libin Song, Zejun Ou, and Guoqing Wang. ToRL: Scaling tool-integrated RL for LLMs. arXiv preprint arXiv:2312.10372, 2023. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. CoRR, abs/2506.01939, 2025b. doi: 10.48550/ARXIV.2506. 01939. URL https://doi.org/10.48550/arXiv.2506.01939. Shumin Wang, Yuexiang Xie, Wenhao Zhang, Yuchang Sun, Yanxi Chen, Yaliang Li, and Yanyong Zhang. On the entropy dynamics in reinforcement fine-tuning of large language models. arXiv preprint arXiv:2602.03392, 2026. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, et al. Bapo: Stabilizing off-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping. arXiv preprint arXiv:2510.18927, 2025. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In International Conference on Learning Representations, volume 2024, pp. 30745–30766, 2024. Wujiang Xu, Wentian Zhao, Zhenting Wang, Yu-Jhe Li, Can Jin, Mingyu Jin, Kai Mei, Kun Wang, and Dimitris N. Metaxas. EPO: entropy-regularized policy optimization for LLM agents reinforcement learning. CoRR, abs/2509.22576, 2025. doi: 10.48550/ARXIV.2509.22576. URL https://doi.org/10. 48550/arXiv.2509.22576. Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Simpletir: Endto-end reinforcement learning for multi-turn tool-integrated reasoning. arXiv preprint arXiv:2509.02479, 2025.

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via selfimprovement, 2024. URL https://arxiv.org/abs/2409.12122. Kai Yang, Xin Xu, Yangkun Chen, Weijie Liu, Jiafei Lyu, Zichuan Lin, Deheng Ye, and Saiyong Yang. Entropic: Towards stable long-term training of llms via entropy stabilization with proportional-integral control. arXiv preprint arXiv:2511.15248, 2025a. Tao Yang, Xiaopu Zhang, Junjie Xiao, Jianzhun Qian, Ning Bian, Jingwei Jia, Boyuan Xie, Huy Manh, Tianyi Zhou, Hongchang Zheng, Zihang Song, Yongqi Yang, Kaihang Liu, Wenhu Huang, Jinxi Guo, Zhilin Xu, Jie Liu, Hao Du, Zhou Zhang, Yicheng Zheng, Zheng Haotian, Xintao Wei, Hua Wu, Qi Liu, and Dong Zhou. AIMO-2 winning solution: Building state-of-the-art mathematical reasoning models with OpenMathReasoning dataset. arXiv preprint arXiv:2307.14047, 2023. Zhihe Yang, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yunjian Xu. Do not let low-probability tokens over-dominate in RL for llms. CoRR, abs/2505.12929, 2025b. doi: 10.48550/ARXIV.2505.12929. URL https://doi.org/10.48550/arXiv.2505.12929. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale, 2025. URL https://arxiv.org/abs/2503.14476. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? CoRR, abs/2504.13837, 2025a. doi: 10.48550/ARXIV.2504.13837. URL https://doi.org/10.48550/ arXiv.2504.13837. Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025b. Wenhao Zeng, Yaoning Wang, Chao Hu, Yuling Shi, Chengcheng Wan, Hongyu Zhang, and Xiaodong Gu. Pruning the unsurprising: Efficient llm reasoning via first-token surprisal, 2026. URL https: //arxiv.org/abs/2508.05988. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. Yuhang Zhou, Kai Zheng, Qiguang Chen, Mengkang Hu, Qingfeng Sun, Can Xu, and Jingjing Chen. Offseeker: Online reinforcement learning is not all you need for deep research agents. arXiv preprint arXiv:2601.18467, 2026. Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in llm reasoning. ArXiv, abs/2506.01347, 2025. URL https: //api.semanticscholar.org/CorpusID:279075301.

Appendix Appendix Contents A Related Work

B Additional Experiments and Analysis on STARE

B.1 STARE vs. GRPO-ds: Detailed Comparison across Diverse Scenarios and Model Scales in RL Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.2 Validation of the High-Surprisal Quantile Proxy. . . . . . . . . . . . . . . . . . . . . . . . .

B.3 Detailed Effects of Single-Polarity and Combined Operations in STARE. . . . . . . . . . .

B.4 Detailed Ablation on Key Hyperparameters and Target-Entropy Gating . . . . . . . . . .

B.5 Effectiveness of Target-Entropy Closed-Loop Gating . . . . . . . . . . . . . . . . . . . . . .

B.6 Details about Emergent Reflection Behaviors. . . . . . . . . . . . . . . . . . . . . . . . . . .

B.7 Fixed vs. Adaptive Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.8 Ablation on Target Entropy Threshold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.9 Ablation on Target-Entropy Gate Granularity . . . . . . . . . . . . . . . . . . . . . . . . . .

B.10 Validation of STARE under Off-Policy Training . . . . . . . . . . . . . . . . . . . . . . . . .

B.11 STARE vs. Fixed-Threshold Low-Probability Token Reweighting . . . . . . . . . . . . . .

C Algorithm: Main STARE Procedure

D Basic Derivations for Sections 2 and 3.1

D.1 Softmax Jacobian Derivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.2 Token-Level Logit Update in the Unclipped GRPO Regime . . . . . . . . . . . . . . . . . .

D.3 Lemma 2.1 (Entropy Gradient with Respect to Logits: Surprisal-Deviation Form) . . . . .

D.4 Theorem 3.1 (Token-Level Entropy Variation) . . . . . . . . . . . . . . . . . . . . . . . . . .

E Complete Proofs for Sections 3.2 and 3.3 and Near-Criticality Analysis

E.1 Positivity of S2 under Non-Uniform Distributions . . . . . . . . . . . . . . . . . . . . . . .

E.2 H > S2 under Non-Degenerate Distributions . . . . . . . . . . . . . . . . . . . . . . . . . .

E.3 Proposition 3.2 (Uniqueness of the Critical Surprisal Threshold) . . . . . . . . . . . . . . .

E.4 Corollary 3.3 (Four-Quadrant Decomposition) . . . . . . . . . . . . . . . . . . . . . . . . .

E.5 Asymmetric Entropy Contributions under Shared Trajectory-Level Advantages . . . . . .

E.6 Theorem 3.4 (Entropy Neutrality Identity) . . . . . . . . . . . . . . . . . . . . . . . . . . . .

E.7 Proposition 3.5 (Entropy Gradient under Token-Level Reweighting) . . . . . . . . . . . . .

E.8 Two Auxiliary Bounds: Lower Bound in the High-Surprisal Region and Upper Bound in the Low-Surprisal Region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

E.9 Statistical Assumptions for the Near-Criticality Analysis . . . . . . . . . . . . . . . . . . .

E.10 Λ Is a Weak Residual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

E.11 Theorem 3.6 (Near-Criticality) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

F Formalization of the Cross-Step Entropy Dynamics in Section 3.4

F.1

Mean-Field Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

F.2

Sign of the One-Step Batch Entropy Change under a Fixed Weight . . . . . . . . . . . . . .

F.3

Open-Loop Self-Reinforcing Entropy Collapse and Recovery . . . . . . . . . . . . . . . . .

F.4

Local Stability of Batch-Level Target-Entropy Gating . . . . . . . . . . . . . . . . . . . . . .

G Single-Polarity Operations and Finer-Grained Closed-Loop Extensions G.1 Definition of Surprisal-Quantile Proxy Sets . . . . . . . . . . . . . . . . . . . . . . . . . . .

G.2 Position-Level Logit Update under STARE Reweighting . . . . . . . . . . . . . . . . . . . .

G.3 Exact Batch-Level Entropy Variation under Reweighting an Arbitrary Token Subset . . .

G.4 Four Single-Polarity Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

G.5 Exact Entropy Variation under Unified Closed-Loop Gating . . . . . . . . . . . . . . . . .

G.6 Unified Instantiations of Three Closed-Loop Granularities . . . . . . . . . . . . . . . . . .

H Combined Reweighting Operations and Adaptive Weights

H.1 Exact Entropy Variation for Arbitrary Two-Subset Reweighting . . . . . . . . . . . . . . .

H.2 Four Representative Combined Reweighting Operations . . . . . . . . . . . . . . . . . . .

H.3 Summary of Single-Polarity and Combined Operations . . . . . . . . . . . . . . . . . . . .

Limitations and Broader Impacts

Appendix Overview The appendices are organized as follows. Appendix A reviews related work on reinforcement learning with verifiable rewards, existing entropy-collapse mitigation strategies, and emerging token-level perspectives, situating STARE within these literatures and clarifying its conceptual and methodological distinctions from prior approaches. Appendix B reports additional empirical results that complement the main paper, including a detailed comparison between STARE and GRPO-ds across diverse scenarios and model scales from 1.5B to 32B (Appendix B.1), an empirical validation of the batch-internal high-surprisal quantile proxy against the theoretical critical threshold (Appendix B.2), ablations of all four single-polarity and four combined token-level reweighting operations (Appendix B.3), ablations on the key hyperparameters W and P together with the target-entropy gate (Appendices B.4 and B.5), an analysis of emergent reflection behaviors elicited by STARE (Appendix B.6), a comparison between fixed and adaptive reweighting schedules (Appendix B.7), ablations on the target-entropy threshold and on the gate granularity (Appendices B.8 and B.9), a validation of STARE under off-policy training (Appendix B.10), and a comparison with fixed-threshold low-probability token reweighting (Appendix B.11). Appendix C presents the complete pseudocode of the default STARE-O1 procedure, integrating surprisalguided entropy-critical token selection, fixed-weight token-level advantage reweighting, and batch-level closed-loop target-entropy gating into a unified algorithm. Appendix D establishes the basic differential identities used in Sections 2 and 3.1 and proves Theorem 3.1. Appendix E proves Proposition 3.2, Corollary 3.3, Theorem 3.4, Proposition 3.5, and Theorem 3.6, provides the detailed asymmetric entropy contribution analysis, and states explicitly the statistical assumptions underlying the near-criticality result. Appendix F formalizes the cross-step entropy dynamics discussed in Section 3.4. Appendix G presents all single-polarity operations referenced in the main text, together with sample-level and token-level closed-loop extensions. Appendix H presents all combined operations and the adaptive weighting scheme. Finally, Appendix I discusses the limitations of our study and its broader societal impacts. To keep the notation unambiguous, Appendices D–F use the exact theoretical sets from Section 3, ∗ Le+ ≜ {(i, t) : Âi > 0, si,t > si,t },

∗ Le− ≜ {(i, t) : Âi < 0, si,t > si,t },

namely, the sets of token positions whose trajectories have positive (resp. negative) advantage and whose token surprisal exceeds the position-dependent critical threshold. By contrast, Appendices G and H use − the quantile-based proxy sets employed in the practical implementation of Section 4, namely L+ q and Lq .

Related Work

Reinforcement Learning with Verifiable Rewards(RLVR). In recent years, large language models (LLMs) have advanced rapidly(Achiam et al., 2023; Anthropic, 2025; Guo et al., 2025; Team et al., 2025b; Xu et al., 2024; Luo et al., 2024c). RLVR has emerged as the dominant post-training paradigm for enhancing the reasoning capability of LLMs, as it leverages verifiable signals to provide precise outcome-level rewards while avoiding the overfitting risks inherent in learned reward models(Schulman et al., 2017; Kazemnejad et al., 2025; Gao et al., 2023). GRPO removes the value network and estimates the advantage through group-normalized rewards, demonstrating strong effectiveness in mathematical reasoning, code generation, and tool-use tasks, and further eliciting emergent behaviors such as long chain-ofthought reasoning and self-reflection(Guo et al., 2025; Shao et al., 2024; Luo et al., 2023; Team et al., 2025b; Wei et al., 2022; Luo et al., 2024a; Chang et al., 2025; Luo et al., 2024b; Shen & Zhang, 2026; Luo et al., 2025). Subsequent studies extend the GRPO framework along several dimensions, including advantage estimation, loss aggregation, and sampling strategies, among which DAPO has become a representative baseline through a combination of asymmetric clipping, dynamic sampling, and tokenlevel loss normalization(Yu et al., 2025; Yue et al., 2025b; Cheng et al., 2025; He et al., 2025c; Chen et al., 2026). As training proceeds over more optimization steps, however, GRPO-style algorithms commonly suffer from policy entropy collapse, in which the entropy decays rapidly, the output diversity vanishes, within-group rollouts become homogeneous, and the number of trainable steps is ultimately capped(Farquhar et al., 2024; Jin et al., 2025; Chen et al., 2025b; Yue et al., 2025a). Existing studies have confirmed the prevalence of this phenomenon and have established empirical correlations between policy entropy and downstream performance; nevertheless, a fine-grained theoretical characterization of the underlying token-level gradient causes of entropy collapse remains absent. Entropy Collapse Mitigation Methods.Existing mitigation strategies fall into three categories. The first category protects low-probability tokens by adjusting the clipping thresholds of the importancesampling ratio, as exemplified by the clip-higher mechanism in DAPO and by subsequent variants such as differentiated clipping and smooth gating; the influence of these mechanisms on entropy is asymmetric and difficult to control precisely, and in the on-policy regime where the sampling ratio remains close to one, clipping is rarely activated, leaving the actual regulatory capacity limited(Yu et al., 18

2025; Yue et al., 2025b; Haarnoja et al., 2018; Chen et al., 2026; Xi et al., 2025; Zhou et al., 2026). The second category applies trajectory-level differentiated weighting between positive and negative samples, including asymmetric importance sampling and the upweighting of rare correct rollouts; since these methods operate at the trajectory granularity, they cannot distinguish the opposing entropy effects of different tokens within the same trajectory(Zhu et al., 2025; Tang et al., 2025; Wang et al., 2025b; Deng et al., 2025; Yang et al., 2025a). The third category couples token-level entropy information into the advantage or loss through entropy-induced advantages(Cheng et al., 2025; Huang et al., 2025; He et al., 2025c; Cui et al., 2025), explicit entropy regularization, or token filtering based on entropy variation; entropy rewards tend to overamplify the signal of high-entropy tokens and induce oscillations, regularization remains highly sensitive to the choice of its coefficient, and entropy-variation-based methods either rely on hard-to-estimate information about unsampled tokens or impose oversimplified binary partitions. In addition, raising the sampling temperature only delays rather than prevents entropy collapse. Overall, existing methods either operate at an excessively coarse granularity or lack a principled understanding of the underlying collapse mechanism. Token-Level Perspectives and STARE. Recent studies have begun to focus on the differentiated contributions of individual tokens(Wang et al., 2025b; Tang et al., 2025; Wang et al., 2026; Chen et al., 2026; Zhu et al., 2025). Some works demonstrate that a small subset of high-entropy tokens dominates the effective learning signal in RLVR and that performing gradient updates only on this subset already yields efficient performance gains(Wang et al., 2025b); another line of work identifies critical tokens at decision branching points along reasoning chains and encourages exploration at these positions(Chen et al., 2026; Tang et al., 2025; Cheng et al., 2025; Yang et al., 2025a). These findings echo the analysis presented in this paper. Most existing methods, however, remain heuristic in nature and either fail to keep the policy entropy stable and controllable or lack comprehensive validation across multiple scenarios, model scales, and long-horizon training settings. Starting from a first-order analysis of token-level entropy dynamics, STARE establishes an advantage-surprisal four-quadrant decomposition that exposes the credit assignment mismatch under shared trajectory-level advantages, together with a near-criticality property; building on this analysis, STARE identifies entropy-critical tokens through a batch-internal surprisal-quantile proxy, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate, thereby achieving principled token-level regulation of the policy entropy through a minimally invasive modification to the GRPO objective. STARE is comprehensively validated across model scales ranging from 1.5B to 32B and across three task families covering Short-CoT, Long-CoT, and tool-use scenarios, where it consistently delivers stable performance improvements and further unlocks the optimization potential of long-horizon RL training.

Additional Experiments and Analysis on STARE

B.1

STARE vs. GRPO-ds: Detailed Comparison across Diverse Scenarios and Model Scales in RL Training

To validate the effectiveness of STARE in long-horizon RL training, we conduct RL training of 5000 steps on Qwen2.5-Math-7B-Base under the Short CoT scenario. Figure 3 compares STARE against GRPO-ds on key metrics, while Figures 5–1 further verify the effectiveness of STARE across model scales ranging from 1.5B to 32B and across diverse task scenarios. Entropy stability and performance evolution. In Figure 3(a), GRPO-ds exhibits entropy collapse during the early training phase (0–1000 steps), in which the policy entropy decreases sharply and approaches zero, which is consistent with the theoretical analysis presented in Section 3. Meanwhile, the accuracy of GRPO-ds on AIME24 and AIME25 peaks around step 1000 and subsequently saturates, fluctuating without further improvement (Figure 3(e)-(f)), thereby indicating premature convergence of the policy distribution. In contrast, STARE stabilizes the policy entropy near Htgt = 0.3 through token-level advantage reweighting and closed-loop entropy gating, and the accuracy of STARE continues to improve beyond step 1000, reaching the optimum at step 5000, which extends the number of trainable steps and unlocks the optimization potential of long-horizon RL training. Exploration-exploitation balance. In Figure 3(g)-(h), the Pass@32 of STARE consistently surpasses that of GRPO-ds throughout training, which indicates that the policy retains sufficient output diversity and effectively mitigates mode-seeking behavior. In Figure 3(b)-(c), the training reward and Full-Solve Ratio of GRPO-ds rise rapidly during the early phase and subsequently plateau, whereas STARE maintains a steady upward trajectory throughout training. Meanwhile, the sustained growth of response length under STARE (Figure 3(d)) suggests that the model addresses complex problems by extending the reasoning depth. Cross-scale and cross-scenario generalization. Figures 5-1 systematically present the behavior of STARE under broader configurations. Across the Short CoT scenario(14B, 32B), the Long CoT scenario(R1-Distill19

Cumulative Net Entropy Change

Record · ID 287128 · SHA-256 1324d9c9bd2d49ff
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.