Experience Augmented Policy Optimization for LLM Reasoning
Jinda Lu 1 Kexin Huang 1 Junkang Wu 1 Shuo Yang 2 Jinghan Li 1 Chiyu Ma 3 Shaohang Wei 2 Xiang Wang 1 Guoyin Wang 4 Jingren Zhou 4
arXiv:2606.30420v1 [cs.LG] 29 Jun 2026
Abstract
reasoning experience that gives rise to advanced behaviors such as chain-of-thought (CoT) reasoning and self-reflection (Guo et al., 2025; Yang et al., 2025) (Figure 1(a)).
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experienceaugmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.
Despite these successes, current RLVR strategies (Yu et al., 2025; Wang et al., 2025; Wu et al., 2025; Shao et al., 2024) typically perform on-policy rollouts and policy optimization from scratch, requiring massive amounts of sampling to explore successful trajectories within large search spaces. Crucially, this paradigm fails to exploit the valuable experience encoded in prior RL-optimized models (πRL ), resulting in substantial inefficient exploration and, more importantly, imposing a fundamental bottleneck on the scalable improvement of reasoning performance. To address this, recent work (Zhan et al., 2025) treats previously generated, static reasoning trajectories as experience by incorporating these trajectories into current rollouts. However, as reinforcement learning progresses, the capabilities and behavior of the policy model evolve, causing earlier trajectories to become increasingly mismatched with the current policy. In this paper, we argue that experience in RLVR should not be reused in the form of fixed reasoning trajectories, but instead represented in a manner that adapts to the evolving policy. Prior studies (Wang et al., 2025; Cheng et al., 2025; Huang et al., 2026) suggest that successful reasoning outcomes are often determined by a small number of pivotal actions (tokens), rather than entire trajectories. This observation motivates an action-level view of experience, where reinforcement learning shapes which decisions are more likely to lead to successful outcomes. Through RL optimization, such experience is implicitly encoded in an RL-optimized policy πRL , which captures a structured prior over critical actions. By comparing the behavior of the current policy with this prior, EAPO identifies decision points where the current policy departs from experience, enabling targeted injection of prior experience during training (Figure 1(b)).
1. Introduction Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated strong effectiveness in improving the reasoning capabilities of large language models (LLMs) (Shao et al., 2024; Jaech et al., 2024; Comanici et al., 2025). Through extensive rollouts and feedback from verifiable rewards, RLVR encourages models to produce longer and more reliable reasoning processes, thereby accumulating 1
University of Science and Technology of China 2 Peking University 3 Dartmouth College 4 Independent Researcher. Correspondence to: Xiang Wang <[email protected]>.
To this end, we propose Experience-Augmented Policy Optimization (EAPO), a framework that leverages a prior RL-optimized policy πRL as an action-level prior to assist the training of the current policy πθ . EAPO operates through two key components: (1) experience-guided action resam-
Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).
1
Experience Augmented Policy Optimization for LLM Reasoning (a) RL from 𝛑𝐁𝐚𝐬𝐞 to 𝛑𝐑𝐋
(c) AIME 24 Comparisons over various backbones
RL 𝛑𝐁𝐚𝐬𝐞
𝛑'(
(b) RL with Experience from 𝛑𝐑𝐋
𝛑𝐁𝐚𝐬𝐞
𝛑'(
(a) AIME’24 score on Qwen-2.5-7B Math
(b) AIME’24 score on Qwen-3-8B Base
Figure 1. Overview of experience-augmented reinforcement learning. (a) Standard reinforcement learning with verifiable rewards (RLVR) optimizes a base policy πBase into an RL-optimized policy πRL through iterative rollouts and policy updates. (b) Instead of training from scratch, EAPO leverages experience from a prior RL-optimized policy πRL during rollout of the current policy by resampling actions at critical decision points, introducing experience that adapts to the evolving policy. (c) Empirical comparisons on AIME’24 show that incorporating experience from πRL significantly accelerates exploration and improves accuracy compared to DAPO (Yu et al., 2025), across various base models (Qwen-2.5-7B-Math (Yang et al., 2024b) and Qwen-3-8B (Yang et al., 2025)).
qb denotes the input question and y b denotes the corresponding ground-truth answer, the model πθ generates an output sequence ob that consists of a reasoning process and a final prediction. The final prediction is enclosed in \boxed{.}, while the intermediate reasoning steps are delimited by <think>...</think>, which enables automated extraction and verification of the predicted answer against the ground truth. RLVR employs a binary reward function R(·), which assigns a reward of 1 if the extracted prediction matches the ground truth and 0 otherwise.
pling during rollout and (2) experience-aware policy optimization for stable learning. Experience-guided action resampling. EAPO identifies critical decision points by comparing the action selection of the current policy πθ with those of a prior RL-optimized policy πRL , and injects experience by resampling actions at these points with πRL during rollout. Experience-aware policy optimization. To ensure stable optimization from experience-augmented rollouts, EAPO employs an experience-aware optimization strategy that adjusts policy updates according to the intensity of experience injection, ensuring robust and effective policy optimization.
The objective of RLVR is to maximize the expected reward over the training distribution, which can be formulated as: JRLVR (θ) = E(qb ,yb )∼D Eob ∼πθ (·|qb ) R(ob , y b ) . (1)
As illustrated in Figure 1(c), experiments on AIME24 with Qwen-2.5-7B-Math (Yang et al., 2024b) and Qwen-3-8B (Yang et al., 2025) demonstrate that, compared to DAPO (Yu et al., 2025), EAPO significantly reduces exploration steps and yields stable and consistent improvements across different backbones.
2.2. RLVR Optimization Algorithms Group Relative Policy Optimization (GRPO). GRPO is a widely adopted optimization strategy for RLVR that stabilizes training by computing advantages relative to a group of responses for the given question (Shao et al., 2024). Specifically, for each input qb , GRPO samples a group of G responses {obi }G i=1 from the old policy πold and normalizes rewards within the group to estimate advantages. The GRPO objective is defined as:
2. Preliminaries In this section, we revisit Reinforcement Learning with Verifiable Rewards (RLVR) and review representative optimization strategies (Shao et al., 2024; Yu et al., 2025).
b
|oi | G 1 X 1 X JGRPO (θ) =E(qb ,yb )∼D E{obi }G b i=1 ∼πold (·|q ) G i=1 |obi | t=1 b b min ri,t · Âbi,t , clip ri,t , 1 − ϵ, 1 + ϵ · Âbi,t (2) #
"
2.1. Reinforcement Learning with Verifiable Rewards Reinforcement Learning with Verifiable Rewards (RLVR) aims to enhance the reasoning capabilities of large language models (LLMs) by aligning model outputs with automatically verifiable answers. Given a batch of B question– answer pairs {(qb , y b )}B b=1 sampled from dataset D, where
− βDKL (πθ ∥πref ) , b Here, ri,t denotes the importance sampling ratio between
2
Experience Augmented Policy Optimization for LLM Reasoning
the current policy πθ and the old policy πold : b ri,t =
πθ (obi,t | qb , obi,<t ) πold (obi,t | qb , obi,<t )
.
Based on this observation, we argue that experience in LLM reasoning should be characterized at the action level, in terms of emphasis over critical decisions. Through extensive on-policy optimization, an RL-optimized model πRL does not merely memorize specific reasoning paths, but instead learns which actions are more likely to lead to successful outcomes under particular decision contexts. Such experience is implicitly encoded in the parameterized distribution of πRL , endowing it with strong generalization capability.
(3)
The advantage Âbi,t is computed by normalizing rewards within each response group: b b Âb = Ri − mean(R ) , i,t b std(R ) (4) Rb = I is equivalent(ob , y b ) , i i
Accordingly, in EAPO, we treat the prior RL-optimized policy πRL as a structured, adaptable representation of experience. By comparing the action distributions of the current policy πθ and πRL , we identify critical decision points where the current policy may deviate from effective decisions, and incorporate prior RL experience at these points with minimal intervention. In this way, EAPO elevates experience from static samples to a generalizable decision prior, while preserving the on-policy nature of policy optimization.
where I(·) is the indicator function. Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO). DAPO (Yu et al., 2025) improves upon GRPO by removing KL regularization and introducing asymmetric clipping, dynamic sampling, and token-level loss normalization, leading to new state-of-the-art performances. Formally, the DAPO objective is given by: " JDAPO (θ) = E(qb ,yb )∼D E{obi }G b PG i=1 ∼πold (·|q )
b
1
|oi | X
3.2. Experience-guided Resampling
min b i=1 |oi | t=1 (5) # b b b b ri,t · Âi,t , clip ri,t , 1 − ϵlow , 1 + ϵhigh · Âi,t ,
EAPO injects experience only when the current policy is overconfident yet disagrees with πRL , enabling sparse and targeted correction during rollout. Critical token identification. At each decoding step t, given the input question q and the current generated prefix y<t , we compare the token-level behavior of the current policy πθ with that of a previously optimized RL policy πRL . Specifically, let yt denote the token sampled by πθ at step t. We define the token-level discrepancy δt as the log-likelihood ratio between current and prior policies:
Here, ϵlow and ϵhigh decouple the clipping thresholds for negative and positive advantages, respectively, allowing more aggressive updates for high-quality responses.
3. Approach In this section, we present Experience-Augmented Policy Optimization (EAPO), which consists of three core components: (1) Experience formalization, where prior RL experience is interpreted as a prior over critical actions; (2) Experience-guided resampling, which incorporates such experience during rollout via sparse, token-level intervention; and (3) Experience-aware optimization, which integrates experience-augmented trajectories into stable onpolicy learning. The algorithm of EAPO is in Alg. 1.
δt = log πθ (yt | q, y<t ) − log πRL (yt | q, y<t ).
(6)
This discrepancy characterizes the degree of disagreement between the current on-policy and prior RL experience. When δt ≈ 0, the two policies assign similar probabilities to the selected token, indicating that the current decision is consistent with RL experience. Similarly, when δt < 0, the RL-optimized policy assigns a higher probability than the current policy, suggesting that the decision is well supported by prior experience.
3.1. Experience as a Prior over Critical Actions Experience has been increasingly recognized as a key driver of continual progress in reinforcement learning, enabling models to move beyond repeatedly exploring from scratch and leading to more stable learning (Silver & Sutton, 2025). In the context of large language models’ reasoning, work in (Zhan et al., 2025) treats previously generated model responses as experience and reuses them during subsequent rollouts. However, as the policy continuously evolves during reinforcement learning, reasoning trajectories generated at earlier stages often become mismatched with the current policy, rendering such static responses unreliable sources.
In contrast, when δt > 0, the current policy assigns high confidence to a token that is strongly disfavored by the RL-optimized policy, exposing a clear mismatch with empirically validated decision patterns. Such overconfident yet unsupported decisions contradict RL experience and tend to arise at pivotal reasoning steps, where an incorrect choice can irreversibly divert the reasoning trajectory and propagate an incorrect final answer. We therefore regard these positions as critical decision points, where incorporating prior RL experience is most beneficial. 3
Experience Augmented Policy Optimization for LLM Reasoning
Resampling. Given the identified critical decision points, EAPO performs experience-guided action resampling during rollout generation. Concretely, we introduce a threshold τ to determine whether experience should be injected at the decoding step t: gt = I(δt > τ ), (7)
policy update direction. In contrast, negative samples are more reliably obtained through on-policy exploration, which better reflects the model’s current uncertainty and failure modes (Liu et al., 2025). To mitigate this, we apply positive experience filtering, restricting policy optimization to experience-augmented trajectories that yield correct predictions. Formally, for the experience-augmented trajectory obexp , we define a binary mask mbexp : ( 1, R(obexp , y b ) = 1, b mexp = (9) 0, R(obexp , y b ) = 0,
where gt = 1 indicates that step t is identified as a critical decision point, and the candidate token yt is discarded and resampled from πRL . Based on this gating, we define the experience-guided sampling trajectory as: ( πθ (· | q, y<t ), if gt = 0, πExp (· | q, y<t ) = (8) πRL (· | q, y<t ), if gt = 1.
ensuring that only trajectories with mbexp = 1 are included in gradient updates.
When gt = 0, tokens are sampled directly from the current policy, preserving on-policy exploration. When gt = 1, experience is injected by resampling the token from the prior RL-optimized policy. Importantly, EAPO does not replace entire trajectories; instead, it intervenes only at a small number of high-risk decision points, enabling fine-grained correction of the reasoning process while maintaining sufficient on-policy exploration.
Resampling-based Importance Sampling. Experienceguided resampling replaces a subset of tokens from πold with a prior RL policy πRL . Consequently, experienceaugmented responses are no longer sampled purely from πold , violating the standard importance sampling assumption (Shao et al., 2024; Schulman et al., 2017).
Acceleration via block-wise verification. Applying experience verification at every decoding step can be computationally expensive. Inspired by speculative decoding (Chen et al., 2023; Leviathan et al., 2023), we amortize verification by processing K decoding steps as a block.
A naive approach would be to modify importance sampling at the token level. However, due to the auto-regressive nature of language models, token-wise correction breaks their causal dependency, since resampling a single token alters the effective distribution of all subsequent tokens.
At step t, the current policy πθ generates a speculative block of K tokens ỹt:t+K−1 . The RL policy πRL is then applied in parallel to evaluate discrepancies {δt+i }K−1 i=0 . If a critical point is detected, we identify the earliest index i such that gt+i = 1, truncate the sequence at t + i, and resample that token from πRL . All subsequent speculative tokens are discarded to preserve causal consistency. This block-wise verification substantially reduces the overhead of experience checking while preserving the original resampling behavior.
Consequently, token-level importance sampling fails to characterize the behavior policy of the generated trajectory faithfully and often leads to unstable optimization.
3.3. Experience-aware Policy Optimization
which measures the proportion of tokens resampled from πRL . This statistic serves as a simple empirical proxy for the overall degree of experience injection in the trajectory. Based on this, we approximate the effective behavior policy b πExp with a smoothed surrogate defined as:
Instead, we adopt a response-level, smoothed importance sampling scheme. For an experience-augmented response obi , we compute the resampling ratio: b
ρbi =
We next describe how experience-guided rollouts are incorporated into policy optimization in a stable manner. Positive Experience Filtering. In EAPO, experience is introduced into only a small number of trajectories, typically a single experience-augmented response per rollout group. While experience-guided resampling often improves response quality, it does not guarantee correct outcomes for each rollout.
|oi | 1 X gb , |obi | t=1 i,t
b π̃Exp = (1 − ρbi ) πold + ρbi πRL .
(10)
(11)
And compute the final importance ratio as: r̃ib =
When an experience-augmented trajectory yields an incorrect prediction, the injected experience may be misaligned with the current policy, making gradient signals that deviate from the on-policy optimization objective. Such erroneous experience-augmented trajectories can therefore distort the
πθ (obi | qb ) . b (ob | qb ) π̃Exp i
(12)
This formulation provides a smooth and stable correction for experience-guided rollouts. When no experience is injected (ρbi = 0), it reduces to the standard importance ratio. As 4
Experience Augmented Policy Optimization for LLM Reasoning
AdamW optimizer with a constant learning rate of 1 × 10−6 , a weight decay of 0.01, and a global batch size of 512. For each training step, we sample a rollout group of size G = 16. Within each group, 15 responses are generated by the current policy πθ to encourage on-policy exploration, while only 1 response is experience-augmented trajectories guided by πRL . For experience annealing, we set the block size to K = 20 and the annealing step to T = 60.
the amount of experience increases, the correction smoothly interpolates toward the experience-guided behavior. Experience Annealing. While experience-guided resampling accelerates early exploration, continued reliance on prior RL experience becomes unnecessary as the current policy improves. Once the policy has sufficiently benefited from experience guidance, further resampling may restrict exploration and hinder independent refinement.
Evaluation Protocols. We evaluate our approach across two complementary dimensions. Mathematical reasoning benchmarks include AIME’24, AIME’25, and AMC. To assess general knowledge and reasoning ability beyond mathematics, we additionally report results on general scientific knowledge benchmarks, including MMLU-Pro (Wang et al., 2024) and GPQA (Rein et al., 2024).
EAPO therefore applies experience-guided resampling only for the first T training steps and subsequently disables it. After annealing, the policy is trained purely with on-policy rollouts, enabling unrestricted exploration and continued improvement. Benefits. EAPO offers three key advantages: (1) Adaptability: experience adapts naturally to prefixes generated by the current policy; (2) Focus on critical decisions: experience is injected only at high-impact decision points; and (3) Balanced exploration: sparse intervention preserves the on-policy nature of learning.
4.2. Ablation Study Hyperparameter Sensitivity (RQ1). We study the sensitivity of the resampling threshold τ , which controls the tradeoff between on-policy exploration and experience guidance. Figure 2 summarizes the effect of τ from three complementary perspectives: (a) AIME’24 performance, reflecting the model’s capability in mathematical reasoning; (b) the resampled ratio, indicating the intensity of experience intervention during training; and (c) the resampled accuracy, which measures the quality of the injected experience.
4. Experiment In this section, we evaluate the effectiveness of our Experience-Augmented Policy Optimization over two base models: Qwen-2.5-Math-7B-Base (Yang et al., 2024b) and Qwen-3-8B-Base (Yang et al., 2025). Specifically, we aim to answer the following Research Questions (RQs):
• Effects of Sparse Intervention. A key observation from Figure 2 (b) and (c) is the high efficiency of experience injection. When τ = 0.5, the resampled ratio remains extremely low (below 4%), indicating that EAPO intervenes at only a small subset of decision points (e.g., approximately 40 tokens within a 1,000-token response). Despite this sparse intervention, the accuracy of experience-augmented responses consistently exceeds 70%. This low-intervention, high-return behavior substantiates our core hypothesis that effective RL guidance is inherently sparse, and that selectively correcting only a limited number of critical tokens is sufficient to reliably steer policy optimization.
❶: Is EAPO sensitive to hyperparameters? ❷: How do different components affect EAPO? ❸: How does EAPO compare to state-of-the-art methods? 4.1. Implementation Details Experimental Setup. All experiments are conducted under the DAPO framework (Yu et al., 2025). Following the standard configuration, we set the clip-higher and clip-lower hyperparameters to ϵhigh = 0.28 and ϵlow = 0.2, respectively. To support long-chain reasoning, the maximum response length is set to 20,480 tokens for Qwen3-8B-Base and 8,192 tokens for Qwen2.5-Math-7B. In our setting, we adopt a two-stage training strategy. In the first stage, an RL-optimized reference policy πRL is obtained by optimizing the base model πBase with the DAPO objective. In the second stage, we initialize the target policy πθ from the same base model πBase and further optimize it using DAPO, while selectively incorporating experience from πRL . Importantly, πRL is only used to provide experience guidance during rollout and does not directly supervise or constrain the optimization objective.
• Impact of the Threshold τ . The threshold τ determines the strictness of experience injection. For moderate values (τ ∈ [0.3, 0.5]), although the resampled ratio gradually decreases as τ increases, both rollout accuracy and AIME’24 performance remain stable and high. This indicates that the most informative experience is preserved even under stricter filtering. In contrast, when τ is set too high (τ ≥ 0.8), the resampled ratio drops sharply, causing many critical decision points to be missed. As a result, both the resampled accuracy (Figure 2 (c)) and the task performance on AIME’24 (Figure 2 (a)) degrade noticeably.
Training Dynamics. All models are fine-tuned on the DAPO-Math-17K dataset (Yu et al., 2025). We use the
• Hyperparameter Setting. To minimize reliance on πRL while maintaining strong performance, we adopt τ = 0.5 as 5
Experience Augmented Policy Optimization for LLM Reasoning
Figure 2. Sensitivity analysis of the resampling threshold τ . (a) AIME’24 performance (Mean@32) under different values of τ , reporting the best performance achieved during training. (b) Evolution of the resampled ratio across training steps, representing the proportion of resampled tokens within the experience-augmented responses. (c) Accuracy of experience-augmented responses across training steps, reflecting the quality of experience injection.
Figure 3. Influence of experience granularity and filtering strategy. (a) AIME’24 performance (Mean@32) comparison between token-level (Tokens, Pos-Only) and trajectory-level (Trajectory, Pos-Only) experience. Following (Zhan et al., 2025; Liu et al., 2025), we consider only positive samples in this comparison, where positive samples refer to trajectories whose final predictions are correct. (b) KL divergence to the RL-optimized experience model πRL , measuring the distributional distance between the learned policy and the experience model under different experience designs. (c) AIME’24 performance (Mean@32) comparison between positive-only (Tokens, Pos-Only) and all (Tokens, Pos & Neg) token-level experience, illustrating the effect of different experience filtering strategies.
the default setting. This configuration achieves a favorable balance between sparse experience usage and effectiveness, reaching a Mean@32 score of 0.381 on AIME’24.
samples leads to a clear degradation in AIME’24 performance. This is accompanied by a larger KL divergence to πRL (Figure 3 (b)), suggesting that negative samples introduce conflicting optimization signals that interfere with effective experience utilization.
Influences of Different Components (RQ2). We analyze two key design choices in experience utilization: the granularity of experience (token-level versus trajectory-level) and the experience filtering strategy (positive-only versus using both positive and negative samples). Here, positive samples refer to trajectories whose final predictions are correct.
Overall, these results indicate that effective experience utilization requires both fine-grained intervention and careful filtering to maintain a moderate distance to the experience model, rather than indiscriminate replay.
• Effects of experience granularity. Following (Liu et al., 2025; Zhan et al., 2025), we compare token-level and trajectory-level experience using only positive samples. As shown in Figure 3 (a), token-level experience consistently outperforms trajectory-level replay on AIME’24. Although trajectory-level experience yields the lowest KL divergence to the experience model πRL (Figure 3 (b)), it provides coarse-grained supervision that is insufficient for effective policy improvement. In contrast, token-level experience enables targeted correction at critical decision points, leading to better performance.
4.3. Comparison with SOTA Methods (RQ3). Compared Methods and Categories. To enable fair comparisons, we re-implement representative state-of-the-art RLVR methods and adapt them to the same training setting as EAPO. The compared methods can be broadly categorized into three groups. (1) Baseline RLVR methods, include GRPO and DAPO, which optimize policies using outcome-level verifiable rewards without explicit experience reuse. Notably, DAPO also serves as the RL-optimized experience policy πRL in EAPO, highlighting that EAPO improves performance by reusing strong RLVR baselines. (2) Token-level supervision methods, include on-policy distillation (OPD) and Multi-Teacher On-Policy Distillation
• Effects of experience quality. We further examine experience filtering by comparing positive-only token-level experience with the inclusion of both positive and negative samples. As shown in Figure 3 (c), incorporating negative 6
Experience Augmented Policy Optimization for LLM Reasoning Table 1. Performance on math reasoning benchmarks. Comparison of RLVR strategies on in-domain math reasoning benchmarks, including AIME’24, AIME’25, and AMC. Due to time constraints and computational cost on the Qwen3-8B-Base model, Trajectory Replay and EAPO w/o sIS are only evaluated on the 7B models. Results are calculated over 32 runs, reported as proportions %. Model
AIME’24
Method
Qwen-2.5 -Math-7B
Qwen-3 8B-Base
AIME’25
Qwen-2.5 -Math-7B
Qwen-3 8B-Base
Average
Pass@1
Pass@16
Pass@1
Pass@16
Pass@1
Pass@16
Pass@1
Pass@16
GRPO DAPO
32.08 37.19
50.24 58.00
11.77 15.62
24.56 30.60
67.39 68.86
81.24 89.59
37.08 40.56
52.01 59.40
MOPD OPD
38.85 38.12
60.28 55.06
15.93 15.21
33.08 29.14
68.56 68.59
87.67 87.75
41.11 40.64
60.34 57.32
Trajectory Replay EAPO w/o sIS EAPO
37.25 39.68 40.14
58.60 63.29 60.44
16.24 15.93 17.08
34.12 35.46 37.04
72.74 72.66 72.81
89.93 88.27 89.74
42.08 42.76 43.34
60.88 62.37 62.41
GRPO DAPO
31.67 36.67
61.98 71.41
23.54 27.39
50.91 48.98
67.73 71.88
86.89 89.87
40.98 45.31
66.59 70.09
MOPD OPD EAPO
37.80 38.23 41.66
74.68 67.48 74.37
29.38 27.81 30.92
52.25 50.90 52.54
72.03 71.16 76.54
89.83 88.17 91.51
46.40 45.73 49.71
72.25 68.85 72.81
Table 2. Performance on general science benchmarks. Comparison of different RLVR strategies on out-of-distribution reasoning benchmarks, including GPQA and MMLU-Pro. All results are averaged over 8 runs, reported as proportions (%). Model
AMC
Method
GPQA
MMLU-Pro
Avg
GRPO DAPO
34.34 39.27
36.94 44.09
35.64 41.68
MOPD OPD
38.76 39.39
43.81 43.34
41.29 41.37
Trajectory Replay EAPO w/o sIS EAPO
39.67 41.98 43.18
46.50 47.35 47.57
43.09 44.67 45.38
GRPO DAPO
50.44 52.27
64.71 65.29
57.58 58.78
MOPD OPD EAPO
50.51 52.27 54.67
65.99 65.91 66.49
58.25 59.09 60.58
marks (Table 1), EAPO consistently outperforms baseline RLVR methods (GRPO, DAPO) and token-level supervision approaches (OPD, MOPD) across model scales. Compared with trajectory-level experience replay, EAPO achieves higher average performance, indicating that selectively injecting experience at critical decision points is more effective than replaying entire trajectories. • Out-of-Domain Science Reasoning. EAPO also demonstrates strong generalization on out-of-distribution science benchmarks (Table 2). On both GPQA and MMLU-Pro, EAPO consistently surpasses all compared baselines under the same training setting, suggesting that it learns transferable reasoning behaviors rather than overfitting to domainspecific patterns. • Effect of Importance Sampling. The consistent performance gap between EAPO and its variant without smoothed importance sampling (EAPO w/o sIS) highlights the importance of stable credit assignment when incorporating experience. Smoothed importance sampling regulates gradient contributions from experience-augmented decisions, leading to more stable optimization even when only a small fraction of tokens are resampled by πRL .
(MOPD), where an RL-optimized policy πRL is used to provide token-level guidance during optimization. (3) Trajectory-level experience replay methods, include trajectory replay, where successful trajectories generated by πRL are reused as experience during training. Moreover, we additionally report results for EAPO without smoothed importance sampling (denoted as EAPO w/o sIS), alongside the full EAPO model.
Overall, EAPO achieves superior performance by combining fine-grained experience injection with careful optimization control, providing a more effective alternative over existing RLVR paradigms.
Overall Performance. Table 1 and Table 2 summarize the comparison between EAPO and representative RLVR baselines on in-domain math reasoning and out-of-distribution science benchmarks. Across all evaluated settings, EAPO consistently achieves the strongest or highly competitive performance, demonstrating its effectiveness as an experienceaugmented policy optimization method.
5. Related Works In this section, we first briefly summarize current RLVR strategies, and then we illustrate related methods for enhancing large language models via experience. Finally, we enumerate the differences.
• In-domain Math Reasoning. On math reasoning bench7
Experience Augmented Policy Optimization for LLM Reasoning
5.1. Reinforcement Learning with Verifiable Rewards.
5.3. Experience in Large Language Models.
Recent advances (Li et al., 2025a; Lu et al., 2023; 2024; 2025b;a), particular Reinforcement Learning with Verifiable Rewards (RLVR) (Guo et al., 2025; Lu et al., 2026b; Li et al., 2026; Lu et al., 2026a), have emerged as a dominant paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). Pioneering works, such as OpenAI’s o1 (Jaech et al., 2024) and DeepSeek-R1 (Guo et al., 2025), have established the paradigm that RLVR can significantly enhance the reasoning capabilities of LLMs. Building on this, recent state-of-the-art models (Team et al., 2025; Yang et al., 2025; Liu et al., 2025) further push the boundaries across diverse and complex scenarios. Representative algorithms, such as Group Relative Policy Optimization (GRPO) (Shao et al., 2024) and its successor, Dynamic Sampling Policy Optimization (DAPO) (Yu et al., 2025), have established themselves as foundational baselines, inspiring a wide range of follow-up studies (Wu et al., 2025; Cheng et al., 2025; Wang et al., 2025). However, we observe that most existing RLVR frameworks rely heavily on pure on-policy sampling, often neglecting the potential of expert experience to guide the optimization process. This motivates our investigation into a more robust experience injection strategy.
As we enter the era of experience (Silver & Sutton, 2025), the focus of reinforcement learning for LLMs is shifting from pure online exploration toward the strategic utilization of experiential knowledge. In agent learning, leveraging early or synthesized experience has been shown to significantly bolster generalization capability across novel domains (Zhao et al., 2024; Zhang et al., 2025b). Similarly, in large reasoning models, recent approaches incorporate experience primarily at the trajectory level. Representative strategies either reuse trajectories from optimized policies (Zhang et al., 2025a) or revisit rollouts from previous onpolicy iterations (Zhan et al., 2025; Li et al., 2025b; Liang et al., 2025). While these methods prove that replaying successful trajectories accelerates convergence, they typically treat experience as holistic, trajectory-level demonstrations, overlooking the sparsity of critical decisions within reasoning processes, and may suffer from trajectory mismatch as the policy evolves. In contrast, EAPO models experience at the action level and injects it at critical decision points, enabling policy-adaptive experience reuse while avoiding trajectory-level mismatch.
5.2. Off-policy Guidance for RLVR.
In this work, we identify a fundamental limitation of existing RLVR methods: they primarily rely on on-policy optimization from scratch and fail to effectively reuse the rich experience accumulated in prior RL-optimized models. To address this limitation, we propose Experience-Augmented Policy Optimization (EAPO), which represents RL experience as an action-level, policy-adaptive prior rather than static trajectories. EAPO selectively injects experience at critical decision points during rollout and integrates experienceaugmented trajectories into on-policy optimization through an experience-aware optimization strategy.
6. Conclusion
To mitigate the high sampling inefficiency and limited exploration of pure on-policy RLVR, recent work has explored incorporating off-policy guidance during RL rollout. Existing approaches can be broadly categorized into two categories. Distillation from stronger models. One line of work attributes the exploration bottleneck to insufficient model capacity and addresses it by distilling stronger models into the current model. Representative approaches, such as (Yan et al., 2025; Lu & Lab, 2025; Jiang et al., 2025; Zhang et al., 2025c), employ more capable models (e.g., Qwen-2.5 32B(Yang et al., 2024a), DeepSeek-R1 (Guo et al., 2025)) to provide high-quality trajectories or token-level guidance, thereby improving the model’s reasoning performance.
Extensive experiments on both mathematical and general reasoning benchmarks demonstrate that EAPO consistently improves reasoning performance and generalization while requiring only sparse experience intervention. Overall, these results suggest that policy-adaptive experience reuse offers an effective and scalable direction for enhancing reinforcement learning with verifiable rewards.
Experience reuse from previous success. Another line of work focuses on reusing experience accumulated from the model’s own reinforcement learning process. Representative methods incorporate successful experiences from earlier RL iterations or optimized policies into current rollouts (Zhan et al., 2025; Li et al., 2025b). Our work belongs to the experience-based category. Different from distillationbased approaches, we believe that the base model already possesses sufficient reasoning capacity, and instead aim to unlock its potential by more effectively exploiting experience accumulated through prior RL optimization.
Limitations and Future Work Despite its effectiveness, EAPO has several limitations that leave room for future research: (1) EAPO relies on a prior RL-optimized policy πRL , and its effectiveness is bounded by the quality and diversity of this policy. Extending EAPO to multiple or dynamically evolving experience policies is a natural next step. (2) EAPO employs predefined mechanisms, such as a fixed resampling threshold and annealing schedule. More adaptive strategies that adjust experience usage based on
8
Experience Augmented Policy Optimization for LLM Reasoning
training dynamics may further improve robustness. (3) Our study focuses on reasoning tasks with verifiable rewards; extending experience-guided resampling to weaker or noisier supervision remains an open question.
learning of llms, 2025. URL https://arxiv.org/ abs/2510.04140. Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274– 19286. PMLR, 2023.
Impact Statement This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Li, J., Gao, Y., Lu, J., Fang, J., Wen, C., Lin, H., and Wang, X. DiffGAD: A diffusion-based unsupervised graph anomaly detector. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview.net/forum?id=AhcY q4CnfF.
Acknowledgement
Li, J., Fang, J., Lu, J., Wang, Y., Guo, X., Zhang, T., Wang, X., and He, X. Enhancing multi-modal LLMs reasoning via difficulty-aware group normalization. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id= jyOgpu5wfC.
This research is supported by the National Natural Science Foundation of China (62572449).
References Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
Li, S., Zhou, Z., Lam, W., Yang, C., and Lu, C. Repo: Replay-enhanced policy optimization. arXiv preprint arXiv:2506.09340, 2025b.
Cheng, D., Huang, S., Zhu, X., Dai, B., Zhao, W. X., Zhang, Z., and Wei, F. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14758, 2025.
Liang, J., Tang, H., Ma, Y., Liu, J., Zheng, Y., Hu, S., Bai, L., and Hao, J. Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model. arXiv preprint arXiv:2507.06892, 2025.
Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. Lu, J., Wang, S., Zhang, X., Hao, Y., and He, X. Semanticbased selection, synthesis, and supervision for few-shot learning. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 3569–3578, 2023.
Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. Deepseekr1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025.
Lu, J., Wang, S., Hao, Y., Liu, H., Wang, X., and Wang, M. Rethinking visual content refinement in low-shot clip adaptation. arXiv preprint arXiv:2407.14117, 2024.
Huang, K., Meng, H., Wu, J., Lu, J., Ma, C., Chen, Z., Wang, X., Ding, B., Wu, J., Wang, X., He, X., Wang, G., and Zhou, J. Beyond magnitude: Leveraging direction of RLVR updates for LLM reasoning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum ?id=r6Pw3RiMYL.
Lu, J., Li, J., Gao, Y., Wu, J., Wu, J., Wang, X., and He, X. Adavip: Aligning multi-modal llms via adaptive visionenhanced preference optimization, 2025a. URL https: //arxiv.org/abs/2504.15619.
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.
Lu, J., Wu, J., Li, J., Jia, X., Wang, S., Zhang, Y., Fang, J., Wang, X., and He, X. Dama: Data-and model-aware alignment of multi-modal llms. In International Conference on Machine Learning, pp. 40726–40740. PMLR, 2025b.
Jiang, Z., Han, J., Li, T., Wang, X., Jiang, S., Liang, J., Dai, Z., Ma, S., Yu, F., and Xiao, Y. Selective expert guidance for effective and diverse exploration in reinforcement
Lu, J., Huang, K., Wu, J., Yang, S., Li, J., Ma, C., Wei, S., Wang, X., Wang, G., and Zhou, J. Experience augmented 9
Experience Augmented Policy Optimization for LLM Reasoning
policy optimization for LLM reasoning. In Forty-third International Conference on Machine Learning, 2026a. URL https://openreview.net/forum?id= QOoQ0Bo2ls.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024a. Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math technical report: Toward mathematical expert model via selfimprovement. arXiv preprint arXiv:2409.12122, 2024b.
Lu, J., Wu, J., Li, J., Huang, K., Yang, S., Wang, G., Wu, J., Wang, X., and He, X. Bridging perception and reasoning: Token reweighting for rlvr in multimodal llms, 2026b. URL https://arxiv.org/abs/2603.25077.
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Lu, K. and Lab, T. M. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml .20251026. https://thinkingmachines.ai/blog/on-policydistillation.
Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L., et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Zhan, R., Li, Y., Wang, Z., Qu, X., Liu, D., Shao, J., Wong, D. F., and Cheng, Y. Exgrpo: Learning to reason from experience. arXiv preprint arXiv:2510.02245, 2025.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhang, H., Fu, J., Zhang, J., Fu, K., Wang, Q., Zhang, F., and Zhou, G. Rlep: Reinforcement learning with experience replay for llm reasoning. arXiv preprint arXiv:2507.07451, 2025a.
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Zhang, K., Chen, X., Liu, B., Xue, T., Liao, Z., Liu, Z., Wang, X., Ning, Y., Chen, Z., Fu, X., et al. Agent learning via early experience. arXiv preprint arXiv:2510.08558, 2025b.
Silver, D. and Sutton, R. S. Welcome to the era of experience. Google AI, 1, 2025. Team, K., Bai, Y., Bao, Y., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y., Chen, Y., Chen, Y., et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025.
Zhang, W., Xie, Y., Sun, Y., Chen, Y., Wang, G., Li, Y., Ding, B., and Zhou, J. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508.11408, 2025c.
Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X., Yang, J., Zhang, Z., et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939, 2025.
Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., and Huang, G. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19632–19642, 2024.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024. Wu, J., Huang, K., Wu, J., Zhang, A., Wang, X., and He, X. Quantile advantage estimation for entropy-safe reasoning. arXiv preprint arXiv:2509.22611, 2025. Yan, J., Li, Y., Hu, Z., Wang, Z., Cui, G., Qu, X., Cheng, Y., and Zhang, Y. Learning to reason under off-policy guidance. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL ht tps://openreview.net/forum?id=vO8LLo NWWk. 10
Experience Augmented Policy Optimization for LLM Reasoning
A. More Ablation Results.
(a) Effect of experience annealing steps. AIME’24 performance under different annealing steps T . Insufficient annealing leads to slightly lower gains.
(b) Component-wise ablation of EAPO. Adding resampling, positive filtering, and smoothed importance sampling yields monotonic AIME’24 performance gains.
Figure 4. Additional ablation results. Left: impact of experience annealing steps. Right: complementary effects of EAPO components.
Effect of experience annealing. Figure 4a studies the impact of the experience annealing steps T . Across all settings, incorporating experience consistently improves performance over the baseline, confirming the general benefit of experienceguided resampling. When the annealing period is too short, insufficient exposure to experience limits its effectiveness, whereas extending experience usage excessively yields diminishing returns as the policy becomes sufficiently optimized. We therefore set T = 60 to leverage experience in the early training stage while avoiding unnecessary intervention for later on-policy refinement. Component-wise analysis. Figure 4b presents a step-by-step ablation starting from the RL-optimized policy πRL . Introducing experience-guided resampling to πθ already yields a clear performance gain over πRL , indicating that injecting action-level experience during rollout is effective and can further improve upon the experience policy itself. Adding positive experience filtering leads to additional gains by stabilizing optimization and mitigating misleading gradients from incorrect experience. Finally, smoothed importance sampling provides a further boost, highlighting the importance of proper credit assignment when integrating experience-augmented trajectories. Together, these results demonstrate that the components of EAPO are complementary and jointly contribute to its overall performance improvements.
11
Experience Augmented Policy Optimization for LLM Reasoning
B. Algorithm of EAPO. Algorithm 1 Experience-Augmented Policy Optimization (EAPO) Require: Current policy πθ , Prior RL policy πRL , Group size G, Threshold τ , Block size K, Annealing step T . 1: Initialize πold ← πθ . 2: for each training step s = 1, 2, . . . do 3: // Phase 1: Rollout (On-policy Rollout + Experience-augmented Rollout) 4: Sample a batch of questions {q b }B b=1 ∼ D. 5: for each question q b in parallel do b 6: Sample G − 1 on-policy trajectories Oon = {ob1 , . . . , obG−1 } from πold . 7: if s ≤ T then 8: Generate experience-augmented trajectory obexp via Accelerated Block-wise Resampling: 9: while not EOS do b 10: Generate speculative block ỹt:t+K−1 ∼ πold . K−1 b 11: Compute discrepancies {δt+i }i=0 by comparing πold and πRL . b > τ then 12: if ∃i s.t. δt+i b b b 13: Find i∗ = min{i | δt+i > τ }, resample yt+i ∗ ∼ πRL , and set gt+i∗ = 1. b 14: Truncate and update prefix y<t+i . ∗ +1 15: else b = 0. 16: Accept block; set gt:t+K−1 17: end if 18: end while b 19: Collect group G b = Oon ∪ {obexp }. 20: else 21: Sample G trajectories from πold to form G b (Experience Annealing). 22: end if 23: end for 24: // Phase 2: Experience-aware Batched Optimization 25: for each group G b do Rib −mean(Rb ) b 26: Compute rewards {Rib }G . i=1 and group-relative advantages Ai = std(Rb ) b b 27: for each trajectory oi ∈ G do 28: if obi is the experience-augmented response obexp then 29: if R(obi , y b ) = 0 then 30: continue {// Positive Filtering: skip failed experience} 31: end if P b b 32: Compute ρbi = |o1b | t gi,t and π̃Exp = (1 − ρbi )πold + ρbi πRL . i
b 33: rib (θ) ← πθ (obi | q b )/π̃Exp (obi | q b ). 34: else 35: rib (θ) ← πθ (obi | q b )/πold (obi | q b ) {// Standard on-policy ratio} 36: end if 37: end for 38: end for 39: Update θ by maximizing JDAPO (θ) = Eb,i min(rib · Abi , clip(rib ) · Abi ) . 40: πold ← πθ . 41: end for
12