Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO

Published as a conference paper at COLM 2026

Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO Hyun Bin Park & Du-Seong Chang Department of Artificial Intelligence, Sogang University Seoul, Republic of Korea tjrtkroal@sogang.ac.kr duseong.chang@gmail.com

arXiv:2609.03941v1 [cs.LG] 3 Sep 2026

Abstract RL-based post-training for reasoning models is increasingly bottlenecked by repeated fresh rollout generation, particularly in agentic settings where environment interaction dominates wall-clock cost. Replay can reduce this burden by reusing past trajectories, but existing methods typically embed it within larger training pipelines involving exploration, experience restructuring, or mixed-policy optimization. This makes replay’s own contribution difficult to isolate. We ask a focused question: how far can principled replay selection alone go? We introduce Headroom-Drift Replay, a group-level replay control primitive for GRPO that separates reuse into two decisions. Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The fresh on-policy stream remains unchanged, and the method adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, this single intervention outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32. In Agentic Search, where environment interaction dominates cost, it delivers comparable quality at materially lower wall-clock time.

Figure 1: Summary of two representative views. (a) In mathematical reasoning, HeadroomDrift Replay leads on Avg Mean@32 over all baselines. (b) In agentic search, HeadroomDrift combines the highest Avg Best@32 with competitive wall-clock cost.

1

Published as a conference paper at COLM 2026

1

Introduction

Reinforcement learning (RL) has become a central ingredient in post-training language models for reasoning (Ouyang et al., 2022; Bai et al., 2022), but it often incurs substantial cost through repeated fresh rollout generation. This burden is especially high in GRPOstyle training and grows further in agentic settings, where interaction with external environments adds significant wall-clock overhead (Wang et al., 2026a; Liu et al., 2026a;b). Replay offers a natural remedy by reusing previously collected trajectories. Yet naive reuse is often insufficient for stable learning. The question is not whether to replay, but how to control replay itself: which stored experience should re-enter training, and under what conditions. A stored replay group can fail for two distinct reasons. It may still contain useful learning signal but have become too stale under the current policy. Conversely, it may remain close to the current policy while offering little room for further learning. Reliable replay therefore requires two separate judgments: whether a group still offers learning value and whether it remains compatible with the current policy. Prior work has shown that replay can improve sample efficiency in reasoning-oriented RL, but existing methods typically couple it with auxiliary mechanisms. This makes replay itself difficult to study as a standalone control problem. We therefore isolate replay-side control within GRPO (Shao et al., 2024; Guo et al., 2025) and ask how far it can go on its own. Headroom-Drift Replay is a group-level replay control primitive for GRPO built around separate judgments of learning value and current-policy compatibility. Headroom measures the remaining room to shift probability toward good actions and away from bad ones. Policy Drift measures how far the current policy has moved from the policy that generated the group and gates out stale groups. Reuse operates on full groups rather than individual responses, preserving the within-group comparison structure central to GRPO. Together, these two controls form the only addition to standard GRPO; the fresh on-policy stream remains unchanged. This design isolates the effect of replay-side control on training dynamics. Empirically, Headroom-Drift outperforms naive replay across mathematical reasoning, multimodal reasoning, and Agentic Search, and matches or exceeds methods that combine replay with additional training machinery. The strongest evidence comes from mathematical reasoning under the fullest baseline set. In Agentic Search, where environment interaction dominates wall-clock cost, the benefit takes a different form: Headroom-Drift outperforms on-policy scaling in both quality and per-step efficiency because the selected trajectories remain informative enough to substitute for costly fresh environment interaction. Multimodal reasoning shows that the same control principle extends beyond text-only settings. Our contributions are threefold: (i) We formulate replay in GRPO as a two-part control problem—retaining learning value while ensuring current-policy compatibility—and instantiate it in Headroom-Drift Replay, a group-level replay control primitive that requires no auxiliary training machinery. (ii) We construct role-aligned baselines that isolate distinct control questions—on-policy budget matching, fresh-data scaling, naive replay volume, strong non-replay alternatives, and broader replay methods with auxiliary machinery—and show across mathematical reasoning, Agentic Search, and multimodal reasoning that this primitive outperforms naive replay and matches or exceeds broader replay methods. (iii) We analyze how principled replay control behaves across training, covering samebuffer counterfactual comparisons, replay-age compatibility and lifetime exposure concentration, multi-reward ingress dynamics, and the relationship between replay and entropy collapse. 2

Published as a conference paper at COLM 2026

2

Related work

Classical reinforcement learning motivates the two control axes in our design. Prioritized Experience Replay showed that stored samples differ substantially in learning value, and that replay is more effective when guided by expected training utility rather than random selection (Schaul et al., 2016). A complementary line of off-policy correction methods addresses policy mismatch, showing that past experience can remain useful when reuse is appropriately weighted or truncated under the current policy (Munos et al., 2016; Espeholt et al., 2018). These works operate on transitions or trajectories in classical RL rather than on groups in LLM post-training, but they establish the same two requirements that drive our design: replay should account for both learning value and current-policy compatibility. Headroom provides a PER-style learning-value priority over full GRPO groups; Headroom-Drift applies this priority only among groups admitted by the Drift-based current-policy compatibility gate. In reasoning-oriented LLM training, replay has moved from a peripheral heuristic to a more central optimization component. RePO (Li et al., 2025), for example, incorporates replay into the GRPO training loop to improve sample efficiency beyond fully on-policy updates. This shift confirms that replay carries real value in modern reasoning settings where rollout generation is expensive. Several recent methods strengthen replay by coupling it with broader training mechanisms. EFRame (Wang et al., 2025) combines replay with exploration and filtering to determine which trajectories are retained and revisited. ExGRPO (Zhan et al., 2026) organizes reasoning trajectories by correctness and entropy signals before mixed-policy reuse. BAPO (Wan et al., 2026) adopts a buffer-centric off-policy design that pairs historical hard samples with adaptive batch construction. These methods reinforce the value of replay in modern reasoning RL, but because replay is embedded within larger pipelines, the two control questions central to our work—which stored groups still offer learning value, and which remain compatible with the current policy—are rarely made explicit as separable design choices. Our work treats these two questions as the primary intervention. The goal is not to replace broader replay-centric designs, but to provide a composable primitive within that larger design space: one that can stand alone and whose effects can be independently evaluated.

3

Method

3.1

Method overview.

Headroom-Drift Replay is a group-level replay control primitive for GRPO that decomposes reuse into two explicit control axes. Rather than treating reuse as a single inclusion decision, it separates learning-value prioritization from current-policy compatibility control through Headroom-based ranking and Policy-Drift gating. The minimum unit of reuse is the full group rather than individual responses, preserving the within-group comparison structure central to GRPO. The fresh on-policy stream is left untouched: the only modification to standard GRPO is that selected replay groups are added to the mixed batch. Any change in training dynamics therefore traces to replay-side control alone. 3.2

GRPO setup and replay state.

We first specify the GRPO state needed to express replay groups within the same update form as fresh on-policy groups. For a prompt Q, the current policy πθ generates n responses G1 , . . . , Gn , which together form a group g = ( Q, G1 , . . . , Gn ). Let T ( g) denote the set of token positions in the group, and let ai,j and si,j denote the j-th generated token in response Gi and its conditioning context. The current policy assigns token probability πθ ( ai,j | si,j ). GRPO computes rewards at the response level and derives a group-relative advantage Ai for each response Gi ; this same Ai is then applied to all token positions belonging to that response. 3

Published as a conference paper at COLM 2026

Figure 2: Headroom-Drift Replay in one GRPO step. The current policy generates fresh on-policy groups. Pre-existing buffered groups are ordered by Headroom and filtered by the Policy Drift gate; up to Krep accepted replay groups are merged with the fresh groups to form the mixed batch for the GRPO update. After the update, fresh groups with nonidentical rewards are appended to the FIFO replay buffer. What distinguishes replay from the fresh on-policy case is that each stored group is paired with the policy state under which it was originally generated. We denote this reference policy by πgen( g) . For a freshly generated group, πgen( g) coincides with the usual rollouttime old policy in GRPO. For a replayed group, however, it is the generation-time policy stored with that group in the buffer. Accordingly, the replay buffer maintains immutable per-group reference state: the stored actions, their generation-time log-probabilities under πgen( g) , and the response-level advantages Ai . This frozen reference state anchors each replayed group to the rollout under which it was originally collected. Using this reference state, both fresh and replayed groups can be handled within the same GRPO update form. For a group g, the token-level importance ratio is ri,j (θ; g) =

πθ ( ai,j | si,j ) , πgen( g) ( ai,j | si,j )

so the difference between fresh and replayed groups lies only in which reference policy appears in the denominator. 3.3

Headroom: learning-value prioritization.

Headroom provides the learning-value signal in our replay-side control primitive. It measures the remaining directional correction room of a stored group under a reference policy. Intuitively, replay is more useful when positively advantaged stored actions still have room for probability increase, or negatively advantaged stored actions still have room for probability decrease. For a policy π and a stored token position (i, j), define the token-level Headroom contribution by  1 − π ( ai,j | si,j ), Ai > 0, hi,j (π; g) = π ( ai,j | si,j ), Ai < 0,  0, Ai = 0, and aggregate it over the group as Headroom( g; π ) =

1 ∑ hi,j (π; g). |T ( g)| (i,j)∈T ( g)

When evaluated under the generation-time policy πgen( g) , Headroom defines the valueside replay priority stored for group g. 4

Published as a conference paper at COLM 2026

3.4

Policy drift: current-policy compatibility control.

Policy Drift provides the current-policy compatibility control in our replay-side control primitive. A stored group may still have high Headroom while already being too far from the current policy for reliable reuse (Munos et al., 2016; Espeholt et al., 2018). To measure this mismatch for a stored group g, define the token-level log-probability drift on the stored action by ∆i,j ( g; θ ) = log πθ ( ai,j | si,j ) − log πgen( g) ( ai,j | si,j ). We then aggregate these tokenwise deviations into a group-level Policy Drift score by averaging their squared values over the group: 1 Drift( g; θ ) = ∑ ∆i,j ( g; θ )2 . |T ( g)| (i,j)∈T ( g) A group is replayed only if its Policy Drift does not exceed a threshold τ. This choice is motivated by the role of Policy Drift as a replay-compatibility gate rather than a full distribution-distance objective. GRPO commonly monitors policy deviation through approximate-KL proxies (Guo et al., 2025; Yu et al., 2025) built from sampled log-ratios on generated tokens. For replay gating, however, what matters is mismatch magnitude on the stored token sequence, not a signed average: if some tokenwise shifts are positive while others are negative, they can offset each other in the average even when the overall sequence has moved substantially under the current policy. We therefore use a non-cancelling aggregation of tokenwise drift, and adopt a squared (L2-style) form so that larger local mismatches are penalized more strongly. An L1 variant yields a tighter formal bound but is less sensitive to concentrated per-token mismatch; in practice, the L2 gate achieves higher late-stage validation while filling less of the replay budget, indicating that it wins through better-compatible selection rather than replay volume (Appendix G.1). This squared aggregation also provides a formal link between the two control axes: when a group passes the Drift gate, √ its current-policy Headroom can deviate from the stored Headroom priority by at most τ, so the gate directly limits how much staleness can distort the stored Headroom priority (proof in Appendix A.3, Proposition 2). 3.5

One GRPO step with replay-side control

Phase A: Generate fresh groups and identify replay ingress candidates. At each training step, the current policy generates fresh grouped rollouts, and rewards and group-relative advantages are computed as in standard GRPO. Replay ingress candidates It are then identified from these fresh groups. Phase B: Order pre-existing buffered groups by Headroom. The pre-existing buffered groups are retrieved and ordered by Headroom, which defines the value-side priority for replay candidate scanning. Phase C: Apply the Policy Drift gate. The Headroom-ordered candidates are scanned under the current policy. For each scanned group, the model reevaluates the group via parallel teacher-forced log-probability computation, and admits the group for replay only if its Policy Drift does not exceed τ. Scanning stops once the replay budget Krep is filled or no further admissible groups remain. Phase D: Form the mixed batch and run the GRPO update. The accepted replay groups are merged with the fresh on-policy groups to form the mixed batch for the GRPO update. Phase E: Append valid ingress groups and maintain the FIFO buffer. To prevent samestep replay, the previously identified ingress candidates It are appended to the replay buffer only after the GRPO update. The buffer is maintained as a fixed-capacity FIFO queue, so once capacity is exceeded, the oldest groups are evicted regardless of Policy Drift or Headroom. Operational note. Replay requires no fresh autoregressive generation and no fresh environment interaction. Because reevaluation is teacher-forced on fixed stored token se5

Published as a conference paper at COLM 2026

Algorithm 1 H EADROOM -D RIFT R EPLAY as a replay-side control augmentation to standard GRPO. Blue lines mark replay-specific additions; the fresh on-policy rollout path remains unchanged. Input: current policy πθt , prompt batch Xt , replay buffer Bt , replay budget Krep , drift threshold τ 1: for each training step t do 2: Sample prompt batch Xt 3: Generate grouped fresh rollouts under πθt 4: Compute rewards and group-relative advantages for the fresh groups 5: + Identify mixed-outcome fresh groups as replay ingress candidates It 6: + Order pre-existing buffered groups by Headroom (descending) 7: for each candidate group g in Headroom order do 8: if |Rt | ≥ Krep then 9: break 10: end if 11: + Reevaluate the stored actions of g under πθt ▷ single forward pass on fixed trajectories 12: + Compute Policy Drift and refresh Headroom for g from the same log-probability pass 13: + if Policy Drift( g) ≤ τ then add g to Rt 14: end for 15: + Form mixed actor batch At ← Gton ∪ Rt 16: Run the standard GRPO update on At 17: + Append ingress groups It to Bt after selection ▷ no same-step replay 18: + If capacity is exceeded, evict the oldest groups from Bt (FIFO) 19: end for

quences, it can be parallelized over stored tokens. For efficiency, the current-policy logprobabilities computed for Policy Drift are reused to refresh Headroom in the same pass. Because scanning stops once the replay budget is filled, cached Headroom for unscanned groups can become stale. The two mechanisms therefore act complementarily: Policy Drift gating protects admission from overly stale groups, while same-pass Headroom refresh keeps the cached priority accurate for groups that are actually revisited. Appendix A makes the cached/reference/runtime Headroom distinction precise and formalizes replay selection in a form that directly mirrors the implementation above. Appendix F details the engineering choices required for stable and efficient execution, including frozen reference-state management, replay-prefix batch layout, and sequence-length balancing across GPUs.

4

Experiments

4.1

Experimental setup

We evaluate Headroom-Drift Replay in three GRPO-style settings with fixed, verifiable rewards: mathematical reasoning, Agentic Search, and multimodal reasoning. For mathematical reasoning, we use AIME24, AMC23, MATH500, Minerva, and OlympiadBench. For Agentic Search, we evaluate on NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle. For multimodal reasoning, we report a compact threebenchmark view covering Geometry3K (Geo3K), MathVista, and MathVision. We use Mean@32 as the headline metric for the main cross-domain comparisons. For their crossbenchmark summaries, we report Avg Mean@32. Best@32 and Weighted Mean@32 are reported in Appendix B for completeness. We additionally include a 7B Agentic Search scale check. Our baselines are chosen to answer distinct control questions, and we refer to them by role rather than by raw budget alone. GRPO on-policy matched keeps the fresh rollout budget aligned with replay and isolates the effect of replay reuse itself. GRPO on-policy larger tests whether replay gains can be explained by simple fresh-data scaling. GRPO + replay matched and, where available, GRPO + replay larger test whether buffered reuse alone is sufficient, or whether principled replay control is necessary. DAPO serves as a strong nonreplay comparator, and where directly comparable we also include broader replay/offpolicy methods such as ExGRPO and BAPO. In mathematical reasoning, we include the full comparison set. In multimodal reasoning, we report GRPO on-policy matched, GRPO on6

Published as a conference paper at COLM 2026

Figure 3: Cross-domain summary with Mean@32. policy larger, and DAPO in a simplified three-benchmark view. In Agentic Search, the main role-aligned comparisons are GRPO on-policy matched, GRPO on-policy larger, GRPO + replay matched, and DAPO. We do not include ExGRPO or BAPO in this setting because their published implementations target prompt-conditioned reasoning trajectories rather than externally interleaved tool-calling trajectories, and a like-for-like adaptation would require substantial re-engineering rather than a clean baseline comparison. Because replay cost varies across settings—negligible in single-turn mathematical reasoning but substantial in Agentic Search where environment interaction dominates wall-clock time—we interpret Mean@32 together with acquisition cost. Table 1 reports mean per-step wall-clock time as the main cost-side comparator. 4.2

Evaluation across domains

4.2.1

Mathematical reasoning

Figure 3 reports the mathematical-reasoning held-out results under the fullest comparison set. Headroom-Drift leads on Avg Mean@32 across all baselines. Relative to GRPO on-policy matched, the improvement confirms that principled replay adds value beyond purely on-policy training at matched fresh-rollout budget. Relative to GRPO on-policy larger, Headroom-Drift still leads despite using fewer fresh responses, ruling out simple fresh-data scaling as an explanation. Against GRPO + replay matched and GRPO + replay larger, the margin widens further, indicating that replay quality depends on principled subset selection rather than on replay volume. Among the broader comparators, DAPO, ExGRPO, and BAPO are all surpassed on Avg Mean@32; this is the one domain where the primitive matches or exceeds every baseline under a single headline metric. Figure 4 provides supporting training-dynamics evidence. The replay trajectory stays above GRPO on-policy matched over a broad mid-to-late window and tracks above GRPO on-policy larger through much of the same range, indicating that the Avg Mean@32 advantage is reflected in the training trajectory rather than confined to a single checkpoint. A training-score-matched analysis in Appendix E further shows that replay delays the onset of entropy collapse: at comparable score levels, the replay run enters the low-entropy regime later and remains there longer than GRPO on-policy larger, suggesting that principled replay contributes to more sustained learning dynamics. 4.2.2

Agentic search

Agentic search serves a different role from mathematical reasoning: the main question is not the largest standalone gain in Mean@32, but whether replay-side control remains worthwhile when training is dominated by multi-turn environment interaction cost. Against naive replay baselines, the picture depends on the metric. On the headline Avg Mean@32, Headroom-Drift leads GRPO + replay matched by only 0.0029 (0.3577 vs. 0.3548), a gap within noise range. But on Avg Best@32, the margin is substantially larger 7

Published as a conference paper at COLM 2026

Figure 4: Training-score trajectories on mathematical reasoning.

(0.4879 vs. 0.4623, as shown in Figure 1(b)), suggesting that principled selection shifts the upper tail of trajectory quality more than it shifts the average. As an additional scale check, we conduct a controlled Search-R1 comparison between Headroom-Drift and GRPO + replay matched with Qwen2.5-7B-Instruct. To make 7B training and agentic evaluation feasible within the available compute budget, both methods use an 8-bit optimizer and we evaluate four samples per input; we therefore report Mean@4 and Best@4 rather than their @32 counterparts. Under this matched setup, Headroom-Drift improves Avg Mean@4 from 0.3737 to 0.3955 and Weighted Mean@4 from 0.4158 to 0.4298, leading on six of seven benchmarks. Full per-benchmark Mean@4 and Best@4 results are reported in Appendix B, Table 6. Method

Step Time (s)

GRPO on-policy matched (b128) GRPO + replay matched (b128+r64) Headroom-Drift (b128+r64) GRPO + replay larger (b128+r128) GRPO on-policy larger (b192) DAPO

146.4 154.4 166.3 187.3 197.2 232.5

Table 1: Mean per-step wall-clock time on agentic search (Search-R1).

The more revealing comparison is against on-policy scaling. GRPO on-policy larger uses 1.5× the fresh rollout budget yet achieves lower Avg Mean@32 (0.3212 vs. 0.3577) at higher per-step cost (197.2 s vs. 166.3 s). Headroom-Drift surpasses it on both axes because principled selection substitutes reevaluation of stored trajectories for costly fresh environment interaction (Table 1). This is a Pareto improvement over on-policy scaling. 4.2.3

Multimodal reasoning

Multimodal reasoning serves as breadth evidence. The question is whether the same replay-side control principle generalizes beyond text-only settings. In this simplified threebenchmark view, Headroom-Drift leads on Avg Mean@32 over GRPO on-policy matched, GRPO on-policy larger, and DAPO (0.4137 vs. 0.3986, 0.4005, and 0.4056 respectively). 8

Published as a conference paper at COLM 2026

Figure 5: (A) Same-buffer counterfactual against GRPO + replay. (B) Initial stored Headroom and lifetime replay. Bars show mean replay count Nrep , and the line shows P( Nrep ≥ 1). (C) Conceptual view of replay source staleness under Headroom-Drift versus GRPO + replay.

4.3

Mechanism analysis of replay selection

We now ask whether the two control axes operate as designed. Figure 5 examines this question through three complementary views: step-level subset quality, lifetime reuse concentration, and temporal diversity of replay sources. Figure 5(a) compares, at each training step, the replay subset actually selected by Headroom-Drift against the subset that GRPO + replay would select by recency alone, on matched pre-step buffer states and matched subset sizes. Each point plots the compatibility gain (x: positive means our subset has lower Policy Drift) against the learningvalue gain (y: positive means higher current-policy Headroom). The majority of steps fall in the jointly improved quadrant, confirming systematic advantage on both axes. The gain is asymmetric: policy compatibility improvement is consistent, while learning-value improvement weakens in late-stage steps where the admissible pool contracts. This pattern is expected—Policy Drift acts as a hard gate, while Headroom prioritizes within the feasible set that the gate admits. Figure 5(b) shows that these step-level decisions accumulate into a sharp lifetime reuse pattern. Groups with initial stored Headroom below 0.8 are rarely replayed (mean Nrep < 0.5; replay probability < 24%), while groups above 0.8 are replayed frequently (mean Nrep ≈ 2.0; > 56%). Headroom ranking preferentially surfaces high-value groups, but lower-Headroom groups are not excluded; they still enter replay when the Policy Drift gate judges them sufficiently compatible. Figure 5(c) reveals the temporal consequence. GRPO + replay draws exclusively from the immediately preceding step (100% at age 1), since each step’s buffer ingress exceeds the replay target. Headroom-Drift distributes replay across ages 1–7, with 44% coming from sources three or more steps back. This multi-age mixture arises because Headroom ranking surfaces older high-value groups that still pass the Policy Drift gate, producing a temporally diverse corrective signal rather than uniform one-step reuse. In practice, we calibrate the Drift threshold τ with a short log-spaced sweep instead of a fine-grained full-training search. On Search-R1, three decade-spaced settings yield mean replay acceptance rates of 6.4%, 40.2%, and 94.4%; only the middle setting filters selectively while keeping replay KL stable. Collapsing acceptance under a tight threshold and rising replay KL under a loose one provide early signals for ruling out unsuitable values without running every candidate to completion. Across the evaluated runs, the selected values fall into two task-family settings, and the Search-R1 threshold is reused from 3B to 7B without scale-specific retuning. Appendix G.2 reports the exact thresholds and full analysis. 9

Published as a conference paper at COLM 2026

Appendix D provides further empirical detail on these reuse patterns. Three findings stand out: age alone does not determine replay compatibility, as the Drift gate rejects a nonnegligible fraction of even the youngest groups; scan depth grows over training as the admissible set contracts; and most buffered groups are never replayed, while a small repeatedly reused subset accounts for the majority of total replay exposure. A component ablation supports the role of both axes. On MATH-500, the full Headroom-Drift combination achieves the highest Mean@32, followed by Headroom-only (the PER-style priorityonly analogue) and then Drift-only (Appendix G.3, Table 17).

5

Conclusion and future directions

We introduced Headroom-Drift Replay, a replay-side control primitive that decomposes reuse into learning-value prioritization (Headroom) and current-policy compatibility control (Drift). Across mathematical reasoning, agentic search, and multimodal reasoning, this single intervention outperforms naive replay. In mathematical reasoning under the fullest baseline set, it matches or exceeds broader replay methods on Avg Mean@32. In agentic search, it delivers a Pareto improvement over on-policy scaling. Replay, when controlled along the right axes, functions as a capable optimization primitive in its own right. We studied this primitive in isolated form, but its two explicit control axes are designed to compose with the exploration, filtering, and adaptive batch construction strategies in recent replay-centric methods (Wang et al., 2025; Zhan et al., 2026; Wan et al., 2026). The primitive does not compete with these methods; it provides a separable control layer within them. Preliminary training-score evidence on a CISPO-style objective suggests that the two-axis decomposition transfers beyond GRPO (Appendix H), though held-out evaluation across objective families, including PPO-, GSPO-, and CISPO-style formulations (Zheng et al., 2025; MiniMax et al., 2025; Yuan et al., 2026; Xi et al., 2026), remains open.

10

Published as a conference paper at COLM 2026

Acknowledgments This research was conducted as part of the Sovereign AI Foundation Model Project (GPU Track), organized by the Ministry of Science and ICT (MSIT) and supported by the National IT Industry Promotion Agency (NIPA), S. Korea (PJT-26-010017). This research was supported by the MSIT (Ministry of Science, ICT), Korea, under the Top-Tier AI Global HRD invitation program (RS-2025-25461932) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation).

Ethics Statement This work studies replay selection mechanisms for reinforcement-learning post-training of reasoning models. The experiments use standard benchmark datasets and simulated training environments; no new human-subject data were collected. Potential risks include misuse of improved reasoning capability and increased compute consumption in large-scale training. To mitigate overclaiming, we report results with explicit scope limits and emphasize quality–cost trade-offs under the evaluated settings. During camera-ready preparation, the authors used a large language model to revise prose and assist with drafting. The authors reviewed the resulting text and take responsibility for all content in the paper.

References Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al. Constitutional AI: Harmlessness from AI feedback, 2022. URL https://arxiv.org/abs/2212.08073. Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actorlearner architectures. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1407–1416, 2018. URL https://proceedings.mlr.press/v80/espeholt18a.html. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633–638, 2025. doi: 10.1038/ s41586-025-09422-z. URL https://www.nature.com/articles/s41586-025-09422-z. Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang, and Chaochao Lu. RePO: Replay-enhanced policy optimization, 2025. URL https://arxiv.org/abs/2506.09340. Ben Lipkin, Aleksei Petrenko, Kevin Chen, Erik Wijmans, Marco Cusumano-Towner, Raja Giryes, and Philipp Krähenbühl. Entropy-preserving reinforcement learning. In International Conference on Learning Representations (ICLR), 2026. doi: 10.48550/arXiv.2603.11682. URL https://iclr.cc/virtual/2026/poster/10010707. Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Jianbin Jiao, and Junge Zhang. Agentic reinforcement learning with implicit step rewards. In International Conference on Learning Representations (ICLR), 2026a. doi: 10.48550/arXiv.2509.19199. URL https://iclr.cc/virtual/2026/poster/10007373. Zichen Liu, Anya Sims, Keyu Duan, Changyu Chen, Simon Yu, Xiangxin Zhou, Haotian Xu, Shaopan Xiong, Bo Liu, Chenmien Tan, Weixun Wang, Hao Zhu, Weiyan Shi, Diyi Yang, Michael Qizhe Shieh, Yee Whye Teh, Wee Sun Lee, and Min Lin. GEM: A gym for generalist LLMs. In International Conference on Learning Representations (ICLR), 2026b. URL https://openreview.net/forum?id=vsqQ1lG52a. Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, and Dacheng Tao. The energy loss phenomenon in RLHF: A new perspective on mitigating reward hacking. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 44076–44105, 2025. URL https://proceedings. mlr.press/v267/miao25c.html. 11

Published as a conference paper at COLM 2026

MiniMax, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, et al. MiniMax-M1: Scaling test-time compute efficiently with lightning attention, 2025. URL https://arxiv.org/abs/2506.13585. Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, volume 29, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/ c3992e9a68c5ae12bd18488bc579b30d-Abstract.html. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pp. 27730–27744, 2022. doi: 10.52202/ 068431-2011. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ b1efde53be364a73914f58805a001731-Abstract-Conference.html. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations (ICLR), 2016. doi: 10.48550/ arXiv.1511.05952. URL https://arxiv.org/abs/1511.05952. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300. Xu Wan, Yansheng Wang, Wenqi Huang, and Mingyang Sun. Buffer Matters: Unleashing the power of off-policy reinforcement learning in large language model reasoning. In International Conference on Learning Representations (ICLR), 2026. doi: 10.48550/arXiv.2602. 20722. URL https://iclr.cc/virtual/2026/poster/10009473. Chen Wang, Lai Wei, Yanzhi Zhang, Chenyang Shao, Zedong Dan, Weiran Huang, Yuzhi Zhang, and Yue Wang. EFRame: Deeper reasoning via exploration-filter-replay reinforcement learning framework, 2025. URL https://arxiv.org/abs/2506.22200. Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, and Zhenzhe Ying. Information gain-based policy optimization: A simple and effective approach for multi-turn LLM agents. In International Conference on Learning Representations (ICLR), 2026a. doi: 10.48550/arXiv.2510.14967. URL https://iclr.cc/virtual/ 2026/poster/10007215. Shumin Wang, Yuexiang Xie, Wenhao Zhang, Yuchang Sun, Yanxi Chen, Yaliang Li, and Yanyong Zhang. On the entropy dynamics in reinforcement fine-tuning of large language models, 2026b. URL https://arxiv.org/abs/2602.03392. Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, Xun Deng, Zhikai Lei, Miao Zheng, Guoteng Wang, Shuo Zhang, Peng Sun, Rui Zheng, Hang Yan, Tao Gui, Qi Zhang, and Xuanjing Huang. BAPO: Stabilizing off-policy reinforcement learning for LLMs via balanced policy optimization with adaptive clipping. In International Conference on Learning Representations (ICLR), 2026. URL https://openreview.net/forum?id=jIeJJqG7dz. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. In Advances in Neural Information Processing Systems, volume 38, pp. 113222–113244, 2025. doi: 10.52202/ 085713-3775. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/ a4277440d50f1f15d2cb4c14f7e0c0d2-Abstract-Conference.html. Dun Yuan, Di Wu, and Xue Liu. Escaping policy contraction: Contraction-aware PPO (CaPPO) for stable language model fine-tuning. In International Conference on Learning Representations (ICLR), 2026. URL https://openreview.net/forum?id=vDlkJewkDu. 12

Published as a conference paper at COLM 2026

Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F. Wong, and Yu Cheng. ExGRPO: Learning to reason from prior successes. In International Conference on Learning Representations (ICLR), 2026. doi: 10.48550/arXiv.2510.02245. URL https: //iclr.cc/virtual/2026/poster/10011339. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization, 2025. URL https://arxiv.org/abs/2507.18071.

13

Published as a conference paper at COLM 2026

A

Formal interpretation, mixed-batch objective, and replay-step formalization

This appendix makes precise the quantities and replay-step construction used by Headroom-Drift Replay. We first fix replay-state conventions, including the distinction between reference, cached, current-policy, and runtime Headroom. We then formalize the mixed-batch objective, define Headroom and Policy Drift, prove a reference-to-current Headroom mismatch bound, introduce lifetime replay exposure quantities, and finally write the replay-augmented GRPO step in set notation. A.1

Notation and replay-state conventions

For any buffered group g and step t, we distinguish four Headroom-related quantities. The reference Headroom is the immutable generation-time score H ref ( g) := Headroom( g; πgen( g) ).

(1)

The current-policy Headroom is the conceptual value of the same policy-conditioned quantity evaluated at the current policy, Htcur ( g) := Headroom( g; πθt ).

(2)

b t ( g) is the priority stored in the buffer at the beginning of step t and The cached Headroom H used to order replay candidates: b t ( g) := Headroom( g; πθ ), H r ( g) t

(3)

where rt ( g) is the last step at which the cached priority of g was refreshed. Finally, for a buffered group g that is actually scanned under the current policy at step t, the runtime Headroom is Htrun ( g) := Headroom( g; πθt ). (4) Thus, whenever g is scanned at step t, we have Htrun ( g) = Htcur ( g).

(5)

At insertion time, the cached priority is initialized at the generation-time value, so a newly b t ( g) = H ref ( g). If a buffered group is scanned at step t, then its buffered group starts with H cached priority is refreshed after selection using the runtime value from the same currentpolicy reevaluation pass; otherwise the cached value remains unchanged. Consequently, b t ( g), not by a freshly recomputed currentcurrent-step replay ordering is governed by H policy Headroom over the entire buffer. A.2

Mixed-batch objective as constrained replay augmentation

Let Gton denote the fresh on-policy groups generated at step t, and let Rt denote the accepted replay groups selected from the pre-existing buffer. The mixed actor batch is

At = Gton ⊎ Rt ,

(6)

where ⊎ denotes disjoint union. For any group g, let

ℓGRPO g; θ, πgen( g)

(7)

be the usual group-level GRPO objective evaluated with the generation-time reference policy attached to g. The one-step mixed-batch objective is then

Lmix t (θ ) =

1 ℓGRPO g; θ, πgen( g) . |At | g∑ ∈A t

14

(8)

Published as a conference paper at COLM 2026

Symbol

Type

Meaning Core GRPO and replay state

g T ( g) Ai

group index set scalar

πgen( g)

policy

π θt t0 ( g ) aget ( g)

policy integer scalar

grouped rollout associated with one prompt token positions contained in group g response-level group-relative advantage for response i generation-time reference policy that originally produced group g current policy at training step t step at which group g is freshly generated age of buffered group g at step t, defined as t − t0 ( g) Headroom and Policy Drift

Headroom( g; π )

scalar score

H ref ( g) Htcur ( g)

scalar score scalar score

b t ( g) H

scalar score

Htrun ( g)

scalar score

rt ( g)

integer

∆i,j ( g; θt )

scalar

Drift( g; θt )

scalar score

policy-conditioned remaining directional correction room of group g under policy π reference Headroom Headroom( g; πgen( g) ) conceptual current-policy Headroom Headroom( g; πθt ) cached Headroom priority stored in the buffer at the beginning of step t and used for buffer ordering runtime Headroom actually computed for a group scanned under the current policy at step t last refresh step index of the cached Headroom of g, b t ( g) = Headroom( g; πθ ) so that H rt ( g) token-level stored-action log-probability drift under the current policy group-level Policy Drift, i.e., squared-average tokenwise log-probability mismatch relative to πgen( g) Replay-step notation

Bt Gton It

ordered queue set set

St

ordered set

Ft (τ )

set

Rt At

set batch-level collection scalar scalar scalar

Krep τ C

replay buffer at the beginning of step t freshly generated on-policy groups at step t ingress candidate set from Gton determined by the task-specific ingress rule Headroom-ordered scan prefix actually reevaluated at step t Drift-feasible buffered groups satisfying Drift( g; θt ) ≤ τ accepted replay groups at step t mixed actor batch combining fresh on-policy groups and accepted replay groups replay budget per training step Policy Drift threshold replay-buffer capacity Lifetime reuse and exposure

Rt ( g) rep NT ( g) use NT ( g)

indicator scalar scalar

WT ( g)

scalar

equals 1 if group g is replayed at step t, else 0 cumulative replay count of group g up to step T total training uses of group g up to step T, including its initial fresh use cumulative mixed-objective exposure weight of group g up to step T

Table 2: Notation used in Appendix A. The key state distinction is between reference Headb t ( g) used for current-step ordering, conceptual currentroom H ref ( g), cached Headroom H policy Headroom Htcur ( g), and runtime Headroom Htrun ( g) obtained only for groups actually scanned at step t.

15

Published as a conference paper at COLM 2026

Define the fresh and replay parts separately as

Lon t (θ ) =

1 ℓGRPO g; θ, πgen( g) , ∑ on |Gt | g∈G on

(9)

t

rep

Lt (θ ) =

1 ℓGRPO g; θ, πgen( g) ∑ |Rt | g∈R

(|Rt | > 0),

(10)

t

with αt :=

|Gton | . |At |

Proposition 1 (Exact mixed-objective decomposition).

(11) If |Rt | > 0, then rep

on Lmix t ( θ ) = α t L t ( θ ) + (1 − α t ) L t ( θ ).

(12)

on If |Rt | = 0, then Lmix t ( θ ) = L t ( θ ).

Proof. The identity follows immediately by splitting the sum in (8) over the disjoint union (6) and normalizing by |At |. This decomposition makes explicit that replay augments rather than replaces the fresh onpolicy stream: the fresh rollout path remains intact, while replay contributes an additional constrained term determined by buffer-side selection. A.3

Headroom and policy drift

For a stored group g and any policy π, define the token-level Headroom contribution as  1 − π ( ai,j | si,j ), Ai > 0, hi,j (π; g) = π ( ai,j | si,j ), (13) Ai < 0,  0, Ai = 0, and the group-level Headroom as Headroom( g; π ) =

1 ∑ hi,j (π; g). |T ( g)| (i,j)∈T ( g)

(14)

Thus Headroom is a policy-conditioned measure of remaining directional correction room on the stored action sequence. The reference Headroom H ref ( g) is obtained by evaluating (14) at the immutable generation-time policy. To measure current-policy mismatch on the stored actions, define the token-level logprobability drift ∆i,j ( g; θt ) := log πθt ( ai,j | si,j ) − log πgen( g) ( ai,j | si,j ),

(15)

and the group-level Policy Drift Drift( g; θt ) :=

1 ∑ ∆i,j ( g; θt )2 . |T ( g)| (i,j)∈T ( g)

(16)

Replay admissibility is controlled by the hard gate Drift( g; θt ) ≤ τ. Remark (Policy Drift as a sampled log-ratio magnitude signal). The term ∆i,j ( g; θt ) is the tokenwise sampled log-ratio on the stored action sequence underlying approximateKL-style diagnostics in GRPO-like training. For replay gating, what matters is mismatch magnitude on the stored sequence rather than a signed average that can cancel across tokens. This is why Policy Drift uses the non-cancelling squared aggregation in (16). 16

Published as a conference paper at COLM 2026

Proposition 2 (Policy-Drift control of Headroom mismatch). For any stored group g and current policy πθt , q Htcur ( g) − H ref ( g) ≤ Drift( g; θt ). (17) Proof. By the definitions of Htcur ( g) and H ref ( g), Htcur ( g) − H ref ( g) ≤

1 ∑ hi,j (πθt ; g) − hi,j (πgen(g) ; g) . |T ( g)| (i,j)∈T ( g)

For any token (i, j), the case definition in (13) implies hi,j (πθt ; g) − hi,j (πgen( g) ; g) ≤ πθt ( ai,j | si,j ) − πgen( g) ( ai,j | si,j ) . Now write x = log πθt ( ai,j | si,j ),

y = log πgen( g) ( ai,j | si,j ). u Because x, y ≤ 0 and the derivative of u 7→ e is at most 1 on (−∞, 0], the exponential map is 1-Lipschitz on this domain. Hence, πθt ( ai,j | si,j ) − πgen( g) ( ai,j | si,j ) ≤ | x − y| = ∆i,j ( g; θt ) . Substituting this bound and applying Cauchy–Schwarz gives 1 Htcur ( g) − H ref ( g) ≤ ∑ ∆i,j ( g; θt ) |T ( g)| (i,j)∈T ( g) v u q u 1 2 = ≤t ∆ ( g; θ ) Drift( g; θt ). t i,j ∑ |T ( g)| (i,j)∈T ( g) This proves (17). Let g and h be two stored groups. If q q H ref ( g) − H ref (h) > Drift( g; θt ) + Drift(h; θt ),

Corollary 2.1 (Ranking certificate).

(18)

then

Htcur ( g) > Htcur (h). (19) Thus a sufficiently large margin in reference Headroom guarantees preservation of the current-policy Headroom ordering. Corollary 2.2 (Drift-gated Headroom distortion bound). If a replayed group g passes the Policy Drift gate at step t, i.e., if g ∈ Rt , then √ (20) Htcur ( g) − H ref ( g) ≤ τ. Equivalently,

√ Htcur ( g) ≥ H ref ( g) − τ. (21) This gives a conservative lower bound on current-policy Headroom for accepted replay groups. Remark (reference Headroom versus cached and runtime Headroom). Proposition 2 and Corollaries 2.1–2.2 are stated against the immutable reference quantity H ref ( g) and the conceptual current-policy quantity Htcur ( g) because Policy Drift is defined relative to the generation-time policy. The buffer ordering actually used at step t is determined by the b t ( g), which is initialized at H ref ( g) and refreshed opportunistically only cached priority H when a group is scanned. For scanned groups, the runtime value equals the conceptual current-policy value by (5). Since current-step selection does not recompute current-policy Headroom for the entire buffer, Proposition 2 should be read as a principled control of reference-to-current mismatch, not as a direct theorem about a uniformly refreshed buffer cache. 17

Published as a conference paper at COLM 2026

A.4

Replay multiplicity and lifetime exposure

A buffered group can contribute to training more than once: it is used once as a fresh onpolicy group at generation time, and may later re-enter training multiple times via replay. To make this explicit, define the replay indicator R t ( g ) : = 1 [ g ∈ R t ],

(22)

which equals 1 if group g is replayed at step t and 0 otherwise. The cumulative replay count of group g up to step T is T

∑

rep

NT ( g) :=

R t ( g ),

(23)

t=t0 ( g)+1

where the sum starts at t0 ( g) + 1 because same-step replay is disallowed. The corresponding total training-use count is rep

NTuse ( g) := 1 + NT ( g),

(24)

where the leading 1 accounts for the initial fresh on-policy use at generation time. To connect replay reuse to the mixed-batch objective, define the cumulative mixedobjective exposure weight 1[ g ∈ At ] WT ( g) := ∑ . (25) |At | t≤ T This quantity records how much objective mass group g receives over time under the stepwise mixed-batch objective. These definitions make explicit that replay selection induces a structured reuse process over buffered groups. A group’s lifetime replay count and cumulative exposure are jointly shaped by (i) the ingress rule that determines whether it enters the buffer, (ii) finite FIFO residence, (iii) Headroom-ordered prefix truncation during scanning, and (iv) Policy Drift admissibility under the current policy. In later appendices, these quantities are summarized empirically through replay age, replay count, and scan-prefix statistics. A.5

Replay-step formalization

Let It ⊆ Gton denote the ingress candidate set produced by the task-appropriate ingress rule at step t. Replay selection operates only on the pre-existing buffer Bt ; current-step ingress is appended only after replay selection and the GRPO update. Let

(1)

(2)

(|Bt |)

Ct = Sort↓ Hbt (Bt ) = gt , gt , . . . , gt

(26)

be the buffer ordered by descending cached Headroom at the beginning of step t. Define the Drift-feasible buffered subset

Ft (τ ) := { g ∈ Bt | Drift( g; θt ) ≤ τ }.

(27)

bt : The accepted replay set is the top-Krep subset of Ft (τ ) under the ordering induced by H
bt Rt := TopK↓KH Ft (τ ) . rep

(28)

By construction,

|Rt | ≤ Krep ,

∀ g ∈ Rt , Drift( g; θt ) ≤ τ.

(29)

Proposition 3 (Equivalent constrained characterization of replay selection). At each training step t, the accepted replay set is equivalently characterized as a solution of

Rt ∈ arg

max

∑ Hbt ( g).

S⊆Ft (τ ), |S|≤Krep g∈S

18

(30)

Published as a conference paper at COLM 2026

That is, replay selection chooses the cardinality-constrained subset with maximum total cached Headroom among the Drift-feasible candidates. b t and taking the first at most Krep elements is exactly Proof. Ordering Ft (τ ) by descending H the solution of (30). For the implementation-aware view, define the cumulative admissible count over the ordered buffer prefix as m   (u) ct (m) := ∑ 1 Drift( gt ; θt ) ≤ τ . (31) u =1

Let the stopping index be  min{m : ct (m) = Krep }, if such an m exists, ⋆ mt := | B t |, otherwise.

(32)

Then the actual scan prefix is

 (1) (m⋆ ) S t : = gt , . . . , gt t , and the accepted replay set can equivalently be written as  Rt = g ∈ St : Drift( g; θt ) ≤ τ .

(33) (34)

This makes explicit the Headroom-ordered prefix truncation induced jointly by cached priority ordering and the replay budget Krep . For every scanned group g ∈ St , the same current-policy reevaluation pass used to compute Drift( g; θt ) also yields the runtime Headroom Htrun ( g) = Htcur ( g). The cached priority is then updated after selection according to  run H ( g ), g ∈ S t , + b (35) Ht ( g) := b t Ht ( g), g ∈ Bt \ St . Thus runtime Headroom affects future candidate ordering through cache maintenance, but it does not retroactively alter the current-step selection order. The mixed actor batch used by the GRPO optimizer is then

At = Gton ⊎ Rt ,

(36)

and the step objective is the mixed-batch objective in (8). Finally, the updated cache state is appended with the current-step ingress groups and maintained under FIFO capacity C:
Bt+1 = FIFOAppendC B+ (37) t , Ord(It ) , where B+ t denotes the ordered buffer with refreshed cached priorities from (35), and each newly appended ingress group g ∈ It is initialized with cached Headroom b t+1 ( g) = H ref ( g). H

(38)

Because appending happens only after selection, groups generated at step t cannot be replayed in the same step.

B

Detailed benchmark tables

B.1

Metric definitions

Let each input instance produce n sampled responses with scalar scores in [0, 1]. • Best@n: for each input, take the maximum score among its n samples, then average over inputs. • Mean@n: for each input, average the n sample scores, then average over inputs. 19

Published as a conference paper at COLM 2026

Metric

GRPO on-pol. matched (b256)

GRPO on-pol. larger (b384)

DAPO

ExGRPO

AIME24↑ 0.3083 / 0.0888 0.2833 / 0.0833 0.2750 / 0.0966 0.3417 / 0.0758 AMC23↑ 0.8000 / 0.4531 0.8500 / 0.4633 0.8500 / 0.5023 0.8750 / 0.4469 MATH500↑ 0.8760 / 0.6927 0.8800 / 0.6895 0.9100 / 0.6944 0.9060 / 0.6701 Minerva↑ 0.3529 / 0.1994 0.3566 / 0.1957 0.3824 / 0.1790 0.3934 / 0.1545 OlympiadBench↑ 0.4095 / 0.2437 0.4095 / 0.2403 0.4347 / 0.2319 0.4318 / 0.2220 Macro Avg↑ 0.5494 / 0.3356 0.5559 / 0.3344 0.5704 / 0.3409 0.5896 / 0.3139 Weighted Avg↑ 0.5473 / 0.3696 0.5486 / 0.3664 0.5722 / 0.3636 0.5772 / 0.3448 Fresh Responses↓ 1,843,200 2,764,800 2,445,312 1,785,600

Metric

BAPO

GRPO+r matched (b256+r128)

GRPO+r larger (b256+r256)

AIME24↑ 0.3083 / 0.0891 0.1417 / 0.0813 0.1333 / 0.0792 AMC23↑ 0.9000 / 0.5117 0.7000 / 0.5312 0.6250 / 0.4813 MATH500↑ 0.9060 / 0.7026 0.8200 / 0.7165 0.8160 / 0.7160 Minerva↑ 0.3971 / 0.1911 0.2721 / 0.1976 0.2721 / 0.1976 OlympiadBench↑ 0.4362 / 0.2400 0.3323 / 0.2459 0.3234 / 0.2474 Macro Avg↑ 0.5895 / 0.3469 0.4166 / 0.3117 0.3922 / 0.3053 Weighted Avg↑ 0.5778 / 0.3712 0.4525 / 0.3595 0.4421 / 0.3595 Fresh 1,843,200 Responses↓ 2,027,520 1,843,200

Headroom-Drift 0.2750 / 0.0943 0.8750 / 0.5117 0.8820 / 0.7104 0.3456 / 0.2022 0.4050 / 0.2477 0.5565 / 0.3533 0.5455 / 0.3792 1,843,200

Table 3: Held-out benchmark results on mathematical reasoning. Each cell reports Best@32 / Mean@32. Arrows indicate metric direction (↑ higher is better, ↓ lower is better). Top-1 values are bold, and Top-2 values are underlined. GRPO + replay matched (b256+r128) and GRPO + replay larger (b256+r256) denote recency-only replay using the latest 128/256 buffered groups without Headroom ranking or Policy Drift gating.

Metric

GRPO on-pol. matched (b128)

GRPO on-pol. larger (b192)

DAPO

Headroom-Drift

Geo3K↑ MathVista↑ MathVision↑ Macro Avg↑ Weighted Avg↑

0.5870 / 0.4927 0.7600 / 0.5695 0.3727 / 0.1336 0.5732 / 0.3986 0.4839 / 0.2740

0.5753 / 0.4864 0.7740 / 0.5747 0.3865 / 0.1404 0.5786 / 0.4005 0.4945 / 0.2788

0.5827 / 0.4952 0.7720 / 0.5860 0.3852 / 0.1357 0.5800 / 0.4056 0.4941 / 0.2793

0.6245 / 0.5111 0.7720 / 0.5825 0.4086 / 0.1475 0.6017 / 0.4137 0.5148 / 0.2883

Table 4: Multimodal benchmark results (simplified 3-benchmark view). Each cell reports Best@32 / Mean@32. Top-1 values are bold, and Top-2 values are underlined. • Macro Avg Mean@n: unweighted average of benchmark-level Mean@n values across benchmarks. • Weighted Mean@n: weighted average of benchmark-level Mean@n values, with benchmark weights proportional to benchmark size. Unless noted otherwise, the paper’s Avg Mean@n notation is identical to Macro Avg Mean@n. As shown in Table 1, Headroom-Drift achieves a lower per-step time than GRPO on-policy larger (b192) despite including replay-selection overhead, suggesting that in agentic set20

Published as a conference paper at COLM 2026

GRPO on-pol. matched (b128)

Metric

GRPO on-pol. larger (b192)

DAPO

NQ↑ 0.4908 / 0.4026 0.5020 / 0.4152 0.4548 / 0.3461 TriviaQA↑ 0.6569 / 0.5711 0.6420 / 0.5626 0.6533 / 0.5488 PopQA↑ 0.4885 / 0.4055 0.4907 / 0.4196 0.4786 / 0.3916 HotpotQA↑ 0.4013 / 0.2895 0.4050 / 0.3017 0.3906 / 0.2684 2WikiMultiHopQA↑ 0.3913 / 0.2312 0.3702 / 0.2306 0.3877 / 0.2200 Musique↑ 0.1528 / 0.0890 0.1546 / 0.0909 0.1483 / 0.0838 Bamboogle↑ 0.3370 / 0.2480 0.3477 / 0.2280 0.3136 / 0.1985 Macro Avg↑ 0.4169 / 0.3196 0.4160 / 0.3212 0.4038 / 0.2939 Weighted Avg↑ 0.4733 / 0.3674 0.4669 / 0.3719 0.4646 / 0.3486 Fresh 309,248 Responses↓ 204,800 307,200 GRPO+r matched (b128+r64)

Metric

GRPO+r larger (b128+r128)

NQ↑ 0.5319 / 0.4496 0.5507 / 0.4413 TriviaQA↑ 0.6785 / 0.5984 0.6856 / 0.5903 PopQA↑ 0.5114 / 0.4339 0.5221 / 0.4239 HotpotQA↑ 0.4539 / 0.3321 0.4771 / 0.3407 2WikiMultiHopQA↑ 0.4862 / 0.2906 0.5033 / 0.2906 Musique↑ 0.1986 / 0.1132 0.2247 / 0.1239 Bamboogle↑ 0.3760 / 0.2660 0.4000 / 0.2560 Macro Avg↑ 0.4623 / 0.3548 0.4805 / 0.3524 Weighted Avg↑ 0.5201 / 0.4062 0.5346 / 0.4028 Fresh Responses↓ 204,800 204,800

Headroom-Drift 0.5449 / 0.4269 0.6940 / 0.5889 0.5301 / 0.4375 0.4721 / 0.3320 0.5115 / 0.3083 0.2305 / 0.1241 0.4320 / 0.2860 0.4879 / 0.3577 0.5399 / 0.4084 204,800

Table 5: Agentic Search results. Arrows indicate metric direction (↑ higher is better, ↓ lower is better). Top-1 values are bold, and Top-2 values are underlined.

Benchmark

GRPO+r matched Headroom-Drift GRPO+r matched Headroom-Drift Mean@4 Mean@4 Best@4 Best@4

NQ TriviaQA PopQA HotpotQA 2WikiMultiHopQA Musique Bamboogle

0.4057 0.6107 0.4069 0.3596 0.3394 0.1436 0.3500

0.4432 0.6228 0.4292 0.3748 0.3378 0.1587 0.4020

0.5244 0.7108 0.5002 0.5018 0.5789 0.2491 0.5280

0.5604 0.7172 0.5253 0.5114 0.5597 0.2627 0.5200

Macro Avg Weighted Avg

0.3737 0.4158

0.3955 0.4298

0.5133 0.5556

0.5224 0.5638

Table 6: Additional 7B Agentic Search scale check on Search-R1 with Qwen2.5-7B-Instruct. Both methods use identical configurations and an 8-bit optimizer, and the controlled comparison is based on one run per method with four samples per input. The better value in each metric pair is bold. tings where environment interaction dominates rollout cost, replay can reduce wall-clock training time in addition to improving sample quality.

21

Published as a conference paper at COLM 2026

C

Replay ingress in the multi-reward Geo3K setting

C.1

Motivation and problem setup

In this section, we analyze a multi-reward refinement of the replay-ingress rule in Headroom-Drift Replay, using Geometry3K as a focused case study. As stated in the main method description, the binary-reward setting uses mixed-outcome-only ingress; the present section clarifies how that same ingress principle should be specialized when multiple reward components jointly determine the final reward. The key issue is not datasetspecific: reward heterogeneity can distort the semantics of "success" and therefore distort replay-ingress decisions. To make the scope of this analysis explicit, we frame the central question as follows. RQ1. Under a multi-reward setting, which groups should be admitted to the replay buffer in order to preserve a useful training signal for subsequent replay learning? The original replay ingress rule was designed to admit a group into the replay buffer only when its success count was neither zero nor full. The motivation is that if all samples within a group receive the same reward, the group-relative advantage in GRPO becomes zero, so the group provides no useful signal for policy updates. Accordingly, our goal was to prioritize replaying groups that still preserve within-group reward differences and therefore retain a useful training signal. In binary-reward settings, the original sample-level success indicator si+ = 1[ri > 0] served as an exact indicator of correctness. Under that setting, excluding groups that were allfailure or all-success under si+ functioned as an appropriate replay-ingress criterion. In Geometry3K, however, the final reward combines answer correctness and formatting quality, so si+ no longer corresponds one-to-one to answer correctness. The problem is that the same si+ -based definition used in single-reward settings was applied unchanged to this multi-reward setting. Condition wrong answer + format failure wrong answer + format success correct answer + format failure correct answer + format success

Final reward 0.0 0.1 0.9 1.0

Table 7: Geometry3K reward structure. As shown in Table 7, the final reward in Geometry3K jointly reflects answer correctness and formatting quality. Accordingly, si+ = 1[ri > 0] no longer corresponds exclusively to correctness: an incorrect response may still receive 0.1 for satisfying the format requirement, whereas a correct response may receive 0.9 even when the format requirement is violated. Therefore, carrying over the si+ -based success definition from single-reward settings changes the replay ingress dynamics: the dynamics in a multi-reward setting are no longer the same as those in a single-reward setting. C.2

Engineering observation: replay ingress drying

Applying the original si+ -based success definition to Geometry3K leads to a rapid drying of replay ingress. An analysis of the training logs shows that, as training progresses, the number of groups newly entering the replay buffer decreases sharply. Figure 6 summarizes this pattern. The same logs also indicate that, in later stages of training, an increasing number of groups are filtered out because all responses in the group are judged either all unsuccessful or all successful under the si+ -based definition. This phenomenon did not simply mean that the model had become correct on almost all problems. Figure 7 shows that, already after the early stage of training, many groups 22

Published as a conference paper at COLM 2026

Figure 6: Declining replay-buffer ingress over the course of training in Geometry3K. The batch size is 128.

were classified as all-success groups under the original si+ -based definition. Yet these allsuccess groups were usually not groups in which all responses were actually answercorrect. Rather, many of them contained multiple answer-incorrect samples that nevertheless received positive reward from the format component alone.

Figure 7: Comparison between the number of exact-8 groups in which all responses satisfy the original success indicator si+ = 1[ri > 0] and the number of such groups that contain at least one answer-positive response ( ai = 1 ⇐⇒ ri ≥ 0.9). Under the si+ -based definition, all such groups are treated uniformly as all-success groups, even though they differ in whether they contain answer-correct responses.

Thus, in Geometry3K, the label all-success under si+ no longer uniquely identifies answercorrect groups; the si+ -based definition collapsed the distinction among format-only, answer-only, and fully correct responses by treating all of them uniformly as success. Accordingly, the direct cause of ingress collapse was not that the model had made almost all samples correct, but that the si+ -based definition failed to separate answer and format signals and counted format-only positives as success. As a result, even groups that still contained a meaningful mix of correct and incorrect responses in terms of ai could be treated as fully positive groups and excluded from replay ingress. 23

Published as a conference paper at COLM 2026

C.3 C.3.1

Why a correctness-aligned ingress criterion is necessary Format-only reward variance

In Geometry3K, variation in the final reward is not sufficient by itself to determine whether a group is useful for replay. Because formatting contributes to the final reward, two responses can receive different rewards even when both are answer-incorrect. The example below illustrates this failure mode. Example 1. Format-only reward variance without answer-correctness variance Case A B

Final reward 0.0 0.1

Answer correctness incorrect incorrect

Format reward fail success

Original response A (score = 0.0) <think> Let's call the missing side length "x". Since the sum of angles in a triangle is 180 degrees, we can use the angle sum property to find angle x. x + 20° + 135° = 180° Simplifying the equation, we get: x + 155° = 180° Subtracting 155° from both sides, we get: x = 25° Therefore, angle x is equal to 25°. \boxed{25°} </think>

Example 1 (continued). Original response B (score = 0.1) <think> To find the value of x, we can use the sine function in a right triangle. The sine function relates the angle of a side (adjacent) to the ratio of the opposite side to the hypotenuse. Since angle 20° is adjacent to x, we can write: sin(20°) = (18) / x Now, let's solve for x: x = (18) / sin(20°) x = (18) / (0.3420) x 52.90 </think> \boxed{52.90}

Case A is format-incorrect because the boxed final answer remains inside the <think> block, whereas Case B is format-correct because the boxed final answer appears outside <think>. This example highlights the core mismatch. From the perspective of final reward, the group contains non-trivial variation, and such variation would indeed induce a non-zero grouprelative advantage in GRPO. However, that advantage is not necessarily aligned with answer correctness. In this case, the resulting update would favor a format-correct but still answer-incorrect response over a response that is both format-incorrect and answerincorrect. Therefore, a replay criterion based only on final-reward variation can retain 24

Published as a conference paper at COLM 2026

groups whose internal preference structure is driven primarily by formatting rather than by answer quality. The same issue appears at run level. Because GRPO updates the policy from relative reward differences within a group, format-driven variance can be repeatedly reinforced even when answer quality does not improve. Figure 8 shows that, even under the baseline onpolicy setting, format reward rises quickly whereas answer reward improves more gradually. This suggests that formatting can be learned sufficiently from on-policy data, while replay capacity should be reserved more selectively for groups informative about answer correctness.

Figure 8: Format reward saturates much earlier than answer reward in Geometry3K. Consistent with this interpretation, Figure 9 shows that the original ingress rule can continue to admit groups composed only of 0.0/0.1 rewards. These groups contain no answercorrect samples, and their within-group reward variance is driven entirely by formatting. As a result, replay-buffer capacity can be consumed by groups that do not contribute directly to answer-level improvement. These observations imply that the main issue is not the existence of a formatting reward, but the ingress criterion itself: the original rule treats answer-driven and format-driven variation as equally replay-worthy. We therefore redesign ingress around the answer-based indicator ai = 1[ri ≥ 0.9], so that limited replay capacity is preferentially allocated to groups that contain answer-relevant variation. To make this design choice explicit and reproducible, we now formalize the answer-based ingress rule used in Geometry3K. C.4

Formal definition of answer-based ingress

The discussion above motivates the need for an answer-based ingress rule, but the term answer-based must be made precise in order to avoid ambiguity. In particular, our analysis involves both sample-level and group-level notions, as well as both the original si+ -based criterion and the revised ai -based criterion. We therefore summarize the notation and terminology first, and then define the answer-based ingress rule formally. Using the notation above, we define answer-based ingress at both the sample and group levels. Let a group consist of G sampled responses, and let i ∈ {1, . . . , G } index the responses within the group. For each response, we define an answer-based indicator ai ∈ {0, 1}, where ai = 1 if the response is correct under the answer criterion, and ai = 0 otherwise. 25

Published as a conference paper at COLM 2026

Figure 9: The count and fraction of groups that can enter the replay buffer under the original ingress rule while containing only 0.0 and 0.1 rewards. Such groups contain no answercorrect samples at all, and their within-group reward variation is driven entirely by formatting rather than answer correctness. As the figure shows, these groups are not confined to the very early stage of training, but continue to appear as ingress candidates well into the middle and later stages.

Level Sample Sample Sample Group Group Group

Symbol / term ri si+ ai C+ ( g) S( g) answer-mixed group

Meaning final reward of response i original success indicator, si+ = 1[ri > 0] answer-based indicator, ai = 1[ri ≥ 0.9] original success count, C + ( g) = ∑iG=1 si+ answer-based success count, S( g) = ∑iG=1 ai a group satisfying 1 ≤ S( g) ≤ G − 1

Table 8: Notation and terminology used in the Geometry3K replay-ingress analysis.

26

Published as a conference paper at COLM 2026

For a group g = { ai }iG=1 , we then define the answer-based success count as G

S ( g ) = ∑ ai . i =1

Under the answer-based ingress rule, a group is admitted to the replay buffer if and only if 1 ≤ S( g) ≤ G − 1. That is, replay ingress excludes both answer-all-failure groups (S( g) = 0) and answer-allsuccess groups (S( g) = G), and retains only answer-mixed groups in which correct and incorrect responses coexist. In the Geometry3K experiments, the group size is fixed to G = 8. The answer-based indicator is instantiated from the task reward structure as ai = 1[ri ≥ 0.9], where ri denotes the final reward of response i. Under the reward structure in Table 7, this means that responses with rewards 0.9 and 1.0 are treated as answer-correct, whereas responses with rewards 0.0 and 0.1 are treated as answer-incorrect. This definition makes the role of answer-based ingress precise. It ensures that replay ingress is determined by whether answer correctness actually varies within the group, rather than by whether the final reward varies for any reason. As a result, groups whose internal variation is driven only by formatting do not satisfy the answer-based criterion, whereas groups containing both correct and incorrect responses remain eligible for replay. C.5 C.5.1

Empirical effect of answer-based ingress Improved training dynamics

Figure 10: Comparison of training-score trajectories under the original ingress rule and the answer-based ingress rule over the shared early-stage window. The top panel shows the raw trajectories (faint) together with their running averages (bold), while the bottom panel shows the score difference between the two runs over the same window. Figure 10 shows that over the shared early-stage window, the answer-based ingress rule tends to yield higher training scores than the original ingress rule. The upper panel indicates that the smoothed score trajectory remains above that of the original rule for most 27

Published as a conference paper at COLM 2026

of the comparison range, while the lower panel shows a mostly positive gap rather than a single-point spike. These results suggest that aligning replay ingress with answer correctness improves training dynamics in Geometry3K. C.5.2

Ingress remains viable

Figure 11: Number of groups newly entering the replay buffer under the answer-based ingress rule during the shared early-stage window. Despite the stricter criterion, replay ingress remains sufficiently active and does not collapse prematurely. A natural concern with the answer-based ingress rule is that the stricter ai -based definition may reduce ingress so aggressively that replay no longer operates reliably. Figure 11 shows that this is not what happens. Even in the early stage of training, the number of groups newly entering the replay buffer remains sufficiently large. In other words, strengthening the ingress criterion around answer correctness does not cause replay-buffer supply to dry out prematurely within this window. These results indicate that answer-based ingress is well motivated in Geometry3K. It is associated with higher training scores while preserving enough replay ingress for replay to remain operational. More broadly, this supports a general design point: when final rewards mix heterogeneous components, replay ingress should be aligned with the target semantic signal (here, answer correctness), not with a coarse positivity test alone. Answer to RQ1. In multi-reward settings such as Geometry3K, replay ingress should prioritize groups whose within-group variation reflects the target task signal (answer correctness) rather than auxiliary components (e.g., formatting). This answerbased ingress is associated with better training-score dynamics while maintaining sufficient replay ingress.

28

Published as a conference paper at COLM 2026

D

Replay age and reuse statistics

Appendix A introduced replay age, replay indicators, lifetime replay counts, and cumulative mixed-objective exposure as formal quantities associated with buffer-side replay selection. This section summarizes how those quantities behave empirically in training. Our goal here is descriptive: we make visible how Headroom ranking, Policy Drift gating, and replay-budget truncation shape the actual reuse patterns observed in training. Unless otherwise noted, the statistics below are reported for the mathematical-reasoning run. D.1

Replay age and drift compatibility

A natural first question is whether absolute replay age alone is sufficient to characterize replay compatibility. To examine this, we compare the age distributions of replay candidates accepted by the Policy Drift gate and those rejected by it. Figure 12 summarizes the resulting age statistics in three complementary views. Figure 12 shows a clear age gradient in compatibility: older groups are less likely to pass the Policy Drift gate. Accepted groups have a lower mean absolute replay age than rejectedby-drift groups, and the age-conditioned acceptance rate decreases sharply as replay age increases. This is consistent with the intended role of Policy Drift as a current-policy compatibility filter. At the same time, the figure also shows that age alone is not sufficient to determine compatibility. The accepted and rejected empirical CDFs overlap substantially in the low-age regime, and rejection remains visible even for the youngest replay ages. Thus, while older groups tend to be less reusable, recent groups are not automatically safe. This is precisely why replay compatibility is controlled by Policy Drift rather than by absolute recency alone. The thin tail at large replay ages is not produced only by the Drift gate; it is also mechanically constrained by finite FIFO residence in the replay buffer. Accordingly, the main message of Figure 12 is not that old groups disappear solely because of gating, but that conditional on being scanned, older groups are less likely to remain compatible, while age alone still does not fully determine acceptance. D.2

Headroom-ordered prefix truncation in practice

The replay rule in Appendix A can be viewed as a Headroom-ordered prefix truncation procedure: buffered groups are ordered by cached Headroom, scanned in that order, and admitted only if they remain Drift-compatible, with selection terminating once the replay budget is filled. Figure 13 shows how this mechanism behaves in practice. Figure 13 shows two complementary properties of the replay-selection mechanism. First, accepted replay is strongly concentrated near the top of the Headroom-ordered buffer. In Panel (B), a relatively small top prefix already accounts for a large share of all accepted replay, indicating that cached Headroom ordering is not merely decorative but materially shapes which groups are reused. Second, the scan depth needed to satisfy the replay budget grows over training. As shown in Panel (A), early steps can often fill the replay target after scanning only a shallow prefix, whereas later steps increasingly require deep scans and sometimes fail to fill the target at all. This pattern is consistent with a tightening effective admissible set under the Policy Drift gate: as training progresses, fewer buffered groups remain simultaneously highpriority and sufficiently near-policy. Taken together, the two panels show that replay selection is not well described as either a fixed top-K rule or a uniform scan over the entire buffer. Rather, it is an adaptive prefixtruncation process in which Headroom ordering determines where replay mass is concentrated, while Policy Drift and the replay budget determine how deep the scan must go to recover enough admissible groups. 29

Published as a conference paper at COLM 2026

Figure 12: Replay age and Drift-gate compatibility in mathematical-reasoning. (A) Age histogram normalized separately within the accepted and rejected populations. (B) Empirical CDF of replay age for accepted and rejected candidates. (C) Age-conditioned acceptance rate, with the gray bars indicating the number of scanned candidates in each age bucket. Accepted replay is concentrated on younger groups overall, but the two age distributions overlap substantially, and rejection remains non-negligible even at the smallest replay ages. The raw tail counts at large ages are additionally constrained by finite FIFO turnover, so the age-conditioned acceptance rate in Panel (C) is the most direct view of compatibility.

Figure 13: Headroom-ordered prefix truncation in practice. (A) Actual scan depth per training step, measured as the number of buffered candidates examined before replay selection terminates. Blue circles indicate steps that filled the replay target, and orange diamonds indicate underfilled steps. (B) Cumulative share of accepted replay recovered from the top Headroom-ordered buffer prefix. Accepted replay is strongly concentrated near the top prefix of the Headroom-ordered buffer, but later training increasingly requires deeper scans because the Policy Drift gate becomes more selective.

D.3

Initial stored headroom and lifetime replay pressure

We next ask whether stored Headroom acts as a genuine replay-selection pressure over training, rather than merely serving as a definitional score attached to buffered groups. To test this, we bucket groups by the decile of their initial stored Headroom at buffer entry, then measure how often those groups are replayed over their lifetime. Figure 14 shows a clear association between initial stored Headroom and subsequent lifetime replay exposure. Low-Headroom deciles exhibit both small mean replay counts and low replay-at-least-once probabilities, whereas the upper deciles show markedly stronger 30

Published as a conference paper at COLM 2026

Figure 14: Initial stored Headroom and lifetime replay pressure. Bars show the mean lifetime replay count Nrep for groups in each initial stored-Headroom decile, and the line shows the probability that a group is replayed at least once. Higher initial stored Headroom is associated with both more frequent reuse and a higher probability of entering replay at least once, indicating a real Headroom-driven selection pressure rather than a purely definitional score.

reuse. Thus, the stored Headroom score is not merely a conceptual ranking variable: it induces a real selection pressure on how often groups re-enter training. Importantly, the relationship is strong but not perfectly deterministic. The uppermost deciles are clearly favored, but they do not form a strictly monotone sequence, and some lower-decile groups are still replayed. This is expected: lifetime reuse is shaped not only by Headroom ranking, but also by Policy Drift admissibility, scan-prefix competition, replaybudget truncation, and finite buffer residence. Higher initial stored Headroom is therefore associated with stronger lifetime replay exposure, though it does not fully determine the reuse process alone.

D.4

Replay multiplicity and exposure concentration

Finally, we examine the overall distribution of replay reuse across buffered groups. The rep quantities NT ( g) and WT ( g) from Appendix A allow us to distinguish two different views of replay concentration: how many groups fall into each replay-count cohort, and how much total replay exposure mass those cohorts account for. Figure 15 shows that replay is far from uniformly distributed across buffered groups. Most groups are never replayed at all, and the cohort share decreases rapidly as the lifetime replay count grows. In this sense, Headroom-Drift Replay is highly selective at the cohort level: the majority of buffered groups never re-enter training through replay. However, the exposure-mass view in Panel (B) tells a more informative story. Although high-Nrep cohorts are small in population share, they account for a much larger share of total replay-use events. Thus, replay selection is not merely deciding one-shot inclusion versus exclusion; it is redistributing lifetime training exposure toward a comparatively small subset of repeatedly reused groups. 31

Published as a conference paper at COLM 2026

Figure 15: Replay multiplicity and exposure concentration. (A) Cohort share by lifetime replay count Nrep , showing what fraction of groups are replayed 0, 1, 2, . . . times. (B) Exposure-mass share by lifetime replay count, showing what fraction of total replay-use events comes from each Nrep cohort. Most groups are never replayed, but a much smaller subset of repeatedly reused groups accounts for a disproportionately large share of total replay exposure. This distinction between cohort share and exposure mass is important for interpreting replay as a training mechanism. A replay rule may appear sparse if viewed only through the fraction of groups that are ever replayed, yet still exert substantial influence on optimization if the repeatedly reused groups receive a large share of total replay exposure. In this sense, Headroom-ranked, Drift-gated replay induces a structured concentration of lifetime exposure rather than a uniform spreading of replay across all buffered groups.

32

Published as a conference paper at COLM 2026

E

Replay dynamics and the delay of entropy collapse

E.1

Why entropy collapse matters in GRPO

In GRPO-style reinforcement learning, late-stage failures such as entropy collapse, policy contraction, reward hacking, and exploration failure are widely reported. Terminology differs across studies, but the core concern is consistent: even when reward or score improves, the policy distribution can become overly concentrated, making subsequent training brittle. From this perspective, entropy collapse is not merely a benign sharpening of the policy; it is a practical indicator of broader training failure (Miao et al., 2025; Yuan et al., 2026; Xi et al., 2026; Wang et al., 2026b). Prior work addresses this issue from several complementary directions, including optimizer and clipping design (Yuan et al., 2026; Xi et al., 2026), explicit regularization against reward hacking and late-stage degradation (Miao et al., 2025), and theoretical analyses of entropy dynamics in reinforcement fine-tuning (Wang et al., 2026b). Taken together, these studies establish entropy collapse as an informative signal of broader training failure. Building on this line of work, this appendix treats replay as an operational factor and asks whether replay changes collapse timing itself, rather than merely shifting the learning curve in absolute training steps. In line with Contribution 3, we keep the analysis observational and focus on how replay shapes training dynamics through gate behavior, mismatch, and entropy/performance evolution. In our experiments, the replay run appeared to enter entropy collapse later, even after reaching a comparable training-score level. However, this pattern alone is insufficient evidence of more stable learning. A run can also appear to collapse later in absolute-step time simply because it learns more slowly and reaches high-training-score regimes later. We therefore reinterpret replay effects under training-score-matched comparison and ask whether replay merely shifts the trajectory in time or meaningfully delays collapse onset. RQ2. Under training-score-matched comparison, does replay meaningfully delay entropy collapse? E.2

Experimental setting and comparison axes

We compare three training configurations: the standard on-policy baseline (GRPO on-policy matched (b256)); Headroom-Drift Replay (b256+r128), which adds Headroom-Drift Replay to the same on-policy setup; and GRPO on-policy larger (b384), which increases the on-policy batch size without replay. All three runs use the same Qwen2.5-Math-1.5B model, GRPO training framework, and dataset. They differ only in replay usage, actor-batch composition, and the number of GRPO mini-batches per update.

Setting Replay On-policy batch size Replay target groups GRPO mini-batch size Number of GRPO mini-batches per update

GRPO on-policy Headroom-Drift Replay GRPO on-policy matched (b256) (b256+r128) larger (b384) No Yes No 256 256 384 0 128 0 64 64 64 4

6

6

Table 9: Runs compared in the entropy-collapse analysis. A GRPO mini-batch here denotes one sequential subdivision of the full actor batch within a single update step. Concretely, Headroom-Drift Replay (b256+r128) and GRPO on-policy larger (b384) both use six GRPO mini-batches per update, whereas GRPO on-policy matched (b256) uses four. We therefore treat GRPO on-policy larger (b384) as the stricter on-policy control, because it better matches Headroom-Drift Replay (b256+r128) in per-update optimization 33

Published as a conference paper at COLM 2026

Entry delay Rebound delay ∆entry ∆rebound 318 121

Run GRPO on-policy matched (b256) Headroom-Drift Replay (b256+r128) GRPO on-policy larger (b384)

237 207

261 180

Table 10: Entry and rebound delays for the three runs under the representative thresholds defined in the text. load. This comparison helps separate the structural effect of replay from the effect of simply processing more mini-batches. E.3

Empirical characterization of the observed delay

Although the replay run appears to enter entropy collapse later in absolute training progress, this observation alone does not establish collapse-free, stable learning. A run may look delayed simply because it progresses more slowly: if it reaches comparable trainingscore levels later, its transition into low-entropy regions will also appear later on the raw progress axis, even when the underlying collapse dynamics are unchanged. We therefore move beyond raw-progress comparison and use training-score-matched analysis to distinguish two possibilities: replay may merely shift the trajectory in time, or it may delay collapse onset at matched performance levels. We first state the limitation of raw-progress comparison and then summarize the training-score-matched patterns across the three runs. E.3.1

From raw-progress comparison to matched-score analysis

To remove learning-speed confounds, we compare runs at matched training-score levels rather than at absolute progress indices. Let S denote the training score and H denote policy entropy. For a training-score anchor s0 , define the first matching index as tfirst (S ≥ s0 ). We then define entry into and exit from the low-entropy regime as tentry ( H ≤ θ ),

trebound ( H ≥ θ ′ ).

To reduce sensitivity to short-lived fluctuations, tentry is detected using 10 consecutive log points, and trebound using 5 consecutive log points. Using these events, we define two training-score-conditioned delays: ∆entry (θ | s0 ) = tentry ( H ≤ θ ) − tfirst (S ≥ s0 ), ∆rebound (θ, θ ′ | s0 ) = trebound ( H ≥ θ ′ ) − tentry ( H ≤ θ ). ∆entry measures how long a run takes to enter the low-entropy regime after reaching the shared training-score anchor, and ∆rebound measures how long it remains in that regime before rebound. E.3.2

Observed patterns across the three runs

In this analysis, we use the representative score anchor S ≥ 0.40, the low-entropy plateau entry threshold HL = 0.03, and the exit threshold HU = 0.04. Table 10 summarizes the score-conditioned delays of the three runs under these criteria, and Figure 16 visualizes the corresponding entropy trajectories and marker timings. Figure 16 shows that Headroom-Drift Replay (b256+r128) reaches the shared score anchor S ≥ 0.40 earlier than GRPO on-policy larger (b384). The replay run therefore does not lag in entering the high-score regime. Yet the same trajectory indicates later entry into the low-entropy regime, suggesting that score improvement and entropy compression are not strictly synchronized. 34

Published as a conference paper at COLM 2026

Figure 16: Entropy trajectories for the three runs. The gray GRPO on-policy matched (b256) is shown only as a reference. Diamonds denote first arrival at the shared training-score anchor, circles denote entry into the low-entropy regime, squares denote rebound, and the highlighted segments indicate the retained low-entropy window. The matched-score comparison reinforces this point and extends beyond entry timing. Under HL = 0.03, GRPO on-policy larger (b384) shows a shorter post-anchor entry delay than Headroom-Drift Replay (b256+r128) (207 vs. 237 in Table 10). Under HU = 0.04, GRPO onpolicy larger (b384) also rebounds sooner, while Headroom-Drift Replay (b256+r128) sustains the low-entropy regime longer. In this run pair, replay reaches the shared high-score regime earlier, but enters the low-entropy regime later and remains there longer. Taken together, these observations indicate that the delay cannot be explained only by slower learning in the replay run. At matched score levels, both low-entropy entry and rebound are delayed under replay. The observed pattern is therefore more consistent with a change in post-anchor entropy dynamics than with a simple shift in optimization speed. E.4

Interpretation of the observed delay

The previous subsection suggests that the delayed collapse in the replay run is not simply a by-product of slower optimization; entropy dynamics differ even after runs reach comparable score levels. This subsection develops a working interpretation of that difference and then examines why the effect weakens in the late stage. E.4.1

Replay as a near-policy corrective mechanism

As shown above in Figure 16 and Table 10, the replay run reaches the shared score anchor earlier, yet enters and exits the low-entropy regime later. This pattern makes a purely slower-learning explanation unlikely and motivates a working interpretation. Figure 5(c) provides the corresponding conceptual pattern: replay under Headroom-Drift remains concentrated on recent sources, but is not reducible to trivial previous-step reuse. Table 11(A)1 , together with the conceptual view in Figure 5(c), indicates that replay in this run is neither trivial one-step reuse nor indiscriminate long-range reuse. The replay groups actually used for updates span source ages from 1 to 7 steps; 66.00% come from sources at least 2 steps old, and 44.06% from sources at least 3 steps old. At the same time, the used subset is more recent than the replay pool as a whole (mean source staleness: 2.67 vs. 3.32 1 Source staleness is reconstructed from checkpoint snapshots because replay-group insertion times are directly available only in checkpointed replay states. Since checkpoints are saved every 10 training steps in this run, the reported statistics summarize checkpoint-aligned observations rather than every individual step.

35

Published as a conference paper at COLM 2026

(A) Source staleness summary Mean Median P90 Max % age ≥ 2 % age ≥ 3

Population Replay pool Used replay subset

3.32 2.67

3 2

6 5

8 7

73.51 66.00

60.22 44.06

(B) Mixing structure of the used replay subset Statistic

Value

Mean distinct source ages per update Fraction of updates with at least 3 source ages Fraction of updates with both age = 1 and age ≥ 3

4.07 85.71% 76.79%

Table 11: True source staleness of replay groups. Panel (A) shows that the replay groups actually used for updates are more concentrated on recent source samples than the replay pool as a whole, while still spanning a non-trivial short horizon. Panel (B) shows that the used replay subset typically mixes multiple source ages within the same update. steps), indicating a clear recent-source preference rather than uniform reuse across the full buffer. Importantly, this recency bias does not collapse to a single age: Table 11(B) shows a mixed-age structure (mean distinct source ages per update: 4.07; updates with at least three ages: 85.71%; updates containing both age = 1 and age ≥ 3: 76.79%). These statistics are consistent with a near-policy corrective interpretation. Replay mostly reuses recent samples but still incorporates selected older ones, so each update reflects a short-horizon multi-age mixture rather than only the immediately preceding rollout. Relative to pure on-policy learning, replay therefore introduces trajectories generated at nearby policy states that remain useful under current criteria. Within this process, Headroom prioritizes higher-value groups in the replay pool, while the Policy Drift gate filters out trajectories that have drifted too far from the current policy. Under this interpretation, replay is less a pull toward stale policies than a constrained reintroduction of still-compatible trajectories from nearby policy snapshots, which can moderate one-step acceleration while retaining informative signal. E.4.2

Late-stage transition and failure mode Metric Early learning Score-rise Pre-collapse Collapse Mean source age 2.77 3.26 2.24 1.26 % age = 1 21.15 22.90 42.13 76.47 % age ≥ 3 48.08 59.35 33.50 2.94 Mean distinct ages / update 4.33 4.83 3.80 1.67 % updates with ≥ 3 ages 100.00 100.00 85.00 16.67 % updates with age = 1 and age ≥ 3 66.67 87.50 85.00 16.67

Table 12: Phase-wise contraction of the used replay mixture. The replay subset is most diverse during the score-rise regime and collapses toward a narrow age-1/2-dominated mixture near the failure phase. Figure 17 shows that the late-stage transition is not well explained by a simple ingress collapse. The number of newly ingressed groups does not drop sharply, even in the late stage. By contrast, the amount of replay actually used in updates falls markedly in the same region, and steps that fail to fill the replay target become frequent near collapse. The transition is therefore better interpreted as a rapid contraction of the subset that remains reusable under current criteria. This interpretation is reinforced by the phase-wise mixture changes in Table 12. During the score-rise phase, the used replay subset has mean source age 3.26, share age ≥ 3 of 36

Published as a conference paper at COLM 2026

Figure 17: Replay ingress and actual replay use over training. The gray curve shows the number of newly ingressed groups added to the replay buffer at each step, while the orange curve shows the number of replay groups actually used for the update. Although ingress remains substantial into the late stage, actual replay use drops sharply near collapse, suggesting that the late-stage transition is better understood as a contraction of the usable replay subset rather than as a simple ingress collapse. 59.35%, mean distinct ages per update of 4.83, and 100% of updates with at least three ages, indicating a rich multi-age mixture during rapid improvement. This structure weakens in pre-collapse and then contracts sharply in collapse: mean source age falls to 2.24 and then 1.26, the age = 1 share rises to 42.13% and then 76.47%, the age ≥ 3 share drops to 33.50% and then 2.94%, and mean distinct ages decrease to 3.80 and then 1.67. The late-stage issue is therefore not only reduced replay volume; the corrective mixture itself narrows from a multi-age structure to a highly concentrated age range. This transition also co-occurs with larger policy movement in the late stage. As shown in Supplementary Figure 18, on-policy GRPO-KL and gradient norm rise together during collapse. At the same time, the main logs show a sharp drop in Drift-gate acceptance rate relative to pre-collapse, along with repeated steps that fail to fill the replay target. This combination is consistent with increasing mismatch between the current policy and replayed trajectories, which shrinks the replay groups admissible under the Policy Drift threshold. Taken together, the late-stage failure in this run is consistent with a phase in which the nearpolicy corrective mechanism can no longer be sustained in sufficiently rich form. In the early-to-middle stage, replay appears to combine useful trajectories from multiple recent policy steps, coinciding with both score improvement and delayed entropy collapse. In the late stage, replay ingress remains present, but the used replay subset drops sharply and its internal source-age mixture contracts into a narrow age-1/2-dominant range. We therefore interpret the transition as being linked less to replay presence per se and more to the duration over which a usable corrective mixture can be maintained.

37

Published as a conference paper at COLM 2026

Figure 18: Supplementary policy-drift proxies over training. The upper panel shows onpolicy GRPO-KL, and the lower panel shows gradient norm on a log scale. Both quantities rise sharply in the late stage, consistent with a phase in which the current policy begins to move more abruptly while the Drift-gate-admissible replay subset simultaneously contracts.

38

Published as a conference paper at COLM 2026

F

Implementation details for headroom-drift replay

This appendix provides implementation-level detail for the replay-augmented training procedure described in Algorithm 1. The main text focuses on the conceptual design of replay selection (Headroom ranking, Policy Drift gating, FIFO eviction); this section supplements it with the engineering choices required to run the system stably in practice. Algorithm 2 expands each step of Algorithm 1 to this level, with concrete instantiations from e t |=128 replay groups, |G on |=256 on-policy groups, the mathematical reasoning setting (|R t mini-batch size 64, 8 GPUs). Algorithm 2 Two-column implementation view of Algorithm 1. Left: pseudocode. Right: concrete instantiation in the mathematical reasoning setting. Left: General Procedure (Pseudocode)

Right: Concrete Instantiation (Math Setting)

A. Rollout, Reward, Advantage Same as Algorithm 1.

A/C. Shared with Alg. 1 Rollout, reward, advantage, ingress, and retrieval are unchanged.

B. Frozen Reference State On-policy groups: compute old log-probs via forward pass. Replay groups: reuse stored log πgen( g) . Replay advantages: reuse stored Ai .

B. Frozen references Replay groups: 128 groups reuse stored log πgen( g) and Ai (no recomputation).

D. Selection with Early Termination Sort buffer by cached Headroom (descending). Initialize selected ← ∅. for each candidate g in sorted order do if |selected| ≥ Krep then break Compute log πθt for g. Compute Drift( g; θt ) using current and stored log-probs. Refresh cached Headroom from the same current-policy pass. if Drift( g; θt ) ≤ τ append g, else reject g (cached Headroom still refreshed). end for

D. Selection Stop when Krep =128 is filled (early termination). Use one parallelized forward pass per scanned group to compute both Policy Drift and refreshed cached Headroom.

E. Batch Construction e t ← selected[ 1 : u⌊m/u⌋ ]. R e t ; G on ]. actor_batch ← [ R t Split into k mini-batches (preserving order).

E. Batch construction u=1 (no truncation). [128 replay; 256 on-policy] = 384 total groups. k =6 mini-batches: MB 1–2 replay, MB 3–6 on-policy.

F. Sequential PPO Update + Eviction for i = 1, . . . , k do Partition MBi across GPUs with length balancing. Compute log πθ on stored trajectories. Compute r = πθ /πgen( g) , clipped PPO objective, and update θ. end for Append ingress groups and apply FIFO eviction.

F. PPO update Compute numerator log πθ only, then clipped PPO update; ingress append + FIFO eviction after update.

F.1

Frozen reference state

When a replay group is admitted to the buffer, its generation-time per-token logprobabilities log πgen( g) ( ai,j | si,j ) and response-level advantages Ai are stored as immutable state (Algorithm 2, Frozen Reference State). The importance-ratio denominator is therefore always anchored at the generation-time policy πgen( g) ; only the numerator is recomputed under the current policy πθt at each PPO update. Advantages are likewise reused without recomputation. As described in Section 3.3, the Policy Drift gate measures 39

Published as a conference paper at COLM 2026

how far this frozen reference state has drifted from the current policy and blocks groups that exceed threshold τ. Replay-buffer storage. In the mathematical-reasoning configuration, the CPU-resident FIFO buffer holds at most 512 query groups (8,192 responses). At the configured maximum sequence lengths, the stored token sequences, generation-time log-probabilities, and GRPO advantages total approximately 448 MiB at full capacity. F.2

Selection with early termination and cached-headroom refresh

Replay selection scans buffer candidates in descending cached-Headroom order and accepts groups satisfying Drift( g; θt ) ≤ τ until the replay budget Krep is filled, at which point the scan terminates immediately (Algorithm 2, Selection with Early Termination). This early termination ensures that selection cost scales with Krep and the number of candidates actually scanned, rather than with the full buffer size. When the budget cannot be filled—for instance, when τ is tight and many candidates are rejected—the accepted count is used as-is. For each scanned candidate, the current-policy forward pass—which is parallelized over stored trajectories—simultaneously computes Policy Drift and the runtime Headroom needed to refresh cached Headroom (Section 3.3). This cached-Headroom refresh is applied to every scanned group regardless of whether it passes or fails the Policy Drift gate, and requires no additional forward pass. As a result, the selection procedure performs Policy Drift gating and cached-Headroom refresh in a single same-pass reevaluation. F.3

Replay-prefix layout

When forming the mixed actor batch At , replay groups are placed before on-policy groups (Algorithm 2, Batch Construction). Because the actor batch is split into mini-batches by sequential slicing, this prefix layout causes replay groups to occupy the earlier mini-batches and on-policy groups to occupy the later ones. In the mathematical reasoning setting, for example, the six mini-batches are partitioned as MB 1–2 (replay) and MB 3–6 (on-policy). Since PPO processes mini-batches sequentially and updates the policy parameters after each one, the relatively more off-policy replay samples are consumed first within each training step. F.4

Batch alignment

Micro-batch truncation. The number of accepted replay groups may not be a multiple of the micro-batch processing unit u. To avoid misalignment during gradient accumulation, the accepted sequence is truncated to the largest multiple of u that does not exceed the accepted count (Algorithm 2, Batch Construction). Formally, given the accepted replay sequence Rt = ( g1 , . . . , gm ), the groups actually used for training are e t = ( g1 , . . . , gu⌊m/u⌋ ). R Because cached-Headroom ordering is preserved, only the lowest-priority tail is removed. The number of discarded groups is at most u − 1; when u = 1, no truncation occurs. Sequence-length balanced partitioning. Each mini-batch is distributed across GPUs for data-parallel processing (Algorithm 2, Sequential PPO Update). A naive sequential partition can concentrate long sequences on a single GPU, leaving other GPUs idle while that GPU finishes. This imbalance is amplified in the mixed batch because replay groups and on-policy groups are generated at different training steps and may have different responselength distributions. To mitigate this, each mini-batch is repartitioned so that the total sequence length per GPU is approximately balanced. In practice, this reduces the maximumminus-minimum sequence-length gap across GPUs from approximately 73K tokens to under 5K tokens. This repartitioning does not change which samples are trained on or the objective used; it affects only the mapping of samples to GPUs within each mini-batch, reducing idle time caused by uneven workloads. 40

Published as a conference paper at COLM 2026

G

Ablation studies

G.1

L1-style versus L2-style policy drift: formal interpretation and empirical comparison

The main text adopts a squared (L2-style) aggregation for Policy Drift and notes that an L1 variant yields a tighter formal bound but underperforms empirically (§3.3). This appendix develops both sides of that claim: we first show that the two gates define qualitatively different compatibility regions, and then compare them empirically on mathematical reasoning. G.1.1

Formal interpretation: variance-sensitive gating

The main method defines Policy Drift as Drift L2 ( g; θt ) :=

1 ∑ ∆i,j ( g; θt )2 . |T ( g)| (i,j)∈T ( g)

The corresponding L1-style alternative replaces the squared terms with absolute values: Drift L1 ( g; θt ) :=

1 ∑ ∆i,j ( g; θt ) . |T ( g)| (i,j)∈T ( g)

A natural question is whether the squared gate is simply a stricter version of the absolute gate. It is not. The squared gate is better understood as a variance-sensitive compatibility score. Proposition (mean–variance decomposition). Let µ g denote the mean absolute tokenwise drift and v g the within-group variance of absolute tokenwise drift. Then Drift L2 ( g; θt ) = µ2g + v g . Proof. Immediate from the standard second-moment identity E[ X 2 ] = (E[ X ])2 + Var( X ). Corollary (acceptance-region geometry). An L1 gate with threshold τ1 accepts a group whenever µ g ≤ τ1 . The L2 gate with threshold τ2 accepts whenever µ2g + v g ≤ τ2 . The L2 gate is therefore not uniformly stricter: it may reject a group with small mean drift if that drift is concentrated on a few tokens, while accepting another group with larger mean drift if the tokenwise mismatch is diffuse. Illustrative example.

Consider two four-token groups with identical L1 drift:

(0.02, 0.02, 0.02, 0.02) vs. (0.08, 0, 0, 0). Both have Drift L1 = 0.02, but Drift L2 = 0.0004 for the uniform case and 0.0016 for the spike case—a fourfold difference invisible to the L1 gate. In our experiments, the L1 threshold is τ1 = 0.005 and the L2 threshold is τ2 = 0.001. Because the L2 gate operates on a second moment, its threshold is more naturally com√ pared on the root-mean-square scale: τ2 ≈ 0.032. The two gates therefore define different compatibility regions rather than tighter or looser versions of the same region. G.1.2

Empirical comparison on mathematical reasoning

We compare L1-style and L2-style Drift gating under otherwise identical training configurations on mathematical reasoning. Figure 19 summarizes step-aligned trajectories over 500 steps. Figures 20–21 characterize the replay groups selected by the L2 gate. 41

Published as a conference paper at COLM 2026

Figure 19: Step-aligned comparison of L1-style and L2-style Drift gating. L2 attains better late-stage validation accuracy while maintaining lower replay mismatch, despite matched or lower replay-budget fulfillment. L2 wins through selection quality, not replay volume. The two gates achieve similar replay-budget fulfillment through the middle phase of training. At step 350, both fill the full budget, yet L2 already yields higher validation accuracy (Mean@4: 0.3636 vs. 0.3565) with an order-of-magnitude lower replay KL (8.5 × 10−5 vs. 5.9 × 10−4 ). The gap widens in the late phase. Averaged over steps 350–500, L2 achieves higher Mean@4 (0.3613 vs. 0.3430) despite lower average budget fulfillment (0.81 vs. 0.88), while maintaining replay KL roughly nine times smaller (7.0 × 10−5 vs. 6.3 × 10−4 ). L2 does not use more replay; it uses better-compatible replay. Selection dynamics within the L2 gate. Figure 20 shows how the L2-selected replay subset evolves. Early in training (step 50), the selected groups have much lower success rate (0.080 vs. 0.423 in the candidate pool) and higher Headroom (0.863 vs. 0.592), indicating strong concentration on hard, high-value cases. This concentration weakens as training progresses: by step 350, the selected subset approaches the pool boundary (success rate 0.390 vs. 0.474 and Headroom 0.643 vs. 0.555), and by step 500 the gap has largely closed. Once the high-value frontier is depleted, the L2 gate contracts replay usage rather than forcing the budget full with less compatible groups. Figure 21 visualizes this pattern as snapshots. At steps 50 and 200, selected groups cluster tightly in the low-success, high-Headroom corner. By step 350, coverage broadens as the high-value region thins. By step 500, selection spans a wider range of success rates, reflecting the depleted frontier. The L2 gate adapts its effective selectivity to the available pool rather than maintaining a fixed selection profile.

42

Published as a conference paper at COLM 2026

Figure 20: Aggregate properties of the replay groups selected by L2-style Drift gating. Early training concentrates replay on hard, high-Headroom groups; later training moves toward the remaining frontier, and replay usage contracts as that frontier is depleted.

43

Published as a conference paper at COLM 2026

Figure 21: Selection snapshots in the success-rate / Headroom plane at four training stages. Selected groups (orange) concentrate near the upper-left frontier of the candidate cloud (gray), particularly in early and middle training.

44

Published as a conference paper at COLM 2026

G.2

Practical calibration and sensitivity of the Policy Drift threshold τ

We use τ to place the Policy Drift gate in a viable operating regime, not to locate a narrow performance optimum. Table 13 summarizes the thresholds used across the evaluated runs. The values fall into two task-family settings: the single-turn reasoning runs and the CISPOstyle portability study share one value, while the 3B and 7B Search-R1 runs share another. In particular, the 7B Search-R1 run reuses the 3B threshold without scale-specific retuning. Evaluated setting

Scope

Mathematical reasoning

τ

CISPO-style portability

Single-turn reasoning Single-turn reasoning Objective portability

Search-R1 (3B) Search-R1 (7B)

Agentic Search Agentic scale check

Multimodal reasoning

10−3

Reuse note Shared single-turn setting

10−3 Shared single-turn setting 10−3 Replay hyperparameters unchanged 10−2 Agentic setting 10−2 Reused without scale-specific retuning

Table 13: Drift thresholds used across the evaluated settings. The values reduce to two taskfamily settings, and the 7B Search-R1 run reuses the 3B threshold without scale-specific retuning. We then use a controlled Search-R1 sweep to examine how a viable regime can be identified without a dense full-training search. Agentic Search is a useful testbed because its multiturn interaction with an external retrieval environment makes replay valuable while also exposing training instability more clearly than the single-turn settings (Wang et al., 2026a; Liu et al., 2026a;b). RQ. How can a useful operating regime for τ be identified without an exhaustive full-training sweep, and what diagnostic signatures appear when the gate is too tight or too loose? G.2.1

Experimental setup

Experiments are conducted on Agentic Search (Qwen2.5-3B-Instruct, 8×H100). The three decade-spaced conditions are designed to probe distinct gate states rather than locate a narrow performance optimum. All other training settings—including batch size, learning rate, replay buffer size, n, and Krep —are held fixed. G.2.2

Gate behavior and calibration diagnostics

We interpret replay acceptance and replay KL jointly: acceptance measures whether the gate supplies enough replay, while replay KL measures whether the admitted groups remain compatible with the current policy. Table 14 summarizes the three conditions.

Condition

Gate rejections per update Replay-KL behavior

τ

Mean acceptance

TIGHT

0.001

6.4%

450.3

MODERATE

0.01 0.1

40.2% 94.4%

119.4 1.9

LOOSE

0.0005 → 0.00003 as replay vanishes Stable at approximately 0.002 0.0045 → 0.0170

Table 14: Diagnostic summary of the Search-R1 threshold sweep. Acceptance rate and gate rejections per update are training averages; replay-KL entries summarize the early-to-final trajectory. 45

Published as a conference paper at COLM 2026

The three settings produce qualitatively different gate states. Under TIGHT, only 6.4% of candidates are accepted, effectively starving replay. Its low replay KL does not indicate better compatibility; it results from admitting very few replay groups. Under LOOSE, 94.4% of candidates pass, leaving the gate nearly inactive, and replay KL rises over training. MOD ERATE instead admits replay selectively while keeping replay KL stable at approximately 0.002. Neither diagnostic is sufficient alone: a useful threshold must preserve replay supply and control policy mismatch at the same time. G.2.3

Accumulation of distribution mismatch

The clearest difference across τ settings appears in replay KL. Replay KL is computed at each training step over the replay samples and measures how far the current policy πθ has drifted from the policy πθold that generated each replay sample. Specifically, it is the mean of DKL (πθ ∥πθold ) across token positions in the replay batch. A higher value indicates that the replay data lies farther from the current policy distribution. This replay KL is a related but distinct diagnostic from the Policy Drift gate quantity itself: the gate uses token-level squared log-probability drift on stored actions (the Policy Drift score), whereas replay KL summarizes mismatch over the accepted replay batch. Figure 22 shows the replay KL trajectories, and Table 15 reports interval-wise averages.

Figure 22: Replay KL over training for the three τ conditions (running average, window = 15). Under MODERATE, replay KL remains stable throughout training. Under LOOSE, it increases monotonically. Interval

TIGHT

MODERATE

LOOSE

LOOSE / MOD .

Early interval Middle interval Final interval

0.0005 0.0001 0.00003

0.0021 0.0023 0.0022

0.0045 0.0089 0.0170

2.1× 3.8× 7.7×

Table 15: Mean replay KL by training interval. Under MODERATE, replay KL remains flat at ∼0.002 throughout training, whereas under LOOSE it grows monotonically and reaches its largest gap from MODERATE in the final interval. Under MODERATE, replay KL remains stable at approximately 0.002 throughout training. Under LOOSE, it increases monotonically and reaches a large multiple of the MODERATE level in the final interval. This reflects the continued admission of replay data that has progressively diverged from the current policy, without filtering by the gate. We report this growth as a distributional diagnostic; the extent to which it translates into gradient-level effects after clipping is not isolated in this analysis. 46

Published as a conference paper at COLM 2026

The replay KL under TIGHT falls below 0.0001 late in training. This does not indicate superior distribution compatibility; rather, it reflects the scarcity of replay samples admitted to the training batch. G.2.4

Performance and entropy dynamics

Figure 23 compares the training score and policy entropy trajectories, and Table 16 reports interval-wise averages.

Figure 23: Training score (upper) and policy entropy (lower) over training for the three τ conditions (running average, window = 15).

TIGHT (τ =0.001)

MODERATE (τ =0.01)

LOOSE (τ =0.1)

Interval

Score

Entropy

Score

Entropy

Score

Entropy

Early interval Middle interval Final interval

0.206 0.365 0.432

0.783 0.527 0.271

0.272 0.392 0.434

0.757 0.447 0.300

0.269 0.382 0.424

0.756 0.441 0.255

Table 16: Training score and policy entropy by interval for the three τ conditions. TIGHT records the lowest score in the middle interval (0.365 vs. 0.392 for MODERATE and 0.382 for LOOSE). This appears to be due to the near-absence of replay, which prevents the run from benefiting from replay-based acceleration. TIGHT approaches MODERATE by the final interval, suggesting that on-policy learning alone can converge gradually, though with slower progress in the early-to-mid phase.

The score gap between MODERATE and LOOSE is approximately 1 percentage point, which is modest in absolute terms but consistent across all intervals in favor of MODERATE. 47

Published as a conference paper at COLM 2026

All three conditions exhibit a common decreasing trend in entropy over the course of training. In the final interval, the ordering of entropy across conditions (MODERATE 0.300 > TIGHT 0.271 > LOOSE 0.255) coincides with the ordering of score (MODERATE 0.434 > TIGHT 0.432 > LOOSE 0.424). MODERATE maintains both the highest score and the highest entropy among the three conditions at this stage. We note that this alignment does not imply a simple monotonic relationship between entropy and score across all training phases: in the middle interval, TIGHT exhibits the highest entropy yet the lowest score, reflecting slow learning due to the near-absence of replay rather than a beneficial preservation of exploration. Nonetheless, the co-occurrence of faster entropy decline and lower score under LOOSE is consistent with prior findings that rapid entropy reduction in LLM reinforcement learning tends to be associated with degraded learning outcomes (Lipkin et al., 2026; Wang et al., 2026b; Yu et al., 2025).

G.2.5

Summary

Answer to RQ. A short log-spaced sweep is sufficient to identify a useful operating regime for τ. Collapsing acceptance rules out thresholds that starve replay, while rising replay KL rules out thresholds that leave the gate too permissive. The remaining regime admits replay selectively and keeps replay KL stable. Because these signals are available before final performance is known, unsuitable candidates need not be trained to completion. Together with threshold reuse across settings and model scales, these results support treating τ as a coarse calibration choice rather than a finely tuned performance parameter.

The same diagnostics could support an adaptive schedule that loosens the gate when replay is starved and tightens it when replay KL rises.

G.3

Component ablation on mathematical reasoning

Table 17 reports a focused component ablation on MATH-500 at the evaluation endpoint. Headroom-only is the closest PER-style priority-only analogue in this group-level setting: it ranks buffered groups by Headroom without the current-policy compatibility gate. Under the same training configuration, the full Headroom-Drift combination records the highest endpoint values on both Best@32 and Mean@32, while Headroom-only and Policy-Driftonly are lower.

Variant Policy Drift only Headroom only Headroom-Drift Replay (full)

Best@32 (↑)

Mean@32 (↑)

0.7820 0.8100 0.8200

0.6794 0.7140 0.7215

Table 17: MATH-500 ablation results at the evaluation endpoint. Arrows indicate metric direction (↑ higher is better). Best@32 denotes the average of the highest score among 32 samples per input, and Mean@32 denotes the average score across all 32 samples per input. 48

Published as a conference paper at COLM 2026

Figure 24: Training-score curves for the mathematical-reasoning component ablation. In the figure legend, M2 denotes Policy Drift only, ZVP denotes Headroom only, and M2 + ZVP denotes full Headroom-Drift Replay. Figure 24 provides the corresponding training-score trajectories for the same three variants. All three curves improve over training, and the late-stage ordering is consistent with Table 17: the full variant remains on top, Headroom-only follows closely, and Policy-Driftonly stays lower. The margin between full and Headroom-only is modest, while the gap to Policy-Drift-only is visibly larger in the later phase. These trajectories provide complementary context to the endpoint comparison. G.4

Sign-only versus advantage-weighted headroom

The main text defines Headroom as a token-level score that depends only on the sign of the response-level advantage Ai : tokens in positively-advantaged responses contribute 1 − π ( ai,j | si,j ), tokens in negatively-advantaged responses contribute π ( ai,j | si,j ), and the group score is a flat average over all tokens. A natural question is whether reweighting each response’s contribution by | Ai | would produce a better ranking. What changes mathematically. Let s g denote the group success rate, and let Hg+ , Hg− denote the mean per-token headroom of the positive-advantage and negative-advantage responses respectively. Under equal response lengths, the two variants reduce to Zsign ( g) ≈ s g Hg+ + (1 − s g ) Hg− ,

Zadv ( g) ≈ 21 Hg+ + 12 Hg− .

The approximation for Zadv follows because, under group-centered binary rewards, positive responses carry | Ai | ≈ 1 − s g and negative responses carry | Ai | ≈ s g , so the count imbalance between the two sides is cancelled to first order by the advantage magnitudes. The key difference is that Zsign contains the success ratio s g directly as a mixing weight. In failure-heavy groups (s g ≪ 0.5), the negative branch dominates the score, and highconfidence wrong tokens (large π on negatively-advantaged responses) can push the group to the top of the ranking. Zadv removes this structural dependence on s g , so groups are ranked more by branch-specific correction room than by failure ratio. In short, the two variants do not define slightly different weightings of the same ranking; they define qualitatively different ranking geometries along the success-rate axis. Empirical comparison. We conducted a small-scale preliminary experiment comparing the two variants under otherwise identical training configurations. Over the shared evalu49

Published as a conference paper at COLM 2026

ation window (steps 1–38), sign-only Headroom achieves a higher average training score (0.3288 vs. 0.3022). Sign-only leads on 32 of 38 steps; advantage-weighted leads on 5. The gap is negligible in the earliest steps (at step 10: 0.2754 vs. 0.2783) but widens once replay becomes active. Over steps 19–38, where both runs use replay, sign-only leads on every step (mean 0.3863 vs. 0.3397). Representative checkpoints: Step

Sign-only

Adv-weighted

∆

10 20 35 38

0.2754 0.3481 0.4177 0.4119

0.2783 0.3108 0.3572 0.3501

−0.0029 +0.0373 +0.0605 +0.0618

The mathematical analysis above suggests a possible explanation. By removing the structural dependence on s g , advantage weighting deprioritizes failure-heavy groups that signonly would rank highly. In this setting, those failure-heavy groups appear to carry useful corrective signal; deprioritizing them reduces the effectiveness of replay. We retain signonly Headroom in the main method accordingly.

H

Preliminary portability to a CISPO-style objective

Figure 25: Training-score trajectories under a CISPO-style objective. Headroom-Drift denotes full Headroom-Drift Replay (b256+r128). Baseline (b256) is the on-policy matched control; Baseline (b384) is the on-policy larger control. Running average with window 10. The Conclusion identifies objective-level portability as an open question: whether the same two-axis decomposition transfers beyond GRPO-style clipped objectives. We provide preliminary training-score evidence on a CISPO-style objective using the same mathematical reasoning setup as the main experiments (Qwen2.5-Math-1.5B, identical dataset and evaluation protocol). The only change is the policy-optimization objective; Headroom ranking, Policy Drift gating, buffer mechanics, and all replay hyperparameters remain unchanged. Figure 25 shows three patterns consistent with the GRPO results in the main text. First, Headroom-Drift Replay outperforms the on-policy matched baseline (b256) across nearly the entire training trajectory, confirming that principled replay adds value beyond the matched fresh-rollout budget under CISPO as well. Second, Headroom-Drift tracks above the on-policy larger baseline (b384) in the late phase and reaches a higher raw peak score (0.522 vs. 0.510 at steps 380 and 343 respectively), despite using fewer fresh responses. 50

Published as a conference paper at COLM 2026

Third, the on-policy larger baseline exhibits a sharp score collapse after step 400, whereas the replay trajectory remains more stable through the same region—a pattern that mirrors the delayed entropy-collapse observation under GRPO (Appendix E).

51

Related documents

• Beyond Class Marginals: Bounding Rehearsal Gaps without Freezing Class Co-occurrence#1061369
• EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance#158570
• Motion Forcing: A Decoupled Framework for Robust Video Generation in Motion Dynamics#1117340
• WARP-RM: A Warp-Augmented Relative Progress Reward Model for Data Curation#974474
• Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation#258659