SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts

SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts Nhat Minh Pham1,2,* Duy Tung Doan2,* Thi Duyen Ngo1 Vinh Van Nguyen1 Khac-Hoai Nam Bui2 1

VNU University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam 2 Viettel AI, Viettel Group, Hanoi, Vietnam {minhpn.266, duytunghanam}@gmail.com * Equal contribution Abstract

arXiv:2608.04962v1 [cs.LG] 5 Aug 2026

Reinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but autoregressive rollout generation remains a major efficiency bottleneck. Speculative decoding can accelerate generation, yet applying it during RL is difficult because the target policy continually evolves: static proposers become stale, while frequent drafter updates add substantial overhead. We introduce SpecRoll, a speculative rollout engine that preserves the target model’s sampling distribution while adapting at two timescales. Lightweight future-token heads generate parallel proposals, while our proposed Reflex module uses delayed verifier feedback to perform bounded, trajectory-local hidden-state corrections without backpropagation. A complementary slow path updates the head parameters only when sustained degradation is detected. SpecRoll combines these mechanisms with concurrency-aware sparse-tree verification and exact target verification, leaving the target rollout distribution and GRPO objective unchanged. Across five models from 1.5B to 14B and three mathematical reasoning datasets, SpecRoll achieves 1.26×–2.15× generation speedup and 1.21×–2.04× end-toend speedup over vanilla GRPO. It also outperforms FastGRPO in both generation and end-to-end time across all 15 matched settings, with an average pairwise end-to-end gain of 1.18×. Controlled ablations show that the fast and slow adaptation paths provide complementary benefits. Our source code is available at https://anonymous.4open. science/r/SpecRoll-26062006.

1

Introduction

Reinforcement learning with verifiable rewards has become a standard approach for improving mathematical reasoning in large language models. Group Relative Policy Optimization (GRPO) is particularly attractive because it forms relative ad-

Train Faster. Spend Less. (Llama-3.1-8B on DAPO-Math-17K)

Wall-clock training time SpecRoll (Ours)

45.3 h $311.7

FastGRPO

47.4 h $326.1

GRPO

92.5 h $636.6

vs FastGRPO

vs GRPO

save $14

save $325

= 2.1 GPU-hours

= 47.2 GPU-hours

Figure 1: SpecRoll delivers plug-and-play rollout acceleration for RL. On a single NVIDIA B200 at $6.88 per GPU-hour, SpecRoll saves approximately $14 and $325 per run relative to FastGRPO and GRPO, respectively.

vantages from groups of sampled responses without requiring a learned value model (Shao et al., 2024; DeepSeek-AI, 2025). However, each update still requires multiple long autoregressive responses, making rollout generation a major wallclock bottleneck. Pipeline systems such as PipelineRL improve utilization by overlapping generation and optimization through asynchronous execution and inflight weight updates (Piché et al., 2025). Such methods reduce inter-stage idle time but not the token-by-token cost of generating each response. Decoder acceleration is therefore complementary: it reduces the rollout work itself rather than hiding it across pipeline stages. Speculative decoding attacks this cost by proposing multiple future tokens and verifying them in parallel. Exact target correction preserves the target sampling distribution, but apply-

ing speculation to RL presents two challenges. First, rollout concurrency decreases as responses finish, making a fixed verification tree inefficient across the full rollout. Second, policy updates continually shift the target distribution, causing a static proposer to become stale. FastGRPO addresses these challenges through concurrencyaware verification and online training of a standalone EAGLE-style drafter (Zhang et al., 2025b). While effective, this approach requires backward computation, optimizer state, and synchronization for an additional autoregressive model, and does not distinguish transient local mismatch from persistent policy drift. We explore a different design based on the observation that delayed verifier errors can remain predictive over nearby positions within a trajectory. This motivates Reflex, a gradient-free fast path that converts mature verifier feedback into bounded, trajectory-local hidden-state corrections. Reflex activates only when a conservative alignment gate indicates that past errors predict future ones, with candidate-boundary safeguards suppressing harmful interventions. Because local memory cannot absorb recurring drift, SpecRoll adapts at two timescales: Reflex corrects transient mismatch without backpropagation, while a slow path updates persistent proposal parameters only after sustained degradation is detected. SpecRoll implements this design using lightweight future-token heads that predict multiple horizons from the target hidden state, avoiding a separate autoregressive drafter and KV cache. Their proposals form a concurrency-aware sparse tree whose budget increases as active concurrency decreases. Exact target verification remains authoritative, preserving the rollout distribution and GRPO objective. Across five models from 1.5B to 14B and three mathematical reasoning datasets, SpecRoll achieves 1.26×–2.15× generation speedup and 1.21×–2.04× end-to-end speedup over vanilla GRPO. It outperforms FastGRPO in both generation and end-to-end time across all 15 matched model–dataset settings, with an average pairwise end-to-end gain of 1.18×. Controlled ablations show that the fast and slow adaptation paths are individually beneficial and strongest when combined. Our main contributions are: • We introduce Reflex and a two-timescale

adaptation mechanism that corrects transient proposal mismatch without backpropagation and reserves persistent updates for sustained drift. • We develop SpecRoll, a lightweight exact speculative rollout framework combining future-token heads, Reflex, selective persistent adaptation, and concurrency-aware verification. Experiments across five models and three datasets demonstrate consistent end-toend gains and validate the complementary roles of the two adaptation timescales.

2

Related Work

Efficient RL post-training systems. DeepSpeed-Chat and OpenRLHF improve RL post-training through integrated training– inference optimizations, scheduling, and resource utilization (Yao et al., 2023; Hu et al., 2024). ReaL dynamically reallocates parameters and parallelization strategies, HybridFlow supports flexible orchestration and efficient resharding, and PipelineRL overlaps rollout generation with optimization (Mei et al., 2024; Sheng et al., 2025; Piché et al., 2025). These systems mainly reduce communication, placement, resharding, and inter-stage idle costs. SpecRoll instead reduces sequential target-model decoding within rollout generation, making decoder-level acceleration complementary to system-level optimization. Speculative decoding and online adaptation. Classical speculative decoding combines a lightweight autoregressive drafter with exact target correction (Leviathan et al., 2023; Chen et al., 2023). Medusa introduces parallel future-token heads, while EAGLE and HASS improve feature prediction, draft-tree construction, and target alignment (Cai et al., 2024; Li et al., 2024b,a, 2025; Zhang et al., 2025a). Online Speculative Decoding, OnlineSpec, and Test-Time Speculation reuse verifier feedback to adapt proposers under changing requests or targets (Liu et al., 2024; Qian et al., 2026; Kumar et al., 2026), primarily through persistent parameter optimization. SpecRoll instead uses Reflex for immediate, trajectory-local hidden-state correction without backpropagation, reserving parameter updates for sustained degradation.

Figure 2: Overview of SpecRoll across one GRPO iteration. Future-token heads construct proposals from the current target hidden state, and Reflex optionally applies trajectory-local corrections before sparse-tree verification. Mature verifier feedback updates the fast memory and reliability records, while persistent head parameters are updated only after sustained drift is detected. Reward computation and the GRPO policy update remain unchanged.

Speculative decoding for RL rollouts. FastGRPO combines concurrency-aware verification with continuous online training of a standalone EAGLE-style drafter (Zhang et al., 2025b). RLHFSpec couples workload-aware drafting with sample reallocation, while other work integrates speculation into synchronous and asynchronous RL pipelines (Wang et al., 2025; Iso et al., 2026). Distribution-Aware Speculative Decoding builds a nonparametric suffix-tree drafter from previous rollouts, whereas EfficientRollout derives a quantized self-drafter from the target (Shao et al., 2026; Kim et al., 2026). SpecRoll focuses on lightweight future-token heads and separates adaptation across two timescales: gated, gradient-free memory handles transient mismatch, while recurring drift is selectively consolidated into persistent parameters. Exact target verification remains authoritative, preserving the rollout distribution and GRPO objective.

3

Method

3.1

Problem Formulation

At optimization stage s, GRPO samples response groups from the current policy πθs and updates its parameters using within-group relative advantages (Shao et al., 2024). We consider an exact speculative proposal engine E within this sampling stage. Let At denote the accepted progress at verification round t. An ideal engine should provide substantial accepted progress at low computational cost while preserving the target rollout distribution: max E

E[At ] Cost(E)

subject to

X ∼ πθs .

(1) Thus, useful speculation must jointly maintain low proposal overhead, sufficient acceptance, and unbiased target sampling. Finding an optimal proposal engine online is generally intractable and could itself dominate the rollout critical path. Instead, SpecRoll starts from a lightweight proposal family and asks: how can the errors of an inexpensive proposer be corrected with minimal delay, while reserving persistent pa-

rameter updates for mismatch that genuinely survives across trajectories? This question is especially important in GRPO, where successive updates continually move πθs and make proposal accuracy inherently non-stationary. Figure 2 summarizes the two-timescale adaptation design within the overall GRPO training loop. SpecRoll instantiates the proposal engine with lightweight MEDUSA-style future-token heads that produce a sparse draft tree. Concurrencyaware construction adapts this tree within each verification round; gradient-free Reflex corrects transient errors within a trajectory; and a slow path consolidates persistent mismatch across trajectories. Exact target verification remains authoritative throughout, so these mechanisms improve rollout efficiency without changing the committed-token distribution, rewards, or GRPO objective. 3.2

a

Distribution error Target p

zt,h = gϕh (ht ) + ht , qt,h = softmax(W zt,h ),

h = 1, . . . , H. (3)

If nt,h nodes are allocated to horizon h, the resulting tree satisfies H X h=1

nt,h ≤ Nt ,

t2

t3

✓

✓

×

C={

(5)

a

} (

t4

× )

normalized mass within C

pC

t1 0.56

qC

b

t2 0.44

t1 0.75

t2 0.25

Coverage error Target p

Proposal q b (0.00, 0.20)

0.45

dcov

0.60 0.35 0.15

0.15 0.20

ddist 0.05 0.05

qC

(2)

where head h predicts xt+h+1 and reuses the target vocabulary projection W . Their candidates form a prefix-closed sparse tree Tt rooted at the anchor, avoiding a second autoregressive backbone and drafter KV cache. Low proposal cost alone does not guarantee efficient rollout. Under the high and varying concurrency of GRPO, a fixed verification tree may add unnecessary work when many responses are active and underuse the hardware as responses finish (Zhang et al., 2025b). Let Bt be the number of unfinished responses and Chw the profiled verification capacity of the model–hardware pair. SpecRoll assigns each response the node budget    Chw Nt = clip , Nmin , Nmax . (4) Bt

|Tt | = 1 +

t1

t1

t2

t3

✓

×

✓

C={

where xt+1 is an exact target-sampled anchor. From the same hidden state, H lightweight horizon heads predict subsequent positions in parallel:

ddist

0.15 0.15

0.05 0.05

pC

xt+1 ∼ pt .

0.35 0.20

At prefix x≤t , the target backbone produces pt = softmax(W ht ),

(0.19, 0.00)

dcov

0.45

Concurrency-Aware Exact Rollout

ht = fθs (x≤t ),

Proposal q

0.60

} (

t4

× )

normalized mass within C t1 0.56 t2 0.44 t1 0.75 t3 0.25 t1 0.75

t3 0.25

Figure 3: Two proposal errors under a fixed candidate budget. Top: The correct candidates are selected, but the proposal mass is misallocated, producing distribution error. Bottom: A target-important token is omitted, producing coverage error.

where the anchor is counted once. The ratio Chw /Bt captures the batch-wide envelope Bt |Tt | ≤ Chw within the profiled operating range. The tree determines only which target conditionals are evaluated together. At every visited node, exact rejection sampling and targetresidual fallback remain authoritative. Consequently, concurrency-aware construction changes verification cost and accepted progress, but not the committed-token distribution. 3.3

Verifier-Guided Proposal Error Analysis

When a horizon-h proposal matures, exact verification exposes the target distribution p alongside the retained proposal distribution q. We use this feedback to analyze proposal quality in terms of two complementary sources of acceptance loss: distribution error and coverage error. Conservative acceptance surrogate. Consider a matured proposal node with candidate set C ⊆

V, where |C| = K. The maximum target mass attainable under the same candidate budget is achieved by ⋆ CK ∈ arg max p(A) = TopK (p).

(6)

A⊆V |A|≤K

For p(C), q(C) > 0, define the conditional target and proposal distributions on C as pC (v) =

p(v) , p(C)

qC (v) =

q(v) , q(C)

v ∈ C.

(7) Their overlap coefficient is X ΩC = min{pC (v), qC (v)} = 1−TV(pC , qC ),

Reflex converts the two proposal errors into temporary hidden-space feedback that is reused only at later prefixes of the same trajectory. The proposal-head parameters remain unchanged.

(9)

S = C ∪ TopKk (q) ∪ TopKk (p) ∪ {y ⋆ }, (12)

v∈C

Because the realized multi-candidate acceptance probability is difficult to decompose directly, we use the following local surrogate. As shown in Appendix B.4, if Aver C denotes the realized node-wise verifier acceptance probability, then eC := p(C)ΩC A = p(C) [1 − TV(pC , qC )]

Reflex: Immediate Trajectory-Local Correction

(8) where 1X |pC (v) − qC (v)| . 2

3.4

Verifier feedback direction. Consider a horizon-h proposal created at position τ of trajectory i. For feedback computation, we use the common sparse support

v∈C

TV(pC , qC ) =

set, while the bottom panel isolates missing targetimportant tokens despite correct allocation within the retained set. Motivated by this analysis, the next subsection converts these signals into the corresponding hidden-space corrections rdist and rcov , which form the core design of Reflex.

(10)

≤ Aver C ≤ p(C).

where k is the support budget and y ⋆ is the realized target token. Let pS and qS be the target and proposal distributions renormalized on S, and let WS contain the corresponding output-projection rows. Distribution feedback is dist = WS⊤ (pS − qS ), ri,τ,h

(13)

which moves the retained-token logits toward the ⋆ \ C contain the target. For coverage, let O = CK omitted target-important tokens and let b⋆ be the weakest retained candidate. We use

eC is a conservative local acceptance surroThus, A gate that captures both the target mass covered by C and the proposal–target agreement within that support.

b⋆ = arg min q(b), b∈C X
cov ri,τ,h = pS (y) Wy − Wb⋆ ,

Distribution–coverage decomposition. Under this surrogate, the gap from the budget-matched target ceiling decomposes exactly as

which promotes omitted tokens relative to the topK selection boundary. We RMS-normalize each direction before combining them:

⋆ eC = p(C) TV(pC , qC ) p(CK )−A | {z }

ddist ⋆ + p(CK ) − p(C) .

|

{z

dcov

(11)

}

Here, ddist measures probability misallocation among the retained candidates, whereas dcov measures target mass lost by omitting better candidates under the same budget. Figure 3 illustrates these complementary failure modes: the top panel isolates misallocation within the correct candidate

(14)

y∈O

x ri,τ,h =

x ri,τ,h x RMS(ri,τ,h )+ϵ

,

x ∈ {dist, cov},

(15) where ϵ > 0 is a numerical stabilizer. The unified feedback is dist cov gi,τ,h = ddist ri,τ,h + dcov ri,τ,h ,
gi,τ,h ei,τ,h = ddist + dcov . RMS(gi,τ,h ) + ϵ

(16)

Thus, ei,τ,h combines both correction directions ddist + dcov .

Trajectory-local recurrent memory. When a proposal record matures, Reflex updates its trajectory–horizon memory: mi,h ← ρh mi,h + (1 − ρh )ei,τ,h ,

0 ≤ ρh < 1, (17) where ρh controls decay. Consistent errors reinforce one another, while inconsistent directions cancel; the memory is reset when trajectory i ends.

Reliability from delayed alignment. Because the current verifier error is delayed, Reflex estimates whether the memory remains useful from earlier matured records. At proposal time, it stores a normalized memory sketch sj . Once record j matures, it computes the realized alignment aj = cos(sj , ξj ) against the normalized verifierfeedback sketch ξj , where aj > 0 means that the memory predicted the subsequent error direction. For each trajectory–horizon pair (i, h), let ni,h , āi,h , and sa,i,h be the count, mean, and sample standard deviation of matured alignments. With sufficient observations, the conservative reliability score is sa,i,h di,h = āi,h − zδ √ Rel , ni,h

(18)

where zδ > 0 penalizes noisy evidence. Gated hidden-state correction. At a later prefix position t, let zi,t,h be the uncorrected state produced by horizon head h. Reflex applies the normalized memory only when its reliability lower bound is positive: mi,h , RMS(mi,h ) + ϵ n o di,h > 0 αi,t,h RMS(zi,t,h )ui,h , ∆zi,t,h = 1 Rel ui,h =

zei,t,h = zi,t,h + ∆zi,t,h .

(19) Here αi,t,h is bounded, and relative-RMS scaling keeps the correction proportional to the current state magnitude. Reflex abstains when the memory is negligible, insufficient alignment observations have matured, or the reliability score is non-positive. It uses only detached feedback from earlier verified proposals, requiring no additional target forward or backward pass; exact target verification remains authoritative. Appendix C.1 gives the practical checks, while Appendices A and E analyze the feedback directions and reliability gate.

3.5

Drift-Triggered Persistent Consolidation

Reflex adapts within a trajectory, whereas mismatch that recurs across trajectories should update the proposal heads. For each horizon head h, SpecRoll maintains a bounded reservoir Rh of matured verifier records. It compares the current sparse proposal–target discrepancy and acceptance rate with calibrated per-head baselines, and updates a head only when both degradation signals persist across repeated checks. Evidence thresholds, cooldown, and head selection are specified in Appendix C.1. For a selected head, a record j ∈ Rh provides a sparse support Sj , a target distribution pSj ,j , and a head distribution qSj ,j (ϕh ) renormalized on the same support, together with the realized target token yj⋆ . SpecRoll minimizes h
Laux (ϕh ) = Ej∼Rh λdist ℓdist pSj ,j , qSj ,j (ϕh )  i + λtok − log qj (yj⋆ ; ϕh ) ). + λprox Ω(ϕh , ϕbefore h (20) Here ℓdist (p, q) = DKL (p∥q) is the restrictedsupport distillation loss, whereas qj (yj⋆ ; ϕh ) denotes the probability of the realized token under the head’s full-vocabulary proposal distribution. The nonnegative λ coefficients weight the three ) penalizes departure terms, while Ω(ϕh , ϕbefore h from the pre-update parameters. Only the heads are updated, and the target backbone, vocabulary projection, and GRPO policy are fixed and receive no gradient from Laux .

4

Experiment

We evaluate SpecRoll through four research questions: RQ1 Can it consistently reduce rolloutgeneration and end-to-end GRPO training time across model scales and reasoning datasets? RQ2 Do these gains result from more effective use of the proposal and verification budget, rather than simply from producing larger trees or longer accepted continuations? RQ3 Do trajectory-local Reflex correction and reliability-triggered persistent consolidation provide complementary gains? 4.1

Setup

We evaluate Qwen2.5-1.5B, 3B, 7B, and 14B (Qwen Team, 2025), together with Llama-3.1-8B (Grattafiori et al., 2024). Each model is trained

GSM8K Model

SimpleRL L3–5

DAPO-Math-17K

Average

Measure

Metric FastGRPO SpecRoll FastGRPO SpecRoll FastGRPO SpecRoll FastGRPO SpecRoll

Speedup

Gen. E2E

1.33 1.30

1.59 1.53

1.07 1.02

1.29 1.24

1.27 1.25

1.46 1.42

1.22 1.19

1.45 1.40

Acceptance

AAL Acc.

1.692 0.086

1.540 0.111

1.796 0.082

1.526 0.102

1.137 0.108

1.503 0.101

1.542 0.092

1.523 0.105

Speedup

Gen. E2E

1.40 1.35

1.60 1.54

1.29 1.22

1.46 1.42

1.22 1.20

1.29 1.26

1.30 1.26

1.45 1.41

Acceptance

AAL Acc.

2.240 0.073

1.730 0.092

1.779 0.065

1.726 0.093

1.950 0.081

1.640 0.082

1.990 0.073

1.699 0.089

Speedup

Gen. E2E

1.08 1.06

1.94 1.85

1.24 1.20

1.49 1.43

1.38 1.33

1.53 1.46

1.23 1.20

1.65 1.58

Acceptance

AAL Acc.

1.989 0.060

1.920 0.113

2.150 0.096

1.820 0.103

2.124 0.101

1.850 0.107

2.088 0.086

1.863 0.108

Speedup

Gen. E2E

1.15 1.14

1.46 1.43

1.21 1.19

1.49 1.45

1.95 1.87

2.09 1.99

1.44 1.40

1.68 1.62

Acceptance

AAL Acc.

2.050 0.081

2.110 0.124

2.110 0.078

1.970 0.115

1.954 0.074

1.875 0.102

2.038 0.078

1.985 0.114

Speedup

Gen. E2E

1.18 1.16

1.38 1.33

1.15 1.13

1.26 1.21

1.98 1.95

2.15 2.04

1.44 1.41

1.60 1.53

Acceptance

AAL Acc.

1.927 0.055

1.840 0.108

1.380 0.068

1.805 0.093

1.890 0.061

1.623 0.076

1.732 0.061

1.756 0.092

Qwen2.5-1.5B

Qwen2.5-3B

Qwen2.5-7B

Qwen2.5-14B

Llama-3.1-8B

Table 1: Pairwise comparison between FastGRPO and SpecRoll. Gen. and E2E report generation and end-toend speedups over matched vanilla-GRPO runs; bold denotes higher speedup. AAL is the number of accepted speculative tokens per target-verification round, and Acc. is the fraction of tested speculative candidates accepted by the verifier. Averages are arithmetic means across the three datasets.

on GSM8K (Cobbe et al., 2021), SimpleRL-AbelLevel3to5 (Zeng et al., 2025), and DAPO-Math17K (Yu et al., 2025). All runs use NVIDIA B200 accelerators. We compare vanilla GRPO, FastGRPO, and four variants that share the same future-token heads and concurrency-aware verifier: Heads only, which disables both adaptation paths; Heads + Reflex, which enables only the gradient-free fast memory; Heads + Aux, which disables Reflex but retains reliability-triggered persistent head updates; and full SpecRoll, which combines both paths. The FastGRPO drafter and all SpecRoll variants are pretrained on the same ShareGPTderived data. Within each model–dataset pair, all methods share the target checkpoint, prompt order, reward function, group size, sampling temperature, response limit, stopping rules, and target sampling implementation. Timings cover proposal construction, verification, feedback processing, and triggered auxiliary work. We report generation and end-to-end speedups, average acceptance length(AAL), and token acceptance rate.

4.2

Overall Training Efficiency

RQ1: Does SpecRoll consistently accelerate GRPO training? Table 1 compares SpecRoll with vanilla GRPO and FastGRPO across 15 model–dataset settings. Relative to vanilla GRPO, SpecRoll achieves 1.26×–2.15× generation speedup and 1.21×–2.04× end-to-end speedup, with averages of 1.57× and 1.51×, respectively. By comparison, FastGRPO averages 1.33× generation speedup and 1.29× endto-end speedup. SpecRoll is faster than FastGRPO on both measures in every setting, although the magnitude of the gain varies across models and datasets rather than increasing monotonically with model size. Appendix D further shows that this advantage persists throughout a representative training run rather than arising from isolated timing fluctuations. Overall, SpecRoll improves both generation and end-to-end efficiency in all 15 settings, showing that its rollout-level gains survive proposal, verification, and adaptation overheads. RQ2: Where do the efficiency gains come from? SpecRoll does not uniformly maximize

FastGRPO

Heads only

+ Reflex

+ Aux

Full SpecRoll

Model

Gen.

E2E

AAL

AR

Gen.

E2E

AAL

AR

Gen.

E2E

AAL

AR

Gen.

E2E

AAL

AR

Gen.

E2E

AAL

AR

1.5B 3B 7B 14B

1.07 1.29 1.24 1.21

1.02 1.22 1.20 1.19

1.796 1.779 2.150 2.110

0.082 0.065 0.096 0.078

1.07 1.19 1.21 1.18

1.06 1.17 1.18 1.16

1.439 1.570 1.610 1.740

0.085 0.071 0.076 0.083

1.21 1.39 1.37 1.40

1.19 1.35 1.32 1.36

1.473 1.617 1.623 1.780

0.089 0.078 0.087 0.089

1.13 1.28 1.24 1.31

1.12 1.25 1.21 1.27

1.468 1.584 1.617 1.750

0.088 0.072 0.081 0.086

1.29 1.46 1.49 1.49

1.24 1.42 1.43 1.45

1.526 1.726 1.820 1.970

0.102 0.093 0.103 0.115

Table 2: FastGRPO and the four-way SpecRoll ablation on Qwen2.5 models using SimpleRL-Abel-Level3to5. The four SpecRoll variants share the pretrained future-token heads, tree budget, verifier, and target configuration; FastGRPO is included as an external drafter-based reference. Gen. and E2E are speedups over matched vanilla GRPO, AR denotes acceptance rate, and bold marks the best result for each model and metric.

average acceptance length. Instead, its higher token acceptance rate indicates that a larger fraction of the tested proposal budget contributes to accepted progress. It achieves a higher acceptance rate than FastGRPO in 14 of the 15 settings, even though FastGRPO occasionally attains a higher AAL. For example, on Qwen2.514B/SimpleRL, FastGRPO obtains a higher AAL (2.110 versus 1.970) but a substantially lower acceptance rate (0.0777 versus 0.115), while also remaining slower end to end. This distinction arises because AAL measures committed progress per verification round, whereas acceptance rate measures how efficiently the tested candidates are converted into accepted tokens. Thus, the results attribute SpecRoll’s advantage to more effective use of its proposal budget, rather than uniformly longer accepted continuations. 4.3

Complementarity of Fast and Slow Adaptation

RQ3: Are fast and slow adaptation complementary? Table 2 isolates the two adaptation paths while holding the heads, verifier, and training setup fixed. Within the SpecRoll variants, either Reflex or auxiliary adaptation improves the Heads-only baseline at every scale, with Reflex providing the larger individual gain. Their combination is consistently strongest on generation speedup, end-to-end speedup, and acceptance rate. For Qwen2.5-14B, for example, end-to-end speedup rises from 1.16× with Heads only to 1.36× with Reflex and 1.27× with Aux, while full SpecRoll reaches 1.45×; it also raises AAL from 1.740 to 1.970 and acceptance rate from 0.083 to 0.115. FastGRPO obtains higher AAL in all four settings, suggesting that its standalone draft model can produce longer accepted continuations. Our claim is therefore not that lightweight future-token heads always draft better. Rather, small heads

combined with Reflex and Aux remain effective enough to achieve higher acceptance rates and end-to-end speedups in every row, while avoiding a separate autoregressive drafter. The ablation supports the intended division of labor: Reflex corrects immediate trajectory-local mismatch, whereas Aux consolidates errors that persist across trajectories.

5

Conclusion

We introduced SpecRoll, an exact speculative rollout engine for GRPO that combines lightweight future-token heads with reliability-gated fast–slow adaptation. Reflex converts delayed verifier errors into temporary hidden-space corrections without backward computation, while a separate slow path absorbs persistent mismatch into the head parameters. Across five models and three reasoning datasets, SpecRoll achieves 1.26×–2.15× generation speedup and 1.21×–2.04× end-to-end speedup over vanilla GRPO, and it is faster in both generation and end-to-end time than FastGRPO in all 15 matched settings. Controlled ablations show that Reflex-only and auxiliary-only variants both improve the same future-token-head baseline, while their combination gives the strongest throughput, AAL, and acceptance rate. These results support a precise novelty claim: verifier feedback is split into a gradient-free trajectorylocal memory and a selectively triggered persistent learner rather than being used only for online parameter training.

Limitations Our experiments cover five models up to 14B and three mathematical reasoning datasets. Broader evaluation on larger models, more challenging and diverse tasks, and additional domains such as code and multilingual reasoning would further establish the generality of SpecRoll. Runtime gains

may also vary across hardware platforms, execution stacks, concurrency settings, and responselength distributions, motivating additional profiling and system-specific tuning. Future work will study longer training runs and larger-scale deployments. These extensions concern empirical coverage and systems optimization; the exact targetverification semantics remain unchanged.

References Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5209–5235. PMLR. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948. Aaron Grattafiori et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Weikai Fang, Xianyu, Yu Cao, Haotian Xu, and Yiming Liu. 2024. OpenRLHF: An easy-to-use, scalable and high-performance RLHF framework. arXiv preprint arXiv:2405.11143. Hayate Iso, Tiyasa Mitra, Sudipta Mondal, Rasoul Shafipour, Venmugil Elango, Terry Kong, Yuki Huang, Seonjin Na, Izzy Putterman, Benjamin Chislett, Maor Ashkenazi, Joseph Guman, Gerald Shen, Tugrul Konuk, Ashwath Aithal, Ritika Borkar, Ran Zilberstein, and Bita Rouhani. 2026. Accelerating RL post-training rollouts via systemintegrated speculative decoding. arXiv preprint arXiv:2604.26779. Minseo Kim, Minjae Lee, Seunghyuk Oh, Kevin Galim, Donghoon Kim, Coleman Hooper, Harman Singh, Amir Gholami, Hyung Il Koo, and Wonjun Kang. 2026. EfficientRollout: System-aware selfspeculative decoding for RL rollouts. arXiv preprint arXiv:2606.18967.

Avinash Kumar, Sujay Sanghavi, and Poulami Das. 2026. Test-time speculation. arXiv preprint arXiv:2605.09329. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024a. EAGLE-2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421– 7432, Miami, Florida, USA. Association for Computational Linguistics. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024b. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28935–28948. PMLR. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. EAGLE-3: Scaling up inference acceleration of large language models via trainingtime test. In Advances in Neural Information Processing Systems, volume 38. Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2024. Online speculative decoding. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 31131–31146. PMLR. Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. 2024. ReaL: Efficient RLHF training of large language models with parameter reallocation. arXiv preprint arXiv:2406.14088. Alexandre Piché, Ehsan Kamalloo, Rafael Pardinas, Xiaoyin Chen, and Dzmitry Bahdanau. 2025. PipelineRL: Faster on-policy reinforcement learning for long sequence generation. arXiv preprint arXiv:2509.19128. Yu-Yang Qian, Hao-Cong Wu, Yichao Fu, Hao Zhang, and Peng Zhao. 2026. When drafts evolve: Speculative decoding meets online learning. arXiv preprint arXiv:2603.12617. Qwen Team. 2025. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Zelei Shao, Vikranth Srivatsa, Sanjana Srivastava, Qingyang Wu, Alpay Ariyak, Xiaoxia Wu, Ameen Patel, Jue Wang, Percy Liang, Tri Dao, Ce Zhang, Yiying Zhang, Ben Athiwaratkun, Chenfeng Xu, and Junxiong Wang. 2026. Beat the long tail: Distribution-aware speculative decoding for RL

training. In Proceedings of Machine Learning and Systems, volume 8. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.

a sufficient condition for descent of a current verifier surrogate. Third, descent of a conditional KL divergence on the verifier support is related to an explicit lower bound on first-slot acceptance. All statements concerning supports are conditional on the indicated supports remaining fixed. A.1

Pointwise surrogate representation

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. HybridFlow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems. Association for Computing Machinery.

Let V be a finite vocabulary, let W ∈ R|V|×dz , and denote by Wv ∈ Rdz the transpose of the row of W indexed by v ∈ V. Fix z0 ∈ Rdz , a nonempty support S ⊆ V, a distribution pS on S, and a temperature τ > 0. Define

Siqi Wang, Hailong Yang, Junjie Zhu, Xuezhu Wang, Yufan Xu, and Depei Qian. 2025. RLHFSpec: Breaking the efficiency bottleneck in RLHF training via adaptive drafting. arXiv preprint arXiv:2512.04752.

qS (z) = softmax(WS z/τ ), dist LS (z) = DKL (pS ∥qS (z)) , rSdist (z0 ) = WS⊤ (pS − qS (z0 )) .

Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, and Yuxiong He. 2023. DeepSpeedChat: Easy, fast and affordable RLHF training of ChatGPT-like models at all scales. arXiv preprint arXiv:2308.01320.

Let O, B ⊆ V be finite sets with B ̸= ∅. Let ω Py ≥ 0 for y ∈ O, and let βb ≥ 0 for b ∈ B satisfy b∈B βb = 1. Define X WB = βb Wb ,

Qiying Yu et al. 2025. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.

The coverage direction in Equation 14 is recovered by taking ωy = p(y), B = {b⋆ }, and βb⋆ = 1. If a conditional target mass is used by the implementation instead, then ωy must be defined by that same mass in both Equation 22 and the surrogate below. For y ∈ O, b ∈ B, and a fixed γ ∈ R, set

Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. SimpleRL-Zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892.

(21)

b∈B cov rO,B =

X

ωy W y − W B .

(22)

y∈O

cy,b (z) = γ + Wb⊤ z − Wy⊤ z,

Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. 2025a. Learning harmonized representations for speculative sampling. In International Conference on Learning Representations.

ℓy,b (z) = log(1 + exp(cy,b (z))) , ωy βb κy,b (z0 ) = . sigmoid(cy,b (z0 ))

Yizhou Zhang, Ning Lv, Teng Wang, and Jisheng Dang. 2025b. FastGRPO: Accelerating policy optimization via concurrency-aware speculative decoding and online draft learning. arXiv preprint arXiv:2509.21792.

Since cy,b (z0 ) is finite, sigmoid(cy,b (z0 )) ∈ (0, 1); hence every coefficient in Equation 23 is finite and nonnegative. The coefficients, together with O, B, and (βb )b∈B , are treated as constants with respect to z. Define XX Lcov (z) = κy,b (z0 )ℓy,b (z). (24) z0

A

Theoretical Analysis of Reflex

This section establishes three results. First, the instantaneous Reflex feedback is represented as the negative gradient of a pointwise surrogate with all data-dependent supports and weights held fixed. Second, a finite-sample bound is derived for an exponentially weighted average of delayed feedback from a drifting predictable field; this bound yields

(23)

y∈O b∈B

Proposition A.1 (Pointwise surrogate representation). At z = z0 , rSdist (z0 ) = −τ ∇z Ldist S (z0 ),

(25)

cov rO,B = −∇z Lcov z0 (z0 ).

(26)

Consequently, for fixed λd , λc ≥ 0, the vector cov r = λd rSdist (z0 ) + λc rO,B

(27)

Proof. In Equation 16, each direction is multiplied by its nonnegative severity weight and divided by a strictly positive scalar. The final RMS normalization multiplies the resulting vector by a nonnegative scalar. Proposition A.1 therefore applies.

(28)

A.2

satisfies r = −∇z Φz0 (z0 ), Φz0 (z) = λd τ Ldist S (z)

Delayed-memory tracking under policy drift

+ λc Lcov z0 (z). If r ̸= 0, then r is a strict descent direction for Φz0 at z0 . Proof. The z-dependent term in DKL (pS ∥qS (z)) is X − pS (v) log qS (v; z). (29)

Fix a trajectory–horizon pair and suppress these indices. Let (Fs )s≥0 be a filtration, and let Xs ∈ Rdz be an integrable, Fs -measurable feedback vector associated with proposal round s. Define gs = E[Xs | Fs−1 ],

ξs = Xs − gs .

(35)

v∈S

The Jacobian of the temperature-scaled logsoftmax gives ∇z Ldist S (z0 ) =

1 ⊤ W (qS (z0 ) − pS ) , τ S

(30)

which proves Equation 25. For every (y, b) ∈ O × B, ∇z ℓy,b (z0 ) = sigmoid(cy,b (z0 ))(Wb − Wy ). (31) It follows from Equation 23 that XX −∇z Lcov ωy βb (Wy − Wb ) z0 (z0 ) = y∈O b∈B

Then (ξs )s≥0 is a martingale-difference sequence. The vector Xs may include the severity weighting and RMS normalizations in Equation 16. Fix integers d0 ≥ 1, H ≥ 1, and t ≥ d0 +H −1. Suppose that exactly one record matures per round after delay d0 . For a fixed ρ ∈ [0, 1), the zeroinitialized EMA after H matured records is mt = (1 − ρ)

H−1 X

ρj Xt−d0 −j = (1 − ρH )m̄t ,

j=0

(36) m̄t =

H−1 X

aH,j Xt−d0 −j ,

aH,j =

j=0

(1 − ρ)ρj . 1 − ρH

! =

X

ωy

y∈O cov = rO,B ,

Wy −

X b∈B

(32)

which proves Equation 26. Linearity of the gradient proves Equation 28. Finally, ∇z Φz0 (z0 )⊤ r = −∥r∥22 < 0

(37)

βb Wb

The weights are deterministic, nonnegative, and sum to one. Define ℓH (ρ, d0 ) =

= d0 +

whenever r ̸= 0.

cov g = λd rSdist (z0 ) + λc rO,B

(34)

for some λd , λc ≥ 0. Hence g is the negative gradient at z0 of the surrogate in Equation 28. Any strictly positive scalar multiple of g is a strict descent direction for that surrogate whenever g ̸= 0.

aH,j (d0 + j)

j=0

(33)

Corollary A.1 (RMS-normalized feedback). Suppose that the distribution and coverage directions are evaluated at z0 , and that the nonnegative severity weights and RMS denominators in Equation 16 are held fixed. Then the pre-normalized combined vector in Equation 16 has the form

H−1 X

νH (ρ) =

H−1 X

ρ HρH − , 1 − ρ 1 − ρH

(38)

a2H,j

j=0

=

(1 − ρ)2 (1 − ρ2H ) . (1 − ρH )2 (1 − ρ2 )

(39)

Let Lt : Rdz → R be differentiable, let zt ∈ d R z , and set ht = −∇Lt (zt ). Assume the following conditions. (A1) Local smoothness. There exists L > 0 such that Lt is L-smooth on the segment {zt + s∆zt : s ∈ [0, 1]}.

(A2) Bounded predictable-field drift. For every consecutive pair of indices needed below, ∥gs+1 − gs ∥2 ≤ Γ

almost surely.

(40)

(A3) Conditional sub-Gaussian noise. For every unit vector v ∈ Rdz and every λ ∈ R, h i E exp(λv ⊤ ξs ) | Fs−1 (41) ≤ exp(λ2 σ 2 /2) almost surely.

The probability statement is pointwise in (t, H); a simultaneous statement over several proposal rounds requires an additional union bound or a time-uniform concentration inequality. Proof. By Equation 35, m̄t − gt =

H−1 X j=0

+

H−1 X

aH,j ξt−d0 −j

j=0

(A4) Current-field calibration. ∥ht − gt ∥2 ≤ b

aH,j (gt−d0 −j − gt )

almost surely.

=: Bt + Nt .

(42)

(50)

For each j, repeated application of (A2) yields Assumption (A4) is not implied by the pointwise identities in Proposition A.1: Xs may contain normalization, severity weighting, sparse-support restriction, coverage feedback, and randomness not present in the gradient of Lt . Theorem A.1 (Delayed-memory tracking and current-surrogate descent). For a fixed pair (t, H) and δ ∈ (0, 1), define 2 ζδ = dz log 5 + log , δ p εt,H (δ) = ΓℓH (ρ, d0 ) + 2σ 2νH (ρ)ζδ ,

(43)

et,H (δ) = b + εt,H (δ).

(44)

∥gt−d0 −j − gt ∥2 ≤ Γ(d0 + j)

almost surely. (51) The triangle inequality and the nonnegativity of the weights therefore give ∥Bt ∥2 ≤ Γ

on Et,H,δ .

(52) Let N be a 1/2-net of the Euclidean unit sphere in Rdz with |N | ≤ 5dz . Fix v ∈ N . Since the coefficients aH,j are deterministic, iterated conditioning in increasing order of the time indices and (A3) imply E exp(λv ⊤ Nt )   2 σ 2 H−1 X λ ≤ exp a2H,j  2 j=0   2 2 λ σ νH (ρ) . = exp 2

(45)

Assume additionally (A1) and (A4), mt ̸= 0, and αt > 0. Write the Reflex perturbation as mt = ηt dt , (46) RMS(mt ) + ϵ mt m̄t dt = = , ηt = ∥∆zt ∥2 . ∥mt ∥2 ∥m̄t ∥2 (47)

∆zt = αt RMS(zt )

On Et,H,δ , Lt (zt + ∆zt ) − Lt (zt )   Lηt ≤ −ηt ∥m̄t ∥2 − et,H (δ) − . (48) 2 Consequently, on the same event, strict descent holds whenever ∥m̄t ∥2 > et,H (δ), 2(∥m̄t ∥2 − et,H (δ)) 0 < ηt < . L

aH,j (d0 + j) = ΓℓH (ρ, d0 ).

j=0

Under (A2)–(A3), there exists an event Et,H,δ satisfying P(Et,H,δ ) ≥ 1 − δ such that ∥m̄t − gt ∥2 ≤ εt,H (δ)

H−1 X

(53)

Thus, for every u > 0,   P |v Nt | ≥ u ≤ 2 exp − 

⊤

u2 2σ 2 νH (ρ)

 . (54)

Applying the union bound over N with u = p σ 2νH (ρ)ζδ gives an event of probability at least 1 − δ on which p max |v ⊤ Nt | ≤ σ 2νH (ρ)ζδ . (55) v∈N

For a 1/2-net, ∥x∥2 ≤ 2 maxv∈N |v ⊤ x|. Hence, on this event, p ∥Nt ∥2 ≤ 2σ 2νH (ρ)ζδ . (56)

(49) Equations 52 and 56 establish Equation 45.

On Et,H,δ , (A4) and the triangle inequality imply ∥ht − m̄t ∥2 ≤ ∥ht − gt ∥2 + ∥gt − m̄t ∥2 ≤ et,H (δ). (57) Because mt = (1 − ρH )m̄t and 1 − ρH > 0, the equality of the two unit directions in Equation 47 holds. Therefore, ⊤ ⊤ h⊤ t dt = m̄t dt + (ht − m̄t ) dt

≥ ∥m̄t ∥2 − ∥ht − m̄t ∥2 ≥ ∥m̄t ∥2 − et,H (δ).

(58)

Proof. For fixed ρ < 1, Equation 60 follows directly from Equations 38 and 39. Substituting ρ = 1 − x into the limiting expressions gives d0 +

ρ 1 = + d0 − 1, 1−ρ x

Consequently, r p x 2σ 2ζδ = 2σ ζδ x (1 + O(x)) 2−x √ = Cδ σ x + O(σx3/2 ),

2

≤ −ηt

Lηt ∥m̄t ∥2 − et,H (δ) − 2

Cδ σ Γ f ′ (x) = − 2 + √ . x 2 x

(65)

The equation f ′ (x) = 0 has the unique solution in Equation 62. At this point, f ′′ (x) = 3Γ/(2x3 ) > 0, which proves the minimization claim.

Lηt2 ≤ −ηt h⊤ t dt + 

(64)

which proves Equation 61. Finally,

By the descent lemma, (A1), and ht = −∇Lt (zt ), Lt (zt + ηt dt ) − Lt (zt )

1−ρ x = . 1+ρ 2−x (63)

 .

(59)

This proves Equations 48 and 49. Corollary A.2 (Sequential infinite-memory scaling). For each fixed ρ ∈ [0, 1) and fixed d0 , ρ , H→∞ 1−ρ 1−ρ lim νH (ρ) = . H→∞ 1+ρ lim ℓH (ρ, d0 ) = d0 +

(60)

After taking this H → ∞ limit, set x = 1 − ρ. If Γ, σ > 0, then, as x ↓ 0, the limiting tracking radius satisfies √ Γ + Cδ σ x x + Γ(d0 − 1) + O(σx3/2 ), p Cδ = 2 ζδ .

ε∞ (x) =

(61)

The unique stationary point of the leading-order √ proxy f (x) = Γ/x + Cδ σ x is

A.3

Transfer to exact first-slot acceptance

Let p be a probability distribution on V, let ∅ ̸= C ⊆ V, and let qC be a probability distribution supported on C, extended by zero on C c . Assume p(C) > 0 and supp(p|C ) ⊆ supp(qC ). Define µC = p(C c ), p(v) pC (v) = , v ∈ C, 1 − µC KC = DKL (pC ∥qC ).

(66)

Proposition A.2 (Candidate-support acceptance certificate). If the first proposal Y is distributed according to qC and the exact verifier accepts it with conditional probability min{1, p(Y )/qC (Y )}, then its mean acceptance probability is X A1 (p, qC ) = min{p(v), qC (v)} (67) v∈V = 1 − TV(p, qC ). Moreover,

 xproxy =

2Γ Cδ σ

2/3 .

(62)

It is the minimizer of f on (0, ∞). On the admissible interval x ∈ (0, 1], the corresponding proxy minimizer is min{1, xproxy }. Equation 62 is an asymptotic design rule when xproxy ≪ 1; it is not asserted to minimize the finite-H radius in Equation 43.

" A1 (p, qC ) ≥ (1 − µC ) 1 −

r

KC 2

# .

(68)

+

Proof. Since qC (v) > 0 for every v that can be sampled from qC ,   X p(v) A1 (p, qC ) = qC (v) min 1, qC (v) v∈C X = min{p(v), qC (v)}. (69) v∈V

For probability distributions a and b on a common finite space, X min{a(v), b(v)} = 1 − TV(a, b). (70)

DKL (pS ∥qS ) does not, in general, imply either an increase in A1 (p, qC ) or an increase in the certificate associated with C, even when p and C remain fixed.

v

This proves Equation 67. Let a = 1 − µC ∈ (0, 1]. For all x, y ≥ 0, min{ax, y} ≥ a min{x, y}. Therefore, X A1 (p, qC ) = min{apC (v), qC (v)} ≥a

min{pC (v), qC (v)}

v∈C

= a (1 − TV(pC , qC )) .

(71)

Pinsker’s inequality gives TV(pC , qC ) ≤ p KC /2. Combining this inequality with A1 (p, qC ) ≥ 0 proves Equation 68. Corollary A.3 (Fixed-support certificate transfer). Suppose that p and C remain fixed during the Reflex perturbation and define

qA (v) = P

uv

,

w∈A uw u′ ′ qA (v) = P v ′ , w∈A uw

(77) v ∈ A.

These restricted distributions can be realized by a common softmax model. Specifically, take dz = 3, W = I3 , τ = 1, and z = (log u1 , log u2 , log u3 ),

Lt (z) = DKL (pC ∥qC (z)) .

(72)

If Theorem A.1 applies to this loss and Equation 49 holds on Et,H,δ , then KC′ := DKL (pC ∥qC (zt + ∆zt )) < KC

on Et,H,δ .

If KC < 2, define the certificate value h i p BC (K) = (1 − µC ) 1 − K/2 . +

(76)

Let u = (1, 1, 1/5) and u′ = (5, 4, 10). For every nonempty A ⊆ V, define

v∈C

X

Proof. Let V = {1, 2, 3} and   1 2 1 p= , , , 2 5 10 S = {1, 2}, C = {1, 3}.

(73)

(74)

z ′ = (log u′1 , log u′2 , log u′3 ).

(78)

Direct calculation gives       1 1 5 4 5 4 , , qS = , , qS′ = , , pS = 9 9 2 2 9 9       5 1 1 2 5 1 ′ , , qC = , , qC = , . pC = 6 6 6 6 3 3 (79) Hence, by strict positivity of KL divergence away from equality,

Then BC (KC′ ) − BC (KC ) ! r r (75) KC′ KC = (1 − µC ) − > 0. 2 2 This conclusion concerns the certified lower bound; it does not, by itself, imply A1 (p, qC (zt + ∆zt )) > A1 (p, qC (zt )). Proof. The strict inequality KC′ < KC is Equation 48 under Equation 49. Because p and C are fixed, µC is unchanged. Since KC′ < KC < 2, the clipping operation in BC is inactive at both arguments. Subtraction yields Equation 75. Proposition A.3 (Failure of support-agnostic transfer). If the diagnostic support S differs from the verifier support C, strict descent of

DKL (pS ∥qS′ ) = 0 < DKL (pS ∥qS ).

(80)

On the other hand, Equation 67 yields     1 5 1 1 3 A1 (p, qC ) = min , + min , = , 2 6 10 6 5     1 1 1 2 13 ′ A1 (p, qC ) = min , + min , = . 2 3 10 3 30 (81) ′ ) < A (p, q ) despite strict KL imThus A1 (p, qC 1 C provement on S. Moreover, the initial candidatesupport KL equals zero whereas the corrected candidate-support KL is strictly positive, so the certificate associated with C also decreases. This proves the claim.

B

Exact Tree Verification

This appendix specifies the sampling semantics used in all rollout experiments. The future-token tree is used to batch target evaluations; token commitment follows the stochastic node-wise procedure below, not longest-prefix matching or greedy branch selection. B.1

Tree Proposal Law

Let a ∼ p0 = πθ (· | x≤t ) be an exact targetsampled anchor token. The speculative tree is rooted at prefix x≤t a. A node u at speculative depth d ∈ {0, . . . , K − 1} represents the unique prefix path(u) = (a, v1 , . . . , vd ). (82) Its next speculative position is proposed by head k = d + 1. Let q̄u denote that head’s base distribution after Reflex. In the current implementation, nodes at the same depth share the same head logits, so q̄u may be numerically independent of path(u). This does not affect correctness: an arbitrary distribution that ignores part of its conditioning context is still a valid conditional proposal. The tree builder selects a finite support Cu under the concurrency budget and samples an ordered list of mu distinct children without replacement. If yu,1:j−1 have already been drawn, define the remaining set Ru,j = Cu \ {yu,1 , . . . , yu,j−1 }

(83)

and the slot-specific conditional proposal 1[v ∈ Ru,j ]q̄u (v) qu,j (v) = P . w∈Ru,j q̄u (w)

(84)

Then yu,j ∼ qu,j . The probability of the ordered child list is Pr(yu,1:mu ) =

mu Y

qu,j (yu,j ),

(85)

j=1

and the proposal factor associated Q with a branch that accepts slot jℓ at node uℓ is ℓ quℓ ,jℓ (vℓ ). Exactness is nevertheless established node by node and does not require forming a single branch-level acceptance ratio. Within a parent, sampling without replacement prevents duplicate token children. The same token label may appear below different parents because those nodes represent different autoregressive prefixes. If an implementation generates duplicate aliases for an identical full prefix, they are

canonicalized before target evaluation and retain parent-specific proposal bookkeeping. For every node u, the packed target verification forward computes pu (v) = πθ (v | x≤t , path(u)) .

(86)

Tree packing is valid only when these logits equal the logits from an ordinary autoregressive target forward at the same prefix, up to numerical precision. B.2

Node-Wise Acceptance and Residuals (0)

At node u, initialize the target residual ru = pu . Candidate yu,j is accepted with probability ! (j−1) ru (yu,j ) Au,j = min 1, . (87) qu,j (yu,j ) After rejection, update the full-vocabulary residual (j−1)

ru(j) (v) = Zu,j =

[ru

(v) − qu,j (v)]+ , Zu,j

X [ru(j−1) (w) − qu,j (w)]+ .

(88)

w

If a candidate is accepted, the verifier commits it and recurses into its child. If all siblings are rejected, it samples one fallback token from the final residual and ends the round. When the deepest speculative candidate is accepted, it samples one bonus token from the target distribution at the resulting leaf. Algorithm 1 Exact node-wise tree verification 1: function V ERIFY N ODE(u) 2: r ← pu 3: for j = 1, . . . , mu do 4: y ← yu,j and q ← qu,j 5: Draw U ∼ Uniform(0, 1) 6: if U ≤ min{1, r(y)/q(y)} then 7: Commit y 8: if y has a speculative child subtree then 9: return y ∥ V ERIFY N ODE(child(u, y)) 10: else 11: Draw bonus b ∼ pchild(u,y) 12: return (y, b) 13: end if 14: else P 15: r(v) ← [r(v)−q(v)]+ / w [r(w)−q(w)]+ 16: end if 17: end for 18: Draw fallback f ∼ r 19: return f 20: end function

B.3

B.4

Sampling Exactness

Proposition B.1 (One-node correction). Conditioned on the prefix represented by u, Algorithm 1 emits its first committed token from pu . Proof. Consider slot k conditioned on all previous candidate draws and rejections. The candidate is distributed as Yj ∼ qu,j . Its accepted contribution to output token v is ! (j−1) ru (v) qu,j (v) min 1, qu,j (v) = min{qu,j (v), ru(j−1) (v)}.

(89)

The probability of rejection is Zu,j , and the conditional continuation distribution in Equation 88 contributes Zu,j ru(j) (v) = [ru(j−1) (v) − qu,j (v)]+ .

(90)

The sum of the accepted and rejected contributions (j−1) is exactly ru (v). Backward induction over the remaining slots, with the final fallback sampled from the last residual, shows that the node emits (0) ru = pu . Theorem B.1 (Exact target continuation). Let all node probabilities satisfy Equation 86, and let candidate lists be generated according to Equation 84. The anchor followed by Algorithm 1 has the same joint distribution as ordinary autoregressive sampling from πθ for every committed token in the round. Proof. The anchor is sampled from the target by construction. Proposition B.1 shows that the first speculative or fallback token below any visited node has target conditional distribution pu . Conditional on accepting token v at node u, the child represents exactly the extended prefix (path(u), v), and the same proposition applies recursively. A rejected node terminates with a target-residual fallback whose marginal is already included in pu ; an accepted deepest node terminates with a direct target bonus sample. Induction on remaining tree depth and the autoregressive chain rule therefore give the target joint distribution for the complete committed continuation. Sampling versus greedy decoding. The exactness claim applies to the stochastic verifier above. A longest-accepted-branch or greedy tree rule may preserve a greedy output under separate conditions, but it is not used for the sampled GRPO rollouts reported here.

Justification of the Conditional-Overlap Surrogate

We first recall the maximal-coupling interpretation of the overlap coefficient and then relate the proposed surrogate to the exact node-wise verifier. Proposition B.2 (Conditional overlap and verifier acceptance). Let C ⊆ V satisfy p(C), q(C) > 0, and write s = p(C). Let Aver C be the probability that the exact node-wise verifier accepts at least one speculative candidate from C, conditioned on reaching the node. Suppose that the first candidate is drawn from qC , as in Equation 84. Then p(C)ΩC ≤ Aver C ≤ p(C),

(91)

where ΩC =

X

min{pC (v), qC (v)} = 1−TV(pC , qC ).

v∈C

(92) Consequently, 0 ≤ Aver C − p(C)ΩC ≤ p(C) TV(pC , qC ). (93) Moreover, ΩC is the largest agreement probability attainable by any coupling of pC and qC . Proof. For any nonnegative a and b, min{a, b} =

a + b − |a − b| . 2

(94)

Because both pC and qC sum to one, 1X (pC (v) + qC (v) − |pC (v) − qC (v)|) 2 v∈C 1X =1− |pC (v) − qC (v)| 2

ΩC =

v∈C

= 1 − TV(pC , qC ).

(95)

To prove the coupling statement, consider any coupling (X, Y ) with marginals pC and qC . For each v ∈ C, Pr(X = Y = v) ≤ min{pC (v), qC (v)}.

(96)

Summing over v gives Pr(X = Y ) ≤ ΩC .

(97)

This bound is achievable. Assign common probability mass m(v) = min{pC (v), qC (v)}

(98)

to the event X = Y = v. The remaining pC - and qC -masses have disjoint supports and can therefore be coupled without further agreement. This constructs a coupling satisfying Pr(X = Y ) = P v m(v) = ΩC . It remains to relate this overlap to verifier ac(1) ceptance. Let AC be the probability that the first candidate is accepted. Because the first candidate Y is distributed as qC and is accepted using the exact rejection-sampling rule,   X p(v) (1) AC = qC (v) min 1, qC (v) v∈C X = min{qC (v), p(v)}. (99) v∈C

Since p(v) = spC (v) for v ∈ C and s ≤ 1, we have qC (v) ≥ sqC (v). Hence, X (1) AC ≥ min{sqC (v), spC (v)} v∈C

=s

X

min{qC (v), pC (v)}

v∈C

= p(C)ΩC .

(100)

Acceptance at the first slot implies acceptance somewhere in the node, so (1)

Aver C ≥ AC ≥ p(C)ΩC .

C

End-to-End Algorithm and Implementation

Algorithm 2 One exact speculative round for sequence i 1: Compute ht = fθ (x≤t ) and sample anchor a ∼ πθ (· | x≤t ). 2: for h = 1, . . . , H do 3: Predict position t + h + 1 with zt,h = gϕh (ht ). 4: if feedback, alignment, and coherence gates pass then 5: Form bounded ∆zt,h from mi,h and apply the boundary safeguard. 6: else 7: ∆zt,h ← 0. 8: end if 9: Form the corrected head distribution and store a detached proposal record. 10: end for 11: Set Nt from active concurrency; build ordered, withoutreplacement child lists using Equation 84. 12: Evaluate pu for every unique tree prefix in one packed target forward. 13: Append a ∥ V ERIFY N ODE(root) using Algorithm 1. 14: for each record whose target position has matured do 15: Build S; compute rdist , rcov , w; update memory, alignment statistics, and reservoir. 16: end for 17: If persistent drift and guardrails pass, run the bounded auxiliary update.

C.1

Implementation and Reproducibility Details

Training protocol. We train Qwen2.5-1.5B, 3B, 7B, and 14B and Llama-3.1-8B on GSM8K, For the upper bound, let X be the first token SimpleRL-Abel-Level3to5, and DAPO-Mathemitted by the exact node-wise verifier. By exact 17K. For each dataset, training uses seed 42 and verification, X ∼ p. Whenever a speculative canproceeds for one epoch. Each prompt produces didate is accepted, the emitted token belongs to C. eight responses. Rollouts use stochastic sampling Therefore, with temperature 1.0, nucleus threshold 0.95, no top-k truncation, a maximum prompt length Aver (102) of 2048 tokens, and a maximum total sequence C ≤ Pr(X ∈ C) = p(C). length of 2048 tokens. Combining the two bounds proves Equation 91. The per-device prompt batch sizes for Qwen2.5Finally, 1.5B, 3B, 7B, 14B, and Llama-3.1-8B are 16, 8, 8, 4, and 8, respectively. The corresponding ver AC − p(C)ΩC ≤ p(C) − p(C) [1 − TV(pC , qC )] gradient-accumulation factors are 2, 4, 4, 8, and = p(C) TV(pC , qC ), (103) 4, yielding an effective batch of 32 prompts, or 256 sampled responses, per policy update. Groups which proves Equation 93. with zero reward variance do not contribute to the GRPO update. The reward is (101)

R(x) = Racc (x) + 0.2Rfmt (x),

(104)

where Racc is symbolic answer correctness and Rfmt indicates compliance with the prescribed reasoning format.

The target policy is optimized with AdamW at learning rate 10−6 . GRPO uses KL coefficient β = 0.04 and clipping parameter ϵ = 0.1. Policy adaptation is restricted to LoRA parameters with rank 64, scale 32, and zero dropout. LoRA modules are attached to the query, key, value, output, gating, up-projection, and down-projection transformations. These target-side settings are identical for vanilla GRPO, FastGRPO, and all SpecRoll variants. Future-token proposer. We instantiate three horizon-specific heads for positions t+2, t+3, and t + 4. Each head applies a residual transformation gϕh (h) = h + SiLU(Ah LN(h) + bh )

(105)

and uses the frozen target vocabulary projection to obtain proposal logits. The target backbone and vocabulary projection are not updated during head pretraining or auxiliary adaptation. The heads are pretrained for one epoch on the complete ShareGPT-V4.3-derived corpus using assistant-token cross-entropy. The three horizon losses are weighted by 1, 0.8, and 0.82 . Pretraining uses AdamW with learning rate 3 × 10−4 , zero weight decay, 100 linear warm-up steps, gradientnorm bound 1.0, and maximum sequence length 1024. The model-specific microbatch and accumulation pairs are (16, 2), (8, 4), (4, 8), (2, 16), and (4, 8) in the model order stated above, giving a common effective pretraining batch size of 32. For comparison, the FastGRPO drafter is pretrained independently on the same ShareGPTderived corpus for one epoch, with maximum sequence length 2048, learning rate 5 × 10−5 , and a 5% linear warm-up fraction. Its online drafter learning rate is 10−4 . Thus, the two proposers share their pretraining data source, although their architectures, objectives, and sequence lengths differ. Concurrency-aware sparse tree verification. The per-response tree budget is adapted to the number of unfinished responses Bt as Nt = clip(⌊Cpeak /Bt ⌋ , Nmin , Nmax ) .

(106)

We set Cpeak = 512, Nmin = 1, and Nmax = 10, with the root included in Nt . The nominal nonroot allocation across the three prediction horizons is (5, 4, 0) and is truncated when necessary to satisfy the available budget. All represented prefixes are evaluated jointly in a single packed target-model forward pass.

Verifier-derived feedback. For each matured proposal, the feedback support is the deduplicated union of the proposal top-48 tokens, target top-48 tokens, the realized target token, and the raw candidate identifiers. The union is truncated to at most 48 entries. Both target and proposal distributions are renormalized on this common support. Distribution and coverage directions are combined with respective weights 0.15 and 1.25. Feedback severity is
w = clip 0.3 TV(pS , qS ) + 0.7 pout , 0, 1 , (107) and observations with w < 0.03 are discarded. The coverage term uses the four proposal tokens nearest the selection boundary, and the candidate-boundary safeguard requires a positive logit-margin change with margin 0.15. Reflex configuration. Reflex maintains an independent FP32 memory for every trajectory– horizon pair. Its memory update uses horizonspecific decay ρk . Proposal-time and verifierderived directions are represented by normalized random projections of rank 24, generated with seed 29. Alignment means and variances are accumulated using the exact online sample moments; no exponential averaging is applied to these alignment statistics. Let [αkmin , αkmax ] denote the base relative-RMS interval. Conditional on an eligible reliability score, the correction ratio lies in [αkmin , αkmax ],

(108)

before any counterfactual safety reduction. Table 3 reports the shared horizon-specific configualign are the miniration. The quantities nfb k and nk mum effective feedback and alignment counts, respectively. Counterfactual safety statistics compare raw and corrected candidate mass and candidate wins. The minimum probe count is 128 for all models. The candidate mass and net-win deadbands are 5 × 10−4 and 2 × 10−3 . One adverse window reduces the active ratio by a factor of 0.5; two favorable windows permit gradual recovery. Auxiliary head adaptation. Auxiliary adaptation updates only the horizon-specific proposal heads using detached verifier records; the target backbone and vocabulary projection remain frozen. An auxiliary-update event is triggered only

Models

(ρ1 , ρ2 , ρ3 )

All models (.85,.90,.95)

align fb fb (nfb , nalign , nalign ) Correction intervals (%) 1 , n2 , n3 ) (n1 2 3

(1,2,4)

(6,16,48)

zδ

Feedback stride

(1.0–2.5, .4–1.0, .25–.6) .60

4

Table 3: Shared horizon-specific Reflex hyperparameters. Correction intervals are expressed as percentages of the proposal hidden-state RMS.

if there exists a horizon k satisfying all of the following conditions: (i) drift is detected in three consecutive monitoring windows of 64 matured records, where restricted TV exceeds its calibrated baseline and either acceptance falls below its baseline or candidate regret rises above its baseline; (ii) the reservoir contains at least Nmin = 64 matured records for horizon k; (iii) at least Gmin = 10 complete rollout batches have elapsed since the preceding auxiliary-update event; and (iv) the projected cumulative auxiliary time does not exceed 1.8% of rollout-generation time. Formally, (3)

Eligiblek = Dk ∧ [Nk ≥ 64] ∧ [g − glast ≥ 10]   Taux + b cupd ∧ ≤ 0.018 . Troll (109) where g is the current completed-rollout-batch index, glast is the index of the most recent auxiliary update, and b cupd is the estimated cost of the pending update. If multiple horizons are eligible, only the one with the largest calibrated degradation is updated. The selected update uses at most 256 mature records. For the selected horizon, we minimize Laux = DKL (pS ∥qSraw ) + 0.1DKL (pS ∥qSeff ) + 1.5Lrank + λp Lprox , (110) using temperature 1.0 and ranking margin 0.20. Each event performs one AdamW step with zero weight decay, numerical constant 10−6 , and gradient-norm bound 1.0. The resulting parameter displacement is contracted by 1/(1 + λp ). After three warm-up updates, the record budget is multiplied by 0.75 after two consecutive budget violations and by 1.20 after four consecutive compliant updates. Metrics and timing. Let Tr be the candidate tree and Pr the committed target-consistent path in verification round r, both including the target-

sampled root. The reported metrics are R

1 X |Pr |, R r=1 PR (|Pr | − 1) AR = Pr=1 . R r=1 (|Tr | − 1)

AAL =

(111) (112)

Thus, AAL includes the exact root, whereas acceptance rate excludes the root but counts every non-root candidate placed in the tree. All experiments use a single NVIDIA B200. Generation time includes target prefill, futuretoken-head evaluation, tree construction, packed target evaluation, accepted-path cache extraction, verifier feedback, and Reflex bookkeeping. Complete wall-clock time is measured from the first training batch through the final checkpoint and further includes reward computation, policy optimization, auxiliary head updates, and checkpointing; model and dataset initialization are excluded. Each reported table entry corresponds to one completed matched run and is normalized by the corresponding vanilla-GRPO run.

D

Runtime Scaling with Training Progress

To examine whether the reported end-to-end measurements reflect stable runtime accumulation throughout training, rather than an advantage concentrated in a particular training phase, we record cumulative wall-clock time at 20% intervals for Qwen2.5-1.5B on SimpleRL-AbelLevel3to5. FastGRPO and SpecRoll use the same target checkpoint, training examples and their order, GRPO configuration, response limit, reward computation, and hardware environment. The rollout engine is the only method-level difference. Figure 4 shows that cumulative runtime grows approximately linearly with the number of processed training examples for both rollout engines. A least-squares linear fit over the five checkpoints gives R2 = 0.998 for SpecRoll and R2 = 0.988 for FastGRPO. SpecRoll remains faster at every observed checkpoint, and the cumulative runtime gap increases from 0.74 hours after 20% of the

Model Qwen2.5-1.5B Qwen2.5-3B Qwen2.5-7B Qwen2.5-14B Llama-3.1-8B

LR −5

8×10 7×10−5 5×10−5 3×10−5 4×10−5

Records Rollout gap Buffer Batch Overhead 256 256 256 256 256

10 10 10 10 10

4096 4096 4096 4096 4096

64 64 64 64 64

1.8% 1.8% 1.8% 1.8% 1.8%

λp .020 .020 .020 .020 .020

Table 4: Auxiliary-adaptation hyperparameters. Records denotes the maximum number of mature records used per update.

Figure 4: Cumulative end-to-end runtime as a function of the fraction of training examples processed for Qwen2.51.5B on SimpleRL-Abel-Level3to5. Runtime is recorded at 20% intervals from the same matched training run. Both methods exhibit approximately linear cumulative scaling, while SpecRoll remains faster than FastGRPO at every checkpoint.

data to 4.49 hours after the complete run. Thus, the final throughput advantage is not caused by a short-lived gain or a late-stage timing anomaly. The near-linear traces also make partial-run completion estimates possible, although such estimates should be interpreted as approximate rather than exact. Fitting a line to the observations available through 40% of training predicts completion times of 23.53 hours for SpecRoll and 28.19 hours for FastGRPO, compared with the observed 21.41 and 25.90 hours. Using observations through 60% predicts 22.67 and 29.95 hours, respectively. These estimates recover both the ordering and the approximate scale of the final runtimes, but their accuracy can still be affected by changes in response length, active-batch composition, and the timing of auxiliary updates.

E

Predictive Validity of the Reflex Reliability Gate

The reliability gate is designed to reuse a stored memory direction only when past delayed feedback suggests that it will remain useful for future proposals. To test this claim, we run a Reflexonly diagnostic on Qwen2.5-1.5B with SimpleRLAbel-Level3to5, with auxiliary adaptation disabled. We use the first prediction horizon and keep the target model, pretrained proposal heads, candidate budget, verifier, and training configuration fixed. For each proposal record j, we save the reliability score and a normalized sketch sj of the memory direction before its verifier error is available. When the record later matures, we obtain the raw

(a) Predictive alignment

(b) Counterfactual shadow utility 0.8

0.60

0.7

0.55

0.6

0.50

0.5

0.45 0.40 0.35

Mean raw-error cosine Positive-alignment rate

0.9 0.8

0.60

0.7

0.55

0.6

0.50

0.5

0.45

0.4

, 0) 01) 03) 05) 07) .1) 15) .2) inf) [-inf [0, 0.0.01, 0.0.03, 0.0.05, 0. [0.07, 0[0.1, 0. [0.15, 0 [0.2, [ [ [

1.0

Beneficial rate

0.65

Mean KL Beneficial rate

0.65

Mean restricted KL

0.70

0.9 Positive rate

Mean raw-error cosine

0.75

0.4

, 0) 01) 03) 05) 07) .1) 15) .2) inf) [-inf [0, 0.0.01, 0.0.03, 0.0.05, 0. [0.07, 0[0.1, 0. [0.15, 0 [0.2, [ [ [

Proposal-time gate score

Proposal-time gate score

Figure 5: Predictive validity of the proposal-time Reflex reliability score on Qwen2.5-1.5B with SimpleRL-AbelLevel3to5. Panel (a): alignment between the stored memory direction and the verifier error observed after the proposal record matures. Panel (b): restricted-KL improvement from a counterfactual shadow correction. Points d = 0. show averages within each proposal-time score bin, and the dashed line marks the operational threshold Rel

verifier-error direction eraw and measure j
craw = cos sj , eraw . j j

(113)

A positive value means that the stored memory direction predicted the direction of the error revealed later. We also test whether following the stored direction would improve the proposal distribution. For this purpose, we construct a small shadow correction zjshadow = zjraw + αshadow RMS(zjraw ) mj × , RMS(mj ) + ϵ with αshadow = 0.005, and compute
raw ∆KLshadow = DKL pS ∥qS,j j   shadow − DKL pS ∥qS,j .

(114)

(115)

A positive ∆KLshadow means that the shadow step j moves the proposal distribution closer to the target. This state is used only for analysis and never affects the actual rollout. As shown in Figure 5, proposals in the highest score bin exhibit the strongest subsequent alignment, reaching a mean cosine of about 0.62 and a positive-alignment rate of about 0.78. The same bin also produces the largest counterfactual gain, with a mean restricted-KL improvement of about 0.59 and a beneficial rate close to 0.78. The lowand intermediate-score bins show smaller differences, while the clearest separation appears for

Top-k

Gen.

E2E

AAL

AR

16

1.43

1.39

1.720

0.089

24

1.44

1.40

1.724

0.090

32

1.35

1.33

1.723

0.087

48

1.46

1.42

1.726

0.093

64

1.43

1.39

1.722

0.089

Table 5: Sparse-support sensitivity on Qwen2.5-3B with SimpleRL-Abel-Level3to5. Speedups are relative to matched vanilla GRPO; AR denotes acceptance rate.

strongly positive scores. These results indicate that the reliability score can identify memory directions that are more likely to remain useful when delayed verifier feedback arrives, supporting its use as a conservative trigger for Reflex.

F

Sparse-Support Top-k Ablation

We vary the support top-k budget in Equation 12 on Qwen2.5-3B with SimpleRL-Abel-Level3to5. The k = 48 row is the setting used in the main result. All other components, including candidatetree budget and Reflex configuration, are unchanged. The quality diagnostics remain within a narrow range but are no longer flat. Increasing k from 16 to 24 raises AAL by 0.004 and acceptance rate by 0.001; at k = 48, the gains relative to k = 16 reach 0.006 AAL and 0.004 acceptance rate. The k = 32 run instead drops to 0.087 acceptance rate, and k = 64 returns to the k = 16 acceptance rate. Across the sweep, AAL varies by 0.006 and ac-

4ZHQ% *60.5DZ5HZDUG

 

0HDQUHZDUG

    

9DQLOOD*532 6SHF5ROO

 







7UDLQLQJVWHS









4ZHQ% *60.5DZ5HZDUG



0HDQUHZDUG

     

9DQLOOD*532 6SHF5ROO 











7UDLQLQJVWHS









Figure 6: Training-reward dynamics on GSM8K. SpecRoll and vanilla GRPO exhibit comparable raw-reward trajectories for Qwen2.5-1.5B and Qwen2.5-3B.

ceptance rate by 0.006. This suggests that the support budget has a modest, non-monotonic effect on the restricted error geometry and candidate acceptance. Runtime is likewise non-monotonic. Relative to k = 16, k = 24 improves end-to-end speed by about 0.7%, and k = 48 by about 2.2%. The k = 32 run is approximately 4.3% slower end to end and also has the lowest acceptance rate, while k = 64 returns to the k = 16 timing. The k = 48 configuration is best on both throughput and acceptance diagnostics, although the absolute differences remain small and the intermediate settings are not ordered by support size. We therefore use k = 48 in the main experiments because it gives the strongest observed generation speedup, endto-end speedup, AAL, and acceptance rate in this sweep. The non-monotonic pattern should nevertheless be interpreted as a sensitivity result rather than a general scaling law.

G

Training-Reward Dynamics

SpecRoll and vanilla GRPO on GSM8K. Across both Qwen2.5-1.5B and Qwen2.5-3B, the two methods exhibit comparable reward trajectories and similar overall learning trends. This result indicates that replacing the standard rollout procedure with SpecRoll does not visibly alter or de-

Dataset GSM8K SimpleLR Abel L3–5 DAPO Math

Triggers

Median

Mean

Maximum

46 53 54

0.247 s 0.519 s 0.052 s

0.844 s 0.830 s 0.333 s

12.596 s 2.879 s 2.988 s

Table 6: Frequency and latency of auxiliary updates.

grade the underlying GRPO optimization dynamics.

H

Auxiliary-Update Overhead

Figure 7 visualizes the latency of individual auxiliary updates, while Table 6 summarizes their frequency and latency statistics. The auxiliary path is triggered only occasionally, with 46 updates on GSM8K, 53 on SimpleLR Abel L3– 5, and 54 on DAPO Math. The median update times are 0.247 s, 0.519 s, and 0.052 s, respectively, while the mean remains below 0.85 s for all three datasets. The larger maximum of 12.596 s on GSM8K results from a rare outlier rather than persistent overhead. Overall, auxiliary adaptation introduces sparse and predominantly sub-second computation.

D*60.

$X[XSGDWHWLPHV

E6LPSOH/5$EHO/ 8SGDWHVn = 46 0HDQV 0HGLDQV



F'$320DWK

8SGDWHVn = 53 0HDQV 0HGLDQV



8SGDWHVn = 54 0HDQV 0HGLDQV



  







 



 





(ODSVHGWLPHK











(ODSVHGWLPHK











(ODSVHGWLPHK



Figure 7: Wall-clock latency of triggered auxiliary updates. Each point denotes one update; dashed and dotted lines indicate the mean and median latency, respectively.

Related documents

• TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning#394431
• RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning#310784
• ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning#1130021
• Explainable Reinforcement Learning via Physics-Aware Policy Distillation#405644
• LOLLA: Deep Reinforcement Learning for Closed-Loop Link Adaptation Towards a GPU-Accelerated AI-RAN#299860