Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation Qian Tan1 , Huaifei Liang1 , Xuanyu Zhu3 , Lei Jiang1 , Yuqiang Li2,† , 1
University of Science and Technology of China 2 Shanghai Artificial Intelligence Laboratory 3 Shanghai Jiao Tong University [email protected],[email protected]
arXiv:2607.29494v1 [cs.LG] 31 Jul 2026
Abstract On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost, particularly when a few long responses delay batch completion. Existing acceleration methods typically control rollout length using fixed budgets or absolute teacher–student agreement thresholds, which may not reflect learning progress across different models and training stages. We propose Adaptive FastOPD, a progress-aware strategy that expands the rollout horizon only when learning near the current boundary region has plateaued and the current horizon is sufficiently utilized. The former is determined from four teacher–student signals measured relative to their values upon entering each horizon, making expansion responsive to stage-specific progress rather than a predefined step interval or an absolute threshold on the raw agreement signals, while the latter prevents a small number of long responses from triggering increases in rollout cost. Across two teacher–student pairs, Adaptive FastOPD achieves the highest average performance while reducing training time by 49.1– 71.2% relative to OPD 15K, and remains robust across a range of hyperparameter settings.
Introduction On-policy distillation (OPD) has emerged as an important post-training approach for transferring reasoning behavior from a strong teacher model to a student. OPD samples sequences from the current student and uses the teacher’s tokenlevel distributions on student-generated prefixes as supervision (Agarwal et al. 2024). This on-policy interaction supplies dense token-level supervision along a reasoning trace. Despite these advantages, the online training loop remains a major efficiency bottleneck. The cost grows with the rollout horizon. Response lengths can also vary within a batch, introducing additional inefficiency. This problem is severe when a few long responses continue decoding after most have terminated. Since a synchronous training iteration cannot proceed until its rollout batch is completed, these long-tail responses can become stragglers that determine the wall-clock time of the rollout phase (Shao et al. 2026; Khan et al. 2026). To alleviate these issues, recent work has explored shortening, progressively expanding, or selectively continuing OPD rollouts (Zhang et al. 2026a; Yang et al. 2026; Ziheng et al. 2026; Zhang et al. 2026c; Zhao et al. 2026). FastOPD observes that useful supervision is often concentrated near
the beginning of a response and progressively increases the horizon with a fixed schedule (Zhang et al. 2026a). PruneOPD detects local student–teacher drift and truncates supervision after an overlap-based reliability score crosses a threshold (Yang et al. 2026). Early Stopping Rollout (ESR) argues that teacher supervision becomes less corrective at later positions and therefore restricts rollout generation to a short predefined rollout horizon (Ziheng et al. 2026). Collectively, these studies show that truncating or selectively extending rollouts can accelerate OPD while matching the performance of full-rollout training. Despite these advances, existing rollout-length adaptation methods remain sensitive to hyperparameters that directly determine rollout expansion or truncation. Fixed-truncation methods require the cutoff length to be selected in advance (Ziheng et al. 2026), while FastOPD (Zhang et al. 2026a) expands the rollout horizon at a fixed interval. PruneOPD relies on an overlap threshold γ to control reward pruning and response-length adaptation (Yang et al. 2026). PrefixGuided OPD (PG-OPD) instead requires a continuation budget that determines how many candidate trajectories receive long rollouts (Zhao et al. 2026). The appropriate values of these control parameters may differ across models, datasets, and training stages. Applying the methods to a new setting may require new hyperparameter searches, whose additional computational cost can partially offset the intended acceleration, or suboptimal settings may degrade downstream performance. These limitations motivate the development of a more adaptive and flexible approach. To this end, we propose Adaptive FastOPD, a progressaware rollout horizon expansion strategy for efficient onpolicy distillation. Starting from a short rollout horizon, Adaptive FastOPD continuously monitors learning progress in the boundary region and expands the horizon only when this progress has plateaued and the current horizon is utilized. Instead of comparing raw teacher–student signals against a globally fixed target, Adaptive FastOPD establishes a stagespecific baseline upon entering each horizon and measures subsequent changes relative to the initial state. This selfreferenced design reduces reliance on manually specified absolute thresholds for the raw teacher–student signals. Specifically, within the current boundary region, we measure four complementary signals and convert these signals into normalized badness scores, conservatively aggregate them using
Figure 1: Overview of Adaptive FastOPD. The student first generates responses under the current horizon H, defined as the maximum response length at the current stage. Within the current boundary region, we measure four complementary OPD signals: top-k overlap (O), shared probability mass (P ), shared-greedy penalty (G), and unshared-token penalty (U ). These signals are normalized against the baselines established when the current horizon is first entered, aggregated through the maximum normalized badness score, and smoothed with an exponential moving average to detect the plateau. In parallel, the horizon-hit rate and boundary-reach rate verify whether the current horizon is sufficiently utilized. The horizon is expanded from H to H + ∆H only when progress has plateaued and at least one utilization criterion is satisfied. their maximum, and smooth the resulting aggregate badness score with an exponential moving average. The horizon is expanded only after the smoothed score has remained on a plateau for several consecutive updates, and the current horizon is sufficiently utilized, as measured by the horizon-hit rate or boundary-reach rate. This utilization gate ensures that the progress estimate for the boundary region is supported by enough rollouts and prevents horizon expansion when only a few long responses would dominate batch completion time, thereby reducing end-to-end rollout time (Shao et al. 2026; Khan et al. 2026). Motivated by prior findings that teacher supervision is more reliable on shorter prefixes and becomes less corrective as prefix drift accumulates (Li et al. 2026; Ziheng et al. 2026; Zhang et al. 2026a), we monitor training progress only in the current boundary region and expand horizon gradually, reducing the risk that errors accumulated in insufficiently aligned prefixes weaken teacher supervision at later positions. Experiments on DeepSeek-R1-Distill-Qwen-1.5B (Guo et al. 2025) and Qwen3-1.7B-Base (Yang et al. 2025) show that Adaptive FastOPD achieves better average performance than vanilla OPD and fixed-schedule FastOPD with shorter end-to-end training time, while showing stable performance across a range of adaptation hyperparameters. Our contributions can be summarized as: • We propose Adaptive FastOPD, a progress-aware rollouthorizon adaptation framework, allowing OPD to allocate rollout computation according to the student’s evolving
learning dynamics. • We develop a robust horizon-expansion criterion that jointly considers multi-signal learning progress and boundary utilization, enabling reliable length transitions while avoiding inefficient expansion caused by sparsely occurring long rollouts. • Experiments across two teacher–student pairs demonstrate that Adaptive FastOPD improves the accuracy– efficiency trade-off over vanilla OPD and fixed-schedule FastOPD, while maintaining stable performance across the tested adaptation settings.
Related Work On-policy distillation and long-tail rollout inefficiency. On-policy distillation (OPD) samples trajectories from the current student policy and queries the teacher for tokenlevel predictive distributions along these trajectories. GKD provides a general formulation for autoregressive models with multiple divergence objectives, while MiniLLM studies reverse-KL distillation under the student-induced distribution (Agarwal et al. 2024; Gu et al. 2024). Subsequent analyses show that OPD learning is largely driven by highprobability tokens shared by the teacher and student (Li et al. 2026; Luo et al. 2026). A major cost of OPD comes from online rollout generation. Responses in the same batch may have different lengths, creating uneven generation times. This inefficiency is especially severe when most responses finish early while a small number continue for much longer. Infer-
ence and RL systems alleviate rollout inefficiency through techniques such as continuous batching (Yu et al. 2022; Kwon et al. 2023; Sheng et al. 2025; Shao et al. 2026; Khan et al. 2026). However, such system-level techniques cannot fully remove the straggler effect, since rollout completion may still be dominated by the longest unfinished sequences. Efficient and adaptive OPD. Recent methods reduce OPD cost by shortening, pruning, or adaptively allocating rollout computation. FastOPD progressively increases the rollout horizon according to a predefined prefix-to-suffix schedule, motivated by the observation that short prefixes retain much of the benefit of full-horizon OPD (Zhang et al. 2026a). ESR uses a fixed prefix budget, while other studies explore partialrollout supervision and short-to-long training (Ziheng et al. 2026; Zhang et al. 2026c,b). Prune-OPD truncates individual trajectories based on local top-k agreement, whereas PG-OPD allocates longer continuations using prefix-level teacher–student agreement (Yang et al. 2026; Zhao et al. 2026). Lightning OPD instead avoids online teacher serving by precomputing teacher targets (Wu, Han, and Cai 2026). ADWIN uses delayed full-rollout probes to adjust subsequent windows according to update admissibility (Liang et al. 2026). In contrast to methods that trigger expansion or truncation directly from a fixed rollout budget, a predefined step interval, or an absolute agreement threshold, Adaptive FastOPD determines horizon expansion from stage-relative training progress observed at the current horizon.
Method Preliminaries: On-Policy Distillation Let xi ∼ D denote a prompt, πθ the trainable student policy, πθ̄ the frozen student policy used to generate the current rollout batch, and πT the teacher policy. Given a maximum rollout horizon H, OPD samples a student response yi = (yi,0 , . . . , yi,Li ) ∼ πθ̄ (· | xi ) with Li ≤ H. Each valid response position j defines a student-visited state si,j = (xi , yi,<j ). Following the top-k OPD configuration used in our experiments, we construct the candidate set Si,j = TopKk (πθ̄ (· | si,j )) from the student distribution. For each v ∈ Si,j , we compute student log-probability ℓSi,j,v = log πθ̄ (v | si,j ) and the corresponding teacher logprobability ℓTi,j,v = log πT (v | si,j ). The student is then optimized with the token-level clipped OPD objective LOPD (θ) = P
1
X
i,j µi,j i,j
µi,j
X
ℓclip (ρi,j,v (θ), Ai,j,v ) ,
v∈Si,j
(1) where µi,j is the response mask, ρi,j,v (θ) = exp log πθ (v | si,j ) − ℓSi,j,v , is the policy ratio, and exp(ℓS i,j,v ) Ai,j,v = wi,j,v ℓTi,j,v − ℓSi,j,v , wi,j,v = P exp(ℓS ) u∈Si,j
i,j,u
is the candidate-level advantage with student-probability weighting. Here, ℓclip denotes the clipped policy surrogate used by the underlying OPD implementation. Vanilla OPD uses the full rollout horizon Hmax throughout training. FastOPD instead starts from a short horizon H0 and
increases it by ∆H after a fixed interval of F steps: n−1 ∆H , Hn = min Hmax , H0 + F
(2)
where n denotes the optimization step, F is the predefined expansion interval, and ∆H is the length increment. Although this schedule reduces the cost of early rollouts, its fixed expansion interval implicitly assumes that all horizons require the same number of optimization steps, which may be inconsistent with the student’s learning dynamics.
Adaptive FastOPD Adaptive FastOPD replaces the fixed expansion in Equation (2) with a progress-aware horizon-expansion rule. Starting from a short horizon H0 , the method expands the current horizon by one fixed-size chunk only when two conditions are simultaneously satisfied: 1. the teacher–student signals at the current boundary region have stopped improving; and 2. the current boundary or horizon is reached by a sufficient fraction of rollouts. Let Plateaun denote the progress condition and Usen the length-utilization condition. The horizon update is min{Hn + ∆H, Hmax }, Plateaun ∧ Usen Hn+1 = Hn , otherwise, (3) Thus, Adaptive FastOPD determines the duration of each horizon from observed training progress and evaluates its signals relative to their stage-specific initial values, avoiding a fixed step interval or an absolute threshold on the raw teacher–student signals while reducing sensitivity to modeland dataset-dependent signal scales. We next detail the two conditions that jointly govern horizon expansion. Stage-Relative Progress and Plateau Detection At optimization step n, let H = Hn denote the current rollout horizon. We monitor its final chunk, which we refer to as the boundary region: BH = [H − ∆H, H),
(4)
where the monitored chunk size is identical to the horizon increment ∆H. Motivated by prior work (Li et al. 2026; Ziheng et al. 2026; Zhang et al. 2026a) which shows that teacher supervision becomes less reliable at later positions as small mismatches accumulate and shift student prefixes away from trajectories that the teacher would naturally follow, we monitor only the most fragile boundary region and expand the horizon progressively so as to reduce unreliable teacher signals on unaligned student-generated prefixes. Specifically, at each optimization step n, our implementation evaluates four signals. We define these signals below. For clarity, we omit the optimization-step superscript (n). Let VH = {(i, j) | µi,j = 1, j ∈ BH } denote the valid response positions within the boundary region, where µi,j is the response mask at position j of rollout i. Progress statistics are updated only at optimization steps satisfying |VH | > 0; otherwise, the previous monitoring state is retained. For each
(i, j) ∈ VH , let Si,j and Ti,j = TopKk (πT (· | si,j )) denote the student and teacher top-k candidate sets at state si,j , respectively. Top-k overlap (O). We measure the fraction of student top-k candidates that also appear in the teacher top-k set: X X 1 OH = 1[v ∈ Ti,j ]. (5) |VH |k (i,j)∈VH v∈Si,j
Here, 1[·] denotes the indicator function, which equals 1 when the condition holds and 0 otherwise. A larger OH indicates greater agreement between the student and teacher candidate sets. Shared probability mass (P ). Let Ii,j = Si,j ∩ Ti,j and I I define VH = {(i, j) ∈ VH : Ii,j ̸= ∅} . When VH is empty, we skip the current progress observation and retain the existing monitoring state. We first compute the average student and teacher probability mass assigned to the shared candidates: X X 1 S PH = I (6) πθ̄ (v | si,j ), |VH | I v∈I (i,j)∈VH
1 T PH = I |VH |
X
i,j
X
πT (v | si,j ).
(7)
I v∈Ii,j (i,j)∈VH
For each quantity, a lower value indicates better alignment. The scale of each signal may vary across model pairs, datasets, and horizon stages. We therefore evaluate each metric relative to its own state at the beginning of the current horizon. Let M = {O, P, G, U } denote the set of monitored metrics. For every metric m ∈ M, we collect the first Nbase valid boundary observations and define its stagespecific baseline: ( ) Nbase 1 X base (r) qm = max ϵs , q , (12) Nbase r=1 m where ϵs > 0 avoids division by zero. At training step n, the (n) raw badness qm is normalized as ! (n) qm (n) bm = log 1 + base . (13) qm Its evolution across training steps is used to track training progress at the current horizon: a continued decrease indicates ongoing improvement, whereas the absence of a new minimum over successive observations indicates that progress has plateaued. The logarithm preserves the ordering while reducing the numerical effect of unusually large values, making the subsequent smoothing and plateau detection less sensitive to transient spikes. We then aggregate the active metrics using
The shared probability-mass signal is S T PH = min{PH , PH }.
b(n) = max b(n) m . (8)
This signal complements top-k overlap by measuring whether the shared candidates also receive high probability under both policies. Shared-greedy penalty (G). For each position with Ii,j ̸= ∗ ∅, let vi,j = arg maxv∈Ii,j πθ̄ (v | si,j ) denote the student’s highest-probability shared candidate. Using the candidatelevel OPD advantage Ai,j,v , we compute X 1 ∗ GH = max 0, − I Ai,j,vi,j . (9) |VH | I (i,j)∈VH
GH examines the OPD training signal on the student’s highest-probability shared candidate. A smaller GH indicates better alignment. Unshared-token penalty (U ). Let ΩH = {(i, j, v) : (i, j) ∈ VH , v ∈ Si,j \ Ti,j } denote the studentonly candidates. Their remaining negative advantage is summarized as X 1 Ai,j,v . (10) UH = max 0, − |ΩH | (i,j,v)∈ΩH
When ΩH is empty, we set UH = 0. UH captures the remaining negative OPD signal on candidates selected only by the student. A smaller UH indicates better alignment. We then convert the four signals into badness scores: qO = 1 − O H ,
qP = 1 − PH ,
qG = GH ,
qU = UH . (11)
m∈M
(14)
The maximum defines the aggregate score by the largest normalized badness at each optimization step, so progress tracking is governed by the least aligned metric. The aggregate score is smoothed using an exponential moving average: zn = βzn−1 + (1 − β)b(n) . (15) ∗ Let zn denote the best smoothed score observed at the current horizon. After each horizon expansion, all monitoring statistics are reset for the newly exposed boundary chunk. Upon receiving the first valid progress observation at step n, ∗ we initialize zn = b(n) and set zn∗ = zn . If zn < zn−1 , we ∗ update zn and reset the plateau counter cn = 0; otherwise, the counter is incremented. The progress condition is Plateaun = 1 [cn ≥ Kpat ] , (16) where Kpat is the the required number of consecutive nonimproving observations. This stage-relative and progress-aware criterion allows different horizons to receive different numbers of optimization steps according to their observed learning progress. (n)
Length-Utilization Gate Let Li denote the generated length of rollout i at optimization step n, and let Nroll be the number of rollouts in the batch. We define Nroll h i 1 X (n) (n) ηhit = 1 Li ≥ H , (17) Nroll i=1 (n)
ηreach =
Nroll h i 1 X (n) 1 Li ≥ H − ∆H . Nroll i=1
(18)
Method
MATH-500
Minerva
Olympiad
Avg
Time
DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5B Base (Instruct) 23.3 26.2 71.0 + OPD 7K 34.7 46.2 83.7 + OPD 15K 33.9 43.5 84.6 + FastOPD (Fixed) 33.8 42.7 86.3 + Adaptive FastOPD 34.5 45.2 85.9
AIME25
AIME24
AMC23
83.0 86.0 86.2 85.8 86.1
28.0 33.2 33.5 33.2 32.8
44.7 52.1 52.1 52.5 51.8
46.0 56.0 55.6 55.7 56.1
– 7h32min 12h18min 7h14min 6h16min
Qwen3-1.7B-Base / Qwen3-8B-Base Base 1.4 + OPD 7K 3.9 + OPD 15K 3.3 + FastOPD (Fixed) 3.5 4.0 + Adaptive FastOPD
25.5 47.7 47.9 47.5 47.6
7.5 12.4 11.9 11.9 12.3
9.8 21.8 21.6 21.1 21.3
9.9 19.9 19.1 19.4 20.1
– 4h43min 9h05min 4h58min 2h37min
1.6 5.8 4.5 5.0 5.4
13.7 27.9 25.4 27.1 30.0
Table 1: Main results on mathematical reasoning benchmarks (Avg@16). Adaptive FastOPD achieves the highest average score with the shortest training time for both model pairs. Bold and underlined values denote the best and second-best results within each model pair. Time is total wall-clock training time. Method
Length Inc. Step Interval AIME25 AIME24 AMC23 MATH-500 Minerva Olympiad Avg
Time
DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5B 33.9 + FastOPD (Fixed) 1024 5 + FastOPD (Fixed) 1024 10 23.9 + FastOPD (Fixed) 1024 20 33.8 + Adaptive FastOPD 1024 Adaptive 34.5
41.4 28.3 42.7 45.2
83.4 71.2 86.3 85.9
85.7 83.3 85.8 86.1
33.0 28.0 33.2 32.8
52.1 44.9 52.5 51.8
54.9 10h45min 46.6 8h14min 55.7 7h14min 56.1 6h16min
Qwen3-1.7B-Base / Qwen3-8B-Base + FastOPD (Fixed) 1024 + FastOPD (Fixed) 1024 + FastOPD (Fixed) 1024 + Adaptive FastOPD 1024
4.5 1.4 5.0 5.4
27.3 13.2 27.1 30.0
48.4 24.7 47.5 47.6
11.5 7.5 11.9 12.3
21.4 10.0 21.1 21.3
19.5 9.7 19.4 20.1
5 10 20 Adaptive
4.1 1.4 3.5 4.0
8h07min 2h19min 4h58min 2h37min
Table 2: Ablation comparing Adaptive FastOPD with fixed-step rollout expansion. Length Inc. is the number of tokens added per expansion, and Step Interval is the number of training steps between expansions. Fixed-schedule FastOPD is sensitive to the chosen interval, with performance variation across settings. The horizon-hit rate measures the fraction of rollouts that reach the current generation limit, whereas the boundaryreach rate measures how often they enter the boundary. (n−1) (n) + (1 − We smooth both rates using η̄a = β η̄a (n ) (n) β)ηa , a ∈ {hit, reach}, where we initialize η̄a 0 = (n0 ) ηa for the first observation at a newly entered horizon, and define the utilization condition: h i (n) (n) Usen = 1 η̄hit ≥ τhit ∨ η̄reach ≥ τreach . (19) The disjunction allows the utilization condition to be satisfied either when a sufficient fraction of rollouts approaches the generation limit or when broader coverage of the boundary is observed. When few rollouts reach the boundary, the progress estimate is based on a small and potentially unrepresentative subset of the batch. Expanding the horizon in this regime would also primarily extend a few long-running responses, which can dominate batch completion time (Shao et al. 2026; Khan et al. 2026). Requiring sufficient boundary coverage therefore improves the reliability of the progress estimate and avoids length expansion that provides little additional supervision for most rollouts.
Experiments Experimental Setup Training data and implementation. We train all methods on DAPO-Math-17K (Yu et al. 2026) using the verl framework (Li et al. 2026; Sheng et al. 2025), with vLLM for rollout generation (Kwon et al. 2023). We evaluate two teacher–student configurations: JustRL-DeepSeek-1.5B (He et al. 2025) with DeepSeek-R1-Distill-Qwen-1.5B (Guo et al. 2025), and Qwen3-8B-Base with Qwen3-1.7B-Base (Yang et al. 2025). All experiments run on a single node with four NVIDIA H200 140GB GPUs, with the actor trained in FP32. Each generation batch contains 64 inputs and four responses per input. We use a learning rate of 1×10−6 , student-selected top-16 candidates with student-probability weighting, and no additional KL regularization. All runs use one epoch without data shuffling, corresponding to 279 optimization steps. Both adaptive and fixed schedules start from a rollout horizon of 1,024 tokens, use ∆H = 1,024, and allow a maximum horizon of 15,360 tokens. The main Adaptive FastOPD configuration uses Nbase = 3, β = 0.8, Kpat = 5, τhit = 0.1, and τreach = 0.3, while fixed-schedule FastOPD expands the horizon every 20 optimization steps.
Nbase
β
Kpat
τhit
τreach
AMC23
MATH-500
Minerva
Olympiad
Avg
Time
DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5B 2 0.8 7 0.1 0.3 34.7 43.3 3 0.6 7 0.1 0.3 34.7 46.0 3 0.8 5 0.1 0.3 34.5 45.2 3 0.8 7 0.1 0.3 33.9 46.8 3 0.8 10 0.1 0.3 33.3 45.2 33.1 43.7 3 0.8 7 0.0 0.0
AIME25
83.4 83.1 85.9 84.0 86.0 83.7
86.2 86.6 86.1 86.0 86.0 85.8
32.7 33.1 32.8 32.7 33.2 33.3
52.0 52.2 51.8 52.1 51.6 51.8
55.4 55.9 56.1 55.9 55.9 55.2
5h13min 5h54min 6h16min 5h31min 4h55min 5h00min
Qwen3-1.7B-Base / Qwen3-8B-Base 2 0.8 7 0.1 0.3 3 0.6 7 0.1 0.3 3 0.8 5 0.1 0.3 3 0.8 7 0.1 0.3 3 0.8 10 0.1 0.3 3 0.8 7 0.0 0.0
28.4 28.3 30.0 28.1 28.1 28.5
48.6 48.8 47.6 48.3 48.5 48.0
12.2 11.7 12.3 12.1 12.8 12.8
21.7 21.7 21.3 21.8 22.6 22.1
20.3 20.1 20.1 20.0 19.9 19.9
2h40min 2h47min 2h37min 2h49min 2h41min 6h06min
4.6 4.0 4.0 3.8 3.5 3.1
AIME24
6.3 6.0 5.4 5.8 3.8 5.2
Table 3: Hyperparameter ablation of Adaptive FastOPD (Avg@16). Nbase is the number of valid boundary observations at each horizon, β is the EMA coefficient, Kpat is the plateau patience, and τhit and τreach are the utilization thresholds. Adaptive FastOPD shows limited variation in average performance across the tested settings for both model pairs. Baselines and evaluation. We compare Adaptive FastOPD with the original student model (Base), OPD using fixed rollout of 7,168 and 15,360 tokens (OPD 7K/15K), and fixed-schedule FastOPD (FastOPD (Fixed)). The OPD baselines characterize the performance and computational cost of different rollout budgets, while FastOPD (Fixed) isolates the effect of replacing a predefined expansion schedule with our progress-aware strategy. We evaluate on AIME 2025 (Balunovic et al. 2026), AIME 2024 and AMC 2023 (Yang et al. 2024), MATH-500 (Hendrycks et al. 2021), Minerva Math (Lewkowycz et al. 2022), and OlympiadBench (He et al. 2024). For each problem, we sample 16 responses with temperature 0.7, top-p = 0.95, and a maximum generation length of 31,744 tokens. We report the mean exact-match correctness over the 16 samples for each benchmark and the macro average across all six benchmarks (Avg@16).
Main Results Table 1 shows that adaptive expansion improves the accuracy–time trade-off on two teacher–student configurations. On the DeepSeek pair, Adaptive FastOPD reaches the best average score, 56.1, in 6h16min. It exceeds OPD 7K and OPD 15K while using 16.8% and 49.1% less time. It also improves over the fixed-schedule FastOPD baseline by 0.4 average points with 13.4% less time. The per-benchmark results show that Adaptive FastOPD remains competitive across all six benchmarks, indicating that its average advantage is not driven by a single outlier task. The efficiency gain is larger for the Qwen3-1.7B-Base student. Adaptive FastOPD achieves the best average, 20.1, in 2h37min, compared with 19.4 in 4h58min for fixed-schedule FastOPD and 19.1 in 9h05min for OPD 15K. This corresponds to 47.3% and 71.2% less training time. OPD 7K remains strong on several individual benchmarks, but the adaptive schedule obtains a higher macro average while using 44.5% less time. These results suggest that neither a uniformly short horizon nor uniform expansion toward the maximum is consistently optimal.
Ablation Study Sensitivity of Fixed Schedules Table 2 compares different expansion intervals for fixed-schedule FastOPD. For the DeepSeek pair, expanding every 20 steps is strongest among fixed schedules, while the 10-step schedule collapses from 55.7 to 46.6 average. For Qwen3, the 5- and 20-step schedules are similar in accuracy, whereas the 10-step schedule ends early and falls to 9.7. Across the tested fixed schedules, the mean score is 52.4 for DeepSeek and 16.2 for Qwen3. The corresponding differences between the best and worst schedules are 9.1 and 9.8 percentage points, respectively. A fixed schedule can expand before the current prefix is aligned, or spend many updates at a horizon that has already saturated. Adaptive FastOPD avoids choosing a global interval and it is both faster and more accurate than the strongest fixed schedule in the DeepSeek setting. In the Qwen3 setting, the fastest fixed run is 18 minutes shorter but loses 10.4 average points; the adaptive run recovers the strongest accuracy while remaining faster than the competitive fixed alternatives. Robustness and the Value of Length Utilization Table 3 reports model performance under different hyperparameter settings. Despite varying Nbase , β, Kpat , τhit , and τreach , the DeepSeek and Qwen3 runs remain within narrow ranges of 55.2–56.1 and 19.9–20.3, respectively. These ranges are much smaller than those of the fixed schedules in Table 2. Nbase , β, and Kpat affect how quickly evidence accumulates, but the expansion decision depends on the observed training dynamics rather than a fixed absolute threshold on raw teacher–student agreement metric. The last row of each group disables both length thresholds. On DeepSeek, removing the gate lowers performance from 56.1 to 55.2 while reducing training time from 6h16min to 5h00min. On Qwen3, removing the gate both lowers performance and increases time from 2h37min to 6h06min. Inspection of the schedules shows that ungated expansions can expose expensive horizons that are reached by only a small tail of the rollout distribution. The length-utilization gate therefore improves
(a) Adaptive horizon schedule.
(b) Normalized badness score over training.
(c) Dominant badness score by horizon.
Figure 2: Horizon-adaptation diagnostics for DeepSeek-R1-Distill-Qwen-1.5B with (Nbase , β, Kpat , τhit , τreach ) = (3, 0.8, 5, 0.1, 0.3). The resulting adaptive schedule spends different numbers of updates at different horizons, and the identity of the maximum normalized badness score changes across stages. Signals POGU POG OG O
AIME25
AIME24
AMC23
MATH-500
Minerva
Olympiad
Avg
Time
34.5 33.1 35.0 34.4
45.2 45.0 46.0 43.5
85.9 84.5 84.6 80.8
86.1 86.0 85.4 83.9
32.8 33.2 32.9 32.6
51.8 51.7 51.4 49.7
56.1 55.6 55.9 54.2
6h16min 5h27min 3h31min 2h29min
Table 4: Ablation of progress signals in Adaptive FastOPD on DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5B (Avg@16). Using multiple signals yields a more conservative criterion of training progress than relying on overlap alone, leading to better final performance. The gray row denotes the full POGU configuration used in the main experiments. reliability and helps prevent a small number of long rollouts from increasing training time.
smaller signal set, while the full signal set is preferable when final performance is the primary objective.
Why Multiple Signals Using the DeepSeek model as an example, Figure 2 visualizes the actual rollout-horizon schedule during the main experiment, the evolution of the normalized badness score, and the signal that dominates the expansion decision at each horizon. The resulting adaptive schedule is visibly non-uniform: Adaptive FastOPD passes quickly through some early horizons but remains for many updates at 5,120 tokens. Across the run, the total dominance duration decreases approximately in the order OGPU and no single signal dominates every stage. Table 4 evaluates a sequence of nested signal sets constructed by progressively removing the signal with the smallest share of dominant expansion decisions. Full POGU achieves the highest average score of 56.1. Removing U, or removing both P and U, reduces training time but also lowers final performance. Using only the most frequently dominant overlap signal produces the fastest run but decreases the average score to 54.2. These results support the multi-signal aggregation in Equation (14). By taking the maximum normalized badness, the expansion decision is based on the signal that currently indicates the greatest remaining teacher–student discrepancy, while the dominant signal may change across horizons and training stages. Combining complementary signals provides a more comprehensive estimate of training progress and yields better final performance than using overlap alone. The ablation also reveals a clear speed–performance trade-off. The OG variant reaches an average score of 55.9 in 3h31min, close to the 55.7 achieved by fixed-schedule FastOPD in 7h14min. The O-only variant obtains a competitive score of 54.2 using about 20% of the training time required by OPD 15K. Thus, the adaptive strategy can favor greater efficiency by using a
Conclusion We introduced Adaptive FastOPD, a progress-aware rollouthorizon expansion strategy for efficient on-policy distillation. Rather than triggering horizon expansion at a fixed step interval or from an absolute threshold on the raw teacher–student signals, Adaptive FastOPD evaluates learning progress relative to the initial state of each horizon. It combines four signals to monitor optimization near the current boundary region and expands the horizon only after the aggregated progress has plateaued and the available length is sufficiently utilized. The utilization condition also avoids extending the horizon when only a small number of long responses would increase rollout time. Experiments across two teacher–student pairs show that Adaptive FastOPD achieves the highest average performance with lower training time than fixed-schedule FastOPD and OPD 15K. The method is also less sensitive to the tested adaptation hyperparameters, supporting progress-aware adaptation as a more robust method. Future work may investigate additional progress signals and reduce the remaining dependence on other adaptation parameters.
References Agarwal, R.; Vieillard, N.; Zhou, Y.; Stanczyk, P.; Ramos Garea, S.; Geist, M.; and Bachem, O. 2024. Onpolicy distillation of language models: Learning from selfgenerated mistakes. In International Conference on Learning Representations, volume 2024, 21246–21263. Balunovic, M.; Dekoninck, J.; Petrov, I.; Jovanović, N.; and Vechev, M. 2026. Matharena: Evaluating llms on uncontam-
inated math competitions. Advances in Neural Information Processing Systems, 38. Gu, Y.; Dong, L.; Wei, F.; and Huang, M. 2024. Minillm: Knowledge distillation of large language models. In The twelfth international conference on learning representations. Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al. 2025. DeepSeekR1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081): 633–638. He, B.; Qu, Z.; Liu, Z.; Chen, Y.; Zuo, Y.; Qian, C.; Zhang, K.; Chen, W.; Xiao, C.; Cui, G.; et al. 2025. Justrl: Scaling a 1.5 b llm with a simple rl recipe. arXiv preprint arXiv:2512.16649. He, C.; Luo, R.; Bai, Y.; Hu, S.; Thai, Z.; Shen, J.; Hu, J.; Han, X.; Huang, Y.; Zhang, Y.; et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiadlevel bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3828–3850. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Vanschoren, J.; and Yeung, S., eds., Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Khan, A. A.; Ahmed, A.; Fayyaz, Z.; Di, S.; Hong, M.; and Anwar, A. 2026. Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing. arXiv preprint arXiv:2606.02218. Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C. H.; Gonzalez, J.; Zhang, H.; and Stoica, I. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, 611–626. Lewkowycz, A.; Andreassen, A.; Dohan, D.; Dyer, E.; Michalewski, H.; Ramasesh, V.; Slone, A.; Anil, C.; Schlag, I.; Gutman-Solo, T.; et al. 2022. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35: 3843–3857. Li, Y.; Zuo, Y.; He, B.; Zhang, J.; Xiao, C.; Qian, C.; Yu, T.; Huan-ang Gao; Yang, W.; Liu, Z.; and Ding, N. 2026. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. In ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning. Liang, K.; Tang, C.; Bai, C.; Liu, W.; Yang, S.; and Wu, Y. 2026. ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation. arXiv preprint arXiv:2605.28396. Luo, F.; Chuang, Y.-N.; Wang, G.; Xu, Z.; Han, X.; Zhang, T.; and Braverman, V. 2026. Demystifying opd: Length inflation and stabilization strategies for large language models. arXiv preprint arXiv:2604.08527. Shao, Z.; Srivatsa, V.; Srivastava, S.; Wu, Q.; Ariyak, A.; Wu, X.; Patel, A.; Wang, J.; Liang, P.; Dao, T.; Zhang, C.; Zhang, Y.; Athiwaratkun, B.; Xu, C.; and Wang, J. 2026. Beat the long tail: Distribution-Aware Speculative Decoding for RL Training. In Proceedings of the 9th Annual Conference on Machine Learning and Systems (MLSys).
Sheng, G.; Zhang, C.; Ye, Z.; Wu, X.; Zhang, W.; Zhang, R.; Peng, Y.; Lin, H.; and Wu, C. 2025. HybridFlow: A Flexible and Efficient RLHF Framework. Wu, Y.; Han, S.; and Cai, H. 2026. Lightning opd: Efficient post-training for large reasoning models with offline on-policy distillation. arXiv preprint arXiv:2604.13010. Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Yang, A.; Zhang, B.; Hui, B.; Gao, B.; Yu, B.; Li, C.; Liu, D.; Tu, J.; Zhou, J.; Lin, J.; et al. 2024. Qwen2. 5-math technical report: Toward mathematical expert model via selfimprovement. arXiv preprint arXiv:2409.12122. Yang, Z.; Guo, Z.; Song, Y.; Xu, M.; Wang, Y.; Wang, Y.; Liang, X.; and Tang, J. 2026. Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning. arXiv preprint arXiv:2605.07804. Yu, G.-I.; Jeong, J. S.; Kim, G.-W.; Kim, S.; and Chun, B.-G. 2022. Orca: A distributed serving system for {TransformerBased} generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22), 521–538. Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Dai, W.; Fan, T.; Liu, G.; Liu, L.; et al. 2026. Dapo: An opensource llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38: 113222– 113244. Zhang, D.; Yang, Z.; Janghorbani, S.; Han, J.; Ressler II, A.; Qian, Q.; Lyng, G. D.; Batra, S. S.; and Tillman, R. E. 2026a. Fast and effective on-policy distillation from reasoning prefixes. In Findings of the Association for Computational Linguistics: ACL 2026, 25553–25569. Zhang, Q.; Yuan, Q.; Lin, H.; Lu, Y.; Han, X.; Sun, L.; Li, X.; Xu, M.; Li, J.; and Zhao, X. 2026b. ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation. arXiv preprint arXiv:2607.13124. Zhang, Y.; Chai, J.; Fu, Y.; Tu, S.; Wang, X.; Lin, W.; Yin, G.; Zhang, Q.; Zhu, Y.; and Zhao, D. 2026c. Are Full Rollouts Necessary for On-Policy Distillation? arXiv preprint arXiv:2605.31490. Zhao, Q.; Song, H.; Tian, S.; Shao, J.; and Li, X. 2026. PrefixGuided On-Policy Distillation: Mining Golden Trajectories from Rollouts. arXiv preprint arXiv:2606.21994. Ziheng, Z.; Li, J.; Tang, H.; Wu, Y. N.; and Terzopoulos, D. 2026. Less is more: Early stopping rollout for on-policy distillation. arXiv preprint arXiv:2605.27028.