Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Feng Luo 1 Yu-Neng Chuang 1 Guanchu Wang 2 Zicheng Xu 3 Xiaotian Han 4 Tianyi Zhang 1 Vladimir Braverman 3
1.0 Rollout truncation Rollout repetition Accuracy (MATH500)
0.6 0.4 0.2 0.0
0.725 0.700 0.675 0.650 0.625 0.600 0.575 0.550
Accuracy
0.8
On-policy distillation (OPD) trains student models under their own induced distribution while leveraging supervision from stronger teachers. We identify a failure mode of OPD: as training progresses, on-policy rollouts can undergo abrupt length inflation, causing truncated trajectories to dominate the training data. This truncation collapse coincides with abrupt repetition saturation and induces biased gradient signals, leading to severe training instability and sharp degradation in validation performance. We attribute this problem to the interaction between student-induced data collection and the distillation objective, which implicitly favors long and repetitive rollouts. To address this issue, we propose Stable-OPD, a stabilized OPD framework that combines a referencebased divergence constraint with rollout mixture distillation. These together mitigate repetitioninduced length inflation and further stabilize OPD training. Across multiple math reasoning datasets, our approach prevents truncation collapse, stabilizes training dynamics, and improves performance by 7.2% on average.
Ratio
arXiv:2604.08527v1 [cs.CL] 9 Apr 2026
Abstract
0
200
400
Step
600
800
Figure 1. Abrupt length inflation within OPD.
avoids the distribution mismatch inherent to purely offline distillation and enables continual adaptation during training (Agarwal et al., 2024; Lu & Lab, 2025; Yang et al., 2025; Ye et al., 2025). This paradigm has shown promise in domains such as long-form generation and reasoning, where robustness under the student’s evolving policy is critical. Despite the strong performance of OPD, we identify a training pathology that emerges during optimization. As depicted in Fig. 1, abrupt rollout length inflation occurs as training progresses: student-generated rollouts suddenly grow much longer, causing truncated trajectories terminated by a fixed context or token limit to dominate the training data. We show that this behavior is not a generic form of length bias commonly observed in GRPO-style RL training. Instead, it is driven by a primary failure mode specific to OPD, which we term “abrupt repetition saturation”. In this regime, student rollouts unexpectedly enter repetitive generation patterns, where repeated tokens rapidly dominate the generated sequence. Once repetition saturation occurs, subsequent on-policy updates reinforce this behavior, leading to rollout length inflation and eventual training collapse.
1. Introduction On-policy distillation (OPD) (Lai et al., 2020; Czarnecki et al., 2019) has recently gained attention as an effective framework for training student LLMs under their own induced distribution while leveraging supervision from stronger teacher LLMs. By iteratively collecting rollouts from the student policy and applying distillation losses, OPD 1
Department of Computer Science, Rice University, Houston, USA 2 Department of Computer Science, University of North Carolina at Charlotte, Charlotte, USA 3 Department of Computer Science, Johns Hopkins University, Baltimore, USA 4 Department of Computer and Data Sciences, Case Western Reserve University. Correspondence to: Feng Luo <[email protected]>, Yu-Neng Chuang <[email protected]>.
The underlying mechanism lies in OPD’s optimization under the student-induced distribution, guided by reverse-KL advantage signals derived from teacher-student likelihood discrepancy. During abrupt repetition saturation, repetitive tokens receive systematically larger reverse-KL advantages than regular tokens. While these tokens contribute
Preprint. April 10, 2026.
1
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
little when they are rare, their frequency rises sharply once repetition begins, and their disproportionately large advantages then make them increasingly dominant in the gradient update. This creates a self-reinforcing feedback loop: updates increasingly favor repetitive continuations, which further encourages repetition and length expansion. In this sense, student repetition may exploit or hack the teacher’s likelihood-based signal.
rewards (RLVR) for tasks such as math and code (Shao et al., 2024; Guo et al., 2025; Zhang et al., 2025). Given a prompt x, GRPO samples a group of G responses {oi }G i=1 from policy πθold and assigns each response a sequence-level reward ri = R(x, oi ) , which is often binary for verifiable tasks (e.g., correctness). This method then assigns a group-normalized advantage Ai to all tokens k = 1, ...|oi | within response oi . Then, the objective is inherited from the clipped objective proposed by PPO(Schulman et al., 2017),
Empirically, shown in Sec. 3.4, we observe that the onset of repetition saturation and the rise in rollout truncation tightly coincide with severe training instability, including sudden drops in validation accuracy and biased learning gradients. Importantly, this collapse occurs while the teacher model and loss formulation remain fixed, indicating that the instability is intrinsic to OPD’s on-policy dynamics under abrupt truncation and repetition. Together, these observations suggest that truncation-dominated OPD training yields biased gradient signals that destabilize optimization.
JGRPO (θ) = Ex∼P (X),{oi }G i=1 ∼πθ G X
old
|oi |
1 1 X min ρti (θ)Ai , clip(ρti (θ), 1 − ϵ, 1 + ϵ)Ai . G i=1 |oi | t=1 (1) where P (X) refers to the question distribution, ρti (θ) = <t t πθ (oti |x, o<t i )/πθold (oi |x, oi ), πθ is the current policy, and ϵ controls the trust region of policy updates. Despite strong empirical performance, GRPO has two notable limitations. First, the reward signal is sparse and provided only at the sequence level, offering limited token-level guidance on where the model makes mistakes. Second, when sampled responses within a group are all correct or all incorrect, their advantages become zero, yielding no effective update despite the computational cost of group sampling. In this work, we leverage the clipping objective of GRPO with token-level advantages for OPD training.
To address this training challenge, we introduce a unified stabilization framework, Stable-OPD, which stabilizes OPD through two complementary mechanisms. First, we incorporate a reference-based divergence constraint that limits uncontrolled policy drift and curbs excessive rollout expansion. Second, we employ rollout mixture distillation, which blends on-policy student rollouts with reference trajectories to maintain a stable fraction of complete, non-truncated sequences throughout training. Across multiple mathematical reasoning benchmarks, we show that Stable-OPD consistently stabilizes OPD training and improves average accuracy by 7.2% compared to standard OPD baselines. Our contributions are summarized as follows:
2.2. Knowledge distillation for LLM Knowledge distillation(Hinton et al., 2015; Rusu et al., 2015; Kim & Rush, 2016; Gou et al., 2021) trains a student model to learn from a more capable teacher by matching the teacher’s output distribution. Standard knowledge distillation typically trains the student on a fixed set of sequences, such as teacher-generated responses or groundtruth demonstrations. Let the student have learnable parameters θ, with πSθ differentiable with respect to θ. Given a fixed dataset of input-output sequence pairs (X, Y ) and a divergence D, standard distillation minimizes the expected discrepancy between teacher and student next-token distributions πT andh πSθ along the fixedisequences: LSD (θ) = E(x,y)∼P (X,Y ) D πT ∥ πSθ (y | x) . While effective, this off-policy training paradigm introduces a training-inference mismatch: during inference, the student conditions on its own generated prefixes, which may deviate from the prefixes observed in the fixed distillation dataset. OPD(Gu et al., 2023; Agarwal et al., 2024; Lu & Lab, 2025) addresses this issue by training on student-generated responses ŷ ∼ πSθ , thereby aligning the training states with the student’s testtime states. The loss for OPD is:
• Length Inflation: We identify rollout length inflation as an observable training pathology in OPD with a primary failure mode of abrupt repetition saturation. • Rollout Pathology: We empirically show that repetitionsaturated, truncation-dominated rollouts produce biased gradients that destabilize training. • Stabilization Protocol: We identify the specific constraints required to stop this pathology. We show that a dual strategy of divergence constraint and rollout mixture is effective to prevent the student from hacking the distillation objective. • Evaluation: Across six datasets and three LLMs, StableOPD consistently improves accuracy and reduces repetition saturation.
2. Preliminary 2.1. Group Relative Policy Optimization (GRPO) Group Relative Policy Optimization (GRPO) is a reinforcement learning method commonly used in RL with verifiable
h i LOPD (θ) = Ex∼P (X),ŷ∼πSθ D πSθ ∥ πT (ŷ | x) . (2) 2
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
3. Length Inflation in OPD
is sufficiently long and its tail is highly compressible: rep(oi ) = 1 |otail i | > L ∧ CompRatio(oi ) > τ , where we set L = 10,000 and τ = 10 in our experiments. The repetition rate over R P is then defined as the average value N of RepRate(R) = N1 i=1 rep(oi ).
In this section, we formulate OPD for reasoning tasks, define metrics for probing training dynamics, present the empirical failure mode of abrupt truncation-repetition inflation, and analyze it from rollout-level, token-level, and mechanistic perspectives.
In our implementation, RepRate measures the fraction of long rollouts whose tails exhibit extreme compressibility, which correlates well with visibly repetitive, lowinformation continuations.
3.1. OPD Training for LLM Reasoning Reverse KL reward and token-level advantage. Instead of sequence-level rewards ri , we use a teacher model πT to define a token-level reward on each visited state. Following prior work (Lu & Lab, 2025), we define for each student token yi,t the reverse-KL-based reward:
3.3. Empirical Failure Mode: Abrupt Truncation-Repetition Inflation We investigate OPD dynamics on a 13k subset of the OpenR1-Math-220k reasoning data and consider three student-teacher groups that vary both student scale and teacher choice: (i) Qwen2.5-Math-1.5B student with DeepSeek-R1-Distill-7B teacher, (ii) Qwen2.5-Math-7B student with OpenThinker3-7B teacher, and (iii) Qwen2.5Math-7B student with DeepSeek-R1-Distill-7B teacher. For each configuration, we run OPD with the reward setting in Sec. 3.1 and track the truncation and repetition metrics on both training rollouts and MATH500 (Lightman et al., 2023) validation set, together with validation accuracy. The resulting dynamics are shown in Fig. 2.
KL ri,t = log πT (ŷi,t | si,t , ŷi,<t ) − log πθ (ŷi,t | si,t , ŷi,<t ),
which encourages the student to increase the probability of tokens to which the teacher assigns high likelihood. We KL then take the token-level advantage to be Ai,t ≜ ri,t . This token-level advantage contrasts with GRPO, where a single sequence-level advantage Ai is broadcast to all tokens in a response oi . Objective. The overall optimization objective keeps the GRPO-style clipped form of Eq. (1), but replaces the sequence-level advantage Ai with the token-level advantages Ai,t . This GRPO-style OPD objective provides dense token-level guidance from the teacher and serves as our default training setup in this work.
Stable early training stage. Across all three settings, OPD initially behaves as desired. Validation accuracy gradually improves, most student responses finish within the generation budget, and visibly repetitive tails are rare. The rollout truncation rate TruncRate stays around 0.5 for Qwen2.5Math-1.5B and around 0.23 for Qwen2.5-Math-7B, while the validation truncation rate remains near 0.2 and 0.1, respectively. The repetition rate RepRate stays close to zero on both training and validation generations.
3.2. Metrics for Analyzing OPD Dynamics To study the dynamics of OPD training, we monitor two simple metrics computed over a set of model rollouts: truncation rate and repetition rate. Both metrics can be evaluated on on-policy training rollouts and on held-out validation prompts; for clarity we define them on an arbitrary set of student-generated responses R = {oi }N i=1 .
Phase transition and robustness. As training progresses, all three settings exhibit a sharp phase transition. Within a relatively short window of OPD steps (about 30 steps in Fig. 2), the truncation rate on on-policy rollouts rises abruptly toward one, indicating that most generations now hit the maximum length budget without emitting an EOS token. At the same time, the repetition rate RepRate spikes from near zero to about 0.3-0.6, revealing the emergence of long, highly compressible suffixes dominated by repetitive patterns.
Truncation rate. Each rollout oi is generated under a fixed maximum generation length. We say oi is truncated if generation terminates because this length budget is exhausted, rather than because the model emits an EOS token. Let trunc(oi ) ∈ {0, 1} indicate whether oi is truncated. The truncation PN rate over R is the average value of TruncRate = N1 i=1 trunc(oi )(R).
The same qualitative transition appears on the MATH500 validation set: both validation TruncRate and RepRate jump sharply at nearly the same training step, coinciding with a sudden drop in validation accuracy. Observing this truncation-repetition inflation and worse validation performance across all three student-teacher groups suggests that it is a robust OPD failure mode in reasoning, rather than an artifact of a particular model pair or dataset split. We
Repetition rate. To capture degenerate generations with strong local repetition, we use a compression-based repetition metric. For a rollout oi , let bytes(oi ) denote its byte representation, and let c(·) denote zlib compression with a fixed level. We compute the compression ra tail tio as CompRatio(oi ) = bytes(otail i ) / c bytes(oi ) , where otail is the suffix consisting of the last L chari acters of oi . We say oi exhibits high repetition if it 3
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Rollout: repetition vs acc 0.6
0.70
200
400
Step
600
800
Eval: repetition vs acc
0.65
0.4
0.60 0.55
Accuracy Repetition ratio
0.70
Accuracy Repetition ratio
0
Eval: truncation vs acc
1.0
Accuracy Truncation ratio
Rollout: truncation vs acc
0.8
0.70
0.6
0.70
0.65
0.6
0.65
0.4
0.65
0.2
0.60
0.4
0.60
0.2
0.60
0.0
0.55
0.2
0.55
0.0
Rollout truncation
0
200
400
Step
600
800
Rollout repetition
0
200
Eval truncation
400
Step
600
800
Eval repetition
Accuracy
Truncation ratio
1.0 0.8 0.6 0.4 0.2
0
200
400
Step
600
800
0.55
Accuracy (MATH500)
(a) Student: Qwen2.5-Math-1.5B; Teacher: DeepSeek-R1-Distill-7B
0.6
0.4 0
200
400
Step
600
800
0.5
Rollout truncation
Eval: truncation vs acc
0.6 0
200
400
Step
600
800
Rollout repetition
0.5
Eval: repetition vs acc
0.6
Accuracy Repetition ratio
0.6
1.0 0.8 0.6 0.4 0.2
0.7
0.7
0
200
Eval truncation
400
Step
600
800
0.7
Accuracy
Rollout: repetition vs acc
Accuracy Repetition ratio
0.7
0.8
0.2
0.4 0.3 0.2 0.1 0.0
Accuracy Truncation ratio
Rollout: truncation vs acc
Truncation ratio
1.0
0.4
0.6
0.2
0.5
0.0
Eval repetition
0.6 0
200
400
Step
600
800
0.5
Accuracy (MATH500)
(b) Student: Qwen2.5-Math-1.5B; Teacher: OpenThinker3-7B
0.25 0
200
400
Step
600
800
Rollout truncation
Rollout: repetition vs acc 0.85
0.6 0.4 0.2 0.0
Eval: truncation vs acc
1.0 0.8 0.6 0.4 0.2
0
200
400
Step
600
Rollout repetition
800
0.80 0.75 0.70 0.65
0.85 0.80 0.75 0.70 0.65
0.8 0.6 0.4 0.2 0.0
Eval: repetition vs acc
0
Eval truncation
200
400
Step
600
800
Eval repetition
0.85 0.80 0.75 0.70 0.65
Accuracy
0.50
0.80 0.75 0.70 0.65
Accuracy Repetition ratio
0.75
Accuracy Truncation ratio
0.8
Accuracy Repetition ratio
Truncation ratio
Rollout: truncation vs acc 0.85 1.00
0
200
400
Step
600
800
Accuracy (MATH500)
(c) Student: Qwen2.5-Math-7B; Teacher: DeepSeek-R1-Distill-7B Figure 2. Training dynamics of OPD on three groups. Training starts in a stable regime with low truncation and repetition, followed by a sharp phase transition where truncation and repetition increase and remain high while validation accuracy collapses, illustrating a robust truncation-repetition inflation failure mode of OPD.
refer to this phenomenon as abrupt truncation-repetition inflation.
remains larger. This provides direct empirical evidence that the reverse-KL signal is not uniformly distributed across the trajectory and systematically favors locally repetitive continuations.
3.4. Understanding Abrupt Repetition Inflation
Taken together, the rollout-level and token-level evidence suggests that OPD increasingly favors repetitive regions of the trajectory, especially once such tokens become more prevalent during training. We next formalize how favorable local reward and on-policy sampling jointly amplify this effect.
The previous subsection established the failure mode empirically. We now examine the training signals associated with its onset through rollout-level and token-level analyses. Rollout-level Evidence. Fig. 3 tracks the average student log-probability, teacher log-probability, reverse-KL advantage, and response length over training. In the early phase, response lengths remain moderate and all three statistics evolve smoothly. Around the onset of inflation, however, they shift together: response length jumps toward the generation budget, both log-probabilities become much less negative, and the teacher log-probability increases more than the student’s, causing the average advantage to rise sharply. This pattern is consistent across all three studentteacher groups.
3.5. Mechanistic Explanation of Abrupt Repetition Inflation To formalize the intuition above, we write the effective OPD update in a state-action form. For the ease of exposition, we ignore the clipping term and suppress the distinction between the rollout policy and the updated policy. Let dπθ (s) denote the state-visitation distribution induced by the current student policy over prefixes s, and let A(s, y) = log πT (y | s) − log πθ (y | s) denote the token-level reverseKL advantage in Eq. (3). Denoting the resulting policy gradient by g(θ), we have:
Token-level Evidence. To isolate the local reward signal, we compare the average reverse-KL advantage of regular tokens and repetitive tokens during training. As shown in Fig. 4, repetitive tokens receive larger advantages than regular tokens throughout training. Before the inflation phase, however, repetitive tokens are extremely rare and therefore contribute little to the aggregate update. Once inflation begins, their frequency rises sharply while their advantage
g(θ) ∝ Es∼dπθ , y∼πθ (·|s) [A(s, y)∇θ log πθ (y | s)] . (3) Thus, the OPD update is governed by two coupled quantities: how often a state is visited, and how strongly actions at that state are favored by the reverse-KL signal. Let R denote the 4
8000
0.1 0.2 0.3 0.4 0.5 0.6 0.7
aggregate influence on the update can increase disproportionately.
Response Len.
Log prob / Advantage
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
6000 4000
4. Mitigating Repetition Saturation
2000 800 0
As shown in Sec. 3, standard OPD can reach a stage where training rollouts are dominated by long, repetitive, and trun(a) Student: Qwen2.5-Math-1.5B; Teacher: DeepSeek-R1-Distill-7B cated trajectories, leading to an abrupt truncation-repetition collapse and a sharp drop in validation accuracy. In this 8000 0.2 section, we introduce two strategies designed to explicitly 6000 0.4 mitigate this failure mode. 0
100
300
400
Step Teacher log_prob
500
600
700
Advantage
Response Len.
Response Len.
Log prob / Advantage
Student log_prob
200
4000
0.6
2000
0.8 0
100
Student log_prob
200
300
400
Step Teacher log_prob
500
600
700
Advantage
4.1. Mixture Distillation: Combining On- and Off-Policy Supervision
800 0
Response Len.
The training objective of OPD is driven by student-generated rollouts, and there is no explicit control over the distribution of states the student visits. Once the student starts to visit long, repetitive, and truncated trajectories, OPD updates are dominated by these degenerate states, which accelerates truncation-repetition saturation and accuracy collapse.
8000
0.1 0.2 0.3 0.4 0.5 0.6
Response Len.
Log prob / Advantage
(b) Student: Qwen2.5-Math-1.5B; Teacher: OpenThinker3-7B 6000 4000 2000 0
100
Student log_prob
200
300
400
Step Teacher log_prob
500
600
700
Advantage
800 0
Response Len.
(c) Student: Qwen2.5-Math-7B; Teacher: DeepSeek-R1-Distill-7B Figure 3. Rollout-level evidence of abrupt repetition inflation for three student-teacher groups. Around the step where rollout length abruptly inflates, both student and teacher log prob become much less negative, with the teacher’s increase being larger, which induces a sudden jump in the reveser KL advantage.
To address this, we introduce a hybrid training paradigm, mixture distillation, which combines OPD with off-policy supervision on high-quality “golden” data. Intuitively, the golden data serves as an anchor: it maintains a fraction of complete, non-truncated, and non-repetitive trajectories throughout training, and keeps the OPD objective tied to reasonable reasoning behavior by reducing the influence of degenerate on-policy rollouts.
In addition to the on-policy rollouts used to optimize the OPD objective, we maintain a fixed dataset Dgold of inputset of tokens in repetitive tails. We can decompose Eq. (3) output pairs (x, y) with complete and high-quality chain-ofinto contributions from states inside and outside R: thought solutions. At each training step, we sample a set g(θ) = Es∼dπθ [1{s ∈ / R} ∆(s)]+Es∼dπθ [1{s ∈ R} ∆(s)] , of prompts x and, for each x, include both the on-policy (4) rollout generated by the student and a golden solution (x, y) where ∆(s) ≜ Ey∼πθ (·|s) [A(s, y)∇θ log πθ (y | s)]. from Dgold in the same minibatch. Thus the model simultaneously sees its own trajectory and a high-quality target for This decomposition makes the OPD-specific feedback exthe same problem, and we optimize a combined loss over plicit. Once visitation to R increases, the second term octhis mixed supervision. cupies a larger share of the update, and that update further encourages continuations that remain in R. This creates a self-reinforcing loop: repetitive tails need not to be the most Lmix (θ) = LOPD (θ) + λgold E(x,y)∼Dgold LSFT (θ; x, y) , frequent tokens overall, but once they become sufficiently (5) common, their combination of frequency and disproporwhere LSFT is a standard supervised loss and λgold controls tionately large token-level advantages allows them to steer the weight of the golden data. subsequent OPD updates toward repetitive continuations. Recent self-distillation methods (Hübotter et al., 2026; Zhao The empirical evidence in Fig. 4 supports this pattern. Beet al., 2026) have also leveraged high-quality “golden” refore collapse, repetitive tokens are relatively rare, so their sponses, but for a different purpose. In those approaches, total contribution to the update remains limited even if their golden responses are typically used to refine the teacher average advantage is larger. However, around the transition signal itself. As noted by Kim et al. (2026), this can suptheir frequency rises sharply while their advantage remains press the teacher’s uncertainty during reasoning and hurt much larger; in Fig. 4, repetitive tokens account for roughly student performance on complex problems. By contrast, 30% of tokens after collapse, yet their average advantage is mixture distillation leaves the original teacher-derived OPD about 4–9× that of non-repetitive tokens. As a result, their signal unchanged on on-policy rollouts. Golden data is used 5
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Repeated vs Regular Token Avg Advantage
0.00
0.8 repeated token ratio
0.05 average advantage
1.0
repeated token regular token repeated token ratio
0.10
0.6
0.15
0.4
0.20
0.2 0.25 0
100
200
300
400 step
500
600
700
800
0.0
Figure 4. Reverse-KL advantage for regular and repetitive tokens during OPD training. Repetitive tokens receive larger advantages than regular tokens throughout training.
5. Experiment
only through an auxiliary off-policy SFT term that stabilizes training, and thus does not introduce this issue.
In this section, we conduct experiments to answer the following research questions about Stable-OPD: RQ1: How does Stable-OPD affect LLM reasoning performance on mathematical reasoning benchmarks? RQ2: How does StableOPD mitigate truncation-repetition saturation in OPD training? RQ3: What are the individual effects of mixture distillation and KL regularization?
From a distributional perspective, mixture distillation can be viewed as training on a mixture of two state distributions: the on-policy distribution induced by the current student, and a fixed off-policy distribution induced by Dgold . This prevents the OPD objective from being driven solely by truncation-dominated rollouts: gradients are rebalanced toward high-quality, non-truncated reasoning trajectories, and the mixed supervision in turn steers the student to generate better on-policy samples during training.
5.1. Experiment Setup Training. We build training data from OpenR1-Math220k1 , following the filtering procedure in (Yan et al., 2025). Prompts are sourced from NuminaMath 1.5 (LI et al., 2024) and paired with reasoning traces generated by DeepSeekR1 (Guo et al., 2025). Starting from the default 94k-prompt split, we filter out generations that exceed 8192 tokens or are marked incorrect by Math-Verify2 , and result in 46k prompts and high-quality demonstrations.
4.2. KL-Regularized Mixture Distillation While mixture distillation modifies the training distribution by injecting off-policy golden data, it does not directly control the magnitude of the student policy updates at each step. In the standard OPD setup, once the student drifts toward long, repetitive trajectories, the reverse-KL advantages start assigning large positive signals exactly to these states. To control this drift, we add a reference-based divergence constraint. In this work, we adapt KL regularization term on the policy itself.
Evaluation. We evaluate on six widely used mathematical reasoning benchmarks: AIME 2024, AIME 2025, AMC (LI et al., 2024), Minerva (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), and MATH500 (Hendrycks et al., 2021). For AIME 2024, AIME 2025, and AMC, whose test sets are small, we report avg@32. For Minerva, OlympiadBench, and MATH500, we report pass@1. The sampling temperature is 0.6 for testing.
We introduce a reference policy πref (e.g., the initial student checkpoint) and penalize deviations from this reference at visited prefix states. For a prefix state st at step t, we define KL(st ) = DKL πθ (· | st ) ∥ πref (· | st ) as the per-prefix KL between the student and reference policies. The KLregularized Mixture Distillation loss is then defined as LStable-OPD (θ) = Lmix (θ) + βKL Est KL(st ) ,
Implementation Details. For OPD training, we use a rollout batch size of 64 and sample 4 on-policy trajectories per prompt. When applying mixture distillation, each
(6)
1 2
where βKL > 0 controls the strength of the regularization. 6
https://huggingface.co/datasets/open-r1/OpenR1-Math-220k https://github.com/huggingface/Math-Verify
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
5.3. Mitigating truncation-repetition inflation (RQ2)
prompt in the batch is additionally paired with one off-policy golden solution from the dataset. Following recent OPD practice (Lu & Lab, 2025), we first perform supervised finetuning on 33k samples from the filtered dataset, and then run OPD on the remaining 13k samples. We generate rollouts with a sampling temperature of 1.0 and optimize the student with Adam using a learning rate of 1 × 10−6 . All experiments are conducted on 4×H200 GPUs.
To assess how Stable-OPD changes OPD training dynamics, we compare the same student-teacher settings and track rollout and evaluation truncation and repetition on MATH500 over training steps, as shown in Fig. 5 and Fig. 6. Across both student-teacher groups, OPD exhibits the truncationrepetition inflation: after an initially stable phase, rollout and evaluation truncation/repetition curves spike sharply and then remain at much higher levels than in the early stage, with only minor fluctuations. Under Stable-OPD, the training dynamics remain stable. For the Qwen2.5-Math1.5B + OpenThinker3-7B setting, all four curves remain flat throughout training: truncation ratios stay at moderate levels and repetition ratios are near zero on both rollouts and evaluation prompts. For the Qwen2.5-Math-1.5B + DeepSeekR1-Distill-7B setting, we do observe a mild upward drift in truncation and repetition near the end of training, but the increase is much smaller than with OPD and occurs much later than the sharp inflation observed under OPD. These results show that mixture distillation and KL regularization prevent OPD from abrupt truncation-repetition inflation regime.
Baseline Methods. We benchmark Stable-OPD against the following baselines using Qwen2.5-Math-1.5B and Qwen2.5-Math-7B (Yang et al., 2024). For SFT methods, the base model is fine-tuned on the full 46k dataset. For RL methods, we include GRPO (Shao et al., 2024), SimpleRLZero (Zeng et al., 2025), Oat-Zero (Liu et al., 2025), PRIMEZero (Cui et al., 2025), and OpenReasonerZero (Hu et al., 2025). We also compare with standard OPD (Lu & Lab, 2025). In addition, we adopt the same 33k/13k split as above: the model is first supervised fine-tuned on 33k examples and then trained with OPD on the remaining 13k examples. More details are listed in Appendix C. 5.2. Reasoning performance (RQ1)
5.4. Ablation on mixture distillation and KL Regularization (RQ3)
We compare Stable-OPD with supervised and RL baselines on six mathematical reasoning benchmarks, reporting accuracies in Table 1 and Table 3 for both Qwen2.5-Math1.5B and Qwen2.5-Math-7B backbones. Across both model scales, SFT and GRPO substantially improve over the base models, but standard OPD fails to match these gains despite leveraging on-policy samples and dense token-level supervision. For example, on the 7B backbone, OPD achieves 43.8% average accuracy, trailing SFT (44.1%) and GRPO (45.5%). A similar trend appears on the 1.5B backbone. The results suggest that training instability limits the effectiveness of OPD. Stable-OPD addresses this issue by stabilizing OPD with mixture distillation and KL regularization, yielding consistent improvements across scales. As shown in Table 3, it boosts average accuracy from 28.9% to 36.1% (+7.2) on the 1.5B backbone, achieving the best performance. On 7B backbone, Stable-OPD reaches 47.6% average accuracy, surpassing all other methods.
We ablate the two components of Stable-OPD, mixture distillation and KL regularization, to quantify their contributions. We use DeepSeek-R1-Distill-7B as the teacher models. As shown in Table 2, compared to the base model, OPD improves performance to 28.0%, but remains substantially below Stable-OPD due to training instability. Adding KL regularization alone yields a modest but consistent gain (28.0→29.7), indicating that constraining policy drift helps stabilize OPD updates. Combining KL regularization with mixture distillation brings a much larger improvement (29.7→35.7), making it the strongest variant in the 1.5B setting. This suggests that the two components are complementary: KL regularization limits abrupt policy shifts at the token-level, while mixture distillation provides a stable fraction of high-quality trajectories that anchors learning when on-policy rollouts start to degrade.
We further compare Stable-OPD with recent RLVR approaches, such as SimpleRL-Zero, OpenReasoner-Zero, PRIME-Zero and Oat-zero, on Qwen2.5-Math-7B backbone, as shown in Table 1. Stable-OPD achieves the best average accuracy (47.6%), outperforming the strong zerostyle methods. These results indicate that stabilizing onpolicy distillation can surpass carefully engineered RLVR pipelines, providing a simple yet effective alternative for improving mathematical reasoning.
6. Related Work Length Bias in LLM Reasoning. Recent work such as Dr.GRPO (Liu et al., 2025) and DAPO (Yu et al., 2025) has observed that standard GRPO-style objectives implicitly favor longer responses, and proposes objective reweighting or normalization schemes to remove sample-length bias. These methods focus on eliminating length-dependent gradient scaling at the sequence level, and have been shown to stabilize RL training under sparse rewards. In contrast, our work identifies a qualitatively distinct training-time failure mode that arises in OPD. We show that repetitive continua7
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models Table 1. Accuracy on six mathematical benchmarks for Qwen2.5-Math-7B. The best performance on each benchmark is highlighted in bold. For OPD-based methods we use OpenThinkerV3 as teacher models.
AVG
MATH-500
M INERVA
O LYMPIAD
AMC
AIME24
AIME25
Q WEN 2.5-M ATH -7B Q WEN 2.5-M ATH -7B-I NSTRUCT
19.1 37.6
43.6 80.4
7.4 32.7
15.6 41.0
31.3 48.5
11.5 12.5
4.9 10.2
S IMPLE RL-Z ERO O PEN R EASONER -Z ERO PRIME-Z ERO OAT-Z ERO
37.4 41.0 40.8 43.8
76.0 82.4 81.4 78.0
25.0 33.1 39.0 34.6
34.7 47.1 40.3 43.4
54.9 52.1 54.0 61.2
27.0 16.5 17.0 33.4
6.8 15.0 12.8 11.9
SFT GRPO OPD S TABLE -OPD
44.1 45.5 43.8 47.6
82.6 84.4 80.0 84.6
40.8 39.3 37.9 43.4
43.7 46.8 47.5 49.3
52.8 62.0 53.4 58.1
22.2 25.1 21.7 24.7
22.3 15.3 22.2 25.2
Rollout repetition
1.0 0.8 0.6 0.4 0.2
0
200
400
Step
600
800
Eval truncation
0
200
400
Step
600
800
OPD
Eval repetition
0.6
Repetition ratio
Repetition ratio
0.4 0.3 0.2 0.1 0.0
Truncation ratio
Rollout truncation
Truncation ratio
1.0 0.8 0.6 0.4 0.2
M ODEL
0.4 0.2
0
200
Stable-OPD
400
Step
600
800
0.0
0
200
400
Step
600
800
Figure 5. Training dynamics of OPD vs. Stable-OPD. Student: Qwen2.5-Math-1.5B; Teacher: OpenThinker3-7B. Table 2. Ablation studies of KL regularization and mixture distillation on average accuracy.
Method Qwen2.5-MATH-1.5B OPD OPD + KL OPD + KL + Mixture Distillation
On-policy Distillation. OPD trains the student on trajectories sampled from its current policy, while a teacher provides per-token guidance through KL-based regularization or closely related objectives (Agarwal et al., 2024; Lu & Lab, 2025; Xiao et al., 2026; Yang et al., 2025; Gu et al., 2023). By aligning the learning signal with the student’s own visitation distribution, these methods reduce the distribution shift that arises when supervision is collected off-policy. These works connect distillation to classic on-policy data aggregation in imitation learning, such as DAgger (Ross et al., 2011), where an expert supplies corrective supervision on states encountered by the learner under its own policy.
Avg. Acc (%) 16.0 28.0 29.7 35.7
tions receive larger token-level advantages, and on-policy sampling amplifies their contribution once they become frequent, leading to abrupt repetition inflation and length explosion. This mechanism differs fundamentally from previously studied forms of length bias in RLVR and is specific to OPD dynamics.
7. Conclusion We identify a failure mode of OPD characterized by abrupt rollout length inflation, truncation collapse, and repetition saturation. We show that this pathology arises from the interaction between student-induced data collection and the likelihood-based distillation objectives: repetitive tokens receive systematically larger advantages, and once sufficiently frequent, their disproportionate reward signal dominates gradient updates, creating a self-reinforcing feedback loop that implicitly favors increasingly long and repetitive rollouts. To address this issue, we propose Stable-OPD, a stabilized OPD framework that combines a reference-based divergence constraint with rollout mixture distillation. Across multiple mathematical reasoning benchmarks, Stable-OPD consistently stabilizes OPD training and improves performance by 7.2% on average compared to standard OPD baselines.
Knowledge Distillation Knowledge distillation (KD) (Hinton et al., 2015) is a widely used paradigm for model compression, where a student is trained under the guidance of a stronger teacher (Rusu et al., 2015; Gou et al., 2021). In autoregressive generation, a common approach is to match the teacher’s conditional next-token distribution via tokenlevel distillation, typically implemented as minimizing the forward KL divergence between student and teacher distributions at each decoding step (Sanh et al., 2019). An alternative is sequence-level distillation, where the student is trained on full sequences produced by the teacher (Kim & Rush, 2016). While these objectives are stable and easy to optimize, they rely on a fixed supervision distribution, leading to train-inference mismatch once the student drifts from the supervised trajectories. 8
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
References
Kim, Y. and Rush, A. M. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp. 1317–1327, 2016.
Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In The twelfth international conference on learning representations, 2024.
Lai, K.-H., Zha, D., Li, Y., and Hu, X. Dual policy distillation. arXiv preprint arXiv:2006.04061, 2020.
Cui, G., Yuan, L., Wang, Z., Wang, H., Zhang, Y., Chen, J., Li, W., He, B., Fan, Y., Yu, T., et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. Solving quantitative reasoning problems with language models, 2022. URL https://arxiv. org/abs/2206.14858, 1, 2022.
Czarnecki, W. M., Pascanu, R., Osindero, S., Jayakumar, S., Swirszcz, G., and Jaderberg, M. Distilling policy distillation. In The 22nd international conference on artificial intelligence and statistics, pp. 1331–1340. PMLR, 2019.
LI, J., Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Huang, S. C., Rasul, K., Yu, L., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L., Fleureau, Y., Lample, G., and Polu, S. Numinamath. [https: //huggingface.co/AI-MO/NuminaMath-1.5](https: //github.com/project-numina/aimo-progress-prize/blob/ main/report/numina dataset.pdf), 2024.
Gou, J., Yu, B., Maybank, S. J., and Tao, D. Knowledge distillation: A survey. International journal of computer vision, 129(6):1789–1819, 2021. Gu, Y., Dong, L., Wei, F., and Huang, M. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., and Lin, M. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., et al. Olympiadbench: A challenging benchmark for promoting agi with olympiadlevel bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024.
Lu, K. and Lab, T. M. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml. 20251026. https://thinkingmachines.ai/blog/on-policydistillation.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627–635. JMLR Workshop and Conference Proceedings, 2011.
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. Hu, J., Zhang, Y., Han, Q., Jiang, D., Zhang, X., and Shum, H.-Y. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025.
Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., and Hadsell, R. Policy distillation. arXiv preprint arXiv:1511.06295, 2015.
Hübotter, J., Lübeck, F., Behric, L., Baumann, A., Bagatella, M., Marta, D., Hakimi, I., Shenfeld, I., Buening, T. K., Guestrin, C., et al. Reinforcement learning via selfdistillation. arXiv preprint arXiv:2601.20802, 2026.
Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
Kim, J., Luo, X., Kim, M., Lee, S., Kim, D., Jeon, J., Li, D., and Yang, Y. Why does self-distillation (sometimes) degrade the reasoning capability of llms? arXiv preprint arXiv:2603.24472, 2026.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 9
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Xiao, B., Xia, B., Yang, B., Gao, B., Shen, B., Zhang, C., He, C., Lou, C., Luo, F., Wang, G., et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026. Yan, J., Li, Y., Hu, Z., Wang, Z., Cui, G., Qu, X., Cheng, Y., and Zhang, Y. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945, 2025. Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., Lu, K., Xue, M., Lin, R., Liu, T., Ren, X., and Zhang, Z. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Ye, T., Dong, L., Chi, Z., Wu, X., Huang, S., and Wei, F. Black-box on-policy distillation of large language models. arXiv preprint arXiv:2511.10643, 2025. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Fan, T., Liu, G., Liu, L., Liu, X., et al. Dapo: An open-source llm reinforcement learning system at scale, 2025. URL https://arxiv. org/abs/2503.14476, 2025. Zeng, W., Huang, Y., Liu, Q., Liu, W., He, K., Ma, Z., and He, J. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892, 2025. Zhang, K., Zuo, Y., He, B., Sun, Y., Liu, R., Jiang, C., Fan, Y., Tian, K., Jia, G., Li, P., et al. A survey of reinforcement learning for large reasoning models. arXiv preprint arXiv:2509.08827, 2025. Zhao, S., Xie, Z., Liu, M., Huang, J., Pang, G., Chen, F., and Grover, A. Self-distilled reasoner: On-policy selfdistillation for large language models. arXiv preprint arXiv:2601.18734, 2026.
10
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
A. Appendix B. Use of LLMs We used a large language model only for spelling and grammar correction of the manuscript text. The LLM was not involved in research ideation, experimental design, data generation, analysis, or substantive writing beyond copy-editing. All content and claims were authored and verified by the authors, who take full responsibility for the paper. The LLM is not an author.
C. Baseline Methods. We benchmark Stable-OPD against the following baselines using Qwen2.5-Math-1.5B and Qwen2.5-Math-7B (Yang et al., 2024). For SFT methods, the base model is fine-tuned on the full 46k dataset. For RL methods, we include GRPO (Shao et al., 2024), trained on the same dataset with verifiable rewards; SimpleRL-Zero (Zeng et al., 2025), which trains from Qwen2.5-Math-7B using rule-based reward; Oat-Zero (Liu et al., 2025) which trains from Qwen2.5-Math-7B and rule-based reward, proposing to remove the standard deviation in GRPO advantage computation and token-level normalization in policy loss computation; PRIME-Zero (Cui et al., 2025), which uses policy rollouts and outcome labels through implict process rewards; and OpenReasonerZero (Hu et al., 2025) which is an open-source implementation of RLVR methods. We also compare with standard OPD (Lu & Lab, 2025), we adopt the same 33k/13k split as above: the model is first supervised fine-tuned on 33k examples and then trained with OPD on the remaining 13k examples.
D. Additional Experiment Results D.1. Additional Experiment Results on More Base models We also compare Stable-OPD with supervised and RL baselines on six mathematical reasoning benchmarks for Qwen2.5Math-1.5B backbone. Similar to results in Table 1, SFT and GRPO substantially improve over the base models, but standard OPD fails to match these gains despite leveraging on-policy samples and dense token-level supervision. The results suggest that training instability limits the effectiveness of OPD. Stable-OPD addresses this issue by stabilizing OPD with mixture distillation and KL regularization, yielding consistent improvements across scales. On the 1.5B backbone, it boosts average accuracy from 28.9% to 36.1% (+7.2), achieving the best performance.
Table 3. Accuracy on six mathematical benchmarks for Qwen2.5-Math-1.5B. The best performance on each benchmark is highlighted in bold. For OPD-based methods we use OpenThinkerV3 as teacher models.
M ODEL
AVG
MATH-500
M INERVA
O LYMPIAD
AMC
AIME24
AIME25
Q WEN 2.5-M ATH -1.5B Q WEN 2.5-M ATH -1.5B-I NSTRUCT
16.0 35.7
28.0 77.4
9.6 28.7
21.2 39.1
26.4 48.1
7.2 12.1
3.6 8.9
SFT GRPO OPD S TABLE -OPD
31.9 30.1 28.9 36.1
70.6 61.8 56.7 73.9
26.8 26.8 23.4 32.6
31.3 32.0 31.0 37.4
37.8 40.2 35.9 43.0
11.7 11.8 11.1 13.8
13.2 7.7 15.0 16.0
D.2. Additional Experiment Results on More Teacher models To further validate the robustness of Stable-OPD under different supervision sources, we conduct experiments with two teachers: DeepSeek-R1-Distill-7B and OpenThinkerV3. Table 1 reports the accuracy on six math benchmarks for all methods. Overall, Stable-OPD with OpenThinkerV3 achieves the best average performance (36.1%) and yields the strongest results on MATH-500 (73.9%) and Olympiad (37.4%). In contrast, Stable-OPD distilled from the R1-distilled teacher performs best on Minerva (32.7%), AIME24 (14.6%), and AIME25 (17.2%). Comparing teachers for the same student, distillation from the stronger OpenThinkerV3 teacher improves the average accuracy over R1-Distill-7B, aligning with the intuition that higher-quality teachers provide more informative token-level supervision. 11
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models Table 4. Accuracy on six mathematical benchmarks. The best performance on each benchmark is highlighted in bold. †denotes models distilled from DeepSeek-R1-Distill-7B, and ‡denotes models distilled from OpenThinkerV3.
M ODEL
AVG
MATH-500
M INERVA
O LYMPIAD
AMC
AIME24
AIME25
Q WEN 2.5-M ATH -1.5B Q WEN 2.5-M ATH -1.5B-I NSTRUCT
16.0 35.7
28.0 77.4
9.6 28.7
21.2 39.1
26.4 48.1
7.2 12.1
3.6 8.9
SFT GRPO OPD† S TABLE -OPD†
31.9 30.1 28.0 35.7
70.6 61.8 58.4 72.0
26.8 26.8 22.4 32.7
31.3 32.0 26.9 34.9
37.8 40.2 36.2 43.0
11.7 11.8 10.9 14.6
13.2 7.7 13.1 17.2
OPD‡ S TABLE -OPD‡
28.9 36.1
56.7 73.9
23.4 32.6
31.0 37.4
35.9 43.0
11.1 13.8
15.0 16.0
Rollout truncation
Rollout repetition
Truncation ratio
Repetition ratio
Truncation ratio
200
400
Step
600
800
0.4
0.6
0.2
0
Eval repetition 0.6
0.8
0.4
0.0
Eval truncation
1.0
0.6
Repetition ratio
1.0 0.8 0.6 0.4 0.2
0.2
0.4
0
200
400
600
Step
0.2
800
OPD
0
200
400
600
Step
Stable-OPD
800
0.0
0
200
400
Step
600
800
Figure 6. Training dynamics of OPD vs. Stable-OPD. Student: Qwen2.5-Math-1.5B; Teacher: DeepSeek-R1-Distill-7B.
E. More Dynamics Analysis of OPD As shown in Fig. 7, abrupt accuracy changes are often synchronized with sharp shifts in the underlying training signals, especially the advantage and the teacher log-probabilities. These synchronized spikes often co-occur with increased truncation or repetition, suggesting a strong correlation between unstable teacher-guided supervision and sudden performance fluctuations. Accuracy Teacher log prob
Rollout: teacher log prob vs acc
0.70
0.4
0.70
0.15
0.65
0.65
0.20
0.60
0.6
0.25
0
200
400
600
Step
800
Advantages
1000
0.55
Accuracy
Advantages
Rollout: advantages vs acc 0.10
0.60
0.8 0
200
Teacher log prob
400
600
Step
800
1000
0.55
Accuracy (MATH500)
Rollout: advantages vs acc
0.7 0.6 0
100
200
300
Step
Advantages
400
500
600
0.5
Rollout: teacher log prob vs acc
0.6
0.7
0.7
Accuracy
0.15 0.20 0.25 0.30 0.35
Accuracy Teacher log prob
Advantages
(a) Student: Qwen2.5-Math-1.5B; Teacher: DeepSeek-R1-Distill-7B
0.8
0.6
0.9 0
Teacher log prob
100
200
300
Step
400
500
600
0.5
Accuracy (MATH500)
Rollout: advantages vs acc 0.85
0.10 0.15 0
100 200 300 400 500 600
0.80 0.75 0.70 0.65
Rollout: teacher log prob vs acc0.85 0.4 0.6 0.8
Step
Advantages
Teacher log prob
0.80 0.75 0.70 0.65
Accuracy
0.05
Accuracy Teacher log prob
Advantages
(b) Student: Qwen2.5-Math-1.5B; Teacher: OpenThinker3-7B
0
100 200 300 400 500 600
Step
Accuracy (MATH500)
(c) Student: Qwen2.5-Math-7B; Teacher: DeepSeek-R1-Distill-7B Figure 7. Dynamics of OPD across training. Each panel shows truncation/repetition for both rollout and evaluation versus accuracy. Sudden accuracy changes often align with abrupt shifts in teacher log-probabilities and advantage estimates.
12