OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning Yuxiao Yang UNC Chapel Hill [email protected]
Xiaoyun Wang NVIDIA [email protected]
arXiv:2605.12400v1 [cs.LG] 12 May 2026
Abstract
and stronger improvements (Agarwal et al., 2024; Lu and Lab, 2025; Gu et al., 2024; Ko et al., 2025; Li et al., 2026b), and has been adopted or explored in industrial post-training pipelines such as MiMo (Team et al., 2026), Qwen3 (Yang et al., 2025), GLM5 (GLM-5-Team et al., 2026), and DeepSeek-V4 (DeepSeek-AI, 2026). Motivated by the success of OPD, a growing line of work has leveraged LLMs’ self-reflection (Shinn et al., 2023) and self-judgment (Zheng et al., 2023) abilities to enable self-improvement without relying on a larger teacher model. This setting, referred to as on-policy self-distillation (OPSD), removes external teacher guidance and thus enables a more direct comparison with RLVR. In OPSD, the student model itself serves as the teacher, typically by conditioning on external privileged information during training to provide dense tokenlevel guidance without requiring a separate stronger model (Zhao et al., 2026; Hübotter et al., 2026). Although such methods can improve both performance and training efficiency, recent studies have reported that vanilla OPSD may suffer from training collapse and instability (Yang et al., 2026a; Li et al., 2026a; Kim et al., 2026). These observations have motivated hybrid approaches that integrate OPSD with RLVR. However, the mechanism underlying this collapse remains underexplored. As a result, existing hybrid methods largely amount to a straightforward combination of the OPSD loss and the GRPO loss, and therefore fail to fully exploit the fine-grained token-level signals provided by OPSD. This naturally raises the following question:
We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories. Despite the performance gains of OPSD, we identify a common but often overlooked mismatch between teacher and student responses: self-reflected teacher responses can be shifted by reflection-induced bias and response templates, leading to miscalibrated token-level supervision. To mitigate this issue, we propose OGLS-SD, an outcome-guided logit-steering framework that leverages verifiable outcome rewards to contrast successful and failed onpolicy trajectories and calibrate teacher logits. By combining outcome-level correctness with dense token-level guidance through logit steering, OGLS-SD stabilizes self-distillation and improves reasoning performance over standard OPSD and other variants across diverse benchmarks.
1
Weitong Zhang UNC Chapel Hill [email protected]
Introduction
With the rapid progress of large language models (LLMs), reinforcement learning with verifiable rewards (RLVR; Shao et al. 2024; Zhu et al. 2025; Zheng et al. 2025) has become a central paradigm for improving reasoning ability during LLM posttraining. However, many RLVR methods provide only sparse, sequence-level supervision, where all tokens in a sampled trajectory are optimized according to the same outcome reward. This coarse supervision makes fine-grained token-level credit assignment difficult and can lead to inefficient learning (Zhao et al., 2026). To address this limitation, on-policy distillation (OPD; Agarwal et al. 2024) has recently emerged as a promising alternative paradigm, in which the student model leverages its own on-policy generations to obtain dense token-level guidance. Compared with RLVR, OPD can often achieve faster
What causes the training collapse of OPSD? In this paper, we hypothesize that OPSD collapse arises from a distributional mismatch between the “teacher” and “student” responses, even though both are from the same language model. We identify two key sources of this mismatch. First, when 1
1.On-policy Rollout
2.Teacher Logit Polling
3. Steering Direction
4. Distillation Target
Teacher logits from successful rollouts
Dataset S= { }
Student distribution
Avg
-
steering direction
Minimize Divergence
Student Steered teacher distribution
Teacher logits from failed rollouts On policy sampling Avg
Anchor teacher logits
+
Steered teacher logits
... Figure 1: Illustration of OGLS-SD. The student model first generates K on-policy rollouts for each input, which are then verified and partitioned into correct and incorrect pools. We compute the average teacher logits conditioned on each pool, construct the steering direction ∆ by contrasting the two averaged logits, and add it to the anchor teacher logits to obtain the steered teacher distribution for token-level guidance.
the teacher constructs its response by self-reflecting on a single output from a specific reasoning path, the resulting supervision may become concentrated around a narrow output distribution, potentially eroding the student’s original capabilities (Yang et al., 2026a). Second, privileged conditioning can make the teacher distribution overly confident and introduce systematic artifacts during self-reflection, causing the student to imitate miscalibrated tokenlevel targets (Kim et al., 2026). Inspired by this hypothesis, we mitigate the distributional mismatch and stabilize the self-distillation process by proposing OGLSSD: Outcome-Guided Logit Steering for SelfDistillation. As illustrated in fig. 1, OGLS-SD provides a simple framework that systematically incorporates outcome-based rewards into on-policy self-distillation. Specifically, OGLS-SD samples a group of on-policy trajectories, verifies their correctness using reward signals, and then constructs an outcome-guided logit-steering direction by contrasting teacher logits induced by successful and failed rollouts. This steering guidance is used to calibrate token-level supervision for self-distillation. In summary, our contributions are threefold:
collapse in vanilla OPSD. • To mitigate this mismatch, we propose OGLSSD, a framework that effectively integrates RLVR with self-distillation and constructs an outcome-guided logit-steering direction to calibrate overconfident teacher guidance. • Empirically, we show that OGLS-SD improves both stability and performance over standard OPSD and naive integrations of RLVR objectives. By mitigating the distributional mismatch, OGLS-SD enables more efficient selfimprovement of language models.
2
Related Works
2.1
On-policy Distillation
On-policy distillation (OPD) samples rollouts from the student policy and aligns the student’s tokenlevel distributions with teacher logits along these on-policy trajectories. Compared with offline distillation, OPD provides dense supervision on the states actually visited by the student, and often leads to faster and more effective performance improvement (Agarwal et al., 2024; Lu and Lab, 2025; Gu et al., 2024; Xu et al., 2025; Kim and Baek, 2026). It has also been widely adopted in industrial post-training pipelines, where it can reduce the reliance on, or even partially replace, RLVRbased post-training (Yang et al., 2025; Team et al., 2026). Recent OPD variants further explore blackbox teacher guidance (Ye et al., 2026), multi-model
• We investigate a commonly overlooked teacher– student mismatch in on-policy self-distillation, focusing on solution-specific bias and privilegedinformation-induced overconfidence. We find that this mismatch may contribute to training 2
distillation (Chen et al., 2026), and agent-level policy distillation (Jülg et al., 2025). 2.2
haviors. Most prior work focuses on controllable generation, persona editing, safety alignment, refusal control, and hallucination mitigation (Zhao et al., 2025; Parekh et al., 2025; Arditi et al., 2024).
LLM Self-Distillation
Despite its promise, OPD typically requires a more capable teacher model, often a larger model, which may be expensive or unavailable in practice. Onpolicy self-distillation (OPSD) addresses this limitation by using the model itself as the teacher, while providing it with external privileged information such as a reference solution or final answer (Zhao et al., 2026; Hübotter et al., 2026). However, vanilla OPSD has been reported to suffer from unstable training and degradation in reasoning performance (Kim et al., 2026; Yang et al., 2026a). Prior work attributes this behavior to different forms of teacher-student mismatch. Kim et al. (2026) observes that self-distillation can suppress epistemic verbalization, leading to more compressed and overconfident reasoning patterns. From an information-theoretic perspective, Yang et al. (2026a) shows that matching a teacher conditioned on a single privileged solution is illposed for a student that cannot access this solution, and that the population optimum corresponds to a marginal privileged teacher. Combining self-distillation with RLVR has recently emerged as a promising direction (Yang et al., 2026a; Li et al., 2026a; He et al., 2026; Yan et al., 2025; Zhang et al., 2025). For example, prior work uses self-distillation to provide denser credit assignment for GRPO (Yang et al., 2026a), or combines GRPO-style policy optimization with selfdistillation objectives (Li et al., 2026a; Ding, 2026). Our work follows this line but focuses on the pattern mismatch induced by privileged prompting, and proposes outcome-contrasted logit steering to reduce outcome-agnostic artifacts in the privileged teacher signal. 2.3
In contrast, using steering as dense trainingtime guidance for self-distillation remains relatively underexplored, particularly in reasoningoriented post-training. The most related work is EXOPD (Yang et al., 2026b), which introduces a reference policy πref and rewrites the OPD objective to strengthen the teacher signal. However, EXOPD focuses on the standard OPD setting, where the teacher is not induced by privileged prompting. Our work instead studies the OPSD setting, where the teacher is constructed by conditioning the same model on privileged information. In this setting, steering is not only used to amplify the teacher signal, but also to mitigate the pattern mismatch introduced by privileged guidance.
3
Preliminaries
We first introduce the notation used throughout this paper. Let x = (x1 , . . . , xm ) denote a question prompt, and let y = (y1 , . . . , yT ) denote a response generated by the student policy. We use y + and y − to denote correct and incorrect rollouts, respectively, as determined by a verifiable reward signal. In OPSD, each prompt is additionally associated with an expert solution s, which serves as privileged information for the teacher. Thus, the training dataset can be written as D = {(xi , si )}N i=1 , while the student rollouts yi are sampled on policy during training.
3.1
Steering the LLM generation
On-Policy Self-Distillation
In OPSD, for each question xi , the student model first generates an on-policy response yi ∼ πθ (· | xi ). The same model, or a closely related copy of it, is then used as a teacher by conditioning on additional privileged information si . Here, si denotes information that is unavailable at test time, such as a ground-truth solution or an expert reasoning path. The teacher parameters θ′ can be instantiated in several ways, such as the base model before training, the current student parameters, or an exponential moving average of the student for improved stability (Zhao et al., 2026; Hübotter et al., 2026).
Inspired by the observation that semantic and behavioral attributes can often be represented as approximately linear directions in model representations (Mikolov et al., 2013), steering methods have been widely studied as lightweight mechanisms for controlling LLM behavior (Rimsky et al., 2024; Turner et al., 2024; Zhao et al., 2025). Existing methods typically construct steering directions from contrastive examples or learned representations, and apply them to hidden states or output distributions to guide generation toward desired be3
The OPSD training objective is T X
LOPSD (θ) =
next token. If this privileged conditioning induces a solution-agnostic confidence shift, such as sharper next-token distributions or reduced probability on epistemic and exploratory tokens (Kim et al., 2026), then this shift can survive marginalization and be inherited by the student through distillation. At test time, where the ground-truth solution is unavailable, such inherited overconfidence may harm reasoning robustness and final-answer accuracy. As illustrated in fig. 2, directly distilling from positive privileged contexts can increase both correctnessrelated confidence and outcome-agnostic overconfidence. We further illustrate this prompt-induced marginalization mismatch with a simple toy example in section A.
KL πθ (· | xi , yi,<t )
t=1
sg πθ′ (· | xi , si , yi,<t ) . (1) where sg[·] denotes the stop-gradient operation. Intuitively, OPSD assumes that, when provided with privileged information, the same model can act as a more capable teacher and provide finegrained token-level guidance for improving the student model. In this way, OPSD replaces sparse sequence-level rewards with dense distributional supervision along the student’s on-policy trajectory. However, because the student policy πθ (· | x, y<t ) cannot access the privileged solution s at test time, directly forcing it to match the privileged teacher πθ′ (· | x, s, y<t )
3.2
Let ct = (x, y<t ) denote the generation context and v denote a candidate next token. Steering can be viewed as modifying a base next-token distribution with a contrastive likelihood ratio: γ + p (v | ct ) , (3) pγ (v | ct ) ∝ p0 (v | ct ) p− (v | ct )
is ill-posed. As pointed out by (Yang et al., 2026a), in the non-privileged student function class, the population optimum is the marginal privileged teacher distribution
where p0 is the base distribution, p+ and p− correspond to positive and negative conditions, and γ ≥ 0 controls the steering strength. Taking the logarithm shows that steering adds the contrastive direction log p+ (v | ct )−log p− (v | ct ) to the base log-probability, up to a normalizing constant. Since LLMs parameterize next-token distributions through logits, this operation is commonly implemented in logit space. Let zt0 , zt+ , and zt− denote the base, positive-conditioned, and negativeconditioned logits at context ct . The steered logits are
P̄T (· | x, y<t ) = Es∼P (s|x,y<t ) [πθ′ (· | x, s, y<t )] . (2) Therefore, directly applying the OPSD loss to each privileged instance incurs an irreducible information-asymmetry term I(Yt ; S | X, Y<t ). Harmful overconfidence region T⁻ = π(· | x, y⁻)
Overconfidence / unjustified certainty
Δ = T⁺ − T⁻
S = π(· | x)
0
π! Guide
λΔ
Guidance in Large Language Models
T⁺ = π(· | x, y⁺)
ztT = zt0 + γ(zt+ − zt− ), pγ (· | ct ) = softmax(ztT ).
Sₛₜₑₑᵣ = S + λ (T⁺ − T⁻)
In OGLS-SD, we instantiate this contrastive direction using outcome-conditioned teacher logits from correct and incorrect on-policy rollouts.
Accuracy / correctness capability
Figure 2: Illustration of how the steering direction guides the student model toward improved correctness while avoiding increased overconfidence.
4
However, marginalizing over privileged solutions does not by itself guarantee that the resulting teacher matches the behavior of a model solving the problem from scratch. Each component in P̄T is still produced by a teacher that has already observed a solution before predicting the
Method
In OGLS-SD, we integrate RLVR with OPSD following prior work (Yang et al., 2026a; Li et al., 2026a; Ding, 2026). For each input xi , we sample G rollouts from the current policy, {yj }G j=1 ∼ πθ (· | xi ), 4
(4)
and verify each rollout using the ground-truth label or a rule-based verifier provided by the dataset. This gives an outcome-labeled set Y(xi ) = {(yj , rj )}G j=1 ,
common teacher behavior induced by incorrect rollouts, which we use to estimate failure-related or overconfident guidance directions. In practice, the averaged logits are maintained by a running accumulator over the guidance pool. This avoids storing all individual teacher logits and only requires memory for the running accumulator.
(5)
where rj ∈ {0, 1} indicates whether yj is correct. We then partition the rollouts into positive and negative sets: Y + (xi ) = {yj | rj = 1, j ∈ [G]}, Y − (xi ) = {yj | rj = 0, j ∈ [G]}.
4.2
+ − Given the mixed guidance logits z̄j,t and z̄j,t , we construct an outcome-guided steering direction by contrasting positive and negative guidance:
(6)
Here, Y + (xi ) contains verified correct rollouts, while Y − (xi ) contains incorrect rollouts. When clear from context, we omit the dependence on xi . 4.1
+ − ∆j,t = z̄j,t − z̄j,t .
Motivated by the marginal teacher in eq. (2), we aggregate privileged guidance over multiple correct solution contexts rather than distilling from a single privileged solution. Instead of explicitly forming a probability-space mixture, we use a logit-space aggregation surrogate, which is easy to maintain with running accumulators and directly compatible with our steering operation.
0 zj,t = zθ′ (· | xi , yj,<t )
(7)
where λ ≥ 0 controls the steering strength. The steered teacher distribution is then T πT (· | xi , yj,<t ) = softmax zj,t . (14) For each rollout yj , we define the sequence-level steering loss as Lsteer (θ) = j
(8)
(15)
The overall loss is then
(9)
Lsteer (θ) =
G X
1{rj = 0}Lsteer (θ). j
(16)
j=1
Since the contrastive direction is intended to attenuate the shared overconfidence component while retaining the correctness improving component, we can use λ > 1 to amplify the useful steering signal. For rollouts that are already correct, additional steering may unnecessarily perturb the model away from its correct behavior and harm its existing capability. The overall procedure is summarized in algorithm 1.
positive and negative guidance logits are defined as 1 X zθ′ (· | xi , g, yj,<t ). |Pj± | ±
KL πθ (· | xi , yj,<t ) ∥ sg πT (· | xi , yj,<t ) .
For brevity, we write Pj± = Pj± (xi ). The mixed ± z̄j,t =
Tj X t=1
If no dataset-provided solution is available, the term {si } is omitted. Similarly, we define the negative guidance pool as Pj− (xi ) = Y − (xi ).
(12)
denote the base teacher logits without privileged rollout guidance. We define the steered teacher logits as T 0 zj,t = zj,t + λ∆j,t , (13)
where πE (· | xi , yj,<t ) denotes an oracle posterior over correct solutions given the current prefix. Since this posterior is inaccessible, we approximate it using verified correct rollouts from Y + (xi ). For the current rollout yj , we define the positive guidance pool as Pj+ (xi ) = Y + (xi ) ∪ {si }.
(11)
As illustrated in fig. 2, we hypothesize that positive and negative guidance share some privileged prompt induced overconfidence components but differ in their outcome discriminative components. Contrasting them is therefore intended to attenuate shared overconfidence while preserving correctness-improving signals. Let
Maintaining Mixed Guidance Logits
Es∼πE (·|xi ,yj,<t ) [zθ′ (· | xi , s, yj,<t )] ,
Outcome-Guided Logit Steering
(10)
g∈Pj
Averaging positive logits reduces the dependence on any single solution path and provides a more robust approximation to the oracle positive guidance. In contrast, averaging negative logits captures the 5
Algorithm 1: OGLS-SD: Outcome-Guided Logit Steering Input: Training dataset D, policy πθ , teacher πθ′ , verifier R, rollout number G, steering strength λ, training steps N . Output: Updated policy πθ . for i = 1, . . . , N do Sample (xi , si ) from D; // Omit si if no oracle solution is available Sample rollouts {yj }G ∼ π (· | x ); i θ j=1 Verify each rollout by rj ← R(xi , yj ) for j = 1, . . . , G; for j = 1, . . . , G do if rj = 1 then continue; // Only steer incorrect rollouts Set Pj+ ← {yk : rk = 1, k ∈ [G]} ∪ {si } and Pj− ← {yk : rk = 0, k ∈ [G]}; if Pj+ = ∅ or Pj− = ∅ then continue; // Contrastive direction is undefined T
j + − Compute mixed guidance logits {z̄j,t , z̄j,t }t=1 over the sequence using eq. (10); Construct the steered teacher distribution πT (· | xi , yj,<t ) using eq. (14); Compute rollout steering loss Lsteer using eq. (15); j
Gather rollout losses {Lsteer } into Lsteer ; j Update πθ by minimizing Lsteer ; Update or synchronize πθ′ ; // According to the chosen teacher scheme
5
Experiments
5.1
Setup
Method
We evaluate OGLS-SD on Qwen3-1.7B and Qwen3-4B (Yang et al., 2025), following the original OPSD setting (Zhao et al., 2026). We train on OpenThought (Guha et al., 2025) and evaluate on AIME 2024 and AIME 2025 (Zhang and Math-AI, 2024, 2025), two challenging mathematical reasoning benchmarks. For OGLS-SD, we sample 8 rollouts for each question, apply symmetric per-token pointwise divergence clipping, and use a ramp-up schedule for the steering coefficient λ, with a maximum value of 3. Unless otherwise specified, we set the maximum generation length to 8,192 tokens during training and 38,912 tokens during evaluation. All methods are trained with LoRA using the same backbone model and training configuration to ensure a fair comparison with OPSD. 5.2
AIME24
AIME25
Avg.
Qwen3-1.7B Base + SFT + GRPO + OPSD + OGLS-SD
51.5 48.4 51.1 56.3 59.2
36.7 36.3 38.3 40.8 45.0
44.1 42.4 44.7 48.5 52.1
Qwen3-4B Base + SFT + GRPO + OPSD + OGLS-SD
74.9 70.2 75.6 76.3 77.9
66.4 62.3 68.1 67.9 68.8
70.1 66.3 71.9 72.1 73.4
Table 1: Results on AIME benchmarks using Qwen31.7B and Qwen3-4B. Results are reported as mean@8.
5.3
Main Results
As shown in table 1, OGLS-SD consistently outperforms all baselines on both AIME 2024 and AIME 2025. These results suggest that outcome-guided logit steering can provide more effective tokenlevel supervision than directly distilling from privileged teacher guidance. In particular, by contrasting successful and failed rollouts, OGLS-SD mitigates overconfident guidance while preserving correctness-improving signals, leading to more stable self-distillation. To better understand the output patterns of OPSD
Baselines
We compare against three baselines. SFT directly imitates expert trajectories from the dataset. GRPO (Shao et al., 2024) optimizes the policy using outcome-level rewards from verified rollouts. OPSD (Zhao et al., 2026) performs on-policy selfdistillation with privileged teacher guidance. 6
Problem: Subtraction game with moves {1, 4}. Alice moves first. Count n ≤ 2024 where Bob can force a win.
1. Finds the correct ...losing positions are numbers where n ≡ 0 or 2 mod pattern and count 5. Therefore total losing positions: 404 + 405 = 809
OPSD
Correct pattern, wrong role mapping
2. Wrong role mapping
Thus, there are 809 positions where Bob cannot guarantee a win.
3. Overconfident commitment
Therefore answer is 2024 - 809 = 1215 ... Thus I think the answer is 1215. Final Answer: \boxed{1215}
Bob needs to have a strategy that can make the game start from a
1. Reflection at the losing position? Wait, no.Wait, if Alice is first, then if the starting role mismatch position is a losing position, then Bob cannot win? Wait, no...
Ours Reflection fixes the role mapping
2. Correct role mapping
If the original number n satisfies this modular congruence, the starting player is in a losing state, and thus Bob can ensure a win
3. Correct answer
Total numbers divisible by 5 in 1-2024: 404. Total numbers ≡2 mod5: 405. Final Answer: \boxed{809} Total numbers: 404 + 405=809.
Figure 3: Case study comparing OPSD and OGLS-SD. Both methods identify the correct modular pattern and compute the correct intermediate count, but OPSD overconfidently maps the losing-position count to the wrong player and outputs the incorrect complement. In contrast, OGLS-SD revisits the Alice/Bob role assignment and preserves the correct answer.
and OGLS-SD, we count the occurrence frequency of seven epistemic reflection markers, as listed in table 3. As shown in fig. 5, the marker frequency of OPSD first decreases during the early stage of training and then gradually increases after around 100 steps, but remains consistently below that of the untrained model. In contrast, OGLS-SD generally increases the frequency of these markers throughout training. This suggests that OGLS-SD may better preserve explicit epistemic reflection behaviors, whereas OPSD tends to suppress them in the early stage. We further provide a qualitative case study in fig. 3. Both OPSD and OGLS-SD identify the correct intermediate structure, but OPSD overconfidently maps the result to the wrong target, while OGLS-SD revisits the key role assignment and avoids the mistake. 5.4
Figure 4: Avg@8 performance of Qwen3-1.7B on AIME24 across training checkpoints. OPSD reaches its peak performance around 100 steps and then degrades, suggesting instability caused by teacher–student pattern mismatch. In contrast, OGLS-SD yields a more stable training curve and achieves better overall performance.
ment of OGLS-SD mainly come from constructing a less biased mean teacher distribution from multiple positive solutions? and (2) does the improvement simply come from using a steering coefficient λ > 1 to amplify the positive guidance signal? To answer the first question, we consider Pos. Mean
Ablation Study
To understand the effectiveness of OGLS-SD, we conduct ablation experiments on Qwen3-1.7B using AIME 2024, as shown in table 2 and fig. 6. We aim to answer two questions: (1) does the improve7
Table 2: Ablation results on AIME 2024 using Qwen31.7B. Results are reported as mean@8. Method Qwen3-1.7B Base (Instruct) + OPSD + Pos. Mean Teacher + Positive-Only Steering + OGLS-SD
AIME 2024 51.5 56.3 57.1 55.4 59.2
Figure 5: Frequency of explicit epistemic reflection markers in reasoning traces generated by Qwen3-1.7B on AIME 2024.
Teacher, which replaces the single privileged solution in OPSD with an averaged positive teacher distribution. Specifically, we apply an OPSD-like objective LPMT (θ) =
T X
Figure 6: Test performance curves of ablation variants on AIME 2024.
h i KL πθ,t sg π̄θ+′ ,t ,
t=1
contrasting positive and negative rollout-induced teacher patterns is important for improving both performance and stability.
where πθ,t = πθ (· | xi , yi,<t ) and π̄θ+′ ,t = π̄θ+′ (· | xi , yi,<t ) constructed from the reference solution and correct on-policy rollouts. As shown in table 2, this variant slightly improves over OPSD and produces a relatively stable test curve in fig. 6, but it still underperforms OGLS-SD. This suggests that simply averaging multiple positive guidance distributions is helpful but insufficient. To answer the second question, we consider Positive-Only Steering, which amplifies the positive guidance direction without using the negative rollouts. The teacher logits are constructed as + T 0 0 zj,t = zj,t + λ zj,t − zj,t ,
6
Conclusion
In this paper, we propose OGLS-SD, an on-policy self-distillation framework that constructs outcomeguided logit steering from successful and failed rollouts. Instead of directly distilling from a single privileged teacher distribution, OGLS-SD averages teacher logits over outcome-conditioned rollout pools and contrasts them to reduce solutionspecific bias and privileged-information-induced overconfidence. Experiments on mathematical reasoning benchmarks show that OGLS-SD improves over standard OPSD and leads to more stable selfdistillation.
0 denotes the base teacher logits and z + where zj,t j,t denotes the positive teacher logits. This variant strengthens the positive guidance signal, in a form close in spirit to EXOPD (Yang et al., 2026b), but does not explicitly cancel the shared pattern mismatch between positive and negative privileged teachers. As shown in fig. 6, Positive-Only Steering reaches its peak performance at the beginning of training and then gradually degrades, indicating that simply amplifying positive guidance does not yield stable improvement. Overall, these ablations show that the gain of OGLS-SD does not only come from using a less biased mean teacher or a larger steering coefficient. Instead, explicitly
References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. Preprint, arXiv:2406.11717.
8
Hansheng Chen, Kai Zhang, Hao Tan, Leonidas Guibas, Gordon Wetzstein, and Sai Bi. 2026. pi-flow: Policy-based few-step generation via imitation distillation. In The Fourteenth International Conference on Learning Representations.
Minsang Kim and Seung Jun Baek. 2026. Explain in your own words: Improving reasoning via tokenselective dual knowledge distillation. In The Fourteenth International Conference on Learning Representations.
DeepSeek-AI. 2026. Deepseek-v4: Towards highly efficient million-token context intelligence.
Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se-Young Yun. 2025. DistiLLM-2: A contrastive approach boosts the distillation of LLMs. In Forty-second International Conference on Machine Learning.
Ken Ding. 2026. Hdpo: Hybrid distillation policy optimization via privileged self-distillation. Preprint, arXiv:2603.23871.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. Preprint, arXiv:2309.06180.
GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, and 168 others. 2026. Glm-5: from vibe coding to agentic engineering. Preprint, arXiv:2602.15763.
Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. 2026a. Unifying grouprelative and self-distillation policy optimization via sample routing. Preprint, arXiv:2604.02288.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations.
Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. 2026b. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. Preprint, arXiv:2604.13016.
Etash Kumar Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Rea Sprague, Ashima Suvarna, Benjamin Feuer, Leon Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, and 31 others. 2025. Openthoughts: Data recipes for reasoning models. In First Workshop on Foundations of Reasoning in Language Models.
Kevin Lu and Thinking Machines Lab. 2025. Onpolicy distillation. Thinking Machines Lab: Connectionism. Https://thinkingmachines.ai/blog/onpolicy-distillation.
Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, and Sanjeev Arora. 2026. Self-distillation zero: Self-revision turns binary rewards into dense supervision. Preprint, arXiv:2604.12002.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In International Conference on Learning Representations.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
Jayneel Parekh, Pegah KHAYATAN, Mustafa Shukor, Arnaud Dapogny, Alasdair Newson, and Matthieu Cord. 2025. Learning to steer: Input-dependent steering for multimodal llms. In Advances in Neural Information Processing Systems, volume 38, pages 159799–159834. Curran Associates, Inc.
Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. 2026. Reinforcement learning via self-distillation. Preprint, arXiv:2601.20802.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thailand. Association for Computational Linguistics.
Tobias Jülg, Wolfram Burgard, and Florian Walter. 2025. Refined Policy Distillation: From VLA generalists to RL experts. In Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS).
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. 2026. Why does self-distillation (sometimes) degrade the reasoning capability of llms? Preprint, arXiv:2603.24472.
9
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 8634–8652. Curran Associates, Inc.
Songming Zhang, Xue Zhang, Tong Zhang, Bojie Hu, Yufeng Chen, and Jinan Xu. 2025. AlignDistil: Token-level language model alignment as adaptive policy distillation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19791– 19807, Vienna, Austria. Association for Computational Linguistics.
Core Team, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, Gang Xie, Hailin Zhang, Hanglong Lv, Hanyu Li, Heyu Chen, Hongshen Xu, Houbin Zhang, Huaqiu Liu, and 107 others. 2026. Mimo-v2-flash technical report. Preprint, arXiv:2601.02780.
Yifan Zhang and Team Math-AI. 2024. American invitational mathematics examination (aime) 2024. Yifan Zhang and Team Math-AI. 2025. American invitational mathematics examination (aime) 2025. Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. 2025. Mitigating object hallucination in large vision-language models via image-grounded guidance. In Forty-second International Conference on Machine Learning.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. Steering language models with activation engineering. Preprint, arXiv:2308.10248.
Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. 2026. Self-distilled reasoner: On-policy self-distillation for large language models. Preprint, arXiv:2601.18734.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. TRL: Transformers Reinforcement Learning.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. 2025. Group sequence policy optimization. Preprint, arXiv:2507.18071.
Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. 2025. Speculative knowledge distillation: Bridging the teacherstudent gap through interleaved sampling. In The Thirteenth International Conference on Learning Representations.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623. Curran Associates, Inc.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. In Advances in Neural Information Processing Systems, volume 38, pages 117157–117186. Curran Associates, Inc.
Xuekai Zhu, Daixuan Cheng, Dinghuai Zhang, Hengli Li, Kaiyan Zhang, Che Jiang, Youbang Sun, Ermo Hua, Yuxin Zuo, Xingtai Lv, Qizheng Zhang, Lin Chen, Fanghao Shao, Bo Xue, Yunchong Song, Zhenjie Yang, Ganqu Cui, Ning Ding, Jianfeng Gao, and 4 others. 2025. Flowrl: Matching reward distributions for llm reasoning. Preprint, arXiv:2509.15207.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. 2026a. Self-distilled rlvr. Preprint, arXiv:2604.03128. Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. 2026b. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. Preprint, arXiv:2602.12125. Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, and Furu Wei. 2026. Black-box onpolicy distillation of large language models. Preprint, arXiv:2511.10643.
10
A
A Tiny Example of Prompt-Induced Mismatch
Prior work (Yang et al., 2026a) shows that, under the information-asymmetry formulation, the optimal non-privileged student target is the marginal privileged teacher P̄T (· | x, y<t ) = Es∼P (s|x,y<t ) [πθ′ (· | x, s, y<t )] .
(17)
This follows from the law of total probability when the conditional distributions are induced by a common joint process. However, in LLM-based OPSD, πθ′ (· | x, s, y<t ) is implemented by changing the prompt to include the privileged solution s. Thus, it is a prompt-induced kernel rather than necessarily a statistical conditional of the non-privileged generation process. We illustrate this distinction with a simple toy example. We use Qwen3-1.7B and ask the model to generate an integer from 0 to 9. For the non-privileged prompt, we compute the model’s next-token probabilities over the digits 0, . . . , 9, denoted by p0 (y) = πθ (y | ϕ0 ), where ϕ0 is the prompt asking the model to generate an integer. We then construct a privileged prompt by prepending “The target integer is x.” to the same request. This gives a prompt-conditioned distribution ppriv (y | x) = πθ (y | ϕpriv (x)). Given an externally chosen distribution q(x), we form the prompt mixture p̂q (y) =
X
q(x) ppriv (y | x).
(18)
x
Unlike ordinary statistical marginalization, p̂q is a mixture over prompt interventions and is not guaranteed to recover the non-privileged distribution p0 . As shown in fig. 7, the resulting prompt mixtures can be far from the non-privileged distribution. Across different choices of q, we observe DTV (p0 , p̂q ) > 0.5. This toy example does not contradict the probabilistic identity used in prior analyses; rather, it highlights a modeling gap when applying that identity to LLM prompting. In particular, revealing s through the prompt can change the model’s generation mode, so averaging over prompted privileged distributions may still leave a prompt-induced pattern mismatch.
Figure 7: Toy demonstration that prompt-based marginalization need not recover the non-privileged generation distribution. We compare the digit distribution under a non-privileged prompt p0 with prompt mixtures p̂q (y) = P x q(x)ppriv (y | x) induced by different external distributions q.
11
Table 3: Keyword set for measuring explicit self-reflection frequency.
Keywords wait, hold on, oops, not quite, let me rethink, let me reconsider, let me check
B
Experiments Details
B.1
Implementation Details
We first describe the implementation details of OGLS-SD. We generally follow the official OPSD codebase (Zhao et al., 2026) and build our method on top of it. We use LoRA (Hu et al., 2021) for post-training, with learning rate 5 × 10−6 , LoRA rank r = 64, and LoRA scaling factor α = 128. To generate on-policy sequences, we use vLLM (Kwon et al., 2023) for efficient sampling with temperature 1.1, and implement the training pipeline with TRL (von Werra et al., 2020). For each prompt, we sample 8 rollouts with a maximum generation length of 8,192 tokens to obtain final answers. During the distillation phase, we only use the first 1,024 tokens of each trajectory, following OPSD. For clipping, we apply symmetric pointwise divergence clipping with threshold 0.05. To maintain stable guidance throughout training, we keep the teacher model fixed as the original base model. During evaluation, we set the maximum generation budget to 38,912 tokens. All experiments are conducted on 4 NVIDIA A100 GPUs with 80GB memory. B.2
Prompt Details
We first present the prompt format used for the privileged teacher. Prompts for teacher model <|im_start|>user Problem: {problem} Here is a reference solution to this problem: === Reference Solution Begin === {solution} === Reference Solution End === After reading the reference solution above, make sure you truly understand the reasoning behind each step—do not copy or paraphrase it. Now, using your own words and independent reasoning, derive the same final answer to the problem above. Think step by step, explore different approaches, and do not be afraid to backtrack or reconsider if something does not work out: Please reason step by step, and put your final answer within \boxed{}. <|im_end|> We then present the prompts used in the toy experiment in section A. Prompts for section A Non-privileged: Generate one integer from 0 to 9. Return only the integer. Privileged: The target integer is {x}. Generate one integer from 0 to 9. Return only the integer. We also list the keywords used to measure explicit self-reflection frequency in table 3.
12