arXiv CSopen access

UniSD: Towards a Unified Self-Distillation Framework for Large Language Models

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

UniSD: Towards a Unified Self-Distillation Framework for Large Language Models Yiqiao Jin1∗ , Yiyang Wang1∗ , Lucheng Fu1 , Yijia Xiao2 , Yinyi Luo3 , Haoxin Liu1 , B. Aditya Prakash1 , Josiah Hester1 , Jindong Wang4† , Srijan Kumar1† 1 Georgia Institute of Technology

2 University of California, Los Angeles

arXiv:2605.06597v1 [cs.CL] 7 May 2026

3 Carnegie Mellon University

4 William & Mary

Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. However, SD in autoregressive LLMs remains challenging because selfgenerated trajectories are free-form, correctness is task-dependent, and plausible rationales can still provide unstable or unreliable supervision. Existing methods mainly examine isolated design choices, leaving their effectiveness, roles, and interactions unclear. In this paper, we propose UniSD, a Unified framework to systematically study Self-Distillation. UniSD integrates complementary mechanisms that address supervision reliability, representation alignment, and training stability, including multiteacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping. Across six benchmarks and six models from three model families, UniSD reveals when self-distillation improves over static imitation, which components drive the gains, and how these components interact across tasks. Guided by these insights, we construct UniSD∗ , an integrated pipeline that combines complementary components and achieves the strongest overall performance, improving over the base model by +5.4 and the strongest baseline by +2.8. Extensive evaluation highlights selfdistillation as a practical and steerable approach for efficient LLM adaptation without stronger external teachers.

Introduction

As large language models (LLMs) are deployed across increasingly diverse applications, post-training adaptation has become essential for specializing pretrained models to new domains, tasks, and deployment constraints. In practice, adaptation pipelines often rely on stronger external models for supervision, including synthetic data generation [1–3], reinforcement learning [4, 5], and distillation from stronger teacher models [6, 7]. While effective, this dependence introduces practical limitations. Repeated supervision from stronger models can dominate training cost, and continued improvement may depend on models restricted by access, policy, or licensing [8]. Moreover, external teachers may propagate undesirable properties, such as bias or privacy-sensitive content [9]. These limitations motivate a central question: can LLMs improve by learning from self-derived supervision, rather than relying on stronger external teachers? Challenges. Self-Distillation (SD) offers a promising direction, where the model derives supervision from its own behavior rather than from a stronger external teacher. However, effective self-distillation in autoregressive LLMs is fundamentally challenging: 1) Open-Ended Generation. LLM generations are freeform trajectories rather than fixed prediction targets: a prompt may admit multiple valid answers, reasoning paths, explanations, or code implementations, and each generated prefix changes the future conditioning state [10–12]. This makes reliability difficult to assess, since an output can be partially correct, stylistically * Equal contribution. Contact: [email protected], [email protected]. †

Corresponding authors: [email protected], [email protected].

Challenges for Self-Dis1lla1on ⚠ Open-Ended Genera,on ⚠ Reliable & Stable Supervision ⚠ Systema,c Understanding

…

UniSD: Uniﬁed Self-Dis1lla1on Framework

…

Mul,-Teacher Agreement

UniSD Objec+ves

Higher Accuracy No External Models

EMA Teacher

Reliable Self-Improvement Generalizability

Divergence Clipping

Contras,ve Learning

…

Feature Matching

Extensible Framework Science

Coding

Tool Use

Figure 1: Overview of UniSD, a unified framework for self-distillation in LLMs. UniSD integrates agreement, stabilization, clipping, contrastive learning, and feature matching to enable systematic analysis. UniSD∗ further integrates various components to improve LLMs without stronger external teachers. different, or locally misleading even when the final answer appears plausible. 2) Unreliable and Unstable Self-Supervision. Self-derived supervision is inherently noisy and unstable. On-policy trajectories expose the model to its own errors, while real-world demonstrations may contain incorrect labels, weak explanations, or underspecified rationales. Because the teacher signal can evolve with the student, transient mistakes, overconfident predictions, and rare high-divergence tokens may be reinforced across updates. 3) Lack of Systematic Understanding. Existing SD methods usually study self-distillation strategies in isolation. It remains unclear which factors drive self-improvement, how they interact, and when each component is beneficial. This Work. We propose UniSD, the first Unified framework to systematically study Self-Distillation in LLMs. UniSD casts self-distillation as a reliability-aware self-correction process over on-policy trajectories: the student first attempts a completion, then learns through comparison and supervision across multiple teacher views, weighting reliable signals and consolidating the resulting knowledge into its own behavior. This formulation organizes self-distillation mechanisms around three complementary axes. First, supervision reliability identifies which self-derived signals should guide learning. Multi-teacher agreement estimates reliability by measuring cross-view consistency over the same trajectory, while Token-Level Contrastive Learning distinguishes informative supervision from plausible but incorrect alternatives. Second, representation alignment extends self-distillation beyond output distributions: Feature Matching regularizes the student toward teacher representations, promoting structural coherence in the learned solution. Third, training stability governs the magnitude and smoothness of student updates. An EMA teacher supplies a temporally smoothed target, while Divergence Clipping prevents rare high-divergence tokens from disproportionately influencing optimization. Together, these components form a modular framework for analyzing the effectiveness of self-derived supervision and for constructing UniSD∗ , an integrated variant that does not rely on stronger external teacher models. Contributions. Our contributions are as follows. • We propose UniSD, the first Unified and extensible framework for systematically studying Self-Distillation in autoregressive LLMs through three axes: supervision reliability, representation alignment, and training stability. • Leveraging UniSD, we conduct extensive evaluation across six benchmarks and six models from three model families, revealing which components drive self-distillation gains and how their interactions affect robustness, transfer, and retention.

…

↻ Update 𝜃 A

Student Policy Input

Self-Derived Teacher

𝜋#"

…

𝑙$#

…

𝑙$!

𝜋!"

𝜋!" (⋅ |𝑥, 𝑐! , 𝑦(#$ ) 𝑙$! = log 𝜋 %! (𝑦($ |𝑥, 𝑐! , 𝑦(#$ )

𝜋∗" (⋅ |𝑥, 𝑐 ∗ , 𝑦(#$ )

B EMA teacher 𝜃& 𝜃' 𝜃(

…

𝑙$%

…

On-policy Rollout

Token-level

𝜋%"

…

𝑥

𝜋* (⋅ |𝑥, 𝑦(#$ )

Multi-Teacher Agreement

…

𝜃)

𝜃& 𝜃( 𝜃' 𝜃) 𝜃) = 𝛽𝜃)0' + (1 − 𝛽)𝜃)

C Token-level Contrastive Learning 𝑦$/

Downweight Unreliable Signals

𝛿$ = 𝐴 𝑙$& &'%

Sequence-level

Agreement high

𝛿/01 = 𝐴 𝐿& ! &'%

low

Feature matching

(

& 𝑚$ 𝑤$ 𝐷 𝜋) ⋅ |𝑥, 𝑦.*$ ∥ 𝜋∗"(⋅ |𝑥, 𝑐∗, 𝑦.*$ ) $'%

+𝜆,-.ℒ,-.

weighted token-level loss

𝑤$

… …

Divergence Clipping

𝑓$* Student

𝑦$0

…

push pull 𝜋) away closer max 0, 𝛾 + 𝑑$/ − 𝑑$0

𝜅 𝑓$∗ Teacher

∑$ 𝑚$ 𝑓$* − 𝑓$∗

( (

… 6$ = min 𝐷$(-) , 𝜅 𝐷

Figure 2: UniSD is a Unified framework for systematically studying Self-Distillation in autoregressive LLMs. It integrates multiple complementary objectives: Multi-Teacher Agreement, EMA Teacher, Token-Level Contrastive Learning, Feature Matching, and Divergence Clipping. The modular design enables controlled analysis of each component and is extensible to additional strategies. • Guided by these insights, we construct UniSD∗ , an integrated variant that combines complementary components and achieves the strongest overall performance, showing that LLMs can improve in both in-domain and OOD settings using self-derived supervision rather than external teachers.

Method

2.1

Self-Distillation in Autoregressive LLMs

We study self-distillation in autoregressive LLMs, where the model improves using supervision derived from its own behavior rather than from stronger external teachers [10, 11]. As discussed in §1, the task is challenging because LLM generations are open-ended and the resulting self-distillation signals can be unstable. Effective self-distillation must therefore select useful self-distillation signals while estimating when each signal is trustworthy. Let πθ denote the student policy. Given an input x, the student samples an on-policy completion ŷ = (ŷ1 , . . . , ŷT ) ∼ πθ (· | x). Self-distillation supervises this trajectory with a primary teacher π∗T (· | x, c, ŷ<t ), while auxiliary teachers estimate the reliability of the target. Training is performed on on-policy student trajectories: " T # X L = Ex Ey∼πθ (·|x) mt wt D πθ (· | x, ŷ<t ) ∥ π∗T (· | x, c, ŷ<t ) + λaux Laux (θ; x, ŷ, c) . (1) t=1

Here, D(· ∥ ·) is a token-level divergence, such as KL divergence and Jensen-Shannon divergence. wt is a reliability weight. mt is a token-level mask. Laux is an auxiliary objective.

2.2

The UniSD Framework

We propose UniSD, the first Unified framework to systematically study LLM Self-Distillation (Algorithm 1). UniSD studies reliable SD along three axes. First, supervision reliability: since self-derived targets can be noisy, Agreement identifies whether the update is supported by multiple teacher views, while TokenLevel Contrastive Learning separates useful supervision from plausible but incorrect alternatives. Second, representation alignment: beyond output distributions, Feature Matching transfers internal representational structure. Third, training stability: EMA Teacher smooths evolving teacher signals, while Clipping prevents 3

rare high-divergence tokens from dominating training. These choices instantiate the same principle from different angles: improving SD requires controlling what signal is used, what representation is matched, and how strongly each update is applied. Multi-Teacher Agreement. Self-derived supervision signals can be noisy and context-sensitive, and dependent on how the teacher is instantiated. Inspired by the wisdom of the inner crowd [13], we use multiple auxiliary teachers to cross-check the same student behavior from different task-preserving perspectives. The auxiliary teacher views serve as reliability probes that measure the stability of the teacher signal under contextual variation rather than as additional distillation targets. Given the student-sampled completion y, we score the same trajectory under each auxiliary teacher: ℓkt = log πkT ŷt | x, ck , ŷ<t , t ∈ [1, T ]. (2) Variation across {ℓkt } reflects uncertainty in the teacher signal. We estimate disagreement at two complemen tary granularities. 1) Token-level agreement captures local unreliable tokens by computing δt = A {ℓkt }K k=1 , where A(·) is a variability statistic, such as variance or range. 2) Sequence-level P agreement captures global instability of the completion. It first aggregates each teacher view as Lk =

mt ℓkt , then computes t=1 mt

t=1 P T

δseq = A({Lk }K k=1 ). Auxiliary teacher views can be generated by any task-preserving perturbation that offers an alternative perspective on the same student trajectory. We instantiate them through context variation, where each view is computed as πkT (· | x, ck , ŷ<t ). We instantiate ck with retrieved / randomly sampled few-shot examples or induced high-level instructions [14]. All views share one teacher model and are batched across contexts, avoiding extra teacher copies that trigger excessive latency or GPU memory usage. Temporal Stabilization with EMA Teachers. Reliability weighting addresses whether the current teacher signal is trustworthy, but it does not prevent the teacher target itself from drifting across training steps. In self-distillation, such temporal drift can propagate transient errors or overconfident predictions into later updates. We therefore use an exponential moving average (EMA) teacher to provide a temporally smoothed self-derived target. Let n denote the optimization step, θn the student parameters, and θ̄n the EMA teacher parameters. We update the teacher as θ̄n = β θ̄n−1 + (1 − β)θn ,

β ∈ [0, 1].

(3)

The EMA teacher defines the target distribution πθ̄n (· | x, c∗ , ŷ<t ), which replaces the primary teacher π∗T in the self-distillation objective (Equation 1). Thus, agreement and EMA address complementary sources of unreliability. Agreement controls which signals are trusted within the current step, while EMA smooths how the teacher target evolves across steps. Token-level Contrastive Learning. Robust self-distillation should not only reinforce reliable teacher signals, but also contrast them against plausible but incorrect alternatives. This is especially important when positive and negative demonstrations share substantial surface structure, such as code solutions that differ only in key implementation details. We therefore introduce a margin-based token-level contrastive objective. Let y + denote positive supervision and y − a wrong answer or flawed rationale. y − can be constructed by prompting an LLM to generate a plausible incorrect alternative, by corrupting the reasoning in y + , or by applying lexical perturbations through WordNet [15], PPDB [16], and TextAttack [17]. Given an on-policy student completion ŷ = (ŷ1 , . . . , ŷT ), we score the same trajectory under the student and teacher distributions conditioned on y + and y − , and optimize Laux : ℓθt = log πθ (ŷt | x, ŷ<t ), Laux (θ; x, y, c) =

T X

T + ℓ+ t = log π (ŷt | x, y , ŷ<t ),

− mt max(0, γ + d+ t − dt ),

t=1

T − ℓ− t = log π (ŷt | x, y , ŷ<t ).

(4)

− θ d− t = |ℓt − ℓt |.

(5)

+ θ d+ t = |ℓt − ℓt |,

− where d+ t and dt measure token-level distances to the positive- and negative-conditioned teacher signals, respectively. mt ∈ {0, 1} masks completion tokens and γ is the margin. The contrastive condition is c = (y + , y − ). This encourages the student trajectory to be closer to correct supervision than to incorrect alternatives.

Feature Matching. Token-level distillation aligns output distributions, but it does not directly constrain the internal features used to produce them. We therefore add an optional feature-matching term that regularizes selected student features toward teacher features, such as hidden states, layer-wise representations [18], selfattention relations or attention-derived features [19], or other task-relevant internal signals. Given the same on-policy completion ŷ = (ŷ1 , . . . , ŷT ), we extract student and teacher features at the same completion-token positions. Let ftθ and ft∗ denote the selected features at token t. We optimize: Lfeat =

T X

mt ftθ − ft∗

t=1

2 2

(6)

where mt masks valid completion tokens. In our implementation, we match final-layer hidden states on completion tokens, providing a representation-level constraint. Divergence Clipping. Rare high-divergence tokens arising from stylistic features can dominate optimization. We therefore clip each scalar token-level divergence after reducing over the vocabulary and before applying reliability weights. 0 < α < 1 defines a weighted Jensen–Shannon divergence: (α) Dt = αD π∗T (· | x, c∗ , ŷ<t ) ∥ Mt + (1 − α)D(πθ (· | x, ŷ<t ) ∥ Mt ) , (7) Mt = (1 − α)πθ (· | x, ŷ<t ) + απ∗T (· | x, c∗ , ŷ<t ),

(8)

where D(· ∥ ·) denotes KL divergence. We additionally support forward- and reverse-KL objectives as separate endpoint-style alternatives to the weighted JSD objective. We then cap the scalar divergence as et = min(D(α) , κ), where κ is the clipping threshold. With agreement weights wt , the clipped distillation D t objective is PT e t=1 mt wt Dt Lclip = P , (9) T t=1 mt wt where mt denotes the completion-token loss mask. When reliability weighting is disabled, the objective reduces to averaging over valid completion tokens. The clipping only caps each token-level distillation term, leaving teacher construction and agreement estimation unchanged, and recovers the unclipped objective when κ is unspecified.

2.3

UniSD∗ : a Unified Pipeline

We instantiate UniSD∗ as a unified pipeline that integrates all objectives (§2.2). From the supervision perspective, multi-teacher agreement and token-level contrastive learning select reliable self-derived signals and suppress plausible but incorrect alternatives. From the representation perspective, feature matching transfers internal structure beyond output distributions. From the optimization perspective, the EMA teacher and divergence clipping stabilize learning under noisy on-policy trajectories. These components combine signal selection, representation alignment, temporal smoothing, and loss stabilization within the same on-policy training loop.

Evaluation

3.1

Experimental Setup

Datasets. We evaluate on six benchmarks spanning four task categories. Four datasets are used for both training and in-domain evaluation, while two are reserved for out-of-domain generalization. 1) Scientific 5

Table 1: Results of UniSD variants, UniSD∗ , and baselines on ID and out-of-domain (in gray) benchmarks using Qwen2.5-7B as the base model. For agreement variants, we use retrieved contexts. Agree (Tok./Seq.) denotes token-/sequence-level agreement. EMA, Contrast, and Clip denote EMA teacher, token-level contrastive learning, and divergence clipping, respectively. Match (Repr./Joint) denotes representation-only and joint logit–representation matching, respectively. Bold and underline denote the best and second-best setting. Method

ScienceQA MBPP CoS-E ToolAlpaca GPQA HumanEval Overall

Raw

81.5

70.8

81.9

61.8

31.0

80.5

67.9

Baselines SFT SDFT [10] GKD [20] SSD [21] OPSD [22]

80.8 81.6 81.2 80.8 81.2

70.4 71.6 72.8 72.4 72.8

82.6 81.2 81.8 79.9 82.0

66.2 73.5 72.1 55.9 61.8

30.6 34.2 33.0 33.7 31.5

79.3 78.7 82.3 81.1 79.9

68.3 70.1 70.5 67.3 68.2

Variants of UniSD Agree (Tok.) 85.2 Agree (Seq.) 84.4 EMA 84.3 Contrast 83.9 Match (Joint) 84.8 Match (Repr.) 83.7 Clip 82.8

71.2 73.2 73.2 73.5 73.2 73.2 73.2

82.2 81.9 81.4 82.0 81.8 81.7 81.7

75.0 76.5 77.9 75.0 76.5 72.1 70.6

36.2 35.7 35.3 34.6 33.5 35.9 31.5

83.5 83.5 82.9 82.3 82.9 82.3 81.7

72.2 72.5 72.5 71.9 72.1 71.5 70.3

UniSD∗

74.7

82.2

77.9

36.4

83.5

73.3

85.0

Reasoning. S CIENCE QA [23] is a science question-answering benchmark covering natural, social, and language science. GPQA [24] is a test-only dataset with expert-level questions in biology, chemistry, and physics. 2) Commonsense Reasoning. C O S-E [25] extends C OMMONSENSE QA [26] with humanwritten explanations. 3) Code Generation. MBPP [27] contains Python programming problems with unit tests. H UMAN E VAL [28] is a test-only dataset featuring function-completion problems. 4) Tool Usage. T OOL A LPACA [29] features multi-step tool-calling interactions. For OOD evaluation, models trained on S CIENCE QA are additionally tested on GPQA, and those trained on MBPP are also tested on H UMAN E VAL. The dataset statistics and licenses are listed in Table 4. Models. We experiment with six LLMs from three model families. Qwen2.5-7B-Instruct [30] serves as the primary model in all main experiments and ablations. To study the effect of model scale, we additionally experiment with Qwen2.5-0.5/1.5/3B-Instruct. To assess cross-family generalization, we further include Llama-3.1-8B-Instruct [31] and gemma-3-4b-it [32].

3.2

Main Results

Table 1 reports the main results on Qwen2.5-7B [30], comparing UniSD variants, baselines, and the integrated pipeline UniSD∗ . Figure 3a further evaluates these trends across model scales. Static imitation is less reliable than on-policy learning. SFT provides limited overall gains despite improving format-oriented tasks. It improves ToolAlpaca by +4.4 over the raw model, where demonstrations largely specify action formats and argument structures, but degrades ScienceQA, GPQA, MBPP, and HumanEval, with limited gains on CoS-E (+0.7). This suggests that off-policy maximum-likelihood training is effective for learning output conventions, but its mean-seeking behavior can be unreliable when supervision 6

Agree (Seq.) Agree (Tok.)

EMA Contrast

Match (Joint) Match (Repr.)

ScienceQA

Clip UniSD ⋆

GPQA

6 4 2 0 −2 0.5B

1.5B 3B Model Scale

0.5B

1.5B 3B Model Scale

Retention w.r.t. Base Models

Accuracy over Base Models (%)

SFT SDFT

Agree (Seq.) Agree (Tok.)

Contrast EMA

Match Clip

2.0

1.5

1.0

0.5B

1.5B 3B Model scale

Figure 3: Left. Gains over the raw Qwen2.5 model [30] across four size variants on ScienceQA (in-domain) and GPQA (OOD). UniSD∗ reaches the largest gain (+7.06) on Qwen2.5-3B. Right. Base-distribution retention perplexity across the same Qwen2.5 size variants. contains diverse reasoning paths, program implementations, or formats. In contrast, on-policy baselines provide a stronger starting point. SDFT improves ToolAlpaca from 61.8 to 73.5 (+11.7) and GPQA from 31.0 to 34.2 (+3.2). Still, its drops on HumanEval and CoS-E indicate sensitivity to noisy demonstrations. Agreement improves supervision reliability. Multi-Teacher Agreement scores the same on-policy student completion under multiple auxiliary teacher views and down-weights signals with high cross-teacher disagreement. Token-level agreement achieves the strongest ScienceQA result (85.2) and is best or second-best on four out of six datasets, suggesting that local reliability estimates can preserve useful token-level supervision. In contrast, sequence-level agreement is more conservative but more stable, matching or improving Raw on all datasets and achieving a stronger overall score than token-level agreement (72.5 vs. 72.2). This reveals a trade-off: token-level agreement better exploits reliable local signals, while sequence-level agreement provides more robust average performance. We further analyze agreement strength, context number, granularity, and auxiliary-context construction in §3.3. Complementary strategies provide additional gains and stabilization. EMA Teacher is the strongest standalone component, matching Agree (Seq.) for the best overall score among individual variants (72.5). The gains are especially pronounced on ToolAlpaca (77.9, +16.1 over Raw), and extend to coding tasks such as MBPP (+2.4) and HumanEval (+2.4), suggesting that smoothing the evolving teacher target is helpful for generation-heavy tasks with strict output protocols. Token-Level Contrastive Learning is slightly weaker on average (71.9), but is more uniformly positive: it improves all six benchmarks, indicating that negativeconditioned supervision provides a robust way to separate useful teacher signals from plausible but incorrect alternatives. Feature Matching shows that representation alignment is helpful but can further benefit from output-level alignment: representation-only matching reaches 71.5 overall, while joint logit–representation matching improves to 72.1. Divergence Clipping is the most conservative, runtime-efficient (Figure 8), and resource-efficient (Table 3) variant. Its relatively modest gains (+2.4) suggest that clipping mainly serves as a lightweight stabilizer rather than a primary learning signal. Combining complementary strategies performs best. Overall, UniSD∗ achieves the strongest performance, improving the overall score from 67.9 to 73.3 (+5.4) and outperforming the strongest baseline GKD (+2.8). This suggests that feature-level regularization is most effective when anchored by token-level distributional supervision. It is best or tied-best on MBPP, ToolAlpaca, GPQA, and HumanEval, and second-best on ScienceQA and CoS-E, indicating broad gains across both in-domain and OOD benchmarks. These improvements support the design principle of UniSD: effective self-distillation should jointly improve teacher 7

reliability, representation alignment, and update stability. Component-level results in Figures 5b and 12 further show that this improvement is not driven by a single dataset or component. At the dataset level, different components contribute complementary strengths: EMA is particularly effective on ToolAlpaca, Agreement and UniSD∗ lead on ScienceQA and HumanEval, and UniSD∗ gives the largest gains on MBPP and GPQA. These trends support the design principle of UniSD: effective self-distillation should jointly improve teacher reliability, representation alignment, and update stability.

3.3

Effects of Agreement Strategies

We analyze how multi-teacher agreement depends on the number of auxiliary teachers K, agreement strength γ, agreement granularity, and the construction of auxiliary contexts. Sensitivity analysis shows that more contexts do not necessarily improve performance. The sensitivity analyses in Appendix Figures 9 and 10 show that performance changes non-monotonically with K. The best setting depends on both task and granularity: sequence-level agreement peaks at K = 3 on ScienceQA with γ = 0.01 (85.2), and at K = 4 on GPQA with γ = 0.01 (36.2), while token-level agreement peaks at K = 7 on ScienceQA with γ = 0.01 (84.4) and on GPQA with γ = 1.0 (36.8). Adding more auxiliary views helps only when they provide complementary and task-relevant evidence. Otherwise, redundant or conflicting contexts can dilute cross-teacher agreement, making the resulting reliability estimate less informative. This is consistent with prior observations that more context does not necessarily lead to better information use [33, 34].

Performance Across Token-level Agreement Methods MBPP

GPQA

HumanEval 79

Cos-E

ScienceQA

ToolAlpaca Raw Model

Retrieval

Agreement strength controls a stability–adaptivity trade-off. Random Induced The effect of K also depends strongly on γ. Smaller γ applies a weaker disagreement penalty, preserving more context-dependent Figure 4: Comparison of token-level supervision but making performance more sensitive to the specific agreement across 3 auxiliary teacher auxiliary views. On ScienceQA with sequence-level agreement, construction strategies. γ = 0.01 achieves the highest accuracy but varies by 2.20 across K. Larger γ filters disagreement more aggressively and produces flatter curves. With γ = 1.0, the corresponding range decreases to 0.85. Thus, weaker agreement weighting can achieve higher peaks when contexts are useful, whereas stronger weighting improves robustness by using auxiliary views more conservatively. Auxiliary-context construction determines when agreement helps. Figures 4 and 11 show that the benefit of agreement also depends on how auxiliary contexts are constructed. Retrieval-based contexts provide nearest-neighbor examples and are most effective when semantic similarity offers task-specific evidence. Under token-level agreement, retrieval gives the strongest results on ScienceQA (85.2), GPQA (36.2), and HumanEval (83.5). However, retrieval is not uniformly best: on coding and open-ended generation tasks, nearest neighbors may share surface form while differing in valid implementation details, limiting the benefit of cross-context agreement. Random contexts are more diverse and remain competitive across both token- and sequence-level agreement, suggesting that diversity can provide complementary supervision when examples are not misleading. Induced contexts trade example-specific evidence for abstract task guidance. This is especially useful for format-sensitive tasks such as ToolAlpaca, where induced token-level agreement reaches 77.9, but less helpful on CoS-E, where short commonsense questions leave less room for generic induced instructions to add useful information.

Component Effectiveness 85

Method Family Base = 81.5%

Baseline Match Agree (Seq.) Agree (Tok.)

Contrast EMA Clip

Match

EMA

Clip

Loss

Gain Over Base Model (%)

Accuracy (%)

Contrast

Agree

Loss w.r.t. Training Steps

5 84

SFT

1 0

100

Avg. training time (min)

0 Agree Contrast EMA

Match

Clip

UniSD

⋆

200

400

Training step

Figure 5: Left: Training time vs. accuracy. Middle: Component effectiveness analysis. The full framework UniSD∗ outperforms all individual components. EMA and multi-teacher agreement provide the strongest single-component gains. Right: Training loss curve on Qwen2.5-7B. Agreement granularity changes supervision robustness. Token- and sequence-level agreement offer different trade-offs across auxiliary teacher construction strategies. Token-level agreement estimates local reliability, preserving useful supervision when only parts of the completion are consistent across teachers. Thus, it achieves stronger peak performance across strategies and often matches or exceeds sequence-level agreement. Sequence-level agreement assigns one reliability score to the whole completion, making it more conservative when teacher views differ in reasoning path or solution style. This reduces peak performance, but improves stability under the retrieved-context setting in Table 1, where sequence-level agreement achieves a slightly higher overall score.

3.4

Generalization Across Models

To verify that UniSD is not specific to a single model family, we evaluate UniSD∗ alongside baselines across three model families: Qwen2.5 [30], Llama-3.1 [31], and Gemma-3 [32]. Figure 7 visualizes the gain over the original models. UniSD∗ achieves the strongest overall performance across all three families, improving over the base models by +5.4, +3.1, and +2.2 on Qwen2.5, Llama-3.1, and Gemma-3, respectively. It also outperforms GKD in overall score for each family. Across the 18 model-dataset pairs, UniSD∗ improves over the raw models in 15 settings, ties in 2, and regresses in only 1 OOD setting. These results suggest that reliability-aware self-distillation transfers across architectures rather than overfitting to one backbone. Notably, CoS-E shows smaller gains likely because instruction-tuned LLMs already encode significant underlying commonsense knowledge required by the task, and its short-form answers leave limited room for improvement. This also explains why SFT is most useful on CoS-E, where its mean-seeking nature can reinforce the dominant demonstration pattern. SFT can calibrate explanation style and reactivates latent knowledge rather than introducing new reasoning behavior. By contrast, coding tasks have a more multimodal output space, where many structurally different programs can be correct. Optimizing toward a single reference-style trajectory can overemphasize common surface patterns and weaken sharp executable solution modes, making SFT less reliable on both in-domain MBPP and OOD HumanEval.

3.5

Completion Likelihood and Distribution Retention

Task accuracy alone does not reveal whether adaptation changes the model distribution in desirable ways. We therefore evaluate two complementary properties. First, reference-completion fit goes beyond finalanswer accuracy and measures whether the adapted model makes the gold completion more likely under teacher forcing. Second, distributional retention measures whether the adapted model preserves the base model’s original generative behavior while acquiring the target skill [35]. Poor retention reflects a form of catastrophic forgetting, where task-specific updates overwrite previously acquired capabilities [36]. Such

models may appear successful on the target task but become over-specialized, producing generations that are less compatible with the base distribution. Gold-completion fit. Given a prompt–completion pair (x, y), we measure whether adaptation improves the likelihood ofP the gold completion by scoring completion tokens under teacher forcing: PPLfit = P log pθ (yt |x,y<t )

exp − (x,y)∈DP t∈M(x,y) , where M(x, y) denotes completion-token positions. By scoring (x,y)∈D |M(x,y)| only completion tokens, this metric focuses on how well the model supports the desired answer trajectory. Across model families, self-distillation substantially improves reference-completion likelihood. On Qwen2.57B, Agreement variants, EMA, and Contrast reduce perplexity from 20.74 to 5.7–6.1. On Gemma-3-4B, these variants reduce perplexity from 47.07 to 10.57–11.24. Feature matching gives less consistent reductions, supporting its role as an auxiliary regularizer rather than the main supervision signal. Base-distribution retention. We next measure distributional retention, i.e. whether adapted generations remain likely under the original base model. For each prompt x, we sample a completion ŷ x from the adapted P P x ) log π (ŷtx |x,ŷ<t 0 x∈D P t model and score it under the original base model π0 : PPLret = exp − . Lower x x∈D |ŷ | PPLret indicates that adapted generations remain more likely under the base distribution, complementing reference-completion fit by measuring preservation rather than task fit. Table 5 shows that SFT can induce substantial drift: on Qwen2.5-7B, retention perplexity increases from 1.14 for the raw model to 1.68, while on Gemma-3-4B it rises from 1.27 to 3.02. Reliability-aware self-distillation generally avoids this collapse. For Qwen2.5-7B, Agreement, EMA, Contrast, and Clip keep PPLret close to the raw model, with the best values between 1.09 and 1.13. EMA teacher reduces retention perplexity by 33.9% relative to SFT, suggesting that a smoothly evolving teacher provides a more distribution-compatible target. Figure 6 further examines retention at the trajectory level. For each generated completion, we UniSD UniSD SFT SFT compute both base-scored perplexity and the average token-level JSD between the adapted and base next-token distributions along the same trajectory. UniSD∗ improves accuracy from 80.8 to 85.0 while reducing mean token-level JSD from 0.054 for SFT to 0.041. The paired analysis further shows that 0 5 10 0.0 0.1 UniSD∗ has lower JSD than SFT on 70.3% of examToken-Level JSD to Base Base-Scored Perplexity ples, with both the mean and median paired differences below zero. Similarly, the base-log-probability Figure 6: Distribution of base-scored perplexity and comparison shows that UniSD∗ completions receive token-level Jensen–Shannon divergence (JSD). higher base-model log-probability on 60.6% of examples. The gain is not merely that the model produces outputs that the base model finds more plausible. More importantly, its token-level predictive distribution remains closer to the base model in generation.

∗

Probability Density

∗

Related Work

Continual Learning and On-Policy Learning. Continual learning [37] aims to adapt models to new knowledge and skills while preserving existing capabilities, a challenge known as catastrophic forgetting [36, 38]. In LLM post-training, this challenge is closely tied to the learning paradigm. Standard supervised fine-tuning (SFT) is off-policy, as it trains on fixed expert demonstrations rather than trajectories induced by the model’s current policy, creating a training-inference mismatch. On-policy learning reduces this mismatch by applying supervision to trajectories sampled from the current policy [20, 39]. For example, GKD [20] reduces exposure bias with on-policy sampling, while MiniLLM [40] and DistiLLM [41] improve distribution matching through stabilized KL objectives.

Knowledge Distillation and Self-Distillation for LLMs. Knowledge distillation (KD) [7] transfers knowledge from a teacher model to a student by matching predictions, logits, hidden states, generated outputs, or reasoning traces. Prior work distills token-level distributions, attention patterns, intermediate representations, rationales, and step-by-step reasoning traces from stronger models [3, 9, 42]. Recent on-policy variants such as VLA-OPD [43], SCOPE [44], and StableOPD [45] further supervise student-generated trajectories using expert teachers or adaptive stabilization. However, these methods usually depend on an external teacher model. Self-distillation instead derives supervision from the model itself or its variants, making it attractive when external teachers are costly, inaccessible, or undesirable [10, 44, 46]. SDFT [10] uses a demonstration-conditioned version of the base model as the teacher. OPSD [22] distills dense supervision on student-generated trajectories. SDPO [46] uses privileged environment feedback for self-improvement. Unlike prior work that studies individual self-distillation recipes, we propose UniSD, a unified and extensible framework for self-distillation.

Conclusion

We presented UniSD, a unified framework for studying self-distillation in LLMs without stronger external teachers. Across six benchmarks and six models from three families, UniSD identifies which components drive self-distillation gains and how they interact across tasks. These insights motivate UniSD∗ , an integrated pipeline that achieves the strongest overall performance. We hope UniSD serves as a foundation for future work on efficient, controllable self-distillation of LLMs.

References [1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36:34892–34916, 2023. [2] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html, 3(6):7, 2023. [3] Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. In CVPR, pages 9590–9601, 2024. [4] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300, 2024. [5] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. NeurIPS, 37:124198–124235, 2024. [6] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv:2505.09388, 2025. [7] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015. [8] Yuvanesh Anand, Zach Nussbaum, Adam Treat, Aaron Miller, Richard Guo, Benjamin Schmidt, Brandon Duderstadt, and Andriy Mulyar. Gpt4all: An ecosystem of open source compressed language models. In NLP-OSS Workshop, pages 59–64, 2023. [9] Yinyi Luo, Yiqiao Jin, Weichen Yu, Mengqi Zhang, Srijan Kumar, Xiaoxiao Li, Weijie Xu, Xin Chen, and Jindong Wang. Agentark: Distilling multi-agent intelligence into a single llm agent. arXiv:2602.03955, 2026. 11

[10] Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026. [11] Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv:2402.13116, 2024. [12] Yiyang Wang, Chen Chen, Tica Lin, Vishnu Raj, Josh Kimball, Alex Cabral, and Josiah Hester. Companioncast: A multi-agent conversational ai framework with spatial audio for social co-viewing experiences. ACM CHI 2026 Workshop on Human-Agent Collaboration, 2026. [13] Stefan M Herzog and Ralph Hertwig. Harnessing the wisdom of the inner crowd. Trends in cognitive sciences, 18(10):504–506, 2014. [14] Or Honovich, Uri Shaham, Samuel Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. In ACL, pages 1935–1952, 2023. [15] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41, 1995. [16] Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. Ppdb: The paraphrase database. In NAACL, pages 758–764, 2013. [17] John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In EMNLP, pages 119–126, 2020. [18] Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: Task-aware layer-wise distillation for language model compression. In ICML, pages 20852–20867. PMLR, 2023. [19] Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi-head selfattention relation distillation for compressing pretrained transformers. In ACL, pages 2140–2151, 2021. [20] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In ICLR, 2024. [21] Ruixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, and Yizhe Zhang. Embarrassingly simple self-distillation improves code generation. arXiv:2604.01193, 2026. [22] Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026. [23] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022. [24] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In COLM, 2024. [25] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. In ACL, pages 4932–4942, 2019. 12

[26] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In NAACL, pages 4149–4158, 2019. [27] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv:2108.07732, 2021. [28] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021. [29] Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv:2306.05301, 2023. [30] Qwen Team. Qwen2.5: A party of foundation models, September 2024. [31] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv:2407.21783, 2024. [32] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv:2503.19786, 2025. [33] Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. How context affects language models’ factual predictions. In AKBC, 2020. [34] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. TACL, 12:157–173, 2024. [35] Xin Yang, Hao Yu, Xin Gao, Hao Wang, Junbo Zhang, and Tianrui Li. Federated continual learning via knowledge fusion: A survey. TKDE, 36(8):3832–3850, 2024. [36] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989. [37] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. TPAMI, 46(8):5362–5383, 2024. [38] Yiyang Wang, Yiqiao Jin, Alex Cabral, and Josiah Hester. Mascot: Towards multi-agent sociocollaborative companion systems. arXiv:2601.14230, 2026. [39] Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models. arXiv:2604.00626, 2026. [40] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In ICLR, 2024. [41] Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. In ICML, 2024. [42] Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv:2603.07079, 2026.

[43] Zhide Zhong, Haodong Yan, Junfeng Li, Junjie He, Tianran Zhang, and Haoang Li. Vla-opd: Bridging offline sft and online rl for vision-language-action models via on-policy distillation. arXiv:2603.26666, 2026. [44] Binbin Zheng, Xing Ma, Yiheng Liang, Jingqing Ruan, Xiaoliang Fu, Kepeng Lin, Benchang Zhu, Ke Zeng, and Xunliang Cai. Scope: Signal-calibrated on-policy distillation enhancement with dual-path adaptive weighting. arXiv:2604.10688, 2026. [45] Feng Luo, Yu-Neng Chuang, Guanchu Wang, Zicheng Xu, Xiaotian Han, Tianyi Zhang, and Vladimir Braverman. Demystifying opd: Length inflation and stabilization strategies for large language models. arXiv:2604.08527, 2026. [46] Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv:2601.20802, 2026. [47] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In ACL, pages 3645–3650, 2019. [48] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54–63, 2020. [49] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv:2104.10350, 2021. [50] Benoit Courty, Victor Schmidt, Sasha Luccioni, Goyal-Kamal, et al. CodeCarbon: mlco2/codecarbon v2.4.1, May 2024. Software. [51] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv:1910.09700, 2019. [52] Victor Avelar, Dan Azevedo, Alan French, and Emerson Network Power. Pue: a comprehensive examination of the metric. White paper, 49:52, 2012. [53] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021. [54] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. [55] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, pages 611–626, 2023. [56] Guancheng Wan, Lucheng Fu, Haoxin Liu, Yiqiao Jin, Hui Yi Leong, Eric Hanchen Jiang, Hejia Geng, Jinhe Bi, Yunpu Ma, Xiangru Tang, et al. Beyond magic words: Sharpness-aware prompt evolving for robust large language models with tare. In ICLR, 2026.

Table 2: Comparison of teacher-forced conditional perplexity on gold completions (§3.5). Lower values indicate better prediction of the reference completion conditioned on the input prompt. The best and secondbest results for each model are shown in bold and underlined, respectively. Qwen2.5-Instruct

Method Raw SDFT

Llama-3.1 Gemma-3

0.5B 1.5B

7.78 7.16

16.19 20.74 6.27 7.56

7.14 4.99

47.07 18.55

7.09 4.77

Agreement (Token-level) Random 5.43 4.70 Retrieval 5.71 4.79 Induction 5.57 4.83 Agreement (Sequence-level) Random 5.37 4.41 Retrieval 5.47 4.82 Induction 5.40 4.44

5.78 5.82 5.82

5.80 5.80 5.78

4.36 4.33 4.44

10.95 10.92 11.24

5.93 5.76 6.10

5.74 6.14 6.03

4.38 4.39 4.41

11.00 10.84 10.57

EMA Contrast Clip Match (Joint) Match (Rep.)

6.00 5.90 5.70 6.16 12.53 13.39 10.65 12.25 15.91 15.62

4.38 4.37 6.09 4.85 5.77

11.22 10.61 24.90 15.33 26.59

5.55 4.93 7.02 6.04 7.12

4.67 4.48 6.00 5.05 6.37

Algorithm Details of UniSD

The detailed procedure of UniSD is shown in Algorithm 1.

Additional Experiments

B.1

Training Time

Training efficiency. Figure 8 compares the wall-clock training cost of different UniSD variants. For the agreement setting, the main cost driver is not the distillation loss itself, but the number of teacher-conditioned scoring passes required for each on-policy completion. Standard SFT is the cheapest baseline. In contrast, agreement-based methods are substantially more expensive because each sampled completion must be rescored under multiple auxiliary contexts before computing reliability weights. For example, on Qwen2.5-7B, sequence-level agreement takes about 100 minutes, compared with 18.6 minutes for SFT. This suggests that agreement estimation is an effective but compute-intensive reliability mechanism. The comparison also reveals a useful design trade-off. Methods that add lightweight stabilization on top of a single teacher signal, such as clipping or feature matching, incur much smaller overhead than full multi-context agreement. EMA, contrastive learning, and joint matching lie between these extremes because they require additional teacher or auxiliary forward passes, but do not multiply the context-conditioned scoring as aggressively as agreement-based variants. Thus, future self-distillation systems should treat reliability estimation as a budgeted component: expensive multi-view agreement can be reserved for noisy or high-uncertainty examples, while cheaper stabilizers such as clipping, EMA smoothing, or representation matching can be applied broadly. This points to adaptive self-distillation designs that allocate computation according to signal reliability rather than applying the most expensive mechanism uniformly to every example.

Algorithm 1 UniSD is a unified and extensible self-distillation framework. Input: dataset D, student policy πθ , primary condition c∗ , auxiliary conditions C(x) = {ck }K k=1 , optional positive/negative supervision (y + , y − ) 1: Initialize EMA teacher parameters θ̄ ← θ ▷ EMA Initialization 2: while not converged do 3: Sample x ∼ D and rollout an on-policy trajectory ŷ = (ŷ1 , . . . , ŷT ) ∼ πθ (· | x) 4: Laux ← 0 ▷ Initialize Auxiliary Objective 5: if EMA teacher is enabled then 6: Use πθ̄T as the primary teacher under c∗ ▷ EMA Teacher 7: else 8: Use π∗T as the primary teacher under c∗ ▷ Primary Teacher 9: end if 10: for t = 1, . . . , T do (α) 11: Dt ← αD π∗T ∥ Mt + (1 − α)D(πθ ∥ Mt ) ▷ Primary Signal (α) e 12: Dt ← min(Dt , κ) ▷ Divergence Clipping 13: end for 14: for k = 1, . . . , K do 15: Compute ℓkt = log πkT (ŷt | x, ck , ŷ<t ) for t = 1, . . . , T 16: end for T 17: Estimate disagreement from {ℓkt }K ▷ Agreement k=1 and obtain reliability weights {wt }t=1 PT e t=1 mt wt Dt P T t=1 mt wt

18:

L←

19:

20: 21: 22: 23:

▷ Reliability-aware Self-Distillation

t=1

24: 25: 26: 27:

end if if feature matching is enabled then Extract selected student and teacher features ftθ and ft∗ on completion tokens T X Laux ← Laux + mt ∥ftθ − ft∗ ∥22 ▷ Representation Auxiliary Signal t=1

end if 29: L ← L + λaux Laux 30: θ ← θ − η∇θ L 31: if EMA teacher is enabled then 32: θ̄ ← β θ̄ + (1 − β)θ 33: end if 34: end while 28:

▷ Unified Objective ▷ Student Update ▷ EMA Update

Table 3: Estimated resource consumption of UniSD variants per million training tokens. Energy is estimated from wall-clock time using NVIDIA A100 PCIe 80GB TDP (PTDP = 300W), utilization u = 0.7, PUE = 1.2, and carbon intensity 475gCO2 e/kWh. Throughput is reported in million tokens per GPU-hour. All values are estimates for relative comparison, not metered facility-level measurements. Variant

kWh / 1M tok ↓

M tok / GPU-h ↑

Peak Mem. (GB)

2.60 2.56 2.32 3.22 2.74

63.0 59.9 61.7 60.8 55.6

1.58 1.50 1.66

77.2 75.5 75.3

1.48 1.47 1.43

73.3 76.7 73.7

0.96

63.0

Single-teacher stabilizers EMA 0.10 Contrast 0.10 Match (Repr.) 0.11 Match (Joint) 0.08 Clip 0.09 Agreement (Sequence-level) Random 0.16 Retrieval 0.17 Induction 0.16 Agreement (Token-level) Random 0.17 Retrieval 0.17 Induction 0.18 UniSD∗

B.2

0.26

Resource Consumption

As LLM post-training methods become increasingly compute-intensive, accuracy alone is insufficient to characterize their practical trade-offs. Prior work has emphasized that training cost affects not only environmental impact, but also reproducibility and accessibility for researchers with limited compute [47–49]. We therefore complement the wall-clock analysis in Appendix B.1 with estimated resource consumption. Since absolute runtime depends on batching, memory budget, and hyperparameters, we report token-normalized cost: energy per million training tokens and throughput in million tokens per GPU-hour. These metrics capture the compute required by different UniSD variants to generate and score on-policy training tokens. Following the emissions accounting used by CodeCarbon [50] and the MLCO2 Impact Calculator [51], we first estimate energy consumption from runtime and then convert it to CO2 -equivalent emissions using grid carbon intensity. Since facility-level power measurements are unavailable, for each completed training run we compute P · TDP kWh = T · NGPU · · u · PUE, (10) 1000 where T is wall-clock time in hours, NGPU is the number of GPUs, PTDP = 300W is the TDP of an NVIDIA A100 PCIe 80GB GPU, u = 0.7 is the assumed sustained utilization, and PUE = 1.2 is Power Usage Effectiveness. PUE is the ratio between total data-center energy and IT equipment energy, accounting for facility overhead such as cooling and power delivery losses [52]. We then estimate emissions as kgCO2 e = kWh ·

c , 1000

(11)

where c = 475gCO2 e/kWh is the assumed carbon intensity. All values are runtime-derived estimates rather than metered facility measurements, and are used only for relative comparison under fixed assumptions. Token-normalized cost and memory footprint. Table 3 compares the token-normalized cost of UniSD variants. Single-teacher stabilization methods are more efficient. Match (Joint) requires only 0.08 kWh per million tokens, while Contrast, EMA, and Match (Repr.) require 0.10–0.11 kWh per million tokens. 17

SFT

SDFT

Gain over Raw (%)

Qwen2.5-7B

UniSD

GKD

∗

Llama-3.1-8B

Gemma-3-4B

-10 A PP S-E SQ MB Co

o To

l ll A va era PQ HE v G O

l al ol QA al To Ev ver P H G O

A PP S-E SQ MB Co

l al ol QA al To Ev ver P H G O

12.6

14.9

18.2

21.0

21.9

29.4

Random

16.1

26.5

43.2

66.0

103.9

92.0

Retrieval

15.6

26.4

43.0

66.2

101.9

94.0

Induced

17.0

27.0

43.2

62.7

100.5

92.6

Random

13.7

26.6

48.0

66.3

77.8

89.8

Retrieval

16.6

26.5

43.4

66.0

73.5

92.4

Induced

13.1

25.4

46.7

63.1

71.1

92.7

EMA

11.7

18.4

29.0

51.2

82.6

67.6

Contrast

12.3

21.6

37.9

54.8

58.1

70.5

Match (Repr.)

12.3

17.8

27.9

40.9

52.2

65.9

Match (Joint)

12.1

18.4

28.5

42.7

43.0

56.5

Clip

18.0

21.0

27.3

37.5

25.0

31.4

Q-0.5B

Q-1.5B

Q-3B

G-4B

Q-7B

L-8B

100

2.2

2.0

Agreement EMA

Retention PPL

SFT

Training time (min)

Method

Agree (Token)

Agree (Seq.)

Figure 7: Gains over the original model across Qwen2.5, Llama-3.1, and Gemma-3 on ScienceQA (SQA), MBPP, CoS-E, ToolAlpaca (Tool), GPQA, and HumanEval (HEval). UniSD∗ improves 15 out of 18 modeldataset pairs, suggesting that reliability-aware self-distillation generalizes across architectures and task formats.

Clipping Contrastive Matching

1.8 1.6 1.4

1.2 0.5B

1.5B

Model parameters (B)

Base Model

Figure 8: Left. Training time comparison of UniSD variants on ScienceQA. Right. Retention perplexity comparison across UniSD variants. These variants preserve high throughput (2.32–3.22M tokens/GPU-hour), showing that adding representation, contrastive, or temporal stabilization incurs only modest overhead. Agreement-based variants require 0.16–0.18 kWh per million tokens and also increase peak memory by roughly 13–17GB (+21–28%) over single-teacher variants. This overhead is expected: Agreement estimates reliability by re-scoring each on-policy completion under multiple auxiliary contexts, increasing teacher-side forward computation and storing additional prompt–completion tensors, masks, and log-probability buffers. The additional scoring reduces throughput to 1.43–1.66M tokens/GPU-hour, exposing a clear reliability–cost trade-off: Agreement spends more computation and memory to obtain a consistency signal for filtering noisy self-supervision. Implementations with tighter memory budgets can reduce Agreement overhead by scoring auxiliary contexts sequentially rather than jointly.

Additional Experimental Details

Training Configuration Training for all methods uses LoRA [53] (rank 64, alpha 128, dropout 0.05) and AdamW optimizer [54] (β1 = 0.9, β2 = 0.999). Unless otherwise noted, we train for 1 epoch with a 18

Sensitivity to #Contexts (k) & Agreement Weights (γ) Agree (Seq.) (ScienceQA)

Accuracy (%)

Peak: 85.2%

Agreement Weight (γ) γ = 1.00 γ = 0.10

Sensitivity to #Contexts (k) & Agreement Weights (γ) Agree (Seq.) (GPQA)

Peak: 36.2% 35

Agreement Weight (γ) γ = 1.00 γ = 0.10

γ = 0.01

30 3

Number of Contexts (k)

Figure 9: Sensitivity to the number of contexts k and the agreement weight γ. Adding more contexts does not consistently improve accuracy. Category

Dataset

Train

Test

License

Scientific Reasoning

ScienceQA GPQA MBPP HumanEval CoS-E ToolAlpaca

12,726 – 120 – 9,741 4,046

4,241 448 257 164 1,221 68

CC BY-NC-SA 4.0 CC BY 4.0 / MIT CC BY 4.0 MIT BSD-3-Clause Apache-2.0

Coding Commonsense QA Tool Usage

Table 4: Dataset statistics across training and test splits, together with their public licenses. learning rate of 2e-5, cosine decay, 10% warmup, gradient accumulation of 4 steps, and bf16 mixed precision. On-policy completions are generated with vLLM [55] in colocate mode at temperature 0.7. The maximum prompt and completion lengths are 3072 and 1024 tokens, respectively. Evaluation All evaluations use vLLM [55] with greedy decoding (temperature τ = 0.0). We compare UniSD against SFT and state-of-the-art self-distillation baselines, including SDFT [10], GKD [20], SSD [21], and OPSD [22]. For code generation (MBPP and HumanEval), we report pass@1 via sandboxed test execution with a 10-second timeout. For multiple-choice tasks (ScienceQA, CoS-E, GPQA), we report accuracy with automatic answer extraction. For tool use (ToolAlpaca), we report full accuracy defined as exact match on both the action names and all arguments. All experiments are conducted on a server with six NVIDIA A100 80GB GPUs.

Broader Impact

UniSD explores self-distillation as a way for LLMs to improve using supervision derived from their own behavior, rather than relying on stronger external teachers. This may lower the cost and access barriers of post-training, especially for academic groups, smaller organizations, and resource-constrained settings. It can also reduce the need to transmit in-domain data to external models, which makes the approach appealing for privacy-sensitive or local adaptation. Finally, UniSD provides a unified, extensible, reproducible and controllable framework for studying self-distillation.

Sensitivity to #Contexts (k) & Agreement Weights (γ) Agree (Tok.) (ScienceQA)

Sensitivity to #Contexts (k) & Agreement Weights (γ) Agree (Tok.) (GPQA)

Accuracy (%)

Peak: 36.8%

Peak: 84.4%

Agreement Weight (γ) γ = 1.00 γ = 0.10

γ = 0.01

30 3

Number of Contexts (k)

Figure 10: Sensitivity to the number of contexts k and the agreement weight γ. Adding more contexts does not consistently improve accuracy.

Ethical Considerations

Self-distillation inherits the limitations of the underlying base model, including potential factual errors, social biases, and unsafe behaviors. Although UniSD uses reliability weighting, divergence clipping, and stabilization to reduce the reinforcement of unreliable signals, these mechanisms are not substitutes for standard safety procedures. Accordingly, adapted models should be evaluated for safety, bias, factuality, and domain-specific risks before deployment, especially in human-centric applications. Users should obtain base models and benchmark datasets from their original providers and comply with the corresponding licenses, access restrictions, and use policies.

AI Assistants Usage

AI assistants were used as auxiliary tools in preparing this manuscript, primarily for language refinement, clarity, organization, and limited experimental workflows. The experimental design and methodological choices were made by the authors. All results, analyses, and final content were manually checked and verified by the authors.

Limitations and Future Work

This work mainly focuses on single-turn scenarios, which provides a controlled setting for systematically studying and isolating the effects of self-distillation. We view this scope as a starting point for several future directions. Long-Horizon Agentic Settings. A natural extension is to apply UniSD to long-horizon agentic tasks, where success depends on multiple interdependent decisions. These settings introduce sparse and delayed feedback, making them a valuable testbed for studying whether reliability-weighted self-correction can provide stable supervision over extended trajectories. Finer-Grained Trajectory Evaluation. Our evaluation follows standard benchmark protocols that score final answers as correct or incorrect. Future work could develop finer-grained evaluation schemes that credit

Performance Across Sequence-level Agreement Methods MBPP

GPQA

HumanEval 79

ScienceQA

ToolAlpaca

Cos-E Raw Model Random

Retrieval Induced

Figure 11: Comparison of sequence-level agreement across three auxiliary-context strategies: random, retrieval, and induced.

Agree

Contrast EMA Match Component Effectiveness by Dataset

⋆

Clip

UniSD

GPQA*

HumanEval*

Gain over Base Model (%)

ScienceQA

COS-E

MBPP

ToolAlpaca

Figure 12: Per-dataset gains of UniSD variants over the raw Qwen2.5-7B model. Asterisks (∗) denote OOD benchmarks. The results highlight complementary component strengths across tasks, with UniSD∗ achieving the most consistent improvements across in-domain and OOD benchmarks.

Table 5: Comparison of base-distribution retention perplexity across model families. Lower values are better. The best and second-best results for each model are shown in bold and underlined, respectively. Method Raw Model SFT SDFT

Qwen2.5-Instruct

Llama-3.1-Instruct

Gemma-3-IT

0.5B

1.5B

1.71 1.34 1.65

1.91 1.60 2.04

1.41 1.36 1.90

1.14 1.68 1.18

1.23 1.25 1.24

1.27 3.02 1.32

1.51 1.52 1.51

1.11 1.09 1.09

1.21 1.23 1.21

1.33 1.33 1.33

1.33 1.53 1.66 1.49 1.60 1.53 1.46 1.43

1.12 1.12 1.13 1.11 1.10 1.13 1.11 1.10

1.16 1.07 1.15 1.23 1.24 1.19 1.27 1.22

1.34 1.34 1.34 1.33 1.33 1.34 1.32 1.31

Agreement (Token-level) Random 1.64 2.13 Retrieval 1.61 2.06 Induction 1.67 2.14 Agreement (Sequence-level) Random 1.20 1.32 Retrieval 1.48 2.09 Induction 1.22 1.31 EMA 1.63 2.10 Contrast 1.33 1.85 Match (Joint) 2.08 2.09 Match (Repr.) 1.91 1.93 Clip 1.83 1.95

partially correct reasoning or useful intermediate steps, which may better capture the benefits of self-generated supervision beyond final-answer accuracy. Broader Self-Supervision Objectives. UniSD instantiates reliability-aware self-distillation through five complementary mechanisms. The framework is naturally extensible. Promising directions include richer contrastive objectives, alternative disagreement measures across self-derived teacher views, and integration with prompt optimization techniques to improve the quality and diversity of self-derived supervision [56].

Record · ID 168321 · SHA-256 ddce3a5778ca6817
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.