arXiv CS open access

SR-OPSD: Self-Referenced On-Policy Self-Distillation

2026 · arxiv_cs

arXiv CS · Papers · 2026

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

SR-OPSD: Self-Referenced On-Policy Self-Distillation

Zhuo Sun1,2∗ † Entong Li3,∗ Yanlong Zhao4,∗ Xiaoyuan Cheng5,∗ Wenxuan Yuan6 Kaiyu Li5 Che Liu2 Huihang Liu1 Harrison Bo Hua Zhu7,2,8,† Li Zeng9,† 1

Shanghai University of Finance and Economics, 2 Imperial College London, Independent Researcher, 4 University of Science and Technology of China, 5 University College London, 6 Nanyang Technological University, 7 Technical University of Denmark, 8 University of Copenhagen, 9 Peking University

arXiv:2608.09745v1 [cs.LG] 10 Aug 2026

Abstract On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards. However, the self-teacher policy used in OPSD is typically a stop-gradient or exponentialmoving-average copy of the policy conditioned on additional context information, and thus co-evolves with both the student policy and its on-policy context distribution. Directly matching such a moving target with a fixed projection objective can lead to unstable optimization or excessive distributional concentration. This nature of OPSD motivates the proposed Self-Referenced On-Policy Self-Distillation (SR-OPSD). At fixed student-generated contexts, a token-level variational characterization identifies the effective distillation target as a geometric interpolation between the self-teacher policy and a reference policy. Meanwhile, we use the Rényi divergence family to generalize the projection geometry. This formulation separates where the adaptive target is placed from how the student is projected toward it: the interpolation coefficient controls underlying target, while the Rényi order controls the projection geometry and its sensitivity to token-level density ratios. Extensive experiments across scientific evaluation, mathematical reasoning, and coding generation tasks with multiple large language models show that SR-OPSD achieves the state-of-the-art or competitive performance across various settings.

Introduction

Post-training of large language models (LLMs) encompasses supervised instruction tuning, preference optimization, and reinforcement learning [Wei et al., 2022, Sanh et al., 2022, Ouyang et al., 2022, Rafailov et al., 2023]. Among reinforcement-learning methods, RLHF methods commonly optimize a learned reward model using proximal policy optimization (PPO) [Schulman et al., 2017, Ouyang et al., 2022], whereas recent reasoning systems increasingly employ reinforcement learning with verifiable rewards (RLVR), using algorithms such as Group Relative Policy Optimization (GRPO) [Shao et al., 2024] and its variants [Yu et al., 2025]. Methods such as Direct Preference Optimization (DPO) [Rafailov et al., 2023] optimize preference data without explicit reward modeling. In terms of reasoning, RLVR is particularly attractive because the correctness of the outcome can often be assessed automatically through external checks, unit tests, or rule-based verifiers. However, these ∗ Equal Contribution. † Corresponding Author. Correspondence to zhuosunreid@outlook.com, habhuz@dtu.dk,

Preprint.

li.zeng@pku.edu.cn.

𝑦 ∼ 𝜋𝜃𝑘 (⋅ |𝑥) ℎ𝑡 Self-Teacher Policy, 𝑠𝑓

𝑡

𝜋𝜃ഥ 𝑘 (⋅ |ℎ𝑡 )

∗ 𝜋𝛼,𝑘

Probability

EMA self-teacher update ҧ 𝜃𝑘+1 = 𝛽𝜃𝑘ҧ + 1 − 𝛽 𝜃𝑘+1

Student Policy 𝜋𝜃𝑘

Probability

Training prompts (x~𝒟)

𝜋𝜃𝑘 (⋅ |ℎ𝑡 )

Vocabulary probability

Student Context ℎ𝑡

Probability

Raw prompt ℎ𝑡

Self-Teacher Context ℎ𝑡𝑠𝑓 Verified solution Environment feedback Reflection or Successful sibling response 𝑟𝑒𝑓

Reference Context ℎ𝑡

Normalized Geometric Target 𝑠𝑓

𝑠𝑔 𝜋𝜃ഥ𝑘 𝑎 ℎ𝑡

∗ 𝜋𝛼,𝑘 (𝑎) =

(A) On-Policy Context Collection

𝑟𝑒𝑓 1−𝛼

𝜋𝑟𝑒𝑓 𝑎 ℎ𝑡

∗ minD𝜌 (𝜋𝛼,𝑘 ||𝜋𝜃𝑘 ) 𝜃

𝑍𝛼,𝑡

Rényi order (ρ): projection geometry and density-ratio sensitivity

Reference-anchored target Reference, (α = 0)

Self-Teacher, (α = 1)

Reference Policy 𝑟𝑒𝑓 𝜋𝑟𝑒𝑓 (⋅ |ℎ𝑡 )

Reference Policy 𝜋𝑟𝑒𝑓 = 𝑠𝑔[𝜋𝜃0 ]

𝛼

ρ Reference distribution

(B) Self-Referenced Target Construction

Forward computation/ gradient update

(ρ → 1): KL limit

(0 < ρ < 1): power-tempered density ratios

Stop-gradient/ EMA

Figure 1: Illustration of SR-OPSD.

methods typically assigns a scalar-valued reward to an entire response. Although this reward induces policy-gradient updates over the generated tokens, it does not directly identify which intermediate tokens or reasoning steps caused success or failure, resulting in coarse and sparse credit assignment over long sequences. On-policy distillation (OPD) offers an alternative approach by utilizing the dense token-level supervision on trajectories sampled from the student policy [Gu et al., 2023, Agarwal et al., 2024, Zhu et al., 2026, Jin et al., 2026]. A teacher policy evaluates student-generated prefixes, reducing the state-distribution mismatch associated with offline distillation. Existing on-policy distillation methods can be viewed through three interacting design choices including: the divergence objective, the source of supervision, and the mechanism used to stabilize policy optimization. These choices are often designed largely in isolation, which is relatively benign in conventional offline distillation where both the teacher and the training distribution are fixed. This benefit, however, generally requires continued access to an extra teacher policy. On-policy self-distillation (OPSD) instead uses two views of the same evolving model: the policy we aim to optimize is conditioned on the original prompt, whereas the so-called self-teacher policy is the same policy conditioned on feedback, privileged information, or richer contextual signals [Zhao et al., 2026a, Hübotter et al., 2026]. OPSD can therefore convert feedback, reflections, verified traces, or successful responses into dense token-level supervision. Motivation However, in on-policy self-distillation, the policy to be optimized generates its own contexts while a self-teacher policy evolves together with it. Consequently, the target, the context distribution, and the policy being optimized all change throughout training, making these three design choices intrinsically coupled. It is sensitive to the construction of the self-teacher policy, the choice of divergence, the extra context information provided to the self-teacher, and top-K approximation as discussed in [Zhu et al., 2026]. For instance, self-distillation with reverse Kullback–Leibler (KL) divergence may amplify mode-seeking behavior and reduce output diversity [Jin et al., 2026]; as demonstrated in Figure 2 and in Appendix B. This coupling is already evident in the OPSD methods based on the two predominant KL objectives. Forward KL [Zhao et al., 2026a] is mode-covering and takes its action expectation under the self-teacher distribution, whereas reverse KL [Zhao et al., 2026a, Hübotter et al., 2026] is naturally aligned with student sampling but is generally more modeseeking. Hence, in on-policy self-distillation, forward KL may suffer from an action-level expectation mismatch when estimated from student-sampled tokens, while reverse KL can weakly correct modes already underrepresented by the student, potentially reinforcing such biases as the self-teacher coevolves. Existing approaches mitigate parts of these issues through exponential-moving-average (EMA) self-teachers, symmetric divergences [Hübotter et al., 2026], entropy-aware switching [Jin 2

et al., 2026], and regularization-based stabilization [Luo et al., 2026, Yang et al., 2026, Yu et al., 2026]. These motivate an on-policy self-distillation learning objective beyond vanilla forward-KL, reverseKL. We instead seek a formulation that separately controls where the adaptive target is placed and how the student is projected toward it. We therefore propose Self-Referenced On-Policy SelfDistillation (SR-OPSD), illustrated in Figure 1, which offers a new framework with Rényi projection and reference-policy anchoring for on-policy self-distillation. Main Contributions Our main contributions are threefold. (i) We adapt the reference-regularized on-policy self-distillation formulation to frozen-context self-distillation and construct a normalized target from a self-teacher policy and a reference policy. (ii) We optimize this target using Rényi divergence and derive the resulting logit gradient, showing that Rényi order controls the power weighting of self-teacher-to-student probability ratios without changing the unconstrained target distribution. (iii) We evaluate the resulting method across scientific reasoning, mathematical reasoning, and code generation, and open-source the code which supports multiple hardware platforms, including NVIDIA GPUs.

Background

Notation We denote the policy to be optimized by πθ , the self-teacher policy by πθ̄ , and the reference policy by πref . Given a prompt x ∼ D and an on-policy response y = (y1 , . . . , yL ), the student prefix at position t is ht = (x, y<t ). We write hsf t for the corresponding self-feedback or privileged context supplied to the self-teacher and href for the context supplied to the reference policy. t When a statement is token-level and the contexts are fixed, we suppress the position index and write h, hsf , and href . Thus, h, hsf , and href always denote three possibly different fixed contexts. All token-level distributions are defined over the same next-token vocabulary V. If a candidate set Ct ⊆ V is used, all compared distributions are normalized on the same candidate space, preferably with a tail bucket retaining the omitted probability mass. The operator sg[·] denotes stop-gradient. Rényi Divergence For probability distributions p and q with common positive support, the Rényi divergence [Van Erven and Harremos, 2014] of order ρ > 0, ρ ̸= 1, is X 1 Dρ (p∥q) = log p(a)ρ q(a)1−ρ . ρ−1 a∈V

The divergence direction is determined solely by the argument order. We call Dρ (q∥p) the reverse projection when q is the student and p is the self-teacher target, and Dρ (p∥q) the forward projection. Both converge to the corresponding KL as ρ → 1. On-Policy Self-Distillation Given a teacher policy πT , on-policy distillation trains the student on its own sampled prefixes, for example through the reverse-KL loss " L # X E x∼D DKL (πθ (· | ht ) ∥ sg[πT (· | ht )]) . y∼πθold (·|x)

t=1

Here πθold denotes the rollout policy used to collect the current on-policy batch; it coincides with the current student before the optimization step. In on-policy self-distillation, the teacher policy is often replaced by an exponential-moving-average of πθ with additional context information, sg[πθ̄ (· | hsf t )]. The additional context hsf can contain textual feedback, verified solutions, reflections, or successful t sibling responses that are unavailable in the ordinary student context ht . Self-Teacher in OPSD The self-teacher policy πθ̄ may be a stopped copy, a lagged snapshot, or an exponential-moving-average version of the policy to be optimized πθ . We use β ∈ [0, 1) as the exponential moving average coefficient and adopt the convention θ̄k+1 = β θ̄k + (1 − β)θk+1 . Thus, larger β produces a slower-moving self-teacher policy. During each update, θ̄k and the resulting self-teacher policy are held fixed [Hübotter et al., 2026]. 3

Method

Revisiting On-Policy Self-Distillation At each outer iteration, OPSD first samples a batch from the rollout policy πθold and then treats the resulting contexts as fixed while updating the student. A standard reverse-KL frozen-rollout objective is " L # X sf max E x∼D −DKL πθ (· | ht ) sg πθ̄ (· | ht ) . θ

y∼πθold (·|x)

t=1

ref The distinction between the sequence-level contexts (ht , hsf t , ht ) and the fixed-context shorthand (h, hsf , href ) is important: the following characterization is pointwise in fixed contexts and therefore applies directly to the frozen-rollout inner update. Proposition 3.1 (Token-Level Variational Characterization of OPSD). Fix h, hsf , and href , and let α ∈ [0, 1]. Assume that, for every token a ∈P V, πθ (a | h) > 0, sg[πθ̄ (a | hsf )] > α 0, πref (a | href ) > 0. Define Zα (hsf , href ) := a∈V sg[πθ̄ (a | hsf )] πref (a | href )1−α and α

1−α

(a|href ) πα⋆ (a | hsf , href ) := sg[πθ̄ (a|hZsfα)](hsfπref . Then the conditional token-level objective ,href ) sg[πθ̄ (a | hsf )] Jα (θ; h, hsf , href ) := Ea∼πθ (·|h) α log − DKL (πθ (· | h) ∥ πref (· | href )) πref (a | href ) satisfies Jα (θ; h, hsf , href ) = −DKL (πθ (· | h) ∥ πα⋆ (· | hsf , href ))+log Zα (hsf , href ). Consequently, for fixed contexts and frozen target components, maximizing Jα with respect to θ is equivalent to minimizing the reverse KL from the student to πα⋆ . When α = 1, Z1 = 1 and π1⋆ (· | hsf , href ) = sg[πθ̄ (· | hsf )], recovering the token-level objective.

The proposition establishes a conditional variational equivalence, rather than an unconditional equivalence between complete moving-distribution objectives. Although Zα (hsf , href ) is constant in a fixed inner update, its expectation under a context distribution generated by a changing policy need not be constant. The result therefore provides a reference-regularized, trust-region-style interpretation of the practical frozen-rollout on-policy self-distillation update. Self-Referenced On-Policy Self-Distillation Empirically, the combination of a moving target and a mode-seeking reverse KL divergence projection can produce unstable long-budget behavior, as observed in Figure 3, while the associated entropy dynamics are shown in Figure 2. Forward KL and Jensen–Shannon divergence (JSD) have also been considered as alternative divergences for on-policy self-distillation [Zhao et al., 2026a, Hübotter et al., 2026]. At a fixed student-generated prefix, full-vocabulary forward KL, reverse KL, JSD, and Rényi divergence are all well-defined and can be evaluated directly when the corresponding logits are available. The practical difficulty arises when the token objective is approximated using actions sampled from the student or a restricted candidate set. We use the Rényi divergence family to generalize the projection geometry of OPSD. Rényi OPSD does not eliminate expectation computation; instead, it gives a student-expectation density-ratio form whose power is controlled continuously by ρ. The reverse Rényi projection can be written as P P 1 sg[πθ̄ (a|hsf t )] 1−ρ maxθ t −Dρ(πθ (a | ht )∥ sg[πθ̄ (a | hsf ]. t )]) = t 1−ρ log Ea∼πθ (a|ht ) [( πθ (a|ht ) ) The forward Rényi projection is P P 1 sg[πθ̄ (a|hsf t )] ρ maxθ t −Dρ(sg[πθ̄ (a | hsf t )]∥πθ (a | ht )) = t 1−ρ log Ea∼πθ (a|ht ) [( πθ (a|ht ) ) ]. Thus, the two directions share a common likelihood-ratio form but apply different powers to the teacher-to-student ratio. For 0 < ρ < 1, these powers temper extreme ratios and provide a tunable projection geometry. This reformulation concerns the token distribution at an already visited prefix. Proposition 3.2 (Token-Level Variational Characterization of SR-OPSD). Fix h, hsf , and href . Let α ∈ [0, 1] and ρ > 0, ρ ̸= 1, and assume that all token probabilities are strictly positive. Define θ̄ (a|hsf )] the token-level pseudo-reward rα (a; hsf , href ) := α log sg[π πref (a|href ) . Define the Rényi-aggregated toP

(a|h

)ρ π (a|h)1−ρ exp(ρr (a;h ,h

))

α θ sf ref 1 ken functional Rα,ρ (θ; h, hsf , href ) := − ρ−1 log a∈V refP refπθ (a|h) . Let ρπ 1−ρ ref (a|href ) a∈V Jα,ρ (θ; h, hsf , href ) := Rα,ρ (θ; h, hsf , href ) − Dρ (πθ (· | h) ∥ πref (· | href )). Then ρ Jα,ρ (θ; h, hsf , href ) = −Dρ (πα⋆ (· | hsf , href ) ∥ πθ (· | h)) − log Zα (hsf , href ). (1) ρ−1

Algorithm 1 SR-OPSD Require: πθ , πθ̄ , πref 1: for k = 0, 1, . . . do 2: Sample prompts x ∼ D and responses y ∼ πθk (· | x) 3: for t = 1, . . . , |y| do ref 4: Construct ht , hsf t , and ht ⋆ ref 5: Form the frozen target πα,k (· | hsf t , ht ) 6: end for X ⋆ ref 7: Update the policy θk+1 ← Update (θk , −η∇θ Dρ πα,k (· | hsf t , ht ) πθ (· | ht ) t

) θ=θk

8: Update the self-teacher: θ̄k+1 ← β θ̄k + (1 − β)θk+1 9: end for 10: return πθ

Therefore, for fixed contexts and frozen target components, maximizing Jα,ρ with respect to θ is equivalent to minimizing the forward Rényi projection from πα⋆ (· | hsf , href ) to πθ (· | h). See Appendix C.2 for proof. The functional Rα,ρ depends on the policy to be optimized and should be interpreted as a conditional risk-sensitive variational representation, rather than as an environment reward. When α = 1, Z1 = 1 and the proposition reduces to Rényi OPSD with the self-teacher as its target. The variational target suggests explicitly combining the self-teacher policy with a reference ref sf α 1−α policy. At position t, define πα⋆ (a | hsf πref (a | href . A natural fixed t , ht ) ∝ sg πθ̄ (a | ht ) t ) reference is πref := sg[πθ0 ], where θ0 denotes the parameters of the policy before optimization. More generally, the reference policy should provide a useful anchor. The corresponding token-level objective is: max E θ

L X

X 1 [log exp((1 − ρ) log πθ (a | ht ) 1−ρ t=1 a∈V ref + ρα log sg πθ̄ (a | hsf t ) + ρ(1 − α) log πref (a | ht ))].

x∼D y∼πθold (·|x)

(2)

See Algorithm 1 for the pseudo-code, where Update(·) denotes an optimizer such as stochastic gradient descent or Adam. When a candidate-space approximation is used, the same objective is evaluated on a common normalized candidate set, ideally augmented with a tail bucket. The coefficient α controls the location of the self-referenced target: α = 1 recovers the self-teacher, whereas α = 0 recovers the reference policy. In contrast, ρ controls the Rényi projection toward this target. As ρ → 1, the projection recovers the corresponding KL limit. Proposition 3.3 further shows that ρ power-tempers the target-to-student density ratio. Although α and ρ parameterize different components of the objective, their effects interact in the resulting gradient through the product αρ. As shown in Figure 3, the proposed method achieves effective performance throughout training, while Figure 2 shows stable policy-entropy dynamics. Qualitative examples are provided in Appendix B. Proposition 3.3 (Logit Gradient of SR-OPSD). Fix h, hsf , and href , and treat πα⋆ (· | hsf , href ) as frozen during the student update. Let zθ (a | h) denote the student logit such that πθ (a | θ (a|h)) h) = P exp(z For ρ > 0, ρ ̸= 1, define π eα,ρ,θ (a | h, hsf , href ) ∝ πθ (a | exp(zθ (b|h)) . b∈V

Experiments

We evaluate the proposed method across various representative scenarios that capture different feedback regimes and levels of supervision, including: (i) Science Q&A benchmarks without rich environment feedback; (ii) mathematical reasoning on challenging problems with access to groundtruth solutions; and (iii) scaling behavior on coding tasks with rich execution-based feedback; and (iv) ablation study on the effectiveness of each component of the proposed method. All experiments are conducted on NVIDIA H200 and A800 GPUs. 5

4.1

Performance with Bootstrapping on Generated Responses as Feedback

Setting We compare our method against SDPO, on-policy GRPO, which samples trajectories from the current policy, and GRPO, which reweights trajectories from previous policies using importance sampling. For all baselines, we use the same hyperparameter settings as SDPO. We evaluate scientific reasoning on undergraduate-level problems in chemistry, physics, biology, and materials science using the L3 reasoning subset of SciKnowEval [Feng et al., 2024], with a train–test split to assess in-domain generalization. We initialize the policies from Qwen3-8B [Yang et al., 2025] and OLMo3-7B-Instruct [Olmo et al., 2025], and report Avg@16 in Table 1 in terms of wall-clock training time. We also report Maj@16 accuracy and Best@16 accuracy in Figure 4 and Figure 5. All experiments are conducted using the verl framework on NVIDIA H200 GPUs to ensure fair comparisons with Hübotter et al. [2026]. See Table 6 for detailed experimental configurations. Table 1: Performance on SciKnowEval benchmark. We report Avg@16 accuracy. The best and second-best results are shown in bold and underlined, respectively. Chemistry 5h

10h

Qwen3-8B + GRPO + GRPO (on-policy) + SDPO + SR-OPSD (ours)

63.1 51.8 80.5 80.2

41.2 68.3 62.9 80.5 82.8

Olmo3-7B-Instruct + GRPO + GRPO (on-policy) + SDPO + SR-OPSD (ours)

39.1 37.5 79.6 78.7

22.8 45.4 40.3 79.6 79.9

Physics 15h

10h

72.6 68.1 80.5 82.8

62.7 60.8 71.1 80.1

59.2 72.0 62.2 73.4 81.2

53.4 44.4 79.6 79.9

58.9 54.3 67.6 68.1

37.7 61.4 59.1 70.8 72.3

Biology 15h

10h

74.1 62.6 73.4 82.3

35.2 33.8 66.0 58.5

30.8 52.5 35.5 66.0 58.5

62.8 60.9 70.8 74.8

33.1 29.8 51.9 50.1

16.2 38.5 31.6 52.0 51.1

Materials 15h

10h

53.5 36.4 66.0 62.5

74.3 70.4 76.7 77.7

58.9 75.9 73.1 78.3 79.6

42.6 34.9 52.0 51.1

71.4 66.4 77.4 76.2

36.7 73.3 70.7 77.4 76.2

Tool use 15h

10h

15h

79.3 74.8 78.3 80.9

62.6 60.1 63.8 63.6

57.5 66.3 62.5 63.8 64.4

66.3 65.5 63.8 67.5

74.2 73.3 77.4 76.2

56.9 55.3 58.8 58.1

39.3 61.0 57.0 60.4 61.4

61.0 59.6 60.4 62.7

Actor entropy

Avg@16 accuracy

Results Analysis As shown in 2.00 0.80 Table 1, SR-OPSD achieves the 0.70 best 15-hour Avg@16 accuracy in 1.50 seven of the ten cases and outper0.60 1.00 forms both GRPO variants in all 0.50 0.50 ten settings. For Qwen3-8B, SR0.40 OPSD surpasses the strongest com0.00 0 100 200 300 400 0 100 200 300 400 peting baseline by 2.3, 8.2, 1.6, and Training step Training step 1.2 percentage points on Chemistry, Physics, Materials, and Tool Use, Figure 2: Investigation of Entropy of the Policy on Science respectively, while SDPO remains Q&A (Chemistry). Left: Avg@16 accuracy. Right: Training stronger on Biology by 3.5 points. entropy of the policy. Averaged across the five domains, SR-OPSD reaches 75.2 Avg@16 at 15 hours, compared with 72.4 for SDPO and 69.2 for GRPO. On OLMo3-7B-Instruct, SR-OPSD obtains the best results on Chemistry, Physics, and Tool Use, improving over the strongest baseline by 0.3, 4.0, and 1.7 points, respectively. It remains competitive on Biology and Materials, trailing SDPO by only 0.9 and 1.2 points. Its five-domain average reaches 68.9, exceeding SDPO and GRPO by 0.9 and 10.1 points, respectively. Meanwhile, we also investigate the entropy of the policy and corresponding generation quality in Figure 2 and Appendix B. It is found that SR-OPSD also stabilizes the entropy of the policy and has better generation quality than the associated baselines. SR-OPSD SDPO (JSD) GRPO

SDPO (RKL) SDPO (JSD) SR-OPSD (w. ref.) SR-OPSD (wo. ref.)

GRPO (on policy) SDPO (RKL)

In Figure 3, we report Avg@16 validation accuracy over training steps on five reasoning-related benchmarks for Qwen3-8B and Olmo3-7B-Instruct. Compared with the JSD-based SDPO baseline, SR-OPSD generally maintains more stable long-budget performance, whereas SDPO often reaches an early peak and then degrades. These results suggest that reference anchoring and Rényi projection provide a more favorable performance and training stability. 4.2

Performance on Mathematical Reasoning Benchmarks

Setting We then evaluate the performance in a setting where we have access to ground-truth solutions as feedback on very hard mathematical reasoning problems. Following the data construction 6

Chemistry

Physics

Qwen3-8B

Biology

Materials

Tool use

70 65 60 55 50

Olmo3-7B-Instruct

0 80

80 50

70 40

60 30 40

20 0

400

800

Step

400

800

0 0

400

Step

800

Step SR-OPSD

400

800

Step

400

800

Step

SDPO (JSD)

Figure 3: SR-OPSD improves long-budget training stability.

of Agrawal et al. [2026], we use a training collection of 738 particularly challenging mathematics problems. We benchmark our approach against OPSD (forward KL), SDPO (reverse KL), and GRPO on AIME 2024, AIME 2025, HMMT 2025, AMC 2023, and Minerva, with all methods initialized from Qwen3-4B-Instruct-2507. Due to the high computational cost of evaluation, we train all methods for 200 steps and report performance at the final. See Table 6 for detailed experimental configurations. Table 2: Performance on mathematical reasoning benchmarks. The best results are shown in bold, and the second-best results are underlined. Model

Method

Qwen3-4B

Base GRPO SDPO OPSD SR-OPSD (Ours)

AIME24

AIME25

HMMT25

AMC23

Minerva

Avg@64

Pass@64

Avg@64

Pass@64

Avg@64

Pass@64

Avg@64

Pass@64

Avg@64

Pass@64

61.4 58.8 41.3 48.6 59.3

82.0 83.3 80.0 76.7 83.3

50.3 49.2 31.9 37.0 51.8

69.5 73.3 66.7 56.7 80.0

30.3 32.5 19.6 22.3 32.5

48.8 60.0 50.0 46.7 70.0

93.8 88.8 74.5 81.1 91.0

98.8 100.0 97.5 97.5 100.0

43.2 45.9 42.9 45.0 46.2

48.6 59.6 57.7 57.0 58.8

Results Analysis As shown in Table 2, SR-OPSD achieves the strongest overall performance across the five mathematical-reasoning benchmarks. It obtains the best Avg accuracy on AIME25 and Minerva, ties for the best Avg accuracy on HMMT25, and achieves the best or tied-best Pass accuracy on AIME24, AIME25, HMMT25, and AMC23. In particular, compared with GRPO, SR-OPSD improves Pass accuracy by 6.7 points on AIME25 and 10.0 points on HMMT25, while increasing Avg accuracy by 2.6, 2.2, and 0.3 points on AIME25, AMC23, and Minerva, respectively. Averaged uniformly across the five benchmarks, SR-OPSD reaches 56.2 Avg accuracy and 78.4 Pass accuracy, compared with 55.0 and 75.2 for GRPO. The advantage over the self-distillation baselines is more pronounced. SR-OPSD exceeds forward-KL OPSD by 9.4 points in mean Avg accuracy and 11.5 points in mean Pass accuracy, and exceeds reverse-KL SDPO by 14.1 and 8.0 points, respectively. These results suggest that directly matching a self-teacher with KL projection can substantially degrade the performance of the base models in this challenging regime, whereas reference anchoring and Rényi projection better preserve its initial capabilities while incorporating solution feedback. SR-OPSD show the promising performance across these benchmarks, substantially improving AIME25, HMMT25, and Minerva without the severe degradation observed for OPSD and SDPO. 4.3

Performance on Code generation under Model Scaling

Setting We next assess our method on code-generation tasks. Programming provides a representative setting with informative feedback signals, including execution errors and unsuccessful unit 7

tests. Success in this domain depends on effective credit assignment, since the model must locate the precise source of an error and use that information to avoid making the same mistake in subsequent attempts. Following the protocol of [Hübotter et al., 2026], we focus our evaluation on the LCBv6 subset. We use the Qwen3 model family for this investigation and train all methods for 120 iterations. See Table 7 for detailed experimental configurations. Table 3: Performance on LiveCodeBench v6 across Qwen3 model scales. The best result in each column is shown in bold, and the second-best result is underlined. Method Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Base GRPO SDPO SR-OPSD

9.2 18.5 18.7 18.0

14.7 29.3 30.7 32.0

23.1 39.9 44.7 46.2

27.9 41.2 48.8 50.1

Results Analysis As shown in Table 3, all methods improve over their corresponding base models across the four model scales. SR-OPSD achieves the best performance on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, reaching 32.0, 46.2, and 50.1, respectively. Compared with GRPO, these results correspond to improvements of 2.7, 6.3, and 8.9 percentage points, while the gains over SDPO are 1.3, 1.5, and 1.3 points. On Qwen3-0.6B, SR-OPSD remains competitive but trails GRPO and SDPO by 0.5 and 0.7 points, respectively. One potential reason for the slightly degraded performance on the 0.6B model is that we did not tune the hyperparameters due to the expensive computational cost. Nevertheless, it still nearly doubles the base-model pass-all success rate, improving it from 9.2 to 18.0. Moreover, except at the smallest scale, SR-OPSD outperforms SDPO as model scale increases.

4.4

Ablation Study

Setting We conduct a con- Table 4: Ablation studies on SciKnowEval Physics. The best result in trolled ablation to examine each column is shown in bold. the effects of self-reference, divergence choice, and the Optimization Objective Self-reference Rényi Avg@16 Rényi order ρ. All variJSD × × 79.4 ants use the same model JSD ✓ × 77.2 initialization, training data, Forward KL × × 79.1 optimization budget, and Forward KL ✓ × 76.3 evaluation protocol as the Forward Rényi (ρ = 0.95) × ✓ 77.7 main experiment, and perSR-OPSD (ρ = 0.50) ✓ ✓ 79.9 formance is measured usSR-OPSD (ρ = 0.70) ✓ ✓ 80.6 ing Avg@16 accuracy. We SR-OPSD (ρ = 0.95) ✓ ✓ 81.1 first compare two commonly used on-policy selfdistillation objectives, JSD [Hübotter et al., 2026] and forward KL [Zhao et al., 2026a], both with and without the reference-anchored target. We then evaluate the complete SR-OPSD objective, which combines self-reference with Rényi projection, tuning ρ ∈ {0.50, 0.70, 0.95}. Results Analysis Table 4 shows that self-reference does not improve performance when it is combined with the standard JSD or forward-KL objectives. Adding self-reference reduces Avg@16 from 79.4 to 77.2 for JSD and from 79.1 to 76.3 for forward KL, suggesting that target anchoring alone is insufficient and may even be detrimental when paired with an unsuitable projection geometry. By contrast, SR-OPSD, which combines self-reference with Rényi projection, achieves the strongest performance. Its Avg@16 score increases from 79.9 at ρ = 0.50 to 80.6 at ρ = 0.70 and 81.1 at ρ = 0.95, outperforming the best non-Rényi variant by 1.9 points. These results indicate that the benefit of self-reference depends critically on the divergence used to approach the anchored target, and that tuning ρ provides an effective mechanism for controlling this projection. 8

Related Work

On-Policy Distillation On-policy distillation (OPD) uses student-generated trajectories for training, allowing the teacher to provide distributional supervision on prefixes visited by the student Agarwal et al. [2024], Wen et al. [2023], Gu et al. [2023], Lu and Lab [2025], Jin et al. [2026], Luo et al. [2026], Yang et al. [2026], Stein et al. [2026], Oh et al. [2026], Kim et al. [2026]. Gu et al. [2023] optimizes sequence-level reverse KL through policy-gradient estimators, while Agarwal et al. [2024] evaluates forward KL, reverse KL, and JSD on student rollouts. Later work studies entropy-dependent divergence selection, reference-policy constraints, and rollout mixing to improve stability [Jin et al., 2026, Luo et al., 2026]. Yang et al. [2026] further formulates external-teacher OPD as KL-regularized policy optimization, where a teacher-to-reference log-probability ratio defines the dense reward and controls interpolation or extrapolation between the two policies. See [Song and Zheng, 2026] for a detailed review. On-Policy Self-Distillation OPSD replaces the external teacher with a self-teacher derived from the same underlying model and conditioned on training-only privileged or auxiliary context. Existing methods use ground-truth solutions, environment feedback, successful sibling responses, or document context to construct the self-teacher [Zhao et al., 2026a, Hübotter et al., 2026, Penaloza et al., 2026, Ye et al., 2026, Sang et al., 2026, Zhao et al., 2026b, Wang et al., 2026]. Subsequent work controls the amount of additional information exposed to the teacher, restricts distillation to selected spans, or gates token-level supervision according to teacher reliability and uncertainty [Han et al., 2026, Wang et al., 2026, Liu et al., 2026, Ke et al., 2026]. Other studies analyze the effects of additional context information, Top-K approximation, and update schemes of self-teacher policies [Zhu et al., 2026, Guo et al., 2026]. Existing objectives mainly use forward KL, reverse KL, or JSD for direct distribution matching [Zhao et al., 2026a, Hübotter et al., 2026], while preference-based self-distillation replaces direct matching with reward-regularized preference optimization [Yu et al., 2026]. In this work, we regard the additional context information and the construction of the self-teacher policies as given, and focus on the target distribution and divergence geometry used for matching at fixed on-policy contexts. Specifically, we combine the self-teacher policy and the reference policy into the normalized geometric target. This log-linear teacher–reference target is closely related to the target induced by reward-scaled KL-regularized OPD [Yang et al., 2026]. The coefficient α controls interpolation, or extrapolation when permitted, along the log-density path between the reference policy and the self-teacher. We then project the student toward πα⋆ using a Rényi divergence of order ρ.

Conclusion

In this work, we propose SR-OPSD, which constructs a reference-anchored self-teacher target and optimizes it using a R’enyi projection. Our results highlight that the effectiveness of self-reference depends critically on the objective through which it is imposed. Reference anchoring alone is not sufficient, and self-reference can even degrade performance under standard forward-KL or JSD objectives. In contrast, coupling the self-referenced target with an appropriate Rényi projection improves performance and training stability across scientific reasoning, mathematical reasoning, and code generation. These results suggest that feedback in on-policy self-distillation is most effective when both its influence on the target distribution and the policy’s response to that target are explicitly controlled, rather than treating feedback as an unconstrained imitation signal. Several limitations remain. Our theoretical characterization concerns token-level optimization at fixed rollout contexts and does not establish global convergence of the fully coupled student–self-teacher dynamics. In particular, context-distribution shift remains an additional source of error that must be controlled separately. Moreover, the target-interpolation coefficient, Rényi order, and teacher update rate are fixed and selected empirically. Future work could study adaptive choices of these quantities based on teacher reliability or policy drift, develop tighter guarantees for the evolving on-policy dynamics, and evaluate SR-OPSD under noisier feedback and at larger model scales.

References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, 2024. Rishabh Agrawal, Jacob Fein-Ashley, and Paria Rashidinejad. Reinforcement learning from rich feedback with distributional DAgger. In Continual Reinforcement Learning Workshop at RLC 2026, 2026. URL https://openreview.net/forum?id=8gMfue2BTd. Kehua Feng, Xinyi Shen, Weijie Wang, Xiang Zhuang, Yuqi Tang, Qiang Zhang, and Keyan Ding. Sciknoweval: Evaluating multi-level scientific knowledge of large language models. arXiv preprint arXiv:2406.09098, 2024. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: On-policy distillation of large language models. arXiv preprint arXiv:2306.08543, 2023. Haowei Guo, Baolong Bi, Ruicheng Zhang, Bingqian Sun, and Wentao Zhang. When should the teacher move? temporal coupling and stability in self on-policy distillation. arXiv preprint arXiv:2606.03532, 2026. Zihao Han, Tiangang Zhang, Huaibin Wang, and Yilun Sun. Adaptive teacher exposure for selfdistillation in llm reasoning. arXiv preprint arXiv:2605.11458, 2026. URL https://arxiv.org/ abs/2605.11458. Jonas Hübotter, Frederike Lübeck, Lejs Deen Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=QkfkxyRizZ. Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079, 2026. Junlong Ke, Zichen Wen, Weijia Li, Conghui He, and Linfeng Zhang. Respecting self-uncertainty in on-policy self-distillation for efficient llm reasoning. arXiv preprint arXiv:2605.13255, 2026. URL https://arxiv.org/abs/2605.13255. Yeongmin Kim, Donghyeok Shin, Mina Kang, Byeonghu Na, and Il-Chul Moon. Distillation of large language models via concrete score matching. In International Conference on Learning Representations, volume 2026, pages 16649–16682, 2026. Xiaogeng Liu, Xinyan Wang, Yingzi Ma, Yechao Zhang, and Chaowei Xiao. When are teacher tokens reliable? position-weighted on-policy self-distillation for reasoning. arXiv preprint arXiv:2605.21606, 2026. Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. Feng Luo, Yu-Neng Chuang, Guanchu Wang, Zicheng Xu, Xiaotian Han, Tianyi Zhang, and Vladimir Braverman. Demystifying opd: Length inflation and stabilization strategies for large language models. arXiv preprint arXiv:2604.08527, 2026. Minjae Oh, Sangjun Song, Gyubin Choi, Yunho Choi, and Yohan Jo. Kl for a kl: On-policy distillation with control variate baseline. arXiv preprint arXiv:2605.07865, 2026. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3. arXiv preprint arXiv:2512.13961, 2025. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In 10

S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc., 2022. doi: 10.52202/068431-2011. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf. Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models. arXiv preprint arXiv:2602.04942, 2026. URL https://arxiv.org/abs/2602.04942. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. Crisp: Compressed reasoning via iterative self-policy distillation. arXiv preprint arXiv:2603.05433, 2026. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626, 2026. Alex Stein, Furong Huang, and Tom Goldstein. GATES: Self-distillation under privileged context with consensus gating. arXiv preprint arXiv:2602.20574, 2026. URL https://arxiv.org/abs/ 2602.20574. Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE transactions on information theory, 60(7):3797–3820, 2014. Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Zheng Pan, Xin Li, and Lan-Zhe Guo. TRACE: Distilling where it matters via token-routed self on-policy alignment. arXiv preprint arXiv:2605.10194, 2026. URL https://arxiv.org/abs/2605.10194. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id= gEZrGCozdqR. Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10817–10834, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.605. URL https://aclanthology.org/2023. acl-long.605/. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. 11

Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026. Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. arXiv preprint arXiv:2602.12275, 2026. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Yonghui Wu, and Mingxuan Wang. DAPO: An open-source LLM reinforcement learning system at scale. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/ forum?id=2a36EMSSTp. Xin Yu, Liuchen Liao, Yiwen Zhang, Yingchen Yu, Lingzhou Xue, and Qinzhen Guo. Preferencebased self-distillation: Beyond kl matching via reward regularization. arXiv preprint arXiv:2605.05040, 2026. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. In Forty-third International Conference on Machine Learning, 2026a. URL https://openreview.net/forum? id=Jpxfof0EaS. Zhengyang Zhao, Lu Ma, and Wentao Zhang. Training with harnesses: On-policy harness selfdistillation for complex reasoning. arXiv preprint arXiv:2605.08741, 2026b. Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi, and Ge Liu. The many faces of on-policy distillation: Pitfalls, mechanisms, and fixes. arXiv preprint arXiv:2605.11182, 2026. URL https://arxiv. org/abs/2605.11182.

Notation Table Table 5: Summary of the notation. Symbol

Meaning

x, D y, yt , y<t a, V, Ct ht ref hsf t , ht h, hsf , href πθ , θ πθ̄ , θ̄ πθold πref θ0 πT sg[·] ⋆ πα , Zα α, ρ DKL , Dρ β

Prompt and prompt distribution, with x ∼ D. On-policy response, token at position t, and prefix before t. Generic next-token action, full vocabulary, and candidate set at position t. Student prefix context, ht = (x, y<t ). Context information for the self-teacher policy and reference policy at position t, respectively. Fixed-context shorthand obtained by suppressing the position index t. Policy to be optimized (student policy) and its parameters. Frozen or EMA self-teacher policy and its parameters. Rollout policy used to sample the current on-policy batch. Reference policy. Parameters of the student policy before optimization. External teacher policy, when on-policy distillation is discussed. Stop-gradient operator. Reference-anchored self-teacher target and its normalizing constant. Target-interpolation coefficient and Rényi order. KL divergence and Rényi divergence. EMA coefficient.

Qualitative Comparison of Generations

We select representative cases from SciKnowEval that expose recurrent differences in reasoning stability rather than estimate their population frequency. All compared models are Qwen3-8B checkpoints post-trained on SciKnowEval using the corresponding methods. The model post-trained with SR-OPSD generally gives shorter, more direct explanations while preserving the connection between intermediate reasoning and the selected option. Models post-trained with the other methods more often extend an initially valid argument until they introduce a contradiction, factual error, or answer-option drift. The model post-trained with SDPO (RKL) exhibits a different failure mode characterized by terse, repetitive, or generic reasoning. In the examples below, red text marks the first substantive departure from a valid derivation or from the correct answer-option mapping. CASE 1

Math

Answer-Binding Failure After Correct Calculation Prompt. An item costs $89. It is discounted by 30%, then a 12% sales tax is applied. What is the final price? Choices. A. $99.68; B. $69.78; C. $62.30; D. $71. Gold. B Method

Response Excerpt

Quality

SR-OPSD (w. ref.)

A 30% discount reduces $89 to $62.30. Applying 12% tax gives $62.30 + $7.48 = $69.78. ✓ Correct Answer: B

SR-OPSD (wo. ref.)

The final price is approximately $69.78. However, Option B is not sufficiently aligned × Answer drift with the calculation, and only Option D accounts for both discount and tax. Answer: D

SDPO (RKL)

The calculation gives $62.3 * (1 + 0.12) = $69.78, but the final option changes. Answer: D

× Binding failure

Finding. Both models reached the correct value but failed to bind it to the correct option; model from SR-OPSD (w. ref.) preserved calculation–answer alignment. CASE 2

Physics

Nuclear-Decay Reasoning Stability Prompt. An element emits one alpha particle, followed by two beta particles. How does its atomic number change? Choices. A. zero; B. −1; C. −2; D. +1. Gold. A Method

Response Excerpt

Quality

SR-OPSD (w. ref.)

Alpha emission reduces the atomic number by 2; two beta emissions increase it by 2. The ✓ Faithful net change is −2 + 2 = 0. Answer: A

SR-OPSD (wo. ref.)

The response states the correct local rules but claims the total effect is not zero and selects Option D. Answer: D

SDPO (RKL)

It incorrectly states that alpha emission decreases the atomic number by 1 and retains × Rule collapse the wrong conclusion. Answer: D

× Conclusion drift

Finding. Without self-reference, the conclusion drifts despite correct local rules; reverse KL additionally corrupts the underlying decay rule.

CASE 3

Anatomy

Self-Contradiction in Anatomical Knowledge Prompt. In the spinal cord, motor neuron cell bodies are located in which lamina? Choices. A. I; B. II; C. V; D. IX. Gold. D Method

Response Excerpt

SR-OPSD (w. ref.)

Motor neuron cell bodies are located in lamina IX, within the ventral horn responsible for ✓ Stable voluntary movement. Answer: D

Quality

SR-OPSD (wo. ref.)

The response identifies the ventral horn but then claims lamina IX is not part of the motor system and selects lamina I. Answer: A

SDPO (RKL)

It gives a generic statement about spinal laminae and claims motor neurons are located × Hallucination in lamina II. Answer: B

× Contradiction

Finding. SR-OPSD (w. ref.) preserves the correct anatomical entity; the alternatives contradict it or replace it with a generic claim. CASE 4

Nutrition

Repetition and Incorrect Option Selection Prompt. How many kcal are there in one gram of ethanol?

Choices. A. 5.8; B. 4.1; C. 7.1; D. 8.1 kcal/g.

Gold. C

Method

Response Excerpt

Quality

SR-OPSD (w. ref.)

Ethanol provides approximately 7.1 kcal per gram, matching Option C. Answer: C

✓ Correct

SR-OPSD (wo. ref.)

It claims Option A (5.8 kcal/g) is the most scientifically accurate choice and that Option C overestimates the energy. Answer: A

× Plausible error

SDPO (RKL)

It repeats the clue 29.7 kJ per gram but does not convert it consistently. Answer: B

× Repetitive mismatch

Finding. The alternatives either prefer a plausible wrong value or repeat the correct clue without binding it to the final answer. CASE 5

Chemistry

Template Collapse on a Complex Scientific Question Prompt. What is the molar weight of a complex peptide-like molecule specified by a long IUPAC name? Choices. A. 1517.700; B. 1589.900; C. 1536.800; D. 1490.700. Gold. C Method

Response Excerpt

Quality

SR-OPSD (w. ref.)

Accounting for all atoms gives the molar weight represented by Option C (1536.800). Answer: C

✓ Correct

SR-OPSD (wo. ref.)

It discusses the functional groups but concludes that Option A is most accurate and Option C is too high. Answer: A

× Option drift

SDPO (RKL)

It begins with atomic-weight summation but drifts to the irrelevant claim that the groups × Template collapse indicate a hydrophilic molecule. Answer: D

Finding. SR-OPSD (w. ref.) remains aligned with molar-weight estimation; the alternatives drift to a nearby option or a generic chemistry template.

Theoretical Results and Proofs

C.1

Proof of Proposition 3.1

Proof. Fix h, hsf , and href , and abbreviate q(a) := πθ (a | h),

p(a) := sg[πθ̄ (a | hsf )] ,

r(a) := πref (a | href ).

The conditional objective expands as X p(a) q(a) Jα = q(a) α log − log r(a) r(a) a X = q(a) [− log q(a) + α log p(a) + (1 − α) log r(a)] . a

By definition, log πα⋆ (a | hsf , href ) = α log p(a) + (1 − α) log r(a) − log Zα (hsf , href ). Substituting this identity yields X Jα = q(a) [− log q(a) + log πα⋆ (a | hsf , href ) + log Zα (hsf , href )] a

= −DKL (πθ (· | h) ∥ πα⋆ (· | hsf , href )) + log Zα (hsf , href ), P where the last step uses a q(a) = 1. Since the contexts and target components are fixed, Zα is constant with respect to the inner optimization variable θ. When α = 1, Z1 (hsf , href ) =

p(a) = 1

and π1⋆ (a | hsf , href ) = p(a) = sg[πθ̄ (a | hsf )] . The objective therefore reduces to the reverse-KL OPSD objective. The constancy statement is conditional on the fixed contexts: after averaging over a context distribution that itself changes with the policy, the expected normalizer need not be constant. C.2

Proof of Proposition 3.2

Proof. Fix h, hsf , and href . To simplify the notation, define q(a) := πθ (a | h), p(a) := sg[πθ̄ (a | hsf )] , r(a) := πref (a | href ). Because all token probabilities are strictly positive, all logarithms, probability ratios, and real-valued powers appearing below are well-defined. Under these abbreviations, the normalizing constant is X Zα := Zα (hsf , href ) = p(a)α r(a)1−α . a∈V

The normalized self-referenced target satisfies πα⋆ (a | hsf , href ) =

p(a)α r(a)1−α . Zα

Multiplying both sides of above equation by Zα gives p(a)α r(a)1−α = Zα πα⋆ (a | hsf , href ). 16

(3)

We first simplify the numerator of Rα,ρ . By the definition of the pseudo-reward rα , rα (a; hsf , href ) = α log

p(a) . r(a)

Multiplying both sides by ρ yields ρrα (a; hsf , href ) = ρα log

p(a) . r(a)

(4)

Exponentiating both sides of Equation (4) gives p(a) exp(ρrα (a; hsf , href )) = exp ρα log r(a) ρα p(a) = r(a) = p(a)ρα r(a)−ρα .

(5)

Consider one summand in the numerator of Rα,ρ . Using Equation (5), we obtain r(a)ρ q(a)1−ρ exp(ρrα (a; hsf , href )) = r(a)ρ q(a)1−ρ p(a)ρα r(a)−ρα .

(6)

Combining the two powers of r(a) gives r(a)ρ r(a)−ρα = r(a)ρ−ρα = r(a)ρ(1−α) .

(7)

Substituting Equation (7) into Equation (6) yields r(a)ρ q(a)1−ρ exp(ρrα (a; hsf , href )) = q(a)1−ρ p(a)ρα r(a)ρ(1−α) .

(8)

Therefore, Equation (8) becomes r(a)ρ q(a)1−ρ exp(ρrα (a; hsf , href )) ρ = q(a)1−ρ p(a)α r(a)1−α .

(9)

Applying Equation (3) to Equation (9) gives r(a)ρ q(a)1−ρ exp(ρrα (a; hsf , href )) ρ

= q(a)1−ρ [Zα πα⋆ (a | hsf , href )]

= Zαρ πα⋆ (a | hsf , href )ρ q(a)1−ρ .

(10)

Summing both sides of Equation (10) over a ∈ V gives X r(a)ρ q(a)1−ρ exp(ρrα (a; hsf , href )) a∈V

Zαρ πα⋆ (a | hsf , href )ρ q(a)1−ρ .

(11)

a∈V

Because Zα does not depend on the summation variable a, it can be taken outside the sum: X r(a)ρ q(a)1−ρ exp(ρrα (a; hsf , href )) a∈V

= Zαρ

πα⋆ (a | hsf , href )ρ q(a)1−ρ .

(12)

a∈V

By the definition of Rényi divergence, Dρ (πα⋆ (· | hsf , href ) ∥ q) X 1 = log πα⋆ (a | hsf , href )ρ q(a)1−ρ . ρ−1 a∈V

(13)

Multiplying both sides of Equation (13) by ρ − 1 gives (ρ − 1)Dρ (πα⋆ (· | hsf , href ) ∥ q) X = log πα⋆ (a | hsf , href )ρ q(a)1−ρ .

(14)

a∈V

Exponentiating both sides of Equation (14) gives exp((ρ − 1)Dρ (πα⋆ (· | hsf , href ) ∥ q)) X = πα⋆ (a | hsf , href )ρ q(a)1−ρ .

(15)

a∈V

Substituting Equation (15) into Equation (12) yields X r(a)ρ q(a)1−ρ exp(ρrα (a; hsf , href )) a∈V

= Zαρ exp((ρ − 1)Dρ (πα⋆ (· | hsf , href ) ∥ q)) .

(16)

We next simplify the denominator of Rα,ρ . By the definition of Rényi divergence, X 1 log q(a)ρ r(a)1−ρ . Dρ (q∥r) = ρ−1

(17)

a∈V

Multiplying both sides of Equation (17) by ρ − 1 gives X (ρ − 1)Dρ (q∥r) = log q(a)ρ r(a)1−ρ .

(18)

a∈V

Exponentiating both sides of Equation (18) gives exp((ρ − 1)Dρ (q∥r)) =

q(a)ρ r(a)1−ρ .

(19)

a∈V

Using Equations (16) and (19) in the definition of Rα,ρ gives Rα,ρ (θ; h, hsf , href ) Z ρ exp((ρ − 1)Dρ (πα⋆ (· | hsf , href ) ∥ q)) 1 log α . =− ρ−1 exp((ρ − 1)Dρ (q∥r))

(20)

We then obtain Rα,ρ (θ; h, hsf , href ) " 1 =− log(Zαρ exp((ρ − 1)Dρ (πα⋆ (· | hsf , href ) ∥ q))) ρ−1 # − log(exp((ρ − 1)Dρ (q∥r))) .

(21)

We then have " # 1 Rα,ρ (θ; h, hsf , href ) = − log Zαρ + log(exp((ρ − 1)Dρ (πα⋆ (· | hsf , href ) ∥ q))) − log(exp((ρ − 1)Dρ (q∥r))) . ρ−1 We then have " # 1 Rα,ρ (θ; h, hsf , href ) = − ρ log Zα + (ρ − 1)Dρ (πα⋆ (· | hsf , href ) ∥ q) − (ρ − 1)Dρ (q∥r) . ρ−1 Then, we obtain Rα,ρ (θ; h, hsf , href ) = −

ρ log Zα − Dρ (πα⋆ (· | hsf , href ) ∥ q) + Dρ (q∥r). ρ−1 18

By definition, Jα,ρ (θ; h, hsf , href ) = Rα,ρ (θ; h, hsf , href ) − Dρ (q∥r). The Dρ (q∥r) terms cancel, so we have Jα,ρ (θ; h, hsf , href ) = −Dρ (πα⋆ (· | hsf , href ) ∥ q) −

ρ log Zα . ρ−1

Finally, restoring q = πθ (· | h) and Zα = Zα (hsf , href ), we obtain Jα,ρ (θ; h, hsf , href ) = −Dρ (πα⋆ (· | hsf , href ) ∥ πθ (· | h)) −

ρ log Zα (hsf , href ). ρ−1

For fixed h, hsf , and href , both p(a) and r(a) are frozen during the inner student update. Therefore, X Zα (hsf , href ) = p(a)α r(a)1−α a∈V

does not depend on the inner optimization variable θ. Consequently, ρ − log Zα (hsf , href ) ρ−1 is constant with respect to θ. It follows that argmax Jα,ρ (θ; h, hsf , href ) = argmax [−Dρ (πα⋆ (· | hsf , href ) ∥ πθ (· | h))] θ

= argmin Dρ (πα⋆ (· | hsf , href ) ∥ πθ (· | h)) . θ

This proves the stated conditional variational equivalence. When α = 1, Z1 (hsf , href ) =

p(a) = 1,

a∈V

and π1⋆ (a | hsf , href ) = sg[πθ̄ (a | hsf )] . Hence, J1,ρ (θ; h, hsf , href ) = −Dρ (sg[πθ̄ (· | hsf )] ∥ πθ (· | h)) .

C.3

Proof of Proposition 3.3

Proof. By the definition of Rényi divergence, Dρ (πα⋆ (· | hsf , href ) ∥ πθ (· | h)) X 1 = log πα⋆ (b | hsf , href )ρ πθ (b | h)1−ρ . ρ−1

(22)

b∈V

Define Sα,ρ,θ :=

πα⋆ (b | hsf , href )ρ πθ (b | h)1−ρ .

b∈V

(23)

Then Dρ =

1 log Sα,ρ,θ . ρ−1

We first compute the derivative of the student probability with respect to the student logit. By the softmax definition, exp(zθ (b | h)) . c∈V exp(zθ (c | h))

πθ (b | h) = P Therefore,

∂πθ (b | h) 1{a = b} exp(zθ (b | h)) P = ∂zθ (a | h) c exp(zθ (c | h)) exp(zθ (b | h)) exp(zθ (a | h)) − P 2 ( c exp(zθ (c | h))) = πθ (b | h) [1{a = b} − πθ (a | h)] .

(24)

Since πα⋆ is frozen during the student update, only πθ depends on zθ . Hence, X ∂Sα,ρ,θ ∂πθ (b | h)1−ρ = πα⋆ (b | hsf , href )ρ ∂zθ (a | h) ∂zθ (a | h) b∈V

= (1 − ρ)

πα⋆ (b | hsf , href )ρ πθ (b | h)−ρ

b∈V

∂πθ (b | h) . ∂zθ (a | h)

Substituting Equation (24) gives X ∂Sα,ρ,θ = (1 − ρ) πα⋆ (b | hsf , href )ρ πθ (b | h)1−ρ ∂zθ (a | h) b∈V

× [1{a = b} − πθ (a | h)] . Separating the two terms, ∂Sα,ρ,θ = (1 − ρ)πα⋆ (a | hsf , href )ρ πθ (a | h)1−ρ ∂zθ (a | h) X − (1 − ρ)πθ (a | h) πα⋆ (b | hsf , href )ρ πθ (b | h)1−ρ b∈V

1−ρ = −1, ρ−1

(25)

Finally, by definition, α

πα⋆ (a | hsf , href ) =

sg[πθ̄ (a | hsf )] πref (a | href )1−α . Zα (hsf , href )

Raising both sides to the power ρ gives αρ

πα⋆ (a | hsf , href )ρ =

sg[πθ̄ (a | hsf )] πref (a | href )(1−α)ρ . Zα (hsf , href )ρ

Then, the common factor Zα (hsf , href )−ρ appears in both the numerator and denominator and therefore cancels. Hence, αρ

π eα,ρ,θ (a | h, hsf , href ) ∝ sg[πθ̄ (a | hsf )]

× πref (a | href )(1−α)ρ πθ (a | h)1−ρ .

Extra Experimental Details

D.1

Extra Results for Science Q&A Chemistry

Physics

Qwen3-8B

Biology 60

80 70

Materials

Tool use

60 40

Olmo3-7B-Instruct

400

800

400

800

400

800

0 0

400

800

50 70 60

50 800

400

800

40 400

800

40 60

400

50 0

Step

400

800

400

Step

800

0 0

Step SR-OPSD

400

800

Step

SDPO (JSD)

Figure 4: Maj@16 validation accuracy over training. We report Maj@16 validation accuracy across training steps on five reasoning-related benchmarks for Qwen3-8B and Olmo3-7B-Instruct.

Qwen3-8B

Chemistry

Physics

100

Biology

Materials

60 80

Olmo3-7B-Instruct

400

800

400

800

70 400

800

400

800 80

400

800

400

800

0 70

20 0

Tool use

70 40

70 0

400

Step

800

400

800

400

Step

800

Step SR-OPSD

400

Step

800

Step

SDPO (JSD)

Figure 5: Best@16 validation accuracy over training. We report Best@16 validation accuracy across training steps on five reasoning-related benchmarks for Qwen3-8B and Olmo3-7B-Instruct. D.2

Experimental Configurations

Table 6: Hyperparameters for the Science Q&A and mathematical-reasoning experiments. Parameter

SDPO (JSD) Science Q&A

SR-OPSD (Ours) Science Q&A

General Benchmark Model Primary evaluation metrics Evaluation thinking mode

SciKnowEval Qwen3-8B Avg@16 Disabled

Data, rollout, and evaluation Maximum prompt length Maximum training response length Question batch / PPO mini-batch size Per-device batch / gradient accumulation Training rollouts per question Training temperature / top-p / top-k Evaluation samples per question Evaluation temperature / top-p / top-k

2048 8192 32 / 32 – 8 1.0 / 1.0 / −1 16 0.6 / 0.95 / −1

– 16384 – 1/1 1 0.7 / 0.95 / 20 64 0.7 / 0.95 / 20

Self-distillation objective Projection objective Rényi order Self-reference coefficient Frozen-reference anchoring Reference policy Top-K distillation / tail bucket Teacher EMA update rate Distillation IS clip Token-loss clip

Jensen–Shannon – – No – 100 / Yes 0.05 2.0 –

Forward Rényi 0.95 0.90 Yes Frozen initial policy 100 / Yes 0.05 2.0 –

Reverse KL – – No – 100 / No 0.05 – 0.05

Forward Rényi 0.95 0.90 Yes Frozen initial policy 100 / No 0.05 – 0.05

AdamW 1 × 10−5 constant / 10 0.01 1.0

AdamW 5 × 10−6 linear / 0 0 0.1

Optimization Optimizer Learning rate Learning-rate schedule / warmup steps Weight decay Gradient clipping norm

SDPO (RKL) Mathematics

SR-OPSD (Ours) Mathematics

AIME24, AIME25, HMMT25, AMC23, Minerva Qwen3-4B-Instruct-2507 Qwen3-4B-Instruct-2507 Avg@64 and Pass@64 Avg@64 and Pass@64 Enabled Enabled

Table 7: Hyperparameters for the Qwen3 model-scaling experiments on LiveCodeBench v6. Training uses eight rollouts per prompt, whereas validation uses 16 sampled solutions per problem. Parameter

GRPO

SDPO

SR-OPSD (Ours)

General Models Benchmark Training and evaluation thinking mode

Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B LiveCodeBench v6 Disabled Disabled Disabled

Data, rollout, and validation Maximum prompt / response length Question batch / PPO mini-batch size Training rollouts per question Training temperature / top-p / top-k Validation frequency / samples per question Validation temperature / top-p / top-k

2048 / 8192 32 / 32 8 1.0 / 1.0 / −1 5 / 16 0.6 / 0.95 / −1

Method-specific objective Policy/distillation objective Environment feedback in reprompting Rényi order Self-reference coefficient Frozen-reference anchoring Reference policy Top-K distillation / tail bucket Teacher EMA update rate Rollout IS correction / distillation IS clip

GRPO – – – – – – – token / –

Reverse KL Yes – – No – 100 / Yes 0.05 token / 2.0

Forward Rényi Yes 0.95 0.90 Yes Frozen initial policy 100 / Yes 0.05 token / 2.0

Optimization Optimizer Learning rate / schedule Warmup steps Weight decay / gradient clipping norm PPO epochs / loss aggregation

AdamW 1 × 10−5 / constant 10 0.01 / 1.0 1 / token mean

D.3

Extra Details of Ablation Study Table 8: Ablation Study on Science Q&A Physics.

Model

Qwen3-8B

Method

Reference

Physics

ρ Avg@16

Pass@16

Majority@16

Format

JSD JSD FKL FKL

No Yes No Yes

– – – –

79.4 77.2 79.1 76.3

88.3 84.6 91.9 82.3

81.7 79.4 81.8 78.3

99.8 99.8 100.0 100.0

SR-OPSD (Ours) SR-OPSD (Ours) SR-OPSD (Ours)

Yes Yes Yes

0.50 0.70 0.95

79.9 80.6 81.3

88.9 82.4 86.8

80.0 81.5 82.4

100.0 100.0 100.0

SR-OPSD: Self-Referenced On-Policy Self-Distillation

Analyze & explore related research in Conceptio

Related documents