VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation

VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation Zewen Ding1,2∗ , Zezhong Wu1,2∗ , Zhou Tao1,2 , Shida Wang1,2 , Shizhuo Hou1,2 , YongXiang Hua1,2 , Haoyu Cao1,2 , Linli Xu1,2† 1

University of Science and Technology of China State Key Laboratory of Cognitive Intelligence

arXiv:2608.28306v1 [cs.LG] 28 Aug 2026

2

Abstract On-policy self-distillation (OPSD) improves reasoning by training a problem-only student on its own rollouts using dense token-level supervision from a privileged teacher that also sees a reference solution. However, standard OPSD treats the teacher distribution as a fixed target along the student’s rollout and updates only the student – even though privileged conditioning does not guarantee that the teacher always provides the most appropriate target for problem-only reasoning. This one-way supervision can therefore misdirect the student when the teacher distribution is misaligned with valid student reasoning. We therefore introduce Verifier-Informed Studentto-Teacher Adaptation (VISTA), which preserves the standard OPSD student update while using outcome-verified rollouts to adapt the teacher toward the student distribution. Within each verified rollout, VISTA further restricts this adaptation to the top-k positions with the largest teacher–student KL divergence. Notably, VISTA reuses the rollout and loss function from standard OPSD, introducing no additional sampling or separate reward objective. Across AIME24, AIME25, and HMMT25 with Qwen3 models at 1.7B, 4B, and 8B, VISTA achieves the highest Avg@12 at every scale, improving over OPSD by 0.6, 0.7, and 2.1 points, respectively. These results demonstrate the value of student supervision from outcomeverified rollouts and highlight student-to-teacher adaptation as a promising direction for OPSD.

1

Introduction

On-policy self-distillation (OPSD) has recently emerged as a practical way to post-train reasoning models (Zhao et al. 2026; Hübotter et al. 2026). In each OPSD step, a problemonly student samples a rollout using only the information available at inference. A privileged teacher with additional access to a reference solution then provides dense token-level supervision. These on-policy rollouts keep training aligned with inference-time behavior, while the token-level targets provide fine-grained guidance throughout the reasoning process. Although this design is appealing in practice, we identify a structural limitation of standard OPSD, which we term the teacher-superiority assumption: at every position along the student’s rollout, standard OPSD treats the privileged distribution as the better token-level target, even though reference conditioning alone does not guarantee its superiority for problem-only reasoning. This assumption can misdirect ∗ †

These authors contributed equally. Corresponding author.

𝑥

25! − 15! =? (25 − 15)

×

𝑦 ∗ = 625 − 225 = 400 (25 + 15)

=

OPSD

KL loss (25 − 15)

×

(25 + 15)

400

VISTA =

400

Figure 1: A schematic of VISTA. Building on on-policy selfdistillation (OPSD), VISTA additionally introduces a selective student-to-teacher adaptation on verified trajectories: at token positions selected by their KL divergence, the privileged teacher is revised toward the student’s distribution, so that the teacher in turn provides more benign supervision to the student. the student in two ways. First, the teacher may favor tokens that explicitly invoke its privileged input, such as “reference solution.” Training toward such targets can lead the problemonly student to mention a reference it never observes at inference (Yang et al. 2026; Shen et al. 2026b). Second, when a valid token favored by the student departs from the teacher’s preferred continuation, the teacher may assign it too little probability. Using the teacher distribution as the target lowers the student’s probability for that token, suppressing valid reasoning and self-correction (Kim et al. 2026; Shen et al. 2026a; Kaur et al. 2026; Peng et al. 2026). Recent methods reduce these failures on the student side. RLSD anchors update directions to verifier rewards rather than the teacher distribution (Yang et al. 2026), whereas TRACE confines distillation to selected token spans (Wang et al. 2026a), and DemoPSD reweights the student’s distillation target toward the student where teacher and student disagree (Li et al. 2026). These safeguards reduce the student’s exposure to mismatched targets, but supervision remains one-way from teacher to student, leaving valuable signals in student rollouts and their token-level distributions unused. This raises the following question: under appropriate conditions, can feedback from the problem-only student’s distribution improve the privileged teacher, so that the teacher, in turn, supervises the student better? Letting student behavior revise the privileged teacher is not a matter of simply reversing the distillation direction:

the student’s token-level distributions are not in general better targets than the teacher’s, and adopting them wholesale could pull the teacher distribution in the wrong direction. Even when the student can be trusted, a second problem remains: teacher–student disagreement is highly non-uniform across positions, and updating the teacher at every token could rapidly shrink the distributional gap and erode complementary privileged knowledge. The central design question is therefore twofold: which rollouts can provide teacher feedback, and which token positions within them should carry it. We introduce Verifier-Informed Student-to-Teacher Adaptation (VISTA), an effective framework that enables an OPSD teacher to learn from its student. To the best of our knowledge, VISTA is the first to establish student-to-teacher feedback within OPSD while preserving the standard student update. Its simplicity is deliberate: VISTA answers the two questions above through rollout-level and token-level selection. Teacher feedback is enabled only for rollouts that pass the outcome verifier, since the verified outcome is a inexpensive yet reliable signal that the student’s distributions along that rollout are informative and worth learning from. Within each eligible rollout, KL-guided position selection adapts the teacher toward the student at the top-k positions of largest teacher–student disagreement. Concentrating adaptation on these positions targets the teacher distributions most misaligned with problem-only reasoning, while the sparse selection keeps the teacher from converging on the student too quickly and eroding its subsequent supervision. Crucially, the method reuses the same rollouts and loss function as standard OPSD, requiring neither additional sampling nor a separate reward objective. When compared against Base, SFT, GRPO, SDPO and standard OPSD results under the matched OPSD protocol, VISTA achieves the highest average scores (Avg@12 across three benchmarks) at all three Qwen3 scales. Relative to reported OPSD, VISTA improves the three-benchmark average by 0.6, 0.7, and 2.1 points at 1.7B, 4B, and 8B scales, respectively, and achieves state-of-the-art performance in eight of the nine scale-benchmark settings. Ablations validate the contributions of verifier-gated and selective teacher adaptation, while three analytical studies indicate that the adapted teacher better supports validated student reasoning, reduces explicit reference attribution, and preserves teacher capability. Our contributions are threefold: • We formalize the limitations of OPSD’s teachersuperiority assumption and characterize how referenceconditioned distributions can over-support referencespecific next tokens or under-support valid alternatives. • We explore whether an OPSD teacher can learn from its student and introduce VISTA, which enables studentto-teacher adaptation through a verifier-based outcome gate and a top-k token-position mask while preserving standard OPSD student training. • Across three Qwen3 scales and three competition-math benchmarks, VISTA achieves the highest aggregate Avg@12 at every scale and state-of-the-art performance

in eight of the nine scale–benchmark settings under the matched OPSD protocol.

2

Related Work

On-policy distillation with privileged context. Knowledge distillation transfers a teacher’s predictive distribution to a student (Hinton, Vinyals, and Dean 2015; Kim and Rush 2016; Sanh et al. 2019; Gou et al. 2021). Two extensions define our setting. First, privileged distillation lets the teacher exploit side information available only at training time; generalized distillation and context distillation cast this privileged information as a teacher–student transfer (Vapnik and Vashist 2009; Lopez-Paz et al. 2015; Snell, Klein, and Zhong 2022). Second, on-policy distillation supervises the student on its own continuations rather than a fixed corpus (Agarwal et al. 2024; Gu et al. 2024; Ross, Gordon, and Bagnell 2011; Xu et al. 2025). On-policy self-distillation (OPSD) unites both: a problem-only student trains on its own rollouts under dense token-level targets from a teacher privileged with a reference solution (Zhao et al. 2026). Closely related methods share this regime: OPCD conditions the teacher on additional context (Ye et al. 2026), SDPO supplies feedback or a successful sibling attempt (Hübotter et al. 2026), and Privileged Information Distillation applies reward objectives to teacher trajectories (Penaloza et al. 2026). All of these methods retain a fixed privileged-to-unprivileged teaching direction. Although mutual and self-distillation relax this direction, they do so only in plain classification, without privileged information or outcome verification (Furlanello et al. 2018; Zhang et al. 2018; Anil et al. 2018; Zhang et al. 2019). VISTA instead lets outcome-verified student behavior revise the privileged target. Verifier-informed reasoning supervision. In mathematics and code, outcome verifiers determine whether a rollout reaches a valid final result and provide a reliable sequencelevel signal (Cobbe et al. 2021; Uesato et al. 2022; Le et al. 2022). RLVR turns this verifier signal directly into a policy-gradient reward, making outcome verification itself the learning signal (Shao et al. 2024; DeepSeek-AI 2025; Lambert et al. 2024). Different from process supervision, which needs step-level labels or learned reward models (Uesato et al. 2022; Lightman et al. 2024; Wang et al. 2024), this outcome check is cheap to obtain and requires no additional annotation. Self-training and filtering methods such as STaR, RFT, and Reinforce-Rej instead reuse successful completions for supervised updates or outcome-filtered policy optimization (Zelikman et al. 2022; Yuan et al. 2023; Xiong et al. 2025). Verification in these methods ultimately governs updates to the model being optimized. In VISTA, the outcome signal instead gates the teacher-side distributional update: only verified rollouts provide soft student targets for teacher adaptation, while the dense OPSD loss remains active on every rollout. Selective and adaptive privileged supervision. Dense or sustained OPSD can amplify artifacts or degrade continual post-training (Wang et al. 2026b; Shen et al. 2026b; Kaur et al. 2026; Peng et al. 2026), motivating regulation by update

Outcome Verifier

Rollout

𝑦!

𝑦"

𝑦$

𝑦#

…

𝑦%

for each position

Student

P

Distribution Divergence

Problem

Correct

VISTA

𝐷&' (𝑃(% ||𝑃() )

Position-wise Comparison

%

(*!

v

Enable

𝑑!

𝑑"

𝑑$

KL Score

𝑑#

Token Mask 1

0

…

Teacher

Shared Loss Term ( Loss = , 𝐷&'

Position Selection

Hint

Full-Position Update

0

1

… 0

Selected-Position Update

𝑑%

Figure 2: Overview of the VISTA training pipeline. (1) On-policy rollout: A problem-only student samples a solution rollout for a training problem. (2) Outcome gating and position selection: The outcome verifier gates teacher updates based on rollout correctness, while position selection forms the teacher-update mask by selecting the top-k positions with the largest teacher– student KL divergence. (3) Student and teacher updates: Building on standard OPSD, VISTA retains the standard student update at every rollout position and, as explained in (2), updates the teacher toward the student’s distribution at token positions where the teacher-update mask equals one. direction or reasoning span: RLSD signs and scales tokenlevel credit by the outcome advantage (Yang et al. 2026), and TRACE routes KL supervision to annotator-identified reasoning spans (Wang et al. 2026a), while DemoPSD downweights the teacher target where the student and teacher disagree (Li et al. 2026). These apply this selectivity only to the student update, leaving the privileged teacher a fixed target. We instead make the teacher itself adaptive: outcome-verified problem-only rollouts supply the student-distribution targets, and full-distribution disagreement selects the update positions.

3.1

3

Preliminary

Method: VISTA

On-policy self-distillation (OPSD) (Zhao et al. 2026) trains a problem-only student using next-token distributions from a privileged teacher. Let S = {(xi , yi⋆ )}N i=1 be a training set of problems and reference solutions. The student policy πS (· | x) sees only the problem x, while the privileged teacher πT (· | x, y ⋆ ) also sees the reference solution y ⋆ . Training remains on-policy: a rollout y is sampled from the current student, y ∼ πS (· | x), and both policies are evaluated on the same partial student rollout. For position t with prefix y<t , define PtS = πS (· | x, y<t ),

PtT = πT (· | x, y ⋆ , y<t ).

OPSD trains the student to match the privileged teacher along these sampled rollouts:  X clip T S  1 D Pt ∥Pt  , LOPSD = E (x,y⋆ )∼S  |y| t=1 V,τ y∼πS (·|x) 

|y|



(1)

clip where DV,τ is the OPSD pointwise-clipped full-vocabulary forward KL (Kullback and Leibler 1951), clip DV,τ (P ∥Q) =

  P (v) min P (v) log ,τ . Q(v) v∈V X

(2)

Thus, OPSD applies one-way teacher-to-student supervision at every prefix of the sampled rollout. It thereby combines on-policy training with dense token-level feedback, without an external teacher or reward model.

3.2

Problem: The Teacher-Superiority Assumption in OPSD

At each prefix along a student-generated rollout, standard OPSD trains the student to match the teacher’s next-token distribution. However, we argue that this one-way update relies on what we call the teacher-superiority assumption: the teacher must remain a better learning target than the current student throughout the rollout. To formalize this assumption, let PtI denote the unobserved next-token distribution of an ideal problem-only policy at prefix ht = (x, y<t ). OPSD aims to approach this ideal distribution indirectly by moving PtS toward PtT . We define their relative distance to the ideal target as ∆t = DKL (PtI ∥PtT ) − DKL (PtI ∥PtS ),

(3)

where ∆t < 0 means that the teacher is closer to PtI ; for ∆t > 0, the reverse holds. Moving PtS toward PtT can bring it closer to PtI only when the teacher is itself closer to the ideal distribution than the current student. By Eq. (3), this requires ∆t < 0 at every token position, which is precisely the teacher-superiority assumption.

Reference conditioning changes the teacher’s information set, but by itself cannot guarantee that the resulting distribution is uniformly better suited to problem-only deployment. At positions where ∆t > 0, OPSD still applies the same update even though the student is closer to PtI . This creates two risks. For a candidate next token v, when PtT (v) > PtS (v), the update can over-support a reference-specific continuation, whereas when PtS (v) > PtT (v), it can suppress a valid alternative. Figure 3 illustrates such a failure at a shared prefix: the reference-conditioned teacher locally omits a term, while the outcome-verified student follows a valid alternative. Problem R Evaluate I = N (x)/D(x) dx, x ∈ [0, π/2] 4 N (x) = c + sc3 + s2 c2 + s3 c D(x) = s4 + c4 + 2sc3 + 2s2 c2 + 2s3 c, . . . Student rollout prefix y<t . . . D = (s4 + c4 ) + 2(sc3 + s2 c2 + s3 c). We saw in the Teacher continuation Student continuation reference solution that: numerator that: sc3 + s2 c2 + s3 c s4 + c4 = 1 − 2s2 c2 ; remaining terms give = sc(c2 + s2 + sc) × = sc(1 + sc). X 2sc(c2 + s2 ) = 2sc.

Figure 3: A representative case of the teacher-superiority assumption failing. From the same student rollout prefix, the student follows valid reasoning, whereas the teacher emits a reference-specific phrase. This counterexample shows that the privileged distribution is not uniformly a better tokenlevel target along the student’s rollout.

3.3

VISTA Training Objective

A natural mitigation for the problem above is to let the teacher learn from the student at token positions where the student is closer to the ideal problem-only distribution, namely where ∆t > 0. However, because of the unobservable nature of the ideal distribution PtI , ∆t cannot be computed during training. As an initial exploration, we introduce VerifierInformed Student-to-Teacher Adaptation (VISTA), a simple but effective mechanism that enables the student to provide training supervision for the teacher. VISTA uses two simple signals to determine where the teacher update is allowed: outcome verification identifies rollouts with valid outcomes, and KL-guided position selection prioritizes token positions with large teacher–student disagreement. We detail these two selection procedures in Section 3.4. Once a trajectory qualifies for feedback, VISTA forms the teacher update from the current student trajectory and paired teacher–student distributions, avoiding a new sampling loop or an independent reward-optimization phase. To formalize this high-level design as a training objective, we build on the student and teacher next-token distributions defined in §3.1, namely PtS = πS (· | x, y<t ) and PtT = πT (· | x, y ⋆ , y<t ). VISTA retains the OPSD student loss from Eq. (1) and adds a teacher loss over selected token positions. Let γt (x, y, y ⋆ ) ∈ {0, 1} denote the binary mask that determines whether the teacher is updated at token

position t of rollout y. The two trajectory-level losses are |y|

LS (x, y, y ⋆ ) =

1 X clip DV,τ sg[PtT ]∥PtS , |y| t=1 |y|

1 X clip LT (x, y, y ) = γt (x, y, y ⋆ )DV,τ PtT ∥ sg[PtS ] . |y| t=1

(4)

⋆

Here, sg[·] denotes stop-gradient: it blocks gradients through PtT in LS and through PtS in LT , so the two losses update only the student and teacher, respectively. Both losses are evaluated from the same pre-update policy snapshot, with the sampled rollout and teacher-update mask treated as fixed. LS , which places the optimized distribution πS in the second argument of the KL divergence, is commonly known as the forward-KL objective. Its mode-covering tendency encourages the student to retain probability across teachersupported alternatives. By contrast, LT optimizes the first argument of the KL divergence, yielding a reverse-KL objective. This design exploits the mode-seeking tendency of reverse-KL, encouraging the teacher to concentrate more of its probability mass on fewer alternatives in the student’s next-token distribution (Gu et al. 2024). Combining the student and teacher losses yields the complete VISTA objective: Ltotal = E (x,y⋆ )∼S [LS (x, y, y ⋆ ) + λLT (x, y, y ⋆ )] ,

(5)

y∼πS (·|x)

where λ is a positive weight controlling the contribution of the teacher-adaptation loss.

3.4

Outcome Gating and KL-Guided Position Selection

As defined in Sec. 3.3, VISTA incorporates the teacheradaptation loss LT into its training objective. However, it remains to determine on which rollouts and at which token positions the teacher should be updated. Accordingly, we factorize the binary teacher-update mask γt for each generated token yt into rollout-level and token-level components: γt (x, y, y ⋆ ) = γ out (x, y) γtpos (x, y, y ⋆ ).

(6)

While the outcome gate γ (x, y) determines whether the completed rollout y is eligible to train the teacher at all, the position mask γtpos (x, y, y ⋆ ) determines whether the teacher loss is calculated at this token position. out

Outcome gate. Student supervision is not uniformly reliable: when a rollout fails, the verifier reveals that the trajectory contains an error but not where it occurs. Using its token distributions to update the teacher could then reinforce the unidentified error. VISTA therefore enables teacher adaptation only when a deterministic verifier accepts the rollout outcome. In mathematics and code, such verification requires only rule-based answer checks or sandboxed test execution, leaving no need for a learned reward model or process-level annotations. Specifically, it sets γ out (x, y) = 1 if the verifier accepts the outcome of rollout y, and 0 otherwise. Passing this gate provides outcome-level evidence that the student can reach a valid solution from the problem alone. The student distributions along that rollout can therefore serve as feedback for adapting the privileged teacher.

Top-k KL-divergence position mask. Teacher adaptation must remain selective: updating the teacher at every rollout token can quickly shrink the teacher–student gap, weakening supervision in subsequent OPSD steps and potentially hurting the student’s final performance. VISTA therefore evaluates the teacher loss only at rollout positions where the two distributions disagree most, which it identifies by assigning each position a teacher–student KL score:
dt (x, y, y ⋆ ) = DV PtT ∥PtS ,

(7)

It then selects up to k highest-scoring positions along the rollout: min(k,|y|)

Ik (x, y, y ⋆ ) = TopKj∈{1,...,|y|} dj (x, y, y ⋆ ),

γtpos (x, y, y ⋆ ) = I[t ∈ Ik (x, y, y ⋆ )] .

(8)

Consequently, γt (x, y, y ⋆ ) = 1 only at the selected top-k KL-divergence positions on outcome-verified rollouts, and is 0 otherwise.

4

Experiments

We evaluate VISTA on competition-level mathematical reasoning by asking whether student-to-teacher adaptation improves upon OPSD across model scales and how much the verifier-based outcome gate and top-k KL-divergence position mask contribute to these gains. We further conduct three teacher-side analyses, which show that VISTA’s adapted teacher provides stronger token-level support for valid student reasoning, exhibits explicit reference attribution less frequently, and preserves its privileged problem-solving capability.

4.1

Experimental Settings

Models and data. We evaluate VISTA with the instructtuned Qwen3-1.7B, Qwen3-4B, and Qwen3-8B models (Yang et al. 2025). Following OPSD (Zhao et al. 2026), we train on the OpenThoughts mathematical-reasoning corpus (Guha et al. 2025) with the same 100-step on-policy distillation budget and student-training protocol, changing only the student learning rate for our setup. Evaluation. We evaluate on AIME 2024, AIME 2025, and HMMT 2025 (HuggingFaceH4 2025; Lin 2025; MathArena 2025) and report Avg@12, the mean accuracy over 12 sampled completions. Table 1 compares VISTA with the base model, SFT, GRPO (Shao et al. 2024), and standard OPSD using results reported in the OPSD paper (Zhao et al. 2026). Additionally, we train and evaluate SDPO (Hübotter et al. 2026) under the same protocol. Teacher adaptation. For teacher training, we keep most training hyperparameters at the default values used for the OPSD student, while varying only the teacher-loss weight, λ, and the position budget, k. Across the model scales and values tested, we empirically find that settings with λ ≥ 0.75 and k ∈ [16, 32] provide a strong, transferable operating range for teacher adaptation.

4.2

Main Results

4.3

Ablation Studies

Under the matched OPSD protocol, Table 1 shows that VISTA delivers the strongest and broadest performance among all compared methods. It achieves the highest threebenchmark Avg@12 at every Qwen3 scale, improving over reported OPSD by 0.6, 0.7, and 2.1 points at 1.7B, 4B, and 8B, respectively. Across the nine scale–benchmark settings, VISTA sets eight new state-of-the-art results. The advantage is especially pronounced at 8B, where VISTA surpasses reported OPSD by 0.8 points on AIME24, 3.1 on AIME25, and 2.5 on HMMT25. The substantially larger gain at 8B highlights VISTA’s favorable scaling with student capacity. As the student becomes stronger, its successful rollouts provide higher-quality distributions from which the privileged teacher can learn. The improved teacher, in turn, provides stronger supervision for subsequent student updates. This positive loop helps account for VISTA’s pronounced advantage at the largest scale. To verify the effectiveness of VISTA’s outcome-gating and position-selection designs, we conduct the following ablation studies using the Qwen3-8B model under the training and evaluation protocol described in §4.1. Outcome gate. To separate the semantics of outcome gating from the number of teacher updates, we compare VISTA with three controls: adapting the teacher on every rollout, on a random 60% subset that approximately matches VISTA’s activation rate, or only on verifier-rejected rollouts. Even without outcome-aware selection, adapting the teacher improves over standard OPSD, as random gating and all-rollout adaptation raise Avg@12 from 64.8 to 65.6 and 65.8, respectively. By moving the teacher distribution closer to the student’s, adaptation makes the resulting teacher targets easier for the student to learn. We next ask which types of student trajectories are most beneficial for teacher adaptation. The two outcome-directed gates yield opposite results: the reverse gate, which adapts the teacher only on verifier-rejected rollouts, drops to 64.5, below OPSD, whereas VISTA, which uses outcome-verified rollouts, achieves the best result among all variants at 66.9 Avg@12. This contrast shows that outcome verification provides a simple yet effective criterion for selecting highquality student trajectories, along which the student’s token distributions offer reliable supervision for teacher adaptation. Rollout-position selection. To identify the positions along an outcome-verified rollout at which the student next-token distributions provide the most useful supervision for the teacher, we compare four rules: selecting the first k positions, k uniformly sampled positions, the last k positions, and VISTA’s KL-guided top-k positions. Figure 4 reports the per-benchmark results and their average score. All four position-selection variants outperform OPSD’s 64.8 Avg@12, with VISTA’s KL-guided top-(k) rule achieving the best score of 66.9. Large KL divergence identifies the positions of greatest teacher–student disagreement. When these positions occur in an outcome-verified rollout, the

Model

Method

AIME24

AIME25

HMMT25

Average

Qwen3-1.7B

Base SFT GRPO SDPO OPSD VISTA

51.5 48.4 51.1 52.8 57.2 57.8

36.7 36.3 38.3 36.9 43.9 43.3

23.1 22.7 23.7 22.8 29.2 30.8

37.1 35.8 37.7 37.5 43.4 44.0

Qwen3-4B

Base SFT GRPO SDPO OPSD VISTA

74.9 70.2 75.6 76.1 76.4 77.2

66.4 62.3 68.1 65.8 68.3 69.2

42.2 43.4 44.4 42.5 46.1 46.4

61.2 58.6 62.7 61.5 63.6 64.3

Qwen3-8B

Base SFT GRPO SDPO OPSD VISTA

75.8 72.3 76.4 76.4 77.8 78.6

65.6 64.2 68.9 68.1 70.8 73.9

43.9 42.9 46.7 44.4 45.8 48.3

61.8 59.8 64.0 63.0 64.8 66.9

Table 1: Main results on competition-level mathematical reasoning (Avg@12, %). Each benchmark uses its peak checkpoint under the OPSD protocol, and Average is the unweighted mean over the three benchmarks. Base, SFT, GRPO, and OPSD results are from the OPSD paper (Zhao et al. 2026); all post-training methods use the same corpus, with baseline budgets retained as reported. The best and second-best results within each model block are shown in boldface and underlined, respectively. Adaptation rule OPSD

Rate 0%

All rollouts 100% Random gate 60% Reverse gate ∼ 40% VISTA ∼ 60%

Avg@12 ↑ 64.8

65.8 65.6 64.5 66.9

Table 2: Outcome-gate ablation on Qwen3-8B. Rate denotes the fraction of rollouts routed to teacher adaptation; the random gate approximately matches VISTA’s rate, while the reverse gate uses only verifier-rejected rollouts. valid outcome lends credibility to the student’s reasoning. The disagreement is therefore more likely to expose teacher distributions misaligned with valid problem-only reasoning, making the corresponding student distributions especially useful feedback for the teacher. Top-k selection size. Having established the advantage of KL-guided position selection, we next examine how the number of selected positions affects the effectiveness of teacher adaptation. We evaluate k ∈ {8, 16, 32, 64} and an All setting, which applies the teacher loss at every rollout position within the first 1,024 tokens, following OPSD. Figure 5 summarizes the resulting Avg@12 scores on AIME24 and AIME25. VISTA achieves its best performance at k = 16, with Avg@12 scores of 78.6 on AIME24 and 73.9 on AIME25. The k = 32 setting performs comparably. Across the full sweep, both benchmarks exhibit an overall rise-then-fall trend, as selection sizes below 16 and above 32 consistently underperform the two intermediate choices.

Figure 4: Rollout-position selection ablation on Qwen3-8B (Avg@12). We compare OPSD with four teacher-adaptation variants: (A) the first k positions, (B) k uniformly random positions, (C) the last k positions, and (D) the top-k KLdivergence positions. VISTA uses (D) and attains the best three-benchmark average. One plausible explanation is that this trend reflects a tradeoff in teacher adaptation. With too few selected positions, the teacher update remains too sparse to support sufficient adaptation. By contrast, selecting too many positions pulls the teacher distribution too rapidly toward the student’s, which can erode the teacher’s distinct supervisory signal and make subsequent student updates less effective.

4.4

Analytical Studies

Main-task accuracy alone cannot tell us whether adaptation brings PtT closer to the ideal distribution PtI defined in §3.2, and the unobservable nature of PtI makes DKL (PtI ∥PtT )

Figure 5: Avg@12 on AIME24 and AIME25 as the number of teacher-update positions varies.

Figure 6: Teacher-adaptation diagnostics across Qwen3-8B checkpoints: low-probability tokens (πT (yt ) ≤ 0.05) on validated student trajectories and explicit reference attribution in teacher responses. the student’s problem-only distribution.

impossible to compute during training. We therefore turn to three complementary analyses of observable teacher behavior for indirect evidence that adaptation moves PtT in the intended direction. Token-level teacher support for valid student reasoning. We construct a fixed diagnostic set by sampling problems from the training set. Using the base Qwen3-8B model in the problem-only student setting, we generate candidate rollouts and retain 100 trajectories whose final answers are verified to be correct and whose complete reasoning traces are manually confirmed to be sound. The teacher should therefore not assign low probability to too many tokens along these trajectories. We evaluate the teacher checkpoints saved throughout the VISTA run using the same hyperparameters and configuration as in training, thereby mirroring the training setup. At each checkpoint, we count positions satisfying πT (yt | x, y ⋆ , y<t ) ≤ 0.05 and average the count over the 100 trajectories. Figure 6 shows that the mean count decreases over the course of training, reaching 19.32 at the final checkpoint, representing a reduction of 5.03 relative to the base model. This indicates that adaptation provides increasingly effective token-level support for valid student-generated trajectories throughout training. Explicit reference attribution during teacher adaptation. To track the teacher’s undesirable propensity to explicitly invoke the privileged reference, we measure the frequency of explicit reference attribution, defined as the percentage of sampled responses in which the teacher explicitly acknowledges the privileged reference. The evaluation covers the initial Qwen3-8B teacher and checkpoints saved throughout adaptation on 100 randomly sampled training problems. At each evaluation point, responses are sampled from the teacher for every problem, conditioned on both the problem and its reference solution. As shown in Figure 6, the rate of explicit reference attribution decreases over training and reaches 48% at both steps 75 and 100, 15 percentage points below the initial teacher’s 63%. This decline indicates that VISTA’s teacher adaptation weakens its preference for reference-dependent continuations, thereby providing guidance better suited to

Teacher capability during training. Throughout training, we evaluate the teacher’s performance on the validation set by providing it with reference answers. As shown in Figure 7, the full trajectory maintains a perfect Pass@12 throughout training, while Avg@12 shows a gradual upward trend. These results show that adapting the teacher to the student does not compromise the teacher’s own problem-solving performance.

Figure 7: Privileged-teacher validation of the Qwen3-8B teacher during training.

5

Conclusion

VISTA revises the one-way supervision structure of OPSD by enabling selective student-to-teacher adaptation. Within outcome-verified rollouts, VISTA adapts the privileged teacher toward the student next-token distributions at the top-k positions with the largest teacher–student KL divergence, while preserving the standard student update. The method reuses the existing rollout and KL objective without additional sampling or a separate reward objective, and retains only the problem-only student at inference. Across three Qwen3 scales and three competition-math benchmarks, VISTA achieves the highest Avg@12 at every scale. Ablations show that outcome-verified, high-disagreement feedback is more effective than indiscriminate or random teacher adaptation. Teacher-side analyses further indicate better support for validated student trajectories, less explicit reference attribution, and preserved privileged-teacher capability. These results suggest that privileged supervision in OPSD need not remain fixed, but can be improved through selective feedback from the student it trains.

References

Agarwal, R.; Vieillard, N.; Zhou, Y.; Stanczyk, P.; Ramos, S.; Geist, M.; and Bachem, O. 2024. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In International Conference on Learning Representations (ICLR). Anil, R.; Pereyra, G.; Passos, A.; Ormándi, R.; Dahl, G. E.; and Hinton, G. E. 2018. Large Scale Distributed Neural Network Training Through Online Distillation. In International Conference on Learning Representations (ICLR). Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168. DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948. Furlanello, T.; Lipton, Z.; Tschannen, M.; Itti, L.; and Anandkumar, A. 2018. Born again neural networks. In International conference on machine learning, 1607–1616. PMLR. Gou, J.; Yu, B.; Maybank, S. J.; and Tao, D. 2021. Knowledge distillation: A survey. International journal of computer vision, 129(6): 1789–1819. Gu, Y.; Dong, L.; Wei, F.; and Huang, M. 2024. MiniLLM: Knowledge Distillation of Large Language Models. In International Conference on Learning Representations (ICLR). Guha, E.; Marten, R.; Keh, S.; Raoof, N.; Smyrnis, G.; Bansal, H.; et al. 2025. OpenThoughts: Data Recipes for Reasoning Models. arXiv preprint arXiv:2506.04178. Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531. Hübotter, J.; Lübeck, F.; Behric, L.; Baumann, A.; Bagatella, M.; Marta, D.; Hakimi, I.; Shenfeld, I.; Kleine Buening, T.; Guestrin, C.; and Krause, A. 2026. Reinforcement Learning via Self-Distillation. arXiv preprint arXiv:2601.20802. HuggingFaceH4. 2025. AIME 2024. Hugging Face dataset. https://huggingface.co/datasets/HuggingFaceH4/ aime_2024. Kaur, S.; Ri, N.; He, Y.; Fowl, L.; and Arora, S. 2026. Rethinking On-Policy Self-Distillation for Thinking Models. arXiv preprint arXiv:2607.05184. Kim, J.; Jeon, J.; Li, D.; and Yang, Y. 2026. Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR. arXiv preprint arXiv:2605.10781. Kim, Y.; and Rush, A. M. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, 1317–1327. Kullback, S.; and Leibler, R. A. 1951. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1): 79–86. Lambert, N.; Morrison, J.; Pyatkin, V.; et al. 2024. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv preprint arXiv:2411.15124.

Le, H.; Wang, Y.; Gotmare, A. D.; Savarese, S.; and Hoi, S. C. H. 2022. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS). Li, Y.; Shi, H.; Liu, W.; Ruan, M.; Hou, H.; Dai, Z.; Qiu, S.; and Song, L. 2026. DemoPSD: Disagreement-Modulated Policy Self-Distillation. arXiv preprint arXiv:2607.02502. Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2024. Let’s Verify Step by Step. In International Conference on Learning Representations (ICLR). Lin, Y.-T. 2025. AIME 2025. Hugging Face dataset. https: //huggingface.co/datasets/yentinglin/aime_2025. Lopez-Paz, D.; Bottou, L.; Schölkopf, B.; and Vapnik, V. 2015. Unifying distillation and privileged information. arXiv preprint arXiv:1511.03643. MathArena. 2025. HMMT February 2025. Hugging Face dataset. https://huggingface.co/datasets/MathArena/hmmt_ feb_2025. Penaloza, E.; Vattikonda, D.; Gontier, N.; Lacoste, A.; Charlin, L.; and Caccia, M. 2026. Privileged Information Distillation for Language Models. arXiv preprint arXiv:2602.04942. Peng, K.; Li, C.; Ouyang, Y.; Yuan, Y.; and Ding, L. 2026. Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation. arXiv preprint arXiv:2607.10805. Ross, S.; Gordon, G. J.; and Bagnell, J. A. 2011. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS), 627–635. Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019. DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. arXiv preprint arXiv:1910.01108. Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300. Shen, G.; Cheng, X.; Zhao, C.; Huang, L.; Li, J.; Zhao, D.; and Yu, X. 2026a. Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information. arXiv preprint arXiv:2605.11609. Shen, Z.; Tong, J.; Yan, S.; Shen, C.; Chen, H.; Ye, W.; Hu, X.; Miao, R.; Wang, H.; Zhao, J.; Chen, G.; and Ye, J. 2026b. Purified OPSD: On-Policy Self-Distillation Without Losing How to Think. arXiv preprint arXiv:2607.02234. Snell, C.; Klein, D.; and Zhong, R. 2022. Learning by Distilling Context. arXiv preprint arXiv:2209.15189. Uesato, J.; Kushman, N.; Kumar, R.; Song, F.; Siegel, N.; Wang, L.; Creswell, A.; Irving, G.; and Higgins, I. 2022. Solving Math Word Problems with Process- and OutcomeBased Feedback. arXiv preprint arXiv:2211.14275. Vapnik, V.; and Vashist, A. 2009. A new learning paradigm: Learning using privileged information. Neural networks, 22(5-6): 544–557.

Wang, J.; Ouyang, X.; Chen, Z.; Hu, Y.; Pan, Z.; Li, X.; and Guo, L.-Z. 2026a. TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment. arXiv preprint arXiv:2605.10194. Wang, M.; Zhao, H.; Liu, W.; Yang, L.; Liu, G.; Guo, H.; Xie, G.-S.; Meng, G.; Liu, H.; and Zhu, F. 2026b. Denser ̸= Better: Limits of On-Policy Self-Distillation for Continual Post-Training. arXiv preprint arXiv:2607.01763. Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2024. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 9426–9439. Xiong, W.; Yao, J.; Xu, Y.; Pang, B.; Wang, L.; Sahoo, D.; Li, J.; Jiang, N.; Zhang, T.; Xiong, C.; and Dong, H. 2025. A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce. arXiv preprint arXiv:2504.11343. Xu, W.; Han, R.; Wang, Z.; Le, L.; Madeka, D.; Li, L.; Wang, W. Y.; Agarwal, R.; Lee, C.-Y.; and Pfister, T. 2025. Speculative Knowledge Distillation: Bridging the Teacher– Student Gap Through Interleaved Sampling. In International Conference on Learning Representations (ICLR). Yang, A.; et al. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. Yang, C.; Qin, C.; Si, Q.; Chen, M.; Gu, N.; Yao, D.; Lin, Z.; Wang, W.; Wang, J.; and Duan, N. 2026. Self-Distilled RLVR. arXiv preprint arXiv:2604.03128. Ye, T.; Dong, L.; Wu, X.; Huang, S.; and Wei, F. 2026. On-Policy Context Distillation for Language Models. arXiv preprint arXiv:2602.12275. Yuan, Z.; Yuan, H.; Li, C.; Dong, G.; Lu, K.; Tan, C.; Zhou, C.; and Zhou, J. 2023. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models. arXiv preprint arXiv:2308.01825. Zelikman, E.; Wu, Y.; Mu, J.; and Goodman, N. D. 2022. STaR: Bootstrapping Reasoning with Reasoning. In Advances in Neural Information Processing Systems (NeurIPS). Zhang, L.; Song, J.; Gao, A.; Chen, J.; Bao, C.; and Ma, K. 2019. Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation. In IEEE/CVF International Conference on Computer Vision (ICCV), 3713–3722. Zhang, Y.; Xiang, T.; Hospedales, T. M.; and Lu, H. 2018. Deep Mutual Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4320–4328. Zhao, S.; Xie, Z.; Liu, M.; Huang, J.; Pang, G.; Chen, F.; and Grover, A. 2026. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv preprint arXiv:2601.18734.

Algorithm 1 VISTA minibatch training procedure. Require: Training set S; separately parameterized student (πS , θS ) and teacher (πT , θT ); verifier V ; update budget M ; position budget k; clipping threshold τ ; student rate ηS ; teacher-rate argument ηT 1: for m = 1, . . . , M do 2: Sample minibatch B ⊂ S and store the pre-update snapshot (θSm , θTm ) cS ← 0, L cT ← 0, and nT ← 0 3: Set L ⋆ 4: for all (x, y ) ∈ B do 5: Sample y ∼ πS (· | x); compute γ out ← V (x, y); evaluate PtS and PtT for all t P clip cS 6: Add |y|−1 t DV,τ (sg[PtT ]∥PtS ) to L out 7: if γ = 1 then 8: Rank positions by dt = DKL (PtT ∥PtS ) and let Ik contain the largest min(k, |y|) scores P clip cT and add |Ik | 9: Add t∈Ik DV,τ (PtT ∥ sg[PtS ]) to L to nT 10: end if 11: end for cS and, if nT > 0, gT ← ∇θ L cT 12: Compute gS ← ∇θS L T 13: Update θS using gS and ηS ; if nT > 0, update θT using gT and ηT 14: end for Ensure: Trained problem-only student parameters θS

A

Use of AI Systems

Generative AI tools were used only for language editing and organization of this supplementary document. No AI was used to conceive, propose, or formulate the VISTA method. No AI was used to choose experimental settings or to design and plan experiments. The authors reviewed every AI-assisted edit and remain fully responsible for all technical claims, numerical results, citations, and final submitted wording.

B.1

B

Implementation Details

Complete VISTA Training Procedure

The main paper gives the VISTA objective and its two selection rules. This subsection makes the corresponding minibatch procedure and gradient routing explicit. For a training pair (x, y ⋆ ), the problem-only student samples one rollout y. The student and privileged teacher are then evaluated at the same student prefixes. The student loss is active at every rollout position. The teacher loss is active only if the deterministic outcome verifier accepts the completed rollout, and then only at the min(k, |y|) positions with the largest raw teacher-first KL scores. Algorithm 1 shows where this procedure enters the training pipeline. At every minibatch update, VISTA retains the standard OPSD student branch over all rollout positions and adds the verifier-gated, top-k teacher-adaptation branch before the two parameter blocks are updated synchronously. For convenience, the implementation uses the teacher-rate argument ηT instead of λ to encode the same relative update strength. With student rate ηS , the main-paper coefficient is simply λ = ηT /ηS .

B.2

Training Configuration

OPSD lineage and VISTA modification. Our implementation retains the central OPSD training logic and studentside pipeline (Zhao et al. 2026). The key difference from standard OPSD lies in the teacher branch: whereas OPSD keeps the privileged target fixed, VISTA maintains an independent, trainable teacher and adds the verifier-gated, top-k teacher update from Section B.1. Apart from this teacher-side addition, the student branch follows OPSD and is optimized at every valid rollout position. Data, conditioning, and rollouts. Student and teacher are initialized from the same instruct-tuned Qwen3 model at the 1.7B, 4B, or 8B scale (Yang et al. 2025). We draw from a pool of up to 30K problem–reference-solution pairs from the mathematical-reasoning subset of OpenThoughts (Guha et al. 2025). With one rollout per prompt, an effective global batch size of 32, and 100 updates, the reported schedule samples about 3,200 prompt–rollouts in total. The student receives only the problem, with thinking mode disabled. The teacher receives the problem and reference solution, with thinking mode enabled. Verification uses the full response (up to 4,096 new tokens), whereas next-token loss is computed only on its first 1,024 tokens: the student scores these tokens conditioned on the problem-only prompt, and the teacher scores the same tokens conditioned on the problem and reference solution. Table 3 provides a more detailed listing of the implementation hyperparameters and related settings. Prompt templates. We next detail how the student and teacher prompts are constructed during training. We use exactly the same prompt-construction procedure as OPSD (Zhao et al. 2026). In the default launcher path, reason_first=False. The literal message contents are shown below; the student uses thinking mode off, while the teacher uses thinking mode on. Student training prompt (thinking disabled) Problem: {problem} Please reason step by step, and put your final answer within \boxed{}.

Teacher training prompt (thinking enabled) Problem: {problem} === Reference Solution Begin === {reference_solution} === Reference Solution End === After reading the reference solution above, make sure you truly understand the reasoning behind each step – do not copy or paraphrase it. Now, using your own words and independent reasoning, derive the same final answer to the problem above. Think step by step, explore different approaches, and do not be afraid to backtrack or reconsider if something does not work out. Please reason step by step, and put your final answer within \boxed{}.

Parameter

Setting

Model scale Qwen3-1.7B, Qwen3-4B, or Qwen3-8B. Data pool OpenThoughts mathematical-reasoning pool. Student prompt Problem only; thinking off. Teacher prompt Problem plus reference solution; thinking on. Temperature 1.1. Top-p 0.95. Top-k 20. Verification roll- Up to 4,096 new tokens for correctness checkout ing. Scored rollout to- Up to the first 1,024 tokens of each rollout. kens Distillation Full-vocabulary forward KL with β = 0. Clip threshold τ = 0.05. Teacher gate Verifier-accepted rollouts only. Position selection Raw-KL top-k positions; k = 32 for 1.7B/4B and k = 16 for 8B. Learning rates ηS = 4 × 10−6 and ηT = 3 × 10−6 ; λ = ηT /ηS = 0.75. LoRA r = 64, α = 128; targets q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. Adapters Independent student and teacher adapters. Optimizer AdamW; zero weight decay; linear student schedule without warmup. Budget Effective global batch 32; 100 updates. Numerics Bfloat16 and gradient checkpointing. Hardware One node with eight H800 80GB GPUs for every model scale. Checkpoints Saved every 25 updates.

Parameter

Value

Max new tokens Thinking mode Temperature Top-p Top-k Min-p Seed Samples per prompt

38,912 Enabled 1.0 0.95 −1 (disabled) 0.0 42 12

Table 4: VISTA evaluation configuration. For every problem, a single vLLM sampling call draws 12 completions with temperature 1.0, top-p = 0.95, top-k disabled, min-p = 0, and seed 42. Each completion may contain at most 38,912 new tokens. Answer checking and metrics. To score a completion, the reused OPSD evaluator extracts its final balanced \boxed{...} expression and uses math_verify to compare the parsed answer with the ground truth for mathematical equivalence. A completion with no such expression is counted as incorrect. For benchmark b with Nb = 30 problems and R = 12 completions per problem, we report N

Avg@12(b) =

Table 3: Core VISTA training configuration.

N

Pass@12(b) = Compute environment and training cost. The Qwen31.7B, Qwen3-4B, and Qwen3-8B post-training runs were conducted on one node with eight NVIDIA H800 80GB GPUs. Complete 100-update training runs from Qwen3-1.7B through Qwen3-8B took approximately 1–6 hours, depending on model scale. The training software stack comprised Python 3.12, CUDA 12.8, PyTorch 2.8, verl 0.6.1, and vLLM 0.11.0.

B.3

Evaluation Configuration

OPSD evaluation lineage. Our implementation directly reuses the evaluation logic released with OPSD (Zhao et al. 2026), including its problem prompt, Qwen3 chat templating, boxed-answer extraction, mathematical-equivalence checker, and Avg@N /Pass@N aggregation. Benchmarks, prompts, and decoding. We evaluate all 30 problems from each of AIME 2024, AIME 2025, and HMMT 2025 (HuggingFaceH4 2025; Lin 2025; MathArena 2025). Each user prompt contains the problem followed by the following fixed instruction: Evaluation instruction (thinking enabled) Please reason step by step, and put your final answer within \boxed{}.

R

b X 100 X I[completioni,r is correct], Nb R i=1 r=1 (9) b 100 X I[∃r : completioni,r is correct]. Nb i=1 (10)

Checkpoint reporting and baseline provenance. The evaluation phase starts at step 25 and evaluates every saved student checkpoint whose update is a multiple of 25. For the 100-update runs reported in this paper, the evaluated states are therefore steps 25, 50, 75, and 100. All methods use the same evaluation protocol and peak-selection rule. Following the OPSD reporting convention, each benchmark reports its highest Avg@12 over these states; the main table’s Average is the unweighted mean of the three peaks and may combine different steps. OPSD supplies the Base, SFT, GRPO, and OPSD entries (Zhao et al. 2026); SDPO and VISTA are evaluated locally.

C.1

C

Additional Experimental Details

Controlled Ablation Protocols

All controlled ablations use Qwen3-8B and the protocols in Sections B.2 and B.3. Outcome gate. OPSD applies no teacher update. For the three teacher-adaptation controls and VISTA, let V (x, y) ∈ {0, 1} denote the deterministic outcome verifier, with V (x, y) = 1 when the student rollout y for problem x is accepted and V (x, y) = 0 otherwise. The all-rollout control therefore enables the teacher update on every rollout, whereas the random control independently samples

setting retains every valid position. For every setting, candidate positions are restricted to the first 1,024 rollout tokens. This limit is not a separately tuned VISTA choice: it exactly follows the OPSD training protocol, which computes its distillation loss over the first 1,024 rollout tokens (Zhao et al. 2026). The all-position setting applies the teacher loss at every valid position in that same prefix, so comparisons with OPSD and the sparse variants use an identical token horizon. Only AIME 2024 and AIME 2025 were recorded for this sweep.

C.2

Figure 8: Verifier acceptance rate of Qwen3-8B student rollouts during VISTA training, centered near 60%. r ∼ Bernoulli(0.6) for each rollout and enables it when r = 1. The reverse control enables the update only on verifierrejected rollouts, while VISTA enables it only on verifieraccepted rollouts. Equivalently, the rollout-level masks are   1,   r, γ out (x, y) =  1 − V (x, y),   V (x, y),

all-rollout control, random control, reverse control, VISTA.

(11)

We use probability 0.6 for the random control because the verifier accepts roughly 60% of student rollouts during VISTA training, as shown in Figure 8. Position selection. With the gate fixed to verifier-accepted rollouts and m = min(k, |y|), the first-k and last-k controls use the first or last m positions, random-k samples an melement subset uniformly without replacement, and the KL control uses the m largest raw teacher-first KL scores. More explicitly, for each rollout position t, define  dt = DKL PtT

PtS



=

X

v∈V

PtT (v) log

PtT (v) , PtS (v)

(12)

where PtT and PtS are the teacher and student next-token distributions at the same student-visited prefix. The controls select positions as follows: the first-k control uses the first m rollout positions, the last-k control uses the last m positions, and the random-k control samples m positions uniformly without replacement. The KL control ranks all rollout positions by dt and uses the m positions with the largest scores. For this Qwen3-8B position-selection comparison, we fix the position budget to k = 16 for every variant. We do not include a bottom-k control because its near-zero-KL positions have very small teacher-update losses and provide little training signal. Position budget. The sweep evaluates k ∈ {8, 16, 32, 64} and an all-position setting. For each sparse setting, candidate positions are ranked by the raw teacher-first KL scores in Eq. (12) and the top k positions are retained. The all-position

Teacher-Side Analysis Protocols

We evaluate the initial Qwen3-8B teacher and the teacher checkpoints saved at steps 25, 50, 75, and 100 from the specific Qwen3-8B training run whose student result is reported as the SOTA result in the main table. Valid student trajectories. The fixed bank contains 100 problem-only Qwen3-8B trajectories from randomly sampled training problems. Each trajectory is automatically verified as correct and then manually reviewed; the review confirms that the overwhelming majority of its tokens and the resulting reasoning trajectory are coherent, well-formed, and valid for the stated problem. Using exactly the same configuration as in training, we then provide each checkpoint teacher with the paired reference solution and evaluate the token-level probabilities along these same 100 student trajectories. We count the tokens assigned probability at most 0.05 and average the count over the bank. Explicit reference attribution. At each checkpoint, the privileged teacher samples one reference-conditioned response for each of 100 randomly sampled training problems. We use an LLM-as- judge procedure with Opus 4.6 as the judge model to identify responses whose realized text explicitly attributes its reasoning to the privileged reference, including phrases such as “according to the reference” or “as the reference solution ...”. We count only trajectories containing such clear reference- attribution statements. Privileged-teacher capability. The capability probe uses 30 validation problems randomly sampled from the training set and 12 privileged-teacher responses per problem.

D.1

D

Additional Results

Checkpoint-Wise Avg@12 for Qwen3-8B

To characterize how VISTA’s model performance evolves over time, we present the Qwen3-8B student’s Avg@12 trajectory on all three benchmarks throughout training. Table 5 reports the recorded scores at steps 25, 50, 75, and 100, allowing us to examine benchmark-specific trends and the stability of the overall training process. The scores fluctuate only modestly, with no abrupt collapse: HMMT25 improves steadily, while AIME24 and AIME25 remain within comparable ranges. Overall, VISTA maintains a stable performance trajectory over training.

Step

AIME24

AIME25

HMMT25

25 50 75 100

74.4 76.9 78.6 76.4

71.9 69.4 71.9 71.1

43.1 43.9 45.6 48.3

Top-16 positions Mean absolute

Student Adaptation Scale Under Sparse and All-Position Updates

The purpose of this experiment is to show the phenomenon described in the main paper: updating the teacher at all positions, or at too many positions, can make the subsequent student update insufficient. We compare the Qwen38B top-16 and all-position runs at the checkpoints t ∈ {25, 50, 75, 100}. Let q ∈ {16, all} denote the teacher position budget, R ∈ {S, T } the student or teacher, and m ∈ M an adapted LoRA module. The effective LoRA update relative to the frozen base weight is α ∆WR,q,m (t) = BR,q,m (t)AR,q,m (t) r = 2BR,q,m (t)AR,q,m (t).

(13)

with rank r = 64 and scaling α = 128. Since LoRA initializes B(0) = 0, these are changes relative to the common base model, not distances from the preceding checkpoint. Let N be the total number of entries across all adapted modules. We report three base-referenced scale statistics: !1/2 X (2) 2 DR,q (t) := ∥∆WR,q,m (t)∥F , (14) m∈M

1 X (mean) ∥∆WR,q,m (t)∥1 , DR,q (t) := N

(15)

m∈M

(16)

(max)

DR,q (t) := max ∥∆WR,q,m (t)∥max . m∈M

where ∥ · ∥F is the Frobenius norm, ∥ · ∥1 sums entrywise absolute values, and ∥ · ∥max selects the largest entrywise absolute value. We also report the student-to-teacher ratio (2)

ρq (t) :=

DS,q (t) (2)

DT,q (t)

(17)

.

(2)

Student scale DS; q (t)

Thus, D(2) is the global effective-update norm, D(mean) the mean absolute entry, D(max) the largest absolute entry, and ρq the relative student/teacher scale. Top-16 positions

1.6

All positions

+15.7%

1.2 1.0 +3.2%

0.8 0.6 25

50

75

100

Training step

Figure 9: Student effective-weight ℓ2 scale, DS,q (t). (2)

+13.7% 1.00

1.05

1.10

1.15

Relative student update (All positions = 1.00)

Figure 10: Step-100 ratio of top-16 to all-position student (u) (u) effective-weight entry scales, DS,16 (100)/DS,all (100), for u ∈ {mean, max}. Top-16 positions

1:5 £

All positions

1:4 £

1:42 £ 1:31 £

1:3 £ 1:2 £ 1:1 £ 25

50

75

100

Training step

Figure 11: Student-to-teacher effective-weight scale ratio, ρq (t). Results and interpretation. Figure 9 shows that the top16 student has larger D(2) at every checkpoint, with its advantage growing from 3.2% at step 25 (0.637 versus 0.617) to 15.7% at step 100 (1.624 versus 1.404). This widening gap shows that the all-position student accumulates progressively less adaptation as training proceeds. The same pattern appears in the step-100 mean and maximum entry scales, which are respectively 15.2% and 13.7% larger for top-16 (Figure 10), ruling out an effect driven only by the global norm or a few extreme entries. The student-to-teacher ratio is also consistently larger for top-16 (Figure 11); at step 100, ρ16 = 1.42× versus ρall = 1.31×. Thus, relative to how far the teacher has moved, all-position updating leaves the student with less accumulated change. These diagnostics support the claim that all-position teacher adaptation can make subsequent student updates insufficient.

D.3

1.4

+15.2%

Maximum absolute

Scale ratio k¢WS k 2 =k¢WT k 2

Table 5: Qwen3-8B VISTA student Avg@12 (%) by checkpoint.

D.2

All positions

Two-Sided Failure of Teacher Superiority

At each prefix along a student-generated rollout, standard OPSD trains the student to match the teacher’s next-token distribution. This one-way update is beneficial only when the teacher’s next-token distribution remains a better learning target than the student’s current distribution at every step. Reference conditioning alone cannot guarantee this: it can over-support a reference-specific continuation or suppress a valid problem-only alternative. Figures 12 and 13 showcase these two failure modes, respectively, using the actual Qwen3-8B training setup and examples drawn from the

actual training set. Each prefix is generated by the problemonly student; the student and teacher then separately prefill the same prefix and continue generating under their respective contexts. The mismatch is two-sided. In Figure 12, the teacher over-supports a continuation that explicitly invokes the privileged reference, while the student continues from information available in the problem and its own derivation. A one-way student update can then transfer a reference-dependent shortcut into the problem-only policy. In the showcased cases, this dependence appears literally as the teacher opening with “reference solution” or “reference solution said”. The problemonly student nevertheless follows a valid problem-grounded route and reaches the correct answer without access to the reference. In Figure 13, the direction reverses. The student offers a locally coherent problem-only next step, but the teacher assigns it very little probability and favors another continuation. The same forward-KL update can therefore veto a viable alternative, reducing the student’s support for a reasoning path that is available at inference time. In the two showcased cases, the student begins a correct solution with “Add” or “need,” and both problem-only continuations reach the correct answer. The teacher instead opens with “Understand” or “are” and assigns low probability to the student’s first token. These examples make concrete why teacher capability or extra information does not guarantee teacher superiority at every token position, motivating selective teacher adaptation.

(a) Problem

Student Prefix We Student

saw

in

numerator

so

the

the

that:

denominator

becomes:

and Teacher

reference

solution

the

other

so,

total:

Later

that:

terms

can

be

grouped

as

correction:

(b) Problem

A stone is thrown from the ground with speed v at an angle θ to the horizontal. There is no friction and the ground is flat. Find the total distance it travels before hitting the ground. Show that the distance is greatest when

Student Prefix Not Student

Teacher

equal

problem

to

that

This

suggests

not

that

We

now

reference

We

the

Define

-

distance

is

this

suggests

something

the

equation

greatest is

a

is

wrong.

at

when

condition

But

the

." for

it

is

equal

to

1

the

maximum.

try

to

prove

the

condition,

defining

solution .

are

Wait

says:

"Show

when

1!

told

Let's that

said try the

that

this

a

different

maximum

and

expression

the

range

is

maximum,

. equals

1

approach.

occurs

connect

when

it

when:

to

maximum

range.

Figure 12: Reference-conditioned over-support: privileged-information leakage at a shared student prefix. At such a prefix, the privileged teacher may favor a continuation that depends on information unavailable to the problem-only student.

(a) Problem

Seven cookie packages contain 8, 10, 12, 15, 16, 17, and 20 cookies. Find the average number of cookies per package.

Student Prefix

Observed Step

Student

counts:

8,

10,

12,

15,

16,

17,

20

1:

Add

all

the

Then

the

average

numbers

is

.

p_T("Add") = 1.201%

Teacher ?

Understand

the

The

average

First

add

Goal

is

the

the

sum

counts,

of then

all

counts

divide

the

divided result

by by

7. 7.

(b) Problem

1 meter = 3 chi; 1 zhang = 10 chi; 1 kilometer = 2 li. How many zhang are in 1 li?

Student Prefix

Step

1:

Convert

li

to

meters

We Student

need

to

express

1

li

in

meters

p_T("need") = 4.249%

Teacher ?

are

So

told

that:

before

converting

units.

Figure 13: Reference-conditioned under-support: suppression of valid problem-only alternatives at a shared student prefix. At such a prefix, the privileged teacher may assign low probability to a locally coherent continuation available to the problem-only student.

Related documents

• VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation#792321
• VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation#788321
• Data-free On-policy Distillation#1136554
• Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling#1129359
• Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillation#1144087