TIP: Token Importance in On-Policy Distillation
Yuanda Xu∗† Hejian Sang∗ Zhengze Zhou∗ Ran He∗ Zhipeng Wang
Alborz Geramifard
arXiv:2604.14084v1 [cs.LG] 15 Apr 2026
Abstract On-policy knowledge distillation (OPD) trains a student on its own rollouts under token-level supervision from a teacher. Not all token positions matter equally, but existing views of token importance are incomplete. We ask a direct question: which tokens carry the most useful learning signal in OPD? Our answer is that informative tokens come from two regions: positions with high student entropy, and positions with low student entropy plus high teacher–student divergence, where the student is overconfident and wrong. Empirically, student entropy is a strong first-order proxy: retaining 50% of tokens with entropy-based sampling matches or exceeds all-token training while reducing peak memory by up to 47%; under more aggressive retention, memory savings reach up to 58%. But entropy alone misses a second important region. When we isolate low-entropy, high-divergence tokens, training on fewer than 10% of all tokens nearly matches full-token baselines, showing that overconfident tokens carry dense corrective signal despite being nearly invisible to entropy-only rules. We organize these findings with TIP (Token Importance in on-Policy distillation), a two-axis taxonomy over student entropy and teacher–student divergence, and give a theoretical explanation for why entropy is useful yet structurally incomplete. This view motivates type-aware token selection rules that combine uncertainty and disagreement. We validate this picture across three teacher–student pairs spanning Qwen3, Llama, and Qwen2.5 on MATH-500 and AIME 2024/2025, and on the DeepPlanning benchmark for long-horizon agentic planning, where Q3-only training with 20% of tokens surpasses full-token OPD. Our experiments are implemented by extending the open-source OPD repository https://github.com/HJSang/OPSD_OnPolicyDistillation, which provides the practical training base for reproducing this work and supports memoryefficient distillation of larger models under limited GPU budgets.
1
Introduction
Knowledge distillation transfers capability from a large teacher to a smaller student by training on the teacher’s output distributions [Hinton et al., 2015]. In on-policy distillation (OPD), the student generates its own responses and learns from the teacher’s corrections at each token [Agarwal et al., 2024, Gu et al., 2023]. Since the student generates the context, token importance is a property of the student–teacher state at each position. This raises a direct question: which tokens carry the most useful learning signal? Our central claim is simple. In OPD, informative tokens come from two regions of the token state space: (1) positions with high student entropy, where the student is uncertain and still forming its prediction; and (2) positions with low student entropy but high teacher–student divergence, where the student is confident but misaligned with the teacher. The first region is easy to detect with entropy alone: retaining 50% of tokens with entropy-based sampling already matches or improves all-token training while substantially reducing memory. The second is easy to miss. Under more aggressive retention, entropy-only selection begins to lose a small set of low-entropy, high-divergence tokens—positions where the student is sharply peaked on a continuation that the teacher strongly ∗ Equal contribution. † Correspondence to [email protected]
Preprint.
MATH-500
Baseline (100%)
67.6 69.2 67.2 70.0 69.2
. Basent. 50%nt. 20%SO 50%SO 20% E E
15.3
Entropy 50%
AIME'24
17.5
18.5 15.3
Entropy 20%
16.8 8.8
. Basent. 50%nt. 20%SO 50%SO 20% E E
Soft-OR 50%
AIME'25 9.7
10.8
Soft-OR 20%
11.6 12.0
. Basent. 50%nt. 20%SO 50%SO 20% E E
DeepPlanning
12.2 12.6 12.1 12.6 12.3
. Basent. 50%nt. 20%SO 50%SO 20% E E
Figure 1: Cross-task summary: average accuracy by selection method. Each panel shows one benchmark; bar height is the mean accuracy (mean@16) averaged across three teacher–student pairs for mathematical reasoning (Qwen3-8B→4B, Llama-70B→8B, Qwen2.5-14B→1.5B) and across two teacher sizes (14B, 32B) for DeepPlanning. Methods: Base. = all-token OPD (100%); Ent. 50%/20% = entropy-only top-k selection; SO 50%/20% = Soft-OR (Eq. 5) top-k selection. The dashed line marks the all-token baseline. Soft-OR consistently improves over entropy-only selection on the mathematical reasoning benchmarks and remains competitive on DeepPlanning, confirming that augmenting entropy with divergence recovers the Q3 blind spot without sacrificing Q1/Q2 coverage. disfavors. These overconfident tokens carry dense corrective signal despite being nearly invisible to any entropy-based rule. We organize this picture with TIP, a two-axis taxonomy that crosses student entropy and teacher– student divergence to yield four quadrants (Section 4). Theoretically, we show that entropy is a strong first-order proxy but must conflate “confident and correct” with “confident and wrong,” and that a parameter-free Soft-OR score fixes this blind spot (Section 5). Experimentally, we validate each quadrant across three model families and two task domains, showing that the combined score consistently improves over entropy-only selection on mathematical reasoning and remains competitive on agentic planning, where Q3-only selection is strongest (Section 7). Contributions. 1. We propose TIP, a two-axis taxonomy that organizes token importance by student entropy and teacher–student divergence, requiring no verification labels and no extra computation beyond the standard OPD loss. 2. We prove that entropy is a strong first-order proxy but any entropy-only score is structurally blind to overconfident tokens, and that a parameter-free Soft-OR score fixes this blind spot (Propositions 1–2, Remark 2). 3. We validate the taxonomy across several datasets and model families, and show that Soft-OR consistently outperforms entropy-only selection on mathematical reasoning while remaining competitive on long-horizon agentic planning in DeepPlanning [Zhang et al., 2026], where Q3-only selection is strongest.
2
Related Work
Curriculum learning and importance sampling. The idea that not all training examples contribute equally dates to curriculum learning [Bengio et al., 2009] and self-paced learning [Kumar et al., 2010], which order or weight samples by difficulty. Importance sampling extends this to gradient estimation: Katharopoulos and Fleuret [2018] select mini-batch elements by gradient norm, and Ren et al. [2018] learn per-example weights via meta-gradients. These methods operate at the example level. Our work pushes the granularity to individual tokens within a sequence, where the relevant axes are student uncertainty and teacher–student disagreement rather than a scalar difficulty score. Off-policy vs. on-policy distillation. Classical sequence-level KD [Kim and Rush, 2016] trains the student on teacher-generated sequences (off-policy). On-policy distillation [Agarwal et al., 2024, Gu et al., 2023] instead lets the student generate its own rollouts and applies teacher supervision token-by-token, avoiding the train–test distribution mismatch inherent in off-policy data. Sang et al. 2
[2026] further show that on-policy reverse KL self-distillation can compress lengthy reasoning chains into shorter ones. Because token importance in OPD is determined by the student’s own distribution at each position, it cannot be pre-computed from teacher outputs—it must be assessed online. This makes the choice of which tokens to train on a fundamentally different problem from off-policy sample selection. Response-level selection. Several methods operate at the sequence level: Xu et al. [2026b] select responses at the frontier of student competence, and LION [Jiang et al., 2023] uses quality signals. These approaches select rollouts to train on but treat all tokens within a response uniformly. A complementary question—which we address—is which token within a response carry the most signal. Token-level importance in distillation and RL. In RL, Wang et al. [2025b] showed that highentropy “forking tokens” drive most gradient signal, SPINE [Wu et al., 2025] extends this idea to test-time RL by updating only decision-critical branch points with entropy-band regularization, and Xu et al. [2026a] identified overconfident errors as a critical failure mode. In distillation, AdaSwitch [Peng et al., 2025] switches between teacher and student guidance based on divergence, Entropy-Aware OPD [Jin et al., 2026] adapts the loss based on teacher entropy, SelecTKD [Huang et al., 2025] lets the teacher verify student-proposed tokens via a propose-and-verify procedure and masks or down-weights rejected positions, and Xie et al. [2026], Ganguly et al. [2024] adjust token-level weights via distance metrics. Beyond fine-tuning, EntroDrop [Wang et al., 2025a] shows that dropping low-entropy tokens during pretraining improves generalization under multi-epoch training, providing independent evidence that high-entropy positions carry most learning signal. EDIS [Zhu et al., 2026] further demonstrates that the temporal dynamics of token entropy—not just its magnitude—can diagnose correct vs. incorrect reasoning trajectories. Several concurrent works also explore token-level weighting for distillation or compression [Wang et al., 2020, Tavor et al., 2026, Kim and Baek, 2026]. Our work systematically studies all three signal sources—student entropy, teacher entropy, and teacher–student divergence—within a unified two-axis taxonomy, proves that any entropy-only rule is structurally blind to low-entropy, high-divergence tokens (Proposition 2), and proposes a single parameter-free Soft-OR score that explicitly recovers this region without additional hyperparameters or auxiliary objectives, validated across diverse domains including mathematical reasoning and long-horizon agentic planning.
3
Setup
Let T denote a frozen teacher and Sθ a trainable student over vocabulary V . A prompt x ∼ D is drawn, the student generates a rollout y = (y1 , . . . , ym ) ∼ Sθ (· | x), and the teacher scores each position. The context at position t is ct = (x, y<t ). The standard on-policy distillation loss is: m
L=
1 X DKL (PS (· | ct ) ∥ PT (· | ct )) . m t=1
(1)
We characterize each token position by two quantities, both already computed during training: Student entropy. H PS (· | ct ) ht = ∈ [0, 1]. log |V |
(2)
High ht means the student is uncertain; low ht means it is confident. Teacher–student divergence. δt = DKL (PS (· | ct ) ∥ PT (· | ct )) .
(3)
High δt means the teacher disagrees with the student. This is the per-token loss itself—no extra computation. These two quantities define the plane in which we study token importance. The empirical question of this paper is whether useful training signal concentrates in particular regions of the (ht , δt ) plane. 3
Table 1: Token taxonomy. Classification by student entropy ht and teacher–student divergence δt . Quadrant
ht
δt
Learning role
Q1: High entropy, high divergence Q2: High entropy, low divergence Q3: Overconfident Q4: Solved
High High Low Low
High Low High Low
Correct errors or consolidate fragile knowledge Stabilize underconfident predictions Break systematic confident biases Negligible signal
Q3
t
Teacher-student divergence
Q1 + Q2 student-entropy-visible
Q3 blind spot
disagrees
mid
Q1
Confident + disagrees Overconfident errors
Uncertain + disagrees Densest signal
missed by student-entropy-only
student-entropy-visible
Q4
Q2
Confident + agrees Solved tokens
agrees confident
Uncertain + agrees Stabilization signal
mid
Student entropy ht
uncertain
Figure 2: TIP taxonomy as a two-axis map. Entropy determines whether the student is uncertain or confident; divergence determines whether the teacher agrees or disagrees. Q1 and Q2 are visible to entropy-based methods, while Q3 is the low-entropy blind spot that requires divergence to detect.
4
TIP Taxonomy: A Two-Axis View of Token Importance
We organize token importance along two axes already computed during standard OPD training: student entropy ht and teacher–student divergence δt . Crossing them yields four quadrants (Table 1, Figure 2). The quadrants are highly imbalanced: Q4 accounts for roughly 40–47% of all tokens, Q1 and Q2 together make up 40–52%, and Q3 constitutes only 3–15% across model families and datasets in the experimental setup, yet carries disproportionate corrective signal (Section 7.3; Appendix B.4 gives representative token-level examples, especially for Q1 and Q3).
5
Theoretical Analysis
The taxonomy suggests three predictions: high-entropy tokens should dominate learning (Q1/Q2 ≫ Q4); entropy-only selection should miss a specific class of tokens (Q3); and adding divergence should recover them. We formalize these below and test each one experimentally in Section 7. Specifically, we prove: (1) an oracle token weight favors Q1 > Q2 > Q3 ≫ Q4 (Proposition 1); (2) entropy-only scores are structurally blind to Q3 (Proposition 2); and (3) augmenting entropy with divergence restores coverage of all informative quadrants (Remark 2). 5.1
Oracle Token Weight
We want to identify which tokens most accelerate training. We formalize this as: what per-token weights {wt } minimize the expected loss after one gradient step? Let gt = ∇θ ℓt be the per-token gradient, µ̄t = E[gt ], and define ϕ̄t = ⟨∇L, µ̄t ⟩ and M̄t = E[∥gt ∥2 ]. Under β-smoothness and a token-separable approximation that neglects cross-token covariance terms 4
(Appendix A.1), a weighted step ĝ =
P
t wt gt satisfies the surrogate bound: m X η2 β 2 E[L(θ − ηĝ)] − L(θ) ≲ −η wt ϕ̄t + wt M̄t . 2 t=1
(4)
Proposition 1 (Oracle token weight). The bound is minimized at wt∗ = ϕ̄t /(ηβ M̄t ), with per-token descent ∆∗t = −ϕ̄2t /(2β M̄t ). 2
Indeed, the bound is separable across tokens, so each coordinate minimizes −ηwt ϕ̄t + η 2β wt2 M̄t independently. Differentiating gives −η ϕ̄t + η 2 βwt M̄t = 0, hence wt∗ = ϕ̄t /(ηβ M̄t ). Substituting back yields ∆∗t = −ϕ̄2t /(2β M̄t ). This is an oracle quantity (it depends on the population gradient), but it gives a clear interpretation: informative tokens have gradients that align well with descent without excessive energy. Across the four quadrants: • Q1: Large ϕ̄t (strong correction), moderate M̄t (well-conditioned) ⇒ largest wt∗ . • Q2: Moderate ϕ̄t (mild correction) ⇒ moderate wt∗ . • Q3: Positive ϕ̄t (real corrective signal despite low entropy) ⇒ positive wt∗ . • Q4: Near-zero ϕ̄t ⇒ negligible wt∗ . The qualitative ordering is Q1 > Q2 > Q3 ≫ Q4. Remark 1 (Quadrant ordering). The ordering follows from the structure of each quadrant. At high divergence, the teacher’s correction is misaligned with the student’s current prediction, so ϕ̄t is large and positive. At high entropy, the student’s distribution is diffuse, spreading gradient energy across many vocabulary entries; M̄t is moderate, yielding a favorable ϕ̄2t /M̄t ratio for Q1. At low entropy (ht ≈ 0), the distribution is sharply peaked, so M̄t is small—but Q3 still has positive ϕ̄t because the teacher strongly disagrees, giving wt∗ > 0. Q4 has both small ϕ̄t (teacher agrees) and small M̄t , so wt∗ ≈ 0. In practice, wt∗ is unavailable because it depends on population-level quantities. A natural proxy is student entropy ht , but any such score is structurally blind to Q3: Proposition 2 (Blind spot). Let ŵ(ht ) = f (ht ) be any non-decreasing score with f (0) = 0 (e.g., f (h) = h or f (h) = 1[h ≥ τ ]). Then Q3 tokens—which may have wt∗ > 0—receive ŵ(ht ) ≈ 0. Entropy alone cannot distinguish “confident and correct” (Q4) from “confident and wrong” (Q3). Appendix B.4 illustrates this concretely: Examples 1, 3, and 4 show Q3 tokens with ht < 0.4 that an entropy-only rule would discard, while Examples 2 and 5 show the contrasting high-entropy Q1 cases that entropy-based rules do capture. Since divergence δt is already computed as part of the loss, the natural fix is a score that is nonzero whenever either axis is active. We define the Soft-OR score with min-max normalized inputs ĥt , δ̂t ∈ [0, 1]: st = ĥt + δ̂t − ĥt · δ̂t = 1 − (1 − ĥt )(1 − δ̂t ). (5) This is parameter-free: st is nonzero whenever either entropy or divergence is nonzero, without a tuning coefficient. Remark 2 (Soft-OR fixes the blind spot). For any Q3 token with ĥt ≈ 0 and δ̂t > 0, the entropy proxy gives ŵ0 (ht ) ≈ 0 (Proposition 2), but st ≈ δ̂t > 0. Simultaneously, Q4 tokens (ĥt ≈ 0, δ̂t ≈ 0) remain suppressed: st ≈ 0. Q1 tokens retain the highest scores because both ĥt and δ̂t are large (st ≈ 1). The Soft-OR score therefore tracks the oracle ordering Q1 > Q2 > Q3 ≫ Q4 without requiring ϕ̄t or M̄t . Empirical predictions.
6
Table 2 maps each theoretical result to its experimental test.
Method: Type-Aware Token Selection
Given a retention ratio ρ ∈ (0, 1], we retain the top-ρ fraction of tokens by the Soft-OR score st = ĥt + δ̂t − ĥt · δ̂t (Equation 5): T = TopK {st }m (6) t=1 , ⌊ρm⌋ . 5
Table 2: Theoretical predictions and experimental tests. Result
Prediction
Tested in
Proposition 1 Proposition 2 Remark 2
Q1/Q2 carry the most signal; removing Q4 improves efficiency Entropy-only selection misses Q3 tokens Combined score recovers Q3 and outperforms entropy-only
Section 7.2 Section 7.3 Section 7.4
The training loss is: LTIP =
1 X DKL (PS (· | ct ) ∥ PT (· | ct )) . |T |
(7)
t∈T
Setting δ̂t = 0 recovers entropy-only selection; including δ̂t additionally promotes Q3 tokens. The score is parameter-free and both ht and δt are already computed during standard distillation, so the only extra cost is a min-max normalization and the top-k sort—O(m log m) per rollout, negligible compared with forward and backward passes.
7
Experiments
We now validate each prediction of the taxonomy. Table 2 maps each theoretical result to its experimental test; we proceed from the strongest signal (high-entropy tokens, Section 7.2) to the blind spot (Q3, Section 7.3) to the combined score (Section 7.4). 7.1
Experimental Setup
Models. Three teacher–student pairs across three model families for mathematical reasoning, plus one pair for agentic planning: • Qwen3 Small: Qwen3-8B (GRPO) → Qwen3-4B [Yang et al., 2025] • Llama: Llama-3.3-70B-Instruct → Llama-3.1-8B-Instruct [Grattafiori et al., 2024] • Qwen2.5: Qwen2.5-14B-Instruct-thinking → Qwen2.5-1.5B-Instruct [Qwen et al., 2025] (∼9× capacity gap, reasoning teacher) • Qwen3 Agentic: Qwen3-{14B, 32B} → Qwen3-1.7B [Yang et al., 2025] (all with thinking enabled, trained on agentic planning data) Data and evaluation. For mathematical reasoning, training prompts are from DAPO [Yu et al., 2025], with evaluation on MATH-500 [Hendrycks et al., 2021] (500 problems) and AIME 2024/2025 (30 each). For agentic planning, training data is from DeepPlanning [Zhang et al., 2026], a benchmark featuring multi-day travel and multi-product shopping tasks that require proactive information acquisition, local constrained reasoning, and global constrained optimization; the Qwen3 Agentic pair is trained for 15 epochs. All models are trained with AdamW, a cosine schedule, and reverse KL on student-generated rollouts (lr = 1 × 10−6 for Qwen3 and Qwen2.5; lr = 3 × 10−7 for Llama). 7.2
High-Entropy Tokens (Q1/Q2)
We begin with the simplest test of the taxonomy: if Q1/Q2 tokens dominate learning signal while Q4 tokens are negligible, then selecting by student entropy should preserve most of the benefit of OPD. Table 3 (and Figure 3 in the Appendix) confirm this prediction. Across all three model pairs, retaining 50% of tokens with entropy-based sampling matches or outperforms the all-token baseline on most benchmarks, while memory drops substantially. For Qwen3 Small, MATH improves from 76.7 to 78.6; for Llama, from 71.0 to 74.0. This indicates that many low-entropy tokens are effectively solved (Q4) and mainly dilute the gradient; Appendix B.4 gives token-level intuition for the contrasting high-entropy cases that do carry corrective signal (Examples 2 and 5). At the same time, entropy alone is incomplete. As the retention ratio becomes more aggressive, performance often drops below the full-token baseline, suggesting that useful signal remains in the discarded low-entropy region as Proposition 2 indicates. We test this hypothesis directly in the next section by isolating the low-entropy, high-divergence tokens that entropy-only selection discards. 6
Table 3: Entropy sampling across model pairs. Accuracy (%, mean@16 ± std). Sampling selects tokens with probability pt ∝ ht . Bold marks the best per benchmark. Model pair
Benchmark
100%
50%
20%
10%
Qwen3-8B (GRPO) → 4B
MATH-500 AIME’24 AIME’25
76.7 ± 0.7 21.9 ± 1.2 19.4 ± 1.1
78.6 ± 0.6 23.8 ± 1.3 20.7 ± 1.3
74.1 ± 0.9 22.5 ± 1.1 21.5 ± 1.2
70.8 ± 1.0 21.1 ± 1.2 19.2 ± 1.0
Llama-70B → 8B
MATH-500 AIME’24 AIME’25
71.0 ± 0.7 21.5 ± 1.1 4.9 ± 0.9
74.0 ± 0.8 25.3 ± 1.5 7.5 ± 1.1
73.6 ± 0.7 18.8 ± 1.3 10.0 ± 1.2
72.5 ± 0.8 20.6 ± 1.4 9.3 ± 1.0
Qwen2.5-14B → 1.5B
MATH-500 AIME’24 AIME’25
55.1 ± 0.9 2.4 ± 0.7 2.1 ± 0.9
54.9 ± 0.9 3.3 ± 1.4 1.0 ± 0.5
54.0 ± 0.9 4.6 ± 1.3 1.0 ± 0.6
55.1 ± 0.9 2.7 ± 1.1 1.3 ± 0.8
72.0 41.6 35.8
38.1 32.1 19.7
35.5 31.8 15.0
35.3 31.9 14.9
Peak Mem (GB, Qwen3) Peak Mem (GB, Llama) Peak Mem (GB, Qwen2.5)
Table 4: Training on Q3 (overconfident) tokens only. Accuracy (%, mean@16 ± std) when training exclusively on low-entropy, high-divergence tokens. Q3-only training with <10% of all tokens can nearly match the all-token baseline across model pairs. Model pair
Benchmark
Baseline (100%)
Q3 20%
Q3 10%
Qwen3-8B → 4B
MATH-500 AIME’24 AIME’25
76.7 ± 0.7 21.9 ± 1.2 19.4 ± 1.1
75.7 ± 0.8 20.2 ± 1.1 19.2 ± 1.1
76.1 ± 0.9 21.5 ± 1.3 17.1 ± 1.0
Llama-70B → 8B
MATH-500 AIME’24 AIME’25
71.0 ± 0.7 21.5 ± 1.1 4.9 ± 0.9
71.8 ± 1.0 20.2 ± 1.2 4.5 ± 0.7
70.4 ± 1.1 20.8 ± 1.1 4.1 ± 0.7
Qwen2.5-14B → 1.5B
MATH-500 AIME’24 AIME’25
55.1 ± 0.9 2.4 ± 0.7 2.1 ± 0.9
54.6 ± 0.9 2.5 ± 0.9 3.3 ± 1.4
54.1 ± 0.9 3.3 ± 1.4 0.8 ± 0.4
72.0 41.6 35.8
36.4 32.5 19.7
36.2 32.4 19.6
Peak Mem (GB, Qwen3) Peak Mem (GB, Llama) Peak Mem (GB, Qwen2.5)
7.3
Overconfident Tokens (Q3)
We next test the blind-spot prediction directly by constructing the opposite of entropy-based selection: a selector that prioritizes tokens with low entropy but high teacher–student divergence—exactly Q3 in the taxonomy. Q3 selection procedure. We isolate overconfident tokens using a confidence-weighted divergence score: 1. Compute per-token forward KL: δtfwd = DKL (PT (· | ct )∥PS (· | ct )). 2. Compute per-token student entropy ht (Equation 2) and min-max normalize to [0, 1]: ĥt = (ht − hmin )/(hmax − hmin ). 3. Define confidence as conf t = 1 − ĥt (low entropy ⇒ high confidence). 4. Compute the Q3 score: wtQ3 = δtfwd · conf t . Tokens with high wtQ3 are precisely the positions where the student is highly confident while the teacher strongly disagrees. We use forward KL rather than the reverse KL of the taxonomy axis because forward KL penalizes missing mass—teacher-preferred continuations to which the student assigns near-zero probability—more heavily, making it a sharper detector of overconfidence. In practice, the two divergences are strongly correlated on Q3 tokens, so the choice does not materially change which tokens are selected. Table 4 confirms that the Q3 region carries real corrective signal. For Qwen3, training on only 5.7K overconfident tokens (<10% of all tokens) reaches 76.1 on MATH, versus 76.7 for the full-token 7
Table 5: Main results: Baseline vs. Entropy-only vs. Soft-OR. Accuracy (%, mean@16 ± std). Soft-OR uses st = ĥt + δ̂t − ĥt · δ̂t (Eq. 5) with Top-K selection. Bold marks the best per benchmark. Baseline
Entropy-only
Soft-OR
Model pair
Benchmark
100%
50%
20%
50%
20%
Qwen3-8B → 4B
MATH-500 AIME’24 AIME’25
76.7 ± 0.7 21.9 ± 1.2 19.4 ± 1.1
78.6 ± 0.6 23.8 ± 1.3 20.7 ± 1.3
74.1 ± 0.9 22.5 ± 1.1 21.5 ± 1.2
79.1 ± 0.8 25.7 ± 1.4 21.9 ± 1.2
77.6 ± 0.7 24.5 ± 1.2 23.2 ± 1.2
Llama-70B → 8B
MATH-500 AIME’24 AIME’25
71.0 ± 0.7 21.5 ± 1.1 4.9 ± 0.9
74.0 ± 0.8 25.3 ± 1.5 7.5 ± 1.1
73.6 ± 0.7 18.8 ± 1.3 10.0 ± 1.2
74.7 ± 1.0 26.0 ± 1.4 11.5 ± 1.1
74.2 ± 0.7 21.0 ± 1.5 10.9 ± 1.4
Qwen2.5-14B → 1.5B
MATH-500 AIME’24 AIME’25
55.1 ± 0.9 2.4 ± 0.7 2.1 ± 0.9
54.9 ± 0.9 3.3 ± 1.4 1.0 ± 0.5
54.0 ± 0.9 4.6 ± 1.3 1.0 ± 0.6
56.2 ± 1.2 3.8 ± 1.2 1.5 ± 0.7
55.8 ± 0.9 5.0 ± 1.3 1.8 ± 0.6
Table 6: Top 50% vs. bottom 50% by Soft-OR score. Accuracy (%, mean@16 ± std). “Top” trains on the highest-scoring half by st ; “Bot.” trains on the lowest-scoring half. The bottom tokens carry substantially less signal. Qwen3-8B → 4B
Llama-70B → 8B
Qwen2.5-14B → 1.5B
Benchmark
Top 50%
Bot. 50%
Top 50%
Bot. 50%
Top 50%
Bot. 50%
MATH-500 AIME’24 AIME’25
79.1 ± 0.8 25.7 ± 1.4 21.9 ± 1.2
72.3 ± 0.9 15.5 ± 1.3 12.8 ± 1.2
74.7 ± 1.0 26.0 ± 1.4 11.5 ± 1.1
67.4 ± 1.1 17.2 ± 1.3 2.9 ± 0.8
56.2 ± 1.2 3.8 ± 1.2 1.5 ± 0.7
50.3 ± 1.0 1.5 ± 0.7 0.8 ± 0.5
baseline. For Qwen2.5, Q3-only training matches or exceeds the baseline on several benchmarks. These results validate the taxonomy’s prediction that Q3 tokens are informative despite having near-zero entropy. Appendix B.4 provides concrete examples: a student that repeats a generic variable instead of substituting a concrete value (Ex. 1), an arithmetic computation error (Ex. 3), and a confident variable-level misstep in a derivation (Ex. 4)—all near-zero entropy, all strongly corrected by the teacher. 7.4
Type-Aware Selection (TIP)
The prediction of Remark 2 is that combining entropy with divergence should outperform entropy-only selection by recovering Q3 without sacrificing Q1/Q2. Table 5 and Figure 1 present the comparison, and Appendix B.4 provides concrete token-level examples of the Q1/Q3 behaviors that the combined score is designed to retain. Top vs. bottom tokens by Soft-OR score. A natural sanity check is the complementary experiment: instead of training on the top 50% tokens by Soft-OR score, train on the bottom 50%. If the taxonomy is correct, these tokens should be predominantly Q4 (solved) and carry negligible learning signal. Teacher entropy is uninformative. One might expect teacher entropy to drive useful signal. We find the opposite: teacher distributions are near-deterministic across all model pairs (Llama: mean entropy 0.067, std 0.164; Qwen3: mean entropy 0.031, std 0.055; median token probability ≥0.79). A signal that is nearly constant across positions has negligible discriminative power regardless of how it is incorporated—whether as a token-selection criterion, a loss weight, or a sampling probability. We additionally verify this with a concrete instantiation: an adaptive KL loss that up-weights by teacher entropy provides no consistent improvement over standard sampling (Appendix B.1). The useful axes are the student’s state (ht ) and the student–teacher gap (δt ), not the teacher’s uncertainty. 7.5
Beyond Mathematical Reasoning: Agentic Planning
The preceding experiments focus on mathematical reasoning. To test whether TIP generalizes beyond this domain, we apply it to the DeepPlanning benchmark [Zhang et al., 2026] (Section 7.1). 8
Table 7: Agentic planning on DeepPlanning (Qwen3-1.7B student, thinking-enabled, Avg@16 %). Left: reference and entropy-only methods. Right: Q3-only and Soft-OR. Q3-only 20% surpasses full-token OPD; Soft-OR matches or exceeds entropy-only. Method
Teacher 14B
Teacher 32B
Method
Teacher 14B
Teacher 32B
OPD, all tokens (100%) + Entropy-only 50% + Entropy-only 20%
11.7±0.07 12.1±0.06 11.6±0.07
12.8±0.07 13.1±0.07 12.7±0.06
OPD + Q3-only 20% OPD + Soft-OR 50% OPD + Soft-OR 20%
12.6±0.07 12.0±0.06 12.1±0.06
13.6±0.07 13.1±0.08 12.6±0.07
Setup. Using the Qwen3 Agentic pair described in Section 7.1, we train on 80% of the DeepPlanning Travel Planning tasks and evaluate on the remaining 20%. We report the average accuracy (Avg@16) over 16 samples. Following DeepPlanning, we score each plan by the fraction of personalized hard constraints satisfied; these scores are lower than commonsense scores because personalized requirements (e.g., budget limits, dietary restrictions) are more demanding. Table 7 confirms that the TIP taxonomy generalizes to a fundamentally different domain. Entropybased selection preserves most of the signal: at 50% retention, performance matches or exceeds full-token OPD for both teacher sizes (12.1 vs. 11.7 for 14B; 13.1 vs. 12.8 for 32B). The most striking finding is Q3: training on only 20% of overconfident tokens surpasses full-token OPD for both teachers (12.6 vs. 11.7; 13.6 vs. 12.8), confirming that entropy discards exactly the tokens with the densest corrective signal. This aligns with the structure of agentic tasks: a single wrong but confident commitment—booking a closed venue, violating a budget constraint—can invalidate an entire plan, making Q3 corrections especially concentrated. Appendix B.2 provides Best@16 results on the same queries, confirming the same pattern.
8
Discussion and Conclusion
TIP establishes that token importance in OPD is governed by two axes—student entropy and teacher– student divergence—and that both are necessary. All three theoretical predictions (Propositions 1–2, Remark 2) are supported by experiments across three model families and two task domains (Tables 3– 7). The most revealing finding is the Q3 blind spot. Because entropy-only rules are provably unable to distinguish “confident and correct” from “confident and wrong” (Proposition 2), the student’s own measure of uncertainty systematically undervalues the positions most in need of correction. That fewer than 10% of tokens, selected precisely for overconfidence, nearly matches full-training performance shows how densely corrective signal is concentrated in this overlooked region. The agentic planning results sharpen the picture further: on DeepPlanning, Q3-only training with 20% of tokens surpasses full OPD (12.6 vs. 11.7 Avg@16 with the 14B teacher), suggesting that the value of catching overconfident errors scales with how much downstream computation depends on a single committed step. This effect is weaker in mathematical reasoning, where mistakes are often more locally contained. More broadly, the two-axis framing—student uncertainty and teacher–student gap—applies naturally to other settings where on-policy token-level supervision is used, including RLHF, process reward fine-tuning, and speculative decoding. Limitations. (1) Q3 detection requires teacher output distributions, though δt is already part of the standard OPD loss. (2) Soft-OR uses per-batch min-max normalization, which may be sensitive to outlier tokens in a batch; alternatives such as running-average normalization remain to be studied. (3) All experiments use reverse KL supervision; whether the same quadrant ordering holds under forward KL or JSD is an open question. We hope TIP provides a useful conceptual lens for future work on efficient and targeted training of language models.
References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated 9
mistakes. In International Conference on Learning Representations (ICLR), 2024. URL https: //arxiv.org/abs/2306.13649. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. ICML, 2009. Shreyan Ganguly, Roshan Nayak, Rakshith Rao, Ujan Deb, and Prathosh AP. AdaKD: Dynamic knowledge distillation of ASR models using adaptive loss weighting. arXiv preprint arXiv:2405.08019, 2024. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The llama 3 herd of models. 2024. URL https://arxiv.org/abs/2407.21783. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: On-policy distillation of large language models. CoRR, 2023. URL https://arxiv.org/abs/2306.08543. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. NeurIPS, 2021. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. Haiduo Huang, Jiangcheng Song, Yadong Zhang, and Pengju Ren. SelecTKD: Selective tokenweighted knowledge distillation for LLMs. arXiv preprint arXiv:2510.24021, 2025. Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distillation of proprietary large language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https://arxiv.org/abs/2305.12870. Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079, 2026. Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International Conference on Machine Learning (ICML), 2018. Minsang Kim and Seung Jun Baek. Explain in your own words: Improving reasoning via tokenselective dual knowledge distillation. arXiv preprint arXiv:2603.13260, 2026. Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2016. URL https: //arxiv.org/abs/1606.07947. M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. NeurIPS, 2010. Zhen Peng et al. Adaswitch: Adaptive switching between teacher and student for on-policy distillation. arXiv preprint, 2025. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. 2025. URL https://arxiv.org/abs/2412.15115. Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning (ICML), 2018. Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. CRISP: Compressed reasoning via iterative self-policy distillation. arXiv preprint arXiv:2603.05433, 2026. 10
Almog Tavor, Itay Ebenspanger, Neil Cnaan, and Mor Geva. Rethinking selective knowledge distillation. arXiv preprint arXiv:2602.01395, 2026. Jiapeng Wang, Yiwen Hu, Yanzipeng Gao, Haoyu Wang, Shuo Wang, Hongyu Lu, Jiaxin Mao, Wayne Xin Zhao, Junyi Li, and Ji-Rong Wen. Entropy-guided token dropout: Training autoregressive language models with limited domain data. arXiv preprint arXiv:2512.23422, 2025a. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. Advances in Neural Information Processing Systems, 38, 2025b. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep selfattention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL https://arxiv.org/abs/2002. 10957. Jianghao Wu, Yasmeen George, Jin Ye, Yicheng Wu, Daniel F. Schmidt, and Jianfei Cai. SPINE: Token-selective test-time reinforcement learning with entropy-band regularization. arXiv preprint arXiv:2511.17938, 2025. Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu, and Jiangning Zhang. LLM-oriented token-adaptive knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34070–34078, 2026. Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. Overconfident errors need stronger correction: Asymmetric confidence penalties for reinforcement learning. arXiv preprint arXiv:2602.21420, 2026a. Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. PACED: Distillation and self-distillation at the frontier of student competence. arXiv preprint, 2026b. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, WeiYing Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An open-source llm reinforcement learning system. arXiv preprint arXiv:2503.14476, 2025. Yinger Zhang, Shutong Jiang, Renhao Li, Jianhong Tu, Yang Su, Lianghao Deng, Xudong Guo, Chenxu Lv, and Junyang Lin. DeepPlanning: Benchmarking long-horizon agentic planning with verifiable constraints. arXiv preprint arXiv:2601.18137, 2026. Chenghua Zhu, Siyan Wu, Xiangkang Zeng, Zishan Xu, Zhaolu Kang, Yifu Guo, Yuquan Lu, Junduan Huang, Guojing Zhou, et al. EDIS: Diagnosing LLM reasoning via entropy dynamics. arXiv preprint arXiv:2602.01288, 2026.
11
A
Supplementary Theory
A.1
Derivation of the Descent Bound
Assumption 1 (Smoothness). L(θ′ ) ≤ L(θ) + ⟨∇L(θ), θ′ − θ⟩ + β2 ∥θ′ − θ∥2 . Assumption 2 (Token-separable approximation). For tractability, we neglect off-diagonal gradient interactions across token positions. Concretely, for t ̸= s we treat the centered cross-token covariance E[(gt − µ̄t )(gs − µ̄s )⊤ ] as lower-order, so that the quadratic term admits a token-separable approximation. Derivation. Expand L(θ − ηĝ) via smoothness where ĝ =
P
t wt gt :
η2 β ∥ĝ∥2 . L(θ − ηĝ) ≤ L(θ) − η⟨∇L, ĝ⟩ + 2 P Using linearity of expectation and ĝ = t wt gt , X X X E[−η⟨∇L, ĝ⟩] = −η wt E[⟨∇L, gt ⟩] = −η wt ⟨∇L, µ̄t ⟩ = −η wt ϕ̄t . t
t
(8)
(9)
t
For the quadratic term, E[∥ĝ∥2 ] = E
h X
wt gt
2i
=
X
t
wt2 E[∥gt ∥2 ] +
t
X
wt ws E[⟨gt , gs ⟩].
(10)
t̸=s
Write gt = µ̄t + (gt − µ̄t ). Under Assumption 2, we drop the off-diagonal covariance contribution and treat the remaining mean interaction terms as lower-order; absorbing these into the ≲ notation gives the token-separable approximation X E[∥ĝ∥2 ] ≲ wt2 M̄t . (11) t
Combining the two displays gives E[L(θ − ηĝ)] − L(θ) ≲
X
−ηwt ϕ̄t +
t
η2 β 2 w M̄t . 2 t
(12)
The right-hand side is separable in t, so minimizing each term gives η2 β 2 ∂ −ηwt ϕ̄t + w M̄t = −η ϕ̄t + η 2 βwt M̄t = 0, ∂wt 2 t
(13)
which yields wt∗ = ϕ̄t /(ηβ M̄t ). A.2
Entropy-Weighted Sampling: Coverage and Variance
1 Sampling tokens with pt ∝ ht and using the importance-weighted estimator ĝIS = m unbiased whenever pt > 0, with variance
P
t∈S gt /pt is
m
Var(ĝIS ) =
1 X 1 − pt E[∥gt ∥2 ]. m2 t=1 pt
(14)
This makes the tradeoff transparent: entropy sampling preserves nonzero Q3 coverage, but the variance cost grows as 1/pt for low-entropy tokens. A.3
Why Adding Divergence Improves the Ranking
Under entropy-only scoring ŵ0 = ht , Q3 and Q4 are both pushed to the low-score end. The Soft-OR score st = ĥt + δ̂t − ĥt · δ̂t separates them: Q3 receives positive score from divergence (st ≈ δ̂t ) while Q4 remains near zero (both axes small). Because the product term ĥt · δ̂t prevents double-counting, the high-entropy ranking is preserved and the score better tracks the oracle ordering without any tuning parameter. 12
B
Supplementary Experiments
MATH-500
AIME'24
AIME'25
25
Qwen3-8B 4B Llama-70B 8B Qwen2.5-14B 1.5B
65 60
20
20
Accuracy (%)
70
Accuracy (%)
Accuracy (%)
75
15 10
1.0
0.5
Retention Ratio
0.2 0.1
10 5
5
55
15
1.0
0.5
Retention Ratio
0.2 0.1
0
1.0
0.5
Retention Ratio
0.2 0.1
Figure 3: Entropy sampling across retention ratios. Accuracy (mean@16) on three benchmarks as a function of retention ratio. Retaining 50% of tokens with entropy-based sampling matches or outperforms the all-token baseline across model pairs. At very low retention, entropy-only selection begins to plateau or degrade.
B.1
Adaptive KL Does Not Help
Table 8: Adaptive KL vs. standard sampling (Qwen3-8B (GRPO) → 4B, mean@16 %). Adaptive KL up-weights by teacher entropy; it provides no consistent improvement. Method Loss Baseline Ratio 0.5 Ratio 0.2 Ratio 0.1 MATH-500 Standard Adaptive KL
reverse KL adaptive KL
70.7 —
72.6 72.0
68.1 69.7
64.8 62.6
AIME’24 Standard Adaptive KL
reverse KL adaptive KL
17.9 —
19.8 18.1
18.5 16.5
17.1 16.5
AIME’25 Standard Adaptive KL
reverse KL adaptive KL
15.4 —
16.7 18.5
17.5 17.1
15.2 15.6
MATH-DAPO Standard reverse KL Adaptive KL adaptive KL
42.6 —
43.1 43.0
42.0 42.3
40.4 39.6
Teacher entropy is near-zero everywhere (mean 0.031, std 0.055 for Qwen3; mean 0.067, std 0.164 for Llama), so any scheme that conditions on teacher entropy—whether for token selection, loss weighting, or sampling—receives an almost constant input and therefore adds no discriminative information. In the specific case of adaptive KL, the weight reduces to a near-constant times student entropy, simply double-weighting by entropy (once via sampling, once via the loss). 13
B.2
Agentic Planning: Held-Out Queries
Figure 4: Token selection for agentic OPD on 20% held-out travel-planning queries. Top row: Avg@16; Bottom row: Best@16 (Pass@16). Within each row the left panel uses the 14B teacher and the right panel uses the 32B teacher. Q3-only 20% matches or exceeds the full-token baseline in every setting, consistent with Table 7. Best@16 results show the same pattern: overconfident-token training improves the upper tail of performance, not just the mean. Figure 4 complements Table 7 with a finer-grained view. The Avg@16 panels confirm the maintext findings: Q3-only 20% leads for both teacher sizes (12.6 and 13.6 vs. baselines of 11.7 and 12.8), and entropy-only 50% improves over full-token OPD. The Best@16 (Pass@16) panels show the same pattern: Soft-OR 20% achieves the highest Best@16 with the 14B teacher (20.3 vs. 18.9 baseline), while Q3-only 20% leads with the 32B teacher (20.1 vs. 19.7). This indicates that correcting overconfident tokens expands the frontier of problems the student can solve, not just its average performance. B.3
Hyperparameters
Table 9 summarizes all training hyperparameters. Table 9: Training hyperparameters across model pairs. Qwen3 (8B→4B) Llama (70B→8B) Qwen2.5 (14B→1.5B) Optimizer Learning rate Batch size (rollouts) Rollouts per prompt Max response length Max prompt length OPD chunk size OPD max length Tensor parallel size Generation temperature Top-p (generation) GPUs
AdamW 1 × 10−6 8 16 8192 2048 512 16,384 2 1.0 1.0 8×H200
AdamW 3 × 10−7 8 16 8192 2048 512 16,384 2 1.0 1.0 8×H200
14
AdamW 1 × 10−6 8 16 8192 2048 512 16,384 2 1.0 1.0 4×H200
For entropy sampling experiments, tokens are sampled with probability pt ∝ ht and the specified retention ratio determines the number of tokens kept. For Top-K experiments, the top-ρ fraction of tokens by score is selected deterministically. All evaluations use mean@16 (average accuracy over 16 independent samples per problem) with temperature 1.0. B.4
Qualitative Examples Across Quadrants
We present five representative tokens from training on Qwen2.5-14B → 1.5B, spanning different quadrants of the taxonomy. Each example shows the problem, the student’s response with the target token highlighted , and the student vs. teacher top-5 distributions. Example 1: Generic variable vs. concrete substitution (Q3: low entropy, high divergence). Student entropy: 0.02 (extremely confident). Forward KL: 5.27. Overconfidence score: 5.24. Problem Let P be aset of monic polynomials with integer coefficients of the least degree, with root k · cos 4π 7 , as k spans over the positive integers. Let P (x) ∈ P be the polynomial so that |P (1)| is minimized. Find the remainder when P (2017) is divided by 1000. Student response (excerpt) . . . The polynomial with roots of the form k · cos 4π can be constructed by considering the minimal 7 polynomial of cos 4π . . . 7
Token distributions Student top-5: k (99.8%), x (0.1%), e (0.0%), a (0.0%), z (0.0%) Teacher top-5: 2 (49.9%), k (30.3%), x (8.7%), e (6.8%), z (2.8%)
Analysis. The student assigns 99.8% to k, mechanically repeating the generic variable from the problem statement, while the teacher places 49.9% on the concrete value 2—indicating that the next reasoning step should substitute a specific integer rather than restate the variable. An entropy-only rule would assign near-zero weight to this position because ht = 0.02, yet it carries one of the densest corrective signals in the batch (overconfidence score = 5.24). Example 2: Reasoning fork—restating vs. advancing (Q1: high entropy, high divergence). Student entropy: 1.82 (uncertain). Forward KL: 5.31. Problem An arithmetic sequence of positive integers has n ≥ 3 terms, initial term a, and common difference d > 1. Carl wrote down all the terms correctly except for one term, which was off by 1. The sum of the terms he wrote was 222. What is a + d + n? Student response (excerpt) . . . Given ai = a + k where k = 1 if the term a + kd (for some integer k) was off and 1 if the term a + (k + 1)d was off.. . .
Token distributions Student top-5: off (54.4%), missing (15.6%), the (9.5%), not (4.5%), replaced (3.1%) Teacher top-5: written (40.3%), increased (35.6%), off (7.9%), decreased (2.9%), incremented (2.0%)
Analysis. The student favors “off” (54.4%), restating the problem, while the teacher prefers “written” (40.3%) or “increased” (35.6%)—words that advance the solution by characterizing the error direction. 15
The teacher’s distribution pushes toward more precise mathematical reasoning, while the student’s choice leads to circular rephrasing. This is a classic Q1 token: the student is uncertain (entropy = 1.82) and the teacher strongly disagrees—entropy-based selection would catch this token. Example 3: Arithmetic computation error (Q3: low entropy, high divergence). 0.40. Forward KL: 3.54. Overconfidence score: 3.27.
Student entropy:
Problem Find the sum of the real roots of f (x) = x4 + 9x3 + 18x2 + 18x + 4. Student response (excerpt) . . . f (2) = 24 + 9 · 23 + 18 · 22 + 18 · 2 + 4 = 16 + 72 + 72 + 36 + 4 = 19 0 ̸= 0. . .
Token distributions Student top-5: 0 (91.2%), 8 (4.0%), 2 (3.5%), 4 (0.7%), 6 (0.5%) Teacher top-5: 6 (60.0%), 2 (25.0%), 8 (6.3%), 9 (3.8%), 0 (3.0%)
Analysis. The student confidently writes f (2) = 190 with 91.2% on digit 0, committing an arithmetic error in summing 16 + 72 + 72 + 36 + 4. The teacher distributes probability across 6 (60.0%) and 2 (25.0%), clearly disagreeing with 0. Like Example 1, this is a Q3 token: the student is fairly confident (ht = 0.40), yet the teacher strongly disagrees—entropy-based selection would under-weight this position. Example 4: Confident on the wrong variable (Q3: low entropy, high divergence). Student entropy: 0.12 (very confident). Forward KL: 5.58. Overconfidence score: 5.44. Problem If the three interior angles A, B, C of triangle △ABC have cotangents cot A, cot B, cot C that form an arithmetic sequence, and the maximum value of angle B can be expressed as mπ n , find the value of m + n. Student response (excerpt) sin B . . . Next, consider the cotangent of each angle cot B = cos B , based on the arithmetic sequence property that cot B − cot A and cot B − cot C are equal. . .
Token distributions Student top-5: B (98.2%), (B (1.0%), \’ (0.2%), ˆ (0.2%), C (0.1%) Teacher top-5: A (52.4%), (A (11.7%), C (11.7%), B (6.3%), ( (6.3%)
Analysis. The student assigns 98.2% to B, while the teacher prefers A (52.4%). The student writes sin B cos B cot B = cos B , which is mathematically incorrect (cot B = sin B ); the teacher’s preferred continuation reflects a different and more correct reasoning path. Another clear Q3 token: ht = 0.12 would be invisible to entropy-only selection. Note. The original rollout is in Chinese; the problem and response excerpts above are translated for readability. The token identifiers (B, A, etc.) are unchanged from the raw log, as mathematical symbols are language-invariant. Example 5: Wrong mathematical symbol (Q1: moderate entropy, high divergence). Student entropy: 1.38. Forward KL: 4.27. 16
Problem (Same as Example 4) Student response (excerpt) . . . Assume the common difference between two adjacent terms is equal, i.e., b − a = \ Delta (where ∆ is a constant). . .
Token distributions Student top-5: text (53.5%), Delta (25.3%), frac (13.5%), lambda (1.4%), pm (1.0%) Teacher top-5: cot (91.7%), frac (4.6%), text (1.1%), delta (1.0%), Delta (0.8%)
Analysis. The teacher assigns 91.7% to cot—the mathematically relevant function for this problem— while the student splits probability across irrelevant symbols (text, Delta, frac). This is a Q1 token (moderate entropy, high divergence): unlike Q3 tokens, entropy-based selection would catch it, but the teacher’s near-deterministic preference for cot makes it an especially high-value training signal. Note that Examples 4 and 5 come from the same student rollout at different token positions (same Chinese-language response as Ex. 4; translated above), illustrating how a single response can contain both Q3 and Q1 tokens. Summary. Examples 1, 3, and 4 illustrate why divergence is needed: the student’s entropy is low, so any entropy-only rule would skip these tokens, yet the teacher strongly disagrees—a generic variable where a concrete value is needed (Ex. 1), an arithmetic error (Ex. 3), or a confident variable-level misstep in a derivation (Ex. 4). Examples 2 and 5 show Q1 tokens that entropy-based selection handles well—the student is already uncertain, and the teacher provides a clear corrective signal. The taxonomy’s value is precisely that it identifies both regions as informative while distinguishing them from Q4 (solved) tokens where both entropy and divergence are low.
17