Conceptio › Archive › arXiv CS

arXiv CSopen access

Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR

Jeonghye Kim1,2∗ ⋄ , Jiwon Jeon2∗ , Dongsheng Li1 , Yuqing Yang1† 1

Microsoft Research 2 KAIST

Abstract Self-distillation has emerged as a powerful framework for post-training LLMs, where a teacher conditioned on extra information guides a student without it, both from the same model. While this guidance is useful when the student has failed, on successful rollouts, the same mechanism instead overwrites the student’s choices and suppresses it’s own reasoning. Therefore, we propose reading the original self-distillation signal in reverse: when the student succeeds along a path the teacher would not have predicted, these tokens reflect its self-driven reasoning. Building on this, we propose RLRT (RLVR with Reversed Teacher), which augments GRPO by reinforcing these tokens on correct rollouts. We interpret this as a new form of exploration in RLVR: not uniform diversity, but valuable exploration grounded in the student’s own success. Across base, instruction-tuned, and thinking-tuned Qwen3 checkpoints, RLRT substantially outperforms selfdistillation and exploration-based baselines, establishing information asymmetry as a new, principled design axis for RLVR.

Introduction GRPO On correct rollouts, upweight teacher-predicted tokens On correct rollouts, upweight self-driven tokens

Base

Mean Avg@16

0.4 0.3 0.2 0.1

Score

arXiv:2605.10781v1 [cs.LG] 11 May 2026

{jeonghye.kim, jiwon.jeon}@kaist.ac.kr, {dongsli, yuqyang}@microsoft.com

Step

100

(a) Training reward on Qwen3-4B-Base by token weighting

80 60 40 20 0

GRPO

SDPO

SRPO

RLSD

+3.4% +18.0%

+12.0%

Qwen3-4B-Base

Qwen3-8B-Base

Qwen3-4B-Instruct

RLRT

+2.3%

Qwen3-8B (Thinking off)

(b) Mean benchmark score across models and methods

Figure 1: Reversing the teacher signal turns self-distillation into valuable exploration. (a) Training reward on Qwen3-4B-Base under GRPO, upweighting teacher-predicted tokens on correct rollouts, and upweighting self-driven tokens (RLRT). (b) Mean avg@16 score over six math benchmarks (AIME24/25/26, HMMT26, AMC23, MATH500) across four Qwen3 backbones. RLRT consistently outperforms baselines significantly. Full results are in Tables 1 and 3. We skipped detailed comparisons with SDPO/SRPO on the base models because they collapsed early during training (Appendix F.2).

Reinforcement learning with verifiable rewards (RLVR) has become the dominant paradigm for post-training LLMs on reasoning tasks [6, 19], yet it suffers from a credit-assignment bottleneck: the only learning signal is a sparse scalar reward at the end of each trajectory. Self-distillation has recently emerged as a powerful response [9, 20, 32, 25]. Its core mechanism is an information asymmetry between two views of the same model: a teacher view conditioned on additional information (rich ∗ Equal contribution. ⋄ Work done during an internship at Microsoft Research. † Corresponding author.

Preprint.

textual feedback, or a successful peer rollout) and a student view without it. By distilling the teacher into the student, this asymmetry converts the sparse scalar reward into dense token-level supervision. However, the value of distilling the teacher into the student depends on whether the rollout was already correct. On failed trajectories, conditioning the teacher on corrective information is useful: the teacher points the student toward solutions it could not previously reach on its own, and distillation transfers that corrective signal token by token. On already-successful trajectories, the same mechanism inverts its role. Even when the student already reached the correct answer, distilling toward the teacher overwrites the student’s choices with the teacher’s, a problem recently identified as optimization ambiguity in self-distillation [12]. Rather than being corrected, the student is forced to imitate a path it had already solved its own way, undermining the independent reasoning that produced the success. This observation motivates us to reverse the direction of self-distillation on correct rollouts. Consider the tokens where the student’s choice differs most sharply from what the teacher would have predicted. On a correct rollout, these are not arbitrary disagreements. They are the very points where the student exercised its own reasoning, choosing against the teacher and still arriving at the correct answer. Such tokens carry the student’s self-driven reasoning: choices that succeeded despite going against the teacher. Therefore, rather than suppressing them by aligning the student to the teacher, we propose to amplify these self-driven tokens during training. In this way, self-distillation becomes a tool for strengthening the student’s reasoning ability, rather than reducing it to imitation. This perspective also suggests a new angle for tackling the loss of reasoning diversity, a persistent failure mode of RLVR in which probability mass concentrates on trajectories the policy already prefers [29]. Existing methods address this through token-level entropy regulation [4, 18, 7] or sequence-level diversity objectives [8, 23, 21], broadening exploration in the hope that wider sampling will surface correct paths. However, they treat diversity as a uniform target, leaving the RL signal to decide which alternative choices are worth keeping. We take a different stance. Rather than encouraging diversity for its own sake, we identify, within the rollouts the model has already produced, tokens that are simultaneously self-driven (departing from the conditioned teacher) and verified (occurring on correct trajectories), and upweight them during training. This yields what we term valuable exploration: diversity grounded in successful reasoning rather than surface variation. Building on this, we propose RLRT (RLVR with Reversed Teacher), which augments GRPO by reversing the direction of self-distillation on correct rollouts: instead of pulling the student to imitate the teacher, RLRT amplifies the self-driven tokens where the student reasoned differently from the teacher and still reached the correct answer. As shown in Figure 1, across Qwen3-4B/8B-Base, Qwen3-4B-Instruct, and Qwen3-8B, RLRT exhibits faster training-score growth and outperforms self-distillation baselines by an average of 8.9% on six math reasoning benchmarks, including the challenging AIME and HMMT. We summarize our contributions as follows: • A new analysis. We reinterpret the teacher–student gap on correct rollouts: prior self-distillation reads it as an alignment target pulling the student to imitate the teacher, whereas we show that, read in reverse, it localizes the student’s own self-driven reasoning. • A new algorithm. Guided by this analysis, we propose RLRT, which augments GRPO by amplifying these self-driven tokens on correct rollouts, yielding consistent gains over strong RLVR baselines across base, instruction-tuned, and thinking-tuned models. • A broader implication. Beyond a specific algorithm, our findings establish information asymmetry as a principled, intrinsic source of valuable exploration, offering a new design axis for RLVR.

Related Works

2.1

Self-Distillation in LLM Post-Training

A growing line of work improves LLM reasoning through information asymmetry within a single model acting as both teacher and student, where the teacher is conditioned on privileged context. This context takes diverse forms: ground-truth reasoning traces [32], runtime errors or judge evaluations as textual feedback [9, 13], second-turn revisions conditioned on critiques [22], expert demonstrations [20], and prepended in-context knowledge or system prompts [27]. Across these variants, the design intent is alignment: the teacher–student gap is used to pull the student toward the teacher, whether by matching distributions [32, 20, 27], distilling improved second-turn behavior into single2

turn [22], weighting tokens by the magnitude of teacher influence under verifiable rewards [25], or restricting alignment to failed rollouts only [12]. RLRT shares this asymmetric setup but inverts the alignment intent altogether: rather than pulling the student toward the teacher, we use the teacher–student gap in the opposite direction, treating tokens where the student diverged from the teacher on correct rollouts as evidence of self-driven reasoning, that is, choices made against the teacher’s prediction that nonetheless reached the correct answer. 2.2

Reasoning Exploration and Diversity

RLVR is widely observed to suffer from reasoning boundary collapse, where the policy concentrates on a narrow set of high-reward strategies rather than expanding its reasoning capacity [29, 17, 26]. Existing remedies broaden output diversity at two scales: token-level entropy regulation [4, 18, 24, 7, 3, 10] and sequence- or outcome-level objectives over full reasoning traces [8, 23, 21, 2, 5]. Both treat diversity as a uniform target and rely on local stochasticity or heuristic proxies such as embedding similarity, n-gram overlap, or outcome counts, capturing surface variation rather than meaningful reasoning differences. RLRT takes a different route. Rather than treating diversity as a uniform target, it identifies, within already-correct rollouts, the specific tokens at which the student departed from the teacher and yet still reached the correct answer, yielding valuable exploration: diversity grounded in the student’s own successful reasoning rather than heuristic surrogates of variation.

Preliminaries

Notation. Let x be a prompt and y = (y1 , . . . , yT ) a response from policy πθ , with prefix y<t := (y1 , . . . , yt−1 ) and suffix y>t := (yt+1 , . . . , yT ). We write ht := (x, y<t ) for the prefix history, R ∈ {0, 1} for the verifiable reward, and V for the vocabulary. Self-distillation in RLVR. In RLVR with self-distillation, a single model serves as both student and teacher: the student conditions only on ht , while the teacher additionally conditions on a privileged context c (e.g., the ground-truth solution or a successful rollout) hidden from the student [32, 9, 25]. We write PSt (·) := πθ (· | ht ), PTt (·) := πθ (· | ht , c), (1) yielding a token-level log-probability ratio ∆t := sg(log PTt (yt ) − log PSt (yt )) , which measures how much the privileged context c revises the model’s belief about token yt , with sg(·) denoting stop-gradient. Distribution-matching approaches such as on-policy self-distillation (OPSD) [32] use ∆t to drive PSt toward PTt directly. RLSD [25] observes that distribution matching is ill-posed when the student lacks access to c, since the target conditions on c while the student does not. To avoid this, RLSD repurposes the ratio as a magnitude-only credit signal, yielding the RLSD update t sign(A) PT (yt ) wtRLSD = exp sign(A) · ∆t = , (2) PSt (yt ) where A is the group-relative advantage. The sign(A) exponent ensures direction-aware credit assignment: on correct rollouts, tokens with PTt > PSt are amplified (the teacher favors them); on incorrect rollouts, the same tokens are attenuated. Thus, the verifiable reward determines the sign of the update, while the teacher only modulates magnitude across tokens within a trajectory.

Motivation

In RLVR, meaningful reasoning gains come not from rollouts that merely reach the correct answer, but from those that arrive there through novel paths, ones that diverge from the model’s prior reasoning patterns. The teacher–student setup above provides a natural lens for identifying such moments. On correct rollouts, the tokens at which the student departs from the teacher are not merely mistakes to be suppressed, but signs of self-driven reasoning. More formally, we identify self-driven reasoning with tokens at which the student deviates from the teacher’s predictive distribution in ways influential 3

to reaching the correct answer. Such tokens are what push the student toward stronger reasoning, and in this section we discuss how to detect and reinforce them. 4.1

Information Asymmetry as an Exploration Signal

To analyse self-driven reasoning, we define the token-level information asymmetry D̂t at a sampled token yt and the position-level information asymmetry D̄t as its expectation under the student: D̂t (yt ) := log

PSt (yt ) , PTt (yt )

D̄t := Ev∼PSt [D̂t (v)] = KL PSt PTt .

(3)

We claim that D̄t flags which positions matter, while the sign of D̂t marks in which direction the policy should update.

Figure 2: Critical positions and explore/exploit directions. Token shading shows the position-level asymmetry D̄t = KL(PSt ∥ PTt ). At each critical position (right panels), candidate tokens are taken as the union of the teacher’s and student’s top-100 tokens; we display the top four with the largest PSt − PTt (green, D̂t > 0) and the top four with the largest PTt − PSt (pink, D̂t < 0). Figure 2 illustrates D̄t and D̂t on a reasoning trajectory. Most tokens have small D̄t , but a few high-asymmetry tokens mark critical positions where token choice strongly affects the outcome. At these positions, candidates the teacher would have predicted (D̂t < 0, e.g., use, conclude) define the exploit direction, while candidates the student chose against the teacher’s prediction (D̂t > 0, e.g., try, consider) define the explore direction. Additional rollouts exhibiting the same pattern are provided in Appendix E. In the following subsections, we examine D̄t and D̂t in more detail. 4.2

D̄t Identifies Which Positions Matter

Claim. The position-level information asymmetry D̄t is large precisely at positions where the choice of token meaningfully affects the probability of a correct outcome. Theoretical Justification. We justify the claim through a Bayesian view of the teacher. We model the teacher as πθ conditioned on the event R = 1 (success), so that the student and teacher distributions become PSt (·) := πθ (· | ht ),

PTt (·) := πθ (· | ht , R = 1).

(4)

For each token v ∈ V, let f (v) :=

Pr [R = 1 | ht , yt = v],

Y ∼PS

f¯St := Ev∼PSt [f (v)],

denote the per-token correctness probability and its student-mean. Bayes’ rule then yields a single identity that underlies the analysis below. Lemma 1 (Bayesian teacher). At each step t, PTt (v) =

PSt (v)f (v) f¯t S

⇐⇒ D̂t (v) = log f¯St − log f (v).

The proof is deferred to Appendix C.1. The teacher is the student tilted toward tokens with higher f (v); equivalently, D̂t (v) measures how far f (v) falls below f¯St . 4

In RLVR, any policy update at position t acts only on tokens the student actually samples, so the relevant signal is how much f varies among such tokens. We call this the influence of position t: Inf S (t) := Ev∼P t f (v) − f¯t . (5) S

A position is critical when Inf S (t) is large and inert when near zero. While D̂t (yt ) acts pointwise, its student-expectation D̄t = KL(PSt ∥ PTt ) from (3) captures the per-position effect of reweighting. The two scales are tied by a Pinsker-type bound. Theorem 2 (D̄t controls Inf S (t)). At every step t, Inf S (t)2 ≤ 2 D̄t . By contrapositive, D̄t ≈ 0 implies Inf S (t) ≈ 0: small asymmetry guarantees an inert position. The proof bounds Inf S (t) by total variation distance using Lemma 1, then applies Pinsker’s inequality (Appendix C.2). Sign of D̂t Identifies Which Direction to Push

4.3

At a critical position, the sign of D̂t (yt ) determines which way to push. Two regimes follow directly from the definition D̂t (v) := log PSt (v) − log PTt (v): • D̂t (v) < 0: the token v is more likely under the teacher (PTt > PSt ), a choice the teacher would have predicted. Reinforcing such tokens follows the teacher’s path, the exploit direction. • D̂t (v) > 0: conversely, v is a choice against the teacher’s prediction (PSt > PTt ). Reinforcing such tokens moves the student onto a self-driven path consistent with success, the explore direction. While the analysis above defines the teacher through the abstract event R = 1, this event cannot be conditioned on directly. In practice, we realize the teacher by feeding a known correct solution c as the conditioning context, so that PTt (·) = πθ (· | ht , c) serves as one instantiation of πθ (· | ht , R = 1).

log (explore / exploit) log10(count)

To verify that the sign of D̂t captures the explore/exploit direction, we ask which tokens the student systematically chooses against the teacher’s prediction versus which tokens align with it across rollouts from Qwen3-8B on DAPO-Math-17k [28]. We score each token’s polarization between the two sides with the smoothed log-odds z-score of Monroe et al. [16]. Figure 3 shows that exploreleaning tokens open new reasoning paths (wait, another, consider), while exploit-leaning tokens close them with verdicts and conclusions (conclude, correct, final). Full details of the marker selection and the per-category list are provided in Appendix D. 4 3 2

EXPLOIT (teacher > student) incorrect yield 6

EXPLORE (student > teacher)

different wait final but write valid right another result possibly contradiction conclude 2 0 2 4 log (explore / exploit)

think

maybe

5 0 5

ct ld ct ly ly e e lt id s e al n e ht n er ut ly k nt er e te te e ly n it m k e rre yiecorrentual final ntinuermin resu val thunclud finclusiodefin rig olutionsid b ossib checffere noth nalyz wridenoecausativedictio wa hm thinmayb o c s co p b tern ntra di a a co det in co con eve al co

Figure 3: Reasoning markers in the explore/exploit population. (a) Volcano scatter of linguistic tokens: x-axis is the polarization log2 (nexplore /nexploit ), y-axis is the total count log10 (nexplore + nexploit ). Highlighted points (green = explore, pink = exploit) are categorized discourse markers; grey points are uncategorized tokens. (b) Per-marker polarization for these markers, sorted from most exploit-leaning (left) to most explore-leaning (right).

RLRT: RLVR with Reversed Teacher

We now present RLRT (RLVR with Reversed Teacher), an instance of the framework in Section 4 that uses an informed teacher and amplifies, on correct rollouts, tokens with D̂t > 0. RLRT modifies only 5

the token-level credit assignment of standard GRPO [19], leaving the rollout, reward, and trust-region machinery unchanged. Figure 4 provides a conceptual illustration and the training pipeline of RLRT. Rollout

Student

Update

...

y1|y|

R1 = 0

y21

y22

...

y2|y|

R2 = 0

yn1

yn2

yn|y|

Rn = 1

yK1

yK2

yK|y|

RK = 1

A = (R - μ) / σ

...

x An

wn1

wn2

wK1

wK2

...

1 wn|y|

...

Prompt x

...

GRPO

Advantage

...

y12

...

Policy

πθ

Token-Level Advantages

y11

...

wK|y|

RLRT weight = PS/PT Teacher Policy

πref

Successful rollout

Prompt x +

(a) Conceptual illustration.

yK1

yK2

PS( yK|y| ) = πθ( yK|y| ∣x, yK1

yK2 ... yK|y|-1 )

PT( yK|y| ) = πθ( yK|y| ∣x, yK1

yK2 ... yK|y|-1 , Successful rollout )

... yK|y|-1

(b) Training pipeline.

Figure 4: Overview of RLRT. (a) Conceptual illustration of the reversed-teacher signal. (b) Given a prompt x, the student policy πθ generates K rollouts that receive verifiable rewards r ∈ {0, 1} and group-standardized advantages A(k) . A reversed teacher provides token-level signals D̂t that, on correct rollouts, up-weight tokens with D̂t > 0. Reverse Weight as Token-Level Information Asymmetry Credit. For a prompt x, the student (k) policy πθ samples a group of K rollouts {y (k) }K )∈ k=1 , each receiving a verifiable reward r(y {0, 1} and a group-standardized advantage A(k) . RLRT defines a per-token reweighting based on D̂t : t sign(A) PS (yt ) . (6) wtRLRT = exp sign(A) · D̂t = PTt (yt ) On positive-advantage tokens, wtRLRT > 1 exactly when D̂t > 0, i.e., for tokens the student chose against the teacher’s prediction, and the reweighting amplifies these self-driven choices rather than aligning the student to the teacher. The flipping of the teacher/student ratio relative to the RLSD update [25] (Eq. 2) reflects a difference in intent: RLSD treats teacher–student disagreement as a correction to be applied, whereas RLRT treats it as a signal of valuable exploration and amplifies it. Reward-Gated Update. Following the framework’s requirement that token-level information asymmetry be combined with outcome conditioning to target self-driven tokens on correct trajectories, the reverse weight is applied only to correct rollouts: h ( i A(k) · (1 − λ) + λ · clip wtRLRT , 1 − εw , 1 + εw if r(y (k) ) = 1, RLRT,(k) At = (7) A(k) if r(y (k) ) = 0, where λ ∈ [0, 1] controls the strength of the reversed signal (λ = 0 recovers vanilla GRPO, λ = 1 yields full reverse weighting), and the clip εw bounds the per-token advantage perturbation by λ · εw .

Experiments

We design our experiments to verify that RLRT effectively leverages the information asymmetry signal to induce valuable exploration during RLVR training. Concretely, we ask: • (Q1) How does RLRT, which pushes the student away from the teacher on correct rollouts, perform compared to self-distillation methods that pull the student toward the teacher? • (Q2) Does D̄t causally identify critical positions, and does RLRT amplify their effect? • (Q3) Beyond sharpening the base’s confident predictions, does RLRT introduce meaningful change? • (Q4) Does RLRT induce more effective exploration than prior exploration-based methods? 6.1

Benchmark Results

Experimental Setup. To answer (Q1), we use DAPO-Math-17k [28] as the training corpus. Since post-training dynamics depend strongly on the pretrained checkpoint’s inductive biases [31, 30], 6

Qwen3-4B-Base GRPO

SDPO

0.4

RLSD

Score

Step

100

0.2 0

Qwen3-4B-Instruct

SRPO

0.4

0.2 0

Qwen3-8B-Base

0.80 50

Step

100

RLRT

0.50

0.75 25

Qwen3-8B

SRPO (w/o dynamic weighting) 0.75

Step

100

0.25 0

Step

100

Figure 5: Training score curves across four backbones (Qwen3-4B-Base, Qwen3-8B-Base, Qwen3-4B-Instruct, Qwen3-8B). RLRT achieves faster exploration and higher training scores in all settings.

Table 1: Performance comparison across mathematical reasoning benchmarks. We report avg@16 and pass@16 for each benchmark. ∆ denotes the gain of RLRT over the best of the other methods. Due to space constraints, results for Qwen3-4B-Instruct are in Table 3 (Appendix F.1). Method

AIME24

AIME25

AIME26

HMMT26

AMC23

MATH500

Avg@16 Pass@16 Avg@16 Pass@16 Avg@16 Pass@16 Avg@16 Pass@16 Avg@16 Pass@16 Avg@16 Pass@16

Qwen3-4B-Base GRPO RLSD RLRT (Ours) ∆ vs. best

9.6 15.0 13.3 22.5 +7.5

33.3 40.0 40.0 50.0 +10.0

6.9 14.4 11.2 18.5 +4.1

30.0 33.3 33.3 36.7 +3.4

6.5 12.3 9.0 19.8 +7.5

16.7 36.7 26.7 40.0 +3.3

3.6 10.0 6.2 15.9 +5.9

24.2 27.3 27.3 33.3 +6.0

43.3 58.3 55.2 63.9 +5.6

90.0 87.5 82.5 95.0 +5.0

66.8 80.2 77.9 83.8 +3.6

92.2 94.2 91.2 94.2 0.0

Qwen3-8B-Base GRPO RLSD RLRT (Ours) ∆ vs. best

10.4 19.8 17.3 27.9 +8.1

33.3 40.0 40.0 63.3 +23.3

10.2 17.5 15.0 18.8 +1.3

30.3 36.7 33.3 50.0 +13.3

9.8 16.5 13.5 21.9 +5.4

30.0 36.7 36.7 53.3 +16.6

5.3 11.0 8.1 15.9 +4.9

30.3 33.3 27.3 33.3 0.0

56.3 62.5 64.2 67.3 +3.1

85.0 90.0 87.5 97.5 +7.5

74.4 83.6 80.9 84.4 +0.8

93.0 95.4 92.4 95.6 +0.2

Qwen3-8B (Thinking off) GRPO SDPO SRPO RLSD RLRT (Ours) ∆ vs. best

25.2 70.2 26.9 15.4 65.4 70.6 +0.4

63.3 86.7 63.3 26.7 83.3 93.3 +6.6

20.0 59.4 22.3 9.8 57.9 62.9 +3.5

43.3 83.3 40.0 26.7 83.3 86.7 +3.4

15.4 62.9 14.4 8.3 57.7 65.0 +2.1

50.0 86.7 36.7 26.7 83.3 86.7 0.0

20.3 41.7 18.6 9.7 39.2 43.2 +1.5

33.3 66.7 30.3 21.2 51.5 69.7 +3.0

67.0 93.6 72.8 49.2 93.6 95.5 +1.9

95.0 100.0 95.0 77.5 100.0 100.0 0.0

83.7 94.8 82.2 75.0 93.1 94.9 +0.1

95.8 98.2 94.4 90.2 98.2 98.2 0.0

we evaluate on three qualitatively distinct model types: a base model (Qwen3-4B/8B-Base), an instruction-tuned model (Qwen3-4B-Instruct), and a thinking-tuned model (Qwen3-8B). We compare RLRT against GRPO and three self-distillation baselines, SDPO [9], SRPO [12], and RLSD [25]. We adopt SDPO rather than the closely related OPSD [32], since OPSD relies on groundtruth solutions from an external dataset and on a hybrid setup in which the student runs with thinking disabled and the teacher with thinking enabled. SDPO instead operates entirely on the model’s own rollouts, consistent with our self-distillation setup. Details of each algorithm are provided in Appendix G.1. In addition, SDPO collapsed early on Qwen3-4B/8B-Base (Appendix F.2), so we omit a detailed comparison for base models. We use a training batch size of 256, a PPO mini-batch size of 128, and a maximum response length of 20,480 tokens, with asymmetric clipping εhigh =0.28 and εlow =0.2 following Yu et al. [28]. Further hyperparameters are listed in Appendix G.2. Performance Comparison. Figure 5 shows the training curves for each algorithm, and Table 1 presents the evaluation results of the trained models on six math benchmarks using avg@16 and pass@16. As shown in Figure 5 and Table 1, across all four backbones, RLRT substantially outperforms both GRPO and the self-distillation baselines, exhibiting faster training-score growth and yielding significant average benchmark gains of 18.0% (Qwen3-4B-Base), 12.0% (Qwen3-8B-Base), 3.4% (Qwen3-4B-Instruct), and 2.2% (Qwen3-8B) over the baselines. Notably, SRPO, which routes correct rollouts to GRPO and incorrect rollouts to self-distillation, performs even worse than full selfdistillation on math. We conjecture that self-distillation and GRPO promote different reasoning styles (e.g., exploration and exploitation as discussed in Section 4.3), leading to conflicting gradients. The gain is largest on Qwen3-4B-Base and smallest on Qwen3-8B, suggesting that RLRT’s exploration signal is most effective when the policy has not yet been concentrated by instruction tuning. 7

6.2

Causal Intervention via Reflection Injection

We answer (Q2) by injecting the reflection prompt “Wait, let me reconsider.” at a chosen token in a rollout and letting the model continue: if high-D̄t tokens are truly critical branch points, this should flip outcomes there more often than elsewhere. We run this on 100 DAPO-Math-17k problems (8 rollouts each) across Qwen3-8B checkpoints from step 0 (base) to step 100 under both RLRT and GRPO, injecting at three positions: arg maxt D̄t (max_kl), a uniform-random token (random), and arg mint D̄t (min_kl). On the hard (ncorrect ∈ {0, 1, 2}) and easy ({5, 6, 7}) subsets, we report flip→R (wrong→right) and flip→W (right→wrong) rates, respectively.

flip rate

Two findings emerge from Fig. 6. First, on the untuned method: Base GRPO RLRT condition: max_kl random min_kl checkpoint (step 0, ■), flip→R at max_kl is twice that at 40% 40% random or min_kl, confirming Section 4.2’s claim that D̄t marks positions causally affecting correct outcomes. 20% 20% The absence of a comparable flip→W spike (panel b) re0% 0% flects that the reflection prompt is biased toward correct0 20 60 100 0 20 60 100 training step training step ing errors, though max_kl remains higher than random (a) flip R (b) flip W and min_kl. Second, the two algorithms diverge with training: RLRT amplifies the max_kl flip→R gain from Figure 6: Reflection injected at max_kl (■), ∼18% to over 40% by step 100, while GRPO lets it random (•), or min_kl (▲). (a) flip→R on collapse toward random and min_kl. RLRT’s flip→W hard subset; (b) flip→W on easy subset. declines just like GRPO’s, so these gains do not come at the cost of fragility on correct rollouts. This explains RLRT’s edge: its D̄t -weighted updates concentrate exploration credit on these critical positions, whereas GRPO spreads it across mostly inert tokens. 6.3

Does RLRT Lead to More Meaningful Distributional Shifts?

To answer (Q3), we analyze where and how each fine-tuned policy’s next-token distribution πft diverges from the base policy πbase , following Meng et al. [15]. We focus on hard prompts (ncorrect ∈ {0, 1, 2} out of 8 under πbase ) so that any shift reflects how the policy learns to improve on cases the base struggles with, and use 30 such prompts from DAPO-Math-17k. At each token position along a fine-tuned rollout, we measure Jensen–Shannon divergence JS(πft ∥ πbase ), and call positions with JS > 0.1 high-divergence: these are the tokens where πft has changed its mind relative to πbase . GRPO

RLSD

0.1

| |/k

P(JS > x)

RLRT

Fraction

1.0

10 1

0.5

10 4 0.0

0.2

0.4

0.6

JS divergence (a) JS divergence (CCDF)

0.0

k (b) Top-k overlap (JS > 0.1)

0.0 <0.0001 <0.001 <0.01

<0.1

base prob. of ft top-1

Figure 7: Token-level distributional shifts of πft relative to πbase . (a) CCDF of JS(πft ∥πbase ) across all positions; dashed line marks the JS > 0.1 threshold for (b)–(c). (b) Top-k overlap |top-k(πft ) ∩ top-k(πbase )|/k at high-divergence positions (k ∈ [1, 20]): how much the candidate set is reshuffled. (c) Fraction of highdivergence positions whose new top-1 token had base probability below each threshold: how deep into the tail.

The three panels in Figure 7 answer three questions about the shift: • (a) How often does the policy diverge from the base? Panel (a) shows the fraction of positions with JS divergence above threshold x. GRPO and RLSD stay close to πbase at most positions, while RLRT places far more positions in the high-divergence regime. • (b) When it diverges, do new tokens enter the top candidates, or are existing ones re-ranked? Panel (b) measures top-k overlap between πft and πbase at high-divergence positions. GRPO and RLSD retain ∼ 80% of πbase ’s candidates even at k ≥ 3, re-weighting the existing pool. RLRT drops to ∼ 50% at k=20, indicating many top candidates are tokens the base did not surface. • (c) How extreme are these new candidates? Panel (c) reports the fraction of high-divergence positions whose new top-1 token had πbase -probability below each threshold. RLRT promotes tokens with base probability under 10−3 to top-1 over 10× as often as the others, routinely picking tokens the base treated as essentially zero. 8

Together, the three views draw a clear line. GRPO and RLSD sharpen what πbase already prefers, re-weighting its top candidates. RLRT instead reorganizes the candidate set itself, pulling tokens from the base’s tail into top positions: it goes beyond reinforcing what the base knows and produces genuinely new behavior. 6.4

Comparison with Other Exploration Methods

We finally answer (Q4) by comparing RLRT against two representative exploration methods: GRPO with an entropy bonus (GRPO+EB) [3] for token-level entropy regulation, and DIVER [8] for sequence-level diversity.

pass@k

AIME24 AIME26 For each method, we evaluate performance on Qwen3Base GRPO GRPO+EB DIVER RLRT 8B-Base by comparing the pass@k curve for k ∈ 0.8 0.8 {1, 2, . . . , 256} on AIME24 and AIME26. We sample 0.6 0.6 256 responses per problem and compute pass@k using 0.4 0.4 the unbiased estimator of Chen et al. [1]. As shown in Figure 8, GRPO+EB injects only local stochasticity at 0.2 0.2 individual decision points [8] and tracks GRPO closely 1 2 4 8 16 32 64 128256 1 2 4 8 16 32 64 128256 k k across the pass@k curve, even falling below GRPO at small k. DIVER improves on GRPO, most visibly at Figure 8: Pass@k comparison across explolarge k, but its margin remains narrow, suggesting that ration methods on AIME24 and AIME26. its semantic-level diversity heuristic broadens exploration only modestly. RLRT, in contrast, dominates from pass@1 through pass@256, reflecting genuinely broader coverage across reasoning modes rather than within one.

6.5

Ablation Study Training Score

Response Length RLRT-all

10k

0.8 0.75 0

Step

5k 0

Actor Entropy

RLRT

Qwen3-4B-Base GRPO

0.4

0.26

Step

0.24 60 0

RLRT ( w = 0.2)

Step

(a) Reward Gating Ablation (Qwen3-4B-Instruct)

RLRT ( w = 1.0)

0.4

0.2 20

Qwen3-8B-Base

RLRT ( w = 0.5)

40 60 Step

0.2 80 100 0

40 60 Step

80 100

(b) w Ablation (Training Score)

Figure 9: RLRT ablations. (a) reward gating on Qwen3-4B-Instruct: RLRT vs. RLRT-all (no r=1 gating) on training score, response length, and actor entropy. (b) clipping range εw on Qwen3-4B-Base and Qwen3-8BBase, with GRPO as reference.

RLRT without Reward Gating. As described in Section 5, RLRT applies the reverse weight only on correct rollouts (r=1). To isolate the effect of the reward gate, we compare against RLRT-all, which applies the same weight regardless of correctness. As shown in Figure 9 (a), RLRT-all initially tracks RLRT but then diverges: response length and entropy grow unbounded, and training collapses around step 40. This confirms that RLRT’s gain requires restricting the reverse weight to correct rollouts: without the gate, the reverse weight reinforces teacher-divergent tokens on failed rollouts, conflating valuable exploration with spurious divergence. Effect of the Clipping Range εw . The clipping range εw controls how strongly the reverse weight can deviate from 1, and thus how much it reshapes the gradient on correct rollouts. We sweep εw ∈ {0.2, 0.5, 1.0} on Qwen3-4B/8B-Base against a GRPO baseline. Figure 9 (b) shows that tighter clipping keeps the reverse weight near unity and tracks GRPO closely, while looser clipping (εw = 1.0) yields the strongest training score on both backbones. This confirms that RLRT’s gains come from the reweighting itself, not from r=1 filtering alone.

Conclusion

We presented RLRT, which inverts self-distillation on correct rollouts: rather than pulling the student toward a privileged-context teacher, it amplifies tokens where the student diverged from the teacher yet still succeeded. We formalized self-driven reasoning through information asymmetry and 9

demonstrated its effectiveness as an exploration signal both theoretically and empirically. Experiments on base, instruction-tuned, and thinking-tuned Qwen3 yield substantial gains over GRPO, selfdistillation, and exploration baselines. Extending RLRT to noisier rewards, other forms of asymmetry, and broader on-policy distillation beyond self-distillation, where the teacher distribution may come from diverse sources, is left for future work.

Acknowledgments This work was supported by Microsoft Research and in part by grants from the Institute of Information & Communications Technology Planning & Evaluation (IITP), funded by the Korea government (MSIT), under Grant No. RS-2024-00457882 (AI Research Hub Project) and Grant No. RS-2022-II220469 (Development of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones).

References [1] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [2] Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508.10751, 2025. [3] Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 30377–30385, 2026. [4] Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025. [5] Shihan Dou, Muling Wu, Jingwen Xu, Rui Zheng, Tao Gui, and Qi Zhang. Improving rl exploration for llm reasoning through retrospective replay. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 594–606. Springer, 2025. [6] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [7] Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, and Jiawei Chen. Rethinking entropy interventions in rlvr: An entropy change perspective. arXiv preprint arXiv:2510.10150, 2025. [8] Zican Hu, Shilin Zhang, Yafu Li, Jianhao Yan, Xuyang Hu, Leyang Cui, Xiaoye Qu, Chunlin Chen, Yu Cheng, and Zhi Wang. Diversity-incentivized exploration for versatile reasoning. arXiv preprint arXiv:2509.26209, 2025. [9] Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026. [10] Renren Jin, Pengzhi Gao, Yuqi Ren, Zhuowen Han, Tongxuan Zhang, Wuwei Huang, Wei Liu, Jian Luan, and Deyi Xiong. Revisiting entropy in reinforcement learning for large reasoning models. arXiv preprint arXiv:2511.05993, 2025. [11] Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms? arXiv preprint arXiv:2603.24472, 2026. 10

[12] Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Unifying group-relative and self-distillation policy optimization via sample routing. arXiv preprint arXiv:2604.02288, 2026. [13] Zeyuan Liu, Jeonghye Kim, Xufang Luo, Dongsheng Li, and Yuqing Yang. Exploratory memory-augmented LLM agent via hybrid on- and off-policy optimization. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview. net/forum?id=UOzxviKVFO. [14] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=5PAF7PAY2Y. [15] Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, and Jingren Zhou. Sparse but critical: A token-level analysis of distributional shifts in rlvr fine-tuning of llms. arXiv preprint arXiv:2603.22446, 2026. [16] Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16 (4):372–403, 2008. [17] Phuc Minh Nguyen, Chinh D La, Duy MH Nguyen, Nitesh V Chawla, Binh T Nguyen, and Khoa D Doan. The reasoning boundary paradox: How reinforcement learning constrains language models. arXiv preprint arXiv:2510.02230, 2025. [18] Jaesung R Park, Junsu Kim, Gyeongman Kim, Jinyoung Jo, Sean Choi, Jaewoong Cho, and Ernest K Ryu. Clip-low increases entropy and clip-high decreases entropy in reinforcement learning of large language models. arXiv preprint arXiv:2509.26114, 2025. [19] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [20] Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026. [21] Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for llm reasoning. arXiv preprint arXiv:2509.06941, 2025. [22] Yuda Song, Lili Chen, Fahim Tajwar, Remi Munos, Deepak Pathak, J Andrew Bagnell, Aarti Singh, and Andrea Zanette. Expanding the capabilities of reinforcement learning via text feedback. arXiv preprint arXiv:2602.02482, 2026. [23] Zhongwei Wan, Yun Shen, Zhihao Dou, Donghao Zhou, Yu Zhang, Xin Wang, Hui Shen, Jing Xiong, Chaofan Tao, Zixuan Zhong, et al. Dsdr: Dual-scale diversity regularization for exploration in llm reasoning. arXiv preprint arXiv:2602.19895, 2026. [24] Chen Wang, Zhaochun Li, Jionghao Bai, Yuzhi Zhang, Shisheng Cui, Zhou Zhao, and Yue Wang. Arbitrary entropy policy optimization breaks the exploration bottleneck of reinforcement learning. arXiv preprint arXiv:2510.08141, 2025. [25] Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr. arXiv preprint arXiv:2604.03128, 2026. [26] Xinhao Yao, Lu Yu, Xiaolin Hu, Fengwei Teng, Qing Cui, Jun Zhou, and Yong Liu. The debate on rlvr reasoning capability boundary: Shrinkage, expansion, or both? a two-stage dynamic view. arXiv preprint arXiv:2510.04028, 2025. [27] Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. arXiv preprint arXiv:2602.12275, 2026. [28] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. 11

[29] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=4OsgYD7em5. [30] Charlie Zhang, Graham Neubig, and Xiang Yue. On the interplay of pre-training, mid-training, and rl on reasoning language models. arXiv preprint arXiv:2512.07783, 2025. [31] Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: Rl post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912, 2025. [32] Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026.

Limitations and Future Directions

To our knowledge, RLRT is the first to show that reversing the teacher’s signal, rather than aligning to it, can improve RLVR combined with distillation. We propose reading the information-asymmetric signal between teacher and student as a driver of exploration rather than imitation, and provide empirical evidence that this reinterpretation yields consistent gains across diverse model families. However, our setup is limited in two ways: it relies on a self-distillation framework where the teacher and student share parameters, and the experiments are restricted to mathematical reasoning. RLRT opens several directions for future work. One axis is varying the teacher itself: rather than self-distillation, the teacher could be a separate, stronger reasoning model (as in on-policy distillation), or, conversely, a weaker one. A second axis is varying the form of privileged information given to the teacher, e.g., process-level feedback, partial hints, or failed attempts rather than a complete successful rollout. A third axis is characterizing how RLRT behaves under off-policy distillation, in contrast to the on-policy setting we study. A particularly promising direction across these axes is a hybrid that adaptively routes between teacher-guided and self-driven updates depending on the context.

RLRT Algorithm

Algorithm 1 summarizes the full RLRT update. The only structural changes relative to GRPO are (i) the per-token reverse weight in Eq. (6) and (ii) the reward gate in Eq. (7); the rollout, reward, and trust-region mechanisms are otherwise unchanged. Algorithm 1 RLRT: RLVR with Reversed Teacher Require: Student/teacher πθ ; privileged context c for the teacher; prompt x; group size K; mixing λ ∈ [0, 1]; clip radius εw . 1: Sample group {y (k) }K k=1 ∼ πθ (· | x). 2: Compute verifiable reward r(y (k) ) ∈ {0, 1} and group-standardized advantage A(k) for each k. 3: for each trajectory k = 1, . . . , K do 4: if r(y (k) ) = 1 then 5: for t = 1, . . . , |y (k) | do (k) (k) 6: Compute D̂t = log PSt (yt ) − log PTt (yt ) {token-level information asymmetry} (k) 7: wtRLRT ← exp sign(A ) · D̂ {Eq. (6)} t h i RLRT,(k) 8: At ← A(k) · (1 − λ) + λ · clip wtRLRT , 1 − εw , 1 + εw {Eq. (7)} 9: end for 10: else RLRT,(k) 11: At ← A(k) for all t {vanilla GRPO advantage} 12: end if 13: end for RLRT,(k) 14: Update θ with the standard GRPO surrogate using {At }.

Proofs and Supporting Results

C.1

Proof of Lemma 1

By the definition of PTt and Bayes’ rule, PTt (v) = πθ (v | ht , R = 1) =

πθ (R = 1 | ht , yt = v) · πθ (v | ht ) . πθ (R = 1 | ht )

The numerator and denominator simplify using the definitions f (v) := Pr[R = 1 | ht , yt = v], f¯t := Ev∼P t [f (v)] = Pr[R = 1 | ht ], S

together with πθ (v | ht ) = PSt (v), yielding PTt (v) =

f (v) · PSt (v) . f¯t S

Taking logarithms,

log PTt (v) = log f (v) + log PSt (v) − log f¯St .

Applying the definition D̂t (v) := log PSt (v) − log PTt (v) gives D̂t (v) = log f¯St − log f (v).

C.2

Proof of Theorem 2

The proof has two steps: Step 1 expresses Inf S (t) in closed form using Lemma 1 and bounds it by total variation distance; Step 2 applies Pinsker’s inequality. By Lemma 1, f (v) = f¯St · PTt (v)/PSt (v), hence t f¯St t t PT (v) ¯ ¯ f (v) − fS = fS − 1 = PTt (v) − PSt (v) . t t PS (v) PS (v)

Step 1: bound by total variation.

Substituting into the definition of Inf S (t), Inf S (t) = Ev∼PSt f (v) − f¯St X f¯t = PSt (v) · t S PTt (v) − PSt (v) PS (v) v∈V X = f¯St PTt (v) − PSt (v) v∈V

= 2 f¯St · TV PSt , PTt ≤ 2 TV PSt , PTt ,

where the second-to-last equality uses the definition TV(P, Q) := 12 inequality uses f¯St ∈ [0, 1].

v |P (v) − Q(v)|, and the last

Step 2: Pinsker’s inequality. For any two probability distributions P, Q on a common space, Pinsker’s inequality states q 1 2 KL(P ∥ Q).

TV(P, Q) ≤

Applied to P = PSt and Q = PTt , TV PSt , PTt

≤

1 2 D̄t .

Squaring the inequality from Step 1 and applying this bound, 2 Inf S (t)2 ≤ 4 TV PSt , PTt ≤ 4 · 12 D̄t = 2 D̄t , as claimed. 14

Marker Statistics for the Explore/Exploit Reading

Starting from 8 rollouts of Qwen3-8B on each of 100 DAPO-Math-17k problems, we retain one correct and one incorrect trajectory per problem (200 trajectories total). At every position t of each trajectory, we identify two tokens from the entire vocabulary V: arg maxv∈V D̂t (v) (the token most favored by the student over the teacher) is added to the explore corpus, and arg minv∈V D̂t (v) (the token most favored by the teacher over the student) is added to the exploit corpus. Note that these are not the sampled tokens yt ; they are the vocabulary entries where the student–teacher divergence is most extreme in each direction. We score every token type v that appears at least 30 times across the two corpora combined, after restricting to ASCII alphabetic tokens of length 3 to 15 characters. Let ev , xv be the counts of token v in the explore/exploit corpora (totals E, X). We compute polarization with the smoothed log-odds z-score of Monroe et al. [16], δv zv = q , d v) Var(δ

δv = log

ev + α xv + α − log , E − ev + α X − xv + α

(8)

with α = 0.5, where zv ≫ 0 marks reliable explore tokens and zv ≪ 0 marks reliable exploit tokens. We keep tokens with |zv | ≥ 3 (251 explore-side and 171 exploit-side candidates), then remove stopwords using two lists: NLTK English stopwords (198 words) and a domain-specific list (approximately 400 words covering math vocabulary, Greek letters, LaTeX fragments, tokenizer artifacts, English numerals, and generic non-discourse fillers). This yields 38 explore-side and 61 exploit-side markers, all listed in Table 2 with their zv values, ranked by |zv | within each category. Table 2: All explore/exploit markers retained after |zv | ≥ 3 filtering and automatic stoplist removal, grouped by discourse function. zv is the smoothed log-odds polarization score (positive = teachersuppressed; negative = teacher-favored). The “Other” rows list all tokens not mapping to any predefined category (13 explore, 32 exploit). Direction

Explore