Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training
arXiv:2605.12483v1 [cs.LG] 12 May 2026
Hejian Sang∗ Yuanda Xu∗† Zhengze Zhou∗ Ran He∗ Zhipeng Wang
Alborz Geramifard
Abstract In settings where labeled verifiable training data is the binding constraint, each checked example should be allocated carefully. The standard practice is to use this data directly on the model that will be deployed, for example by running GRPO on the deployment student. We argue that this is often an inefficient allocation because it overlooks a reward-density principle: sparse sequence-level reward should train models where exploration is productive, while dense token-level teacher reward should be used where the aim is to compress behavior into a smaller model. In this view, GRPO-style sparse RL and OPD-style dense teacher supervision are not separate recipes; they are different reward-density regimes. The allocation rule is simple: use scarce labeled training data upstream on the strongest model that can turn it into reward-shaped behavior, then transfer that behavior downstream as dense supervision. We evaluate this rule on verifiable math with Qwen3 and Llama models. At fixed Qwen3-1.7B deployment-student size, an RL-improved 8B teacher distilled through the dense bridge outperforms direct GRPO on the same student (79.3% vs. 75.9% on MATH; 25.2 vs. 19.8 on AIME 2024), while transfer from the same teacher before RL underperforms. The bridge is important: a forward-KL warmup on teacher rollouts followed by OPD on student rollouts is consistently strongest on MATH before any post-bridge student-side sparse RL, and also gives the best preStage 3 AIME endpoints for the canonical 8B/14B teachers. The bridge also makes later student-side sparse RL effective: GRPO that is weak on a cold student lifts MATH from 75.4% to 78.5% after the bridge and outperforms a matched replay control by 2.8 points. The teacher-quality ordering—raw-teacher transfer < direct GRPO < RL-teacher transfer—replicates on Llama-3.1-8B-Instruct with a Llama3.3-70B-Instruct teacher. The operational lesson is to avoid using scarce labeled data on the least prepared policy: use sparse reward for teacher-side discovery, dense transfer for student compression, and student-side sparse reward only after the bridge.
1
Introduction
Labeled training data is the bottleneck of language-model post-training. Pretraining text and teacher rollouts can scale with compute; labeled data for verifiable tasks does not scale so easily. Each example needs a problem with a checkable answer and a grader whose errors will not corrupt the reward. In the Qwen experiments below, the labeled training data comes from DAPO-Math-17K [Yu et al., 2025]. The practical question is therefore not which post-training algorithm is best in isolation, but which model should train on each scarce labeled example. ∗ Equal contribution. † Correspondence to [email protected]
Preprint.
Teacher-side discovery
sparse reward should go to the stronger explorer (teacher discovery)
Dense bridge FKL-to-OPD
teacher logits become dense token signal for student compression
Post-bridge student RL
fresh sparse reward refines the bridged, now-trainable student
Figure 1: Where labeled training data should be allocated. The teacher-side path (Stage 1: teacher RL) discovers reward-shaped behavior; the two-stage dense bridge (Stage 2a: FKL warmup, Stage 2b: OPD) converts it into token-level supervision for the deployment student; the optional post-bridge student RL stage (Stage 3) uses any remaining labeled data on a now-trainable student. The default approach is to train the deployment model directly. If a 1.7B model must do well on MATH, run GRPO on the 1.7B model. This paper argues for a different allocation, and for the simple reward-density principle behind it. The reward-density principle. Sparse task reward and dense teacher log-probabilities sit on the same axis of a KL-regularized policy objective. At one end, ordinary task RL (PPO, GRPO) is sparse: a single sequence-level signal arrives after a long trajectory. At the other end, on-policy distillation (OPD) against a teacher is, as Section 2 recalls, maximum-entropy RL with a dense token-level reward rT (s, y) = β log πT (y | s). Sparse reward is unbiased, but it is useful only when the policy already samples successful trajectories often enough to learn from them. Dense teacher reward is biased toward the teacher, but it provides a signal at every token. A small base model has neither advantage: its rollouts are too weak for sparse reward to teach much, and it has no teacher-shaped distribution to imitate. A larger model can turn the same sparse reward into stronger behavior. The central move is therefore to apply sparse reward where it is informative, then turn the resulting reward-shaped policy into dense supervision for the deployment model. Contributions. We evaluate the reward-density principle on verifiable math and make three contributions: 1. Teacher-first allocation. At fixed deployment-student size, a fixed pool of labeled training data yields a stronger student when it is allocated to teacher RL plus dense transfer than when it is allocated to direct student RL. The gain requires a reward-shaped teacher: transferring the same teacher before teacher-side RL underperforms direct GRPO, so scale alone is not the cause (Section 5.1). 2. A two-stage dense bridge. A forward-KL warmup on teacher rollouts followed by OPD on student rollouts outperforms both teacher-sample SFT and OPD-only transfer. The warmup fixes support mismatch so that the subsequent OPD stage is well-conditioned (Section 5.2). 3. Post-bridge student RL. The bridge changes student trainability: sparse-reward GRPO that is weak on a cold student lifts the bridge endpoint above both direct GRPO and a matched replay control that reuses bridge data (Section 5.3). What this changes in practice. The standard post-training pipeline—SFT, then RL on the deployment model—places the scarce labeled data in the least effective position first. The teacher-first view prescribes a different order: allocate the labeled training data to a model large enough to use it, run a two-stage dense bridge into the deployment model, and only then decide whether any held-out labeled data remains worth using on the student. Figure 1 summarizes the resulting pipeline. Scope. The evidence is on verifiable math (MATH, AIME 2024, AIME 2025) with two studentteacher families: Qwen3-family models [Yang et al., 2025] and Llama-family models [Grattafiori et al., 2024]. In the Qwen block, the deployment student is Qwen3-1.7B and the teachers are raw, SFT-trained, and RL-trained Qwen3-8B/14B checkpoints; in the Llama block, the deployment student is Llama-3.1-8B-Instruct and the teacher is Llama-3.3-70B-Instruct. OPD requires a shared tokenizer; 2
“cross-family validation” below means that the recipe is run separately within each family, not that logits are transferred across vocabularies. Terminology. A sparse reward is a sequence-level task reward R(x, y) available only at the end of a trajectory. A dense reward is the token-level teacher signal rT (st , yt ) = β log πT (yt | st ). OPD is reverse-KL distillation on student rollouts. The two-stage bridge (or FKL-to-OPD) is forward-KL on teacher rollouts followed by OPD on student rollouts. Stage 1 is teacher RL on sparse reward; Stage 2 is the bridge; Stage 3 is optional student-side sparse-reward RL. Cold RL is direct Stage 3 on the base student with no Stages 1–2. 1H/2H denote the two halves of DAPO used in the data-split experiments.
2
Sparse and Dense Reward Are One Objective
The teacher-first prescription rests on a useful observation: OPD is not a separate kind of training from RL; it is the same KL-regularized policy objective with a denser reward. Let x be a prompt, y = (y1 , . . . , yT ) a response, and st = (x, y<t ) the autoregressive state. Sparse RL maximizes Ex,y∼πθ [R(x, y)] − βEx KL(πθ ∥πref ), which is satisfied by the reward-tilted policy ∗ ∗ πR ∝ πref exp(R/β). The student never has direct access to πR ; it has to infer it from sparse rollouts, which is precisely why direct student RL is hard. OPD is the same objective with the teacher’s policy substituted for the reward-tilted target. Define the dense token reward rT (st , yt ) = β log πT (yt | st ), (1) and consider maximum-entropy RL with this reward: " # X J0 (θ) = Ex,y∼πθ rT (st , yt ) + β H(πθ ) = −β Ex KL(πθ ∥πT ). (2) t
The derivation is a one-line autoregressive factorization, deferred to Appendix A. The right-hand side is OPD. The teacher provides a full distribution at every token; if the teacher was itself improved by RL, that distribution is a tractable approximation to reward-shaped behavior found at larger scale. Applying sparse reward to the teacher is what makes the dense reward rT informative. The two objectives sit at opposite ends of a reward-density axis: " # X Jλ (θ) = Ex,y∼πθ (1 − λ) rT (st , yt ) + λR(x, y) + β H(πθ ),
λ ∈ {0, 1}.
(3)
t
Setting λ = 0 recovers OPD (Eq. 2); setting λ = 1 recovers sparse-reward RL. Rather than mixing the two signals in a single update, the pipeline in Eq. 6 allocates each endpoint to the model best positioned to use it: the teacher operates at λ = 1 to discover reward-shaped behavior (Stage 1); the student operates at λ = 0 to absorb that behavior as dense supervision (Stage 2), then at λ = 1 on held-out labeled data (Stage 3). The design choice is which model receives which reward density, and in what order. Why OPD alone is not enough. OPD is defined under student-state occupancy dπθ : LR (θ) = Es∼dπθ KL(πθ (· | s)∥πT (· | s)).
(4)
When the student starts far from the teacher’s support, dπθ rarely visits states where πT has useful structure, and the gradient is dominated by low-quality prefixes. A forward-KL phase on teacher rollouts, LF (θ) = Es∼dπT KL(πT (· | s)∥πθ (· | s)), (5) is the off-policy projection onto the same teacher target under teacher occupancy: mode-covering, stable, and precisely the step that moves the student into the region where OPD is well-conditioned. The two stages target the same πT ; they differ in the direction of the KL and in the occupancy under which it is taken. This is why neither stage alone can replace the pair. The student-side path therefore reads LF → |{z} teacher-occupancy warmup
LR ≡ −J0 /β | {z }
dense on-policy teacher reward
3
→
J1 |{z}
sparse task RL (optional)
.
(6)
3
Why the Teacher Is the Right Place for Sparse Reward
Eq. 2 says that the student receives a dense reward proportional to teacher log-probability. The value of that reward is therefore governed by the quality of the teacher distribution. This avoids two failure modes of sparse student RL, while introducing one clear risk. Failure mode 1: weak rollout distribution. Sparse reward can distinguish only the trajectories that the policy already samples with non-negligible probability. A small base model on AIME has near-zero pass rate, so most rollouts receive the same zero reward and the gradient signal collapses. A larger model has a higher base pass rate, so the same labeled training example produces a more informative spread of rewards and a more useful advantage. The same labeled example is worth more to a larger model. Failure mode 2: long-horizon credit assignment. Even when the final reward is non-zero, assigning it to the right token in a 4k-token chain is sample-inefficient. A teacher’s per-token distribution supplies this assignment by construction. Distilling a reward-shaped teacher into the student converts a sequence-level signal into a token-level one. ∗ The risk: teacher bias. Dense teacher reward is biased toward πT , not toward πR . If the teacher was not reward-shaped—if it was only pretrained, or only SFT’d—then dense transfer simply imitates a generic teacher. This is why scale alone is not enough: in Section 5, raw-teacher transfer underperforms direct GRPO, while RL-teacher transfer outperforms it.
The teacher-first prescription is therefore not simply “use a bigger model.” It is to move sparse reward upstream to the model that can turn it into a reward-shaped distribution, then make that distribution dense.
4
The Two-Stage Bridge
The bridge in Eq. 6 is not merely an ordering choice; its two stages address complementary weaknesses. A forward-KL warmup on teacher rollouts is the stage that can move the student into the teacher’s support without sparse-reward feedback. It is supervised next-token training under teacher occupancy, stable and inexpensive. Up to teacher-entropy terms it equals Es∼dπT KL(πT ∥πθ ): a per-state mode-covering projection. Its weakness is that it never visits student-only states. OPD then takes over. On the support neighborhood now reachable by the student, it minimizes KL(πθ ∥πT ) under student occupancy, which is mode-seeking and on-policy. By Eq. 2, it is densereward RL. Its weakness at initialization is precisely what the warmup resolves. Two alternatives in the literature keep only one side of this pair. Teacher-sample SFT (the DeepSeekR1 distillation recipe [Guo et al., 2025]) keeps the off-policy half and drops the on-policy half: the student never receives feedback on its own states. OPD-only [Agarwal et al., 2024, Lu and Thinking Machines Lab, 2025] keeps the on-policy half and drops the support-fixing half. Section 5 shows that both are weaker than the pair on the pre-Stage 3 Qwen transfer endpoints, and that the bridge remains the strongest MATH endpoint after the subsequent student-RL stage.
5
Experiments
The experiments follow the three contributions in turn. Table 1 provides a compact map of the routes and controls, so that each comparison has a named purpose. The training stack builds on verl/HybridFlow [Sheng et al., 2024]; key hyperparameters are in Appendix E. Accuracies are avg@16 (each problem is scored by the mean correctness over 16 independent samples), with ± standard error across evaluation problems. 5.1
Teacher-side vs. student-side sparse reward
The direct comparison considers three uses of the same labeled training data at fixed deploymentstudent size (Qwen3-1.7B): allocate it to student RL, allocate it to raw-teacher distillation, or allocate 4
Table 1: Compact map of the Qwen3 routes and controls. The table is intentionally smaller than a full route grid: it lists only the contrasts needed to interpret the claims. In the half-split rows, 1H and 2H denote the first and second halves of DAPO. Route / control
Sparse-reward placement
Purpose
Cold GRPO (R1)
Direct GRPO on the base Qwen3-1.7B student Full DAPO for teacher RL and twostage bridge; no Stage 3 1H for teacher RL and bridge; 2H for student GRPO Same 1H bridge as R5-half; Stage 3 reuses 1H RL’d teacher; teacher-sample SFT instead of bridge; 2H GRPO SFT-trained teacher; bridge plus 2H GRPO RL’d teacher; OPD without FKL warmup; 2H GRPO
Deployment-student baseline
Full teacher-side bridge (R3-full) Half-split post-bridge RL (R5-half) Replay control (R7) Teacher-sample SFT (R2) SFT-teacher control (R6) OPD-only control (R8)
Teacher-side allocation ceiling Student-side use of the held-out half after transfer Separates new labeled examples from extra updates Tests off-policy hard-label transfer Tests whether teacher RL is needed Tests the value of the forward-KL warmup
Table 2: Direct GRPO across Qwen3 scales, MATH and AIME (avg@16, %). The 1.7B row is the cold-RL baseline that the teacher-first pipeline must beat. Model Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3-14B
MATH
AIME 2024
AIME 2025
75.9 ± 0.9 85.2 ± 0.8 88.4 ± 0.8 89.5 ± 0.7
19.8 ± 1.4 39.2 ± 1.4 47.7 ± 1.5 47.1 ± 1.2
17.1 ± 0.9 34.4 ± 1.0 36.7 ± 1.2 39.0 ± 0.9
it to teacher RL followed by dense transfer. Table 3 reports the full-DAPO endpoints; Table 2 first checks that the 1.7B direct-RL baseline is not an artifact of an under-scaled GRPO recipe. Table 2 sets a strong direct-RL baseline. Larger Qwen3 models reach much stronger GRPO endpoints, so the low 1.7B endpoint is not a sign of a broken optimizer; it is the cost of applying sparse reward to the least capable policy. Three patterns in Table 3 support the main allocation result. (i) Scale alone is not the cause. A raw 8B teacher distilled into the 1.7B student gives 71.5% MATH, four points below direct GRPO. A raw 14B teacher gives 72.8%. The deployment student is not simply waiting for a larger model to imitate; it needs a teacher whose behavior has been shaped by reward. The SFT-trained teacher rows make the same point more precisely. They are better transfer sources than raw teachers, reaching 76.9% and 77.6% MATH, but they still trail the RL-improved 8B/14B teachers. Supervised teacher improvement helps, but it does not replace teacher-side discovery from sparse reward. (ii) Reward-shaped scale is the cause. Once the same 8B and 14B teachers have themselves been trained with sparse reward, the bridge moves the student to 79.3% and 78.6% MATH and 25.2 and 24.6 AIME 2024. For the 8B teacher, this beats direct GRPO by 3.4 MATH points and 5.4 AIME 2024 points; for the 14B teacher, the gains are 2.7 and 4.8 points. The labeled examples are the same examples a direct-RL run would have used; only their placement changes. (iii) Even same-size matters. An RL’d 1.7B teacher distilled into a fresh 1.7B student reaches 76.5% MATH and 20.6 AIME 2024, beating direct GRPO on those two metrics and matching it on AIME 2025. This isolates the dense-reward effect from teacher scale: the same labeled training data produces more useful supervision when its product is a teacher distribution than when it directly updates the deployment student. 5
Table 3: Transfer-only endpoints at fixed deployment student (Qwen3-1.7B), without subsequent student-side RL. In the RL-improved rows, labeled training data is allocated to teacher RL; raw/SFT rows use the same transfer protocol without teacher-side sparse RL. The table includes raw, SFTtrained, and RL-improved teachers, plus one-stage transfer controls on the RL-improved teachers. At matched 8B/14B teacher scale, raw teachers underperform direct GRPO, SFT-trained teachers are intermediate, and RL-improved teachers are strongest. The 1.7B RL’d teacher is a same-size control. Teacher checkpoint
Transfer protocol
MATH
AIME 2024
AIME 2025
—
Direct GRPO (cold student)
75.9 ± 0.9
19.8 ± 1.4
17.1 ± 0.9
raw Qwen3-8B raw Qwen3-14B
two-stage bridge two-stage bridge
71.5 ± 0.9 72.8 ± 0.8
15.0 ± 1.5 16.7 ± 1.4
10.6 ± 1.2 13.5 ± 1.3
SFT’d Qwen3-8B SFT’d Qwen3-14B
two-stage bridge two-stage bridge
76.9 ± 0.9 77.6 ± 0.8
22.1 ± 1.7 23.2 ± 1.6
17.6 ± 1.4 18.4 ± 1.5
RL’d Qwen3-1.7B RL’d Qwen3-8B RL’d Qwen3-14B
two-stage bridge two-stage bridge two-stage bridge
76.5 ± 0.8 79.3 ± 0.7 78.6 ± 0.9
20.6 ± 1.5 25.2 ± 1.6 24.6 ± 1.5
17.1 ± 1.4 20.2 ± 1.3 20.8 ± 1.5
RL’d Qwen3-1.7B RL’d Qwen3-8B RL’d Qwen3-14B
OPD only OPD only OPD only
75.2 ± 0.9 77.6 ± 0.8 77.1 ± 1.0
19.1 ± 1.5 23.0 ± 1.4 22.8 ± 1.5
12.4 ± 1.2 18.9 ± 1.4 18.6 ± 1.7
RL’d Qwen3-1.7B RL’d Qwen3-8B RL’d Qwen3-14B
teacher-sample SFT teacher-sample SFT teacher-sample SFT
73.6 ± 0.9 76.0 ± 0.9 76.5 ± 1.1
16.7 ± 1.4 22.4 ± 1.5 21.5 ± 1.5
11.4 ± 1.0 19.4 ± 1.4 17.0 ± 1.1
The Llama family shows the same ordering with a single canonical teacher (Table 8 in Appendix D): raw-70B transfer underperforms direct GRPO on the 8B student (55.4% vs. 59.8% MATH), while RL’d-70B transfer outperforms it (62.1%). The conclusion is therefore not tied to Qwen alone. 5.2
Transfer protocol ablation: FKL warmup, OPD, and SFT
Table 3 also isolates the bridge. Holding the RL-trained teacher and the labeled training data fixed, the two-stage bridge reaches 79.3% MATH at the 8B teacher; OPD-only reaches 77.6%; teacher-sample SFT reaches 76.0%. The 14B teacher gives the same MATH ordering (78.6% > 77.1% > 76.5%), and the same-size 1.7B controls follow the same ordering on MATH and AIME 2024 (76.5% > 75.2% > 73.6% on MATH). In this pre-Stage 3 transfer comparison, the two-stage bridge is also the best endpoint on AIME 2024 and AIME 2025 among the canonical 8B/14B teachers. This is the pattern predicted by Section 4. Teacher-sample SFT is off-policy and gives no signal on student-only states; OPD-only is on-policy but ill-conditioned at initialization. On MATH, teachersample SFT is the weakest variant and OPD-only is intermediate. Across the AIME cells, both one-stage variants trail the two-stage bridge before Stage 3, although their relative ordering varies. Section 5.4 shows that after Stage 3 the MATH ordering remains clear, while the AIME cells are closer and partly mixed. 5.3
Student RL after the bridge: half-split and replay controls
The first two results could still leave a narrower interpretation: perhaps the bridge is only a better initialization, and any later sparse-reward RL is wasted. We test this directly. Split the DAPO training set into two random halves, 1H and 2H. Train the teacher and the bridge on 1H. Hold the resulting 1.7B checkpoint fixed, then ask whether sparse student RL on 2H adds value over (a) the bridge alone, (b) cold direct GRPO, and (c) a matched replay control that reuses 1H for student RL. Table 4 reports the result with RL-trained teachers, Table 5 with SFT-trained teachers. Student-side sparse RL on the held-out second half lifts the bridge endpoint from 75.4% to 78.5% MATH at the 8B teacher and from 76.3% to 78.7% at the 14B teacher. Both endpoints clear cold direct GRPO (75.9%). The replay control uses the same student-RL data count and update count on already-seen bridge data, yet never improves by more than 0.3 points and sometimes degrades. The gain is not extra updating; it is new labeled examples reaching a student that is now prepared to use 6
Table 4: Student-side sparse RL on fresh labeled data lifts the bridge endpoint above both direct GRPO and a matched replay control. The teacher and bridge are trained on the first DAPO half; student GRPO uses the held-out second half. RL-trained Qwen3 teachers, Qwen3-1.7B student; avg@16 (%). Teacher
Student stage
MATH
AIME 2024
AIME 2025
RL’d Qwen3-8B
After two-stage bridge (1H) + GRPO on held-out 2H + GRPO replay on 1H
75.4 ± 0.8 78.5 ± 0.9 75.7 ± 0.7
22.0 ± 1.6 23.7 ± 1.5 21.6 ± 1.3
16.7 ± 1.4 18.5 ± 1.2 17.0 ± 1.2
RL’d Qwen3-14B
After two-stage bridge (1H) + GRPO on held-out 2H + GRPO replay on 1H
76.3 ± 1.1 78.7 ± 1.1 75.6 ± 1.0
22.7 ± 1.7 23.1 ± 1.7 22.4 ± 1.5
17.3 ± 1.2 19.2 ± 1.3 17.6 ± 1.0
Table 5: The same student-side sparse-RL pattern holds when the teacher is SFT-trained instead of RL-trained, but with lower MATH and AIME 2025 endpoints. Qwen3-1.7B student; avg@16 (%). Teacher
Student stage
MATH
AIME 2024
AIME 2025
SFT’d Qwen3-8B
After two-stage bridge (1H) + GRPO on held-out 2H + GRPO replay on 1H
74.3 ± 1.0 77.2 ± 1.0 74.0 ± 0.8
21.8 ± 1.5 22.9 ± 1.3 22.1 ± 1.1
14.5 ± 1.2 18.4 ± 1.0 14.2 ± 1.0
SFT’d Qwen3-14B
After two-stage bridge (1H) + GRPO on held-out 2H + GRPO replay on 1H
75.8 ± 0.9 76.9 ± 0.8 75.6 ± 0.7
22.0 ± 1.5 23.2 ± 1.3 22.3 ± 1.2
15.1 ± 1.4 18.6 ± 1.1 14.9 ± 1.2
them. The SFT-teacher table shows the same fresh-data-vs-replay pattern, but with weaker MATH endpoints than the RL-teacher pipeline (77.2% vs. 78.5%; 76.9% vs. 78.7%), as the teacher-first allocation predicts: an unshaped teacher gives a weaker bridge.
5.4
Where should the held-out half of the data go?
The previous two subsections establish two facts: teacher-side allocation beats cold student RL, and student-side RL becomes useful after the bridge. The remaining allocation question is simpler than it may appear: after using the first half of DAPO (1H) to train the teacher and bridge, where should the second half (2H) go? We compare two placements of the same 2H data. The teacher-side route (R3-full) uses both 1H and 2H upstream: the full DAPO set trains the teacher and the two-stage bridge, and the resulting student receives no Stage 3 GRPO. The student-side route (R5-half) uses only 1H upstream, holds 2H out from teacher RL and transfer, and then applies 2H as post-bridge student GRPO. Thus both routes use the same total labeled data; the difference is whether 2H is consumed before transfer or after transfer. The teacher-side endpoint is the full-DAPO bridge from Table 3 (79.3% MATH at the RL’d 8B teacher). The student-side endpoint is the half-bridge-plus-held-outGRPO pipeline from Table 4 (78.5% MATH at the same teacher). This is the central fixed-data allocation contrast. The teacher-side route wins, but the margin is small (0.8 MATH points; AIME points are within standard error): upstream use of labeled data is slightly better, while post-bridge student RL recovers most of its value. When teacher-side compute is the binding constraint, the student-side route remains a competitive lower-cost alternative. The fixed-data contrast above uses the two-stage bridge. Table 6 checks whether that bridge choice matters inside the student-side route. It does: the two-stage bridge remains the best MATH starting point for student-side GRPO. The AIME cells are closer and partly mixed: removing either the forward-KL warmup or the on-policy dense stage weakens AIME 2024, while AIME 2025 has one small OPD-only exception at the 14B teacher. This is how the two-stage recipe mitigates OPD failure modes highlighted in recent analyses: the forward-KL warmup first fixes support mismatch, so the subsequent OPD stage is no longer a cold-start reverse-KL update on low-quality student states [Li et al., 2026, Hou et al., 2026]. 7
Table 6: Bridge controls under the student-side route. Each row uses the same RL-trained Qwen3 teacher and the same held-out GRPO data; only the transfer protocol before Stage 3 differs. Qwen31.7B student; avg@16 (%).
6
Teacher
Transfer protocol
MATH
AIME 2024
AIME 2025
RL’d Qwen3-8B
two-stage bridge OPD only teacher-sample SFT
78.5 ± 0.9 77.8 ± 0.8 77.3 ± 0.8
23.7 ± 1.5 22.8 ± 1.2 22.5 ± 1.4
18.5 ± 1.2 16.6 ± 1.3 16.9 ± 1.4
RL’d Qwen3-14B
two-stage bridge OPD only teacher-sample SFT
78.7 ± 1.1 77.5 ± 0.9 77.2 ± 1.2
23.1 ± 1.7 21.9 ± 1.5 21.6 ± 1.8
19.2 ± 1.3 19.5 ± 1.2 19.0 ± 1.4
Discussion
What changes operationally. The standard reading of the post-training literature is a menu of competing methods: SFT, RL, distillation. The reward-density principle turns that menu into an allocation problem. Once OPD is viewed as dense-reward RL (Eq. 2), the design choice is not only which method to run, but which model should receive which density of reward, and in what order. Direct sparse-reward RL on the deployment model is inefficient placement on both axes: sparse reward is given to the policy least prepared to use it. Implication for model-family training. The practical recipe is clearest when a lab trains or maintains a model family rather than a single deployment checkpoint. A larger teacher and a smaller deployment student can be pretrained on the same data distribution, preferably with a shared tokenizer, and kept as parallel post-training targets. The reward-density principle then says that labeled posttraining data should be allocated preferentially to the larger model first, because it can convert sparse reward into a better reward-shaped distribution. The smaller model should receive that distribution through the dense FKL-to-OPD bridge, with student-side sparse RL reserved for held-out labeled data after the bridge. Why the bridge is two-stage. An off-policy stage alone cannot teach the student to recover on its own prefixes; an on-policy stage alone is poorly conditioned at initialization. The two-stage bridge covers both sides. Table 3 shows the clean pre-Stage 3 ordering; Table 6 shows that after Stage 3 the bridge remains best on MATH, with closer and partly mixed AIME cells. Why student-side reward still matters. The post-bridge student-RL result (Section 5.3) keeps the recipe from becoming a rigid “never train the student” rule. After the bridge, sparse reward on the student gives a real 2–3-point lift on MATH and is strictly better than running more updates on bridge data. The right framing is teacher-first with post-bridge student RL; the weaker framing is either “RL the student” or “never RL the student.” Limitations. The evidence is on verifiable math with two student-teacher families at relatively small deployment scale (1.7B and 8B students, with teachers up to 14B and 70B). Whether the teacher-first advantage persists, grows, or shrinks at larger scales—for example, a 70B student with a 400B+ teacher—remains open. The reward-density argument predicts persistence, but the marginal value of sparse reward on a stronger student may shift the allocation balance. The principle itself does not depend on the task; the bridge does require a shared tokenizer between teacher and student. Code, instruction following, and open-ended tasks would need their own verifier-density experiments, and we make no claim about an optimal λ schedule beyond the staged version in Eq. 6. The Llama block is deliberately narrower than the Qwen study: its role is to test the teacher-quality ordering in a second model family, while the half-split allocation, replay, OPD-only, teacher-sample SFT, and SFT-teacher controls remain future cross-family experiments, as noted in Appendix D.
7
Related Work
Post-training reshapes language-model behavior through feedback-based RL and teacher transfer: RLHF uses sparse preference or outcome rewards [Ouyang et al., 2022, Stiennon et al., 2020, Bai 8
et al., 2022], while distillation transfers teacher behavior through dense supervised signals [Hinton et al., 2015]. We position this paper along both axes. Appendix B provides the per-paper detail. Sparse-reward post-training. PPO, GRPO, and SFT-warmup-then-PPO recipes use labeled data to apply sparse reward directly to the deployment model [Schulman et al., 2017, Shao et al., 2024, Luong et al., 2024]. Verifier-filtered SFT uses the reward only as a data filter [Zelikman et al., 2022, Singh et al., 2024]. Recent work increases reward density through self-distillation [He et al., 2026, Yang et al., 2026] or reference-guided trajectories [Wu et al., 2026a]. These methods differ in how they use reward, but they still train the model being optimized on the labeled data. Our point is orthogonal: the same data is often more valuable upstream on a teacher and then densified through the bridge. Distillation and OPD. Knowledge distillation transfers teacher behavior into smaller models [Hinton et al., 2015]; teacher-sample SFT is its off-policy form [Guo et al., 2025]. OPD corrects the student on its own rollouts [Agarwal et al., 2024] and has been framed as dense on-policy teacher-logprob reward [Lu and Thinking Machines Lab, 2025]. Related work connects distillation to entropy-regularized or RL-aware objectives [Liu et al., 2025, Zhang et al., 2026b] and extends OPD through KL scheduling, token importance, chain compression, and offline caching [Xu et al., 2026a,b, Sang et al., 2026, Wu et al., 2026b]. Our Eq. 2 uses the same connection prescriptively: OPD’s dense reward is only as good as the teacher, so sparse reward should first improve the teacher. Reasoning teachers and data allocation. DeepSeek-R1 showed that RL-improved models can teach smaller ones via SFT [Guo et al., 2025]; MiMo-V2-Flash extends this with multi-teacher OPD that integrates domain specialists through on-policy token-level rewards [Xiaomi LLM-Core Team, 2026]. Our focus is different: not whether an RL-improved model can teach, but where a fixed pool of labeled training data should be allocated—teacher-side or student-side. Table 7 (Appendix C) classifies representative methods along this axis.
8
Conclusion
Two axes structure language-model post-training: how dense the reward signal is, and which model receives it. Our experiments support a teacher-first allocation rule for verifiable math: use scarce labeled training data first where sparse reward is most informative, transfer the resulting behavior through the FKL-to-OPD bridge, and reserve student-side GRPO for held-out labeled examples after the bridge. This recipe beats direct student GRPO in the Qwen3-1.7B setting, preserves the teacher-quality ordering in the Llama family, and shows that post-bridge student RL adds value beyond replaying bridge data. The broader lesson is not to avoid student RL, but to apply it after dense transfer has made the deployment policy trainable.
References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, 2024. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Zhiqi Bai, Ken Deng, Jinyang Guo, Congnan Liu, Jiaheng Liu, Jie Liu, Lin Qu, Haoran Que, Wenbo Su, Jiakai Wang, Jiamang Wang, Yanan Wu, Chenchen Zhang, Ge Zhang, Yuanxing Zhang, and Bo Zheng. DDK: Distilling domain knowledge for efficient large language models. In Advances in Neural Information Processing Systems, 2024. Junfeng Fang, Zhepei Hong, Mao Zheng, Mingyang Song, Gengsheng Li, Houcheng Jiang, Dan Zhang, Haiyun Guo, Xiang Wang, and Tat-Seng Chua. Rubric-based on-policy distillation. arXiv preprint arXiv:2605.07396, 2026a. 9
Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituo Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, and Feng Zhao. Flow-OPD: On-policy distillation for flow matching models. arXiv preprint arXiv:2605.08063, 2026b. Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726, 2023. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633–638, 2025. Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, et al. Self-distillation zero: Self-revision turns binary rewards into dense supervision. arXiv preprint arXiv:2604.12002, 2026. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, et al. Uni-OPD: Unifying on-policy distillation with a dual-perspective recipe. arXiv preprint arXiv:2605.03677, 2026. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! Outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, 2023. Hyunseok Lee, Soheil Abbasloo, Jihoon Tack, and Jinwoo Shin. Beyond correctness: Learning robust reasoning via transfer. arXiv preprint arXiv:2602.08489, 2026. Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xin Xie. Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.06726, 2022. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016, 2026. Kun Liang, Clive Bai, Xin Xu, Chenming Tang, Sanwoo Lee, Weijie Liu, Saiyong Yang, and Yunfang Wu. ORBIT: On-policy exploration-exploitation for controllable multi-budget reasoning. arXiv preprint arXiv:2601.08310, 2026. Guanlin Liu, Anand Ramachandran, Tanmay Gangwani, Yan Fu, and Abhinav Sethy. Knowledge distillation with training wheels. arXiv preprint arXiv:2502.17717, 2025. Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/ on-policy-distillation/. Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. ReFT: Reasoning with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773–1781, 2023. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744, 2022. 10
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, 2023. Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. CRISP: Compressed reasoning via iterative self-policy distillation. arXiv preprint arXiv:2603.05433, 2026. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. arXiv preprint arXiv:2409.19256, 2024. Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, 2024. Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626, 2026. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems, volume 33, 2020. Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, Xiaohu Ruan, Xiaoxin Chen, and Honggang Qi. Skill-SD: Skill-conditioned self-distillation for multi-turn LLM agents. arXiv preprint arXiv:2604.10674, 2026a. Jiaqi Wang, Wenhao Zhang, Weijie Shi, Yaliang Li, and James Cheng. TCOD: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. arXiv preprint arXiv:2604.24005, 2026b. Yangzhen Wu, Shanda Li, Zixin Wen, Xin Zhou, Ameet Talwalkar, Yiming Yang, Wenhao Huang, and Tianle Cai. Learn hard problems during RL with reference guided fine-tuning. arXiv preprint arXiv:2603.01223, 2026a. Yecheng Wu, Song Han, and Hai Cai. Lightning OPD: Efficient post-training for large reasoning models with offline on-policy distillation. arXiv preprint arXiv:2604.13010, 2026b. Xiaomi LLM-Core Team. MiMo-V2-Flash technical report, 2026. URL https://arxiv.org/ abs/2601.02780. Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. PACED: Distillation and on-policy self-distillation at the frontier of student competence. arXiv preprint arXiv:2603.11178, 2026a. Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. TIP: Token importance in on-policy distillation. arXiv preprint arXiv:2604.14084, 2026b. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengyuan Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, et al. Self-distilled RLVR. arXiv preprint arXiv:2604.03128, 2026. 11
Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, and Furu Wei. Black-box on-policy distillation of large language models. arXiv preprint arXiv:2511.10643, 2025. Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. arXiv preprint arXiv:2602.12275, 2026. Qiying Yu, Zheng Sun, Xiang Shen, Liang Gao, Ziyi Pan, et al. DAPO: An open-source llm reinforcement learning system. arXiv preprint arXiv:2503.14476, 2025. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35, 2022. Jiaxin Zhang, Xiangyu Peng, Qinglin Chen, Qinyuan Ye, Caiming Xiong, and Chien-Sheng Wu. The illusion of certainty: Decoupling capability and calibration in on-policy distillation. arXiv preprint arXiv:2604.16830, 2026a. Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, and Stefano Soatto. Reinforcement-aware knowledge distillation for LLM reasoning. arXiv preprint arXiv:2602.22495, 2026b.
A
Deriving OPD as Dense-Reward RL
For a fixed prompt x, OPD minimizes KL(πθ ∥πT ) = Ey∼πθ [log πθ (y | x) − log πT (y | x)] . P Using log π(y | x) = t log π(yt | st ), this is " # X X Ey∼πθ log πθ (yt | st ) − log πT (yt | st ) . t
(8)
t
Multiplying by −β yields " # " # X X X β Ey∼πθ log πT (yt | st ) − log πθ (yt | st ) = Ey∼πθ rT (st , yt ) + β H(πθ ), t
(7)
t
(9)
t
i.e. entropy-regularized RL with token reward rT (st , yt ) = β log πT (yt | st ).
B
Extended Related Work
This appendix provides the per-paper detail that the shorter related-work section omits. Sparse-reward post-training. In sparse-reward policy optimization, the reward directly updates the policy through PPO, GRPO, or SFT-warmup-then-PPO recipes such as ReFT [Schulman et al., 2017, Shao et al., 2024, Luong et al., 2024]. Systems work such as verl/HybridFlow makes these RLHF dataflows practical by combining flexible algorithm representation with efficient distributed execution [Sheng et al., 2024]. In verifier-filtered SFT, the reward is a data-construction rule: sample candidate traces, keep correct ones, and then run supervised imitation [Zelikman et al., 2022, Singh et al., 2024, Yang et al., 2024]. DPO and related derivations make explicit the link between reward optimization and KL-regularized policy targets [Rafailov et al., 2023]. Recent RLVR work moves beyond finalanswer correctness by training on more informative intermediate reasoning behavior [Lee et al., 2026]. A related line uses self-distillation to convert sparse binary RLVR rewards into dense token-level supervision [He et al., 2026, Yang et al., 2026]. Reference-guided fine-tuning targets the zero-reward hard-problem regime: partial human reference solutions elicit model-generated positive trajectories before DAPO-style RL, raising the density of rewarding samples on problems the base model cannot initially solve [Wu et al., 2026a]. 12
Distillation and OPD. Knowledge distillation transfers behavior from stronger models into smaller models [Hinton et al., 2015]. Reasoning-distillation work shows that intermediate traces can be more useful than final answers alone [Fu et al., 2023, Li et al., 2022, Magister et al., 2023, Hsieh et al., 2023]. Domain-aware distillation methods adapt transfer to domain knowledge and teacher-student capability gaps [Bai et al., 2024]. Teacher-sample SFT is the off-policy form of this idea: imitate teacher-generated traces, including the DeepSeek-R1 distilled models [Guo et al., 2025]. OPD instead corrects the student on its own rollout distribution rather than only on teacher-generated states [Agarwal et al., 2024]; related variants extend this idea to context distillation and black-box teacher access [Ye et al., 2026, 2025]. Rubric-based OPD pushes the black-box direction further by inducing prompt-specific rubrics from teacher-student contrasts and using weighted rubric pass rates as on-policy rewards [Fang et al., 2026a]. Recent practitioner evidence frames OPD as dense on-policy teacher-logprob reward and reports large compute-efficiency gains over sparse RL and extended off-policy distillation [Lu and Thinking Machines Lab, 2025]. Liu et al. [2025] formulate KD as entropy-regularized value optimization with on-policy and off-policy demonstrations, while Zhang et al. [2026b] propose RL-aware distillation through advantage-aware selective imitation during PPO/GRPO-style updates. Further OPD work studies a forward-then-reverse KL schedule [Xu et al., 2026a], analyzes which student-state tokens carry the strongest learning signal [Xu et al., 2026b], applies on-policy self-distillation to compress overlong reasoning chains [Sang et al., 2026], introduces temporal curricula and skill-conditioned self-distillation for multi-turn agents [Wang et al., 2026b,a], and explores offline OPD through precomputed teacher log-probabilities [Wu et al., 2026b]. Concurrent analyses dissect when OPD succeeds or fails and propose unified recipes across LLM and MLLM settings [Li et al., 2026, Hou et al., 2026], while Flow-OPD adapts OPD-style dense multi-teacher supervision to flow-matching text-to-image alignment [Fang et al., 2026b]. Zhang et al. [2026a] highlight that OPD can systematically miscalibrate confidence even when accuracy improves; for a taxonomy of OPD feedback signals, teacher access regimes, and loss granularity, see Song and Zheng [2026]. Reasoning teachers and data allocation. DeepSeek-R1 showed that large-scale RL can elicit strong reasoning behavior and that smaller models can inherit it through supervised fine-tuning on DeepSeek-R1-generated traces [Guo et al., 2025]. ORBIT studies a different control dimension: it uses multi-stage RL under context-length constraints to discover Pareto-frontier reasoning-effort policies, then fuses those policies by OPD into one controllable model [Liang et al., 2026]. Our allocation question is different: where should scarce labeled training data enter the post-training pipeline? MiMo-V2-Flash makes the OPD connection explicit through Multi-Teacher On-Policy Distillation (MOPD) [Xiaomi LLM-Core Team, 2026]. Its post-training pipeline first runs SFT, then trains domain-specialized teachers through RL or SFT, and finally integrates those teachers by having the student sample from its own on-policy distribution while receiving token-level reverse-KL rewards from the teacher selected for each prompt domain. The formulation is aligned with our reward-density principle: the teacher log-probability ratio becomes a dense per-token advantage. In our taxonomy, MOPD is a scalable multi-teacher OPD mechanism for capability integration, while our paper studies how scarce labeled training data should be allocated before and after such dense transfer.
C
Method Classification
D
Llama Cross-Family Validation
The Llama block repeats the teacher-quality ordering in a second model family: raw-teacher transfer < direct GRPO < RL-teacher transfer. A 9× larger raw teacher is still worse than direct RL; the same teacher after RL is the best source. This supports the paper’s central distinction between teacher size and reward-shaped teacher quality. The corresponding Llama half-split, replay, OPD-only, teacher-sample SFT, and SFT-teacher controls remain future work.
E
Implementation Details
All route comparisons keep the deployment-student size fixed. In the Qwen3 block, the student is Qwen3-1.7B and the teacher checkpoints are the raw, SFT-trained, and RL-trained Qwen3 checkpoints 13
Table 7: Representative methods classified by where sparse reward enters and what signal is used for transfer. Paper names are examples of method classes; route labels such as R2 or R5-half are complete pipelines built from these classes. Method / reference
Sparse reward use
Transfer signal
Student endpoint
InstructGPT et al., 2022]
preference reward
none
RLHF policy
GRPO / DeepSeekMath [Shao et al., 2024]
answer RL
none
RL policy
ReFT / PPO after SFT [Luong et al., 2024]
answer RL after SFT
none
RL policy
Verifier-filtered SFT [Zelikman et al., 2022, Singh et al., 2024] Step-by-step distillation [Fu et al., 2023]
verifier as filter
accepted traces
SFT policy
none
rationales
distilled policy
GKD / OPD [Agarwal et al., 2024] DeepSeek-R1 distilled models [Guo et al., 2025]
teacher-dependent
teacher logits
OPD policy
teacher-side RL
SFT on teacher-generated traces
distilled policy
MiMo-V2-Flash / MOPD [Xiaomi LLM-Core Team, 2026]
specialized teacher RL/SFT plus optional outcome reward
multi-teacher OPD logits
unified model
This work
allocated teacher/student RL
FKL-to-OPD transfer
allocation-aware student endpoint
[Ouyang
post-trained
Table 8: Llama replication (Student = Llama-3.1-8B-Instruct, Teacher = Llama-3.3-70B-Instruct), avg@16 (%). Configuration Direct GRPO (cold student) Two-stage bridge ← raw 70B Two-stage bridge ← RL’d 70B
MATH
AIME 2024
AIME 2025
59.8 ± 0.9 55.4 ± 0.8 62.1 ± 0.8
12.5 ± 1.2 8.8 ± 1.5 14.9 ± 1.8
7.2 ± 1.1 3.1 ± 1.2 9.2 ± 1.4
listed in Tables 2–4. In the Llama block, the student is Llama-3.1-8B-Instruct and the teacher is Llama-3.3-70B-Instruct. OPD is only run within a model family, because the token-level KL in Eq. 4 requires a shared tokenizer and vocabulary. Data splits. The Qwen allocation experiment uses a fixed random split of the DAPO-Math-17K training set [Yu et al., 2025] into two equal halves. The first half (1H) is the teacher-RL and bridge data pool for the half-split rows, and also the replay data pool for the R7 control. The second half (2H) is held out from teacher RL and transfer, then used for Stage 3 GRPO in R5-half. R5-half and R7 therefore start from the same bridge checkpoint and use the same Stage 3 data count and update count; they differ only in whether Stage 3 uses new labeled examples from 2H or replay examples from 1H. R3-full instead trains teacher RL and the bridge on the full DAPO set and has no final student GRPO. All rows are evaluated on MATH-500, AIME 2024, and AIME 2025, not on either DAPO training half. The SFT-teacher rows use the same first-half/second-half construction, replacing only the source teacher. Matched training protocol. Direct GRPO, Stage 3 GRPO, and replay GRPO use the same verifier reward, advantage normalization, optimizer family, batch size, rollout count per prompt, length limit, learning-rate schedule, KL settings, and update count within each matched contrast. In particular, R5-half and R7 are matched in checkpoint initialization, data count, rollout count, update count, and sequence-length limit. R2 keeps the RL-trained teacher and 2H Stage 3 GRPO fixed but replaces the 14
bridge with teacher-sample SFT. R8 keeps the RL-trained teacher and 2H Stage 3 GRPO fixed but removes the forward-KL warmup. Bridge protocol. The two-stage bridge runs a forward-KL warmup on cached teacher rollouts followed by OPD on student rollouts. The forward stage uses cached teacher rollouts and teacher next-token distributions on those rollouts. The OPD stage queries the frozen teacher on the student’s sampled prefixes, so the teacher signal is computed on-policy with respect to the current student distribution. Implementation caches may store these logits for audit and replay, but the teacher checkpoint is not updated. Unless otherwise stated, the forward and OPD stages use the same maximum sequence length and tokenizer as the corresponding student/teacher family, and all teacherlogit temperatures and KL coefficients are fixed across rows inside a contrast. Evaluation and error bars. All reported accuracies are avg@16. For each evaluation problem, the model samples 16 independent completions under the same decoding configuration; the problem score is the mean correctness over those completions, and the table entry is the mean over problems. The reported ± values are standard errors over evaluation problems, not standard deviations across independently retrained checkpoints. Data-split seeds, decoding seeds, training seeds, rollout counts, learning rates, KL coefficients, OPD temperatures, maximum prompt and response lengths, and exact checkpoint identifiers are recorded with the run configuration for each table row, so the route contrasts can be reproduced without changing non-ablation hyperparameters. Table 9: Key GRPO training hyperparameters for direct-RL and Stage 3 student-RL runs. Matched route contrasts use the same settings unless a row explicitly ablates the training stage or data source. Group
Parameter
Value
Algorithm Algorithm Optimizer Optimizer Update Update Update Length Length Loss Loss KL Data Compute Compute Compute Compute
Framework Estimator Optimizer Learning rate GRPO epochs Mini-batch size Micro-batch per GPU Max prompt tokens Max response tokens Clip ratio Gradient clip KL coefficient Validation sets Precision GPUs Rollout engine Tensor parallel size
VERL GRPO AdamW 1 × 10−6 10 8 1 3072 16384 0.2 1.0 5 × 10−4 MATH-500, AIME 2024, AIME 2025 bfloat16 2 × 8× NVIDIA H200 sglang 16
15
Table 10: Key OPD/transfer-stage hyperparameters. The table reports the result-relevant settings from the VERL transfer run configuration; internal checkpoint, profiler, QAT, and logging fields are omitted. Group
Parameter
Value
Algorithm Algorithm Optimizer Optimizer Training setup Update Update Update Length Length Sampling Data Compute Compute Compute
Framework Estimator Optimizer Learning rate Rollouts per prompt Epochs Mini-batch size Micro-batch per GPU Max prompt tokens Max response tokens Temperature / top-p / top-k Validation sets Precision GPUs Rollout engine / TP size
VERL GRPO-style actor rollout AdamW 1 × 10−6 8 1 64 4 2048 16384 1.0 / 1.0 / −1 MATH-500, AIME 2024, AIME 2025 bfloat16 2 × 8× NVIDIA H200 sglang / 2
16