Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation

Paper Under Review

B EYOND D OMAIN -ROUTED S UPERVISION : C ROSS -T EACHER G UARDING FOR M ULTI -T EACHER O N -P OLICY D ISTILLATION

arXiv:2610.13521v1 [cs.AI] 9 Oct 2026

Shi-Yu Tian1,2,∗ , Yong Liu2,∗ , Ximan Liu2,† , Tianrui Zhu2 , Sule Bai2 , Xinwei Yang2 , Ruiqi Liu2 , Kun-Yang Yu1 , Ming Yang1 , Zhi Zhou1,‡ , Yu-Feng Li1,‡ , Yipeng Sun2 , Gu Simiu2 , 1

Nanjing University

∗

†

Equal contribution

2

JD.com

Project leader

‡

Corresponding authors

A BSTRACT Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) by routing domain-specific data to specialized teachers, integrating their capabilities through dense token-level supervision on student-generated trajectories. However, domain routing only distinguishes the supervision sources for different domains; updates from all domains still act on shared parameters. When teachers differ substantially in response style and domain expertise, supervision that is effective in isolation may still lead to persistent gradient conflicts and capability degradation during joint optimization. To address this challenge, we propose Cross-Teacher Guarding for MOPD (ROAD MOPD), which introduces nonrouted teachers as auxiliary cross-domain critics on the same student trajectories and coordinates multi-teacher supervision through token-level credit assignment and gradient updates. Specifically, cross-teacher-guided credit assignment uses teacher agreement to strengthen token-level corrections jointly supported by the teachers and attenuate conflicting signals; preventive multi-teacher gradient projection uses auxiliary gradients from the same trajectories to constrain potentially conflicting updates and adaptively controls projection strength to retain targetdomain learning signals. In experiments, ROAD MOPD mitigates degradation in the disadvantaged domain, gains over 5 percentage points under high conflict, and improves overall performance in low-conflict and three-domain settings.

1

I NTRODUCTION

Reinforcement learning (RL) has emerged as an important paradigm for post-training large language models (OpenAI, 2023; DeepSeek-AI, 2025). However, training with outcome-level rewards lacks direct feedback on intermediate reasoning steps, making it difficult to attribute a response’s success or failure to specific generation decisions. Optimization therefore remains largely dependent on policy exploration, posing challenges for credit assignment and stable optimization. On-policy distillation (OPD) (Gu et al., 2024; Agarwal et al., 2024; Ko et al., 2025) offers a promising alternative for addressing this limitation: it leverages a strong teacher to provide dense token-level supervision on trajectories generated by the student’s current policy, combining on-policy sampling aligned with the student’s current generation distribution with fine-grained teacher feedback. Prior work has improved OPD capability transfer through better distillation objectives and optimization strategies (Li et al., 2026a; Lin et al., 2026; Yang et al., 2026a; Zhang et al., 2026). A natural extension is to consolidate specialized teachers’ complementary capabilities in a single student across domains. Multi-teacher on-policy distillation (MOPD) (Ma et al., 2026; DeepSeek-AI et al., 2026) pursues this goal, typically through direct domain routing: a sample from domain d receives supervision only from its routed teacher Td , while updates from all domains act on the same student model. This design exploits each teacher’s domain expertise but does not explicitly coordinate updates from different domains to shared parameters. Consequently, supervision that is effective in 1

Paper Under Review

50

Medical × Code Avg. 36.6%

Score change (Vanilla MOPD − single-domain OPD)

+3.54

Code

+3.54 +0.62

0

+2

Math

+0.76 +0.42

-7.05

+0.29 -10

-9.26

Medical

10

20

30

40

49

Optimizer update

-14.45 Me d Me QA dX pe Me dM rt GP CQA QA -M ed

1

-15

-2.53

-4

-3.07

-5

+1.35

AIM E2 AIM 4 HM E25 MT HM -Feb MT -No v

0

-0.29 -1.33

0 +2.08

-0.07

Math × Code Avg. 2.2%

Code + Medical +0.46

+2.19 +0.42

0

25

d

Math + Medical

+5

Math

AIM E2 4 AIM E2 5 HM MT -Fe HM b MT -No v Hu ma nE val+ MB PP + LC Bv 6

Conflicting layers (%)

75

c

Math + Code

+4

Code

-6.81 -7.19

-8

Medical

-12

-12.10 Me dQ A Me dX pe rt Me dM CQ GP A QA -M ed

b Medical × Math Avg. 45.2%

Ev al+ MB PP + LC Bv 6

Gradient conflicts 100

Hu ma n

a

Figure 1: Update conflicts reveal a failure mode of domain-routed MOPD. (a) Fraction of conflicting layers during training; labels show training averages. (b–d) Per-benchmark score changes of Vanilla MOPD relative to single-domain OPD. Math–Code remains compatible, whereas both Medical pairings consistently degrade Medical performance.

single-domain training may induce interfering update directions during joint optimization, limiting multi-domain capability integration. Prior MOPD studies have demonstrated the potential of joint learning on domain combinations such as math and code (Yang et al., 2026b; Hou et al., 2026), but these gains may not generalize to other combinations. As shown in Figure 1, joint training on math and code generally preserves or improves upon the corresponding single-domain OPD performance, with relatively few gradient conflicts. In contrast, jointly training medicine with either math or code consistently yields a high proportion of layers with conflicting gradients, and this proportion increases further in later stages of training. In both pairings, medical performance falls below the single-domain OPD baseline, showing that the joint-learning benefits observed for math and code do not extend to these combinations. This contrast highlights compatibility between domain updates as an important factor in MOPD and suggests that allocating supervision through domain routing alone is insufficient to coordinate capability transfer through shared parameters. To address this limitation, we reconsider teachers’ roles in MOPD: can non-routed teachers help anticipate and mitigate update interference while preserving the routed teacher’s domain-specific supervision? We retain this supervision and introduce non-routed teachers as cross-domain critics on the same student-generated trajectories. Their feedback on the same behavior provides local cues about potentially incompatible learning signals. We aim to reduce joint-optimization interference through this feedback while retaining target-domain learning signals. Specifically, we propose Cross-Teacher Guarding for MOPD (ROAD MOPD), to exploit crossteacher feedback at both the token and gradient levels. At the token level, ROAD MOPD compares evidence from the routed and auxiliary teachers relative to a shared frozen reference model, strengthening supervision when their implied correction directions agree and downweighting the corresponding tokens when they conflict. At the gradient level, auxiliary supervision on the same trajectory defines a cross-domain reference direction, and preventive gradient projection suppresses components of the target-domain update that conflict with this direction. In high-conflict settings, ROAD MOPD improves Medical accuracy by 5.6–8.8 percentage points over Vanilla MOPD. It also improves overall performance in low-conflict and three-domain settings. Our contributions are as follows: • We identify a failure mode of domain-routed MOPD: when domain updates are poorly compatible, separately assigned supervision can induce persistent gradient conflicts in shared parameters. These conflicts accompany asymmetric performance degradation, highlighting the importance of update compatibility for joint learning. • We propose ROAD MOPD, using non-routed teachers as cross-domain critics on the same student trajectories. Cross-teacher agreement guides token weighting, while preventive gradient projection adapts its strength under a norm-retention constraint to protect targetdomain learning signals. 2

Paper Under Review

• Across domain combinations and teacher–student configurations, ROAD MOPD improves adversely affected domains under high conflict, preserves low-conflict gains, and generalizes to three domains, outperforming standard MOPD and gradient-conflict correction baselines in overall performance.

2

R ELATED W ORK

2.1

O N -P OLICY D ISTILLATION

Classical knowledge distillation trains a student to match a teacher’s predictive distribution (Hinton et al., 2015). Sequence-level distillation applies this idea to autoregressive models using teachergenerated responses, but creates a mismatch between training and inference prefixes (Kim & Rush, 2016). On-policy distillation (OPD) instead queries the teacher on student-generated trajectories, using reverse-KL, general on-policy, or contrastive objectives (Gu et al., 2024; Agarwal et al., 2024; Ko et al., 2025). Subsequent work analyzes sampled-token estimators (Shao et al., 2026); makes supervision more selective through token filtering, reweighting, verifiable rewards, or reasoning progress (Li et al., 2026a; Lin et al., 2026; Yang et al., 2026a; Zhang et al., 2026); and extends OPD across tokenizers or intermediate generation steps (He et al., 2026; Sun et al., 2026a). Referencebased extrapolation further scales teacher corrections against a frozen student initialization (Yang et al., 2026b). Despite these advances, OPD can generalize unevenly across capabilities (Li et al., 2026b). These methods refine how evidence from one assigned teacher is estimated, selected, or scaled; they do not test its compatibility with capabilities represented by other teachers. We instead study how domain-specialized teachers can jointly assess the same student-generated prefixes. 2.2

M ULTI -T EACHER O N -P OLICY D ISTILLATION

Multi-teacher on-policy distillation (MOPD) transfers capabilities from domain specialists to a shared student, typically routing each rollout to its corresponding teacher for dense supervision (Ma et al., 2026). Domain routing is also used in large-scale and staged multi-domain post-training systems (Xiaomi LLM-Core Team et al., 2026; Yang et al., 2026c). Related methods adjust domain proportions during pretraining or multi-domain reinforcement learning (Xie et al., 2023; Liang et al., 2025), but do not coordinate teacher feedback on the same trajectory. Among recent MOPD methods, Open-MOPD balances token shares and teacher–student gaps while refreshing stale rewards (Gao et al., 2026), and D3 -MOPD schedules domains by their remaining distillation gaps and learning progress (Sun et al., 2026b). Other methods separate capability-recovery and domainpreservation updates (Chen et al., 2026), or calibrate confidence across heterogeneous teachers (Yin et al., 2026). Gradient-conflict methods such as PCGrad and conflict-averse gradient descent instead modify already-computed task gradients (Yu et al., 2020; Liu et al., 2021). None uses non-routed teachers to evaluate the current target-domain trajectory. ROAD MOPD lets routed and auxiliary teachers assess the same prefixes: their agreement guides token-level credit assignment, while auxiliary supervision defines preventive gradient constraints before domain updates are aggregated.

3

M ETHOD

3.1

P ROBLEM S ETUP AND O N -P OLICY D ISTILLATION

Let πθ be the student policy, πT a teacher, and D a prompt distribution. For x ∼ D, the student samples y = (y1 , . . . , yT ) with prefixes ht = (x, y<t ). OPD minimizes the sequence-level reverseKL objective: LOPD (θ; πT , D) = Ex∼D DKL (πθ (· | x) ∥ πT (· | x)) . (1) Following prior work (Yang et al., 2026b), we express OPD as relative-reward maximization with a KL penalty to a frozen reference πref . With reward scale λ > 0, we maximize   πT (y | x) Jλ (θ) = Eθ λ log − DKL (πθ (· | x) ∥ πref (· | x)) . (2) πref (y | x) 3

Paper Under Review

Road MOPD: Cross­Teacher Guarding

Vanilla MOPD: Domain-Routed Supervision Domain A data D A

Domain B data D B

···

x ∼ Dd

···

Σ

e.g. medical domain

Rollout on domain d data

rollout

Student πθ

y2

···

y3

yt

Shared prefixes

e.g. math domain

Routed Teacher Tr

Router

domain-specific

domain-specific

routed

evidence

TB

no interaction

auxiliary

y1

gradient conflict

y2

y3

y4

y5

gB

T

gA gB < 0

···

yt

gate agr

Agreement-modulated loss Ld

shared parameters θ Coupled updates

each domain → its teacher

shared params, conflicting grads

shared baseline

(B) Preventive Multi-Teacher Gradient Projection projected gradient

Targetdomain loss agr d

minimum norm retention

gd

L

g̃d

− g′d

projected gradient

···

>90°

Routed supervision

Frozen Reference π ref

auxiliary assessment

(A) Cross-Teacher-Guided Credit Assignment

Teacher B

TA

Auxiliary Teacher Ta

routed assessment

hard routing

Teacher A

gA

y1

gd

upweight downweight neutral

θ

g′d Auxiliary loss

L′d→a minimum norm retention

Figure 2: Vanilla MOPD and ROAD MOPD. Left: Vanilla MOPD’s domain-routed supervision can produce conflicting updates to shared parameters. Right: ROAD MOPD uses same-trajectory auxiliary-teacher feedback for (A) agreement-guided token credit assignment and (B) preventive gradient projection under a target-gradient norm-retention constraint. where Eθ averages over x ∼ D and y ∼ πθ (· | x). Maximizing this objective recovers standard OPD at λ = 1; 0 < λ < 1 attenuates the teacher’s relative reward, while λ > 1 amplifies it (ExOPD). The reference provides a shared baseline, not an additional teacher. For K domain–teacher pairs {(Dd , πTd )}K d=1 , MOPD routes each prompt to its domain teacher and minimizes the equal-weight objective: K

LMOPD (θ) =

1 X LOPD (θ; πTd , Dd ). K

(3)

d=1

For a fixed target domain d ∈ {1, . . . , K}, denote its routed teacher by Tr = Td and a teacher from an auxiliary domain by Ta . The upright labels r and a denote teacher roles, while d indexes domains. Both teachers and the reference score the same student-generated tokens under teacher forcing, without additional rollouts. We write ℓrt , ℓat , and ℓref t for their respective log-probabilities of yt given ht . 3.2

C ROSS -T EACHER -G UIDED C REDIT A SSIGNMENT

A fixed λ in Eq. (2) applies the same reward scale to every token, although a trajectory may contain both transferable reasoning patterns and domain-specific corrections. The routed teacher alone provides no cross-domain check on these corrections. We therefore query an auxiliary teacher on the same student-generated prefixes and use its feedback to adjust the routed teacher’s token-level reward. This strengthens supervision supported by both teachers and attenuates conflicting signals. Importantly, the auxiliary teacher is not treated as a supervisor for solving the target-domain task. Its relative preference is used only as a compatibility signal for the capability represented by that teacher. To compare teacher preferences, we measure each teacher’s log-probability relative to the shared frozen reference. Their token-wise agreement is  r   a  ℓt − ℓref ℓt − ℓref t t st = tanh × tanh , st ∈ [−1, 1]. (4) τ τ | {z } | {z } routed-teacher evidence

auxiliary-teacher evidence

4

Paper Under Review

Positive st means that both teachers favor the same direction of correction relative to the reference. Negative st indicates opposing preferences, with values near −1 corresponding to strong cross-teacher disagreement. The score approaches zero when either teacher provides little evidence beyond the reference. We use this agreement to replace the global reward scale in Eq. (2) with a token-wise coefficient: λt = clip(1 + ρg st , λmin , λmax ) .

(5)

Here, ρg > 0 controls the modulation strength; the subscript g denotes the agreement gate. When st > 0, λt > 1 strengthens the routed teacher’s relative reward on tokens supported by both teachers. When st < 0, λt < 1 attenuates the reward, with stronger disagreement producing greater attenuation. When st = 0, λt = 1 recovers standard OPD. Clipping limits both amplification and attenuation. For a sampled token yt and a comparison policy q, the k1 log-ratio estimator is k1,t (πθ ∥q) = log πθ (yt | ht ) − log q(yt | ht ). The modulated signal λt (ℓrt − ℓref t ) − k1,t (πθ ∥πref ) is clipped and detached as the advantage in the PPO-style surrogate ℓagr d,i,t (θ). Gradients flow through the policy importance ratio, not the advantage (Appendix A.1). With multiple auxiliary teachers, we average their token-wise agreement with the routed teacher before applying the same reward modulation (Appendix A.1). 3.3

P REVENTIVE M ULTI -T EACHER G RADIENT P ROJECTION

Token-level agreement does not guarantee compatible parameter updates, since token supervision signals are aggregated through shared parameters. We therefore formulate gradient correction as a constrained optimization problem: minimize conflict with an auxiliary teacher on the same trajectories while preserving a minimum fraction of the target-gradient norm. We first compute the target-domain gradient gd = ∇θ Lagr d from the agreement-modulated loss. To construct the auxiliary gradient, we replace the routed teacher with the auxiliary teacher and disable agreement modulation. Let ℓOPD d→a,i,t denote the resulting plain token-level OPD loss. Applying the same response-level reduction gives the auxiliary loss and gradient, which we distinguish from their target-domain counterparts using primes: L′d→a =

Ti 1 X 1 X ℓOPD , |Bd | Ti t=1 d→a,i,t

gd′ = ∇θ L′d→a .

(6)

i∈Bd

Both losses use the same stored domain-d trajectories. Thus, gd′ reflects the auxiliary teacher’s preferences on the current behavior, rather than its training gradient on auxiliary-domain data. Differences in knowledge, response style, or confidence can shape this signal; its usefulness as a constraint is assessed through joint-domain performance. We optimize the corrected gradient separately for each trainable parameter tensor. Let g and g ′ be nonzero slices of gd and gd′ , and let v denote a candidate corrected gradient. For an intended tensorlevel step −ηv with η > 0, the first-order change in the same-trajectory auxiliary loss is −η⟨v, g ′ ⟩. Thus, ⟨g, g ′ ⟩ < 0 indicates local conflict; otherwise, we retain g unchanged. For a conflicting pair, we restrict v to the line segment between g and its full projection onto the hyperplane orthogonal to g ′ (Yu et al., 2020): gfull = g −

⟨g, g ′ ⟩ ′ g, ∥g ′ ∥22

S(g, g ′ ) = conv{g, gfull }.

(7)

Here, conv denotes the convex hull. This restriction preserves the component orthogonal to g ′ and only removes the conflicting component, up to full projection. Full projection eliminates local conflict but can nearly erase g when the gradients oppose each other. We therefore choose the gradient with the least residual conflict subject to a norm-retention floor m ∈ (0, 1]: ge = arg min − ⟨v, g ′ ⟩

subject to

v∈S(g,g ′ )

∥v∥22 ≥ m2 ∥g∥22 .

(8)

On this segment, −⟨v, g ′ ⟩ ≥ 0 measures the predicted auxiliary-loss increase per unit step size, while the constraint preserves the target-gradient norm. The solution can be written as a partial 5

Paper Under Review

projection:

⟨g, g ′ ⟩ ′ g, α⋆ ∈ [0, 1]. (9) ∥g ′ ∥22 Here, α⋆ is the fraction of the conflicting component removed. We use full projection (α⋆ = 1) whenever it satisfies the norm floor; otherwise, we remove as much conflict as the floor permits, with ∥e g ∥2 = m∥g∥2 . For non-conflicting pairs, α⋆ = 0. The closed-form coefficient and its derivation are given in Appendix A.2. These are local statements about the same-trajectory auxiliary loss, not guarantees of auxiliary-domain performance after an optimizer step. Tensors with negligible gradient norm are left unchanged; auxiliary gradients are never applied as separate optimizer updates. The projected tensors form ged . ge = g − α⋆

With multiple auxiliary teachers, we apply the projections sequentially under a shared normretention floor relative to the original target gradient (Appendix A.2). Method-specific hyperparameters are summarized in Appendix B.3. A related approach in multi-task learning is PCGrad (Yu et al., 2020), which performs post-hoc gradient surgery by subtracting conflicting components from independently computed task gradients. Our projection is preventive: the auxiliary teacher assesses the current target-domain trajectory to construct a gradient constraint specifically for that update, using its feedback to constrain potentially incompatible corrections while retaining target-domain learning signals. We empirically compare these two projection strategies in Section 4.3. 3.4

OVERALL O BJECTIVE AND T RAINING

We use sequence-mean aggregation, averaging token losses within each response and then across responses and domains. Auxiliary losses use the same reduction but guide only the gradient correction, so the overall distillation objective includes only target-domain losses: T

K

Loverall (θ) =

i 1 X 1 X 1 X ℓagr (θ). K |Bd | Ti t=1 d,i,t

(10)

i∈Bd

d=1

After applying the preventive projection in Section 3.3, we average only the projected target-domain gradients, using the same domain weights as Eq. (10). The resulting gupdate is globally clipped for one AdamW step: K 1 X gupdate = ged . (11) K d=1

4

E XPERIMENTS

4.1

E XPERIMENTAL S ETUP

Evaluation datasets. We evaluate 11 benchmarks across Code (HumanEval+, MBPP+, and LiveCodeBench) (Liu et al., 2023; Jain et al., 2024), Math (AIME 2024/2025 and HMMT February/November 2025) (MAA, 2024; 2025; Harvard–MIT Mathematics Tournament, 2025a;b), and Medical (MedQA, MedXpertQA, MedMCQA, and SuperGPQA-Med) (Jin et al., 2020; Zuo et al., 2025; Pal et al., 2022; M-A-P Team et al., 2025). We report Avg@4 accuracy for Code, Avg@32 for Math, and Avg@1 for Medical. Avg@n averages correctness over n completions per problem and then over problems, equivalent to the same-sample pass@1 estimate for Math and Code. Baselines. We compare with four representative approaches: (1) Vanilla MOPD routes each onpolicy rollout only to its corresponding domain teacher and applies token-level reverse-KL supervision (Ma et al., 2026); (2) ExOPD constructs an extrapolated teacher target using the frozen student initialization as a reference (Yang et al., 2026b); (3) CaMOPD separates capability-recovery and domain-preservation updates and prioritizes samples with larger teacher–student gaps (Chen et al., 2026); and (4) Open-MOPD mitigates cross-domain imbalance through token-share balancing, dynamic domain loss weighting, and reward refresh (Gao et al., 2026). Implementation details. We use Qwen3-4B and Qwen3-1.7B students with thinking disabled (Yang et al., 2025). Math and Code teachers use specialized Qwen3-4B checkpoints (Yang et al., 6

Paper Under Review

Table 1: Main results across paired-domain settings. Domain Avg. averages benchmarks within each domain; Avg. averages the two domains. Bold/underlined scores mark the best/second-best multi-domain methods per student model. (a) Math × Medical Student Model: Qwen3-4B Math

Method

Medical MedQA

MedXpert QA

All

MedMCQA

SuperGPQA -Med

Domain Avg.

Avg.

AIME24

AIME25

HMMT25 Feb.

Student Single OPD

21.46 58.13

21.77 51.67

10.00 31.56

8.23 35.73

15.37 44.27

64.73 77.06

13.80 15.31

55.63 62.44

29.87 36.81

41.01 47.90

28.19 46.09

Vanilla MOPD ExOPD Open-MOPD CaMOPD Ours

58.75 61.67 56.56 61.15 63.85

55.21 55.62 55.10 54.27 53.96

31.98 33.54 32.60 33.12 33.65

37.92 37.92 37.08 38.02 38.96

45.97 47.19 45.34 46.64 47.60

62.61 69.99 70.31 77.45 77.06

12.24 13.76 13.06 16.61 15.67

55.39 59.50 60.94 61.89 62.83

27.55 32.23 33.76 35.03 37.35

39.45 43.87 44.52 47.75 48.23

42.71 45.53 44.93 47.20 47.92

HMMT25 Nov.

Domain Avg.

Domain Avg.

Avg.

HMMT25 Nov.

Domain Avg.

Student Model: Qwen3-1.7B Math

Method

Medical

All

AIME24

AIME25

HMMT25 Feb.

MedQA

MedXpert QA

MedMCQA

SuperGPQA -Med

Student Single OPD

11.98 39.90

10.21 34.48

6.04 18.75

3.96 17.29

8.05 27.60

46.03 53.81

10.33 13.39

42.55 50.99

22.58 24.50

30.37 35.67

19.21 31.64

Vanilla MOPD ExOPD Open-MOPD CaMOPD Ours

41.46 41.56 37.40 36.35 40.10

35.10 35.42 31.98 33.13 31.88

18.85 18.13 18.96 19.06 17.81

18.02 18.65 18.33 18.23 19.06

28.36 28.44 26.67 26.69 27.21

47.37 49.88 46.58 47.84 58.13

11.39 9.80 11.39 10.16 11.06

45.25 49.49 44.20 44.63 52.35

20.69 22.87 20.04 21.49 26.03

31.18 33.01 30.55 31.03 36.89

29.77 30.72 28.61 28.86 32.05

Domain Avg.

Avg.

(b) Code × Medical Student Model: Qwen3-4B Code

Method

Medical

All

MedQA

MedXpert QA

MedMCQA

SuperGPQA -Med

HumanEval+

MBPP+

LCB

Domain Avg.

Student Single OPD

76.68 84.30

65.08 70.44

17.14 28.00

52.97 60.91

64.73 77.06

13.80 15.31

55.63 62.44

29.87 36.81

41.01 47.90

46.99 54.41

Vanilla MOPD ExOPD Open-MOPD CaMOPD Ours

84.76 85.98 84.45 84.91 87.65

69.11 68.98 70.77 69.58 72.69

27.71 30.00 27.29 27.43 28.71

60.53 61.65 60.83 60.64 63.02

64.96 72.82 76.83 76.12 75.73

12.78 13.63 15.59 15.39 15.96

55.63 60.46 61.68 62.49 62.97

29.62 32.81 35.25 35.10 36.33

40.75 44.93 47.34 47.28 47.75

50.64 53.29 54.09 53.96 55.38

MedQA

MedXpert QA

MedMCQA

SuperGPQA -Med

Domain Avg.

Avg.

Student Model: Qwen3-1.7B Code

Method

Medical

All

HumanEval+

MBPP+

LCB v6

Domain Avg.

Student Single OPD

60.37 73.48

53.44 59.13

12.57 22.43

42.13 51.68

46.03 53.81

10.33 13.39

42.55 50.99

22.58 24.50

30.37 35.67

36.25 43.67

Vanilla MOPD ExOPD Open-MOPD CaMOPD Ours

71.95 75.76 71.95 71.65 73.78

58.27 62.30 58.60 55.29 61.44

23.00 24.43 20.43 23.00 24.43

51.07 54.16 50.33 49.98 53.22

46.11 44.62 45.80 47.53 56.56

11.02 10.37 10.78 10.33 12.24

45.68 44.85 43.46 45.80 51.35

20.91 19.24 19.64 21.71 25.88

30.93 29.77 29.92 31.34 36.51

41.00 41.97 40.12 40.66 44.86

2026b). The Medical teacher is II-Medical-8B, a Qwen3-8B RL checkpoint (Intelligent Internet, 2025). Each paired setting uses 10,000 sampled prompts per domain from DeepMath, Eurus-RLCode, and Medical sources including the MedMCQA training split (Table 5). Full configurations are in Appendix B. 4.2

M AIN R ESULTS

Best overall performance across domain pairs and student sizes. Across Math × Medical and Code × Medical with both student sizes, ROAD MOPD achieves the highest two-domain average among the compared multi-domain methods in all four settings (Table 1), exceeding the strongest overall baseline in each by 0.72–2.89 points. 7

Paper Under Review

Table 3: Three-domain generalization: Math × Code × Medical. Qwen3-4B domain averages (%) and their mean (All).

Table 2: Low-conflict generalization: Math × Code. Qwen3-4B domain averages (%) and their mean (All). Method

Math

Code

All

Student Single OPD

15.37 44.27

52.97 60.91

Vanilla MOPD ExOPD Open-MOPD CaMOPD Ours

46.12 48.28 44.32 46.30 47.73

61.24 61.82 59.82 60.37 63.07

Raw gradient conflict (%)

100

Math

Code

Medical

All

34.17 Student 52.59 Single OPD

15.37 44.27

52.97 60.91

41.01 47.90

36.45 51.03

53.68 55.05 52.07 53.34 55.40

46.98 47.09 44.06 45.96 47.29

61.26 61.52 60.56 60.97 62.94

41.03 45.62 38.83 48.20 48.45

49.76 51.41 47.82 51.71 52.89

Vanilla MOPD ExOPD Open-MOPD CaMOPD Ours

Math domain Medical

75 60 50

Method

46

48

76%

Math

Medical domain

52

40% 34

25 24

26

20 2

3

4

5 Highest

Math auxiliary

... , even if the attending says not to . Or perhaps the correct answer is B , the resident has to tell the attending that they can 't fail to disclose . The question asks which is the correct next action . Maybe the answer is A ? Wait , ...

0 1 Lowest

Medical auxiliary

... 2 0 4 minutes total . But let me just check again , maybe I made a mistake with the equations ? Wait , perhaps I should make sure that the steps are correct . Alternatively , let 's see if there 's another way . Let me check if the value ...

λt 0.5 attenuate

1.0 neutral

1.5 amplify

Teacher disagreement (S− quintile)

Figure 3: Teacher disagreement and local supervision. Left: raw gradient conflict by disagreement quintile (Math: blue; Medical: orange). Right: Math (top) and Medical (bottom) response windows; blue attenuates (λt < 1) and orange amplifies (λt > 1) token supervision.

4B students: Medical recovery accompanies gains in the other domain. In both Math × Medical and Code × Medical, Vanilla MOPD lowers Medical accuracy below the initial student’s 41.01, while ExOPD, Open-MOPD, and CaMOPD mitigate this degradation to varying degrees. ROAD MOPD reaches Medical averages of 48.23 and 47.75, respectively, recovering to at or near the Single OPD reference of 47.90 and exceeding the strongest multi-domain Medical baseline in each setting by 0.48 and 0.41 points. It also achieves the highest Math and Code averages in the respective settings, indicating that Medical recovery accompanies gains in the other domain’s average performance. 1.7B students: Larger Medical gains improve the cross-domain trade-off. With the smaller student, all multi-domain baselines fall below the Single OPD Medical average of 35.67 in both pairs. ROAD MOPD instead reaches 36.89 and 36.51, respectively, exceeding this reference in both settings. Its margins over the strongest multi-domain Medical baseline in each setting increase to 3.88 and 5.17 points. Although its Math and Code averages trail ExOPD by 1.23 and 0.94 points, respectively, the Medical gains yield two-domain averages that exceed ExOPD by 1.33 and 2.89 points, demonstrating a better overall result when domains are weighted equally. 4.3

F URTHER A NALYSIS

Generalizing to lower-conflict settings and more domains. Tables 2 and 3 show that ROAD MOPD also improves joint performance in low-conflict and three-domain settings. For Math × Code, it achieves the best Code and second-best Math averages among the compared multidomain methods. Although its Math average trails ExOPD by 0.55 points, its 1.25-point Code gain yields the highest All Avg. of 55.40, exceeding ExOPD by 0.35 points. With all three domains, ROAD MOPD leads in every domain and achieves the best All Avg. of 52.89, surpassing CaMOPD by 1.18 points. This setting also tests multiple auxiliary teachers, using aggregated agreement for credit assignment and sequential constraints for gradient protection. Full per-benchmark results are in Appendix C.2 (Tables 9 and 10). 8

Paper Under Review

Mean layer cosine

a

PCGrad

1

Math × Medical

0 μ = −0.529

−1 1

Code × Medical

μ = +0.033

0 −1

μ = −0.391

1

20

Training step

Math × Med.

b Accuracy (%)

Ours

μ = +0.076

40 35

46.8

47.6

Math

48.0

58

Code × Med. 60.9 63.0

60 50

40

48.2

Med.

47.6

Code

47.8

Med.

Figure 4: Preventive projection vs. PCGrad. (a) Mean layerwise cosine; labels average shaded steps 39–58. (b) Domain accuracy (%).

Table 4: Ablations of ROAD MOPD. Domain-average accuracy (%). Avg.: mean across the two domains. Math × Med.

Math

Med.

Avg.

w/o Credit Assign. w/o Grad. Protect. w/o Adaptive Ret.

45.78 47.19 45.78

47.74 48.11 48.40

46.76 47.65 47.09

ROAD MOPD

47.60

48.23

47.92

Code × Med.

Code

Med.

Avg.

w/o Credit Assign. w/o Grad. Protect. w/o Adaptive Ret.

59.54 63.01 62.85

47.46 46.87 47.03

53.50 54.94 54.94

ROAD MOPD

63.02

47.75

55.38

Cross-teacher disagreement is associated with gradient conflict, supporting token-level credit assignment. To examine whether cross-teacher agreement reflects update compatibility, we sample response batches at a fixed student checkpoint, averaging max(−st , 0) (Section 3.2) over all valid tokens per batch to obtain disagreement S− . Within each domain, we sort batches by S− from low to high into five equal-sized groups. Figure 3 (left) plots each group’s fraction of batches with negative cosine between raw full-parameter OPD gradients under the two teachers on identical responses. Conflict rates trend upward from 24% to 40% in Math and 46% to 76% in Medical. This suggests that teacher disagreement can signal potential update conflicts, providing empirical support for agreement-guided credit assignment. Figure 3 (right) visualizes token-level credit assignment: relative to the shared reference, opposing teacher corrections on Alternatively in Math and The in Medical attenuate the routed teacher’s relative reward signal, whereas aligned corrections on perhaps and Maybe in the respective domains amplify it. Probe details are provided in Appendix B.5; full prompts and visualizations of extended responses appear in Appendix C.3 (Figure 5). Preventive projection improves joint learning over post-hoc correction. Figure 4 compares preventive projection with PCGrad under matched training settings; both retain cross-teacher credit assignment and the same norm-retention mechanism. PCGrad corrects conflicts between gradients computed independently on each domain’s own trajectories. Our method instead constrains targetdomain updates using auxiliary-teacher gradients on the same rollout. This directly tests compatibility between the target gradient and auxiliary-teacher feedback on the same behavior. Unlike PCGrad’s comparison of independent domain gradients, this constraint can address local conflicts that may persist even when those gradients align. Figure 4a shows negative mean layerwise cosines for same-trajectory pairs during much of training, whereas independently formed domain-gradient pairs are mostly non-negative. These patterns reflect different local compatibility criteria and do not directly measure capability damage. Across Math × Medical and Code × Medical, ROAD MOPD improves all four domain-average accuracies over PCGrad by 0.18–2.12 percentage points (Figure 4b). These gains support the utility of same-trajectory constraints for joint learning. Each component improves joint performance. Table 4 shows that removing either credit assignment or gradient protection lowers all four domain averages across the two domain pairs. Without credit assignment, Math and Code accuracy drop by 1.82 and 3.48 points, respectively; removing gradient protection reduces Medical accuracy by 0.88 points in Code × Medical. Replacing adaptive retention with full projection lowers the two-domain mean by 0.83 and 0.44 points in Math × Medical and Code × Medical, respectively, despite a 0.17-point Medical gain in the former. The full method achieves the highest equally weighted two-domain mean in both settings. These results support adaptive retention as a way to balance conflict reduction with target-domain learning, alongside credit assignment and gradient protection.

9

Paper Under Review

5

C ONCLUSION

We identify update incompatibility as a central challenge in multi-teacher on-policy distillation: domain-routed supervision can produce conflicting updates to shared student parameters. ROAD MOPD addresses this challenge using auxiliary teachers to assess the same student trajectories. Agreement-aware token weighting and preventive gradient projection suppress conflicting supervision and updates while retaining target-domain learning signals. Across mathematics, medicine, and code, ROAD MOPD mitigates high-conflict degradation, retains gains in compatible settings, and achieves the best distilled-model average in each domain under joint three-domain training. Our evaluation is limited to three text-based reasoning domains, and auxiliary scoring and gradient computation add training cost. Extending to broader domains and larger teacher pools calls for efficient teacher selection and compatibility estimation. AI USE STATEMENT We used generative AI tools to assist with English translation, language polishing, and the presentation of figures and tables. This assistance included refining wording, improving clarity, and adjusting captions and layouts. The authors retain full responsibility for the paper’s technical content, results, citations, and conclusions, as well as the accuracy and integrity of the final manuscript.

R EFERENCES Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/ 5be69a584901a26c521c2b51e40a4c20-Abstract-Conference.html. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. HuatuoGPT-o1, towards medical complex reasoning with LLMs. arXiv preprint arXiv:2412.18925, 2024. Tianlei Chen, Jiao Ou, Ziyuan Liu, Ruiming Tang, Jian Liang, and Han Li. Counteraction-aware multi-teacher on-policy distillation for general capability recovery with domain preservation. arXiv preprint arXiv:2605.27115, 2026. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, et al. Process Reinforcement through Implicit Rewards. arXiv preprint arXiv:2502.01456, 2025. URL https: //arxiv.org/abs/2502.01456. DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948, 2025. URL https://arxiv.org/abs/2501. 12948. DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, et al. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348, 2026. URL https://arxiv.org/abs/2606.19348. FreedomIntelligence. Medical-R1-Distill-Data. Hugging Face dataset, 2025. URL https://huggingface.co/datasets/FreedomIntelligence/ Medical-R1-Distill-Data. Huan-ang Gao, Haohan Chi, Yong Yan, Shiyuan Feng, Hanlin Wu, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, and Hao Zhou. Open-mopd: Diagnosing and fixing capability imbalance in multi-teacher on-policy distillation. arXiv preprint arXiv:2608.19098, 2026. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In Proceedings of ICLR, 2024. Harvard–MIT Mathematics Tournament. Archive of february 2025. Official tournament archive, 2025a. URL https://www.hmmt.org/www/archive/282. 10

Paper Under Review

Harvard–MIT Mathematics Tournament. Archive of november 2025. Official tournament archive, 2025b. URL https://www.hmmt.org/www/archive/291. Haonan He, Haodi Lei, Yun Luo, Haoran Zhang, Shunkai Zhang, Yizhuo Li, Shengji Tang, Zhilin Wang, Runzhe Zhan, Lei Bai, Ganqu Cui, Fangchen Yu, Yafu Li, Peng Ye, Ning Ding, and Yu Cheng. Simpleopd: Simple tokenizer-agnostic on-policy distillation for long-context reasoning. arXiv preprint arXiv:2608.14277, 2026. Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. DeepMath-103K: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456, 2025. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531, 2015. URL https://arxiv.org/abs/1503.02531. Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, Chengquan Zhang, Zhuotao Tian, Han Hu, Yi Yang, Fei Wu, and Hehe Fan. Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe. arXiv preprint arXiv:2605.03677, 2026. URL https://arxiv.org/abs/2605. 03677. Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning with Large Language Models. arXiv preprint arXiv:2504.00869, 2025. URL https://arxiv.org/abs/2504.00869. Intelligent Internet. II-Medical-8B: Medical reasoning model. Hugging Face model, 2025. URL https://huggingface.co/Intelligent-Internet/II-Medical-8B. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081, 2020. Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1139. URL https://aclanthology. org/D16-1139/. Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and SeYoung Yun. DistiLLM-2: A contrastive approach boosts the distillation of LLMs. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 31044–31062. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/ko25a.html. Yuying Li, Leqi Zheng, Yongzi Yu, Wenrui Zhou, Xuchang Zhong, Xing Hu, Jing Jin, Hangjie Yuan, and Tao Feng. Filter, then reweight: Rethinking optimization granularity in on-policy distillation. arXiv preprint arXiv:2606.02684, 2026a. Zhaoyi Li, Deyang Kong, Yuan Wei, Evan Yang, Ranran Shen, Mahardika Krisna Ihsani, Ming Yang, Wei Zhang, Chuan Hao, Jian Yang, Ran Tao, Bryan Dai, Shikun Zhang, Wei Ye, Ying Wei, and Defu Lian. Every coin has two sides: On the dual nature of generalization in on-policy distillation of large language models. arXiv preprint arXiv:2608.16647, 2026b. Yiqing Liang, Jielin Qiu, Wenhao Ding, Zuxin Liu, James Tompkin, Mengdi Xu, Mengzhou Xia, Zhengzhong Tu, Laixi Shi, and Jiacheng Zhu. Modomodo: Multi-domain data mixtures for multimodal llm reinforcement learning. arXiv preprint arXiv:2505.24871, 2025. 11

Paper Under Review

Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, and Gao Huang. On-policy distillation with verifiable reward. arXiv preprint arXiv:2608.24696, 2026. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Conference on Neural Information Processing Systems (NeurIPS), 2021, December 2021. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Thirty-Seventh Conference on Neural Information Processing Systems, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/ hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract.html. M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, et al. SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines. arXiv preprint arXiv:2502.14739, 2025. URL https://arxiv.org/abs/2502.14739. Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, Jinhao Dong, Zhifang Sui, and Fuli Luo. Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training. arXiv preprint arXiv:2606.30406, 2026. Mathematical Association of America. 2024 AIME I Problems. Problem sets archived by Art of Problem Solving, 2024. URL https://artofproblemsolving.com/wiki/index. php?title=2024_AIME_I_Problems. Mathematical Association of America. 2025 AIME I Problems. Problem sets archived by Art of Problem Solving, 2025. URL https://artofproblemsolving.com/wiki/index. php?title=2025_AIME_I_Problems. OpenAI. GPT-4 Technical Report. Technical Report arXiv:2303.08774, OpenAI, 2023. URL https://arxiv.org/abs/2303.08774. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pp. 248–260. PMLR, 2022. URL https://proceedings.mlr.press/v174/ pal22a.html. Bing Shao, Jiazheng Zhang, Long Ma, Yujiong Shen, Senjie Jin, Xin Guo, Yuming Yang, Mingxu Chai, Zhiheng Xi, Boyang Liu, Junlin Shang, Tao Gui, Qi Zhang, and Xuanjing Huang. A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation. arXiv preprint arXiv:2608.25643, 2026. URL https://arxiv.org/abs/2608.25643. Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, Yanlong Du, Zifeng Cheng, Ziwei Jiang, and Qing Gu. Step-level onpolicy distillation: Interpolating between on-policy distillation and supervised fine-tuning. arXiv preprint arXiv:2608.16333, 2026a. Zechen Sun, Zhiwei Zhang, Fei Zhao, Juntao Li, Mu Chuan, Huayu Deng, Guojian Zhan, Wenliang Chen, Yao Hu, and Min Zhang. D3 -MOPD: Dynamic Domain ScheDuling for Efficient MultiTeacher Distillation. arXiv preprint arXiv:2608.24987, 2026b. URL https://arxiv.org/ abs/2608.24987v2. Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, Yihan Cao, Hui Ren, Xiang Li, Xiaoxiao Li, and Yuyin Zhou. MedReason: Eliciting factual medical reasoning steps in LLMs via knowledge graphs. arXiv preprint arXiv:2504.00993, 2025. Xiaomi LLM-Core Team, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, et al. MiMo-V2-Flash Technical Report. arXiv preprint arXiv:2601.02780, 2026. URL https:// arxiv.org/abs/2601.02780. 12

Paper Under Review

Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798– 69818, 2023. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/ 2505.09388. Chen Yang, Haiyuan Wan, Rengrong Xiong, Yize Chen, and Danny H. K. Tsang. Beyond imitation: Filtering on-policy distillation by reasoning progress. arXiv preprint arXiv:2608.19408, 2026a. Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026b. URL https://arxiv.org/abs/2602.12125. Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, Renjie Pi, Grace Lam, Nayeon Lee, Alexander Bukharin, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation. arXiv preprint arXiv:2603.19220, 2026c. URL https://arxiv.org/abs/2603.19220. Qixiang Yin, Huanjin Yao, Yuchen Cai, Jianghao Chen, Ziyi Wang, Min Yang, Fei Su, and Zhicheng Zhao. H-opd: Confidence aware heterogeneous multi-teacher multimodal on-policy distillation. arXiv preprint arXiv:2607.02592, 2026. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824–5836, 2020. Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao, and Yuezhi Zhou. Beyond teacher likelihood: Group-calibrated on-policy distillation for long-context reasoning. arXiv preprint arXiv:2608.19181, 2026. Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. MedXpertQA: Benchmarking expert-level medical reasoning and understanding. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 80961–80990. PMLR, 2025. URL https://proceedings.mlr.press/v267/zuo25a.html.

13

Paper Under Review

A PPENDIX C ONTENTS Page A Additional Method Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.1 OPD Derivation and Training Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.2 Derivation of Norm-Constrained Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.3 Complete Training Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 B

Experimental Setup and Reproducibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.1 Models and Teacher–Student Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.2 Training Data and Prompt Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.3 Training Hyperparameters and Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 B.4 Baseline Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B.5 Evaluation Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

C

Additional Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 C.1 Detailed Benchmark Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 C.2 Full Results for Low-Conflict and Three-Domain Settings . . . . . . . . . . . . . . . . . . . . . . . . . . 23 C.3 Extended Token-Level Visualizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.4 Training Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

D

Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

14

Paper Under Review

A

A DDITIONAL M ETHOD D ETAILS

A.1

OPD D ERIVATION AND T RAINING L OSS

From sequence-level reverse KL to token-level supervision. We expand the OPD reformulation in Section 3.1 before specifying its sampled-token implementation. For a prompt x ∼ D, let y = (y1 , . . . , yT ) ∼ πθ (· | x) and ht = (x, y<t ). Write pt = πθ (· | ht ) and qt = πT (· | ht ) for the student and teacher distributions on the same prefix. With Eθ denoting expectation over these prompts and student-generated responses, autoregressive factorization and the KL chain rule give LOPD (θ) = Ex∼D DKL (πθ (· | x) ∥ πT (· | x))   πθ (y | x) = Eθ log πT (y | x) # " T " T # X X pt (yt ) = Eθ = Eθ log DKL (pt ∥ qt ) . qt (yt ) t=1 t=1

(12)

The expectations assume finite divergences and common policy support. Thus, the token-level teacher distributions are evaluated on prefixes visited by the student. Equivalent relative-reward objective. Introducing a frozen reference πref and adding and subtracting its log-probability exposes the teacher’s relative reward (Yang et al., 2026b): −LOPD (θ) = Eθ [log πT (y | x) − log πθ (y | x)] 

 = Eθ log πT (y | x) − log πref (y | x) − log πθ (y | x) − log πref (y | x)   πT (y | x) − Ex∼D DKL (πθ (· | x) ∥ πref (· | x)) ≡ J1 (θ). = Eθ log πref (y | x)

(13)

Consequently, arg minθ LOPD (θ) = arg maxθ J1 (θ): the reward-maximization objective is the negative of the reverse-KL loss. Scaling only the relative-reward term by λ > 0 gives the objective in Eq. (2), whose sampled-response form is   πθ (y | x) πT (y | x) Jλ (θ) = Eθ λ log − log πref (y | x) πref (y | x) " T # (14) X

T ref θ ref = Eθ λ ℓt − ℓt − ℓt − ℓt , t=1

where ℓTt , ℓθt , and ℓref t are the teacher, student, and reference log-probabilities of the sampled token. At λ = 1, the reference terms cancel and recover OPD; 0 < λ < 1 attenuates the relative reward, whereas λ > 1 amplifies it (ExOPD). Domain routing applies this construction to each teacher– domain pair in Eq. (3). Credit assignment with multiple teachers. For the K > 2 extension of Section 3.2, let Pd = {j | j ̸= d} be the peer teachers for target domain d. Here we use explicit teacher indices: for each peer j, the main-text roles are Tr = Td and Ta = Tj , with ℓrt = ℓdt and ℓat = ℓjt . All peers score the same trajectory as the routed teacher, and their token agreements are averaged: !  d  j ref X 1 ℓt − ℓref ℓ − ℓ t t s̄d,t = tanh tanh t . (15) K −1 τ τ j∈Pd

We obtain λd,t by applying Eq. (5) with st = s̄d,t . For K = 2, this recovers Eqs. (4)–(5). For K > 2, the K − 1 peers reinforce or attenuate the routed signal according to their mean evidence; opposing assessments can cancel, recovering standard OPD when s̄d,t = 0. Sampled-token learning signal. The implementation replaces the fixed reward scale in Eq. (14) with the agreement coefficient from Section 3.2, using the sampled-token k1 log-ratio estimator and a PPO-style policy surrogate. Let πold denote the student policy that generates a stored response, 15

Paper Under Review

and let ℓθi,t = log πθ (yi,t | hi,t ) and ℓold i,t = log πold (yi,t | hi,t ). For a response from domain d, the routed teacher and shared frozen reference define ∆di,t = ℓdi,t − ℓref i,t . The advantage supplied to the policy loss is 

 d θ ref Aagr (16) d,i,t = sg clip λd,i,t ∆i,t − ℓi,t − ℓi,t , −10, 10 , where sg denotes stop-gradient. The entire recomputed advantage is clipped to [−10, 10] after agreement modulation and detached before entering the policy surrogate, including its student logprobability term. Teacher and reference parameters are frozen throughout training. When λd,i,t = 1, Eq. (16) reduces to the plain OPD learning signal sg[clip(ℓdi,t − ℓθi,t , −10, 10)]. Thus, the coefficient rescales the teacher’s reward relative to the reference; it does not multiply the entire OPD loss. Clipped policy surrogate and masking. The importance ratio and per-token loss are
ri,t (θ) = exp ℓθi,t − ℓold i,t , n o agr agr ℓagr d,i,t (θ) = − min ri,t (θ)Ad,i,t , clip(ri,t (θ), 1 − ϵPPO , 1 + ϵPPO ) Ad,i,t . P Let Mi,t indicate a valid response token and Ti = t Mi,t . The target-domain loss is P agr 1 X t Mi,t ℓd,i,t agr Ld = . |Bd | Ti

(17) (18)

(19)

i∈Bd

Prompt and padding positions do not contribute to the loss. The recorded training configuration uses ϵPPO = 0.2, without an outcome-based task reward or a separate KL reward added by the trainer. The single-turn rollout mask retains the returned generated tokens, including EOS when returned, and excludes padding. A response reaching the length cap retains supervision on its available prefix. Fully masked responses contribute zero. The agreement coefficient λd,i,t is bounded by [0.5, 1.5]. Auxiliary supervision. For each peer teacher j ̸= d, we reuse the same stored domain-d responses j θ and replace the advantage by AOPD d→j,i,t = sg[clip(ℓi,t − ℓi,t , −10, 10)]. The auxiliary loss uses the same clipped surrogate, with agreement modulation disabled and the same advantage clipping range. For the selected auxiliary teacher Ta = Tj , we write the main-text loss L′d→a as L′d→j . The full method and all ablations retain this range, including when credit assignment is disabled. Advantage clipping is separate from the importance-ratio clipping in Eq. (18). A.2

D ERIVATION OF N ORM -C ONSTRAINED P ROJECTION

For nonzero g and g ′ , let c = ⟨g, g ′ ⟩/(∥g∥2 ∥g ′ ∥2 ). When c < 0, every candidate in S(g, g ′ ) can be written as ⟨g, g ′ ⟩ v(α) = g − α ′ 2 g ′ , 0 ≤ α ≤ 1. (20) ∥g ∥2 The objective in Eq. (8) is −(1 − α)⟨g, g ′ ⟩, which strictly decreases with α. The optimum therefore uses the largest feasible coefficient. Expanding the squared norm gives ∥v(α)∥22 = 1 − (2α − α2 )c2 . ∥g∥22

(21)

√ This decreases monotonically on [0, 1]. Full projection is feasible if 1 − c2 ≥ m; otherwise, the optimum satisfies 1 − (2α − α2 )c2 = m2 . Including the no-conflict case gives  0, c ≥ 0,   √  1, r c < 0 and 1 − c2 ≥ m, α⋆ = (22) 2  √  1 − 1 − 1 − m , c < 0 and 1 − c2 < m. c2 Substitution into Eq. (9) recovers the unique solution of Eq. (8) for a conflicting pair. The norm constraint trades complete conflict removal for retention of the target-domain signal. In particular, ⟨e g , g ′ ⟩ = (1 − α⋆ )⟨g, g ′ ⟩, (23) 16

Paper Under Review

which remains negative when c < 0 and α⋆ < 1. These statements concern the projected tensor before domain aggregation, global clipping, and AdamW preconditioning. They do not guarantee a non-increasing auxiliary loss after the final optimizer step, nor a lower bound on the norm of the merged update. Multi-teacher projection and shared norm retention. For K > 2, each peer j ∈ Pd induces ′ a same-trajectory auxiliary gradient gd→j , computed as in Eq. (6). For each parameter tensor, we select the remaining valid peer with the most negative cosine, project against it, and recompute the cosines; processing stops when no valid negative cosine remains. These projections use the normconstrained form of Eq. (9) with a shared floor relative to the original target tensor: (q)

(0)

∥gd ∥2 ≥ m∥gd ∥2 , (0)

q = 1, . . . , K − 1,

(24)

(q)

where gd is the original target tensor and gd is the tensor after q projections. Once processing is complete, the resulting tensors form ged . The floor is measured against the original tensor throughout the sequence. Applying a fresh local retention ratio m independently at every projection would provide only a product bound and would not implement this shared constraint. In addition, a later projection can change alignment with a previously processed peer; sequential projection alone does not establish simultaneous non-conflict with every peer. For each conflicting peer, the implementation computes the coefficient from the remaining squared(q−1) 2 (0) 2 (q−1) ′ 2 2 norm budget. Write a = ∥gd ∥p , g ⟩ /∥g ′ ∥22 . It uses α = 1 2 , b = m ∥gd ∥2 , and u = ⟨gd if a − u ≥ b; otherwise, α = 1 − 1 − clip((a − b)/u, 0, 1). Thus, a tensor that has exhausted its shared budget receives no further projection. These comparisons use numerical tolerance 10−12 . A.3

C OMPLETE T RAINING P ROCEDURE

Algorithm 1 combines teacher scoring, token modulation, and all-peer gradient projection within one optimizer step, using the domain-mean gradient convention of the main text. All target and auxiliary gradients are evaluated at the same student parameters on stored trajectories; only projected target gradients enter the update. Batch normalization is detailed below, and archived projection configurations are specified in Appendix B.3. Token operations are elementwise over the valid responses in Bd ; stored scores and masks are restricted to that domain when used. M ASKED L OSS computes the loss in Eqs. (18) and (19) using valid response tokens. T ENSORWISE P ROJECT applies Eq. (9) separately to each parameter tensor. P ROJECT P EERS S HARED F LOOR denotes the sequential multi-peer procedure detailed in Appendix A.2, subject to the shared norm floor in Eq. (24). Auxiliary gradients are discarded after projection and never merged into gupdate . Domain means and global-batch normalization. Equation (19) and the algorithm define each gradient from a domain mean, and the algorithm averages the projected gradients across domains as in Eq. (11). In the implementation path supplied with full-batch normalization metadata, masking or slicing to domain d retains the denominator |B|. After microbatch accumulation and data-parallel averaging, its loss contribution is therefore Lbatch = wd Lagr d d , where wd = |Bd |/|B|; auxiliary losses on the same responses receive the same factor. Scaling both gradients by wd > 0 preserves the projection cosines and retention coefficients in exact arithmetic. To express this implementation using the algorithm’s domain-mean gradients, replace its accumulation by gupdate ← gupdate + wd ged . Balanced batches give wd = 1/K, recovering the equal-weight average in Eq. (11); shuffled batches use their observed domain fractions. For imbalanced batches, these weights differ from 1/K before global clipping and AdamW, so the two weighting conventions are not assumed to produce identical optimizer steps. Projection implementation. The paired-domain implementation projects each trainable FSDP parameter tensor separately. Gradient snapshots are stored in BF16, while dot products and squared norms are computed in FP64 and reduced across the data-parallel group. Shards of the same tensor therefore use a common cosine and projection coefficient. Tensors with negligible gradient norms are left unchanged; the numerical stability constant is 10−12 . With multiple peers, each tensor selects the most negative cosine among its unprocessed peers and recomputes these cosines after 17

Paper Under Review

Algorithm 1 One optimizer step of ROAD MOPD Require: Student parameters θ, optimizer state S, rollout policy πold Require: Frozen teachers {πTd }K d=1 , frozen reference πref , labelled prompts X Require: Modulation parameters (τ, ρg , λmin , λmax ), retention m, clip threshold γ Ensure: Updated student parameters θ and optimizer state S 1: B, ℓold , M ← ROLLOUT(πold , X ) 2: ℓ1:K , ℓref ← S CORE S AME T RAJECTORIES({πTd }K d=1 , πref , B) 3: gupdate ← 0 4: for d = 1, . . . , K do 5: Bd ← {i ∈ B : domain(i) = d}; Pd ← {1, . . . , K} \ {d} θ 6: ℓθ ← S TUDENT L OG P ROBS(πθ , Bd ); r ← exp(ℓ − ℓold )

P j d ref ref 1 7: s̄d,i,t ← K−1 tanh (ℓ − ℓ )/τ tanh (ℓ i,t i,t i,t − ℓi,t )/τ j∈Pd 8: λd,i,t ← clip(1 , λmax )  + ρg s̄d,i,td, λminref
 θ ref 9: Aagr ← sg clip λ (ℓ − ℓ d,i,t i,t i,t ) − (ℓi,t − ℓi,t ), −10, 10 d,i,t 10: Lagr ← M ASKED L OSS(r, Aagr d d , Md ) 11: gd ← ∇θ Lagr d 12: for j ∈ Pd do j θ 13: AOPD d→j,i,t ← sg[clip(ℓi,t − ℓi,t , −10, 10)] ′ 14: Ld→j ← M ASKED L OSS(r, AOPD d→j , Md ) ′ ← ∇θ L′d→j 15: gd→j 16: end for 17: if K = 2 then ′ , m), where Pd = {j} 18: ged ← T ENSORWISE P ROJECT(gd , gd→j 19: else ′ 20: ged ← P ROJECT P EERS S HARED F LOOR(gd , {gd→j }j∈Pd , m) 21: end if 1 22: gupdate ← gupdate + K ged 23: end for 24: gupdate ← G LOBAL N ORM C LIP(gupdate , γ) 25: (θ, S) ← A DAM WS TEP(θ, S, gupdate ) 26: return θ, S

each projection. Ties are broken by the configured peer order. Peers with negligible norms are skipped, and processing stops when no valid negative cosine remains. A selected peer is removed from the candidate set, even if the norm budget forces its coefficient to zero; previously selected peers are not revisited.

B

E XPERIMENTAL S ETUP AND R EPRODUCIBILITY

B.1

M ODELS AND T EACHER –S TUDENT C ONFIGURATIONS

Section 4.1 introduces the student and domain teachers. The specific Math and Code teacher checkpoints are Qwen3-4B-Non-Thinking-RL-Math-Step500 and Qwen3-4B-Non-Thinking-RL-CodeStep300, respectively (Yang et al., 2026b). The medical teacher is II-Medical-8B, a reinforcementlearning (RL) checkpoint based on Qwen3-8B (Intelligent Internet, 2025). The student undergoes full-parameter training; all teachers and the shared reference remain frozen and are used only for scoring. The models use compatible Qwen3 tokenizers and vocabularies, allowing the scorers to evaluate identical response token IDs at identical prefixes. Scoring checks require matching token IDs and log-probability tensor shapes. B.2

T RAINING DATA AND P ROMPT C ONSTRUCTION

Each paired training set contains 10,000 prompts per domain, for 20,000 prompts in total, and is randomly shuffled before training. For Math, we filter DeepMath (He et al., 2025) to samples with difficulty level ≥ 6, yielding a pool of 57K candidates, and sample 10,000 training prompts from this pool. For Code, we sample 10,000 prompts from Eurus-RL-Code (Cui et al., 2025). The Medical pool contains 10,000 prompts from four sources, as detailed in Table 5, including 4,000 sampled 18

Paper Under Review

from the MedMCQA training split (Pal et al., 2022). The MedMCQA validation split is used only for evaluation, not for constructing the training pool. Table 5: Composition of the training prompt pools. Counts denote sampled training prompts. Each paired setting uses the two corresponding domain pools. Domain

Source

Math Code

DeepMath (He et al., 2025), difficulty level ≥ 6 (57K candidates) Eurus-RL-Code (Cui et al., 2025)

Sampled prompts 10,000 10,000

Medical

MedMCQA training split (Pal et al., 2022) MedReason (Wu et al., 2025) Medical-R1-Distill-Data (FreedomIntelligence, 2025; Chen et al., 2024) m23k (Huang et al., 2025)

4,000 3,000 2,000 1,000

Domain labels select the routed teacher; only the prompt is supplied as the model input. Groundtruth answers are not used as outcome rewards in the distillation loss. Medical prompts request step-by-step reasoning with the final answer enclosed in \boxed{}. Thinking mode is disabled through the chat template, although generated responses can still contain intermediate reasoning. B.3

T RAINING H YPERPARAMETERS AND I NFRASTRUCTURE

Table 6 records the shared optimization and rollout settings of the archived ROAD MOPD runs. The implementation uses verl with an FSDP student and vLLM for rollout generation and teacher scoring. Baseline-specific update schedules are described in Appendix B.4. Table 6: Shared optimization and rollout settings of ROAD MOPD. Each rollout batch produces one optimizer update. Hyperparameter

Value

Training / PPO minibatch size Rollout batches (paired / three-domain) PPO epochs per rollout batch Responses per prompt Rollout temperature / top-p Maximum prompt / response length Optimizer Learning rate / schedule Adam betas / weight decay Global gradient norm clipping Advantage clipping PPO ratio-clipping parameter Distillation estimator Task reward / separate trainer KL reward Checkpoint interval In-training evaluation

1,008 / 1,008 prompts 58 / 116 1 1 1.0 / 1.0 2,048 / 16,384 tokens AdamW 10−5 / constant, no warm-up (0.9, 0.999) / 0.01 1.0, after merging domain gradients [−10, 10], full method and all ablations 0.2 Sampled-token k1 , policy-gradient surrogate Disabled / disabled 29 optimizer steps Disabled; offline checkpoint evaluation

The shared modulation parameters are τ = 1, ρg = 0.5, and [λmin , λmax ] = [0.5, 1.5], with 10−12 for numerical stability. Paired-domain training uses bidirectional projection with m = 0.6. Because agreement lies in [−1, 1], the unbounded mapping 1 + 0.5st already lies in [0.5, 1.5]; clipping does not further restrict this mapping in exact arithmetic. The paired- and three-domain configurations use one node with eight NVIDIA B200 GPUs. Six GPUs host the FSDP student and three rollout replicas with tensor-parallel size two. The remaining two GPUs host teacher scoring. In paired settings, one 4B teacher shares a GPU with the frozen reference; in three-domain training, the Math teacher, Code teacher, and reference share one GPU, while the Medical teacher occupies the other. Gradient checkpointing, padding removal, and dynamic batching are enabled. The three-domain ROAD MOPD sampler forms balanced batches of 336 prompts per domain. 19

Paper Under Review

The audited ExOPD, Open-MOPD, and CaMOPD runs use 1,008 prompts per rollout batch, with 58 batches in paired settings and 116 in three-domain settings. ExOPD and CaMOPD perform one optimizer update per batch. Open-MOPD divides that batch into four sequential minibatches of 252 prompts, giving 232 and 464 optimizer updates, respectively, while retaining one response per prompt. Thus, matching rollout budgets does not imply matching optimizer-update counts; CaMOPD also changes the allocation of prompts across domains through its alternating schedule. Student sizes, reference models, and Single OPD. For both ROAD MOPD and ExOPD, the reference is a frozen copy of the student’s initialization: Qwen3-4B in 4B experiments and Qwen31.7B in 1.7B experiments. For Qwen3-1.7B, Table 1(a)–(b) reports multi-domain checkpoints at step 58; Student denotes the original, undistilled model. Open-MOPD uses the sampled-token k1 estimator. Within each student size, Math, Code, and Medical Single OPD share the same training configuration and differ only in their domain-specific datasets. All three Qwen3-1.7B Single OPD runs start from the original model and train for 50 steps, evaluated with Avg@32, Avg@4, and Avg@1, respectively. Qwen3-4B Single OPD uses the matching in-domain checkpoints in Table 8. B.4

BASELINE I MPLEMENTATIONS

We implement the baselines within the same MyOPD/verl training stack, using the corresponding Qwen3-4B or Qwen3-1.7B initialization and domain teachers introduced in Section 4.1. Teacher scoring uses the sampled tokens of student-generated responses. The baseline paths use the sampledtoken k1 reverse-KL estimator and a clipped policy-gradient surrogate with clipping parameter 0.2; task rewards, entropy regularization, and a separate trainer KL reward are disabled. The methodspecific implementations are as follows. Vanilla MOPD. Each prompt is routed to its domain teacher (Ma et al., 2026). The token advantage log πTd (yt | ht ) − log πθ (yt | ht ) is clipped to [−10, 10] and detached, without cross-teacher modulation or gradient projection. The archived Math–Code and Code–Medical runs each use 58 optimizer updates, while the three-domain run uses 116. ExOPD. We use a fixed extrapolation coefficient of 1.25. The reference is the frozen student initialization, Qwen3-4B or Qwen3-1.7B according to student size (Yang et al., 2026b). For each sampled token, the detached advantage is 1.25(log πTd − log πref ) − (log πθ − log πref ), with all log probabilities computed at the same response prefix. The cross-teacher gate, gradient projection, and domain-balancing modules are disabled for this baseline. CaMOPD. The paired-domain implementation alternates homogeneous domain batches and selects examples using the rollout-time gap ∆t = log πTd (yt | ht ) − log πold (yt | ht ) (Chen et al., 2026). Recovery examples are scored by the mean of |∆t |, while preservation examples use the mean of max(∆t , 0). Within each branch, examples are ranked by this score and the smallest prefix covering 80% of the total score mass is retained. For Medical–Math and Medical–Code, Medical is the recovery branch: the update ratio is 3:1, and selected-example advantage scales are 2.0 for Medical and 1.0 for the other domain. Math–Code uses Math as recovery and Code as preservation, a 1:1 update ratio, and unit scales for both branches. The three-domain adaptation treats domains as peer branches, using the positive-gap score, the same 0.8 score-mass threshold, and unit selectedexample weights. We sample Medical, Math, and Code data in a 3:1:1 ratio, prioritizing the weaker Medical domain in line with CaMOPD’s capability-recovery objective. Open-MOPD. We adapt its balancing and reward-refresh mechanisms to the sampled-token OPD path (Gao et al., 2026). At rollout time, a domain’s weight is proportional to its uniform target share divided by its observed response-token share, multiplied by its mean absolute teacher–student gap relative to the mean gap across domains. This gap factor uses exponent 1.0 and is clipped to [0.05, 20]. The weights are normalized to have token-weighted mean one and remain fixed within the rollout batch. Dynamic allocation is therefore implemented through domain loss weights. Each batch is split into four sequential optimizer minibatches. On each minibatch, the OPD advantage is recomputed using the current student’s log probabilities and cached teacher scores, providing reward refresh. 20

Paper Under Review

PCGrad. For the two-domain comparison (Yu et al., 2020), separate backward passes mask the shared minibatch to each domain’s own responses. For each trainable parameter tensor, the implementation computes the two gradients’ dot product and squared norms across FSDP shards. If the dot product is negative, both gradients are projected against the original opposing gradient; the two corrected gradients are then summed. This symmetric two-task calculation requires no random task ordering. Gradient snapshots use BF16 and dot products and squared norms use FP64. The comparison applies global gradient clipping after merging, followed by one optimizer step. These gradients are formed on their respective domains’ responses, following the independent-domain comparison in Section 4.3; same-trajectory auxiliary gradients are not used for this projection. The archived PCGrad runs whose benchmark scores appear in Table 7 also enable reward-agreement modulation and are marked with a dagger in that table. B.5

E VALUATION P ROTOCOLS

We evaluate 11 benchmarks across Code (HumanEval+, MBPP+, and LiveCodeBench) (Liu et al., 2023; Jain et al., 2024), Math (AIME 2024/2025 and HMMT February/November 2025) (MAA, 2024; 2025; Harvard–MIT Mathematics Tournament, 2025a;b), and Medical (MedQA, MedXpertQA, MedMCQA, and SuperGPQA-Med) (Jin et al., 2020; Zuo et al., 2025; Pal et al., 2022; M-A-P Team et al., 2025). Generation setup. Offline evaluation uses vLLM with temperature 1.0, top-p 1.0, and a maximum of 16,384 generated tokens per completion. Chat prompts disable thinking mode via enable_thinking=False. Mathematics. Each of the four Math test sets contains 30 questions, with 32 sampled completions per question. We append the instruction “Please reason step by step, and put your final answer within \boxed{}.” The evaluator extracts the last boxed answer and uses math-verify to compare it with the reference answer; a missing or malformed boxed answer is incorrect. The reported Avg@32 is the mean correctness over completions, averaged over questions: Avg@32 =

N 32 1 X 1 X 1[âi,r = ai ]. N i=1 32 r=1

(25)

Here equality denotes equivalence under the answer verifier. Avg@32 equals the pass@1 estimate computed from the same 32 samples. This differs from pass@32, which counts a question as solved when at least one of its 32 completions is correct. Code. All three Code benchmarks use four completions per problem. We report Avg@4, the fraction of passing completions averaged over problems, which equals the pass@1 estimate from the same four samples. HumanEval+ contains 164 problems and MBPP+ contains 378. A completion must pass both the original and additional EvalPlus tests to count as correct. Prompts request a Python solution in a fenced code block. Our local evaluation protocol (v3) prioritizes Python or untagged Markdown code blocks, selects the candidate with the most nonempty lines, and uses treesitter to retain the target entry point and its dependencies. Generation continues until EOS or the token limit, without stopping at the first closing code fence. EvalPlus batches up to 256 generation requests across problems and sets each test’s execution limit to max(5 s, 6tref ), where tref is the reference solution’s runtime for that test. LiveCodeBench. We use the v6 code-generation split: 175 problems with contest dates from January 4 to April 6, 2025, giving 700 completions in total. The local loader selects test6.jsonl, with no additional date filtering. The runner uses the Qwen3-4B non-thinking prompt template and extracts the final fenced code block. Each completion is checked against the problem’s public and private tests with a 60-second per-test timeout. Compilation failures, runtime errors, timeouts, and failed tests count as incorrect completions. Medical. The evaluation uses one completion per question and reports Avg@1 multiple-choice accuracy. It covers the four-option MedQA-USMLE test set (1,273 questions), MedXpertQA Text test set (2,450; ten options), MedMCQA validation set (4,183; four options), and the SuperGPQA subset selected by discipline=Medicine (2,755; ten options). Each prompt includes the question 21

Paper Under Review

and labeled options, and requests step-by-step reasoning with the final answer letter in \boxed{}. The parser first checks the last boxed letter, then falls back to an explicit answer phrase such as “Answer: B” or a terminal “(B)”. Letters are normalized to uppercase and must belong to the question’s available options. Exact match to the reference letter determines correctness; unparseable answers are incorrect. Aggregation. All scores are reported as percentages. Each benchmark weights its questions equally, and domain averages are arithmetic means over the benchmarks within that domain, irrespective of their different numbers of questions. The 11-task average in Table 8 weights all 11 benchmarks equally; it is distinct from averaging the three domain means, because Code contains three benchmarks and the other domains contain four. Teacher-disagreement and gradient-conflict probe. For the fixed-checkpoint probe in Figure 3 (left), we compute S− = meant [max(−st , 0)] over all valid response tokens pooled across each four-trajectory batch, using Eq. (4) with τ = 1. We partition batches into five within-domain quintiles, with 50 batches per quintile. A raw conflict is a negative cosine between the full-parameter routed- and auxiliary-teacher OPD gradients on the same batch, without token modulation, projection, or optimizer updates. This five-bin analysis is exploratory; shading connects pointwise 95% Wilson interval limits. The probe is separate from the token-visualization samples in Appendix C.3.

C

A DDITIONAL E XPERIMENTAL R ESULTS

C.1

D ETAILED B ENCHMARK R ESULTS

Table 7 provides available benchmark breakdowns for the ablations and PCGrad comparison. Table 7: Per-benchmark breakdown of the ablations and PCGrad comparison. Avg. is the benchmark mean; rounding may cause small differences from the mean of displayed scores. Scores are percentages (higher is better). (a) Math × Medical Math

Medical

Method

AIME24 AIME25

HMMT 25 Feb.

HMMT 25 Nov.

Avg.

MedQA

Med Xpert QA

w/o Credit Assign. w/o Grad. Protection w/o Adaptive Retention PCGrad†

59.69 61.46 58.13 62.60

53.54 54.48 55.42 53.54

32.71 33.75 33.02 33.75

37.19 39.06 36.56 37.50

45.78 47.19 45.78 46.85

76.98 76.04 77.53 77.93

15.88 15.39 15.59 14.65

62.08 63.28 63.61 62.54

36.01 37.71 36.84 36.99

47.74 48.11 48.40 48.03

ROAD MOPD

63.85

53.96

33.65

38.96

47.60

77.06

15.67

62.83

37.35

48.23

MedMCQA

GPQA Med

Avg.

(b) Code × Medical Code

Medical

Method

Human Eval+

MBPP+

LCB

Avg.

MedQA

Med Xpert QA

w/o Credit Assign. w/o Grad. Protection w/o Adaptive Retention PCGrad†

84.15 87.65 86.74 84.45

69.05 71.96 73.68 70.11

25.43 29.43 28.14 28.14

59.54 63.01 62.85 60.90

76.20 75.18 76.12 76.90

16.08 14.16 14.08 15.71

61.77 61.53 62.25 62.49

35.79 36.62 35.68 35.17

47.46 46.87 47.03 47.57

ROAD MOPD

87.65

72.69

28.71

63.02

75.73

15.96

62.97

36.33

47.75

MedMCQA

GPQA Med

Avg.

† The archived PCGrad runs also enable reward-agreement modulation.

Single-domain reference results. Table 8 lists the available single-domain checkpoints across all 11 benchmarks, complementing Figure 1.

22

Paper Under Review

Table 8: Cross-domain evaluation of single-domain training checkpoints. Each Qwen3-4B single-domain model is trained with OPD on 10,000 in-domain prompts for 3 epochs and evaluated across all 11 benchmarks. The best and second-best results among the three checkpoints are shown in bold and underlined, respectively. Benchmark Code HumanEval+ MBPP+ LCB v6 Avg. Math AIME24 AIME25 HMMT25 Feb. HMMT25 Nov. Avg. Medical MedQA MedXpertQA MedMCQA SuperGPQA-Med Avg. 11-Task Avg.

C.2

Teacher

Student

Math-only

Medical-only

Code-only

85.98 79.63 27.00 64.20

76.68 65.08 17.14 52.97

81.40 69.71 22.00 57.70

81.10 72.95 22.14 58.73

84.30 70.44 28.00 60.91

57.71 55.00 32.29 39.27 46.07

21.46 21.77 10.00 8.23 15.37

58.13 51.67 31.56 35.73 44.27

59.38 46.15 30.31 36.56 43.10

43.75 37.29 20.00 24.79 31.46

87.90 25.71 71.38 47.19 58.05 55.37

64.73 13.80 55.63 29.87 41.01 34.94

70.86 14.90 60.17 34.70 45.16 48.26

77.06 15.31 62.44 36.81 47.90 49.11

67.71 14.00 57.33 33.32 43.09 43.72

F ULL R ESULTS FOR L OW-C ONFLICT AND T HREE -D OMAIN S ETTINGS

Tables 9 and 10 provide the full per-benchmark Qwen3-4B results underlying the summaries in Tables 2 and 3, respectively. Domain averages weight benchmarks equally. Single OPD uses the matching in-domain checkpoints from Table 8. Table 9: Full results for low-conflict Math × Code. Per-benchmark Qwen3-4B scores underlying Table 2. Scores are percentages (higher is better); Avg. weights benchmarks equally within each domain. Bold and underlining mark the best and second-best multi-domain methods, respectively. Benchmark

Student

Single OPD

Vanilla MOPD

ExOPD

OpenMOPD

CaMOPD

Ours

Math AIME24 AIME25 HMMT25 Feb. HMMT25 Nov. Avg.

21.46 21.77 10.00 8.23 15.37

58.13 51.67 31.56 35.73 44.27

60.21 55.21 31.98 37.08 46.12

63.33 57.71 33.85 38.23 48.28

56.77 53.33 30.52 36.67 44.32

61.04 54.17 31.87 38.12 46.30

60.73 57.40 33.33 39.48 47.73

Code HumanEval+ MBPP+ LCB v6 Avg.

76.68 65.08 17.14 52.97

84.30 70.44 28.00 60.91

85.06 70.37 28.29 61.24

86.59 71.16 27.71 61.82

83.08 70.37 26.00 59.82

84.91 70.50 25.71 60.37

86.89 72.75 29.57 63.07

23

Paper Under Review

Table 10: Full results for generalization to three domains. Per-benchmark Qwen3-4B scores underlying Table 3. Scores are percentages (higher is better); Avg. weights benchmarks equally within each domain. Single OPD uses the matching in-domain checkpoint from Table 8. The best and second-best results among the compared multi-domain methods are shown in bold and underlined, respectively. Benchmark Code HumanEval+ MBPP+ LCB Avg. Math AIME24 AIME25 HMMT25 Feb. HMMT25 Nov. Avg. Medical MedQA MedXpertQA MedMCQA SuperGPQA-Med Avg.

C.3

Student

Single OPD

Vanilla MOPD

ExOPD

OpenMOPD

CaMOPD

Ours

76.68 65.08 17.14 52.97

84.30 70.44 28.00 60.91

86.28 70.63 26.86 61.26

84.76 71.23 28.57 61.52

85.52 70.44 25.71 60.56

84.45 70.90 27.57 60.97

86.59 72.95 29.29 62.94

21.46 21.77 10.00 8.23 15.37

58.13 51.67 31.56 35.73 44.27

61.04 57.50 31.67 37.71 46.98

59.90 55.31 34.90 38.23 47.09

55.73 54.17 31.25 35.10 44.06

59.48 54.90 31.98 37.50 45.96

61.88 54.48 33.54 39.27 47.29

64.73 13.80 55.63 29.87 41.01

77.06 15.31 62.44 36.81 47.90

64.49 12.78 57.14 29.69 41.03

73.29 14.57 60.94 33.68 45.62

60.17 12.78 55.06 27.30 38.83

76.75 16.69 62.54 36.81 48.20

77.61 15.96 63.11 37.10 48.45

E XTENDED T OKEN -L EVEL V ISUALIZATIONS

Figure 5 expands the exact trajectories shown in the right panels of Figure 3 to 256-token contexts. It includes the original prompts and marks the 52-token main-text windows, preserving every displayed token’s original coefficient. These illustrative windows were selected from 20 sampled responses per prompt to show both attenuation and amplification near an answer mention; they do not estimate the overall coefficient distribution. The visualization samples use the token-modulationonly checkpoint at step 58, temperature/top-p 1/1, and a 2,048-token generation cap. Both selected trajectories reached that cap.

24

Paper Under Review

(a) Mathematics AIME2024_0 | rollout 7 | response tokens [1792, 2048)

Every morning Aya goes for a 9-kilometer-long walk and stops at a coffee shop afterwards. When she walks at a constant speed of s kilometers per hour, the walk takes her 4 hours, including t minutes spent in the coffee shop. When she walks s+2 kilometers per hour, the walk takes her 2 hours and 24 minutes, including t minutes spent in the coffee shop. Suppose Aya walks at s+1/2 kilometers per hour. Find the number of minutes the walk takes her, including the t minutes spent in the coffee shop. Please reason step by step, and put your final answer within \boxed{}. Scored response context | outlined tokens: Figure 3 [1988, 2040)

says : " Find the number of minutes the walk takes her , including the t minutes spent in the coffee shop ." So , when she walks at s + · 1 / 2 km /h , the time spent walking is · 9 /( s + · 1 / 2 ) hours . Then add t minutes . But to convert the entire thing to minutes , we need to convert the walking time to minutes and add t . So : · Walking time : · 9 / ( s + · 1 / 2 ) hours . Convert to minutes : · 9 /( s + · 1 / 2 ) * · 6 0 minutes . Then add t minutes , which is · 2 4 . But s + · 1 / 2 is · 2 . 5 + · 0 . 5 = · 3 , as above . So · 9 / · 3 = · 3 hours . · 3 hours is · 1 8 0 minutes . · 1 8 0 + · 2 4 = · 2 0 4 minutes . So that 's · 2 0 4 minutes total . But let me just check again , maybe I made a mistake with the equations ? Wait , perhaps I should make sure that the steps are correct . Alternatively , let 's see if there 's another way . Let me check if the value of s is correct . From the initial Earlier context omitted. Generation limit reached at token 2048.

(b) Medical MedQA_test_0 | rollout 2 | response tokens [869, 1125)

A junior orthopaedic surgery resident is completing a carpal tunnel repair with the department chairman as the attending physician. During the case, the resident inadvertently cuts a flexor tendon. The tendon is repaired without complication. The attending tells the resident that the patient will do fine, and there is no need to report this minor complication that will not harm the patient, as he does not want to make the patient worry unnecessarily. He tells the resident to leave this complication out of the operative report. Which of the following is the correct next action for the resident to take? Options: A. Disclose the error to the patient and put it in the operative report B. Tell the attending that he cannot fail to disclose this mistake C. Report the physician to the ethics committee D. Refuse to dictate the operative report Reason briefly, then state the final answer as a single option letter. Scored response context | outlined tokens: Figure 3 [971, 1023)

the patient . · Alternatively , maybe the resident should follow the attending 's orders if the attending has authority . However , this can get tricky . · Also , if the resident is not disclosing errors , they might be violating their own ethical training . · Wait , the key is that even minor errors should be reported . The attending 's reasoning is perhaps that it 's not harmful , but the mistake itself needs to be documented . So , perhaps the correct answer is A , because the resident should disclose the error to the patient , even if the attending says not to . Or perhaps the correct answer is B , the resident has to tell the attending that they can 't fail to disclose . · The question asks which is the correct next action . Maybe the answer is A ? · Wait , the operative report is part of the medical record . So if the resident is dict ating the operative report , perhaps they can 't omit the error . So if the attending says to not report it , but the resident is in a position to tell the truth , then perhaps the resident has to go against the attending . · Alternatively , maybe the attending is the one who has the authority to decide what is reported . But generally , medical records must be accurate . · Hmm . Maybe the answer is A , because the resident Earlier context omitted. Response continues beyond this excerpt.

attenuation

amplification

λt 0.5

1.0

1.5

Figure 5: Extended Math and Medical examples from the right panels of Figure 3. Outlines identify the main-text windows within the longer contexts. Blue and orange denote attenuation and amplification on the unchanged scale λt ∈ [0.5, 1.5]. Token indices are zero-based, with half-open intervals. Omitted context and the generation limit are marked.

25

Paper Under Review

Additional fixed-window examples. Figures 6 and 7 show three further responses per prompt. We take the first three rollout IDs after excluding the response already shown in Figure 5: Math IDs 0, 1, and 2; Medical IDs 0, 1, and 3. In that order, we display the fixed 192-token windows [0, 192), [928, 1120), and [1856, 2048). This selection does not inspect coefficients or answer correctness. Prompt summaries accompany the plots; the full prompts appear in Figure 5. Mathematics: three additional sampled responses AIME2024_0 | same prompt; distinct rollout IDs

Prompt summary: Aya walks 9 km with a fixed coffee stop. The total time is 4 h at speed s and 2 h 24 min at s + 2. Find the total minutes at speed s + 1/2, including the same stop. The full prompt appears in Figure 5.

(a) Beginning | rollout 0 | tokens [0, 192) Okay , so I need to figure out this problem about A ya 's walk . Let me try to parse it step by step . · First , the problem says that every morning A ya walks · 9 kilometers and then stops at a coffee shop . When she walks at a constant speed of s km /h , the walk takes her · 4 hours , including t minutes in the coffee shop . Then , when she walks at s + · 2 km /h , the walk takes her · 2 hours and · 2 4 minutes , including the same t minutes in the coffee shop . Now , we need to find out how long the walk takes her when she walks at s + · 1 / 2 km /h , including the t minutes in the coffee shop . So , the key here is that the total time includes both the walking time and the time spent in the coffee shop . But the coffee shop time is the same t minutes in both scenarios . So , we need to find Response begins here. Later context omitted.

(b) Middle | rollout 1 | tokens [928, 1120) 1 . 2 5 = · 1 1 . 2 5 ? Wait , let me do that again . · 1 8 divided by · 1 . 6 . Alternatively , · 1 . 6 is · 8 / 5 , so · 1 8 divided by ( 8 / 5 ) is · 1 8 * · 5 / 8 = · 9 0 / 8 = · 1 1 . 2 5 . Yes , correct . So · 1 1 . 2 5 = s (s + · 2 ) So s ^ 2 + · 2 s - · 1 1 . 2 5 = · 0 · Multiply both sides by · 4 to eliminate the decimal ? Let 's see : s ^ 2 + · 2 s - · 1 1 . 2 5 = · 0 · Alternatively , maybe convert · 1 1 . 2 5 to · 4 5 / 4 ? Yes , · 1 1 . 2 5 = · 4 5 / 4 . So equation is s ^ 2 + · 2 s - · 4 5 Earlier context omitted. Later context omitted.

(c) End | rollout 2 | tokens [1856, 2048) t is · 2 4 minutes . Wait , but wait . The problem says " the number of minutes the walk takes her , including the t minutes spent in the coffee shop ." But when she walks at · 3 km /h , the time spent walking is · 3 hours , then adding t minutes ( 2 4 minutes ) for coffee . So total time is · 3 hours + · 2 4 minutes . But the answer needs to be in minutes . So convert · 3 hours to minutes : · 3 * 6 0 = 1 8 0 minutes . Plus · 2 4 minutes , total · 2 0 4 minutes . Wait , but wait , is that correct ? Wait , let me check again . Wait , in the original problem , the total time when walking at s is · 4 hours ( which includes t minutes ). So , when A ya walks at s , her total time is · 4 hours , which is time walking ( 9 /s hours ) plus t minutes . Similarly , Earlier context omitted. Generation cap reached at token 2048.

attenuation

λt

amplification 0.5

1.0

1.5

Figure 6: Three additional Math response excerpts. Panels show the beginning, middle, and end of three distinct sampled responses to AIME2024_0. Original token coefficients and the [0.5, 1.5] color scale are preserved. A dot represents a whitespace-only token; omitted context and the generation cap are marked.

26

Paper Under Review

Medical: three additional sampled responses MedQA_test_0 | same prompt; distinct rollout IDs

Prompt summary: A resident cuts and repairs a tendon; the attending asks that the complication be omitted from the record. Choose: A disclose and document; B tell the attending it must be disclosed; C report to ethics committee; D refuse to dictate. Full prompt: Figure 5.

(a) Beginning | rollout 0 | tokens [0, 192) Okay , so I need to figure out the correct next action for the resident in this scenario . Let 's see . The situation is that a junior orth opa edic surgery resident did a car pal tunnel repair , but accidentally cut a flex or tendon during the procedure . The tendon was repaired without complications , and the attending physician told the resident not to report this minor complication , saying the patient will be fine and there 's no need to worry . The resident is now faced with a dilemma about what to do next . The options are A through D . Let 's break them down . Option A says to disclose the error to the patient and put it in the operative report . But the attending told the resident not to report it , so maybe the resident is conflic ted here . However , the ethical obligation might require disclosure , even if the attending says otherwise . Option B is to tell the attending that he can 't fail to disclose the mistake . This seems like a direct confrontation , Response begins here. Later context omitted.

(b) Middle | rollout 1 | tokens [928, 1120) if the attending says not to . · But the attending says the patient will do fine , and it 's not harmful . If the resident disc loses the error , perhaps the patient might be worried , but the patient has a right to know . · However , the initial instruction is that the attending says there 's no need to worry . So maybe the resident 's next step is to conflict with the attending 's order . · Alternatively , maybe in the United States , under the legal framework , it 's the responsibility of the surgeon to report errors . So , even if the attending says not to , perhaps the resident must report . · But looking for the correct answer . · Alternatively , perhaps the answer is D . Ref use to dictate the operative report . So , the resident wouldn 't follow the attending 's order and instead choose not to dictate the report . So , the resident would write the report himself without including the error . But that seems contradictory . · Alternatively , the correct Earlier context omitted. Later context omitted.

(c) End | rollout 3 | tokens [1856, 2048) proper action is A . · Another approach : the operative report is supposed to be accurate . The resident 's duty is to document the case accurately . So even if the attending tells them to omit the error , they should document it . Therefore , the correct answer would be A . · Therefore , A would be correct . · However , according to the guidelines , sometimes certain errors might not be reported if they are minor , but the attending 's order here may not be considered . · Alternatively , the correct answer is D : Ref use to dictate . · Wait , the operative report is the resident 's responsibility to write . So if the resident is asked by the attending not to mention the error , but the resident knows it was an error , perhaps they should refuse to write the report . · This seems complicated , but if the attending says they should not mention it , perhaps the resident 's best course would be to document it . · In the medical field , errors are supposed Earlier context omitted. Generation cap reached at token 2048.

attenuation

λt

amplification 0.5

1.0

1.5

Figure 7: Three additional Medical response excerpts. Panels show the beginning, middle, and end of three distinct sampled responses to MedQA_test_0, using the same fixed-window rule and unchanged color scale as Figure 6. These are generated reasoning excerpts, not verified answer explanations.

27

Paper Under Review

C.4

T RAINING DYNAMICS

We complement the endpoint evaluations with complete training traces from three archived ROAD MOPD runs: Math × Medical, Code × Medical, and Math × Code. Each run has 58 updates with 1,008 prompts per update, one PPO epoch, and a 16,384-token response cap. Both domains use reward-agreement modulation and tensor-wise adaptive projection with norm-retention floor m = 0.6. All plots retain every logged step without smoothing. Loss and generation behavior. Figure 8 shows the logged signed policy surrogate, per-domain response lengths, and the fraction of responses reaching the cap. The logger captures only the task-0 primary loss contribution: Math in Math × Medical, and Code in the other two runs. This quantity can be negative and is neither the full two-domain objective nor a nonnegative KL estimate. It decreases sharply early in training, while generation lengths and cap rates continue to vary. These single-run traces describe training behavior; they do not establish convergence of both domain objectives or comparative test accuracy.

Code × Medical

Math only

0.1

Math × Code

Code only

Code only

0.01 0 −0.01

Mean response length (tokens)

Signed surrogate loss (symlog)

Math × Medical

16k

Math Medical

Code Medical

Math Code

12k 8k 4k 0

Responses at cap (%, both domains)

30

20

10

0

1

20

40

Training step

58

1

20

40

Training step

58

1

20

40

58

Training step

Figure 8: Training loss and generation behavior across three domain pairings. Columns show individual ROAD MOPD runs. Top: the logged task-0 surrogate contribution on a symmetric-log scale (linear within ±0.01); each panel identifies the recorded domain. Middle: mean response length for each domain. Bottom: percentage of responses across both domains that reach 16,384 tokens. Curves connect all 58 logged updates; negative loss values and fluctuations are preserved.

28

Paper Under Review

Gradient-norm retention during projection. Figure 9 examines when the norm constraint changes the projection strength. For each primary domain, the logger computes two norm ratios from the same pre-projection gradients: the ratio under full projection and the ratio under the chosen adaptive strength. Each curve reports the minimum over conflicting parameter-tensor units at that update (or one if none conflict). These are analytical projection diagnostics before merging and global gradient clipping, rather than measurements of the final optimizer displacement. The adaptive ratios stay at or above 0.6, within numerical precision, in all six domain traces. For example, in the Math branch of Math × Medical, the full-projection estimate falls below 0.6 in 40 of 58 updates and reaches a minimum of 0.253, whereas the adaptive minimum is 0.600. The Medical branch of that same run never crosses the floor, so the two curves coincide. This illustrates how the constraint limits norm loss when needed without changing full projection when its retained norm already satisfies the floor. The full-projection curve uses the gradients encountered by ROAD MOPD; it is not a separately trained baseline.

Math × Medical

Code × Medical

Math × Code

Math primary

Code primary

Math primary

Medical primary

Medical primary

Code primary

Min. norm ratio

1.0 0.8 0.6 0.4 0.2

Min. norm ratio

1.0 0.8 0.6 0.4 0.2

1

20

40

58

1

Training step Adaptive projection

20

40

58

Training step Full projection (estimate)

1

20

40

58

Training step Floor m = 0.6

Figure 9: Minimum norm retention over training. Each panel identifies the primary domain whose gradient is projected; columns use the same runs as Figure 8. Blue shows the logged ratio under adaptive projection; dashed orange shows the analytical full-projection estimate on the same gradients. The dotted line marks m = 0.6. Ratios are minima over conflicting parameter-tensor units, before merged-gradient clipping. Overlapping curves indicate identical minimum retention. All logged steps are shown without smoothing.

29

Paper Under Review

D

L IMITATIONS

Computational overhead. ROAD MOPD incurs additional training overhead compared with Vanilla MOPD because of auxiliary-teacher scoring and gradient processing. Reusing student trajectories avoids additional rollout generation. Future work will explore more efficient teacher evaluation and gradient correction to reduce this overhead. Limited evaluation of domain scalability. Our experiments cover mathematics, code, and medicine, with joint training evaluated on at most three domains (Table 3). Future work will assess whether the observed gains and training stability extend to broader domain collections and larger teacher pools.

30

Related documents

• Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation#1297873
• MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation#1105396
• Contrastive On-Policy Distillation#1143849
• Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation#1049433
• Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation#925371