Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation Tianlei Chen* , Jiao Ou*† , Ziyuan Liu, Ruiming Tang† , Jian Liang, Han Li Kuaishou Technology, Beijing, China [email protected], [email protected], [email protected] * Equal contribution. † Corresponding authors.
arXiv:2605.27115v1 [cs.AI] 26 May 2026
Abstract
2026). However, domain specialization faces a fundamental paradox: the process of specialization itself often weakens the general capabilities that made these models valuable in the first place, creating a need for capability preservation (McCloskey and Cohen, 1989; Kumar et al., 2022). Classical mitigation strategies mix general-domain data or use replay mechanisms to reduce forgetting (LopezPaz and Ranzato, 2017; Rolnick et al., 2019; Dong et al., 2024; Huang et al., 2024). However, these approaches heavily rely on massive amounts of high-quality original data and introduce challenges in optimizing training schedules. Recently, multi-teacher on-policy distillation (MOPD) has attracted attention as a way to recover diverse capabilities into a single student model by supervising student-generated trajectories with token-level teacher feedback (Yang et al., 2026; GLM-5 Team, 2026). However, existing MOPD pipelines often assume teacher-aligned prompt coverage: the prompts used during MOPD should sufficiently match the data distribution that produced the teachers (LLM-Core Xiaomi, 2026; BaichuanM3 Team, 2026). This premise places a heavy burden on data collection and can be expensive, timeconsuming, or impossible when teacher’s original post-training data are unavailable. To avoid over-reliance on costly collection of the teacher’s original data, we consider using proxy prompts to recover general capability. Specifically, starting from an open-source general model, we post-train it into a vertical-domain model, after which MOPD is applied to recover its lost general capability: the original open-source model serves as the general teacher, and the specialized model serves as both the student initialization and the domain teacher. In this situation, domain prompts are available from the specialization pipeline, while the general teacher’s original post-training data are unknown. Instead of attempting to reconstruct or cover this hidden distribution, we use readily
Domain specialization can improve LLM behavior in vertical domains, but often weakens the general capabilities inherited from the original model. Recent Multi-Teacher On-Policy Distillation (MOPD) pipelines recover model capabilities by supervising student-generated trajectories with teacher feedback, but typically assume teacher-aligned prompt coverage, requiring prompts to match the teachers’ training distributions. This assumption is difficult to satisfy when the general teacher is an opensource model whose post-training data are unknown. Instead of attempting to reconstruct this hidden distribution, we study general capability recovery with readily available proxy general prompts. We identify two failure modes of vanilla MOPD in this incomplete-coverage situation: recovery-preservation counteraction from mixing conflicting recovery and preservation gradients, and weak-signal flattening from uniformly averaging samples with unequal correction demand. We propose CounteractionAware Multi-Teacher On-Policy Distillation (CaMOPD), which addresses these issues with decoupled alternating training and gap-based sample selection. CaMOPD gives general recovery dedicated updates, periodically reviews domain prompts for preservation, and selects samples with larger averaged token-level teacher-student log-probability gaps to concentrate correction signals. Across role-play dialogue and medical reasoning QA scenarios, CaMOPD performs best in general recovery over baselines while maintaining domainspecific behavior. Gradient coherence analyses further support the intended effect of CaMOPD in producing more coherent correction signals.
1
Introduction
The remarkable success of large language models (LLMs) has accelerated their deployment across vertical domains, from role-play (Wang et al., 2025) to clinical decision support (Baichuan-M3 Team, 1
preservation step. This alternating schedule gives general recovery dedicated updates while periodically preserving domain behavior, thereby mitigating gradient counteraction between recovery and preservation. Second, CaMOPD uses gapbased sample selection. It selects samples with higher rankings in token-level teacher-student logprobability gaps. This concentrates updates on prompts with stronger correction demand, and our gradient coherence analysis shows that the selected subsets produce more aligned update directions. We validate CaMOPD in two stylistically different vertical-domain settings, role-play dialogue and medical reasoning QA. Evaluations show that, under the premise of preserving the acquired domain behavior, CaMOPD outperforms all baselines in general capability recovery. Furthermore, the success across such distinct scenarios verifies the broad applicability and generalization of our method. Our contributions are summarized as follows:
Figure 1: Conceptual overview and training-dynamics. Top: Two failure modes of MOPD under incompletecoverage teacher feedback: recovery-preservation counteraction and weak-signal flattening within a feedback source. Bottom: Training dynamics of cross-domain gradient dot products for Medical Reasoning QA and Role-Play Dialogue. The persistently negative values empirically support the recovery-preservation counteraction failure mode.
• We claim two MOPD failure modes under proxy prompts: recovery-preservation counteraction and weak-signal flattening. • We propose CaMOPD, integrating decoupling alternating training and gap-based sample selection to prevent gradient counteraction and concentrate high-demand correction signals.
available public general-domain prompts as proxy prompts, which may incompletely cover the opensource teacher’s true training distribution. Under this incomplete-coverage setting, we claim two MOPD failure modes as explanatory hypotheses, as illustrated in Figure 1: recovery-preservation counteraction, where gradients for general capability recovery and domain behavior preservation point in different directions and counteract when averaged in the same update, which is evidenced by persistently negative cross-domain gradient dot products during training; and weak-signal flattening, where a fraction of gradients from different prompts under the same teacher are small and directionally dispersed, flattening the large-norm gradients that are most useful for training. In this paper, we propose CounteractionAware Multi-Teacher On-Policy Distillation (CaMOPD) to address these two failure modes. First, inspired by replay methods that mitigate cross-task gradient interference in continual learning (Lopez-Paz and Ranzato, 2017; Rolnick et al., 2019), CaMOPD uses decoupling alternating training: by default, it performs three general-recovery steps followed by one domain-
• Evaluations across two stylistically distinct vertical domains validate our analysis, demonstrating the effectiveness of CaMOPD.
2
Related work
2.1
On-Policy Distillation
On-policy distillation (OPD) queries a teacher on student-generated outputs, making it a natural fit for LLM post-training on model-generated trajectories (Agarwal et al., 2024; Gu et al., 2024; Qwen Team, 2025; DeepSeek-AI, 2026). However, OPD signals are not uniformly useful across trajectories or tokens. One line of work studies signal imbalance, teacher-student compatibility, and noisy or low-coherence teacher feedback, showing that OPD can introduce unreliable update directions (Li et al., 2026; Fu et al., 2026; Xu et al., 2025, 2026). Another line improves OPD by calibrating, relaxing, selecting, or reweighting local supervision at the token or sample level (Zheng et al., 2026; Ko 2
et al., 2026; Zhang et al., 2026; Kim and Baek, 2026; Huang et al., 2025a; Xie et al., 2026; Xu et al., 2026). Unlike these methods that refine local signals, CaMOPD introduces gap-based sample selection to explicitly improve cross-sample gradient coherence. 2.2
gap-based sample selection (introduced in Section 3.3). 3.1
Problem Setup. As illustrated in Figure 2 (Part A), we consider general capability recovery after domain specialization under an incompletecoverage setting. Given a domain-specialized model, we aim to recover its general capabilities while preserving domain behaviors using proxy general prompts Dg and domain prompts Dd . The original open-source model serves as the general teacher Tg , while the domain-specialized model serves as both the domain teacher Td and the initialization for the target student policy πθ .
Multi-Teacher On-Policy Distillation
Multi-teacher OPD (MOPD) extends OPD by using multiple teachers as token-level guidance sources, and has been used in several recent LLM post-training pipelines. MiMo-V2-Flash and DeepSeek-V4 use MOPD-style distillation for base-model post-training and broad capability integration (LLM-Core Xiaomi, 2026; DeepSeekAI, 2026). Baichuan-M3, Nemotron-Cascade 2, and GLM-5 apply related ideas to verticaldomain or staged post-training, with the latter two using cross-stage distillation to integrate or recover capabilities that degrade across training stages (Baichuan-M3 Team, 2026; Yang et al., 2026; GLM-5 Team, 2026). Unlike these pipelines that assume fully aligned teacher data, CaMOPD targets an incomplete-coverage setting with proxy prompts, focusing on reducing cross-domain gradient conflicts and elevating signal coherence. 2.3
Technical Formulation of MOPD. Under the multi-teacher on-policy distillation (MOPD) paradigm (LLM-Core Xiaomi, 2026), the student policy πθ is optimized on rollouts y = (y1 , . . . , yT ) generated by πθ given prompt x ∼ Db (b ∈ {g, d}). Let ht = (x, y<t ). The token-level teacher feedback defines the advantage function Âb,t (θ) as the teacher-student log-probability gap: πTb (yt | ht ) Âb,t (θ) = sg log , (1) πθ (yt | ht )
Mitigating Catastrophic Forgetting
where sg[·] denotes the stop-gradient operation. The branch-wise Vanilla MOPD loss, with traininginference importance sampling parameter wt (θ), is formulated as: "
Capability degradation during specialization is closely related to catastrophic forgetting in continual adaptation (Luo et al., 2023). Existing mitigation strategies commonly mix domain data with general-domain or pretraining data (Dong et al., 2024; Béthune et al., 2025), or use replay mechanisms to revisit prior-task or synthesized examples (Lopez-Paz and Ranzato, 2017; Rolnick et al., 2019; Lesort et al., 2020; Huang et al., 2024). Inspired by replay, CaMOPD schedules recovery and preservation updates to mitigate gradient counteraction.
3
Preliminaries
LbMOPD (θ) = −Ex∼Db ,y∼πθ (·|x) # T 1X wt (θ)Âb,t (θ) log πθ (yt | ht ) , (2) T t=1
3.2
Failure Modes of Vanilla MOPD
We claim that Vanilla MOPD can induce two optimization failure modes under the incompletecoverage setting.
Method
Recovery-Preservation Counteraction. The first failure mode occurs between the two data streams. Vanilla MOPD unconditionally aggregates gradients from both branches during every parameter update step:
This section aims to address the general capability recovery after domain specialization problem under a practical incomplete-coverage setting. As illustrated in Figure 2, Part A depicts the proxyprompt training setup (detailed in Section 3.1); Part B illustrates the recovery-preservation counteraction and weak-signal flattening failure modes of Vanilla MOPD (analyzed in Section 3.2); and Part C presents our CaMOPD method, which resolves these issues via decoupled alternating training and
gmix (θ) = gg (θ) + gd (θ),
(3)
where gb (θ) = ∇θ LbMOPD (θ). The training dynamics of Vanilla MOPD show that the generalrecovery and domain-preservation gradients have 3
Figure 2: Overview of the general capability recovery after domain specialization problem, the Vanilla MOPD failure modes, and the CaMOPD design. A: General capability recovery after domain specialization under an incomplete-coverage setting. The original open-source model acts as the general teacher on proxy general prompts, while the specialized model serves as both the student and the domain teacher on domain prompts. B: Vanilla MOPD mixes both feedback sources in every update, which can cause recovery-preservation counteraction when general and domain gradients are misaligned, and weak-signal flattening when low-demand or dispersed samples dilute concentrated corrections within a branch. C: CaMOPD addresses these issues with decoupled alternating training, together with gap-based sample selection that concentrates each branch on samples with larger teacher-student correction demand.
3.3
persistently conflicting optimization directions, as reflected by the negative cross-domain gradient dot products in Figure 1. Consequently, part of the recovery direction is implicitly counteracted by the preservation direction within the same mixed update.
To resolve the aforementioned bottlenecks, we introduce a scheduled and score-gated distillation strategy that decouples cross-domain aggregation and refines within-branch correction density while fully preserving the MOPD importance sampling core.
Weak-Signal Flattening. The second failure mode occurs within each individual feedback source. Full-batch averaging uniformly aggregates all sampled examples: X X 1 gg = gg (xi ) + gg (xj ) , (4) |Bg | xi ∈Sg
Counteraction-Aware MOPD (CaMOPD)
Decoupled Alternating Training. Inspired by replay methods that mitigate cross-task gradient interference in continual learning (Lopez-Paz and Ranzato, 2017; Rolnick et al., 2019), CaMOPD prevents recovery-preservation counteraction by decoupling the two signals through a periodic training schedule:
xj ∈S / g
where Sg denotes a subset of samples with strong correction demand. The low-demand tail (xj ∈ / Sg ) flattens the contribution of high-demand samples by occupying batch weight. More critically, examples with weak effective learning signals often exhibit poor cross-sample gradient coherence, meaning their dispersed directions further dilute the large-norm gradients that are most useful for training.
S = G, . . . , G, D, | {z }
(5)
ng
where ng general recovery steps (G) are followed by a single domain preservation step (D). By updating only the active branch b = S(t) at each step, general and domain gradients are computed at different parameter points, reducing direct mixed4
update counteraction between recovery and preservation signals.
The complete end-to-end procedure is summarized in Algorithm 1.
Role-specific Gap-based Sample Selection. To mitigate weak-signal flattening, CaMOPD selects samples based on their correction demand within the active branch. We use the initial token-level gap at rollout generation time, ∆b,i,t = log πTb (yi,t | hi,t ) − log πθ (yi,t | hi,t ), as a label-free indicator. General recovery uses the token-average absolute gap to measure deviation from the general teacher:
Algorithm 1 Counteraction-Aware Multi-Teacher On-Policy Distillation (CaMOPD) Require: Initial student πθ0 , teachers {πTg , πTd }, data streams {Dg , Dd }, schedule S, gap-score mass targets {ρg , ρd }, recovery scale rg 1: for training step t = 0, . . . , T − 1 do 2: Set active branch b ← S(t) Set branch scale rb ← rg if b = g, otherwise rb ← 1 3: 4: Sample prompts Bb ∼ Db 5: Generate on-policy responses yi ∼ πθ (· | xi ) for xi ∈ Bb Compute initial token gaps ∆b,i,τ = log πTb (yi,τ | 6: hi,τ ) − log πθ (yi,τ | hi,τ ) 7: if b = g then P i |∆g,i,τ | 8: Score each sample by si ← T1i Tτ =1 9: else P i [∆d,i,τ ]+ 10: Score each sample by si ← T1i Tτ =1 11: end if 12: Select the smallest prefix length kb covering ρb of cumulative gap-score mass and set Sb = {x(1) , . . . , x(kb ) }
T
sg (xi ) =
i 1 X |∆g,i,t |. Ti
(6)
t=1
Conversely, domain preservation uses the tokenaverage positive gap to estimate preservation demand: Ti 1 X sd (xi ) = [∆d,i,t ]+ . (7) Ti t=1
This asymmetric scoring reflects the different roles of the two branches: general recovery treats any large teacher-student deviation as a useful recovery signal, whereas domain preservation prioritizes tokens that the domain teacher assigns higher likelihood than the current student, avoiding overpenalization of already plausible domain-specific behaviors. Given an active branch b and a gapscore mass target ρb , we sort samples by sb (xi ) in descending order, yielding x(1) , . . . , x(|Bb |) . We first choose the smallest prefix length kb that covers the target cumulative gap-score mass: ( Pk ) j=1 sb (x(j) ) kb = min k : P|B | ≥ ρb . (8) b j=1 sb (x(j) )
13: Compute LCaMOPD (θt ) on Sb using Eq. (9) b 14: Update θt+1 ← θt − η∇θ LCaMOPD (θt ) b 15: end for
4
Experiments
4.1
Experimental Setup
Role-Play Dialogue Instantiation. We use Qwen3-4B-Instruct-2507 as the base model and general teacher Tg . The role-play domain teacher Td is obtained by supervised fine-tuning the same base model on CoSER (Wang et al., 2025). During MOPD, the general-recovery branch uses 10K Nemotron (Bercovich et al., 2025) proxy prompts, while the domain-preservation branch uses 10K held-out long-turn CoSER prompts.
The selected subset is then the corresponding prefix Sb = {x(1) , . . . , x(kb ) }, and the loss calculation is performed on this subset. To intensify the optimization drive on these prioritized instances, we introduce a branch-specific scaling ratio rb (e.g., rg = 2, rd = 1) to scale their generated teacher feedback. The final CaMOPD objective for active branch b is formulated as: " 1 X CaMOPD Lb (θ) = − |Sb | xi ∈Sb # T i 1 X wt (θ) · rb Âb,t (θ) log πθ (yi,t | hi,t ) . Ti
Medical Reasoning QA Instantiation. We use Qwen3-8B as the base model and general teacher Tg . The domain-specialized initialization and domain teacher Td are both Intelligent-Internet/IIMedical-8B (Intelligent Internet, 2025), an opensource medical reasoning model post-trained from Qwen3-8B with medical SFT and DAPO RL (Yu et al., 2025). During MOPD, the general-recovery branch uses 10K Nemotron proxy prompts, and the domain-preservation branch uses 10K prompt-only medical examples. More data and training details are provided in Appendix A.
t=1
(9) 5
Table 1: Role-Play dialogue instantiation results in general capability benchmarks and Role-Play domain benchmarks. Higher is better for all metrics. Bold marks the best result among baselines. CaMOPD denotes our method. Model
General capability
Role-Play domain capability
GPQA Zebra Diamond Logic
HMMT LCB v5 LCB v6 25
IF Eval
Arena HP
Arena CW
LiveBench Story 1125 Cons.
Anthro.
Char. Fid.
Story Qual.
Avg.
General Teacher Role-Play Teacher
61.49 54.29
79.50 31.50
33.12 26.67
34.76 25.09
20.57 16.00
83.18 67.65
36.50 9.70
65.80 4.20
65.60 42.80
34.74 42.27
23.09 33.42
23.89 38.75
41.98 49.89
30.92 41.08
Vanilla MOPD Relaxed OPD SelecTKD
61.99 63.51 60.98
71.30 71.40 70.60
29.17 31.25 31.04
31.54 32.26 34.41
22.29 22.86 24.00
79.67 79.67 78.93
25.50 30.10 27.10
13.70 17.30 16.90
58.90 59.00 58.90
43.57 45.42 43.87
34.91 33.35 32.28
41.45 41.53 39.46
51.50 52.84 51.54
42.86 43.28 41.79
CaMOPD
62.63
76.30
31.04
37.28
24.00
80.41
33.20
33.30
63.50
47.81
33.96
44.65
53.47
45.00
Table 2: Medical Reasoning QA instantiation results in general capability benchmarks and Medical domain benchmarks. Higher is better for all metrics. Bold marks the best result among baselines. CaMOPD denotes our method. Model
General capability GPQA Zebra HMMT Diamond Logic 25 LCB v5 LCB v6
Medical domain capability
IF Eval
Arena Arena LiveBench MedQA MedXpertQA Med. HP CW 1125 USMLE Text Avg.
General Teacher Medical Teacher
62.50 62.37
84.80 71.60
45.00 28.33
58.42 48.75
31.43 27.43
84.66 48.98
24.80 10.90
36.50 36.40
52.10 34.40
79.03 86.17
17.22 22.90
48.13 54.54
Vanilla MOPD Relaxed OPD SelecTKD
60.10 61.49 58.46
72.60 73.20 72.10
33.96 32.92 33.54
51.25 53.76 54.48
24.00 29.14 29.71
48.06 48.61 49.17
14.80 15.40 15.20
33.00 34.90 35.60
42.80 40.70 41.80
86.02 85.94 85.78
23.55 24.12 23.51
54.79 55.03 54.65
CaMOPD
61.99
72.60
34.58
56.63
29.71
59.89
16.40
31.80
49.00
85.78
23.63
54.71
Evaluation. For general capability, we evaluate knowledge and reasoning with GPQADiamond (Rein et al., 2024), ZebraLogic (Lin et al., 2025), and HMMT25 from MathArena (Balunović et al., 2025); coding with LiveCodeBench v5 and LiveCodeBench v6 (Jain et al., 2024); instruction following with IF-Eval (Zhou et al., 2023); preference-based generation with the Hard Prompt and Creative Writing subsets of Arena-Hard v2 (Li et al., 2025); and aggregate live evaluation with LiveBench 1125 (White et al., 2025). For role-play capability, following CoSER (Wang et al., 2025), we evaluate Storyline Consistency, Anthropomorphism, Character Fidelity, and Storyline Quality. For medical reasoning QA capability, we evaluate MedXpertQA Text and MedQA-USMLE 4-option test accuracy. Appendix B details the scoring protocol for the general and medical reasoning QA benchmarks. 4.2
terprets the teacher-student log-likelihood ratio as a token reward and relaxes strict imitation with reward clipping and entropy-based token sampling. SelecTKD (Huang et al., 2025a) uses a proposeand-verify token selection rule, applying full distillation loss to accepted tokens while masking or down-weighting rejected tokens. 4.3
Main Results
Tables 1 and 2 report the main comparisons for role-play and medical reasoning QA specialization on both general benchmarks and domain-specific evaluations. Across these two stylistically different instantiations, CaMOPD achieves strong generalcapability recovery while maintaining the specialized domain behavior, demonstrating the effectiveness and generality of our counteraction-aware design. General Capability Recovery. CaMOPD consistently improves recovery over baselines on the general benchmarks. In the role-play instantiation, it raises LiveCodeBench v5 from 31.54 to 37.28, Arena-Hard v2 Creative Writing from 13.7 to 33.3, and LiveBench 1125 from 58.90 to 63.50. It also obtains the strongest scores on ZebraLogic, IF-Eval, Arena-Hard v2 Hard Prompt, and several other metrics. In the medical reasoning QA instanti-
Baselines
For each domain, we compare Vanilla MOPD, CaMOPD, and two OPD methods reproduced under the Vanilla MOPD training mechanism with the same student initialization, teachers, data streams, batch size, rollout budget, optimizer settings, and training steps. Relaxed OPD (Ko et al., 2026) in6
(a) Role-Play Gen. GCG
(b) Role-Play Dom. GCG
(c) Role-Play Gen. Sel.
(d) Role-Play Dom. Sel.
(e) Medical Gen. GCG
(f) Medical Dom. GCG
(g) Medical Gen. Sel.
(h) Medical Dom. Sel.
Figure 3: Training dynamics and gap-based sample selection analysis. Panels (a)–(d) show Role-Play dialogue, and panels (e)–(h) show Medical Reasoning QA. Within each row, the four panels report general-branch GCG, domain-branch GCG, general-branch selected fraction, and domain-branch selected fraction, respectively. GCG denotes gradient coherence gain of the selected subset over the full candidate batch, and selected-fraction plots report the retained fraction under the same 80% gap-score mass target.
ation, CaMOPD similarly improves over baselines on seven of nine general benchmarks, with large gains on LiveCodeBench v5 (51.25 to 56.63), IFEval (48.06 to 59.89), and LiveBench 1125 (42.80 to 49.00). These gains demonstrate the excellence and generalizability of our method.
the same 80% gap-score mass target. The general branch requires broad coverage in both domains. The domain branch is sparse in role-play dialogue but remains broader in medical reasoning QA, suggesting that domain correction demand can vary substantially across specialized tasks.
Domain Capability Preservation. MOPDbased training preserves the specialized domain behavior across both instantiation scenarios. Our method and the baselines all remain close to or above the corresponding domain teachers on the domain metrics, indicating that the MOPD training mechanism itself is sufficient to protect the acquired domain capability.
Within-branch Gradient Coherence. To examine whether this selection is associated with reduced weak-signal flattening, we measure the within-branch gradient coherence of the parameter updates. For a sample set A, we define P gi , Coh(A) = P i∈A i∈A ∥gi ∥
4.4
where gi is the parameter gradient of sample i. Higher values indicate more aligned gradient directions. We report gradient coherence gain (GCG) as the relative improvement of the selected subset over the full candidate batch:
Training Dynamics and Selection Analysis
We further analyze whether CaMOPD changes the training dynamics in a way consistent with our motivation. The goal of this analysis is to examine whether gap-based sample selection concentrates updates on samples with larger correction demand and reduces within-branch signal dilution.
GCG =
Selection Ratio Under The Same Mass Target. As a process monitoring step, Figures 3c, 3d, 3g, and 3h report the actual selected fraction under
Coh(Aselected ) − Coh(Afull ) × 100%. Coh(Afull )
Figures 3a, 3b, 3e, and 3f plot the GCG of the selected subset over the full candidate batch. Both domains show positive GCG, with especially 7
Table 3: Ablation results on the Role-Play dialogue instantiation. Higher is better for all metrics. w/o sel uses decoupled alternating training but disables gap-based sample selection. Bold marks the best result among the compared variants. w/o dec keeps mixed-update training without decoupled alternating updates, with the recovery-to-preservation data ratio shown in parentheses. Metric
w/o dec w/o dec (1:1) (3:1) w/o sel CaMOPD
General capability GPQA Zebra HMMT25 LCB v5 LCB v6 IF-Eval Arena-HP Arena-CW LiveBench
Figure 4: Bar-chart summary of the hyperparameter study. General Avg. is the arithmetic mean over the nine general benchmarks in Table 7; Role-Play Avg. uses the domain-average scores.
61.99 71.30 29.17 31.54 22.29 79.67 25.50 13.70 58.90
61.49 75.20 31.04 35.13 21.71 78.19 29.50 22.10 60.50
61.74 74.70 27.29 35.48 24.57 78.00 30.20 25.50 61.10
62.63 76.30 31.04 37.28 24.00 80.41 33.20 33.30 63.50
44.75 31.92 41.22 52.50 42.60
47.81 33.96 44.65 53.47 45.00
Role-Play domain capability Story Cons. Anthro. Char. Fid. Story Qual. RP Avg.
large gains in the role-play domain branch and smaller but consistent gains in medical reasoning QA. Together with the selection-ratio analysis, this suggests that gap-based sample selection reduces weak-signal flattening and produces more coherent update directions across both specialization settings. 4.5
43.36 34.49 39.36 50.79 41.99
recovery-heavy mixing helps general capability, but still suffers from mixed-update counteraction. Effect of Gap-based Sample Selection. w/o sel already improves several recovery metrics by using decoupled alternating updates. Full CaMOPD further adds gap-based sample selection and gives the best overall trade-off, achieving the strongest role-play average and the best or tied-best result on most general benchmarks.
Hyperparameter Analysis
We further examine the recovery-to-review schedule, the recovery scaling ratio rg , and the gap-score mass target ρg in the role-play setting. As summarized in Figure 4, the 3-1 schedule gives the best recovery-preservation trade-off; the 5-1 schedule achieves similar general recovery, but domain performance starts to decline. For recovery scaling and gap-based sample selection, rg = 2 and ρg = 0.8 gives the strongest aggregate general recovery, combining broader sample coverage with amplified gradients on selected high-demand samples. We therefore use the 3-1 schedule with rg = 2 and ρg = 0.8 in the main experiments. The full metric table and detailed analysis are provided in Appendix C. 4.6
43.57 34.91 41.45 51.50 42.86
5
Conclusion
We studied general capability recovery after domain specialization under an incomplete-coverage setting, where only proxy general prompts are available for the general teacher. We identified two failure modes of Vanilla MOPD: recoverypreservation counteraction and weak-signal flattening. To address them, we proposed CaMOPD, which combines alternating recovery/preservation updates with gap-based sample selection. Experiments on role-play dialogue and medical reasoning QA show that CaMOPD improves general recovery over baselines while maintaining domain capability. Gradient and selection analyses further suggest that CaMOPD reduces mixed-update counteraction and produces more coherent correction signals.
Ablation Study
Table 3 compares the ablation variants. Effect of Decoupled Alternating Training. Comparing w/o dec (1:1) and w/o dec (3:1) shows that increasing the recovery data ratio improves several general benchmarks, but lowers the role-play average from 42.86 to 41.99. This suggests that 8
Limitations
Louis Béthune, David Grangier, Dan Busbridge, Eleonora Gualdoni, Marco Cuturi, and Pierre Ablin. 2025. Scaling laws for forgetting during finetuning with pretraining data injection. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 4020–4042. PMLR.
Our experiments instantiate CaMOPD with 4B and 8B models in the role-play and medical reasoning QA domains, respectively. Due to computational constraints, we do not evaluate larger model scales in this work; extending CaMOPD to larger specialized models is an important direction for future exploration. In addition, we use a fixed 32K response-length budget for on-policy rollouts. This setting is chosen to avoid artificially limiting the output capacity of long-reasoning models during distillation, but it does not study how different rollout length budgets affect the recovery-preservation trade-off. Future work can examine CaMOPD under shorter and adaptive rollout budgets.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024. HuatuoGPT-o1: Towards medical complex reasoning with LLMs. arXiv preprint arXiv:2412.18925. DeepSeek-AI. 2026. DeepSeek-V4: Towards highly efficient million-token context intelligence. https://huggingface.co/deepseek-ai/ DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf. Technical report. Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. How abilities in large language models are affected by supervised fine-tuning data composition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.
Ethical Considerations This work studies post-training methods for recovering general capabilities while preserving domainspecific behavior. In the medical reasoning QA setting, the models and evaluations are used only for research on medical question answering and educational reasoning benchmarks. The resulting systems should not be interpreted as providing professional medical advice, diagnosis, or treatment recommendations. Any deployment in clinical or patient-facing settings would require substantially stronger validation, expert oversight, safety guardrails, and compliance with applicable medical regulations. Our training and evaluation use publicly available datasets and open-source models. We do not use private patient records or personally identifiable medical information.
Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. 2026. Revisiting on-policy distillation: Empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562. GLM-5 Team. 2026. GLM-5: ing to agentic engineering. arXiv:2602.15763.
From vibe codarXiv preprint
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. MiniLLM: Knowledge distillation of large language models. In International Conference on Learning Representations. Haiduo Huang, Jiangcheng Song, Yadong Zhang, and Pengju Ren. 2025a. SelecTKD: Selective tokenweighted knowledge distillation for LLMs. arXiv preprint arXiv:2510.24021.
References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations.
Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. 2024. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 1416–1428. Association for Computational Linguistics.
Baichuan-M3 Team. 2026. Baichuan-M3: Modeling clinical inquiry for reliable medical decision-making. arXiv preprint arXiv:2602.06570.
Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. 2025b. m1: Unleash the potential of test-time scaling for medical reasoning with large language models. arXiv preprint arXiv:2504.00869.
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. 2025. MathArena: Evaluating LLMs on uncontaminated math competitions.
Intelligent Internet. 2025. II-Medical-8B: Medical reasoning model. https://huggingface.co/ Intelligent-Internet/II-Medical-8B. Model card.
Akhiad Bercovich and 1 others. 2025. LlamaNemotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949.
9
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando SolarLezama, Koushik Sen, and Ion Stoica. 2024. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747. Michael McCloskey and Neal J. Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower, editor, Psychology of Learning and Motivation, volume 24, pages 109–165. Academic Press.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 248–260. PMLR.
Minsang Kim and Seung Jun Baek. 2026. Explain in your own words: Improving reasoning via tokenselective dual knowledge distillation. arXiv preprint arXiv:2603.13260. Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. 2026. Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137.
Qwen Team. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out-ofdistribution. In International Conference on Learning Representations.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof Q&A benchmark. In Conference on Language Modeling.
Timothée Lesort, Andrei Stoian, and David Filliat. 2020. Continual learning: Tackling catastrophic forgetting in deep neural networks with replay processes. arXiv preprint arXiv:2007.00487.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019. Experience replay for continual learning. In Advances in Neural Information Processing Systems, volume 32.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2025. From crowdsourced data to highquality benchmarks: Arena-Hard and BenchBuilder pipeline. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 34209–34231. PMLR.
Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen-tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Wei Wang, Yanghua Xiao, and Shuchang Zhou. 2025. CoSER: Coordinating LLMbased persona simulation of established roles. arXiv preprint arXiv:2502.09082. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh Agrawal, Sandeep Singh Sandha, Siddartha V. Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. LiveBench: A challenging, contamination-limited LLM benchmark. In International Conference on Learning Representations.
Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. 2026. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016. Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. 2025. ZebraLogic: On the scaling limits of LLMs for logical reasoning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 37889–37905. PMLR.
Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, Yihan Cao, Hui Ren, Xiang Li, Xiaoxiao Li, and Yuyin Zhou. 2025. MedReason: Eliciting factual medical reasoning steps in LLMs via knowledge graphs. arXiv preprint arXiv:2504.00993.
LLM-Core Xiaomi. 2026. MiMo-V2-Flash technical report. arXiv preprint arXiv:2601.02780.
Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu, and Jiangning Zhang. 2026. LLM-oriented token-adaptive knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence. Also available as arXiv:2510.11615.
David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30.
10
Wenda Xu, Rujun Han, Zifeng Wang, Long T. Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. 2025. Speculative knowledge distillation: Bridging the teacherstudent gap through interleaved sampling. In International Conference on Learning Representations. Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. 2026. PACED: Distillation and onpolicy self-distillation at the frontier of student competence. arXiv preprint arXiv:2603.11178. Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, Renjie Pi, Grace Lam, Nayeon Lee, Alexander Bukharin, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2026. Nemotron-Cascade 2: Post-training LLMs with cascade RL and multi-domain on-policy distillation. arXiv preprint arXiv:2603.19220. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, and 16 others. 2025. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler, Qian Qian, Gregory D. Lyng, Sanjit Singh Batra, and Robert E. Tillman. 2026. Fast and effective on-policy distillation from reasoning prefixes. arXiv preprint arXiv:2602.15260. Binbin Zheng, Xing Ma, Yiheng Liang, Jingqing Ruan, Xiaoliang Fu, Kepeng Lin, Benchang Zhu, Ke Zeng, and Xunliang Cai. 2026. SCOPE: Signal-calibrated on-policy distillation enhancement with dual-path adaptive weighting. arXiv preprint arXiv:2604.10688. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. MedXpertQA: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362.
11
A
Dataset Appendix: Dataset Details and Processing
A.1
Training Sources
A.2
CoSER Split for Specialization and Domain Preservation
We first rank CoSER conversations by the number of dialogue turns. The 10K conversations with the largest number of turns are reserved as unseen role-play domain data for MOPD. For each reserved conversation, we use the first n − 1 turns as the prompt/context and leave the final continuation to be generated on-policy by the current student. These examples are never used in role-play SFT. The remaining 300K CoSER conversations are used to fine-tune Qwen3-4B-Instruct-2507 and produce the role-play teacher. This split is designed for the on-policy MOPD setting. Since MOPD queries teachers on studentgenerated continuations, held-out prompts provide fresher and more informative domain gradients than replaying prompts that the SFT model has already fit. Selecting long-turn conversations further stresses the preservation problem because the model must maintain persona, dialogue state, and storyline consistency over rich contexts rather than short isolated turns.
For the role-play experiments, we start from Qwen3-4B-Instruct-2507 (Qwen Team, 2025). We use two teachers during MOPD. The general teacher Tg is the original Qwen3-4B-Instruct2507 model, and the role-play teacher Td is obtained by supervised fine-tuning the same base model on CoSER. CoSER (Wang et al., 2025)1 is a role-play dataset built from 771 novels and provides multi-turn, multi-character conversations with character profiles, dialogue context, plot summaries, and character experiences. The official dataset card reports a formatted SFT file, sft_conversations_sharegpt.json, together with a held-out test file and full book-level JSON files. For the medical reasoning QA experiments, we instantiate the same recovery-under-preservation setup with a publicly released medical post-trained model. The general teacher is Qwen/Qwen38B (Qwen Team, 2025), and the domainspecialized initialization and domain teacher are both Intelligent-Internet/II-Medical-8B (Intelligent Internet, 2025)2 . The model card lists II-Medical8B under the Apache-2.0 license and describes it as a medical reasoning model obtained by applying medical supervised fine-tuning and DAPO RL optimization (Yu et al., 2025) on top of Qwen/Qwen38B. We therefore treat it as an already specialized open-source model rather than a model trained from scratch in this work.
A.3
Nemotron Sampling for General Recovery
In principle, prior OPD studies suggest that when the recovery data are sufficiently abundant and well aligned with the teacher’s capability distribution, on-policy distillation can recover broad capabilities and may even surpass the teacher on some evaluations through student-side exploration and selection (Agarwal et al., 2024; Gu et al., 2024; Li et al., 2026; Fu et al., 2026). In our setting, however, the pretraining and post-training data of the open-source general teacher are unknown. A practitioner can only collect a high-quality proxy for the general distribution, such as Nemotron, and the overlap between this proxy and the teacher’s original training distribution is not observable. We therefore choose a sample-efficient recovery setting: from the 25.66M-example Nemotron corpus, we select only 10K prompts while preserving the official category proportions. This makes the recovery task intentionally budget-limited. Strong performance under this limited and imperfectly matched general corpus supports the central claim of CaMOPD: reducing recovery-preservation signal counteraction matters when full teacher-data coverage is unavailable.
For general-domain recovery, we use NemotronPost-Training-Dataset-v1 (Bercovich et al., 2025)3 , a large SFT corpus released with five splits: chat, code, math, stem, and tool calling. The official card reports 25,659,642 examples in total, with 746,622 chat examples, 1,896,395 code examples, 2,044,407 math examples, 20,662,167 stem examples, and 310,051 tool-calling examples. We sample 10K general prompts proportionally from these categories.
1
https://huggingface.co/datasets/Neph0s/CoSER https://huggingface.co/Intelligent-Internet/ II-Medical-8B 3 https://huggingface.co/datasets/nvidia/ Nemotron-Post-Training-Dataset-v1 2
12
Nemotron general prompt example. category: math/code/stem/chat/tool_calling messages: [user] a task prompt, such as a math problem, programming challenge, STEM question, chat request, or tool-use instruction. [assistant] a synthetic reference response generated by public/open models. metadata: source information and, for tool-calling data, tool schema or call metadata.
CoSER Role-Play conversation example. conversations: [system] role instruction and character/background profile, e.g., “Your role is to be [character]” plus scenario context. [user/character] previous dialogue turns and actions. [assistant/character] the next in-character response. auxiliary fields: character profiles, scenario, topic, plot summary, speaking characters, and book-level metadata in the full CoSER files.
Figure 5: Nemotron general prompt structure used for the general recovery branch.
A.4
Figure 6: CoSER Role-Play conversation structure used for Role-Play specialization and domain preservation.
Medical Reasoning QA Model and Domain-Preservation Prompts
Medical domain-preservation prompt template. raw fields: question, optional options or opa/opb/opc/opd prompt: [system] You are a careful medical reasoning assistant. Provide educational medical information, mention uncertainty when relevant, and recommend professional medical care for diagnosis or treatment decisions. [user] {question} A. {opa} B. {opb} C. {opc} D. {opd} Please reason step by step and give the final answer choice at the end. open-ended variant: for sources with only question or prompt/text, the user message is {question} followed by “Please reason step by step and answer clearly. Do not present the answer as a substitute for professional medical advice.” data_type: medical training use: gold answers and explanations are not placed in the prompt field; teacher feedback is computed on student-generated continuations.
The medical reasoning QA instantiation uses II-Medical-8B as both the starting student and the preservation teacher. This mirrors the roleplay setup at the level of the recovery objective: CaMOPD starts from a domain-specialized model, recovers general capability from the corresponding Qwen3-8B teacher, and periodically reviews medical reasoning QA prompts with the domain teacher to reduce domain drift. The medical reasoning QA domain-preservation stream is prompt-only. It does not use gold answers during MOPD/CaMOPD training: the current student first generates on-policy continuations, and the domain teacher provides token-level log-probability feedback on those student trajectories. We construct the preservation prompts from medical question-answering and reasoning sources related to the II-Medical data ecosystem: 4K prompts from MedMCQA (Pal et al., 2022)4 , 3K from MedReason (Wu et al., 2025)5 , 2K English prompts from Medical-R1-Distill-Data (Chen et al., 2024)6 , and 1K from m23k-tokenized (Huang et al., 2025b)7 . After selection and formatting, the current preservation set contains 10K prompts. A.5
Figure 7: Medical domain-preservation prompt template constructed from source dataset fields.
A.6
Resulting Data Roles
Table 4 summarizes how each data source is used in the training pipeline. We separate specialization data, domain-preservation prompts, and proxy general-recovery prompts to avoid mixing their roles in the MOPD/CaMOPD setup.
Data Examples
Figures 5–7 show the field structure and training usage for the general, role-play, and medical reasoning QA streams.
A.7
Training Protocol
All MOPD-based methods generate one response per prompt with the current student and compute token-level teacher feedback on student trajectories. We train for 120 steps with a prompt batch size of 512 and a learning rate of 2 × 10−6 . Prompts are truncated at 4096 tokens and rollouts are capped at 32768 tokens. Unless otherwise specified, CaMOPD uses a 3-1 recovery-to-review
4 https://huggingface.co/datasets/ openlifescienceai/medmcqa 5 https://huggingface.co/datasets/UCSC-VLAA/ MedReason 6 https://huggingface.co/datasets/ FreedomIntelligence/Medical-R1-Distill-Data 7 https://huggingface.co/datasets/UCSC-VLAA/ m23k-tokenized
13
Table 4: Dataset roles in CaMOPD. Domain-preservation prompts provide preservation gradients for the active vertical domain, while the general branch uses a compact proportional sample from Nemotron to recover general capability under a limited MOPD budget. Data
Size
Used for
Notes
CoSER remainder CoSER long-turn heldout Medical prompt mixture Nemotron proportional sample
300K 10K 10K 10K
Role-Play SFT teacher Role-Play domain preservation Medical domain preservation General recovery
Trains Td Unseen by Role-Play SFT Prompt-only; no labels used Proxy general prompts
schedule, a gap-score mass target of ρ = 0.8 for both branches, and a general recovery scale of rg = 2, with the domain scale fixed to 1. We run all training on a 4-node cluster with 8 NVIDIA H100 GPUs per node. Each run uses 32 H100 GPUs for approximately 7 hours, corresponding to about 224 total GPU hours.
B
Qwen2.5-14B-Instruct is used as an answer extractor and equivalence verifier, not as an open-ended quality judge. Concretely, the verifier prompt wraps each example as <prompt>, <response>, and <ground_truth_answer> fields and asks for <extracted_answer> and <decision> tags. A prediction is counted as correct if and only if the parsed decision is yes. For multi-sample settings, the generation scripts sample n candidate responses and the extraction scorer aggregates the binary decisions with k = 1, i.e., a pass@1/accuracy estimate rather than an oracle pass@n score.
Evaluation Protocols
This appendix describes how the general-capability, role-play, and medical reasoning QA benchmarks are scored. For general capability, we use GPQA-Diamond (Rein et al., 2024), HMMT25 from MathArena (Balunović et al., 2025), ZebraLogic (Lin et al., 2025), LiveCodeBench v5 and LiveCodeBench v6 (Jain et al., 2024), IFEval (Zhou et al., 2023), the Hard Prompt and Creative Writing subsets of Arena-Hard v2 (Li et al., 2025), and LiveBench 1125 (White et al., 2025). Table 5 summarizes the generation settings, aggregation rules, and judges or checkers used for these general benchmarks. We distinguish three evaluation modes. First, objective benchmarks with short canonical answers are scored by extracting the model’s final answer and comparing it with the reference answer. Second, code and instructionfollowing benchmarks are evaluated by their official execution or rule-based checkers. Third, preference-based generation benchmarks use an LLM judge because the target quality criteria are preference-based rather than exact-answer based.
Rule-based and Execution-based Benchmarks. ZebraLogic is evaluated with the ZebraLogic grid evaluator, which parses the predicted grid and compares it against the puzzle solution. LiveCodeBench v5 and v6 are evaluated with the benchmark execution pipeline: generated programs are extracted and run against the associated tests, and pass rates are reported following the benchmark protocol. IF-Eval is scored with its official instruction-following checker, which applies deterministic constraints to the generated responses. LiveBench 1125 is evaluated with the official LiveBench ground-truth judging scripts, which route each task family to the appropriate exact, rule-based, symbolic, or execution-based scorer. Preference-based Generation Benchmarks. Arena-Hard v2 requires model-based evaluation because its outputs are judged by pairwise preference. We use GPT-4.1 as the judge and follow the Arena-Hard automatic judging pipeline. We report the Hard Prompt and Creative Writing subsets separately as HP and CW.
Answer Verification for Objective Benchmarks. For GPQA-Diamond and HMMT25, we use an LLM-based answer verifier implemented with Qwen2.5-14B-Instruct. The verifier is given the original prompt, the model response, and the ground-truth answer. It is instructed to identify the final answer in the response and output both an extracted answer and a binary decision indicating whether it matches the reference. The downstream scorer only reads this binary decision. Thus,
B.1
Role-Play Evaluation Protocols
For role-play preservation, we follow the official Given-Circumstance Acting (GCA) evaluation protocol from CoSER (Wang et al., 2025). The protocol simulates role-playing dialogues from the 14
Table 5: Evaluation settings and scoring protocols for the general benchmarks. T denotes the evaluated model’s generation temperature, n denotes the number of generated samples, and k denotes the pass@k aggregation parameter. LLM-based verification is used only for objective answer extraction and equivalence checking; preferencebased generation benchmarks use GPT-4.1 as a pairwise judge. Benchmark
Generation setting
Scoring/aggregation
Judge or checker
GPQA-Diamond
0-shot; T=0.7; n = 4 top_p=0.95; top_k=20 rep.=1.05; max 32K tokens T=0.6; n = 16 top_p=0.95; top_k=20 rep.=1.05; max 32K tokens T=0; n = 1 max 32K tokens 0-shot; T=0; n = 1 max 32K tokens 0-shot; T=0; n = 1 max 32K tokens T=0; n = 1 max 16K tokens T=0; n = 1 max 32K tokens T=0; n = 1 max 32K tokens T=0; n = 1 max 32K tokens
pass@1/accuracy estimate with k = 1
Qwen2.5-14B-Instruct verifier; T=0.2
pass@1/accuracy estimate with k = 1
Qwen2.5-14B-Instruct verifier; T=0.2
Puzzle/grid correctness
ZebraLogic rule-based grid evaluator Official execution-based evaluator Official execution-based evaluator Official deterministic rule checker GPT-4.1 judge via Arena-Hard pipeline; T=0.0 GPT-4.1 judge via Arena-Hard pipeline; T=0.0 Official exact, rule-based, symbolic, and execution scorers
HMMT25
ZebraLogic LiveCodeBench v5 LiveCodeBench v6 IF-Eval Arena-Hard v2 (HP) Arena-Hard v2 (CW) LiveBench 1125
Code pass rate Code pass rate Instruction-following score Pairwise preference score Pairwise preference score Official mixed benchmark score
provided character profiles, scenario, and reference dialogue, and then evaluates the generated dialogue with the same judge and scoring logic as CoSER. In our experiments, only the evaluated actor_model changes across systems; the environment model, next-speaker predictor, and LLM judge are fixed to Qwen2.5-72B-Instruct. We report the four standard GCA dimensions—Storyline Consistency, Anthropomorphism, Character Fidelity, and Storyline Quality—and use their arithmetic mean as the roleplay average. B.2
the predicted option letter from the final-answer pattern and compute exact-match accuracy against the reference label. MedQA-USMLE. MedQA-USMLE (Jin et al., 2021) is a four-option medical exam benchmark. We use the GBaker/MedQA-USMLE-4-options test split, which contains 1,273 examples in the local cache. The model is prompted as a multiple-choice question-answering system and is required to end with Final answer: <letter>. We extract the predicted option and compute exact-match accuracy.
Medical Reasoning QA Evaluation Protocols
C
For the medical reasoning QA instantiation, we evaluate domain preservation with two objective benchmarks: MedXpertQA Text (Zuo et al., 2025) and MedQA-USMLE (Jin et al., 2021).
Hyperparameter Analysis
We further study the training schedule and two hyperparameters in the general recovery branch. The recovery-to-review schedule controls how often general recovery is updated relative to domain review. The recovery scaling ratio rg multiplies the gap signal of selected general-branch samples in Eq. (9). The gap-score mass target ρg controls how much total recovery gap-score mass must be covered by the selected samples in Eq. (8). The maintext configuration uses a 3-1 recovery-to-review schedule, rg = 2, and ρg = 0.8. We compare it with Vanilla MOPD and schedule, scaling, and mass-target variants, keeping the teachers, data,
MedXpertQA Text. MedXpertQA Text is a textonly expert-level medical multiple-choice benchmark from the Text subset of MedXpertQA (Zuo et al., 2025)8 . We evaluate the full test split used by the local runner, which contains 2,450 examples. The model is prompted to choose a single option and end with an explicit final answer. We extract 8
https://huggingface.co/datasets/TsinghuaC3I/ MedXpertQA
15
Table 6: Evaluation protocol for Medical Reasoning QA benchmarks. All scores are objective exact-match metrics. Benchmark
Evaluation set
Scoring/aggregation
MedXpertQA Text
Full text test split; 2,450 examples
MedQA-USMLE
GBaker/MedQA-USMLE-4-options split; 1,273 examples
Multiple-choice exact match from parsed finalanswer letter Four-option exact match from parsed finalanswer letter
and other optimization settings unchanged. Effect of Recovery-to-Preservation Schedule. Under rg = 2 and ρg = 0.8, the 3-1 schedule gives the best recovery-preservation trade-off. The 5-1 schedule achieves similar general recovery, but its role-play average is lower than the 3-1 setting, indicating that domain capability starts to decline when recovery updates become too frequent. The 1-1 schedule is more conservative for recovery and also yields a lower domain average than 3-1. Effect of Recovery Scaling and Gap-score Mass Target. For a fixed 3-1 schedule, rg = 2 and ρg = 0.8 gives the strongest aggregate general recovery. The larger gap-score mass target covers a broader set of informative samples, while the recovery scale amplifies the gradient signal of the selected high-demand samples. Other combinations can improve individual metrics, but their gains are less consistent. Trade-off Across General and Domain Performance. Based on these results, we use the 3-1 schedule with rg = 2 and ρg = 0.8 in the main experiments. This configuration prioritizes robust general recovery while maintaining competitive role-play preservation, rather than optimizing a single benchmark family or the domain average alone.
16
test
Table 7: General and Role-Play capability under different recovery schedules and general-recovery hyperparameters. Higher is better for all metrics.
Metric
5-1 3-1 3-1 3-1 3-1 3-1 3-1 1-1 rg =2 rg =1 rg =1 rg =2 rg =2 rg =3 rg =3 Vanilla rg =2 MOPD ρg =0.8 ρg =0.8 ρg =0.6 ρg =0.8 ρg =0.6 ρg =0.8 ρg =0.6 ρg =0.8
General capability GPQA-Diamond ZebraLogic HMMT25 LiveCodeBench v5 LiveCodeBench v6 IF-Eval Arena-Hard v2 (HP) Arena-Hard v2 (CW) LiveBench 1125
61.99 71.30 29.17 31.54 22.29 79.67 25.50 13.70 58.90
61.11 74.10 31.46 37.99 23.43 79.48 30.60 27.60 63.00
60.23 76.70 29.79 37.28 23.43 79.85 35.20 31.10 63.20
61.87 74.40 31.46 34.41 24.00 78.74 27.60 22.10 59.70
61.62 74.20 30.83 35.13 20.57 78.93 26.60 25.50 61.40
61.62 74.50 31.25 34.77 24.00 80.77 32.60 27.30 62.70
62.63 76.30 31.04 37.28 24.00 80.41 33.20 33.30 63.50
60.98 74.50 30.42 34.41 22.29 79.67 30.00 28.00 63.00
62.50 75.50 31.87 35.48 22.86 80.96 29.30 30.20 62.30
45.29 36.58 41.76 52.10 43.93
45.30 35.98 42.62 51.55 43.86
46.79 34.24 42.18 54.58 44.45
44.46 35.50 42.85 53.22 44.01
46.14 34.34 44.66 53.08 44.56
47.81 33.96 44.65 53.47 45.00
44.58 35.65 44.69 53.88 44.70
47.47 34.05 42.39 54.25 44.54
Role-Play domain capability Storyline Consistency Anthropomorphism Character Fidelity Storyline Quality Average
43.57 34.91 41.45 51.50 42.86
17