Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback Woohyeon Byeon Jiwon Jeon Jeonghye Kim Youngchul Sung* KAIST {woohyeon.byeon, jiwon.jeon, jeonghye.kim, ycsung}@kaist.ac.kr
arXiv:2606.14368v1 [cs.LG] 12 Jun 2026
Abstract We study multi-domain LLM training in which two models, each stronger in a different domain, co-evolve by tutoring each other through on-policy feedback. Unlike one-way distillation or single-model fine-tuning, our goal is mutual Pareto improvement: each model improves across domains without losing its original strength. To this end, we propose OnPolicy Co-Distillation (OPCoD), where each student’s self-distillation is conditioned on its own correct rollout and feedback from its peer. To make feedback exchange effective, OPCoD uses cognizance-based gating to decide when to give feedback and feedback anchoring to ground feedback in the problem. On Science Q&A tasks, OPCoD consistently outperforms baselines and achieves Pareto improvement across all evaluated domain pairs and students.
1
Figure 1: Conceptual illustration of OPCoD
This leads us to ask: how can we leverage multidomain training without falling into negative transfer? To answer this, consider how humans handle a similar challenge, as illustrated in Figure 1: a physics major and a chemistry major face an exam covering physics and chemistry, including physical chemistry at their intersection. Studying alone leaves each student’s blind spots unaddressed, while tutoring each other allows them to exchange complementary knowledge and catch errors they would miss on their own. This benefit can even extend across fields: a chemistry major’s chemical intuition can sometimes help a physics major solve a physics problem they could not crack alone, broadening the reasoning each can draw on. For students to tutor each other effectively, both must exchange feedback throughout the process. Such feedback-driven learning has gained considerable attention in recent LLM research, with methods such as on-policy distillation (Agarwal et al., 2024) and self-distillation (Hübotter et al., 2026; Zhao et al., 2026) training student models on signals from a teacher’s outputs. Our setting, however, differs from these methods in two key respects: we target multi-domain capability rather than single-domain improvement, and we require bidirectional natural language feedback between
Introduction
Large language models (LLMs) are commonly fine-tuned on a single domain, such as science, medicine, or law, to acquire specialized knowledge (Taylor et al., 2022; Singhal et al., 2025; Hu et al., 2025). While such single-domain fine-tuning yields strong in-domain expertise, the resulting specialist often struggles beyond its domain. To broaden this coverage, recent work leverages the capacity of LLMs to absorb diverse knowledge from many domains within a single model (Brown et al., 2020; Bommasani et al., 2021), motivating multi-domain training, where datasets from different fields are combined during fine-tuning (Sanh et al., 2021; Wei et al., 2021; Chung et al., 2024). Despite these advantages, mixing data from different domains often induces negative transfer. Gradients from one domain can interfere with those from another, degrading performance, sometimes even on the model’s original specialty (Cai et al., 2026; Yang et al., 2026c; Ye et al., 2026a). * Corresponding author.
1
two models rather than one-way teacher-to-student supervision. Motivated by these distinctions, we propose On-Policy Co-Distillation (OPCoD), an on-policy co-distillation framework that enables two student models to mutually tutor each other across both domains. OPCoD realizes this through three key components: (1) co-distillation, where the two selfdistillation processes are coupled through bidirectional feedback, co-evolving throughout training; (2) cognizance-based gating, which gates feedback by the tutor’s competence across domains, mitigating negative transfer; and (3) feedback anchoring, which grounds feedback in the specific problem to elicit informative, non-hallucinated responses. Together, these components enable mutual Paretoimprovement: both models gain capability across both domains without sacrificing performance on their original specialties. We evaluate OPCoD on SciKnowEval (Feng et al., 2024) across diverse multi-domain combinations (e.g., physics–chemistry, chemistry–materials), where it consistently outperforms strong baselines and achieves Pareto-improvement on both domains. We also analyze the contribution of each design component: cognizance-based gating in preventing unreliable feedback from corrupting correct rollouts, and feedback anchoring in suppressing hallucinated feedback. Finally, we present examples of the feedback exchanged during training, illustrating how the two models tutor each other through cross-domain reasoning signals. In summary, our main contributions are: • We open a new direction for distillation: bidirectional between two models for multi-domain learning, rather than one-way for single-domain improvement. • We propose three components (co-distillation, cognizance-based gating, and feedback anchoring) that enable mutual tutoring, allowing each model to improve across targeted domains without sacrificing its stronger capabilities. • We empirically demonstrate that OPCoD consistently achieves strong performance across diverse cross-domain combinations on SciKnowEval, outperforming baseline methods.
2
roles: the student learns to match the token-level distribution of a self-teacher that has access to additional privileged information. Let x be a prompt drawn from a dataset D and y = (y1 , . . . , yT ) an on-policy rollout from policy πθ , with y<t = (y1 , . . . , yt−1 ). We denote by c the privileged information available only to the self-teacher. We denote the student as πS (· | x, y<t ) := πθ (· | x, y<t ) and the self-teacher as πT (· | x, c, y<t ) := sg(πθ (· | x, c, y<t )), where sg(·) denotes the stop-gradient. The privileged information c can take various forms, such as environment feedback, a successful rollout, or a groundtruth solution (Zhao et al., 2026; Hübotter et al., 2026; Yang et al., 2026a). The SDPO (Hübotter et al., 2026) loss is LSDPO (πS ) = Ex∼D, y∼πS (·|x)
h
|y|
1 X D(πS (· | x, y<t ) ∥ πT (· | x, c, y<t )) , |y|
i
t=1
where D(· ∥ ·) is a divergence, such as KL or Jensen–Shannon. When rich environment feedback is unavailable, SDPO uses the model’s own verified correct rollout as c, leaving c empty when no such rollout exists. Pareto Criteria. For two evaluated domains A and B, we say π Pareto-dominates π ′ if π achieves no lower score than π ′ on both domains and a higher score on at least one domain (Hayes et al., 2022). Given training from an initial policy π0 to a learned policy π, we say the learned policy achieves Pareto improvement if π Pareto-dominates π0 . For two students, mutual Pareto improvement means that each learned student policy achieves Pareto improvement over its initial policy, respectively. Our goal is to achieve mutual Pareto improvement through bidirectional peer feedback, without relying on any external teacher. Formal definitions are provided in Appendix A.
3
OPCoD: On-Policy Co-Distillation
We introduce On-Policy Co-Distillation (OPCoD), an on-policy co-distillation framework that drives several student models toward mutual Paretoimprovement by exchanging natural-language feedback during training. Each model performs onpolicy self-distillation, with its self-teacher conditioned on both its own correct rollout and feedback from its peer. As training proceeds, each model’s updates also affect its peer through the feedback
Preliminaries
On-Policy Self-Distillation. On-policy selfdistillation (OPSD) is a training paradigm in which a single policy plays both the teacher and student 2
Bidirectional Tutoring Student 1
Student 2
Optimization
Rollout
Feedback A B
Feedback Student 1
Selected correct rollout
A B
Prompt x
Student 1 (Tutee)
y1
...
yk
...
yn yn
Policy Update round r+1
Student 2 (Tutor) Feedback
A B
Training
A B
Feedback
Anchoring
Cognizance
Gate
Self-Teacher 1: Prompt x + y1 + fk + Feedback
yk <t
On-Policy
Self-Distillation
f1 ...
round r
fn Student 1 (Tutee): Prompt x +
Fixed
yk <t
Figure 2: OPCoD overview. (Left) Mutual tutoring scheme: students tutor each other through bidirectional feedback, driving policy updates. (Right) Within-round mechanism: for a prompt x, the tutee samples on-policy rollouts {y 1 , . . . , y n }. The tutor generates feedback {f 1 , . . . , f n } only if it passes the cognizance gate. The tutee is updated by matching its policy to a self-teacher conditioned on x, y 1 (any correct rollout), and f k . Each round applies this procedure in both directions by swapping tutee/tutor roles; for visual clarity, only Student 1 as tutee is shown.
it provides. The two self-distillation processes are thus coupled rather than independent, yielding a co-evolving training dynamic. OPCoD controls the tutor’s feedback process along two complementary axes: when to give feedback (Section 3.2), via cognizance-based gating, and how to give feedback (Section 3.3), via feedback-anchoring. Figure 2 illustrates the overall pipeline; Section 3.1 formalizes the co-distillation objective before we turn to each axis. Pseudo-code is provided in Appendix B. 3.1
responses to each training prompt, and the tutor generates natural-language feedback on each response. The tutee then updates its student policy πSi to match a self-teacher πTi conditioned on two anchors that instantiate the privileged information as c = (s, f ), where s is a correct response from the tutee’s own rollouts (empty when none exists) and f ∼ π −i (· | x, y) is the tutor’s feedback on the response. For each i ∈ {1, 2}, when π i serves as a tutee, it minimizes h
i Li (πS ; π −i ) = Ex∼D, y∼πi (·|x)
Co-Distillation Framework
1 X i D πS (· | x, y<t ) πTi (· | x, s, f, y<t ) |y|
Problem Statement. We consider two domains A and B with their respective training sets DA and DB . We are given two student models, π 1 and π 2 . Our goal is mutual Pareto-improvement: both models should improve across both domains, without access to an external teacher.
(1)
S
|y|
i
,
t=1
where D is a divergence (e.g., Jensen–Shannon). 3.2
When to Give Feedback: Cognizance-Based Gating
Not all tutor feedback helps the tutee. A tutor that is insufficiently reliable in the problem’s relevant area may mislead the self-teacher and degrade the distillation signal. Figure 3 illustrates such a case, where inappropriate tutor feedback corrupts the response of the tutee’s self-teacher and can mislead the tutee’s learning. We therefore let the tutor give feedback only when it is sufficiently cognizant of the relevant domains, as measured by its relative performance gap.
Multi-Round Training. Training proceeds over R rounds. Within each round, the two models alternate as tutee and tutor: each model is updated for K on-policy self-distillation steps as a tutee, while the other model provides feedback as the tutor. Across rounds, each model’s updated state shapes the feedback it provides next, realizing the coupled training dynamics. Within One Round. Each round consists of bidirectional updates. First, π 1 acts as the tutee and is updated for K on-policy self-distillation steps using feedback from π 2 as the tutor. Then the roles are swapped: π 2 becomes the tutee and is updated for K steps using feedback from π 1 . We describe the loss for one such directional update below; the same procedure is applied symmetrically after swapping the roles. Let π i be the current tutee and π −i the current tutor (i ∈ {1, 2}). The tutee samples on-policy
Cognizance-Based Gating. At the start of each round, both π 1 and π 2 are evaluated on held-out validation sets for each domain. Let sid denote model π i ’s validation score on domain d, and let s∗d = max{s1d , s2d } denote the highest score on domain d. To evaluate the suitability of each π i as a tutor, we define the cognizance gap of π i as the total relative shortfall from s∗d : for i = 1, 2, ∆i =
X d∈{A,B}
3
s∗d − sid . s∗d
(2)
Problem summary
misleading
correct chemistry
For a large aromatic heterocyclic molecule with a sulfone and an amide-like nitrogen, predict the octanol/water distribution coefficient log D7.4. Choices: A: 2.56 B: 0.05 C: 2.12 D: 3.96 (ground truth: A)
Tutee's original answer (correct) The molecule is large, aromatic, and mostly neutral, so its lipophilicity should be high. [ ]
At pH 7.4, the sulfoneandamide-like nitrogen are not strongly basic, so the molecule remains largely neutral. [ ]
D: 3.96 is somewhat high for a molecule that still contains a sulfoneand an amide-like nitrogen, whichadd polarity and reduce the overall logD.
Incognizant tutor's feedback (misleading)
Tutee's answer with feedback (misled by feedback)
[ ]
The molecule is a very large, aromatic, heterocyclic scaffold withmultiple heteroatoms. Such a structure is expected to have a much higher log D7.4 than A: 2.56.
The molecule is a large aromatic heterocyclic scaffoldwith a sulfone and an amide-like nitrogen.This kind of structure is very lipophilic, especially when the molecule remainsneutral at pH 7.4.
Therefore, the best fit is D: 3.96 .
A: 2.56 is too low for a molecule this large and aromatic.
The peer s reasoning is flawed.
[ ]
[ ] [ ]
Its log D7.4 should be very high, around 3 to 4 . So the correct answer is D: 3.96 .
[ ]
Therefore the best fit is A: 2.56 .
[ ]
Correct chemistry judgment
Why this feedback is misleading
How feedback misled reasoning
The tutee balances hydrophobic scaffold size againstpolar functional groups, ruling out the overly high D: 3.96.
The tutor overweights the large aromatic scaffold and downplays the polar-group caveat, pushing the answer toward D: 3.96.
The 2nd answer adopts the feedback s high-logD frameand flips the original chemistry judgment from A: 2.56 to D: 3.96.
Figure 3: Necessity of cognizance-based gating: Incognizant tutor’s feedback can break an initially correct answer. The tutee first rules out the overly high log D7.4 value by considering polar functional groups, but the incognizant tutor’s feedback overemphasizes the large aromatic scaffold. The second answer then adopts this high-log D frame and flips from the correct choice A to the incorrect choice D.
Intuitively, each term measures the π i ’s relative gap from the best model on a domain, so a smaller ∆i means π i is broadly closer to the best across domains. We consider π i cognizant if ∆i ≤ τ and incognizant otherwise, where τ is a predefined threshold. At the start of each round, if π i is cognizant, it gives feedback as a tutor for this round, providing the privileged information c = (s, f ) for the tutee’s OPCoD update (Eq. (1)); if π i is incognizant, it gives no feedback for this round, and the tutee falls back to self-distillation (c = s). This criterion is relative rather than absolute: it asks whether the tutor is likely to add value beyond what the tutee can already obtain on its own. For example, if two models have scores 1 (sA , s1B ) = (100, 70) and (s2A , s2B ) = (70, 100) with τ = 0.2, both have ∆ = 0.3 and give no feedback as tutors for each other: although a score of 70 indicates a reasonably capable tutor in absolute terms, the tutee already has stronger self-generated rollouts on its own high-scoring domain.
Feedback Anchoring. Feedback anchoring uses a two-step prompt. First, the tutor must identify a single technical concept from the question and output it in <concept>...</concept> tags. Second, the tutor writes a short critique of the tutee’s reasoning without revealing the final answer. The extracted concept also serves as a verification signal: if it does not appear explicitly in the question text, we discard the feedback as ungrounded. Verified feedback is then sanitized to remove direct answer reveals, and the concept tag is stripped before the feedback is given to the tutee. The full prompt and filtering details are provided in Appendix C.
3.3
4
the question and the tutee’s response, then provide constructive feedback. In practice, this can produce feedback disconnected from the actual problem, addressing irrelevant content rather than the tutee’s specific reasoning. We refer to this phenomenon as feedback hallucination. Figure 4 illustrates this failure mode and how anchoring mitigates it.
How to Give Feedback: Feedback-Anchoring
Experiments
We organize our experiments around four aspects of OPCoD. First, we evaluate whether co-distillation achieves Pareto improvement across paired do-
A natural starting point for feedback generation is to prompt the tutor with a generic instruction: read 4
Problem
problem key off-topic (hallucinated) on-topic (anchored)
Feedback from naive prompt
Feedback from feedback-anchoring prompt
The student correctly identified that the most favorablematerial for stabilityis the one with the best combination of density and first ionization potential.
The material specialist s reasoning focuses on the physical meaning of the damped frequency compared to the natural frequency. The specialist correctly identifies the key quantity as thefrequency ratio
A damped harmonic oscillator loses amplitude to 1/e after four cycles ; find the damped-to-natural frequency ratio. [ ]
[ ]
[ ]
A also has ahigh density. Therefore, A is the best overall stability candidate.
[ ]
[ ]
Why this is hallucination
Why this stays anchored
The problem asks about adamped oscillator, but the feedback discussesmaterial stability, density, andfirst ionization potential.
The feedback stays anchored to the problem by identifyingdamped frequency, natural frequency, and the frequency ratio.
Figure 4: Necessity of feedback anchoring: Example of feedback hallucination and feedback anchoring. The left panel shows problem-irrelevant feedback generated by the naive prompt, while the right panel shows feedback generated by our anchoring prompt, which stays grounded in the problem.
mains. Second, we examine whether cognizancebased gating mitigates the risk that feedback corrupts previously correct rollouts. Third, we analyze whether feedback anchoring preserves enough tutor feedback while filtering problem-irrelevant responses. Finally, we examine how peer feedback can supply complementary reasoning insights that help a student correct its own mistake. 4.1
Each student is trained for 100 update steps. We use n = 8 rollouts per prompt and the cognizance threshold τ = 0.2. We report avg@16 on the final checkpoint; remaining hyperparameters, including the divergence and learning rate, are provided in Appendix D.3. 4.2
Experimental Setting
Results: Multi-Domain Science Q&A
Table 1 reports avg@16 scores across the three domain pairs. OPCoD achieves mutual Pareto improvement in every pair: both models in each pair improve on both domains. Moreover, OPCoD Pareto-dominates the GRPO and SDPO baselines, achieving the highest score in every column.
We use the Science Q&A subset from the L3 split of SciKnowEval (Feng et al., 2024), which spans four scientific domains: chemistry, physics, materials science, and biology. Following Hübotter et al. (2026), we partition each domain’s data into training and test splits. In our experiments, we use three students based on Qwen3-8B (Yang et al., 2025) with different domain strengths, whose construction details are deferred to Appendix D. To cover a range of cotraining scenarios, we pair two students from distinct domains and jointly train them on the union of their respective training splits. Because the biology split contains substantially fewer examples than the other three, we focus on three pairs that exclude biology: chemistry–materials, materials–physics, and physics–chemistry. As baselines, we train each student individually on the same union dataset via GRPO (Shao et al., 2024) and SDPO (Hübotter et al., 2026), isolating the effect of peer feedback by contrasting solo and joint training under matched data.
Mutual Pareto improvement, our main goal, is not achieved by all baselines. In SDPO runs, the per-pair average increases because gains on the non-native domain offset losses on the native domain. However, SDPO degrades native-domain performance in three of the six (agent, pair) configurations: the Mat-stronger agent’s Mat score drops from 65.1 to 62.2 in Mat–Phys and from 65.1 to 60.0 in Chem–Mat, and the Phys-stronger agent’s Phys score drops from 51.6 to 51.1 in Phys–Chem. This is exactly the negative-transfer pattern: independent training on the union of domains can improve a model’s non-native domain at the cost of its original specialty. OPCoD’s gating prevents a tutor from giving feedback when it is not sufficiently reliable, eliminating this failure mode while still delivering strong non-native-domain gains. 5
Mat-Phys Mat.
Phys.
Chem-Mat Avg.
Chem.
Mat.
Phys-Chem Avg.
Phys.
Chem.
Avg.
Student 1 + GRPO + SDPO + OPCoD
65.1 67.8 62.2 70.5
Mat-stronger 48.1 56.6 53.8 60.8 52.3 57.2 55.3 62.9
56.3 62.0 70.9 71.3
Chem-stronger 56.1 56.2 65.6 63.8 65.8 68.4 69.8 70.6
51.6 52.6 51.1 54.1
Phys-stronger 37.3 44.5 55.4 54.0 56.7 53.9 58.9 56.5
Student 2 + GRPO + SDPO + OPCoD
56.0 66.0 58.8 66.1
Phys-stronger 51.6 53.8 53.9 60.0 51.9 55.4 54.3 60.2
37.6 57.3 55.4 57.5
Mat-stronger 65.1 51.4 65.2 61.2 60.0 57.7 66.4 62.0
48.6 54.3 56.4 58.8
Chem-stronger 56.3 52.5 62.8 58.6 69.9 63.2 70.2 64.5
Table 1: Avg@16 results across three domain pairs. OPCoD is highlighted. Each block label (e.g., Mat-stronger) indicates the student with the highest score in that domain among the three initial students.
4.3
Cognizance-Based Gating Mitigates the Risk of Feedback Corruption
remains substantially riskier than cognizant feedback. We evaluate this domain-selective alternative as an ablation in Figure 7, further supporting our gating rule in Section 3.2.
Tutor feedback directly affects training through the self-teacher’s conditioning (Eq. (1)); misleading feedback can therefore corrupt the student’s training signal. To validate that cognizance-based gating mitigates this risk, we define the break-rate for each (tutee, tutor) pair as the fraction of the tutee’s previously correct rollouts that become incorrect after the tutor’s feedback is added to the self-teacher’s conditioning. Across the three domain pairs, incognizant tutors (which gating excludes) break correct rollouts at 2.4× the rate of cognizant tutors (which gating admits), confirming the effectiveness of gating’s selection rule. Detailed settings and aggregate break-rates are provided in Appendix E. Gating’s benefit is robust across problem difficulty. As shown in Table 2, cognizant tutors have lower break-rates than incognizant tutors in every difficulty range. The gap appears even on easier problems (1.13% vs. 4.75% on the easiest range) and remains substantial on harder ones (5.42% vs. 13.73% on the hard range and 13.79% vs. 21.88% on the hardest). Gating’s safety advantage is therefore consistent across difficulty levels, and especially valuable on harder problems where correct rollouts are scarce. A natural alternative to our gating rule (which silences all incognizant feedback for the round) is to admit feedback from an incognizant tutor only for problems in its stronger domain. However, even within the tutor’s stronger domain, incognizant tutors still break correct rollouts at 1.4× the rate of cognizant tutors. This indicates that domain restriction alone is insufficient: incognizant feedback
Difficulty
Cognizant (%)
Incognizant (%)
Easiest Easy Hard Hardest
1.13 2.35 5.42 13.79
4.75 (×4.22) 5.54 (×2.36) 13.73 (×2.53) 21.88 (×1.59)
Table 2: Break-rate by difficulty, defined by incorrect pre-feedback rollouts among 8: very easy (0–1), easy (0–4), hard (5–7), and hardest (7).
4.4
Feedback Anchoring Suppresses Hallucinations
Figure 6 reports the breakdown of filter outcomes for tutor-generated feedback under feedback anchoring at the initial and final rounds of training. As described in Section 3.3, the anchoring prompt requires the tutor to extract a key concept from the question in <concept> tags. We then classify each feedback by rule-based string matching into one of five categories. Kept means the extracted concept appears explicitly in the question text, while no_match means it does not, signaling ungrounded feedback. The remaining categories (no_concept_tag, empty, and all_generic) are cases where the concept tag is absent, empty, or too generic to verify against the question, so we discard them to prioritize precision over recall. Detailed criteria for each category are in Appendix C.2. Two observations stand out. First, the kept rate stays consistently above 70% and slightly increases 6
Problem summary
physics chemistry
For I2(g) , the molar constant-volume heat capacity is CV, m = 28.6 J mol 1 K 1 . Assuming that the electronic contribution is negligible, what is the vibrational contribution to the heat capacity? Choices: A: 7.82 B: 7.68 C: 7.96 D: 8.10 (J mol 1 K 1)
Tutee's physics-style failure
Tutor's chemistry feedback
Answer after tutor feedback
[ ]
The physics specialist correctly identifies that the vibrational contribution is the part of the total heat capacity not explained by translational and rotational degrees of freedom. However, the specialist makes a key chemistry mistake : they treat vibrational degrees of freedom using the same formulas as translational and rotational ones , which is not correct for vibrational contributions. Thus, the reasoning contains a significant conceptual error in how the vibrational contribution is calculated.
The total molar heat capacity at constant volume is given as CV, total = 28.6 J mol 1 K 1 . For a diatomic gas like I2(g) , the translational and rotational contributions at room temperature are approximately CV, trans + rot 52 R . Using R = 8.314 J mol 1 K 1 , 5 R = 2.5(8.314) = 20.785 J mol 1 K 1 . 2 So the vibrational contribution is the remaining part : CV, vib = CV, total CV, trans + rot . CV, vib = 28.6 20.785 = 7.815 J mol 1 K 1 . Rounding: CV, vib 7.82 J mol 1 K 1 . So the correct answer is A .
For a diatomic molecule, the total constant-volume molar heat capacity is CV, m = 2f R , where f is the number of degrees of freedom . For I2(g) , the degrees of freedom are: 3 translational , 2 rotational , 2 vibrational . So total f = 3 + 2 + 2 = 9 .
[ ]
Since there are 2 vibrational degrees of freedom , each contributes R2 . So total vibrational contribution is 2 × R2 = R . Therefore the vibrational contribution is R = 8.314 J mol 1 K 1 . The closest option is D .
[ ]
Key chemistry insight
Failure mode
The physics student is trapped in equipartition-style degree counting , and incorrectly treats vibration like translation/rotation.
Molecular vibrations should not be counted like translational/rotational motion; their heat capacity must be isolated from the measured CV.
[ ]
Rescue pattern
The rescue combines the molecular-vibration insight with the physics calculation CV, vib = CV, total 52 R.
Figure 5: Case study of peer tutoring. Left: a physics-stronger tutee attempts a heat-capacity problem and reaches an incorrect answer through physics-style equipartition reasoning. Middle: a chemistry-stronger, physics-cognizant tutor localizes the error and supplies the missing physical-chemistry insight without revealing the answer. Right: given the same problem with the feedback, the tutee revises its reasoning by combining its physics fluency with the tutor’s physical-chemistry insight to reach the correct answer.
from 72.8% to 74.6%, showing that the anchoring filter preserves sufficient training signal. Second, no_match remains a rare category at about 2.5%, suggesting that feedback anchoring effectively suppresses problem-irrelevant feedback. Initial round Final round0
20
72.8%
21.1% 2.6% 0.6% 3.0%
74.6%
18.9% 2.5% 0.4% 3.6%
40
60
%
80
of-freedom counting. It counts translational, rotational, and vibrational degrees of freedom, then treats the vibrational part as if each vibrational degree contributed in the same way as translation or rotation. Thus, the failure is not that the tutee ignores vibration, but that it handles molecular vibration with the wrong heat-capacity interpretation. The tutor feedback identifies this issue without revealing the answer. Its key insight is that the vibrational contribution should be isolated from the measured heat capacity, rather than counted like translational or rotational motion. With this feedback, the tutee combines the tutor’s molecularvibration insight with its own physics fluency: it treats the vibrational contribution as the remainder of the measured heat capacity after accounting for translational and rotational motion, thereby recovering the correct answer.
100
kept no_match all_generic no_concept_tag empty Figure 6: Filtering statistics under feedback anchoring at the initial and final training rounds.
4.5
A Case Study of Cross-Domain Tutoring
Figure 5 illustrates how co-training can help a student recover even when the error lies within its stronger domain. The problem is a physics-labeled heat-capacity question, and the tutee is the student that is stronger in physics than the tutor. Nevertheless, the tutee initially answers incorrectly. The tutor is stronger in chemistry, and its feedback supplies a complementary physical-chemistry perspective that fills a gap in the tutee’s reasoning. The tutee’s initial reasoning is physics-style in the sense that it relies on equipartition and degree-
This example illustrates why co-training diverse students can be useful. Even when a problem lies in one student’s stronger domain, another student may supply a complementary perspective that pinpoints the reasoning gap and enables the original student to complete the reasoning correctly. Additional examples are provided in Appendix F. 7
5
6
Ablation
Cognizance-Based Gating Ablation. We ablate when tutor feedback is given. Figure 7 compares four gating strategies on the physics–chemistry pair: Always give, Never give, Domain-selective, and Cognizance-based gating. Domain-selective gating still allows feedback from an incognizant tutor only on its stronger domain, testing whether a tutor’s expertise can compensate for its overall incognizance (Section 4.3). Never give and Domain-selective improve the chemistry score of the physics-stronger student, but reduce its physics score below the initial score, showing negative transfer and failing Pareto improvement. Although Always give achieves Pareto improvement, Cognizance-based gating Pareto-dominates it and achieves Pareto improvement with higher scores. Initial student Domain-selective
Never give Cognizance-based (Ours)
Multi-Domain LLMs. Multi-domain training improves cross-task performance (Wei et al., 2021; Chung et al., 2024), but can induce negative transfer that weakens domain-specific expertise. Existing approaches mitigate this by modifying gradients, adapters, or training schedules (Cai et al., 2026; Yang et al., 2026c; Ye et al., 2026a), while multi-teacher distillation aggregates supervision from several teachers into a single student (Xiao et al., 2026; Yang et al., 2026b). In contrast, OPCoD addresses negative transfer through codistillation with cognizance-based gating, rather than one-way transfer into a single model, without relying on external teachers. On-Policy Self-Distillation. On-policy selfdistillation trains an LLM from its own rollouts, with the same model acting as both student and selfteacher. Recent methods condition the self-teacher on privileged information such as successful rollouts, environment feedback, or richer feedback signals (Zhao et al., 2026; Hübotter et al., 2026; Kim et al., 2026; Song et al., 2026; Ye et al., 2026b). Unlike these single-model methods, OPCoD couples two self-distillation processes through bidirectional feedback in a multi-student, multi-domain setting. Multi-LLM Co-Training. Existing work trains multiple LLM agents mainly to improve inferencetime collaboration, such as debate, verifier-scored discussion, or persuasion-balanced dialogue (Park et al., 2025; Liao et al., 2025; Stengel-Eskin et al., 2025; Subramaniam et al., 2025). In contrast, OPCoD uses peer interaction only during training: students exchange feedback to transfer complementary reasoning signals, but are evaluated independently across all paired domains.
Always give
avg@16 score (%)
70 60 50 40 30 Phys domain Chem domain Phys-stronger
Phys domain Chem domain Chem-stronger
Figure 7: Feedback gating strategy ablation on the physics–chemistry pair, shown for the physics-stronger student (left) and chemistry-stronger student (right).
55 50 45 40 Initial student 51 52
5r x 20s 2r x 50s 1r x 100s
53
54
Chem score (avg@16)
Chem score (avg@16)
Round-Step Ablation. We ablate how to allocate a fixed number of OPCoD training steps across rounds. Figure 8 compares 5 × 20, 2 × 50, and 1 × 100 schedules on the physics–chemistry pair, where r × s denotes r rounds with s training steps per round. All schedules improve over the initial student, showing robustness to round-step choice. The 2 × 50 schedule yields strong trajectories for both students, suggesting that both multi-round training (which 1×100 lacks) and sufficient withinround updates (which 5 × 20 lacks) are important. 70.0 67.5 65.0 62.5 60.0 57.5 55.0
Initial student
50
Related Works
7
Conclusion
We presented OPCoD, an on-policy co-distillation framework where student LLMs improve together through peer feedback. In OPCoD, each student performs on-policy self-distillation with a selfteacher conditioned on both its own correct rollout and peer feedback, where the feedback is controlled by cognizance-based gating and feedback anchoring. On Science Q&A tasks, OPCoD achieves mutual Pareto improvement across all evaluated domain pairs, outperforming baselines. Our analyses show that properly gated and anchored feedback can provide complementary reasoning cues while avoiding unreliable or problem-irrelevant feedback.
5r x 20s 2r x 50s 1r x 100s
55
Phys score (avg@16) Phys score (avg@16) Figure 8: Round-step ablation on the physics–chemistry pair, showing evaluation trajectories: physics-stronger student (left), chemistry-stronger student (right).
8
Limitations
Kehua Feng, Xinyi Shen, Weijie Wang, Xiang Zhuang, Yuqi Tang, Qiang Zhang, and Keyan Ding. 2024. Sciknoweval: Evaluating multi-level scientific knowledge of large language models. arXiv preprint arXiv:2406.09098.
First, our experiments focus on Science Q&A tasks, and it remains unknown whether the method is effective for more distant, non-science domain combinations. Second, the current approach is restricted to pairwise co-distillation between two students, and settings involving more than two agents remain unexplored. Third, the framework relies on prompt-based feedback generation, meaning that the detailed wording of the prompt may still affect feedback quality and downstream performance.
Conor F Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, and 1 others. 2022. A practical guide to multi-objective reinforcement learning and planning: Cf hayes et al. Autonomous Agents and Multi-Agent Systems, 36(1):26. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3.
Potential Risks Because OPCoD trains students using feedback generated by other students, its reliability depends on the trustworthiness of the participating models. If a malicious or compromised student is included, it may introduce misleading feedback that is later distilled into another student. This motivates using trusted participants, feedback validation, and monitoring when applying OPCoD beyond controlled experimental settings.
Yinghao Hu, Leilei Gan, Wenyi Xiao, Kun Kuang, and Fei Wu. 2025. Fine-tuning large language models for improving factuality in legal question answering. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4410–4427, Abu Dhabi, UAE. Association for Computational Linguistics. Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and 1 others. 2026. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802.
References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pages 21246–21263.
Jeonghye Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. 2026. Rebellious student: Reversing teacher signals for reasoning exploration with self-distilled rlvr. arXiv preprint arXiv:2605.10781.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, and 1 others. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. 2025. Marft: Multi-agent reinforcement finetuning. arXiv preprint arXiv:2504.16129. Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman E Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. 2025. Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 30215–30248.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. Min Cai, Yu Liang, Longzheng Wang, Yan Wang, Yueyang Zhang, Long Xia, Zhiyuan Sun, Xi Ye, and Daiting Shi. 2026. Advancing general-purpose reasoning models with modular gradient surgery. arXiv preprint arXiv:2602.02301.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, and 1 others. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
9
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, and 1 others. 2025. Toward expert-level medical question answering with large language models. Nature medicine, 31(3):943–950.
conflicts in multi-task lora via orthogonal gradient projection. arXiv preprint arXiv:2601.09684. Hua Ye, Siyuan Chen, Haoliang Zhang, Weihao Luo, Yanbin Li, and Xuan Zhang. 2026a. Synergy over discrepancy: A partition-based approach to multidomain llm fine-tuning. Advances in Neural Information Processing Systems, 38:18893–18923.
Yuda Song, Lili Chen, Fahim Tajwar, Remi Munos, Deepak Pathak, J Andrew Bagnell, Aarti Singh, and Andrea Zanette. 2026. Expanding the capabilities of reinforcement learning via text feedback. arXiv preprint arXiv:2602.02482.
Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. 2026b. On-policy context distillation for language models. arXiv preprint arXiv:2602.12275. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. 2026. Self-distilled reasoner: On-policy selfdistillation for large language models. arXiv preprint arXiv:2601.18734.
Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. 2025. Teaching models to balance resisting and accepting persuasion. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8108–8122. Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. 2025. Multiagent finetuning: Self improvement with diverse reasoning chains. In International Conference on Learning Representations, volume 2025, pages 10840–10862. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, and 1 others. 2026. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. 2026a. Self-distilled rlvr. arXiv preprint arXiv:2604.03128. Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, and 1 others. 2026b. Nemotron-cascade 2: Post-training llms with cascade rl and multi-domain on-policy distillation. arXiv preprint arXiv:2603.19220. Ziyu Yang, Guibin Chen, Yuxin Yang, Aoxiong Zeng, and Xiangquan Yang. 2026c. Disentangling task
10
A
B
Formal Definitions of Pareto Criteria
Consider two students indexed by i ∈ {1, 2} and two domains d ∈ {A, B}. Let sd (π) denote the evaluation score of policy π on domain d, where higher scores indicate better performance.
Algorithm 1 On-Policy Co-Distillation (OPCoD) Input: Initial students π11 , π12 , threshold τ , rounds R, steps K
Definition A.1 (Pareto dominance). A policy π Pareto-dominates a policy π ′ , denoted π ≻P π ′ , if sd (π) ≥ sd (π ′ ), ′
sd (π) > sd (π ),
for r = 1, . . . , R do Compute cognizance gaps ∆1r , ∆2r at round r using validation scores (Eq. (2)) π̄r1 ← πr1 , π̄r2 ← πr2 // Frozen tutors
∀d ∈ {A, B}, and ∃d ∈ {A, B}.
Definition A.2 (Pareto improvement). Training from an initial policy π0 to a learned policy π achieves Pareto improvement if
Pseudo-Code
for i ∈ {1, 2} do πtutee ← πri , πtutor ← π̄r−i
1 2
for k = 1, . . . , K do foreach x ∈ D do 5 y 1 , . . . , y n ∼ πtutee (· | x) π ≻P π0 . s ← correct rollout in {y j }nj=1 , if any; otherwise ∅ Definition A.3 (Mutual Pareto improvement). 6 if ∆tutor ≤τ // Cognizant r Training from initial policies π01 , π02 to learned poli7 Generate and sanitize feedcies π 1 , π 2 achieves mutual Pareto improvement back using feedback anchorif ing (Appendix C) π i ≻P π0i , ∀i ∈ {1, 2}. f j ∼ πtutor (·|x, y j ), ∀j f j ← Sanitize(f j ), ∀j 8 else // Incognizant 9 f 1, . . . , f n ← ∅ 10 end 11 end 12 Update πtutee using Li (πtutee ; πtutor ) (Eq. (1)) 13 end i ← πtutee 14 πr+1 15 end end 3 4
1 2 Return: πR+1 , πR+1
C
Feedback Anchoring Process
C.1
Prompts for Feedback Generation
Prompt for feedback generation with generic instruction You are a domain expert reviewing a student's response to a conversation or problem. Read the full context below, then evaluate the student's final response. \ Provide concise, constructive feedback: point out what is correct, what is wrong \ or missing, and how the student should improve their reasoning. Do NOT directly reveal the correct answer. Context: {problem}
11
ences to answer options inside explanatory text. Second, we validate the concept anchor produced by the feedback-anchoring prompt: the extracted <concept>...</concept> field must contain at least one non-generic word that appears in the problem text. If this validation fails, the feedback is dropped. Finally, for validated feedback, we strip the concept tag before injecting the feedback, so the tutee receives only the feedback content.
Student's response: {response} Expert feedback:
Prompt for feedback generation with feedback anchoring You are a {peer_domain} reasoning specialist. You excel at {peer_domain} concepts and {peer_domain} reasoning. Your peer is a {agent_domain} specialist. They are strong at {agent_domain} concepts but may make mistakes when the problem involves {peer_domain} reasoning.
Meaning
match:<word> no_concept_tag
Kept; <word> appears in the problem text. Dropped; the concept tag is absent or malformed. Dropped; the concept tag is present but empty. Dropped; the concept only contains generic words. Dropped; the concept does not appear in the problem. Dropped; feedback generation returned no output.
empty
Below is the question and your peer's response. Your task has TWO steps.
all_generic no_match
STEP 1: Anchor to the problem: Read the QUESTION carefully. Identify ONE key technical term or concept word (1-3 words) that is central to the problem. Use the EXACT wording as it appears in the question text. Do NOT use generic words such as "problem", "question", " answer", "value", "calculation", "formula ", "student", "peer", or "specialist" -these will be rejected. Output the concept in <concept>...</concept> tags BEFORE writing any other text.
fb_none
Table 3: Outcome categories for feedback anchoring process.
STEP 2: Provide feedback: After the concept tag, write your feedback as a short paragraph of 3 to 5 sentences ( no more than about 80 words total). Focus on the {peer_domain} aspects of the reasoning where relevant. Point out what is correct and what is wrong or missing.
D
Experimental Details
D.1
Setup for Multi-Domain Science Q&A Experiment
We follow the data construction of Hübotter et al. (2026). For each domain in SciKnowEval (Feng et al., 2024), we split the data into train and test sets with a 9:1 ratio. For OPCoD, we additionally sample a small validation set from the training portion, yielding disjoint train, validation, and test partitions. To construct the initial students, we start from Qwen3-8B (Yang et al., 2025) and supervised finetune a separate model for each domain using chainof-thought (CoT) data from that domain. The resulting students are not intended to be perfect solvers; rather, each student is relatively stronger on its target domain than the other domain-tuned students. We use these three single-domain SFT models as the initial students for OPCoD. Since SciKnowEval provides problem–answer pairs without CoT rationales, we generate rationalized solutions by prompting gpt-5-mini with each training problem and its gold answer. SFT is performed with LoRA (Hu et al., 2022) using rank 8 on all linear layers, learning rate 2 × 10−4 with cosine scheduling and 10% warmup, batch size 16, three epochs, and bf16 precision.
Do NOT solve the problem yourself. Cite specific steps or lines in the peer's response. Do NOT directly reveal the correct answer letter. Question: {problem} The {agent_domain} specialist's response: {response} Your response (begin with <concept>...</ concept>, then a 3-5 sentence paragraph):
C.2
Tag
Sanitizing Feedback
Before tutor feedback is injected into the tutee’s reprompt, we apply a lightweight sanitization and validation pipeline. First, we remove explicit answerrevealing patterns, such as boxed answer letters, bolded answer letters, and phrases like “Final answer: X” or “The correct option is X”. We do not remove numeric boxed values or ordinary refer12
D.2
Implementation Details
Parameter
Value
General
We implement GRPO and SDPO using the official codebase of Hübotter et al. (2026), utilizing their codebase in full compliance with its Apache-2.0 license for academic research. OPCoD is built on top of SDPO: each student is trained with the same self-distillation objective, but the self-teacher is additionally conditioned on tutor feedback. Here, the self-anchor is the student’s earliest correct rollout for the same problem when available, and is empty otherwise. The feedback is generated by the frozen tutor model, then passed through sanitization and feedback anchor validation before being used.
Model Thinking Data Max prompt length Max response length Max model length
2048 4096 18944
Batching / Rollout Question batch size Mini batch size Number of rollouts Inference engine Temperature / Top-p
32 32 8 vLLM 1.0 / 1.0
Evaluation
Relative to SDPO, OPCoD adds four implementation components. First, a feedback collector runs the tutor model in a separate vLLM instance and generates feedback for the student responses in each training batch. Second, concept-anchor validation discards feedback whose extracted <concept> tag does not appear in the problem text, filtering ungrounded feedback. Third, sanitization removes direct answer-revealing patterns, such as boxed answer letters, to prevent shortcut imitation. Fourth, dynamic cognizance-based gating evaluates both agents on a validation set at the end of each round; if the tutor is classified as incognizant, feedback collection is skipped in the next round and training falls back to self-distillation without feedback.
Number of rollouts Temperature / Top-p
16 0.6 / 0.95
SDPO loss Full-logit distillation Top-K distillation Tail bucket Distillation divergence Teacher regularization Teacher EMA update rate Rollout correction
True 100 True JSD (α = 0.5) EMA 0.05 Token-level IS, threshold 2.0
Training Optimizer Learning rate Warmup steps Weight decay Gradient clip norm Total steps per agent
Each OPCoD round consists of two directional phases. In the first phase, one student is updated while the other is loaded as a frozen tutor; in the second phase, their roles are swapped. At the end of each phase, we save both FSDP shards and a HuggingFace-merged checkpoint, which is used to load the tutor model for the next phase. All experiments are run on a single node with 2× NVIDIA H200 GPUs. The actor uses FSDP across the two GPUs, while the tutor vLLM process is colocated on the same GPUs with limited memory utilization.
D.3
Qwen/Qwen3-8B False
AdamW 1 × 10−5 , constant 5 0.01 1.0 100
Table 4: Hyperparameters for the SDPO.
E
Break-Rate Analysis
We diagnose whether tutor feedback can change the correctness of the tutee’s response. For each domain pair, we take the two initial students and evaluate them on a held-out subset of the pair’s training distribution. For each problem, a student first generates n = 8 rollouts. The other student then generates feedback, which is injected back into the reprompt together with the original problem and previous correct rollout if exists. The same student then re-answers the problem, which takes a role of self-teacher. For every rollout, we record whether the pre-feedback answer is correct or wrong, and whether the post-feedback answer is correct or wrong. The break-rate is the fraction of originally correct rollouts that become wrong after
Hyperparameters
We report the main hyperparameters used in our experiments. Table 4 lists the SDPO hyperparameters, which also serve as the base configuration for OPCoD. Table 5 reports the additional hyperparameters specific to OPCoD. Table 6 lists the GRPO baseline hyperparameters. 13
Parameter
Value
Parameter
Training
General
# rounds
Phys–Chem (2) Chem–Mat (2) Mat–Phys (5) 100
Total steps per agent
Model Thinking
Peer inference backend Max tokens for feedback Max model length Max reprompt length
Max prompt length Max response length Max model length
vLLM 1024 8192 10240
Samples per validation problem
Question batch size Mini batch size Number of rollouts Inference engine Temperature / Top-p
0.2 60 problems balanced per domain 16
32 32 8 vLLM 1.0 / 1.0
Evaluation Number of rollouts Temperature / Top-p
Table 5: Additional hyperparameters for OPCoD. The underlying SDPO hyperparameters are the same as in Table 4.
16 0.6 / 0.95
Algorithm Normalize advantage by std Critic KL loss Rollout correction
adding feedback: BreakRate =
2048 8192 10240
Batching / Rollout
Feedback processing and gating Threshold τ Validation set
Qwen/Qwen3-8B False
Data
Data
#(C → W ) . #C
False Disabled Disabled Token-level IS, threshold 2.0
Training Total steps per agent Optimizer Learning rate Warmup steps Weight decay Gradient clip norm
We group problems by difficulty using the number of wrong pre-feedback rollouts among the eight sampled rollouts: very easy (0–1), easy (0–4), hard (5–7), and hardest (7). Across the three domain pairs, the aggregate break-rate is 2.77% for cognizant tutors and 6.53% for incognizant tutors; restricting to problems in the tutor’s stronger domain gives 3.56% and 5.12% respectively.
100 AdamW 1 × 10−5 , constant 5 0.01 1.0
Table 6: Hyperparameters for the GRPO.
G F
Value
Walltime Analysis
Additional Case Studies OPCoD adds tutor-feedback generation on top of the SDPO training pipeline, which can introduce additional walltime cost; we analyze this cost on the physics–chemistry experiment. Figure 9 reports the average walltime per training step. We report the overall OPCoD average, and also separate OPCoD steps where feedback generation is enabled (FB-on) from those where feedback is skipped by the gating mechanism (FB-off).
We present additional examples demonstrating how tutor feedback effectively guides the tutee in resolving diverse conceptual errors. Figure 10 shows a chemistry-domain example where the tutee’s failure comes from a chemistry misconception rather than from being stuck in physics-style reasoning. The chemistry-stronger tutor corrects this misconception, allowing the tutee to revise its answer. Figure 11 shows a physics-domain example where the physics-stronger tutee applies relevant formulas but misses the phase-equilibrium interpretation of the quantity being asked. The chemistrystronger tutor points out this missing interpretation, enabling the tutee to combine it with its physics calculation and correct the answer.
The overall per-step walltime increases only modestly from 6.25 min/step for SDPO to 6.66 min/step for OPCoD. As expected, FB-on steps are slower due to feedback generation, while FB-off steps have a cost comparable to SDPO. Thus, the feedback mechanism introduces a limited overhead in the overall training pipeline. 14
8
6.25
6.66
7.20
SDPO
OPCoD (all)
OPCoD (FB-on)
6.12
min/step
6 4 2 0
OPCoD (FB-off)
Figure 9: Average walltime per training step on the physics–chemistry experiment. FB-on and FB-off denote OPCoD phases with and without feedback generation, respectively.
15
Problem
chemistry insight
misconception
For the molecule CCCCCl , choose the correct logarithmic aqueous solubility value. A: 2.9 B: 2.03 C: 8.4 D: 0.71
Tutee's flawed heuristic
Tutor's chemistry feedback
Answer after tutor feedback
To estimate the aqueous solubility of CCCCCl , we look for the option that is chemically reasonable. CCCCCl is a long-chain chlorinated hydrocarbon. Longer hydrocarbon chains typically negative log result in more solubility values. Among the choices, 8.4 is the most negative and therefore the best estimate.
The peer correctly identifies that CCCCCl is a chlorinated hydrocarbon and should have low aqueous solubility. However, the key chemistry issue is that 8.4 is far too low for a molecule with only one chlorine substituent a heavily chlorinated and not aromatic system.
CCCCCl means a carbon chain of four carbons with one chlorine atom attached. So the molecule is 1-chlorobutane. 1-chlorobutane is a relatively small, nonpolar organic molecule with only one chlorine substituent. Such a molecule typically has moderate, not extremely low, aqueous solubility. Among the choices, 8.4 is too negative, while 2.03 is chemically reasonable.
[ ]
[ ]
The correct answer is C .
[ ]
[ ]
So the reasoning is flawed in the final chemical judgment.
[ ]
[ ]
The correct answer is B .
Failure mode
Key chemistry insight
Rescue pattern
The tutee lacks structure solubility understanding and overestimates how insoluble CCCCCl should be.
CCCCCl is a small mono-chlorinated alkyl chain, not a heavily chlorinated aromatic compound; 8.4 is too extreme.
The rescue replaces the flawed solubility heuristic with structure-based chemistry: CCCCCl is 1-chlorobutane, not a heavily chlorinated aromatic molecule.
Figure 10: Chemistry-domain example where tutor feedback corrects a chemistry misconception in the tutee’s original reasoning.
Figure 11: Physics-domain example where chemistry-perspective feedback helps the tutee identify the missing phase-equilibrium interpretation and revise its answer.
16