arXiv CSopen access

Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

May 6, 2026

Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe Wenjin Hou1,∗ , Shangpin Peng3,∗ , Weinong Wang3,† , Zheng Ruan3 , Yue Zhang1 , Zhenglin Zhou1 Mingqi Gao3 , Yifei Chen3 , Kaiqi Wang3 , Hongming Yang3 , Chengquan Zhang3 , Zhuotao Tian2 Han Hu3, ‡ , Yi Yang1 , Fei Wu1 , Hehe Fan1, B 1 Zhejiang University 2 Shenzhen Loop Area Institute 3 LLM Department, Tencent [email protected]

[email protected]

On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model. Despite its empirical success, the conditions under which OPD yields reliable improvement remain poorly understood. In this work, we identify two fundamental bottlenecks that limit effective OPD: insufficient exploration of informative states and unreliable teacher supervision for student rollouts. Building on this insight, we propose Uni-OPD, a unified OPD framework that generalizes across LLMs and MLLMs, centered on a dual-perspective optimization strategy. Specifically, from the student’s perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training. From the teacher’s perspective, we show that reliable supervision hinges on whether aggregated token-level guidance remains order-consistent with the outcome reward. To this end, we develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories. We conduct extensive experiments on 5 domains and 16 benchmarks covering diverse settings, including single-teacher and multi-teacher distillation across LLMs and MLLMs, strong-to-weak distillation, and cross-modal distillation. Our results verify the effectiveness and versatility of Uni-OPD and provide practical insights into reliable OPD.1

Teacher

46.6

42.9

Uni-OPD

Student

58.3

Math

47.0

OPD

64.0

63.9

41.2

62.7

15.9

54.5

15.9

36.0 0

60.2

60.5

65.6

60.8 59.0

53.5

64.1

64.3

60.4 61.3

53.5

100

46.3 42.7 44.6

46.6

200

500

300

400

63.6 60.2

Code

60.8

RL 47.0

15.9 (Student)

63.6

OPD

47.0 48.5

Math

Uni-OPD 48.5

Code

arXiv:2605.03677v1 [cs.LG] 5 May 2026

Abstract

59.1

59.9 60.5

56.3 53.5 (Student)

Multi-Teacher Distillation Strong-to-Weak Distillation Cross-Modal Distillation

100 150 200 250 300 350

Optimization Steps

Figure 1: Overall performance comparisons and convergence behavior. Results are shown for settings including multi-teacher, strong-to-weak, and cross-modal distillation on math reasoning and code generation tasks. Uni-OPD consistently outperforms OPD and converges faster than RL, demonstrating its effectiveness across diverse settings.

Introduction

Injecting complex reasoning abilities, domain knowledge, and human preferences into LLMs and MLLMs remains a core challenge in the post-training stage. Conventional approaches typically follow a two-stage paradigm: supervised fine-tuning (SFT) first, followed by reinforcement learning (RL) (Guo et al., 2025a; Xu et al., 2025a; Zeng et al., 2026; Zhao et al., 2026a). While SFT leverages expert data for training, its inherently off-policy nature introduces substantial exposure bias (Qin et al., 2025; Song & Zheng, 2026). Entering rarely covered erroneous states during inference may lead to compounding errors. Alternatively, on-policy RL (e.g., GRPO (Shao et al., 2024b)) alleviates distribution shift through online sampling. However, it mainly relies on sequence-level or terminal rewards, making fine-grained credit assignment difficult and limiting the stability of long-term training (Team et al., 2026). ∗ Equal contribution. † Project leader.

⋆ Work was done when Wenjin Hou and Shangpin Peng interned at Tencent.

‡ Project supervisor.

B Corresponding author.

1 Code is available at https://github.com/WenjinHou/Uni-OPD.

Recently, on-policy distillation (OPD) has emerged as a promising post-training paradigm for efficiently transferring the knowledge and capabilities of domain experts into a single, unified model. It combines the strengths of RL and SFT, namely on-policy sampling and token-level supervision. Concretely, OPD trains the student on its own sampled trajectories with teacher feedback under a reverse KL objective (Lu & Lab, 2025; DeepSeek-AI, 2026). Despite its empirical success, current OPD research remains largely confined to LLM distillation (Zhou et al., 2025; Yang et al., 2026b; Xiao et al., 2026; Yang et al., 2026c; Wu et al., 2026). Although a few recent works extend OPD to MLLMs, they are restricted to limited subsets of tasks within a single modality, such as video (Li et al., 2026a) or speech (Cao et al., 2026). To this end, we first aim to develop a unified OPD framework for both LLMs and MLLMs, enabling effective knowledge distillation across tasks and modalities. Key observations. Beyond unifying the framework, we raise a more fundamental question: what makes OPD a reliable optimization paradigm? We posit that effective OPD depends on two factors. First, the student must sufficiently explore informative states, i.e., diverse and appropriately difficult self-generated trajectories. Second, the teacher’s token-level supervision must remain reliable when applied to student rollouts. In particular, the reliability of token-level guidance is significantly enhanced when its trajectory-level aggregation remains order-consistent with outcome reward (i.e., correct trajectories receive higher aggregated scores than incorrect ones). The outcome reward thus provides a global anchor for calibrating unreliable teacher supervision. These observations motivate a dual-perspective optimization strategy that jointly improves student exploration and the reliability of teacher signals. Our recipe. Building on these insights, we introduce Uni-OPD, a dual-perspective strategy for optimizing OPD from the fundamental roles of the student and the teacher. In this unified framework, we adopt two complementary data-balancing strategies, namely offline difficulty-aware and online correctness-aware balancing, to promote exploration of informative student-generated states. We further present a novel outcome-guided margin calibration mechanism to obtain reliable teacher supervision. Extensive experiments on LLMs and MLLMs verify our recipe. To summarize, our contributions are threefold: •

Key bottlenecks of OPD. We identify two core bottlenecks in OPD: insufficient exploration of informative states and unreliable teacher supervision for student rollouts. Our analysis reveals that reliable teacher supervision largely depends on whether token-level guidance remains order-consistent with the outcome reward. • Dual-perspective optimization recipe. We present a dual-perspective optimization recipe for unified OPD that jointly improves student exploration and teacher supervision. Concretely, we combine offline and online data balancing with an outcome-guided margin calibration mechanism, leading to more effective optimization. • Comprehensive experimental validation. We conduct extensive experiments on 5 domains and 16 benchmarks covering diverse settings, including single-teacher and multi-teacher distillation across LLMs and MLLMs, strong-to-weak distillation, and cross-modal distillation (i.e., combining text-only and multimodal tasks). Our results verify the effectiveness and versatility of Uni-OPD and provide practical insights into reliable OPD.

Related Work

Knowledge distillation for LLMs and MLLMs. Knowledge distillation (Hinton et al., 2015; Xu et al., 2024) aims to transfer knowledge from a larger teacher model to a smaller student model. Conventional approaches typically rely on off-policy forward Kullback–Leibler (KL) divergence on a static dataset to align the student’s generation distribution with that of the teacher (Liu et al., 2024d; Guo et al., 2025b; He et al., 2025a; Liu & Zhang, 2025; Ko et al., 2025). Another line of work treats supervised fine-tuning (SFT) on tokens generated by the teacher as an alternative off-policy distillation strategy for eliciting reasoning capabilities during LLM and MLLM post-training (Guo et al., 2025a; Zhang et al., 2025c; Bansal et al., 2025; Zhang et al., 2025b; Team et al., 2026; Xiao et al., 2026). Though effective, these off-policy methods essentially imitate the teacher’s behavior, limiting the student’s ability to surpass the teacher and making the student prone to exposure bias (Song & Zheng, 2026). On-policy distillation. OPD (Agarwal et al., 2024; Lu & Lab, 2025) allows a superior teacher to provide feedback on the student’s on-policy trajectories. This paradigm effectively alleviates exposure bias and elevates the student’s upper performance bound. Owing to these merits, OPD has become an efficient way to merge capabilities from multiple experts into a single student model (Xiao et al., 2026; Yang et al., 2026c), as well as to support strong-toweak distillation (Bai et al., 2025a; Zeng et al., 2026). Building on this paradigm, current studies on OPD have branched into several key directions. From the lens of the teacher, recent work explores teacher-free self-distillation paradigms (Kujanpää et al., 2024; Shenfeld et al., 2026; Zhao et al., 2026b; Hübotter et al., 2026; Ye et al., 2026; Zhang et al., 2026a; Stein et al., 2026), develops black-box OPD methods (Ye et al., 2025; Xiong et al., 2026), and facilitates distillation across different model families (Patiño et al., 2025). Complementary efforts focus on unified training frameworks (Zhang et al., 2026b) and stable optimization strategies (Jin et al., 2026; Kim & Baek, 2026; Li et al., 2026b; Xu et al., 2026) combined with RL (Yang et al., 2026a; Qu et al., 2026; Jang et al., 2026; Wang et al., 2026). Few works extend OPD to multimodal domains (Bousselham et al., 2025; Ko et al., 2026; Li et al., 2026a; Cao et al., 2026). In this work, we push OPD with a dual-perspective recipe that promotes student exploration and teacher reliability, generalizing across LLMs and MLLMs. More detailed related work is provided in the §E. 2

Preserve data diversity & difficulty spectrum Ensure meaningful signals & explore broader solution space Stronger student policy by merging knowledge from multi-expert teachers

Better student exploration

Student Perspective

Teacher Perspective

Uni-OPD

Offline Difficulty-aware Data Balancing Data difficulty distribution

Domain Data

Teacher Model Pool Student Policy

Balanced distribution

LLM LLM MLLM MLLM MLLM Math Code Math Logic Doc Teacher Teacher Teacher Teacher Teacher

Upsampling mid-difficulty

Domain Expert

Outcome-guided Margin Calibration Order-consistency based on outcome reward

Online Correctness-aware Data Balancing

Promoting Exploration

Reliable Teacher Supervision

Rollout Outcome Imbalance lack sufficient outcome diversity

rollout

Calibration More reliable teacher supervision

Student balanced composition

ratio control

Reward Margin calibration

Margin shift

Practical Benefits Preserve data diversity & difficulty spectrum Ensure meaningful signals & explore broader solution space

Stronger student policy by absorbing capabilities from multiple domain-specific teachers

Better student exploration

Figure 2: Overview of the Uni-OPD framework. (Left) Offline difficulty-aware and online correctness-aware data balancing promote student exploration. (Right) Outcome-guided margin calibration mechanism improves the reliability of teacher supervision. (Middle) The resulting student policy merges complementary capabilities from multiple domain-specific teachers more effectively than standard OPD, leading to stronger overall performance.

Methodology

We propose Uni-OPD, a unified framework that advances OPD across LLMs and MLLMs, as shown in Fig. 2. Our design is driven by two fundamental bottlenecks in OPD: insufficient exploration of informative student-generated states and unreliable teacher supervision for student rollouts. Uni-OPD addresses them with a dual-perspective recipe that enhances student exploration and calibrates teacher supervision to align with the outcome reward. We first introduce the preliminaries in §3.1, followed by an overview of Uni-OPD in §3.2. We then detail the exploration strategy in §3.3 and the supervision calibration mechanism in §3.4. 3.1

Preliminaries

On-policy distillation. OPD retains the on-policy nature of optimization while providing token-level credit assignment, enabling effective post-training. During training, the student policy πθ samples its trajectories and is optimized by minimizing the reverse Kullback-Leibler (KL) divergence to the teacher policy πT over these samples: h i JOPD (θ) = min Eq∼ D, τ ∼πθ (·|q) DKL πθ (τ | q) πT (τ | q) , (1) θ

where o<t denotes the prefix before step t. The gradient naturally induces a token-level reward at step t, analogous to standard RL: πT (ot | q, o<t ) . πθ (ot | q, o<t ) This formulation provides fine-grained credit assignment signals at the token level. rtOPD = log πT (ot | q, o<t ) − log πθ (ot | q, o<t ) = log

(3)

Analyzing teacher supervision in OPD. As shown in Eq. (3), OPD relies on the teacher to provide fine-grained supervision for student-generated trajectories. For effective optimization, this signal should align with overall trajectory correctness. In practice, this alignment is not guaranteed and can fail in several typical ways: (a) OOD degradation: when student rollouts enter sparse or out-of-distribution regions relative to the teacher, log πT (ot | ·) may become noisy, disrupting the ranking between correct and incorrect trajectories. (b) Overestimation of incorrect trajectories: incorrect trajectories may receive abnormally high scores when their local token patterns align with the teacher’s high-confidence regions. (c) Underestimation of correct trajectories: correct trajectories may receive abnormally low scores when their generation paths deviate from the teacher’s dominant regions, thereby suppressing useful reasoning paths. These phenomena suggest that teacher supervision is not always reliable, motivating us to introduce an outcome reward as a global anchor for calibrating trajectory-level supervision. 3.2

The Overview of Uni-OPD

In this work, we propose Uni-OPD, a unified OPD framework that generalizes across both LLMs and MLLMs, as illustrated in Fig. 2. Formally, given expert teachers {πT1 , πT2 , . . . , πTN } who specialize in different domains, and 3

Training data difficulty distribution

Filtering out easy or hard samples

Data filtering: hurts data diversity Data balancing: helps exploration

Difficulty-aware data balancing

Training data difficulty distribution

Balancing Strategies and Their Impact

Filtering out easy or hard samples

Difficulty-aware data balancing

Data filtering: hurts data diversity Data balancing: helps exploration

Figure 3: Data difficulty distribution and its impact on OPD performance. (Left) Training data often exhibits mirrored J-shaped or U-shaped difficulty distributions. (Right) A naive strategy is to filter out overly easy or overly hard samples (i.e., all-correct or all-wrong cases), but this reduces diversity. In contrast, our difficulty-balancing strategy upsamples mid-difficulty samples to preserve a balanced spectrum and empirically outperforms filtering. letting wi denote the weight assigned to teacher πTi , we define the objective as: N JUni-OPD (θ) = ∑ wi DKL πθ ∥ πTi ,

(4)

i =1

This formulation provides a unified objective for both single-teacher and multi-teacher distillation by aggregating supervision from multiple experts. Building on this objective, we optimize OPD from the two fundamental roles. From the student’s perspective, we introduce a data-balancing strategy that promotes exploration via offline difficultyaware and online correctness-aware selection. From the teacher’s perspective, we develop an outcome-guided margin calibration mechanism to correct unreliable token-level supervision by enforcing consistency with outcome rewards. These designs stabilize optimization and improve the reliability of OPD. 3.3

Joint Offline and Online Data Balancing Strategy for Student Exploration

From the student’s perspective, sufficient diversity and an appropriate level of difficulty in the generated trajectories are essential for effective OPD. To this end, based on our empirical study, we propose complementary data-balancing strategies for both offline data construction and online sampling. Offline difficulty-aware data balancing. A prevalent practice in RL is to estimate prompt difficulty via multiple rollouts and then filter out samples that are either overly easy (i.e., always correct) or overly hard (i.e., always incorrect) (An et al., 2025; Zhou et al., 2023a). However, for small-scale models, training data often exhibits a mirrored J-shaped or U-shaped distribution (see Fig. 3). Strictly removing these easy or hard samples can substantially reduce data diversity and limit exploration of informative student-generated states. Our empirical findings show that such filtering leads to substantial performance degradation in OPD. Based on this observation, we adopt a difficulty-aware balancing strategy that selectively upsamples mid-difficulty samples (i.e., correct in only some of multiple rollouts). As shown in Fig. 3, this strategy reshapes the data distribution into a more uniform form while preserving both diversity and difficulty. In addition, it consistently improves performance on math reasoning and code generation. Overall, these results show that maintaining data diversity and a balanced difficulty spectrum enables the student to generate more informative trajectories, thereby exploring a broader solution space.

Code Score

Math Score

45 Online correctness-aware data balancing. After applying offline Learn Without Effective Range Ratio 62 difficulty-aware balancing, we further observe that insufficient explo- 44 Correct Rollout Control 43.8 Only 43.5 43.4 ration can cause the model to collapse to local optima during training, 61 60.4 60.6 43.2 42.7 especially when rollout groups lack sufficient outcome diversity (e.g., 43 60 only incorrect trajectories). To mitigate this issue, we explicitly en- 42 42.4 59.0 42.0 59 force a balanced composition of correct and incorrect trajectories 59.2 41 58.6 58.5 within each rollout group during training. This prevents degenerate 40.9 58 58.1 4B Math Student cases in which all samples share the same outcome and thus yield 40 4B Code Student 57 57.3 uninformative gradients. By maintaining such a balance, we ensure Baseline 0.5 0.75 1.0 1.2 1.5 2.0 (Only Correct) Correct / Incorrect Ratio that the student consistently receives meaningful contrastive signals for stable on-policy learning. As shown in Fig. 4, an appropriate outcome Figure 4: Impact of online correct and inbalance achieves better performance than using only correct samples correct ratio on student final performance. or an excessively high correct/incorrect ratio.

Question: In triangle ABC, the incircle with center O touches the sides AB, BC, and AC at points M, N, and K respectively. Given that angle AOC is four times the size of angle MKN, determine the measure of angle B. Label: 108^\circ

Correct rollout with high average advantage

Example: Question:

Outcome-guided Margin Outcome-guided Margin Calibration Calibration

Unreliable Teacher Supervision

Ground Truth:

Student preferred

feedback Teacher

Student

Incorrect trajectory (R = 0)

rollout

Low distill return Correct outcome High distill return Incorrect outcome

failure mode violate ordering

Student

Teacher

Example of Violating Order Consistency

...

Correct trajectory (R = 1)

rollout

Teacher preferred

Compute return & Validate correctness

Margin Calibration Margin Mask

feedback

Reliable Teacher Supervision

Margin Shift

Effective Student Policy Optimization

Incorrect rollout with high distillation return

Correct rollout with low distillation return

Incorrect reasoning but with high reward

Correct reasoning but with low reward Result: 30^\circ (Incorrect) Distill. return: -0.11 (High)

Result: 108^\circ (Correct) Distill. return: -0.19 (Low)

violate ordering

Figure 5: Demonstration of unreliable teacher supervision and outcome-guided margin calibration mechanism. (Left) Standard teacher supervision in OPD suffers from misalignment between trajectory-level return and outcome rewards, yielding unreliable supervision signals. (Right) Our method uses outcome rewards as a global anchor to calibrate returns through margin-based adjustment, restoring order consistency and improving optimization stability. 3.4

Outcome-guided Margin Calibration for Teacher Supervision

A basic premise of OPD is that the teacher exhibits a directional likelihood preference over positive and negative trajectories. In particular, relative to the student, the teacher should assign higher likelihood to correct trajectories and lower likelihood to incorrect ones. Under this premise, the resulting distillation signal should remain consistent with outcome-level correctness at the trajectory level. We next formalize this principle through a trajectory-level distillation return and develop an outcome-guided calibration strategy based on it. Trajectory-level distillation return. To characterize the overall supervision signal along a rollout trajectory, we define the trajectory-level distillation return as the average log-probability gap between the teacher and the student: |τ |

GOPD (q, τ ) ≜

|τ |

π (ot | q, o<t ) 1 1 = log T ∑ rtOPD , |τ | t∑ π ( o | q, o ) | τ | t < t θ =1 t =1

(5)

This quantity measures the teacher’s average log-likelihood preference over the student along trajectory τ. When GOPD (q, τ ) > 0, the teacher assigns higher confidence than the student on average, encouraging the student to move toward this trajectory. Conversely, when GOPD (q, τ ) < 0, the student is discouraged from moving toward this trajectory. The normalization by trajectory length ensures comparability across trajectories of different lengths. Order consistency as a trajectory-level criterion. For a given question q, let R(q, τ ) ∈ {0, 1} denote the outcome reward of a sampled trajectory τ, where R(q, τ ) = 1 indicates that the final answer in τ is correct for question q, and R(q, τ ) = 0 otherwise. We then define the positive and negative trajectory sets as: S+ (q) ≜ {τ | R(q, τ ) = 1},

S− (q) ≜ {τ | R(q, τ ) = 0}.

(6)

Following the trajectory-level bandit formulation in (Ouyang et al., 2022), we treat the prompt as the context and the entire generated trajectory as a macro-action. Under this view, the associated outcome reward naturally serves as a one-step trajectory-level return, denoted as GRL (q, τ ) = R(q, τ ). Therefore, the outcome-level RL return induces the following oracle ordering: GRL (q, τ+ ) ≥ GRL (q, τ− ) ,

∀τ+ ∈ S+ (q), ∀τ− ∈ S− (q) .

(7)

The derivation process is provided in §A.3. This motivates a trajectory-level reliability criterion for OPD. Under the distillation premise, the trajectory-level distillation return GOPD (q, τ ) should preserve the same outcome-induced ordering as GRL (q, τ ). Specifically, for any prompt q, we expect: GOPD (q, τ+ ) ≥ GOPD (q, τ− ) ,

∀τ+ ∈ S+ (q), ∀τ− ∈ S− (q) .

(8)

Teacher supervision may violate ordering. In practice, however, the teacher’s supervision is not always reliable. As discussed in §3.1, teacher scoring may degrade in sparse out-of-distribution regions, overestimate incorrect trajectories, or underestimate correct ones due to spurious local patterns. Such failures may persist even after 5

Table 1: Performance of Qwen3-4B Student under math reasoning and code generation benchmarks. Teacher models (i.e., Qwen3-4B-Math-RL and Qwen3-4B-Code-RL) are developed through domain-specific RL. The performance of teacher models is denoted by the “RL” type. Bold values indicate the best score within each group. Avg. denotes the average score within each domain. Math Reasoning

Code Generation

Method

AIME 2024

AIME 2025

HMMT 25 Feb.

HMMT 25 Nov.

Avg.

Human Eval+

MBPP+

LCB

Avg.

Student (4B) Teacher (RL)

23.0 60.1

19.3 55.1

12.3 32.5

9.2 38.5

15.9 46.6

77.4 85.2

65.3 69.8

17.7 26.6

53.5 60.5

ExPO OPD ExOPD Uni-OPD

58.7 57.9 62.7 63.3

55.2 52.4 56.1 57.0

Single–Teacher Distillation 32.4 37.0 45.8 30.2 37.8 44.6 33.9 39.3 48.0 34.8 39.8 48.7

84.8 82.6 86.9 88.3

70.2 68.8 70.7 71.6

28.0 25.7 28.6 29.7

61.0 59.0 62.1 63.2

SFT ExPO OPD ExOPD Uni-OPD

58.5 57.5 60.9 61.0 62.3

53.3 54.5 55.2 56.0 57.2

Multi–Teacher Distillation 30.7 34.8 44.3 31.7 36.3 45.0 33.4 38.3 47.0 34.4 39.2 47.7 34.9 39.6 48.5

86.4 86.7 86.3 86.3 88.0

69.6 72.0 70.9 70.6 72.6

26.4 29.0 23.4 29.0 30.1

60.8 62.6 60.2 62.0 63.6

token-level supervision is aggregated to the trajectory level. A mean-based criterion is therefore insufficient, since the mismatch is often concentrated in a few extreme samples: a single overly confident negative trajectory or a severely underestimated positive trajectory can already distort the supervision signal for the entire prompt group. Outcome-guided margin calibration. Based on the above analysis, during OPD training, the constraint in Eq. (8) should hold between positive and negative trajectories within each prompt. To this end, we consider the margin between the lowest-scoring correct trajectory and the highest-scoring incorrect trajectory, which directly characterizes whether the ordering is violated in the most adversarial case. We define the prompt-level margin as m(q) ≜ min GOPD (q, τ ) − max GOPD (q, τ ) . τ ∈ S+ ( q )

τ ∈ S− ( q )

(9)

By construction, m(q) ≥ 0 indicates strict order consistency on prompt q, since even the worst positive trajectory still outperforms the best negative one (see Fig. 5). Thus, m(q) ≥ 0 means that all positive trajectories are ranked above all negative ones for prompt q. To improve robustness, we further require: m(q) ≥ δ ,

(10)

where δ > 0 defines a safety margin against estimation noise and finite-sample fluctuations. Since S+ (q) and S− (q) are determined by outcome rewards, this criterion uses the outcome signal as a global anchor to calibrate the teacher’s trajectory-level scores. This formulation enables direct interventions on the margin, allowing us to suppress ordering violations or enlarge the separation between positive and negative trajectories. Margin calibration strategy. Based on Eq. (10), we present two calibration strategies: margin mask and margin shift. Specifically, the margin mask keeps only the prompt groups satisfying m(q) ≥ δ and discards the rest, so that training is performed only with reliable supervision. Margin shift instead repairs an unreliable group with the smallest additive correction. For groups with m(q) < δ, we define: λ ( q ) = δ − m ( q ),

eOPD (q, τ ) = GOPD (q, τ ) + λ(q) 1{ R(q, τ ) = 1}. G

(11)

This shift preserves the relative ordering within S+ (q) and guarantees eOPD (q, τ ) − max G eOPD (q, τ ) = δ . min G

τ ∈ S+ ( q )

τ ∈ S− ( q )

(12)

In this way, margin shift restores outcome-consistent ordering with a minimal group-level correction, while margin mask provides a more conservative alternative when the supervision signal is too unreliable to calibrate.

Experiments and Analysis

In this section, we conduct comprehensive experiments across both textual and multimodal domains to evaluate the effectiveness of Uni-OPD. We first detail the experimental configurations (§4.1). Subsequently, we assess how the proposed recipe improves OPD performance across diverse distillation scenarios for LLMs and MLLMs, including single-teacher and multi-teacher distillation (§4.2), strong-to-weak distillation (§4.3), and cross-modal distillation (§4.4). Finally, we provide a rigorous ablation study to further analyze the core strategies of our method (§4.5). 6

Table 2: Performance of Qwen3-VL-4B-Instruct Student under math reasoning, logic reasoning, and document understanding benchmarks. Teacher models (i.e., Qwen3-VL-4B-Instruct-Math-RL, Qwen3-VL-4B-Instruct-LogicRL and Qwen3-VL-4B-Instruct-Document-RL) are developed through domain-specific RL. Bold values indicate the best score within each group. Avg. denotes the mean score within each category. Math Reasoning Method

4.1

Logic Reasoning

Document Understanding

Math Dyna We LogicVista LogicVista Visu Chart Doc Info Avg. Avg. AI2D Avg. Vision Math Math Accuracy Format Logic QA VQA VQA

Student (4B) Teacher (RL)

33.8 47.2

62.2 65.3

67.5 54.5 79.5 64.0

49.9 52.5

66.4 73.8

25.1 27.4

47.0 81.7 51.2 82.5

73.5 76.4

94.9 95.1

79.8 82.5 81.6 83.9

OPD Uni-OPD

47.5 47.8

64.8 65.4

77.5 63.3 78.3 63.9

Single–Teacher Distillation 49.8 73.0 26.1 53.1 73.8 28.2

49.6 82.4 51.7 82.6

75.4 75.8

95.2 95.2

81.4 83.6 81.2 83.7

OPD Uni-OPD

41.0 45.5

60.9 62.3

71.7 57.9 76.1 61.0

Multi–Teacher Distillation 51.3 72.3 26.3 54.0 75.2 27.5

50.0 82.6 52.5 83.0

75.0 75.7

95.1 95.3

81.3 83.4 81.6 83.9

Experimental Setup

Models. We conduct experiments on the Qwen3 family (Yang et al., 2025; Bai et al., 2025a). For textual experiments, we use Qwen3-4B and Qwen3-1.7B as student models. In the same-sized setting, we apply domain-specific RL to Qwen3-4B to obtain specialized teachers. In the strong-to-weak setting, we use Qwen3-30B-A3B-Instruct-2507 as the strong teacher. For multimodal experiments, we use Qwen3-VL-2B-Instruct and Qwen3-VL-4B-Instruct as student models, and obtain multimodal teachers through domain-specific RL. Detailed training setups are in §B.1. Training datasets. We use task-specific training data to construct and distill specialized teachers. For textual tasks, we use 57K math reasoning samples filtered from DeepMath (He et al., 2025b) (difficulty level ≥ 6) and 25K code generation samples from the Code subset of Eurus-2-RL-Data (Cui et al., 2025). For multimodal tasks, we use math reasoning, logic reasoning, and document understanding data mainly from OpenMMReasoner-RL-74K (Zhang et al., 2025b). Detailed training data configurations are provided in §B.2. Baselines. We compare Uni-OPD against several representative baselines for LLM distillation: (1) SFT, which performs supervised fine-tuning on teacher-generated trajectories via cross-entropy loss; (2) ExPO (Yang et al., 2026b), a weight-space extrapolation method that merges domain-specific teachers and extrapolates their weights relative to the student model; (3) ExOPD, a reward-level extrapolation approach that scales the reward factor (> 1) to enable the student to surpass the performance boundaries of its teachers. For MLLM experiments, since OPD remains largely underexplored in this setting, we use vanilla OPD as the primary baseline. Evaluation benchmarks. We evaluate Uni-OPD on a comprehensive benchmark suite spanning textual and multimodal capabilities, organized along five capability axes: Textual Math Reasoning: AIME24 (AI-MO, 2024), AIME25 (OpenCompass, 2025), HMMT25 (February and November) (Balunović et al., 2025); Textual Code Generation: HumanEval+ (Liu et al., 2023b), MBPP+ (Liu et al., 2023b), and LiveCodeBench (v6 only, Feb. 25∼May 25) (Jain et al., 2024); Multimodal Math Reasoning: MathVision (Wang et al., 2024a), DynaMath (Zou et al., 2024), and WeMath (Qiao et al., 2025); Multimodal Logic Reasoning: LogicVista (Xiao et al., 2024) and VisuLogic (Xu et al., 2025b); Document Understanding: AI2D (Kembhavi et al., 2016), ChartQA (Masry et al., 2022), DocVQA (Mathew et al., 2021), and InfoVQA (Mathew et al., 2022). Detailed information is in §C.1. 4.2

Single-Teacher and Multi-Teacher Distillation on LLMs and MLLMs

As an effective and flexible paradigm for consolidating capabilities from one or multiple teachers into a unified student model, we first evaluate Uni-OPD on both LLMs and MLLMs across diverse domains. Specifically, for LLMs, following G-OPD (Yang et al., 2026b), we conduct experiments on math reasoning and code generation. For MLLMs, we further consider three domains: math reasoning, logic reasoning, and document understanding. Main results. As shown in Table 1, Uni-OPD achieves the best overall performance on LLM distillation under both single-teacher and multi-teacher settings. In single-teacher distillation, Uni-OPD consistently outperforms OPD and ExOPD, obtaining the highest scores of 48.7 on math reasoning and 63.2 on code generation. More importantly, under multi-teacher distillation, Uni-OPD effectively merges the distinct capabilities of multiple teachers into a single student model, yielding gains of 1.5% and 3.4% over OPD on math reasoning and code generation. A similar trend is observed for MLLMs in Table 2. Under single-teacher distillation, Uni-OPD delivers the best average performance in all three domains, reaching 63.9 on math reasoning, 51.7 on logic reasoning, and 83.7 on document understanding. For multi-teacher distillation, Uni-OPD consistently outperforms OPD, improving the

Table 3: Results for strong-to-weak distillation setting under math reasoning and code generation benchmarks. The teacher model is Qwen3-30B-A3B-Instruct-2507, and the student models are the smaller Qwen3-4B and Qwen31.7B. Bold values indicate the best score within each group. Avg. denotes the average score within each domain. Math Reasoning

Code Generation

Method

AIME 2024

AIME 2025

HMMT 25 Feb.

HMMT 25 Nov.

Avg.

Human Eval+

MBPP+

LCB

Avg.

Teacher

72.1

61.4

42.5

57.1

58.3

81.9

77.2

23.4

60.8

Student OPD Uni-OPD

23.0 56.5 55.9

19.3 46.4 50.2

12.3 28.5 29.8

Qwen3-4B Student 9.2 15.9 33.4 41.2 35.6 42.9

77.4 82.9 83.1

65.3 72.4 71.3

17.7 21.6 28.0

53.5 59.0 60.8

Student OPD Uni-OPD

13.9 35.7 35.2

11.1 27.6 30.7

5.6 17.2 17.7

Qwen3-1.7B Student 4.9 8.9 14.6 23.8 16.4 25.0

61.9 67.1 71.5

53.4 56.7 58.6

11.9 23.4 28.0

42.4 49.1 52.7

average score from 57.9 to 61.0 on math reasoning, from 50.0 to 52.5 on logic reasoning, and from 83.4 to 83.9 on document understanding. The consistent gains across settings validate the robustness of Uni-OPD. 4.3

Strong-to-Weak Distillation

Strong-to-weak distillation is particularly important for the practical post-training of small models (Bai et al., 2025a). We further investigate whether Uni-OPD can better facilitate the transfer of reasoning capabilities from a larger, stronger teacher model (e.g., Qwen3-30B-A3B-Instruct-2507) to significantly smaller students (e.g., Qwen3-4B and Qwen3-1.7B). In this setting, the student is trained on both math and code data, with teacher feedback provided across both domains, which can be viewed as a multi-teacher scenario. Main results. The results for the strong-to-weak distillation setting are presented in Table 3. Notably, Uni-OPD yields significant performance gains across both the 4B and 1.7B student settings. When distilled from the highly capable 30B teacher, Uni-OPD consistently outperforms standard OPD. Specifically, for the 4B student, Uni-OPD achieves average scores of 42.9 in mathematical reasoning and 60.8 in code generation, surpassing standard OPD by 1.7 and 1.8 points, respectively. This trend holds even for the highly constrained 1.7B student, where Uni-OPD lifts performance to 25.0 on math reasoning and 52.7 on code generation. These results demonstrate that Uni-OPD effectively bridges the capacity gap, enabling smaller students to more effectively absorb and replicate complex reasoning behaviors from superior teachers. 4.4

Cross-Modal Distillation

Table 4: Results for cross-modal distillation under textual code generation and multimodal math reasoning benchmarks. The student model is Qwen3-VL-4B-Instruct. The teacher models are developed from the same MLLM backbone via domain-specific RL on textual code and multimodal math domains, i.e., Qwen3-VL-4BInstruct-Code-RL and Qwen3-VL-4B-Instruct-Math-RL, respectively. Bold values indicate the best score within each group. Avg. denotes the average score within each domain. Code Generation (Textual) Method

Math Reasoning (Multimodal)

Human Eval+

MBPP+

LCB

Avg.

Math Vision

Dyna Math

We Math

Avg.

Student Teacher

76.8 82.2

70.0 70.5

37.0 40.1

61.3 64.3

33.8 47.2

62.2 65.3

67.5 79.5

54.5 64.0

OPD Uni-OPD

83.1 84.1

70.6 71.4

38.6 41.3

64.1 65.6

46.1 46.6

65.4 66.5

76.6 78.5

62.7 63.9

Cross-modal distillation is an important yet underexplored setting in OPD. Unlike conventional distillation settings, where capability transfer typically occurs within the same modality, here we investigate whether textual and multimodal capabilities can be unified into a single student policy. Specifically, we use Qwen3-VL-4B-Instruct as the student model, and construct domain-specific teachers from the same MLLM backbone via RL on textual code data and multimodal math data, respectively. As a result, although the student is multimodal, one of the transferred capabilities is learned from a teacher specialized in a purely textual domain, enabling capability transfer across modality boundaries. This setting is beneficial for integrating and transferring cross-modal capabilities.

Main results. As shown in Table 4, Uni-OPD achieves consistent gains over standard OPD across both textual code generation and multimodal math reasoning in this cross-modal setting. Specifically, it improves the average score from 64.1 to 65.6 on code generation and from 62.7 to 63.9 on math reasoning. On the textual side, the gains are consistent across all three code benchmarks, with the largest improvement on LCB (38.6 → 41.3). On the multimodal side, Uni-OPD further improves MathVision (46.1 → 46.6) and DynaMath (65.4 → 66.5), while maintaining strong performance on WeMath. These results suggest that Uni-OPD can effectively absorb and coordinate capabilities originating from both textual and multimodal domains within a unified student model, rather than improving one domain at the expense of the other. For a broader view of cross-modal distillation, we further provide results on code and logic reasoning in §D. 4.5

Ablation Study

Table 5: Results of Uni-OPD variants with a Qwen3-4B Student on math reasoning and code generation. We ablate core strategies (i.e., offline data balancing, online data balancing, and margin calibration) to assess their effectiveness using the Qwen3-4B-RL and Qwen3-30B-A3B-Instruct teacher models. Math Reasoning

Code Generation

Configuration

AIME AIME HMMT HMMT Human Avg. MBPP+ LCB Avg. 2024 2025 25 Feb. 25 Nov. Eval+

OPD Uni-OPD w/o offline data balancing w/o online data balancing w/o margin calibration

60.9 62.3 62.6 62.5 63.0

Qwen3-4B RL Teacher 55.2 33.4 38.3 47.0 57.2 34.9 39.6 48.5 56.5 32.5 38.5 47.5 56.7 33.2 38.9 47.8 54.7 33.4 38.1 47.3

86.3 88.0 88.0 88.0 86.4

70.9 72.6 71.1 71.8 71.6

23.4 30.1 27.9 28.0 25.7

60.2 63.6 62.3 62.6 61.2

OPD Uni-OPD w/o offline data balancing w/o online data balancing w/o margin calibration

56.5 55.9 57.1 57.0 54.9

Qwen3-30B A3B-Instruct Teacher 46.4 28.5 33.4 41.2 50.2 29.8 35.6 42.9 46.3 28.8 36.8 42.2 47.6 26.8 37.0 42.1 48.1 29.1 35.8 42.0

82.9 83.1 80.6 81.6 82.8

72.4 71.3 70.3 71.4 70.4

21.6 28.0 28.0 28.0 25.7

59.0 60.8 59.6 60.3 59.6

In Table 5, we conduct comprehensive ablation studies to evaluate the individual contributions of each strategy in our Uni-OPD. Applying our proposed operations results in a significant improvement in accuracy over the vanilla OPD. In particular, the average gains reach +1.5/+3.4 points on math/code with the Qwen3-4B-RL teacher, and +1.7/+1.8 points with the Qwen3-30B-A3B-Instruct teacher. Offline and online data balancing address insufficient exploration: without either of them, the student policy struggles to be exposed to diverse and challenging trajectories. Margin calibration improves supervision reliability: without it, token-level feedback can become misaligned with outcome rewards, leading to less stable training and suboptimal performance. Margin mask vs. margin shift. We consider various Table 6: Comparison results for different margin calstrategies to calibrate the return signals for improving ibration. We directly incorporate them into OPD to teacher supervision. In this work, we explore two sim- examine which strategy better benefits OPD training. ple variants, namely margin mask and margin shift. As AIME AIME HMMT HMMT shown in Table 6, directly incorporating either mech- Method Avg. 2024 2025 25 Feb. 25 Nov. anism into OPD yields consistent performance gains Student (4B) 23.0 19.3 12.3 9.2 15.9 over the baseline, underscoring the necessity of reliable OPD 57.9 52.4 30.2 37.8 44.6 teacher supervision. Among them, margin shift achieves 56.2 34.3 38.1 47.7 slightly better results and is therefore adopted in our + margin mask 62.3 + margin shift 62.7 56.3 34.4 39.2 48.1 main experiments. More ablations are in §D.3. 4.6

Qualitative Evaluation

To intuitively illustrate the effectiveness of our outcome-guided margin calibration, we use a token-level reward heatmap for visualization. As shown in Fig. 6, we display the two failure modes under the same question: the overestimation of incorrect trajectories (top-left) and the underestimation of correct trajectories (bottom-left). Each token is colored by its reward value: blue tokens indicate student-preferred (rtOPD < 0), and red tokens indicate teacher-preferred (rtOPD > 0), with saturation proportional to magnitude. On the top-left, an incorrect rollout still accumulates a high distillation return: most of its tokens are saturated red, since they fall on regions where the teacher dominates the student. On the bottom-left, a correct rollout receives a low distillation return: its tokens are already well-covered by the student, so the teacher provides little additional return (predominantly faint colors with some blue). The right column shows the same two rollouts after our outcome-guided margin calibration. Concretely, the per-token rewards are uniformly shifted so that the trajectory-level aggregation aligns with the outcome reward. 9

Question: Find all functions $ f: \mathbf{N} \rightarrow \mathbf{N} $ such that for every $ n \in \mathbf{N} $, $ f(n) < f(n+1) $, and $ f(f(n)) = n + 2k $, where $ k \in \mathbf{N} $. 38.png Please reason step by step, and put your final answer within \boxed{}. Student preferred

Answer: \boxed{f(n) = n + k}

Teacher preferred

Outcome-guided margin calibration (Margin shift)

Incorrect rollout with high distillation return Incorrect reasoning but with high reward

Result: \boxed{f(n) = n + 2a} (Incorrect) Distill. return: -0.17 (High)

Distill. return: -0.39 (Low)

Correct rollout with low distillation return

Correct reasoning but with low reward

Result: \boxed{f(n) = n + k} (Correct) Distill. return: -0.22 (Low)

Distill. return: 0.006 (High)

Figure 6: Heatmap visualization of failure modes in OPD and the effect of margin shift. Left: an incorrect rollout with a high distillation return (top) and a correct rollout with a low one (bottom). Right: the same two rollouts after our margin shift, with the outcome ordering restored. 4.7

Analysis and Takeaways

Based on our comprehensive and systematic study on both LLMs and MLLMs across single-teacher, multi-teacher, strong-to-weak, and cross-modal distillation settings, we deliver three takeaways to further advance OPD. •

Balancing reasoning capability and efficiency. Uni-OPD achieves the best performance with substantially fewer optimization steps than RL (Fig. 1), and consistently delivers strong reasoning capability across diverse domains (Tables 1–4, and D.1–D.3 in the Appendix). • Teacher value comes from the capability gap, not absolute strength alone. In OPD, even with the same 4B backbone, a domain-specific RL teacher injects new capabilities and knowledge that drive the student to improve and even surpass the teacher (Tables 1 and 2). Moreover, our dual-perspective recipe further translates this gap into student gains, consistently boosting performance across all model sizes. • OPD distills reasoning as a modality-agnostic capability. Trained jointly on textual and multimodal data, the multimodal student under Uni-OPD improves textual code generation and multimodal math/logic reasoning (Tables 4 and D.3). The per-token signal carries reasoning patterns largely independent of modality, enabling a unified, single-stage path that enhances both textual and multimodal reasoning within one multimodal model. • OPD cleanly merges specialized capabilities, with related ones reinforcing each other. Beyond two teachers, Uni-OPD extends to three, jointly improving all capabilities (Tables 2 and D.2). OPD thus offers a scalable path for merging many specialists into one reasoner, with related ones synergizing via shared reasoning structure. Reproducibility statement. To facilitate a clear understanding of our contributions and support broader adoption of our work, we provide extensive materials. In the main text, we detail the key components of our method in §3 and report the main experimental results in §4. In the supplementary materials, we further elaborate on Method Details (§A), Training Details (§B), and Evaluation Details (§C), which together should be sufficient to reproduce our results. All code, training data, complete scripts, and model checkpoints will be open-sourced upon publication to accelerate future research.

Conclusion and Future Work

In this paper, we present Uni-OPD, a unified OPD framework that generalizes across LLMs and MLLMs. We identify two key bottlenecks for effective OPD: insufficient student exploration of informative states and unreliable teacher supervision for student rollouts. To address them, we propose a dual-perspective optimization strategy: (i) offline difficulty-aware and online correctness-aware data balancing for student exploration, and (ii) outcome-guided margin calibration for teacher supervision. Extensive experiments on 16 benchmarks covering multi-teacher, strong-to-weak, and cross-modal settings demonstrate the effectiveness and versatility of Uni-OPD. We hope this work can provide a practical foundation for future research on scalable and reliable distillation across models, teachers, and modalities. For future work, our findings suggest several promising directions: (1) extending Uni-OPD to larger-scale teacher distillation settings; (2) applying Uni-OPD to broader capability merging scenarios, such as agentic planning, tool use, and long-horizon decision making; and (3) uncovering the mechanistic principles of OPD, particularly how it shapes training dynamics and parameter geometry. 10

References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The twelfth international conference on learning representations, 2024. AI-MO. AIME 2024. https://huggingface.co/datasets/AI-MO/aimo-validation-aime, 2024. AI@Meta.

Introducing Llama 3.1: Our most capable models to date.

https://ai.meta.com/blog/

meta-llama-3-1, 2024a.

AI@Meta. Llama 3 model card. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md, 2024b. Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. POLARIS: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. URL https://hkunlp.github.io/blog/2025/Polaris. Anthropic.

Claude 2, 2023a.

URL https://www-files.anthropic.com/production/images/

Model-Card-Claude-2.pdf.

Anthropic. Introducing Claude, 2023b. URL https://www.anthropic.com/index/introducing-claude. Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku, 2024. URL https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization. Text Reading, and Beyond, 2023. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025a. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025b. Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating LLMs on uncontaminated math competitions. arXiv preprint arXiv:2505.23281, 2025. Hritik Bansal, Devandra Singh Sachan, Kai-Wei Chang, Aditya Grover, Gargi Ghosh, Wen-tau Yih, and Ramakanth Pasunuru. Honeybee: Data recipes for vision-language reasoners. arXiv preprint arXiv:2510.12225, 2025. Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard. https://huggingface.co/spaces/ HuggingFaceH4/open_llm_leaderboard, 2023. Walid Bousselham, Hilde Kuehne, and Cordelia Schmid. VOLD: Reasoning transfer from LLMs to vision-language models via on-policy distillation. arXiv preprint arXiv:2510.23497, 2025. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020. Di Cao, Dongjie Fu, Hai Yu, Siqi Zheng, Xu Tan, and Tao Jin. X-OPD: Cross-modal on-policy distillation for capability alignment in speech llms. arXiv preprint arXiv:2603.24596, 2026. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. ArXiv, 2018. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025. 11

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning, 2023. DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: On-policy distillation of large language models. arXiv preprint arXiv:2306.08543, 2023. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025a. Yiju Guo, Wenkai Yang, Zexu Sun, Ning Ding, Zhiyuan Liu, and Yankai Lin. Learning to focus: Causal attention distillation via gradient-guided token pruning. arXiv preprint arXiv:2506.07851, 2025b. Changyi He, Yifu Ding, Jinyang Guo, Ruihao Gong, Haotong Qin, and Xianglong Liu. DA-KD: difficulty-aware knowledge distillation for efficient large language models. In Forty-second International Conference on Machine Learning, 2025a. Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. DeepMath-103K: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456, 2025b. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. Wenjin Hou, Wei Liu, Han Hu, Xiaoxiao Sun, Serena Yeung-Levy, and Hehe Fan. Seeing is believing? a benchmark for multimodal large language models on visual illusions and anomalies. arXiv preprint arXiv:2602.01816, 2026. Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggu Lim, and Taesup Kim. Stable on-policy distillation through adaptive target reformulation. arXiv preprint arXiv:2601.07155, 2026. Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079, 2026. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European conference on computer vision, pp. 235–251. Springer, 2016. Minsang Kim and Seung Jun Baek. Explain in your own words: Improving reasoning via token-selective dual knowledge distillation. arXiv preprint arXiv:2603.13260, 2026. Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se-Young Yun. DistiLLM-2: A contrastive approach boosts the distillation of LLMs. arXiv preprint arXiv:2503.07067, 2025. Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137, 2026. Kalle Kujanpää, Pekka Marttinen, Harri Valpola, and Alexander Ilin. Efficient knowledge injection in LLMs via self-distillation. arXiv preprint arXiv:2412.14964, 2024. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 12

Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: Reasoning segmentation via large language model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024. Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. Jiaze Li, Hao Yin, Haoran Xu, Boshen Xu, Wenhui Tan, Zewen He, Jianzhong Ju, Zhenbo Luo, and Jian Luan. Video-OPD: Efficient post-training of multimodal large language models for temporal video grounding via on-policy distillation. arXiv preprint arXiv:2602.02994, 2026a. Jingyao Li, Senqiao Yang, Sitong Wu, Han Shi, Chuanyang Zheng, Hong Xu, and Jiaya Jia. Logits-based finetuning. arXiv preprint arXiv:2505.24461, 2025. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016, 2026b. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In ACL, 2022. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024a. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 2023a. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024b. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, 2024c. URL https://llava-vl.github.io/blog/ 2024-01-30-llava-next/. Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, et al. DDK: Distilling domain knowledge for efficient large language models. Advances in Neural Information Processing Systems, 37:98297–98319, 2024d. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems, 36:21558–21572, 2023b. Lingyuan Liu and Mengxiang Zhang. Less is more: Selective reflection for compatible and efficient knowledge distillation in large language models. arXiv preprint arXiv:2508.06135, 2025. Yijun Liu, Jiequan Cui, Zhuotao Tian, Senqiao Yang, Qingdong He, Xiaoling Wang, and Jingyong Su. Typicalnessaware learning for failure detection. arXiv preprint arXiv:2411.01981, 2024e. Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, pp. 2263–2279, 2022. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200–2209, 2021. Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. InfographicVQA. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697–1706, 2022. Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 2024. OpenAI. GPT-4V(ision) system card, 2023. OpenCompass. AIME 2025. https://huggingface.co/datasets/opencompass/AIME2025, 2025. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 2022. 13

Carlos Miguel Patiño, Kashif Rasul, Quentin Gallouédec, Ben Burtenshaw, Sergio Paniego, Vaibhav Srivastav, Thibaud Frere, Ed Beeching, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Unlocking on-policy distillation for any model family, 2025. Shangpin Peng, Senqiao Yang, Li Jiang, and Zhuotao Tian. Mitigating object hallucinations via sentence-level early intervention. In Proceedings of the IEEE International Conference on Computer Vision, 2025. Shangpin Peng, Weinong Wang, Zhuotao Tian, Senqiao Yang, Xing W, Haotian Xu, Chengquan Zhang, Takashi Isobe, Baotian Hu, and Min Zhang. Uni-DPO: A unified paradigm for dynamic preference optimization of LLMs. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=G7DBGlgjjp. Runqi Qiao, Qiuna Tan, Guanting Dong, MinhuiWu MinhuiWu, Chong Sun, Xiaoshuai Song, Jiapeng Wang, Zhuoma Gongque, Shanglin Lei, Yifan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 20023–20070, 2025. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. A survey of multilingual large language models. Patterns, 6(1), 2025. Tianyuan Qu, Longxiang Tang, Bohao Peng, Senqiao Yang, Bei Yu, and Jiaya Jia. Does your vision-language model get lost in the long video sampling dilemma? arXiv preprint arXiv:2503.12496, 2025. Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov, and Aviral Kumar. POPE: Learning to reason on hard problems via privileged on-policy exploration. arXiv preprint arXiv:2601.18779, 2026. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021. Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of CLIP for training-free open vocabulary semantic segmentation. In European Conference on Computer Vision, 2024a. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024b. Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatronlm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626, 2026. Alex Stein, Furong Huang, and Tom Goldstein. GATES: Self-distillation under privileged context with consensus gating. arXiv preprint arXiv:2602.20574, 2026. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. Hunyuan Vision Team, Pengyuan Lyu, Xingyu Wan, Gengluo Li, Shangpin Peng, Weinong Wang, Liang Wu, Huawen Shen, Yu Zhou, Canhui Tang, Qi Yang, Qiming Peng, Bin Luo, Hower Yang, Xinsong Zhang, Jinnian Zhang, Houwen Peng, Hongming Yang, Senhao Xie, Longsha Zhou, Ge Pei, Binghong Wu, Kan Wu, Jieneng Yang, Bochao Wang, Kai Liu, Jianchen Zhu, Jie Jiang, Linus, Han Hu, and Chengquan Zhang. HunyuanOCR technical report, 2025. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2.5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026.

Zhuotao Tian, Michelle Shu, Pengyuan Lyu, Ruiyu Li, Chao Zhou, Xiaoyong Shen, and Jiaya Jia. Learning shape-aware embedding for scene text detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2019. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. Junjie Wang, Bin Chen, Yulin Li, Bin Kang, Yichi Chen, and Zhuotao Tian. DeCLIP: Decoupled learning for open-vocabulary dense perception. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2025. Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024a. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024b. Yinjie Wang, Xuyang Chen, Xiaolong Jin, Mengdi Wang, and Ling Yang. OpenClaw-RL: Train any agent simply by talking. arXiv preprint arXiv:2603.10165, 2026. Yecheng Wu, Song Han, and Hai Cai. Lightning opd: Efficient post-training for large reasoning models with offline on-policy distillation. arXiv preprint arXiv:2604.13010, 2026. Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026. Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973, 2024. Jing Xiong, Hui Shen, Shansan Gong, Yuxin Cheng, Jianghan Shen, Chaofan Tao, Haochen Tan, Haoli Bai, Lifeng Shang, and Ngai Wong. OVD: On-policy verbal distillation. arXiv preprint arXiv:2601.21968, 2026. Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, et al. Redstar: does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284, 2025a. Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504.15279, 2025b. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024. Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. PACED: Distillation at the frontier of student competence. arXiv preprint arXiv:2603.11178, 2026. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024a. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled RLVR. arXiv preprint arXiv:2604.03128, 2026a. Senqiao Yang, Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. LiDAR-LLM: Exploring the potential of large language models for 3d LiDAR understanding. arXiv preprint arXiv:2312.14074, 2023a. Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240, 2023b. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. VisionZip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467, 2024b. Senqiao Yang, Zhuotao Tian, Li Jiang, and Jiaya Jia. Unified language-driven zero-shot domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024c. 15

Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026b. Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, et al. Nemotron-Cascade 2: Post-training LLMs with cascade RL and multi-domain on-policy distillation. arXiv preprint arXiv:2603.19220, 2026c. Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, and Furu Wei. Black-box on-policy distillation of large language models. arXiv preprint arXiv:2511.10643, 2025. Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. arXiv preprint arXiv:2602.12275, 2026. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. URL https://aclanthology.org/P19-1472. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D Lyng, Sanjit Singh Batra, and Robert E Tillman. Fast and effective on-policy distillation from reasoning prefixes. arXiv preprint arXiv:2602.15260, 2026a. Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. LMMs-Eval: Reality check on the evaluation of large multimodal models. In Findings of the Association for Computational Linguistics: NAACL 2025, 2025a. Kaichen Zhang, Keming Wu, Zuhao Yang, Bo Li, Kairui Hu, Bin Wang, Ziwei Liu, Xingxuan Li, and Lidong Bing. OpenMMReasoner: Pushing the frontiers for multimodal reasoning with an open and general recipe. arXiv preprint arXiv:2511.16334, 2025b. Songming Zhang, Xue Zhang, Tong Zhang, Bojie Hu, Yufeng Chen, and Jinan Xu. KDFlow: A user-friendly and efficient knowledge distillation framework for large language models. arXiv preprint arXiv:2603.01875, 2026b. Yi Zhang, Bolin Ni, Xin-Sheng Chen, Heng-Rui Zhang, Yongming Rao, Houwen Peng, Qinglin Lu, Han Hu, Meng-Hao Guo, and Shi-Min Hu. Bee: A high-quality corpus and full-stack suite to unlock advanced fully open mllms. arXiv preprint arXiv:2510.13795, 2025c. Shiwan Zhao, Zhihu Wang, Xuyang Zhao, Jiaming Zhou, Caiyue Xu, Chenfei Liu, Liting Zhang, Yuhang Jia, Yanzhe Zhang, Hualong Yu, et al. Large language model post-training: A unified view of off-policy and on-policy learning. arXiv preprint arXiv:2604.07941, 2026a. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026b. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. Zhisheng Zhong, Chengyao Wang, Yuqi Liu, Senqiao Yang, Longxiang Tang, Yuechen Zhang, Jingyao Li, Tianyuan Qu, Yanwei Li, Yukang Chen, et al. Lyra: An efficient and speech-centric framework for omni-cognition. arXiv preprint arXiv:2412.09501, 2024. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. Advances in Neural Information Processing Systems, 36: 55006–55021, 2023a. Guorui Zhou, Honghui Bao, Jiaming Huang, Jiaxin Deng, Jinghao Zhang, Junda She, Kuo Cai, Lejian Ren, Lu Ren, Qiang Luo, et al. OpenOneRec technical report. arXiv preprint arXiv:2512.24762, 2025. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023b. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836, 2024. 16

Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe Supplementary Material Appendix Outline This material provides supplementary details to the main paper, including the following sections: •

(A) Method Details - (A.1) Offline Difficulty-Aware Data Balancing - (A.2) Online Correctness-Aware Data Balancing - (A.3) Order Consistency of Trajectory-level Returns - (A.4) Outcome-Guided Margin Calibration

•

(B) Training Details - (B.1) Training Setup - (B.2) Training Data - (B.3) Training Reward Acquisition - (B.4) Training Pseudocode - (B.5) Training Dynamics - (B.6) Training Complexity

•

(D) Further Evaluations - (D.1) More Evaluation Results - (D.2) Downstream Task Evaluation - (D.3) Further Ablation

•

(E) Related Work - (E.1) Multimodal Large Language Models - (E.2) Reinforcement Learning - (E.3) On-Policy Distillation

•

(F) Case Studies

Method Details

In this section, we provide a detailed exposition of the key components of our proposed Uni-OPD framework, including its formulations and implementations. A.1

Offline Difficulty-Aware Data Balancing

In this section, we provide a detailed description of our offline difficulty-aware data balancing strategy. Offline rollout sampling. Before training, we perform a one-time offline rollout pass over the entire training set using the student model (e.g., Qwen3-4B). For each training instance, the student is prompted to generate N = 8 independent candidate responses, which serve as the basis for subsequent difficulty estimation. The rollouts are produced with vLLM (Kwon et al., 2023) under the same prompt template that will later be used at training time, so that the estimated difficulty reflects the actual input format the student will see. The decoding configuration is kept fixed throughout this offline phase: we use temperature = 1.0, top-p = 0.95, top-k = 50, and a maximum response length of 16,384 tokens. For each instance, we then verify the correctness of its N candidate responses with the task-specific verifier (§B.3) and record the number of correct ones. The resulting empirical pass rate k/N serves as our proxy for the instance’s difficulty: a lower pass rate indicates a harder example, while a higher pass rate indicates an easier one. Limitations of aggressive difficulty filtering. Prior work on online RL optimization, such as GRPO, often relies on a heuristic pre-training filter that simply discards “trivial” samples such as all-correct cases, because these instances yield zero advantage and therefore provide essentially no learning signal. POLARIS (An et al., 2025), for example, reports that removing the easiest samples leads to consistent performance gains, and argues that keeping an unfiltered dataset can actively hinder training. In the token-level reward OPD setting, however, we find that such aggressive filtering is, in fact, counterproductive. Empirically, removing any specific difficulty tier, whether the easiest or the hardest, consistently hurts final performance. A plausible explanation is that each tier contributes a distinct pattern of token-level credit: easy instances calibrate the student’s baseline behavior, intermediate instances provide the richest contrastive signals between correct and incorrect trajectories, and hard instances expose the student to diverse, non-trivial solution paths. Dropping any tier, therefore, both distorts the overall distribution of token-level credit and narrows the space of solution patterns to which the student is exposed. Difficulty-aware data balancing. Motivated by this observation, we adopt a difficulty-aware balancing scheme that deliberately preserves the full spectrum of difficulty while reweighting its different regions, rather than truncating them. Concretely, after the offline rollout pass, we examine the empirical distribution over the number of correct responses out of N. Across our training sources, we observe two recurring shapes: (i) a U-shaped distribution, where both very easy and very hard instances dominate while intermediate ones are sparse; and (ii) a mirrored-J-shaped distribution, where easy instances dominate and the mass decays toward the hard end. We treat the two shapes slightly differently. For U-shaped distributions, we upsample instances of intermediate difficulty, namely those with 1–7 correct responses out of N = 8, so as to fill in the under-represented middle region. For mirrored-J-shaped distributions, we instead upsample all non-trivial instances, i.e., everything with 1–8 correct responses, to counteract the long tail of easy samples. In both cases, the effect of the reweighting is to flatten the overall difficulty distribution and to ensure that the token-level credit signals arriving during training are more evenly spread across difficulty levels. Empirically, we find that this simple rebalancing consistently leads to better final performance than either no filtering or the conventional drop-the-easy-cases strategy. A.2

Online Correctness-Aware Data Balancing

In this section, we detail the online correctness-aware data balancing strategy that operates during rollout. While the offline difficulty-aware balancing in §A.1 controls the prompt-level difficulty distribution before training, the composition of correct and incorrect trajectories within a rollout group still varies dramatically as the student evolves. This subsection describes how we regulate such intra-group composition online. Motivation. In OPD, for each prompt q we sample G on-policy trajectories {τi }iG=1 and split them into a positive set S+ (q) and a negative set S− (q) based on the outcome reward Ri . As training proceeds, many prompts exhibit degenerate outcome distributions: either |S− (q)| ≪ G (the student nearly masters q) or |S+ (q)| ≪ G (the student often fails on q). In both cases, the outcome-level contrast vanishes and the outcome-guided margin calibration in §A.4 cannot provide any corrective signal, since the prompt-level margin m(q) is undefined. If left unregulated, such degenerate groups dominate the batch and drive the student into local optima with shrinking exploration.

Online correctness-aware balancing. To preserve sufficient outcome diversity throughout training, we maintain a target correct-to-total ratio γ⋆ ∈ (0, 1) at the batch level (we use γ⋆ ≈ 0.5 by default, so positive and negative trajectories are roughly balanced). At each training step, given a freshly rolled-out batch B , we let γ(B) = ∑τi∈B 1{ Ri = 1}/|B| denote the current correct-to-total ratio across the whole batch. Whenever |γ(B) − γ⋆ | > ϵ for a tolerance ϵ, we downweight the over-represented side (correct or incorrect trajectories) by subsampling within each group, so that the overall batch ratio is pulled back to the γ⋆ ± ϵ interval. Subsampling is performed uniformly inside each group, which keeps the intra-group difficulty distribution intact and avoids biasing the prompt-level difficulty spectrum inherited from offline balancing. A.3

Order Consistency of Trajectory-level Returns

This section provides a brief explanation for the order-consistency conditions in Eqs. (7) and (8) of the main paper. The key observation is two-fold. First, treating the entire reasoning rollout as a single macro-action gives GRL (q, τ ) = R(q, τ ), so GRL respects the outcome-induced ordering by construction. Second, under the distillation premise, the trajectory-level distillation return GOPD (q, τ ) is expected to preserve the same ordering, although this is a desideratum rather than a definitional consequence. Trajectory-as-one-action view of outcome-based RL. In outcome-based RL for reasoning, supervision is provided only at the trajectory level: a rollout τ receives a single scalar reward R(q, τ ) determined by the final answer. Under this view, the trajectory-level return reduces to the outcome reward itself, i.e., GRL (q, τ ) = R(q, τ ) .

(13)

Order consistency under binary rewards. For the binary outcome reward adopted in this work, any τ+ ∈ S+ (q) satisfies R(q, τ+ ) = 1, while any τ− ∈ S− (q) satisfies R(q, τ− ) = 0. Combined with Eq. (13), we have GRL (q, τ+ ) = 1 ≥ 0 = GRL (q, τ− ) ,

(14)

for all τ+ ∈ S+ (q) and τ− ∈ S− (q), which recovers Eq. (7) directly. Extension to soft outcome rewards. The same argument extends to soft outcome rewards, where R(q, τ ) ∈ [0, 1] (or any bounded interval) measures a graded notion of correctness, e.g., partial credit or a verifier’s confidence score. As long as the trajectory partition is defined by thresholding the outcome reward, i.e., S+ (q) = {τ | R(q, τ ) ≥ η } and S− (q) = {τ | R(q, τ ) < η } for some threshold η, then by Eq. (13) every positive trajectory attains a return no smaller than that of any negative trajectory, and Eq. (7) still holds. In particular, the binary case is recovered as the special instance η = 1, R ∈ {0, 1}. From RL return to distillation return. The distillation return GOPD (q, τ ) defined in Eq. (5) plays the same role for OPD training as GRL does for outcome-based RL: it is the trajectory-level supervision signal broadcast to all tokens in the rollout. The distillation premise in §3.4 posits that, relative to the student, the teacher assigns a higher log-likelihood to correct trajectories than incorrect ones. In other words, the teacher’s trajectory-level preference is expected to be aligned with the outcome reward, so that GOPD should inherit the same outcome-level ordering as GRL , leading to Eq. (8). Unlike the RL return, however, GOPD is derived from the teacher–student log-probability gap rather than the outcome reward itself, so the ordering is a desired property rather than a guaranteed one. The order-consistency condition in Eq. (8) provides a principled target, and subsequent margin mask and margin shift strategies (§A.4) are designed to enforce it whenever the teacher’s supervision violates this property in practice. A.4

Outcome-Guided Margin Calibration

In this section, we describe the details of the two outcome-guided margin calibration strategies introduced in §3.4: Margin Mask and Margin Shift. Both strategies operate on the trajectory-level distillation returns { GOPD (q, τi )}iG=1 within a rollout group of a prompt q, with the common goal of enforcing the order-consistency condition m(q) ≥ δ (Eq. (10)). They differ in how they repair violations: Margin Mask removes the most adversarial trajectories until the condition holds, whereas Margin Shift applies a minimal additive correction to restore the margin in closed form. Margin choices: MinMax vs. Mean. Following the prompt-level margin in Eq. (9), we define the margin between S+ (q) and S− (q) in two modes: the MinMax mode uses minτ ∈S+ GOPD − maxτ ∈S− GOPD and characterizes the worst-case ordering violation; the Mean mode uses meanτ ∈S+GOPD − meanτ ∈S−GOPD and reflects the averagecase ordering tendency. MinMax is more conservative (it forces every positive to outrank every negative), while Mean is more lenient and less sensitive to individual outliers. Detailed implementation of margin mask. The margin mask strategy discards unreliable trajectories until the prompt-level margin is restored. We implement its fine-grained, data-efficient variant as Greedy Margin Mask, which removes the single most adversarial trajectory in each iteration rather than discarding the entire group. Specifically, given the rollout group {τi }iG=1 of prompt q with trajectory-level returns { GOPD (q, τi )}iG=1 , we sort the positives in ascending order of GOPD (so the worst correct trajectory comes first) and the negatives in descending order 3

Algorithm 1 Greedy Margin Mask Inputs: Prompt q with rollout group {τi }iG=1 , outcome rewards { Ri }iG=1 with Ri ∈ {0, 1}, min retention ratio ρ, trajectory-level distillation returns { GOPD (q, τi )}iG=1 , target margin δ, mode ∈ {MinMax, Mean}. Output: Keep-mask {k i }iG=1 ∈ {0, 1}G ▷ ki = 1 means “keep trajectory τi ” and ki = 0 means “drop it”. Notation: For any two subsets A ⊆ S+ (q) and B ⊆ S− (q), we define the prompt-level margin M ARGIN ( A, B; MinMax) = minGOPD (q, τ ) − maxGOPD (q, τ ), τ∈A

τ ∈B

M ARGIN ( A, B; Mean) = meanGOPD (q, τ ) − meanGOPD (q, τ ), τ∈A

τ ∈B

1: function G REEDY M ARGIN M ASK(q, {τi , Ri , GOPD (q, τi )}iG=1 , δ, ρ, mode) 2: ▷ Step 1: split the group by outcome correctness. 3: S+ (q) ← {τi | Ri = 1}, S− (q) ← {τi | Ri = 0} 4: N+ ← |S+ (q)|, N− ← |S− (q)| 5: k i ← 1, ∀i = 1, . . . , G ▷ initialize: keep all trajectories 6: if N+ = 0 or N− = 0 then 7: return {k i }iG=1 ▷ ordering is not defined; no masking 8: end if 9: 10: ▷ Step 2: sort each side so that the most ordering-violating trajectory is at the front. 11: L+ (q) ← sort S+ (q) by GOPD (q, ·) ascending ▷ L+ (q)[1] = correct trajectory with lowest return 12: L− (q) ← sort S− (q) by GOPD (q, ·) descending ▷ L− (q)[1] = incorrect trajectory with highest return 13: 14: ▷ Step 3: iteratively drop the trajectory whose removal increases the margin the most. 15: while M ARGIN ( L+ (q), L− (q); mode) < δ do 16: if | L+ (q)| ≤ ⌈ρN+ ⌉ and | L− (q)| ≤ ⌈ρN− ⌉ then 17: break ▷ minimum retention ratio reached on both sides 18: end if 19: 20: ▷ Margin gain when the worst correct trajectory L+ (q)[1] is dropped. 21: ∆+ ← M ARGIN ( L+ (q)\{ L+ (q)[1]}, L− (q); mode) − M ARGIN ( L+ (q), L− (q); mode) 22: ▷ Margin gain when the best incorrect trajectory L− (q)[1] is dropped. 23: ∆− ← M ARGIN ( L+ (q), L− (q)\{ L− (q)[1]}; mode) − M ARGIN ( L+ (q), L− (q); mode) 24: if max(∆+ , ∆− ) ≤ 0 then 25: break ▷ no single removal can further improve the margin 26: end if 27: if ∆+ > ∆− and | L+ (q)| > ⌈ρN+ ⌉ then 28: τdrop ← P OP F RONT ( L+ (q)) ▷ greedy drop on the positive side 29: else 30: τdrop ← P OP F RONT ( L− (q)) ▷ greedy drop on the negative side 31: end if 32: k idx(τdrop ) ← 0 ▷ exclude this trajectory from the subsequent gradient update 33: end while 34: return {k i }iG=1 35: end function

(so the best incorrect trajectory comes first). At each iteration, we compute the margin improvement obtained by removing the front of each sorted list and greedily dropping the side that yields the larger improvement. The iteration terminates once (i) the target margin m(q) ≥ δ is satisfied, (ii) no further beneficial removal exists, or (iii) a minimum retention ratio ρ ∈ (0, 1) is reached to prevent excessive data loss. The masked trajectories are excluded from the eOPD (q, τi ) = k i · GOPD (q, τi ), where subsequent gradient update by setting their trajectory-level return to zero, i.e., G k i ∈ {0, 1} is the keep mask. In distributed training, the trajectory-level statistics are aggregated across all ranks via AllReduce so that the masking is deterministic and consistent across devices. The procedure is in algorithm 1. Detailed implementation of margin shift. The margin shift strategy applies a minimal additive correction to the trajectory-level returns so that the margin exactly meets the target δ, rather than discarding any sample. Given the rollout group {τi }iG=1 of prompt q, we first compute the current margin m(q) with the chosen mode (Mean by default). If m(q) < δ, we define the required shift as λ(q) = δ − m(q) > 0 and distribute it across trajectories in one of three eOPD (q, τ ) = GOPD (q, τ )+ λ(q)1{r (q, τ ) = 1}, directions: (i) Lift: add λ(q) to every positive trajectory, i.e., G which matches Eq. (11) in the main text; (ii) Suppress: subtract λ(q) from every negative trajectory, i.e.,

Algorithm 2 Margin Shift Inputs: Prompt q with rollout group {τi }iG=1 , outcome rewards { Ri }iG=1 with Ri ∈ {0, 1}, trajectory-level distillation returns { GOPD (q, τi )}iG=1 , target margin δ, mode ∈ {MinMax, Mean}, direction ∈ {Lift, Suppress, Spread}. eOPD (q, τi )}G Output: Calibrated trajectory-level returns { G i =1 1: function M ARGIN S HIFT(q, {τi , Ri , GOPD (q, τi )}iG=1 , δ, mode, direction) 2: ▷ Step 1: split the group by outcome correctness. 3: S+ (q) ← {τi | Ri = 1}, S− (q) ← {τi | Ri = 0} 4: if S+ (q) = ∅ or S− (q) = ∅ then eOPD (q, τi ) ← GOPD (q, τi )}G 5: return { G i =1 6: end if 7: 8: ▷ Step 2: summarize each side and compute the prompt-level margin m(q). 9: if mode = MinMax then 10: G+ (q) ← min GOPD (q, τ ) τ ∈ S+ ( q )

▷ ordering is not defined

▷ worst-scoring correct trajectory

11:

G− (q) ← max GOPD (q, τ )

▷ best-scoring incorrect trajectory

12: 13:

else G+ (q) ← mean GOPD (q, τ )

▷ average correct score

14:

G− (q) ← mean GOPD (q, τ )

▷ average incorrect score

τ ∈ S− ( q )

τ ∈ S+ ( q )

τ ∈ S− ( q )

15: end if 16: m(q) ← G+ (q) − G− (q) 17: 18: ▷ Step 3: additive correction when the margin is below the target. eOPD (q, τi ) ← GOPD (q, τi ), ∀i = 1, . . . , G 19: G 20: if m(q) < δ then 21: λ(q) ← δ − m(q) 22: if direction = Lift then eOPD (q, τ ) += λ(q), ∀τ ∈ S+ (q) 23: G 24: else if direction = Suppress then eOPD (q, τ ) −= λ(q), ∀τ ∈ S− (q) 25: G 26: else eOPD (q, τ ) += λ(q)/2, ∀τ ∈ S+ (q) 27: G eOPD (q, τ ) −= λ(q)/2, ∀τ ∈ S− (q) 28: G 29: end if 30: end if eOPD (q, τi )}G 31: return { G i =1 32: end function

▷ start from the uncalibrated returns ▷ amount by which the margin falls short of δ ▷ pull all correct trajectories up ▷ push all incorrect trajectories down ▷ split: half up on the positive side, . . . ▷ . . . and half down on the negative side

eOPD (q, τ ) = GOPD (q, τ )− λ(q)1{r (q, τ ) = 0}; and (iii) Spread: split the correction symmetrically, adding G λ(q)/2 to positives and subtracting λ(q)/2 from negatives. All three variants (a) preserve the relative ordering within S+ (q) and within S− (q) respectively, and (b) guarantee that the calibrated margin equals δ, i.e., eOPD − maxτ ∈S G e minτ ∈S+ G − OPD = δ. In distributed training, the aggregation of trajectory-level statistics and the computation of λ(q) are done via AllReduce to ensure consistency across devices. The procedure is in algorithm 2.

Training Details

In this section, we present details related to training, including the training setup (§B.1), the training datasets (§B.2), the training reward acquisition (§B.3), the training pseudocode (§B.4), the training dynamics (§B.5), and the training complexity analysis (§B.6). These details are provided to enhance the reproducibility of Uni-OPD.

B.1

Training Setup

To support multi-teacher OPD for both LLMs and MLLMs, we build Uni-OPD upon a widely used training framework, Miles2 . Specifically, we use Megatron-LM3 (Shoeybi et al., 2019) as the training backend and SGLang4 as the rollout inference engine. For teacher models, we deploy them as independent SGLang services that can be accessed via HTTP from arbitrary locations to obtain token-level rewards, enabling flexible teacher extensions and scalable multi-teacher integration. Each teacher is served behind a pool of SGLang endpoints with client-side shuffled round-robin load balancing, and a lightweight task-to-teacher routing table dispatches every prompt to the teacher best matched to its domain (e.g., math reasoning or code generation), so that new teachers or new tasks can be plugged in by simply extending the registry without touching the training loop. Because each teacher only needs to expose its prefill-time input token logprobs, no gradient, KV cache, or parameter is shared with the student, which keeps teachers fully stateless and decouples their deployment from the trainer. As a result, teacher scoring overlaps with student generation and contributes negligible overhead to the overall training throughput. General training hyperparameters. All general training settings, including the batch size, rollout numbers, learning rate schedule, optimizer choice, and so on, are identical to those used in ExOPD5 (Yang et al., 2026b), ensuring a fair and controlled comparison. The prompts used for training are provided in prompt 1. Training Prompt Template

Math Reasoning <|im start|>user {question} Please reason step by step, and put your final answer within \boxed{}.<|im end|> <|im start|>assistant

Code Reasoning <|im start|>user {question} Write Python code to solve the problem. Present the code in ‘‘‘python Your code ‘‘‘ at the end. You need to think first then write the Python code.<|im end|> <|im start|>assistant

Multimodal Math Reasoning <|im start|>user <image> {question} Please solve the problem step by step and put your answer in one \boxed{}.<|im end|> <|im start|>assistant

RL training setup. Teacher models are trained using reinforcement learning (RL). Detailed training settings of the teacher models are provided in Table B.1. OPD training setup. For OPD, we inherit most hyperparameters (e.g., learning rate, optimizer, and sequence lengths) from the teacher RL setup in Table B.1, so that the student is trained under the same optimization regime as its teachers. The OPD-specific entries, including the training batch size, the number of on-policy samples per prompt, the online correctness-aware filter, and the margin calibration configuration, are summarized in Table B.2. Concretely, we use a training batch size of 64 and sample n = 16 on-policy rollouts per prompt, which we find provides a good trade-off between return estimation quality and computational efficiency (see the ablation in Table D.6). The online correctness-aware filter is applied in sample filter mode with a target correct-to-incorrect ratio of 1:1 within each training batch, following §A.2. For margin calibration (§A.4), we adopt group-level mean normalization in both domains, while the shift direction and target margin are tuned per domain: for the textual domain, we use Spread with δ = 0.4, and for the multimodal domain, we use Lift with δ = 0. 2 https://github.com/radixark/miles 3 https://github.com/nvidia/megatron-lm 4 https://github.com/sgl-project/sglang 5 https://github.com/RUCBM/G-OPD

Table B.1: Teacher model training configuration with GRPO. Group

Setting

Value

Base model Model Training steps

LLM: Math, Code: Qwen3-4B MLLM: Math, Logic, Document: Qwen3-VL-4B-Inst. LLM: Math, Code: 500, 300 MLLM: Math, Logic, Document: 300, 300, 160

Optimization

Tensor Parallelism (TP) Micro batch size / GPU Training batch size Learning rate Warm-up steps LR schedule ZeRO stage Optimizer

2 1 128 1 × 10−6 0 Constant 3 Adam

Sequence

Max prompt length Max response length

2048 16384

RL Algorithm

Advantage estimator GRPO clip ratio Use KL in reward KL loss coefficient Entropy coefficient

GRPO 0.2 False 0.0 0.0

Rollout

Samples per prompt (n) Temperature Top-p Top-k

8 1.0 0.95 50

Hardware

GPUs

16× NVIDIA H20

Table B.2: OPD training configuration. Most hyperparameters inherit from the teacher RL setup in Table B.1; only the entries that differ between OPD and RL are listed here.

B.2

Group

Setting

Textual

Multimodal

Optimization

Training batch size Samples per prompt (n)

64 16

Online filter

Filter mode Correct/Incorrect ratio

Sample filter 1:1

Margin calibration

Scope Mode Direction Target margin δ

Group Mean S PREAD 0.4

Group Mean L IFT 0

Training Data

Textual math reasoning data. We use a subset of the DeepMath dataset (He et al., 2025b) with difficulty level ≥ 6 to train mathematical reasoning ability, comprising 57.0K samples. Textual code generation data. We use the Code subset of the Eurus-2-RL-Data dataset (Cui et al., 2025) with 25.3K samples to train code generation ability. Multimodal math reasoning data. For multimodal math reasoning tasks, we draw 14.8K samples from the OpenMMReasoner-RL dataset6 , covering MMK12, WeMath-Standard, and WeMath-Pro subsets. Multimodal logic reasoning data. We collect 14.8K samples spanning AlgoPuzzle, PuzzleVQA, and ThinkLiteVL-Hard subsets from the OpenMMReasoner-RL-74K dataset. Multimodal document understanding data. We include 14.6K document understanding samples, obtained by 15% sampling from the TQA subset of OpenMMReasoner with ChartQA (Masry et al., 2022) and InfographicsVQA (Mathew et al., 2022) training sets. 6 https://huggingface.co/datasets/OpenMMReasoner/OpenMMReasoner-RL-74K

OPD

12000

Average Response Length

Response Correct Ratio (%)

Uni-OPD

0.42

10000

0.40

Entropy

0.38

0.36

0.34 0.32

0.30

30 0

0.28

8000 6000 4000 2000 0

Optimization Step Optimization Step Optimization Step Figure B.1: Training dynamics of OPD and Uni-OPD for multi-teacher distillation. We track three indicators along the optimization trajectory: response correctness (%), policy entropy, and average response length. B.3

Training Reward Acquisition

In this section, we describe how training rewards are obtained for different data sources. For textual math reasoning tasks, we use the rule-based verifier provided by DeepMath7 to determine whether generated answers are correct. For textual code generation tasks, we use the rule-based verifier provided by PRIME8 to evaluate the correctness of generated code. For multimodal tasks, we use the verifier released by OpenMMReasoner9 to assess whether generated answers are correct. B.4

Training Pseudocode

The full training procedure of Uni-OPD is summarized in algorithm 3. In brief, the procedure (1) samples a prompt batch with offline difficulty-aware balancing (§A.1); (2) rolls out G trajectories per prompt and computes the trajectory-level distillation return GOPD from teacher–student log-probability differences (Eq. (5)); (3) applies online correctness-aware balancing across the batch (§A.2); (4) calibrates GOPD via the prompt-level margin m(q) (Eq. (9)) using either G REEDY M ARGIN M ASK (algorithm 1) or M ARGIN S HIFT (algorithm 2); and (5) broadcasts the calibrated returns to token-level advantages and updates the student πθ. B.5

Training Dynamics

Fig. B.1 demonstrates the effectiveness of Uni-OPD along three complementary axes. From a comparable starting point (∼35% correct, entropy ∼0.33, length ∼1.6k), Uni-OPD converges to a substantially higher response-correct ratio than OPD, peaking at 80.6% versus 75.2% and averaging 75.5% over the final 10 steps versus OPD’s 69.1% (+6.4 absolute points). Crucially, this accuracy gain is not obtained by sacrificing exploration: policy entropy rises mildly under both methods, with Uni-OPD maintaining a marginally higher steady-state value, ruling out the entropy-collapse failure mode that typically plagues teacher-driven training. Meanwhile, the average response length grows from ∼1.6k to ∼8k tokens, with Uni-OPD producing slightly longer outputs than OPD (7.8k vs. 7.1k), indicating that the model learns to perform more elaborate reasoning rather than collapsing to short, high-confidence shortcuts. Together, these trends suggest that Uni-OPD provides a consistent improvement over OPD without adverse effects on exploration or response length. B.6

Training Complexity

Beyond vanilla OPD, Uni-OPD introduces lightweight components on top of the standard per-iteration cost during training: online correctness-aware data balancing (per batch; §A.2), and outcome-guided margin calibration via Margin Mask / Shift (per prompt; §A.4). Let B be the training batch size (number of prompts) and G be the rollout group size. The online balancing only resamples prompts based on their precomputed { Ri }, costing O( BG ) per iteration. Margin Mask and Margin Shift both operate on the G trajectory-level returns within each prompt group: Margin Shift is O( G ) per prompt, while the greedy variant of Margin Mask is at most O( G2 ) per prompt in the worst case (typically G ≤ 16 in our setup). In contrast, the dominant per-iteration cost of OPD comes from two stages whose complexity scales linearly with the total number of rollout tokens Ttok = ∑iBG =1 | τi | and cubically with the hidden size d: (i) sampling BG on-policy rollouts from the student, and (ii) running a teacher prefill pass over these rollouts to obtain tokenlevel log-probabilities, each of order O( Ttok d2 ) for transformer forward passes. Typical numbers in our setup (Bs = 64, N = 16, average length ∼8k) give Ttok on the order of 8 × 106 tokens per iteration. All of Uni-OPD’s 7 https://github.com/zwhe99/DeepMath 8 https://github.com/PRIME-RL/PRIME 9 https://github.com/EvolvingLMMs-Lab/OpenMMReasoner

Algorithm 3 Uni-OPD: Outcome-guided Policy Distillation with Margin Calibration Input: Teacher πT , student πθ , dataset D , group size G, target margin δ, calibration mode ∈ { M ASK, S HIFT }, learning rate η. Output: Updated student parameters θ. 1: function U NI OPD(πT , πθ , D , G, δ, mode, η) 2: ▷ Offline difficulty-aware data balancing (once before training; see §A.1). 3: Sample a prompt batch B ⊂ D with rebalanced difficulty distribution 4: 5: while not converged do 6: ▷ Rollout and token-level scoring (per prompt). 7: for all prompt q ∈ B do 8: Rollout G trajectories {τi }iG=1 ∼ πθ (· | q) 9: for i = 1, . . . , G do 10: Obtain outcome reward Ri = r (q, τi ) ∈ {0, 1} 11: for all token ot ∈ τi do 12: rtOPD (τi ) ← log πT (ot | q, o<t ) − log πθ (ot | q, o<t ) ▷ token-level OPD reward 13: end for 14: ▷ Trajectory-level distillation return (Eq. (5)). 1 |τ | 15: GOPD (q, τi ) ← ∑ i rOPD (τi ) |τi | t=1 t 16: end for 17: Partition: S+ (q) ← {τi | Ri = 1}, S− (q) ← {τi | Ri = 0} ▷ correct / incorrect trajectory sets 18: end for 19: 20: ▷ Online correctness-aware data balancing (across the batch; see §A.2). 21: B ← O NLINE C ORRECTNESS AWARE DATA BALANCING B , { Ri }q,i 22: 23: ▷ Outcome-guided margin calibration (per prompt; Eqs. (9) and (10)). 24: for all prompt q ∈ B do 25: Compute prompt-level margin m(q) = minτ ∈S+ (q) GOPD (q, τ ) − maxτ ∈S− (q) GOPD (q, τ ) 26: if mode = M ASK then 27: {k q,i }iG=1 ← G REEDY M ARGIN M ASK (q, {τi , Ri , GOPD (q, τi )}iG=1 , δ, ρ, mode) ▷ algorithm 1 eOPD (q, τi ) ← k q,i · GOPD (q, τi ), ∀i = 1, . . . , G 28: G ▷ zero out masked trajectories 29: else eOPD (q, τi )}G ← M ARGIN S HIFT (q, {τi , Ri , GOPD (q, τi )}G , δ, mode, direction) 30: {G ▷ algorithm 2 i =1 i =1 31: end if 32: end for 33: 34: ▷ Token-level broadcasting and policy update. 35: for all prompt q ∈ B , rollout i = 1, . . . , G, token ot ∈ τi do et (q, τi ) ← G eOPD (q, τi ) 36: A ▷ broadcast calibrated trajectory return to all tokens 37: end for h i et (q, τi ) log πθ (ot | q, o<t ) 38: L(θ) = − Eq,τi ,t A 39: θ ← θ − η ∇θ L(θ) 40: end while 41: return θ 42: end function

▷ one gradient step on the student

additional computation scales with the number of trajectories rather than the number of tokens, involves only scalar comparisons and additions, and is therefore several orders of magnitude cheaper than the rollout and teacher-scoring stages that OPD already pays. In practice, we observe that enabling all three components adds less than 1% wall-clock overhead per iteration relative to vanilla OPD, while delivering the accuracy improvements reported in §B.5 and the main experiments. Thus Uni-OPD offers a favorable accuracy–compute trade-off: a negligible compute surcharge in exchange for consistently better final performance.

Evaluation Details

C.1

Evaluation Benchmarks

We evaluate our Uni-OPD on a comprehensive benchmark suite spanning textual and multimodal capabilities, organized along five capability axes: •

Textual Math Reasoning: - AIME (2024/2025) (AI-MO, 2024): A prestigious high school mathematics competition featuring challenging 9

problems that test deep mathematical reasoning. - HMMT25 (Feb & Nov) (Balunović et al., 2025): Contest-level benchmarks designed to rigorously evaluate advanced reasoning across algebra, geometry, combinatorics, and other domains. • Textual Code Generation: - HumanEval+ (Liu et al., 2023b): A set of 164 hand-written programming problems evaluating functional correctness, covering language understanding, reasoning, algorithms, and basic mathematics. - MBPP+ (Liu et al., 2023b): A collection of ∼1,000 crowd-sourced Python tasks targeting entry-level programming skills, including fundamentals and standard library usage. - LiveCodeBench (v6) (Jain et al., 2024): A contamination-free and continuously updated benchmark assessing not only code generation but also execution, self-repair, and test prediction. • Multimodal Math Reasoning: - MathVision (Wang et al., 2024a): A curated dataset of 3,040 visual problems sourced from real competitions, spanning 16 disciplines and multiple difficulty levels for evaluating multimodal mathematical reasoning. - DynaMath (Zou et al., 2024): A dynamically generated benchmark based on 501 seed question generators, enabling diverse and scalable evaluation through multiple sampled variants. - WeMath (Qiao et al., 2025): A large-scale benchmark with 6.5K visual math problems organized into 67 hierarchical knowledge concepts, designed to analyze problem-solving processes. • Multimodal Logic Reasoning: - LogicVista (Xiao et al., 2024): A benchmark for evaluating multimodal logical reasoning across 5 task types and 9 capabilities using annotated multiple-choice questions with human reasoning. - VisuLogic (Xu et al., 2025b): A challenging visual reasoning benchmark focusing on reasoning directly from visual inputs, with tasks that are difficult to express textually and expose gaps in current MLLMs. • Document Understanding: - AI2D (Kembhavi et al., 2016): A diagram understanding benchmark focusing on parsing diagram structure and reasoning over relationships between components via question answering. - ChartQA (Masry et al., 2022): A benchmark for question answering over charts, requiring complex visual and logical reasoning over both chart structure and underlying data. - DocVQA (Mathew et al., 2021): A large-scale document visual question answering dataset over document images, emphasizing structural and textual understanding. - InfoVQA (Mathew et al., 2022): A benchmark on infographic understanding that requires joint reasoning over layout, text, and visual elements with an emphasis on multi-step reasoning. C.2

Evaluation Setup

Textual evaluations. For all textual evaluations, we use a sampling temperature of 1.0, top-p of 1.0, a maximum generation length of 16,384 tokens, and a fixed random seed of 42. We use the vLLM inference engine to perform sampling. For math reasoning benchmarks, we sample N = 32 solutions per problem, while for code generation benchmarks, we sample N = 4 solutions per problem. For evaluation, we adopt Math-Verify10 as a rulebased verifier for math reasoning tasks. For code generation, we use the EvalPlus11 and LiveCodeBench12 frameworks to assess functional correctness. For all main results, we report the average accuracy across sampled solutions (i.e., pass@1), and compute pass@k as: pass@k = 1 −

( Nk−c)

, ( Nk ) where N is the number of samples and c is the number of correct solutions.

(15)

Multimodal evaluations. For multimodal evaluations, we adopt the widely used LMMs-Eval13 (Zhang et al., 2025a) framework and strictly follow its official evaluation protocols and configurations. The reported evaluation metrics are summarized in Table C.1.

Further Evaluations

D.1

More Evaluation Results

Single-teacher and multi-teacher distillation on LLMs and MLLMs. We further evaluate Uni-OPD under 10 https://github.com/huggingface/Math-Verify 11 https://github.com/evalplus/evalplus 12 https://github.com/livecodebench/livecodebench 13 https://github.com/evolvinglmms-lab/lmms-eval

Table C.1: Reported evaluation metrics for different benchmark datasets. We summarize the primary metrics used for performance reporting across math, logic, and document understanding tasks. Tasks

Filter

N-Shot

Reported Metric

Multimodal Math Reasoning

MathVision Test DynaMath Reasoning WeMath TestMini Reasoning

none none none

0 0 0

mathvision standard eval dynamath average acc score

Multimodal Logic Reasoning

LogicVista Reasoning LogicVista Reasoning VisuLogic

none none none

0 0 0

acc score format score visulogic acc

Document Understanding

AI2D ChartQA DocVQA Val InfoVQA Val

flexible-extract none none none

0 0 0 0

exact match relaxed human split anls anls

Category

Table D.1: Performance of Qwen3-1.7B Student under math reasoning and code generation benchmarks. Teacher models (i.e., Qwen3-4B-Math-RL and Qwen3-4B-Code-RL) are developed through domain-specific RL. The performance of teacher models is denoted by the “RL” type. Math Reasoning Method

Code Generation

AIME 2024

AIME 2025

HMMT 25 Feb.

HMMT 25 Nov.

Avg.

Human Eval+

MBPP+

LCB

Avg.

Student Teacher

13.9 60.1

11.1 55.1

5.6 32.5

4.9 38.5

8.9 46.6

61.9 85.2

53.4 69.8

11.9 26.6

42.4 60.5

OPD Uni-OPD

42.3 42.6

35.4 35.1

Single–Teacher Distillation 18.4 19.1 28.8 71.8 20.8 20.9 29.9 73.0

58.2 60.0

26.7 28.1

52.5 53.7

OPD Uni-OPD

40.3 44.0

32.4 35.1

Multi–Teacher Distillation 20.0 20.3 28.3 73.2 19.5 19.8 29.6 72.9

59.1 60.5

25.7 28.0

52.7 53.8

Table D.2: Performance of Qwen3-VL-2B-Instruct Student under math reasoning, logic reasoning, and document understanding benchmarks. Teacher models (i.e., Qwen3-VL-4B-Instruct-Math-RL, Qwen3-VL-4BInstruct-Logic-RL and Qwen3-VL-4B-Instruct-Document-RL) are developed through domain-specific RL. Avg. denotes the mean score within each category. Math Reasoning Method Student Teacher

Logic Reasoning

Document Understanding

Math Dyna We LogicVista LogicVista Visu Chart Doc Info Avg. Avg. AI2D Avg. Vision Math Math Accuracy Format Logic QA VQA VQA 11.1 47.2

49.1 48.6 36.3 65.3 79.5 64.0

32.4 52.5

59.1 73.8

6.4 32.6 73.4 27.4 51.2 82.5

66.1 76.4

92.8 72.4 76.2 95.1 81.6 83.9

OPD 24.4 Uni-OPD 25.5

54.5 64.8 47.9 55.2 65.0 48.6

Single–Teacher Distillation 35.3 61.6 26.8 41.2 76.1 36.8 65.2 27.6 43.2 76.7

66.0 93.0 72.2 76.8 66.6 92.9 72.6 77.2

OPD 15.2 Uni-OPD 18.7

50.2 57.6 41.0 51.2 58.7 43.9

Multi–Teacher Distillation 38.0 65.2 27.2 43.4 76.2 42.0 69.8 27.0 46.3 76.0

66.1 92.9 72.5 76.9 66.5 93.0 72.6 77.0

both single-teacher and multi-teacher distillation settings on LLMs and MLLMs. As shown in Tables D.1and D.2, our method consistently outperforms the standard OPD baseline across all domains and settings. On the LLM student (i.e., Qwen3-1.7B), Uni-OPD improves the average scores on both math reasoning and code generation under single-teacher and multi-teacher distillation. On the MLLM student (i.e., Qwen3-VL-2B-Instruct), it delivers consistent gains across math reasoning, logic reasoning, and document understanding. Further, it narrows the gap to the teacher ensemble under multi-teacher distillation. Consistent improvements in smaller students provide strong empirical evidence for our dual-perspective approach, confirming that student exploration and teacher reliability are indeed the fundamental drivers of successful and reliable distillation. Cross-modal distillation on code generation and logic reasoning. Beyond the cross-modal distillation on math reasoning and code generation, we further conduct cross-modal distillation on code generation and logic reasoning. Specifically, we combine text-only code data with multimodal logic reasoning data, and jointly distill from two domain-specific teachers (Qwen3-VL-4B-Instruct-Code-RL and Qwen3-VL-4B-Instruct-Logic-RL) into a single 11

Table D.3: Performance of Qwen3-VL-4B-Instruct Student under code generation and logic reasoning benchmarks. Teacher models (i.e., Qwen3-VL-4B-Instruct-Code-RL and Qwen3-VL-4B-Instruct-Logic-RL) are developed through domain-specific RL. The performance of teacher models is denoted by the “RL” type. Code Generation Method

Logic Reasoning

Human Eval+

MBPP+

LCB

Avg.

LogicVista Accuracy

LogicVista Format

Visu Logic

Avg.

Student Teacher

76.8 82.2

70.0 70.5

37.0 40.1

61.3 64.3

49.9 52.5

66.4 73.8

25.1 27.4

47.0 51.2

OPD Uni-OPD

79.0 79.4

68.5 69.2

Multi–Teacher Distillation 39.6 62.4 50.0 41.4 63.3 52.0

69.3 73.8

27.3 28.0

48.9 51.3

Qwen3-VL-4B-Instruct student. As shown in Table D.3, Uni-OPD outperforms the standard OPD baseline on both the code generation and logic reasoning averages, with the largest gain on LCB (39.6 → 41.4) and LogicVista Accuracy (50.0 → 52.0). These results confirm that Uni-OPD effectively integrates heterogeneous text-only and multimodal data under a single training run, further supporting its applicability to cross-modal distillation. D.2

Downstream Task Evaluation

Table D.4: General downstream task performance. Evaluation on 8 general benchmarks to ensure general-purpose capabilities are maintained after OPD. Model

MMLU ARC HellaSwag TruthfulQA Winogrande GSM8K CommonsenseQA IFEval Avg.

Qwen3-4B

68.3

80.7

68.4

54.8

66.6

84.2

75.8

88.9

73.5

Math Teacher Code Teacher OPD Uni-OPD

68.4 68.3 68.3 68.3

80.8 80.2 80.3 80.3

68.5 68.3 68.4 68.3

54.3 54.8 54.6 54.6

66.0 65.7 66.5 66.0

86.7 85.8 88.6 88.6

75.4 75.7 75.5 75.7

89.2 89.7 89.2 89.2

73.7 73.6 73.9 73.9

Evaluation on general capabilities. To assess the impact of OPD on general downstream performance of the policy model, we evaluate the models on a diverse set of benchmarks from the Hugging Face Open LLM Leaderboard (Beeching et al., 2023) following recent studies (Peng et al., 2026; Meng et al., 2024). Specifically, we report results on MMLU (Hendrycks et al., 2020), ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), TruthfulQA (Lin et al., 2022), Winogrande (Levesque et al., 2012), GSM8K (Cobbe et al., 2021), CommonsenseQA (Talmor et al., 2019), and IFEval (Zhou et al., 2023b). We strictly follow the standard evaluation protocols provided by the lm-evaluation-harness system14 . For IFEval, we report the inst level loose acc. The results are presented in Table D.4. Overall, Uni-OPD not only outperforms OPD and domain-specific teachers on math reasoning and code generation benchmarks demonstrated in the main text, but also retains strong performance across a wide range of downstream tasks. These results suggest that OPD serves as a general and effective framework for improving LLM performance beyond task-specific settings. D.3

Further Ablation

Table D.5: Effectiveness validation of margin shift across different hyperparameters. We conduct single-teacher distillation experiments with a Qwen3-4B Student using individual math and code teachers. Math Reasoning Configuration

Code Generation

AIME AIME 2024 2025

HMMT HMMT 25 Feb. 25 Nov.

Avg.

Human MBPP+ Eval+

LCB

Avg.

OPD (no shift) Global + Mean + Lift Global + MinMax + Lift Group + MinMax + Spread

57.9 61.8 62.4 63.4

52.4 55.2 57.3 56.7

30.2 34.8 32.2 33.4

37.8 39.4 38.2 39.0

44.6 47.8 47.5 48.1

82.6 85.7 85.8 86.9

68.8 71.4 71.8 70.6

25.7 25.7 26.7 26.7

59.0 60.9 61.4 61.4

Group + Mean + Spread (ours)

62.7

56.3

34.4

39.2

48.2

88.3

72.3

26.7

62.4

Hyperparameter analysis for margin shift. As shown in Table D.5, we compare four variants of margin shift against the OPD baseline across math reasoning and code generation benchmarks. The shift scope (Global vs. 14 https://github.com/EleutherAI/lm-evaluation-harness

Group), normalization mode (Mean vs. MinMax), and shift direction (Lift vs. Spread) are ablated systematically. All shift variants consistently outperform the vanilla OPD baseline, demonstrating the general effectiveness of margin shift. Among the variants, Group + Mean + Spread achieves the best average performance on both code generation (62.4) and math reasoning (48.2), indicating that group-level mean normalization with bidirectional shifting provides a more calibrated return signal. Applying the shift to both correct and incorrect responses (Spread) proves beneficial over unidirectional shifting (Lift), and group-level statistics generalize better than global ones when reward distributions vary across prompts. Furthermore, we observe that MinMax-based normalization and global-scope statistics are susceptible to outlier return values, as extreme return values within a batch can distort the shift magnitude and destabilize training. In contrast, group-level mean normalization produces more robust and consistent return estimates, contributing to stable optimization throughout training. Hyperparameter analysis for rollout number n. As Table D.6: The effects of rollout number. The global shown in Table D.6, we ablate the rollout number n in batch size is fixed at n × bs = 1024 throughout. OPD while keeping the global batch size fixed at 1024 (i.e., AIME AIME HMMT HMMT n × bs = 1024), so that increasing n comes at the cost Method Avg. 2024 2025 25 Feb. 25 Nov. of a smaller per-step batch size bs. For the OPD baseline, 23.0 19.3 12.3 9.2 15.9 performance remains largely stable across all values of n Student (4B) (44.3–44.6 avg.), suggesting that the base method is relOPD atively insensitive to this trade-off. In contrast, OPD with n = 4, bs = 256 60.1 55.1 32.5 29.6 44.3 margin shift benefits notably from larger rollout groups: n = 8, bs = 128 59.8 52.9 29.6 35.8 44.5 30.2 37.8 44.6 average performance improves from 45.3 at n=4 to 48.2 n = 16, bs = 64 57.9 52.4 30.6 36.9 44.3 at n=16, as more responses per prompt yield more reli- n = 32, bs = 32 58.3 51.2 able relative return estimation for the margin-based calOPD + Margin shift ibration. We find that increasing n from 16 to 32 yields n = 4, bs = 256 57.9 52.4 33.2 37.8 45.3 comparable performance. Considering return estimation n = 8, bs = 128 62.5 55.4 31.9 39.2 47.3 34.4 39.2 48.2 quality, training stability, and computational efficiency, n = 16, bs = 64 62.7 56.3 n = 32, bs = 32 63.1 55.4 34.2 39.6 48.1 we therefore set n=16 as our default.

Related Work

E.1

Multimodal Large Language Models

Large Language Models (LLMs) have undergone rapid development in recent years (Touvron et al., 2023; Achiam et al., 2023; AI@Meta, 2024b; Hurst et al., 2024; Yang et al., 2024a; AI@Meta, 2024a; Yang et al., 2025; Brown et al., 2020; Team et al., 2024; Anthropic, 2023b;a; 2024; Liu et al., 2024a; Guo et al., 2025a; Li et al., 2025), significantly improving reasoning capabilities. Meanwhile, MLLMs have also seen substantial progress (Radford et al., 2021; Shao et al., 2024a; Wang et al., 2025; Tian et al., 2019; Liu et al., 2024e; Yang et al., 2024c; Peng et al., 2026; Team et al., 2025). Leveraging advances in LLMs, multimodal large language models (MLLMs) further integrate visual and textual representations through cross-modal learning, achieving strong multimodal understanding and generation capabilities. A key driver of this success lies in the combination of large-scale self-supervised pre-training on diverse corpora and subsequent high-quality supervised fine-tuning (SFT), which enables LLMs and MLLMs to exhibit strong generalization and emergent capabilities in real-world tasks (Wang et al., 2024b; Bai et al., 2023; 2025b; Liu et al., 2023a; 2024b;c; Dai et al., 2023; OpenAI, 2023; Zhu et al., 2023; Qu et al., 2025; Yang et al., 2023b; Zhong et al., 2024; Yang et al., 2023a; 2024b; Lai et al., 2024; Peng et al., 2025; Hou et al., 2026). Building upon these foundations, KD has emerged as an important paradigm for transferring sophisticated reasoning capabilities from teacher models to more efficient students. Among various distillation strategies, OPD has recently emerged as a mainstream post-training paradigm for both LLMs and MLLMs. In the on-policy setting, however, the effectiveness of distillation is tied to both the quality of student exploration and the reliability of teacher feedback. In this work, we present a dual-perspective optimization strategy from both the student and teacher sides to improve data suitability and training stability in OPD. E.2

Reinforcement Learning

By optimizing trajectories sampled from the current policy, on-policy RL alleviates distribution mismatch and is often instantiated with verifiable or outcome-based rewards in reasoning tasks. Notable methods include GRPO (Shao et al., 2024b) for critic-free grouped optimization and GSPO (Zheng et al., 2025) for sequence-level stable optimization. Recently, some works have also combined RLVR with OPD, such as Self-Distilled RLVR (Yang et al., 2026a) and OpenClaw-RL (Wang et al., 2026). In our work, we use GRPO to obtain stronger domain-specific teachers and use the corresponding reward models as global guidance for return calibration in OPD.

E.3

On-Policy Distillation

Early OPD work, such as MiniLLM (Gu et al., 2023) and GKD (Agarwal et al., 2024), establishes the basic paradigm of using teacher feedback on student-generated trajectories under a reverse KL objective. Recent studies further broaden this paradigm from multiple perspectives. In self-distillation methods, OPSD (Zhao et al., 2026b) uses privileged information; SDFT (Shenfeld et al., 2026) allows the student to absorb knowledge from retrieved demonstrations while reducing forgetting. SDPO (Hübotter et al., 2026) treats the current model itself as a selfteacher; OPCD (Ye et al., 2026) internalizes context knowledge into model parameters by minimizing reverse KL between the student and a context-conditioned teacher on the student’s trajectories. Regarding teacher access, black-box OPD (Ye et al., 2025) introduces a discriminator-guided framework that does not require teacher logits. Several works also focus on improving optimization and efficiency. ExOPD (Yang et al., 2026b) reformulates OPD as weighted dense RL; Fast and Effective OPD (Zhang et al., 2026a) improves efficiency through prefixonly distillation; KDFlow (Zhang et al., 2026b) provides an extensible distillation framework supporting both off-policy and on-policy training; MiMo-V2-Flash (Xiao et al., 2026) introduces multi-teacher OPD, enabling effective capability merging across domains. Li et al. (Li et al., 2026b) rethink OPD in terms of its phenomenology, mechanisms, and training recipes. Recently, OPD has also begun to extend beyond text-only settings. VOLD (Bousselham et al., 2025) transfers reasoning ability from text teachers to vision-language students through a two-stage pipeline that combines cold-start alignment with GRPO and OPD. Video-OPD (Li et al., 2026a) adapts OPD to long-video grounding and introduces a curriculum that filters unreliable teacher signals. X-OPD (Cao et al., 2026) further extends OPD to speech through cross-modal alignment. In contrast, our work focuses on developing a unified OPD framework with an open recipe for both LLMs and MLLMs.

Case Studies

We provide qualitative case studies of Uni-OPD, standard OPD, and the Student model across both LLM and MLLM benchmarks, covering textual math reasoning, code generation, logical reasoning, multimodal math reasoning, and chart understanding. We first revisit the math reasoning case in Fig. F.1, and provide a detailed output comparison of standard OPD and our Uni-OPD. Standard OPD assigns high returns to incorrect trajectories and low returns to correct ones. Furthermore, the code generation case in Fig. F.2 highlights Uni-OPD’s ability to balance algorithmic efficiency and code readability. These case studies demonstrate how our dual-perspective optimization–specifically by restoring order consistency through margin calibration–leads to more reliable and high-quality model outputs. Across the multimodal case studies in Fig. F.3–F.9, our observations reveal three consistent patterns: (a) Uni-OPD demonstrates superior efficiency on complex reasoning problems, producing more concise outputs while maintaining correctness, whereas the Student model and standard OPD frequently generate excessively long responses that are truncated before reaching a final answer; (b) Uni-OPD achieves higher correctness than the Student model, often succeeding on questions where the Student model fails; and (c) Our data-balancing strategies encourage exploration of informative student-generated states during training, improving Uni-OPD’s ability to tackle challenging visual and mathematical reasoning problems that the Student model cannot solve on its own.

Question: Find all functions $ f: \mathbf{N} \rightarrow \mathbf{N} $ such that for every $ n \in \mathbf{N} $, $ f(n) < f(n+1) $, and $ f(f(n)) = n + 2k $, where $ k \in \mathbf{N} $. Please reason step by step, and put your final answer within \boxed{}. 38.png Answer: \boxed{f(n) = n + k} OPD solution 1:

… (Omitted for brevity.) ### Step 5: Are there **other** solutions? Suppose $ f $ is not of the form $ f(n) = n + 2a $. Let’s suppose that it is **strictly increasing**, $ f(n) \in \mathbb{N} $, and such that $ f(f(n)) = n + 2k $, for some $ k \in \mathbb{N} $. Suppose that $ f(n) = n + a(n) $, where $ a(n) $ is a strictly increasing function (since $ f(n) = n + a(n) $ and $ f $ is strictly increasing, then $ a(n) $ must be strictly increasing). Then $ f(f(n)) = f(n + a(n)) = n + a(n) + a(n + a(n)) $ So: $$ f(f(n)) = n + a(n) + a(n + a(n)) = n + 2k \Rightarrow a(n) + a(n + a(n)) = 2k $$ This is only possible if $ a(n) $ is **constant**. Because otherwise the left-hand side is not consistent across $ n $. So unless $ a(n) = 2a $, a constant, the right-hand side is not of the form $ 2k $, but depends on both $ n $ and $ a(n) $. Therefore, the only way this can hold is if $ a(n) = 2a $, i.e., $ f(n) = n + 2a $.

Incorrect reasoning but with high return

Hence, **the only solutions** are functions of the form: $$ f(n) = n + 2a \quad \text{for } a \in \mathbb{N} $$ … (Omitted for brevity.) ### Final Answer: $$ \boxed{f(n) = n + 2a \text{ for some } a \in \mathbb{N}} $$ All such functions are of the form $ f(n) = n + 2a $, for some positive integer $ a $.<|im_end|>

OPD solution 2:

… (Omitted for brevity.) Let’s try a known function that satisfies this condition. ### Step 4: Try the function $ f(n) = n + 1 $ Then $ f(f(n)) = f(n+1) = n+2 $, so this function **satisfies**: $$ f(f(n)) = n + 2 $$ So this function is:

Correct reasoning but with low return

$$ f(n) = n + 1 $$ … (Omitted for brevity.)

### Final Answer: $$ \boxed{f(n) = n + k \text{ for some fixed } k \in \mathbb{N}} $$ That is, the **only** functions $ f: \mathbb{N} \to \mathbb{N} $ that are strictly increasing and satisfy $ f(f(n)) = n + 2k $ (for some fixed $ k \in \mathbb{N} $) are the linear functions of the form: $$ \boxed{f(n) = n + k} $$<|im_end|>

Uni-OPD :

… (Omitted for brevity.) Thus, the only solution that satisfies both the functional equation and the strictly increasing property is the linear function $ f(n) = n + k $, where $ k \in \mathbb{N} $.

--### **Conclusion** The only functions $ f: \mathbb{N} \to \mathbb{N} $ that are strictly increasing and satisfy the functional equation $ f(f(n)) = n + 2k $ for a fixed $ k \in \mathbb{N} $ are the functions of the form: $$ f(n) = n + k $$

Correct reasoning but with proper return

Therefore, the answer is: $$ \boxed{f(n) = n + k} $$ Where $ k \in \mathbb{N} $, but since the problem asks to "find all such functions," we can write the general form with $ k $ as an arbitrary natural number.

Figure F.1: Comparison of math reasoning outputs between OPD and Uni-OPD. In this case, standard OPD assigns high returns to incorrect reasoning trajectories and low returns to correct ones. In contrast, our Uni-OPD performs outcome-guided margin calibration to restore order consistency between correct and incorrect trajectories, yielding a reliable supervision signal that ultimately improves both efficiency and correctness of the generated solutions. On this question, we further measure pass@1 accuracy over 64 rollouts: standard OPD reaches 79.69%, while our Uni-OPD attains 82.81%, further validating the effectiveness of the proposed strategy.

Question: Write a python function to find the element of a list having maximum length.

38.png

assert Find_Max([['A'],['A','B'],['A','B','C']]) == ['A','B','C'] Student: def Find_Max(lst): if not lst: return []

Correct but without any comments (poor readability).

max_list = lst[0] for sublist in lst: if len(sublist) > len(max_list): max_list = sublist return max_list

OPD: def Find_Max(lst): # Find the maximum length of any sublist in the list max_length = max(len(sub) for sub in lst)

Correct with comments, but inefficient (two passes).

# Find all sublists that have this maximum length max_sublists = [sub for sub in lst if len(sub) == max_length] # Return the first one found (or any, but the test case expects the first one with max length) return max_sublists[0]

Uni-OPD: def Find_Max(lst): # Initialize the max_length with the length of the first element max_length = len(lst[0]) # Initialize the result with the first element max_element = lst[0] # Iterate through the list

Correct, well-commented, and efficient (single pass).

for element in lst: # Check if the current element has a longer length if len(element) > max_length: max_length = len(element) max_element = element return max_element

Figure F.2: Comparison of code generation for the Find Max task. While the Student model produces correct logic with limited readability, the OPD baseline introduces redundant computation (two passes) despite adding comments. Our Uni-OPD generates a superior solution that is both computationally efficient (single pass) and well-commented, demonstrating its effectiveness in aligning with complex task requirements.

Question: Which option from A-E should replace the question mark to follow the pattern present above the separating line? (select from A-E)

38.png

Answer: E

Student: To solve this, let’s analyze the sequence of shapes above the line:\n\nWe have 5 shapes in a row, each a diamond divided into two triangles (top and bottom). The pattern seems to involve the **color** and **position** of the shaded region.\n\nLet’s label the positions:\n\n1. First shape: **Top triangle black**, bottom triangle white.\n2. Second shape: **Entire diamond gray** (so both triangles shaded gray).\n3. Third shape: **Divided by a vertical line**, top triangle white, bottom triangle white — but wait, that doesn’t make sense unless it’s a different interpretation.\n\nWait — perhaps we should look at the **shading pattern** more carefully.\n\nActually, let’s consider the **shading** as a sequence:\n\n- Shape 1: Top half black, bottom half white.\n- Shape 2: Entire diamond gray (so both halves gray).\n- Shape 3: Diamond with a vertical line dividing it — top half white, bottom half white? That doesn’t match.\n\nWait — perhaps the **shading is applied to the top or bottom triangle**, and the pattern is **alternating** or **rotating**.\n\nLet’s look at the **color** of the shaded region:\n\n- Shape 1: Top triangle black.\n- Shape 2: Entire diamond gray — so both triangles gray.\n- Shape 3: Diamond with a vertical line — but the top triangle is white, bottom triangle is white? That doesn’t make sense unless it’s a different interpretation.\n\nWait — perhaps the **shading is applied to the top triangle only**, and the bottom triangle is always white? Let’s check:\n\nShape 1: Top triangle black, bottom white → ✅\nShape 2: Entire gray → so top gray, bottom gray → ✅\nShape 3: Top triangle white, bottom triangle white? That contradicts.\n\nWait — perhaps the **shading is applied to the bottom triangle**?\n\nShape 1: Top black, bottom white → bottom white? No.\n\nWait — let’s look at the **pattern of shading** as a sequence:\n\nShape 1: Top triangle black.\nShape 2: Entire gray — so both triangles gray.\nShape 3: Diamond with a vertical line — but the top triangle is white, bottom triangle is white? That doesn’t make sense.\n\n … (Omitted for brevity.) Wait — perhaps the **shading is applied to the top triangle**, and the **color changes** in a pattern.\n\nShape 1: Top triangle black.\nShape 2: Top triangle gray.\nShape 3: Top triangle white? But the shape is divided by a vertical line — maybe it’s not shaded at all? No, the shape is white.\n\nWait — perhaps the **shading is applied to the bottom triangle**?\n\nShape 1: Bottom triangle white — no shading.\nShape 2: Bottom triangle gray.\nShape 3: Bottom triangle white? No.\n\nThis is confusing.\n\nLet’s try a different approach: **look at the position of the shaded region**.\n\nShape 1: Shaded top triangle.\nShape 2: Shaded entire diamond (both triangles).\nShape 3: Shaded bottom triangle? But the bottom triangle is white — no.\n\nWait — perhaps the **shading is applied to the top triangle**, and the **color changes** in a pattern: black → gray → white → black → gray?\n\nShape 1: Top triangle black.\nShape 2: Entire gray — so both triangles gray.\nShape 3: Top triangle white? But the shape is divided by a vertical line — maybe it’s not shaded at all? No.\n\nWait — perhaps the **shading is applied to the bottom triangle**.\n\nShape 1: Bottom triangle white — no shading.\nShape 2: Bottom triangle gray.\nShape 3: Bottom triangle white? No.\n\nThis is not working.\n\nLet’s look at the **sequence of shapes** again:\n\nShape 1: Top triangle black, bottom white.\nShape 2: Entire gray.\nShape 3: Diamond with a vertical line — top triangle white, bottom triangle white? That doesn’t make sense.\n\nWait — perhaps the **shading is applied to the top triangle**, and the **color changes** in a pattern: black → gray → white → black → gray?\n\nShape 1: Top triangle black.\nShape 2: Top triangle gray.\nShape 3: Top triangle white? But the shape is divided by a vertical line — maybe it’s not shaded at all? No.\n\nWait — perhaps the **shading is applied to the bottom But the shape is divided by a vertical line — maybe it’s not shaded at all? No.\n\nWait — perhaps the **shading is applied to the bottom triangle**.\n\nShape 1: Bottom triangle white — no shading.\nShape 2: Bottom triangle gray.\nShape 3: Bottom triangle white? No.\n\nThis is not working.\n\nLet’s try to look at the **sequence of shapes**

OPD: To determine the correct option, let’s analyze the pattern in the sequence of shapes above the separating line.\n\nWe have five shapes in a row:\n\n1. A diamond with a black top half and white bottom half.\n2. A diamond with a gray top half and white bottom half.\n3. A diamond with a white top half and black bottom half.\n4. A diamond with a black top half and white bottom half (same as shape 1).\n5. A diamond with a gray top half and white bottom half (same as shape 2).\n\nWait — this doesn’t seem to follow a clear pattern. Let’s look again.\n\nActually, let’s consider the **position of the shaded area** and **its color**.\n\nShape 1: Top half shaded black.\nShape 2: Top half shaded gray.\nShape 3: Bottom half shaded black.\nShape 4: Entire diamond shaded black.\nShape 5: Bottom half shaded gray.\n\nNow, if we look at the **color progression**:\n\n- Black → Gray → Black → Black → Gray\nWait — perhaps the pattern is **shading the top half in black, then gray, then bottom half in black, then entire black, then bottom half in gray** — but that’s not a clear cycle.\n\nAlternatively, maybe the pattern is **shading the top half, then bottom half, then entire, then top half, then bottom half** — but the colors don’t match.\n\nWait — let’s look at the **shading location**:\n\n1. Top half — black\n2. Top half — gray\n3. Bottom half — black\n4. Entire — black\n5. Bottom half — gray\n\n … (Omitted for brevity.) Now, if we look at the **color sequence**:\n\nBlack → Gray → Black → Black → Gray\n\nThat’s not a clear pattern.\n\nWait — perhaps the pattern is **shading the top half in black, then gray, then bottom half in black, then entire black, then bottom half in gray** — but that’s not a clear cycle.\n\nAlternatively, maybe the pattern is **shading the top half, then bottom half, then entire, then top half, then bottom half** — but the colors don’t match.\n\nWait — let’s look at the **shading location**:\n\n1. Top half — black\n2. Top half — gray\n3. Bottom half — black\n4. Entire — black\n5. Bottom half — gray\n\nNow, if we look at the **color sequence**:\n\nBlack → Gray → Black → Black → Gray\n\nThat’s not a clear pattern.\n\nWait — perhaps the pattern is **shading the top half in black, then gray, then bottom half in black, then entire black, then bottom half in gray** — but that’s not a clear cycle.\n\nAlternatively, maybe the pattern is **shading the top half, then bottom half, then entire, (Comment: truncated)

Uni-OPD: To determine the correct option, let’s analyze the pattern in the sequence of shapes above the separating line:\n\n1. The first shape is a diamond divided into two triangles: the top triangle is black, the bottom is white.\n2. The second shape is a solid gray diamond.\n3. The third shape is a diamond with a white horizontal line dividing it into two triangles — meaning both triangles are white.\n4. The fourth shape is a solid black diamond.\n5. The fifth shape is a diamond with the bottom triangle shaded gray and the top triangle white.\n\nNow, observe the pattern in terms of color and position:\n\n- The sequence alternates between “top black”, “gray”, “white”, “black”, “gray” — but more importantly, the shading is changing in a specific way.\n\nLet’s look at the position of the shaded area:\n\n- 1st: top shaded black\n- 2nd: entire shape gray (no internal division)\n- 3rd: no shading (white only)\n- 4th: entire shape black\n- 5th: bottom shaded gray\n\nSo the pattern seems to be:\n\n- Black top → Gray whole → White whole → Black whole → Gray bottom → ?\n\nWhat’s next? The pattern might be alternating between “shaded top” and “shaded bottom”, but with a shift.\n\nAlternatively, consider the sequence as a cycle of 5 steps, and look for symmetry or repetition.\n\nBut perhaps a simpler approach: look at the **color** and **position** of the shaded area.\n\nLet’s list the shaded areas:\n\n1. Top black\n2. Entire gray (no internal division)\n3. No shading (white only)\n4. Entire black\n5. Bottom gray\n\nNow, if we look at the **next** shape (the 6th), what should it be?\n\nIf we consider the pattern of shading:\n\n- 1: top black\n- 2: gray (whole)\n- 3: white (no shading)\n- 4: black (whole)\n- 5: bottom gray\n- 6: ?\n\nPerhaps the pattern is: top black → gray → white → black → bottom gray → **top white**?\n\nBut that’s speculative.\n\nAlternatively, notice that the 2nd and 4th shapes are solid (gray and black), and the 1st, 3rd, and 5th have internal divisions.\n\nBut the 3rd shape is white only — no shading.\n\nLet’s look at the **color transitions**:\n\n- 1: black top\n- 2: gray\n- 3: white\n- 4: black\n- 5: gray bottom\n\nSo the colors are: black → gray → white → black → gray → ?\n\nIf we follow the color sequence: black, gray, white, black, gray — then the next should be **white**.\n\nBut where? The 6th shape should have **white** shading.\n\nLooking at the options:\n\nA: Gray diamond (whole) — matches 2nd or 4th\nB: White top, black bottom — doesn’t match\nC: Black whole — matches 4th\nD: Gray bottom, white top — matches 5th\nE: White whole — matches 3rd\n\nSo if the pattern is: black → gray → white → black → gray → **white**, then the 6th should be **white whole**.\n\nThat matches option **E**.\n\nAlternatively, if we look at the **position** of shading:\n\n1. Top black\n2. Whole gray\n3. Whole white\n4. Whole black\n5. Bottom gray\n6. ? — likely **top white**\n\nBut option E is white whole — which could be interpreted as top white.\n\nLooking at the options, E is a white diamond — no shading — which fits the “white whole” pattern.\n\nSo the most consistent pattern is:\n\n1: black top\n- 2: gray whole\n- 3: white whole\n- 4: black whole\n- 5: gray bottom\n- 6: white whole (to complete the cycle)\n\nThus, the answer is:\n\n<answer>E</answer>

Figure F.3: Example output of LogicVista. The Student model produces an incorrect reasoning trace and arrives at logicvista the wrong answer. Standard OPD overthinks the problem, generating an excessively long response that is truncated without producing a final answer. In contrast, Uni-OPD reasons concisely and correctly answers the question.

Question: In 1996, total output from all fuels was 200TWh. If output for Nuclear in 2006 was twice that for Coal in 1996, what was the output for Nuclear in 2006? Select from A, B, C, and D. (A) 140TWh (B) 400TWh (C) 38.png 64TWh (D) 96TWh Answer: A

Student: To solve this, we need to:\n\n1. Find the percentage of Nuclear output in 1996 from the chart.\n2. Find the percentage of Coal output in 1996.\n3. Use the given total output for 1996 (200 TWh) to calculate the actual Coal output in 1996.\n4. Double that to get the Nuclear output in 2006 (as per the problem statement).\n5. Use the 2006 Nuclear percentage to verify or calculate the actual output (though the problem gives a direct relationship, so we can proceed with step 4).\n\n---\n\n**Step 1: Nuclear in 1996**\n\nFrom the chart, Nuclear in 1996 is 16% of total output.\n\nSo, Nuclear 1996 = 16% of 200 TWh = 0.16 × 200 = **32 TWh**\n\n---\n\n**Step 2: Coal in 1996**\n\nFrom the chart, Coal in 1996 is 35% of total output.\n\nSo, Coal 1996 = 35% of 200 TWh = 0.35 × 200 = **70 TWh**\n\n---\n\n**Step 3: Nuclear in 2006 = 2 × Coal in 1996**\n\nNuclear 2006 = 2 × 70 = **140 TWh**\n\n---\n\n**Step 4: Verify with 2006 chart**\n\nFrom the chart, Nuclear in 2006 is 19% of total output.\n\nTotal output in 2006 is not given, but we can check if 19% of total equals 140 TWh.\n\nSo, total 2006 = 140 / 0.19 ≈ 736.8 TWh — which is not needed since the problem gives a direct relationship.\n\n---\n\nThus, the output for Nuclear in 2006 is **140 TWh**.\n\n<answer>A</answer>

OPD: To solve this, we first need to find the output for Coal in 1996 from the chart. In 1996, Coal is represented by the orange segment, which is 35% of the total output. Since total output in 1996 was 200 TWh, Coal output in 1996 is:\n\n35% of 200 TWh = 0.35 × 200 = 70 TWh.\n\nThe question states that Nuclear output in 2006 was twice that of Coal in 1996. So:\n\nNuclear output in 2006 = 2 × 70 TWh = 140 TWh.\n\nThus, the correct answer is:\n\n<answer>A</answer>

Uni-OPD: To solve this, we first need to find the output for Coal in 1996 from the chart. In 1996, Coal is shown as 35% of total output. Since total output in 1996 was 200 TWh, Coal output in 1996 is:\n\n35% of 200 TWh = 0.35 × 200 = 70 TWh.\n\nThe question states that Nuclear output in 2006 was twice that of Coal in 1996. So:\n\nNuclear in 2006 = 2 × 70 TWh = 140 TWh.\n\nLooking at the options, (A) 140TWh matches this value.\n\n<answer>A</answer>

Figure F.4: Example output of LogicVista. All three models correctly answer this multi-step arithmetic reasoning question. OPD and Uni-OPD both reason concisely, with Uni-OPD being slightly more token-efficient.

logicvista

Question: The left image shows a polyhedron composed of 3 equally-sized gray cubes and 16 white cubes, which can be divided into three smaller polyhedra, labeled as ①, ②, and ③. Which polyhedron could ③ represent: A: A B: B 38.png C: C D: D Answer with the option letter only. Answer: A Student: C

OPD: A

Uni-OPD: A

Question: Choose the most appropriate option from the four given choices to fill in the question mark, so that a certain pattern is exhibited: A: A B: B C: C 38.png D: D Answer with the option letter only. Answer: B Student: A

OPD: A

Uni-OPD: B

Figure F.5: Example output of VisuLogic. Uni-OPD correctly answers both questions, demonstrating that our training recipe encourages student exploration to improve its ability for challenging visual reasoning problems.

18 Visulogic

Question: Which cube can be formed by folding the given shape? Select from A, B, C, and D. (A) (B) (C) (D)

38.png

Answer: D

Student: To determine which cube can be formed by folding the given net, we need to analyze the positions and orientations of the patterns (lines and dots) on the faces of the net.\n\nThe net has:\nA central square with a **dot** in the center.\n- Adjacent squares with various **lines** and **dots**.\n- The key is to check which option correctly shows the relative positions and orientations of these patterns when folded.\n\nLet’s examine the options:\n\n**Option A**: Shows a face with a **horizontal line** and a **vertical line** on adjacent faces. The dot is not visible on any face — but in the net, the dot is on the central face, which becomes the front face in this view. So this is inconsistent.\n\n**Option B**: Shows a **dot** on the **bottom face** and a **horizontal line** on the **front face**. But in the net, the dot is on the central face, which becomes the front face — so the dot should be on the front face, not the bottom. Also, the lines don’t match the net’s configuration.\n\n**Option C**: Shows a **dot** on the **front face** and a **horizontal line** on the **side face**. This matches the net: the central face (with the dot) becomes the front, and the adjacent face with the horizontal line becomes the side. The orientation of the lines and the dot’s position are consistent.\n\n**Option D**: Shows a **dot** on the **top face** and a **horizontal line** on the **side face**. But in the net, the dot is on the central face, which becomes the front face — not the top. So this is incorrect.\n\nThus, **Option C** is the only one that correctly reflects the folding of the net.\n\n<answer>C</answer>

OPD: To determine which cube can be formed by folding the given net, we need to analyze the positions and orientations of the symbols (dots and lines) on the faces of the net.\n\nThe net has:\n- A central square with a dot in the center.\n- Adjacent squares with various lines and dots.\n\nWhen folding:\n- The central square becomes the front face.\n- The square with the dot above it becomes the top face.\n- The square with the dot to the left becomes the left face.\n- The square with the dot to the right becomes the right face.\n- The square below becomes the bottom face.\n- The square to the far left becomes the back face.\n\nNow, let’s check the options:\n\nA: Shows a line on the top face and a line on the front face — but the top face should have a dot, not a line. So A is incorrect.\n\nB: Shows a dot on the top face and a line on the front face — but the top face should have a dot, and the front face should have a line. However, the dot is on the top face, which matches, but the line is on the front face, which also matches. But wait — the dot is on the top face, and the line is on the front face — this is consistent with the net. But let’s check others.\n\nC: Shows a dot on the front face — but in the net, the dot is on the top face. So C is incorrect.\n\nD: Shows a dot on the front face — again, incorrect.\n\nWait — let’s re-examine.\n\nActually, in the net:\n- The central square has a dot.\nThe square above it has a dot.\n- The square to the left has a dot.\n- The square to the right has a dot.\n\nBut in the net, the dot is on the central square, and the other dots are on adjacent squares.\n\nWhen folded:\n- The central square becomes the front.\n- The square above becomes the top.\n- The square to the left becomes the left.\n- The square to the right becomes the right.\n- The square below becomes the bottom.\n- The square to the far left becomes the back.\n\nSo, the dot should be on the front face (central square), and the other dots should be on the top, left, right, or back.\n\nBut in option B, the dot is on the top face — which is incorrect, because the top face should have a dot from the square above the central square.\n\nWait — let’s look again … (Omitted for brevity.) Option A: the front face has a line — which is incorrect.\n\nSo none seem correct.\n\nWait — perhaps the correct answer is B.\n\nBecause in option B, the dot is on the top face — which is correct.\n\nThe front face has a line — which is incorrect.\n\nBut in the net, the front face has a dot — so it should be a dot.\n\nSo B is incorrect.\n\nPerhaps the correct answer is C.\n\nBut C has a line on the top face — which is incorrect.\n\nSo perhaps I’m misreading the net.\n\nLet me look at the net again.\n\nThe net has:\n\n- Central

Uni-OPD: To determine which cube can be formed by folding the given net, we need to analyze the positions and orientations of the patterns (dots and lines) on the faces of the net.\n\nLooking at the net:\n- There is a central square with a dot in the center.\n- Adjacent to it, there are squares with various lines and dots.\n- Specifically, the dot is on the central square, and there is a dot on the top square (above the center).\n- Also, there is a dot on the left square (to the left of the center).\n\nWhen folding, the top square (with the dot) will be on top, and the left square (with the dot) will be on the left side. The central square becomes the front face.\n\nNow, let’s evaluate the options:\n\nA: Shows a line on the top and a line on the front — doesn’t match the net’s pattern.\nB: Shows a dot on the bottom and a line on the front — doesn’t match.\nC: Shows a dot on the front and a line on the side — doesn’t match.\nD: Shows a dot on the top and a line on the front — matches the net’s pattern.\n\nTherefore, the correct answer is D.\n\n<answer>D</answer>

Figure F.6: Example output of LogicVista. On this challenging visual pattern reasoning puzzle, both the Student model and OPD fail to produce a final answer due to overthinking. Uni-OPD, however, identifies the correct pattern and selects the right answer. logicvista Question: Is the median value of all the leftmost bars greater than 50? Answer the question with a single word.

38.png

Answer: Yes

Student: No

OPD: No

Uni-OPD: Yes

Question: What's the difference in value of biggest and smallest segment? Answer the question with a single word.

38.png

Answer: 77

Student: 77

OPD: 77

Uni-OPD: 77

Figure F.7: Example output of ChartQA. All models answer the simpler chart question correctly, while only Uni-OPD answers the more complex one correctly. chartqa

Question: Please solve the problem step by step and put your answer in one \"\\boxed{}\". How many points are there in the three unseen sides of dice? 38.png Answer: 11

Student: We are given a die (cube) with dots on its visible faces. The image shows **three visible faces**:\n\n- **Top face**: 6 dots (since it’s a standard die, and we can infer from the pattern — 3 dots on top, 3 on bottom, etc. — but actually, looking at the image: the top face has 6 dots arranged in two rows of three. So top = 6.\n- **Front face**: 5 dots (centered, with 4 around it — 1 in center, 4 around? Actually, standard die: 5 dots = center dot with 4 around? But in the image, front face has 5 dots: one in center, and four around? Wait — let’s count carefully.\n\nActually, looking at the image:\n\n- **Top face**: 6 dots — arranged in two rows of three. So 6.\n- **Front face**: 5 dots — arranged as a center dot with four dots around it? But in the image, it’s shown as: one dot in center, and then dots at top-left, top-right, bottom-left, bottom-right? That’s 5 dots. So front = 5.\n- **Right face**: 3 dots — arranged in a diagonal? Actually, looking at the image: the right face has 3 dots — one at top, one in middle, one at bottom? Or is it arranged as a vertical line? Actually, in the image, the right face has 3 dots: one at top, one in middle, one at bottom — so 3.\n\nWait — let me re-express the image:\n\nThe die is shown with:\n\n- **Top face**: 6 dots (two rows of three)\n- **Front face**: 5 dots (center dot, and four around? Actually, no — in the image, front face has 5 dots: one in center, and then dots at top-left, top-right, bottom-left, bottom-right? That’s 5. But standard die 5-dot face is usually arranged as a cross? But in this drawing, it’s 5 dots: one center, and four around — yes, that’s 5.\n- **Right face**: 3 dots — arranged vertically? Or diagonally? Actually, in the image, the right face has 3 dots: one at top, one in middle, one at bottom — so 3.\n\nBut wait — in a standard die, opposite faces sum to 7. So if top is 6, then bottom is 1. If front is 5, then back is 2. If right is 3, then left is 4.\n\nSo the three **unseen** sides are:\n\n- **Bottom**: 1 dot\n- **Back**: 2 dots\n- **Left**: 4 dots\n\nSo total unseen dots = 1 + 2 + 4 = **7**\n\nBut let’s verify with the visible faces:\n\nVisible: top=6, front=5, right=3 → total visible = 6+5+3 = 14\n\nTotal dots on a die = 1+2+3+4+5+6 = 21\n\nSo unseen = 21 - 14 = **7**\n\n✅ So the answer is 7.\n\n---\n\n**Final Answer: $\\boxed{7}$**

OPD: We are given a die (cube) with dots on its visible faces. The die is shown with three faces visible:\n\n- Top face: 3 dots\n- Front face: 5 dots\n- Right face: 1 dot\n\nWe need to find the total number of dots on the **three unseen sides**.\n\nA standard die has 6 faces. The sum of dots on opposite faces of a standard die is always 7.\n\nSo, we can find the dots on the unseen faces by subtracting the visible dots from the total possible dots on all faces.\n\nTotal dots on a die: 1+2+3+4+5+6 = **21**\n\nVisible dots: 3 (top) + 5 (front) + 1 (right) = **9**\n\nTherefore, unseen dots = 21 - 9 = **12**\n\nAlternatively, we can find the unseen faces by using the opposite face rule:\n\n- The face opposite the top (3 dots) is the bottom. Since 3 + x = 7 → x = 4\n- The face opposite the front (5 dots) is the back. 5 + y = 7 → y = 2\n- The face opposite the right (1 dot) is the left. 1 + z = 7 → z = 6\n\nSo, unseen faces have: 4 + 2 + 6 = **12**\n\nThus, the total number of points on the three unseen sides is:\n\n\\boxed{12}

Uni-OPD: We are given a die (cube) with dots on its visible faces. From the image, we can see:\n\n- The **top face** has **4 dots**.\n- The **front face** has **5 dots**.\n- The **right face** has **1 dot**.\n\nWe know that a standard die has faces with numbers: 1, 2, 3, 4, 5, 6, and opposite faces sum to 7.\n\nSo, let’s find the numbers on the **unseen** faces:\n\n1. **Opposite of the top face (4 dots)** → 7 - 4 = **3 dots**.\n2. **Opposite of the front face (5 dots)** → 7 - 5 = **2 dots**.\n3. **Opposite of the right face (1 dot)** → 7 - 1 = **6 dots**.\n\nNow, add the dots on the unseen sides:\n\n3 + 2 + 6 = **11 dots**.\n\n\\boxed{11}

Figure F.8: Example output of MathVision. All three models follow the required format, but only Uni-OPD produces correct reasoning and reaches the right answer. mathvision

Question: As shown in the figure, two identical largest circles can be cut out from a rectangular piece of paper. What are the side lengths of the rectangle, AB = ( ) cm, AD = ( ) cm? A. 2, 4; B. 8, 4; C. 4, 4; 38.png D. 4, 8; E. No correct answer Answer: D Student: To determine the side lengths of the rectangle, let’s analyze the figure.\n\nWe are told that **two identical largest circles** can be cut out from the rectangle. The circles are inscribed such that they touch each other and the sides of the rectangle.\n\nFrom the diagram, each circle has a radius of **2 cm**, as indicated by the dashed line from the center to the side of the circle.\n\n---\n\n### Step 1: Find the width of the rectangle (AB)\n\nSince the circles are identical and placed side by side, the **width of the rectangle (AB)** must be equal to the **diameter of one circle**.\n\nDiameter = 2 × radius = 2 × 2 = **4 cm**\n\nSo, AB = 4 cm.\n\n---\n\n### Step 2: Find the height of the rectangle (AD)\n\nThe height of the rectangle (AD) must be equal to the **diameter of one circle**, because the circles are inscribed vertically and touch the top and bottom of the rectangle.\n\nSo, AD = 4 cm.\n\n---\n\n### Final Answer:\n\nAB = 4 cm, AD = 4 cm → This matches option **C. 4, 4**\n\n<answer>C. 4, 4</answer>

OPD: To determine the side lengths of the rectangle, we analyze the figure:\n\n- Each circle has a radius of 2 cm, as indicated by the dashed line.\n- Therefore, the diameter of each circle is 2 × 2 = 4 cm.\n- Since two identical circles are placed side by side within the rectangle, the length of the rectangle (AB) must be equal to the sum of the diameters of the two circles: 4 cm + 4 cm = 8 cm.\n- The width of the rectangle (AD) must be equal to the diameter of one circle, which is 4 cm.\n\nThus, AB = 8 cm and AD = 4 cm.\n\n<answer>B. 8, 4</answer>

Uni-OPD: The figure shows two identical largest circles cut from a rectangle. Each circle has a radius of 2 cm, as indicated by the dashed line. Therefore, the diameter of each circle is 4 cm.\n\nSince the circles are placed side by side within the rectangle, the width of the rectangle (AB) must be equal to the diameter of one circle, which is 4 cm.\n\nThe length of the rectangle (AD) must accommodate the diameters of both circles placed side by side, so AD = 4 cm + 4 cm = 8 cm.\n\nThus, AB = 4 cm and AD = 8 cm.\n\nLooking at the options, this matches option D.\n<answer>D. 4, 8</answer>

Figure F.9: Example output of WeMath. This geometry problem requires correctly identifying which side accommodates two circle diameters. Both the Student model and OPD confuse the orientation of AB and AD, while Uni-OPD correctly answers the question.

20 wemath

Record · ID 155314 · SHA-256 e02ef831564b4183
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.