When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon
arXiv:2606.30445v1 [cs.LG] 29 Jun 2026
Huaqing Zhang* Tsinghua University [email protected] Juno Kim UC Berkeley [email protected]
Jingchu Gai* Carnegie Mellon University [email protected]
Bingbin Liu Harvard University [email protected]
Andrej Risteski Carnegie Mellon University [email protected]
Abstract Online imitation learning (IL), particularly on-policy distillation, has emerged as a strong LLM posttraining approach, often outperforming offline supervised fine-tuning (SFT). Yet a principled understanding of when and why online interaction helps remains unclear. In this work, we challenge the view that error accumulation is the main source of online IL’s advantage, and instead show that the benefits of online interaction depend critically on whether the setting is realizable, i.e., whether the student policy class can represent the expert policy. Under realizability, we empirically find that offline IL already matches expert performance. In contrast, in non-realizable (misspecified) settings, we prove that offline IL encounters an information-theoretic bottleneck even when horizon H = 1, and propose a structural characterization of misspecification relative to the reward, under which online IL provably achieves high performance despite a large distributional mismatch between the expert and student policies.
1
Introduction
Imitation learning (IL) trains a student policy from expert demonstrations, enabling the student to achieve strong performance without access to explicit reward signals. For post-training of large language models (LLMs), the canonical offline imitation learning method is supervised fine-tuning (SFT), where the learner imitates an expert policy using a fixed dataset of expert demonstrations, such as human-written responses or traces generated by a stronger model. However, such offline approaches are reported to suffer from limitations such as poor generalization [7, 21, 30] and severe catastrophic forgetting [5]. Online imitation learning offers a promising alternative for mitigating these issues in post-training. In the online IL setting, the learner actively interacts with the environment and queries an expert to obtain feedback [48]. A prominent online IL approach for LLM post-training is on-policy distillation, where the student samples from its current policy and minimizes the reverse-KL divergence to the expert distribution [2, 20, 29]. Although recent studies report the empirical superiority of online IL in LLM post-training [35, 63], a principled understanding of when and why this advantage arises remains limited. A classical explanation for the advantage of online IL is its ability to mitigate error accumulation: offline IL can suffer from errors that compound quadratically over the horizon, whereas DAgger-style online algorithms improve this dependence under a recoverability assumption [46, 48]. However, recent theory by Foster et al. [15] shows that under the standard log-likelihood objective used in SFT, offline IL can match the sample-complexity guarantees of DAgger in realizable settings, where the expert policy is contained in the * Equal contribution.
1
student policy class. This result calls into question whether error accumulation alone can fully explain the practical advantage of online IL in LLM training. In this work, we identify non-realizability (or misspecification) as an orthogonal source of online IL’s advantage: the student policy class is too restricted, e.g., due to a much smaller model size, to represent the expert policy. Unlike the classical error-accumulation view, which concerns long-horizon compounding, this advantage can already arise in contextual bandits with horizon H = 1. We first study the realizable setting as a sanity check, and empirically show that offline IL already suffices in this setting (Section 4). Specifically, we consider both synthetic (Countdown [39]) and real-world math reasoning (GSM8K [8] and DeepScaleR [36]) tasks. For each task we first apply reinforcement learning (RL) to a base model to obtain the expert, and then compare SFT and on-policy distillation initialized from the same base model. By construction, this setup guarantees realizability, since the expert policy is itself obtained within the same model family. Our results show that the SFT-trained student matches the expert’s performance, and on-policy distillation offers no further gains in either accuracy or training speed. This finding is consistent with the theory of Foster et al. [15], which predicts no statistical-complexity gap between offline and online IL in realizable settings, and further indicates that other potential sources of online IL advantage, such as optimization effects or problem-specific structures, do not yield an empirical advantage in this regime either. Taken together, these results further motivate our focus on the non-realizable setting as the key regime for understanding the benefits of online imitation learning. In non-realizable settings, we develop an account of online IL’s advantage that is orthogonal to the classical error-accumulation argument. We focus on a simplified yet informative contextual-bandit framework with horizon H = 1, where long-horizon compounding is absent by construction, yet online IL still gains an advantage from misspecification. We first show, both theoretically and empirically, that prior analyses that control the performance gap via distributional discrepancies between the expert and student policies [15, 42, 45] can be overly pessimistic under misspecification, and reward-dependent analyses are necessary. For offline IL, we identify an information-theoretic limitation specific to non-realizable settings: the expert policy may place too little probability mass on responses that the restricted student class can learn. For online IL, we further introduce a structural condition that explicitly captures the misspecification structure relative to the reward (Assumption 1), under which online IL can provably achieve small performance suboptimality despite large distributional discrepancy. At a high level, the condition states that the expert’s signal should either be aligned with the actual reward, or deviate from it only in ways that the restricted student class cannot realize. To summarize, our main contribution is to identify non-realizability as a key factor underlying the advantage of online over offline imitation learning in language model post-training. Specifically, 1. In realizable settings, we empirically verify that offline IL (SFT) fully recovers expert performance, and that online IL (on-policy distillation) yields neither accuracy gains nor faster learning (Section 4). 2. In non-realizable settings, we show that prior discrepancy-based analyses can be overly pessimistic (Section 5.1), and reward-dependent analyses are necessary to explain the success of IL. 3. By analyzing the reward-dependent structure of misspecification, we identify an information-theoretic limitation of offline IL in this regime (Section 5.2) and propose a structural condition characterizing when online IL can remain effective despite misspecification (Section 5.3).
2
Preliminaries
For a set X , let ∆(X ) denote the set of all probability distributions over hX . For two i distributions P, Q over X , P (x) := their Kullback–Leibler divergence is defined by DKL (P ∥Q) Ex∼P log Q(x) , and their total variation P := distance is defined by DTV (P, Q) supA⊆X |P (A) − Q(A)| = 12 x∈X |P (x) − Q(x)|. We use standard asymptotic notation throughout, and we write a ≲ b as shorthand for a = O(b), and a ≳ b as shorthand for a = Ω(b). 2
2.1
Problem Setting
To provide a clean theoretical sandbox, we abstract language model post-training as a contextual bandit problem [16, 41, 62]. Let X represent the set of all possible prompts (contexts) and Y the set of generated responses (actions). A policy π : X → ∆(Y) specifies a conditional distribution π(y | x) over responses y given a prompt x. Let r(x, y) : X × Y → [0, 1] be a reward function. We aim to find a policy π within a policy class Π that achieves high expected reward V (π) := Ex∼Dx Ey∼π(·|x) r(x, y), where Dx is the distribution over prompts. We follow standard convention in RL theory and assume the policy class |Π| < ∞ is finite [14, 15, 45]. This finite-class abstraction provides a clean testbed for studying statistical generalization without imposing stronger structural assumptions, such as linearity, that need not hold for neural networks. [15]. Realizability. Realizability is a widely studied assumption in RL theory that asks whether the student policy class is expressive enough to represent the expert policy. In this work, we identify realizability as a crucial factor governing the advantage (or lack thereof) of online IL over offline IL. Definition 1 (Realizability and Misspecification). We say that the expert policy π e is realizable with respect to the student policy class Π if π e ∈ Π. Otherwise if π e ∈ / Π, the setting is non-realizable (or misspecified).
2.2
Imitation Learning
Imitation learning is a learning paradigm in which the learner improves its policy by leveraging information provided by an expert policy π e , without direct access to the reward feedback during training. In this work, we study two settings: offline and online imitation learning. Offline Imitation Learning. The learner is given a static dataset D = {(xi , yi )}ni=1 of i.i.d. expert demonstrations, where xi ∼ Dx and yi ∼ π e (· | xi ). The standard approach in this setting is behavior cloning (BC), which converts IL to a supervised learning problem with the following objective: Definition 2 (Population Objective of Behavior Cloning). Given an expert policy π e and a loss function ℓ : ∆(Y) × Y → R, the population behavior cloning objective is defined as LBC (π) := Ex∼Dx Ey∼πe (·|x) ℓ(π(· | x), y) . (1) The goal of behavior cloning is to find a policy π ∈ Π that minimizes LBC (π). A standard instantiation of behavior cloning in LLM post-training uses the negative log-likelihood loss ℓ(π(· | x), y) = − log π(y | x), which corresponds to the supervised fine-tuning (SFT) objective. Online Imitation Learning. In the online setting, the learner can actively query the expert for feedback. Learning proceeds in episodes: in the ith episode, the learner observes xi ∼ Dx , executes a certain policy πi to generate a response yi ∼ πi (· | xi ), and receives expert-provided information evaluated on this response. In existing online imitation learning formulations based on MDPs (e.g., [15, 46]), the expert provides step-wise guidance along a trajectory. In contrast, we adopt a simpler contextual bandit formulation: in each episode, the learner produces a complete response yi (a token sequence), and the expert reveals its conditional probability π e (yi | xi ) on that response. This form of probability access is natural in language model distillation, where the expert model’s logits are available. Furthermore, it captures the core interaction pattern of widely used on-policy distillation algorithms [2, 20, 35], and enables a meaningful characterization of the benefit of online access to expert information beyond a fixed offline dataset (Section 5). While the objective of online IL can be flexible, in this work we focus on the following formulation:
3
Definition 3 (Population Objective of Online Imitation Learning). Given an expert policy π e and a shaping function f : R → R that maps the expert density to a scalar score, the population objective of online imitation learning is Jon (π) := Ex∼Dx Ey∼π(·|x) [f (π e (y | x))] . (2) The goal of online imitation learning is to find a policy π ∈ Π that maximizes Jon (π). The key difference to the offline objective (Equation (1)) is that the inner expectation is taken over the student policy π, rather than the expert policy π e . A prominent instantiation of online imitation learning in language model post-training is on-policy distillation [2, 20], which uses a reverse-KL objective and is optimized in an online, on-policy manner: max −Ex∼Dx [DKL (π(· | x)∥π e (· | x))] . π∈Π
(3)
This reverse-KL objective is an instance of Definition 3 with f (π e (y | x)) = log (π e (y | x)), together with an additional entropy regularization term that promotes exploration and prevents policy collapse:1 − Ex∼Dx [DKL (π(· | x)∥π e (· | x))] = Ex∼Dx Ey∼π(·|x) [log π e (y | x)] + Ex∼Dx [H(π(· | x))] , where H(π(· | x)) := −Ey∼π(·|x) [log π(y | x)] denotes the policy entropy. Remark 1 (Contextual bandit formulation). Previous analyses of imitation learning are typically framed in the Markov Decision Process (MDP) setting, which involves sequential decision-making, stochastic state transitions, and rewards assigned at each step. In this work, we focus on autoregressive language generation, which is a simpler special case: the state is accumulative with deterministic transition st = (x, y≤t ) = (st−1 , yt ), and the reward (e.g., whether a math problem is solved correctly) is defined over the entire trajectory. As a result, this setup is essentially a contextual bandit: given a context x, the policy π specifies the distribution over the entire response trajectories y. This formulation also matches the practical objectives used in LLM post-training. In particular, both SFT (Equation (1)) and on-policy distillation (Equation (3)) are defined at the full-trajectory level, and can therefore be naturally captured within the contextual bandit framework. Moreover, in this work, we identify non-realizability as a source of advantage of online over offline imitation learning that is distinct from the classical error-accumulation view. Unlike the latter, which attributes the benefit of online interaction to the mitigation of compounding errors over long horizons [46], the advantage arising from non-realizability already appears in contextual bandits with H = 1. By removing the multi-step structure, the contextual bandit formulation provides a cleaner setting for isolating this source of advantage.
3
Related Work
Imitation Learning for LLM Post-Training. Imitation learning plays an important role in languagemodel post-training. Supervised fine-tuning (SFT) can be viewed as offline IL, where the model is trained with a log-likelihood objective on the demonstration data [3, 38, 40]. However, empirical studies show that SFT can struggle to improve a weak student when the expert is a substantially stronger reasoning model. In these settings, expert responses may be too compressed or rely on reasoning patterns beyond the student’s representational capacity, making them difficult for the student to imitate effectively [27, 32]. Recently, online imitation learning has emerged as a powerful paradigm for LLM post-training. A representative instantiation in this context is on-policy distillation, where the learner actively samples responses from the current student model and queries the expert for probabilities along these self-generated trajectories [20, 29]. Unlike supervised fine-tuning that optimizes a forward-KL objective on fixed data, online distillation typically minimizes the reverse-KL divergence between the student and expert distributions [20]. 1 The shaping function f can be generalized to depend on both π e (y | x) and π(y | x) without affecting our analysis. This extension fully captures the reverse-KL objective. For simplicity, we focus on shaping functions that depend only on the expert density.
4
Recent work has further extended on-policy distillation to black-box settings without access to expert logits [65], multi-expert distillation [60], and self-distillation [68]. These methods have proven effective in practice [35, 64] and have been adopted in industrial-scale post-training pipelines [60, 63], while more recent work studies their mechanisms and failure modes [33]. However, a theoretical understanding of when and why online methods outperform offline SFT remains limited, as discussed below. Theoretical Analysis on Online and Offline IL. Seminal work Ross & Bagnell [46] shows that offline IL incurs error accumulation: If the learner incurs a per-step error ϵ under the expert distribution, then when rolled out under the learner’s own policy, the compounding error can scale quadratically with the horizon, as O(H 2 ϵ). Online IL algorithms such as DAgger [47, 48] improve this dependence to O(µHϵ), where a small recoverability coefficient µ indicates that the expert can recover effectively from suboptimal intermediate actions. However, this line of work gives only algorithm-dependent lower bounds, rather than an end-to-end separation between offline and online IL [15, 42]. Indeed, Foster et al. [15] show that, under realizability, DAgger-like algorithms offer no worst-case statistical improvement over offline IL with log-likelihood loss. While online interaction could theoretically yield improvements under additional assumptions (see the discussion in their Section 4), our empirical results do not provide evidence of such gains under realizability (Section 4). Prior work has also studied the minimax sample complexity of imitation learning in tabular MDPs and policy classes [42, 43, 55]. In non-realizable settings, Rohatgi et al. [45] formulate IL as minimizing the Hellinger distance between the student and the expert. They show that a specific offline behavior cloning algorithm, RhoEstimatorBC, already achieves optimal statistical complexity for this objective [45, Theorem 3.1], and no computational advantage of online interaction is established. This suggests that discrepancy-based analyses can be overly pessimistic and may not explain the practical advantage of online IL. Another line of work is inverse reinforcement learning (IRL), which aims to recover an underlying reward function from expert demonstrations [1, 4, 56, 70]. Recent work shows that interaction with the environment can enable efficient imitation learning under misspecification through IRL, given certain structural assumptions [10]. However, these results typically assume that the reward function lies in a finite and realizable hypothesis class, and the IRL algorithms are not commonly adopted in LLM post-training. Finally, a recent study [53] also empirically identifies misspecification, rather than sampling error, as a more fundamental cause of behavior cloning’s suboptimal performance in partially observed MDPs.
4
Offline IL Suffices Under Realizability
Classical analyses attribute the advantage of online IL to mitigating long-horizon error accumulation [46, 48]. However, recent theoretical work [15] shows that, for the log-likelihood objective used in SFT, offline behavior cloning already achieves optimal statistical complexity in realizable settings, leaving no worst-case statistical advantage for online IL. That said, this theory does not cover every possible advantage of online IL in realizable settings. For example, it studies statistical complexity but does not consider optimization dynamics, and it does not rule out the possibility that online IL may still benefit from favorable problem-specific structure. In this section, as a sanity check for the realizable setting, we empirically demonstrate that SFT (offline IL) is indeed sufficient to fully recover expert performance in this regime, and on-policy distillation does not yield further gains.
4.1
Experimental Setup
We conduct experiments on three tasks: a synthetic Countdown [39] task2 and two math reasoning datasets: GSM8K [8] and DeepScaleR [36] (evaluated on MATH [24] and AIME benchmarks). We use Qwen2.5-3B2 Countdown asks the model to use each given integer exactly once, with operators {+, −, ×, ÷}, to form an expression matching a target; e.g., for [8, 3, 2, 1] and target 24, 8 × 3 × (2 − 1) = 24.
5
Countdown
GSM8K
Performance
0.6 0.4
DeepScaleR (MATH)
0.8
0.850
0.7
0.800
0.825
0.2 0.0 0
100
200
Steps
300
0.25
0.750
0.5
0.20
0.725 0
SFT (realizable expert) SFT (non-realizable expert)
100
200
Steps
300
DeepScaleR (AIME)
0.30
0.775
0.6
0.35
0
250
500
Steps
On-policy distillation (realizable expert) RL
750 1000
0
250
500
Steps
750 1000
Realizable expert performance
Figure 1: Offline IL suffices with realizable experts. Across different tasks, SFT with a realizable expert fully matches expert performance, and on-policy distillation yields no accuracy gains or faster training. In contrast, when the expert is non-realizable, SFT exhibits a significant performance gap. Instruct [57] as the base model for Countdown, Llama-3.2-3B [19] fine-tuned on OpenR1 [11] for GSM8K, and DeepSeek-R1-Distill-Qwen-1.5B [9] for DeepScaleR (details in Appendix C). We compare four training paradigms: • RL (GRPO): We train the model using GRPO [49] with verifiable outcome rewards to obtain the realizable expert model for SFT and on-policy distillation. • SFT with realizable expert: The student model is trained by SFT on samples generated by the RL-trained model. By construction, this setup guarantees realizability, since the expert is obtained within the same model family. • SFT with non-realizable expert: For the Countdown task, the student is trained directly on ground-truth solutions. For GSM8K, the student is supervised by a highly capable non-realizable expert, DeepSeek-V3.2-Exp [34], which achieves over 95% accuracy on GSM8K. • On-policy distillation with realizable expert: The student model is trained by on-policy distillation using the same RL-trained expert described above.
4.2
Experimental Results
As shown in Figure 1, on both the synthetic Countdown task and real-world math reasoning tasks, offline IL (SFT) with a realizable expert matches expert performance, while online IL (on-policy distillation) improves neither final performance nor training efficiency. This provides no evidence of the classical error-accumulation phenomenon of offline IL in the realizable setting. Moreover, SFT is substantially more sample-efficient than RL training. In contrast, when the expert is non-realizable, SFT falls well short of the expert’s performance. This indicates that misspecification is a key ingredient in online IL outperforming offline IL in practice. We further evaluate out-of-distribution (OOD) generalization on the Countdown task and catastrophic forgetting during GSM8K training. We find that both SFT and on-policy distillation with a realizable expert achieve OOD performance close to that of the expert and exhibit minor forgetting. By contrast, SFT with a non-realizable expert exhibits substantially worse OOD generalization and more severe forgetting. Detailed settings and results are provided in Appendix B.
5
Non-Realizability as a Source of Online IL Gains
Since the advantage of online IL in LLM post-training does not arise under realizability (Section 4), we turn to the non-realizable setting, where the expert policy cannot be represented by the student policy class, i.e. 6
π e ̸∈ Π. In this setting, online on-policy distillation has been empirically shown to significantly outperform offline IL (SFT) [20, 63, 64]3 . However, despite these successes, a theoretical understanding of why and when online interaction helps in non-realizable settings remains limited. Previous work framed imitation learning as minimizing the distributional discrepancy (e.g., Hellinger distance) between the student and expert policies [45]. In this section, we first discuss and empirically verify that such discrepancy-based analyses can be overly pessimistic under non-realizability. Furthermore, discrepancy analysis can be misaligned with the true reward objective, and therefore does not adequately explain the widely observed empirical advantage of online over offline IL methods. Motivated by these limitations, we develop a new perspective on the advantage of online over offline IL in non-realizable settings by directly accounting for the reward structure. For offline IL, we identify a fundamental information-theoretic barrier under non-realizability even in contextual bandits – a barrier distinct from classical error-accumulation arguments [46, 48]. We show in Theorem 1 that the sample complexity of e any offline IL algorithm that learns from i.i.d. expert demonstrations scales with a coverage coefficient C∞ (Equation (5)), which can be interpreted as a measure of the mismatch between the expert and student policies. For online IL, we characterize the interplay between (i) the expert signal, (ii) the misspecified student policy class, and (iii) the reward function, and identify a general condition under which online IL is provably efficient despite severe misspecification (Theorem 2).
5.1
Limitations of Discrepancy-based Analyses in Non-Realizable Settings
A common approach to deriving IL guarantees is to upper bound the performance suboptimality by a distributional discrepancy between the expert policy π e and the learned policy π̂ ∈ Π [15, 42, 45]. For instance, one typically argues that V (π e ) − V (π̂) ≤ Ex∼Dx [DTV (π e (· | x), π̂(· | x))] ,
(4)
where DTV can be further controlled by other statistical distances or divergences, such as KL, Hellinger, or χ2 divergences.4 In realizable settings, the approximation term vanishes: inf π∈Π Ex∼Dx [DTV (π e (· | x), π(· | x))] = 0, so minimizing discrepancy drives the performance suboptimality V (π e ) − V (π̂) to zero. In non-realizable settings, however, even the best-in-class discrepancy is strictly positive, so discrepancy-based bounds cannot certify suboptimality below this irreducible distance. While such bounds are tight in the worst case over all possible reward functions, they can be overly pessimistic for “average-case” reward functions: a large expert-student discrepancy does not necessarily imply a large performance gap. This is especially relevant for language models, where discrepancies may arise from wording, style, or reasoning traces that are not related to correctness. For example, a strong expert may solve a math problem directly, while a weaker student may reach the same answer through trial and error; their response distributions can differ substantially despite similar success rates. We verify this empirically in LLM post-training below. Furthermore, discrepancy-based analyses do not explain the practical gains of online IL over offline IL. Prior work has shown that offline ρ-estimator behavior cloning is already statistically optimal for minimizing the discrepancy even in non-realizable settings [45, Theorem 3.1]; see Section 3 for more details. This suggests that the advantage of online interaction, when it arises, is not in identifying the policy closest to the expert under a distributional discrepancy, but in identifying the student policy with higher reward under a particular reward function. Empirical Validation. We empirically validate that the distributional discrepancy between an expert and a student learned via imitation learning can be substantially larger than their performance gap. Specifically, 3 Gu et al. [20], Yang et al. [63], and Yang et al. [64] study strong-to-weak distillation (distilling a smaller model from a much larger expert), and their setup therefore falls into the non-realizable setting. 4 Foster et al. [15] gives a tighter upper bound based on Hellinger distance than the direct TV bound for deterministic experts or experts with bounded reward variance (Theorems 2.1 and 3.1). Our discussion is not sensitive to this distinction.
7
Response length
Length-induced TV distance Keyword-frequency gap (wait) Keyword-frequency gap (maybe) Keyword-frequency gap (thus) Keyword-frequency gap (alternatively)
29.99 29.12 23.29 24.88 6.20
Performance gap
5.46
ve
s
ati
alt
ern
wa
0.00 1 2 3 4 5 6 7 8 9 10111213 0.0 Tokens per response (×5 × 103)
Value (%)
ly
0.2
e
0.05
0.4
thu
0.10
yb
0.15
Metric
0.6
it
0.20
R1 R1-Distill
0.8
ma
Proportion
0.25
Keyword Counts
1.0
R1 R1-Distill
Proportion
0.30
(a) Distributional discrepancy proxies.
(b) Discrepancy vs. performance gap.
Figure 2: Bounding performance gap by expert-student discrepancy can be overly pessimistic. We compare DeepSeek-R1-0528 (expert) and DeepSeek-R1-0528-Qwen3-8B (student) on AIME benchmark. (a) The expert and student differ substantially in response-length distributions and keyword frequencies. (b) Length-induced TV distance and keyword-frequency gaps are lower bounds of the TV distance between response distributions, yet already far exceed the performance gap. This suggests that discrepancy-based guarantees can be overly loose for explaining reward performance. we take DeepSeek-R1-0528, a 692B-parameter model, as the expert, and DeepSeek-R1-0528-Qwen3-8B as the student [9], which is distilled from DeepSeek-R1-0528 using Qwen3-8B-Base [63] as the base model. We evaluate both models on the AIME 2024 and AIME 2025 benchmarks. For each benchmark, each model generates 14 responses per question, yielding 840 responses per model in total. As shown in Figure 2, the two models exhibit substantial differences in their response distributions, as reflected by two simple and interpretable features: response length (discretized into 5,000-token bins, and measured with the Qwen3-8B tokenizer for both models) and the frequency of keywords associated with reasoning behavior in the responses [18, 67]. These metrics are lower bounds of the total variation distance between the response distributions, since distinct responses may share the same length bin or selected keywords. Thus, the true expert-student discrepancy can only be larger than reported. Since even these lower bounds already exceed the observed performance gap by a wide margin, our results provide empirical validation that distributional discrepancies can be much larger than performance differences. These limitations motivate us to go beyond the reward-agnostic discrepancy-based analysis in non-realizable settings, and instead ask when offline IL becomes inefficient (Section 5.2) and when online IL can yield improvements (Section 5.3) under a specific reward function.
5.2
Information-theoretic Lower Bounds for Offline IL in Non-Realizable Settings
We now present an information-theoretic limitation of offline IL in non-realizable settings. Intuitively, the hard case arises when the expert’s responses cannot be faithfully represented by the student policy class, while correct responses that are representable by the student are rarely revealed in expert samples. We formalize this intuition in Theorem 1, showing that even when the student class Π contains an optimal policy and the expert π e itself achieves perfect performance, learning from i.i.d. expert demonstrations can still be statistically hard in non-realizable settings. Specifically, for any offline IL algorithm that observes only i.i.d. expert demonstrations, the sample complexity required to achieve a vanishing performance suboptimality e must scale with the coverage coefficient C∞ defined as e := C∞
π ⋆ (y | x) , e x∈X ,y∈Y π (y | x) sup
(5)
e where π ⋆ ∈ arg maxπ∈Π V (π) [6, 37, 44]. This barrier is specific to non-realizability: C∞ = 1 under ⋆ e realizability since one can take π = π , but can be arbitrarily large under misspecification, serving as a measure of the mismatch between the expert and π ⋆ .
8
Limitation of offline IL in non-realizable settings e Theorem 1. For any integer K ≥ 2, any coverage parameter C∞ ≥ 1, and any (possibly randomized) e offline imitation learner A with access to N i.i.d. samples (xi , yi )N i=1 with xi ∼ Dx and yi ∼ π (· | xi ), e there exist a contextual bandit problem (Dx , r), an expert policy π , and a student policy class Π such that the following hold:
1. The expert is optimal, i.e., V (π e ) = 1. 2. There exists an optimal student policy π ⋆ ∈ Π with V (π ⋆ ) = 1. e 3. The expert satisfies a pointwise coverage condition with respect to π ⋆ , i.e., C∞ < ∞.
Moreover, the policy π̂ output by A satisfies C e log |Π| 1 − E[V (π̂)] ≳ min 1, ∞ N , where the expectation is over the randomness of the learner and sampling process. The proof constructs a hard instance where, at each context, the expert puts most mass on an optimal-reward action that is unavailable to the student class due to misspecification, while the unique optimal student action e appears in expert samples only with probability 1/C∞ . The argument follows Theorem 6.2 of Rajaraman et al. [42]; see Appendix A.1.
5.3
A Structural Condition for Effective Online IL in Non-Realizable Settings
Online interaction can alleviate the information bottleneck of learning from i.i.d. expert demonstrations by querying the expert on learner-induced inputs. However, these queries may fall in regions rarely sampled by the expert, where the expert signal can be unreliable. Thus, a high-performing expert is not necessarily a useful teacher for a misspecified student; its feedback must help the learner identify high-performing policies within the restricted student class Π. Motivated by Section 5.1, we seek structural conditions characterizing the interplay of misspecification structure and reward structure, under which online IL achieves small performance suboptimality maxπ∈Π V (π) − V (π̂), even when the learned policy remains far from the expert in distributional discrepancy. We consider a setting in which the feedback on a response y is given by the expert’s conditional probability π e (y | x), and study a specific class of online imitation learning objectives (Definition 3): maxπ∈Π Jon (π) := Ex∼Dx Ey∼π(·|x) [f (π e (y | x))] , where f : R → R is a shaping function. As an example, the widely used on-policy distillation algorithm for language model post-training uses f (π e (y | x)) = log(π e (y | x)) together with an additional entropy regularization term (Section 2.2). A Motivating Synthetic Example. We use a simple toy 2D Gaussian example (Figure 3) to motivate our condition. Actions are a = (x, y) ∈ R2 , policies are Gaussians πµ = N (µ, I2 ), and the reward is r(x, y) = 1[y ≥ 0]. The expert is π e = N ((−2, 2), I2 ), and we compare two student classes ΠA = {N (µ, I2 ) : µx = 0} and ΠB = {N (µ, I2 ) : µy = µx }. Both classes are misspecified, yet the expected rewards resulting from IL (with reverse-KL objective as in on-policy distillation) differ sharply. Let π̂A , π̂B be the reverse-KL projections of π e onto ΠA and ΠB , respectively. Then V (π̂A ) ≈ 0.977, V (π̂B ) = 0.5, corresponding to excess risks 0.023 and 0.5, respectively. The gap between ΠA and ΠB arises from their different misspecification structures relative to the reward. The reward r(x, y) = 1[y ≥ 0] depends only on the y-coordinate. In ΠA , misspecification is confined to the reward-irrelevant x-direction: the class fixes µx = 0 but leaves µy free. Thus, the reverse-KL projection matches the expert’s reward-relevant mean µey = 2, yielding high expected reward. In contrast, ΠB couples the two coordinates through µy = µx . As a result, matching the expert distribution trades off the reward-relevant y-direction against the reward-irrelevant x-direction, leading to suboptimal expected reward. 9
4 2
reward = 1 [y 0]
reward = 1 [y 0]
4
x=0
2.5 0.0
x
2.5
y=
2
x
0
2.5 0.0
x
2.5
Samples from expert Samples from student Feasible mean set of the student policy class Figure 3: Synthetic example: misspecification relative to reward. Actions are (x, y) ∈ R2 and policies are Gaussians; the reward is r(x, y) = 1[y ≥ 0] (shaded region marks r(x, y) = 1). Expert π e = N ((−2, 2), I2 ). Left: student class ΠA : µx = 0 (red dashed line) yields a reverse-KL projection with V (π̂A ) ≈ 0.977. Right: student class ΠB : µy = µx yields V (π̂B ) = 0.5. The performance gap arises from different misspecification structures relative to the reward.
Figure 4: Estimated suboptimality bound. We empirically estimate the subˆ optimality bound in Theorem 2 ∆(ϵ) := minα>0 ϵ+δ̂αα (ϵ) through Monte Carlo. Consistent with Theorem 2, the expert leading to a better student (i.e., Skywork-7B) has a smaller suboptimality bound at every ϵ.
A General Sufficient Condition: Misspecification–Reward Alignment. As illustrated in the above toy example, to characterize when IL is effective in non-realizable settings, one must account for misspecification structure between the student and expert under a particular reward function. We now propose a general condition, which we term the Misspecification–Reward Alignment Condition for the online IL objective considered in Definition 3. Under this condition, approximate optimality for the online IL objective translates into a guarantee on the expected reward suboptimality. Misspecification-Reward Alignment Condition
Without loss of generality, we can decompose the expert signal as: f (π e (y | x)) = α r(x, y) + h(x, y) + b(x),
(6)
where α > 0, h : X × Y → R, and b : X → R. The term αr(x, y) is the reward-aligned component, with α quantifying the strength of alignment. The residual h(x, y) captures expert preferences beyond reward, and b(x) is an action-independent offset. Assumption 1. Let π ⋆ ∈ Π be a reference policy. For any ϵ > 0, assume that for all π ∈ Π satisfying Jon (π) ≥ Jon (π ⋆ ) − ϵ, Ex∼Dx Ey∼π(·|x) h(x, y) − Ey∼π⋆ (·|x) h(x, y) ≤ δα (ϵ). The decomposition in Equation (6) can be made for any fixed α > 0: for any expert signal f (π e (y | x)) and any x-dependent offset bα (x), one can define h(x, y) := f (π e (y | x)) − αr(x, y) − bα (x). Therefore, the decomposition is without loss of generality; the actual assumption lies in the alignment condition on the residual term h. Crucially, Assumption 1 requires that within the ϵ-optimal set under Jon , the residual term cannot inflate Jon (π) for a reward-suboptimal policy. This condition allows us to translate near-optimality for the online IL objective into a guarantee on the reward performance of the learned policy.
10
Effectiveness of online IL in non-realizable settings
Theorem 2. Under Assumption 1, let π ∈ Π be any policy that is ϵ-suboptimal for the online IL objective Equation (2), i.e., Jon (π) ≥ maxπ′ ∈Π Jon (π ′ ) − ϵ. Then π satisfies: V (π) ≥ V (π ⋆ ) − ϵ+δαα (ϵ) . The proof of Theorem 2 is provided in Appendix A.2. To better understand when the Misspecification–Reward Alignment condition (Assumption 1) holds, we highlight two representative scenarios. 1. When the reshaped expert signal f (π e (y | x)) is uniformly aligned well with the reward r(x, y), Assumption 1 holds essentially independently of the misspecified student class. Concretely, if |h(x, y)| ≪ α for all x ∈ X , y ∈ Y, then the suboptimality bound satisfies ϵ+δ(ϵ) ≪ 1 as ϵ → 0. α 2. Although the expert signal f (π e (y | x)) encodes preferences beyond the reward, captured by the residual h(x, y), the misspecified student is unable to represent these preferences. For example, the expert may prefer more concise responses, while any student policy that achieves high performance requires longer chains of reasoning due to limited capability. In this case, the reward-irrelevant preferences (e.g., response length) in the expert signal do not substantially distinguish among near-optimal policies in the student class. Formally, for any π1 , π2 ∈ Π satisfying Jon (πi ) ≥ maxπ∈Π Jon (π) − ϵ for i = 1, 2, we have Ex∼Dx Ey∼π1 (·|x) h(x, y) − Ey∼π2 (·|x) h(x, y) ≪ 1. This regime also explains why the expert signal is effective for ΠA in the 2D Gaussian example. It is worth noting the comparison between Theorem 2 and the classical view of online IL as mitigating error accumulation. A classic account is that online interaction mitigates the compounding of per-step prediction errors over long horizons, leading to DAgger-style algorithms [35, 46, 48]. In contrast, our work studies the information bottleneck that arises in non-realizable settings, an orthogonal limitation of offline imitation learning that already appears in contextual bandits, i.e., when H = 1. We further characterize how the reward function interacts with the misspecification structure to enable online imitation learning in non-realizable settings, a perspective absent from previous DAgger-style analyses. Finite-Sample Guarantee. The efficient training of online and on-policy algorithms for LLM post-training (including RL and on-policy distillation) crucially relies on a strong base model, so that the model can receive positive feedback with nontrivial probability [9, 67]. This requirement can be characterized theoretically through coverage coefficients defined with respect to the base policy (e.g., Foster et al. [16], Huang et al. [25, 26], Xie et al. [61]): ⋆ 2 ⋆ π (y|x) base ⋆ base ⋆ := C∞ (π ) := supx∈X ,y∈Y ππbase(y|x) , C (π ) E E , base x∼D base y∼π 2 x (y|x) π (y|x) where π ⋆ ∈ Π is a reference policy (e.g., an optimal student policy). Besides coverage, we also require a boundedness condition on the shaped expert signal, which plays the same role as the standard bounded-reward condition in finite-sample RL and off-policy evaluation analyses. Assumption 2 (Bounded shaped expert signal). For all x ∈ X and y ∈ Y, |f (π e (y | x))| ≤ Vmax . Using standard pessimism techniques [17, 54, 59], one can extract a finite-sample result for the objective Equation (2) under such a condition. Below we present the finite-sample guarantee, and defer details and proof to Appendix A.3. Theorem 3. Suppose Assumption 2. For a base model π base , a reference student policy π ⋆ , with probability 1 − δ, the learned policy π̂ in Equation (7) satisfies r base ⋆ 2 C2base (π ⋆ )Vmax log(|Π|/δ) C∞ (π )Vmax log(|Π|/δ) ⋆ Jon (π ) − Jon (π̂) ≲ + . N N 11
e Compared to the lower bound for offline IL (Theorem 1), the coverage dependence improves from C∞ to base ⋆ C∞ (π ). Since the base policy is the model before fine-tuning, it is expected to cover policies reachable during training better than an external expert. While a strong base model appears necessary for efficient online algorithms, it remains open whether its role can be characterized by a finer notion than these coverage coefficients.
We also note that the hard instances in Theorem 1 can also be constructed to satisfy Assumption 1 and Assumption 2, yielding a clean offline–online separation within the same regime. See Remark 2 for further discussion. Empirical Validation. We further provide an empirical validation of the misspecification–reward alignment condition (Assumption 1). Following the setup of Li et al. [33], we start from the same base model, DeepSeekR1-Distill-Qwen-1.5B [9], and perform on-policy distillation with two different experts: DeepSeek-R1-DistillQwen-7B (R1-Distill-7B) [9] and Skywork-OR1-Math-7B (Skywork-7B) [22]. On AIME 2024 and AIME 2025, distillation from Skywork-7B improves the student’s average performance from 25.0% to 32.8%, whereas distillation from R1-Distill-7B improves it to 27.1% (averaged over three runs). To empirically assess Assumption 1, we estimate the suboptimality bound in Theorem 2. The decomposition in Equation (6) is not unique: each choice of α > 0 induces a residual upper bound δα (ϵ). Since the bound in Theorem 2 depends on this choice through (ϵ + δα (ϵ))/α, we further minimize the estimated bound over α. We estimate the bound using the following procedure. For each expert, we perform three on-policy distillation runs. We collect checkpoints from all distillation trajectories across the two experts, b We take the checkpoint with checkpoints saved every 10 steps, and use their union as a proxy policy class (Π). ⋆ with the highest empirical reward as the reference policy π b . Then, for each expert separately and for each ϵ n> 0, we consider the set of checkpointso whose empirical online IL objective is within ϵ of that of π b⋆ : b ϵ := π ∈ Π b : Jbon (π) ≥ max ′ b Jbon (π ′ ) − ϵ . We estimate the residual upper bound δα (ϵ) in Assumption 1 Π π ∈Π b over Πϵ by Monte Carlo, and denote the resulting estimate by δbα (ϵ). We then report the estimated bound ϵ + δbα (ϵ) b ∆(ϵ) := min . α>0 α Details are provided in Section C.5. As shown in Figure 4, Skywork-7B yields a much smaller estimated bound than R1-Distill-7B across ϵ, consistent with its larger distillation improvement. The relatively large range of ϵ is expected, since the online IL objective is based on sequence-level log probabilities, whose scale grows approximately linearly with response length.
6
Conclusion and Discussion
In this work, we show that misspecification is a key factor driving the empirical performance gap between offline and online IL methods in LLM post-training. Our results help clarify what leads to the empirical performance gap between offline and online IL methods in LLM post-training, and suggest several directions that may be worth exploring in future work. Empirically, in realizable settings, offline IL already recovers expert performance, leaving no room for online interaction to help. In this sense, the gap often observed between SFT and online methods such as RLVR or on-policy distillation does not necessarily come from some inherent algorithm properties. Instead, it can arise from the offline data distribution induced by a non-realizable expert, which may be poorly suited for the target student class. On the theoretical front, we characterize an information-theoretic limitation of offline IL under misspecification, and provide structural conditions under which online IL can still be effective even when the discrepancy between expert and student is large. This perspective complements prior explanations based on error accumulation. An 12
important question not fully explained by the current framework is why on-policy distillation can empirically achieve better training efficiency than RLVR. Answering this question may require analyzing the multi-step structure of language generation and characterizing how dense reward signals improve learning efficiency in this setting.
Acknowledgments The authors thank Audrey Huang, Abhishek Panigrahi, Aditi Raghunathan, Dhruv Rohatgi, Kaiyue Wen and Chen Wu for valuable discussions and feedback. B.L. is supported by the Kempner Fellowship from the Kempner Institute at Harvard. A.R. and J.G. are supported in part by NSF awards IIS-2211907, CCF-2238523, IIS-2403275, an Amazon Research Award, ONR award N000142512124, a Google Research Scholar Award, and an OpenAI Superalignment Fast Grant.
References [1] Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004. [2] Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024. [3] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [4] Chang, J., Uehara, M., Sreenivas, D., Kidambi, R., and Sun, W. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems, 34: 965–979, 2021. [5] Chen, H., Razin, N., Narasimhan, K., and Chen, D. Retaining by doing: The role of on-policy data in mitigating forgetting. arXiv preprint arXiv:2510.18874, 2025. [6] Chen, J. and Jiang, N. Information-theoretic considerations in batch reinforcement learning. In International conference on machine learning, pp. 1042–1051. PMLR, 2019. [7] Chu, T., Zhai, Y., Yang, J., Tong, S., Xie, S., Schuurmans, D., Le, Q. V., Levine, S., and Ma, Y. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025. [8] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv: 2110.14168, 2021. [9] DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948. [10] Espinosa-Dice, N., Choudhury, S., Sun, W., and Swamy, G. Efficient imitation under misspecification. arXiv preprint arXiv:2503.13162, 2025. [11] Face, H. Open r1: A fully open reproduction of deepseek-r1, 2025. [12] Foster, D. and Rakhlin, A. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International conference on machine learning, pp. 3199–3210. PMLR, 2020.
13
[13] Foster, D., Agarwal, A., Dudík, M., Luo, H., and Schapire, R. Practical contextual bandits with regression oracles. In International Conference on Machine Learning, pp. 1539–1548. PMLR, 2018. [14] Foster, D. J. and Rakhlin, A. Foundations of reinforcement learning and interactive decision making. arXiv preprint arXiv:2312.16730, 2023. [15] Foster, D. J., Block, A., and Misra, D. Is behavior cloning all you need? understanding horizon in imitation learning. Advances in Neural Information Processing Systems, 37:120602–120666, 2024. [16] Foster, D. J., Mhammedi, Z., and Rohatgi, D. Is a good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration. arXiv preprint arXiv:2503.07453, 2025. [17] Gabbianelli, G., Neu, G., and Papini, M. Importance-weighted offline learning done right. In International Conference on Algorithmic Learning Theory, pp. 614–634. PMLR, 2024. [18] Gandhi, K., Chakravarthy, A., Singh, A., Lile, N., and Goodman, N. D. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307, 2025. [19] Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [20] Gu, Y., Dong, L., Wei, F., and Huang, M. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024. [21] Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., and Song, D. The false promise of imitating proprietary language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=Kz3yckpCN5. [22] He, J., Liu, J., Liu, C. Y., Yan, R., Wang, C., Cheng, P., Zhang, X., Zhang, F., Xu, J., Shen, W., et al. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312, 2025. [23] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021. [24] Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. [25] Huang, A., Block, A., Foster, D. J., Rohatgi, D., Zhang, C., Simchowitz, M., Ash, J. T., and Krishnamurthy, A. Self-improvement in language models: The sharpening mechanism. In The Thirteenth International Conference on Learning Representations, 2025. [26] Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishnamurthy, A., and Foster, D. J. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. In The Thirteenth International Conference on Learning Representations, 2025. [27] Jiang, W., Lu, Y., Lin, H., Han, X., and Sun, L. Teach small models to reason by curriculum distillation. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 7412–7422, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.376. URL https://aclanthology.org/2025.emnlp-main.376/. [28] Kim, J., Yun, J., Lee, J. D., and Jun, K.-S. Coverage improvement and fast convergence of on-policy preference learning. arXiv preprint arXiv:2601.08421, 2026. [29] Kim, Y. and Rush, A. M. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp. 1317–1327, 2016.
14
[30] Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: //openreview.net/forum?id=UYneFzXSJWh. [31] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [32] Li, Y., Yue, X., Xu, Z., Jiang, F., Niu, L., Lin, B. Y., Ramasubramanian, B., and Poovendran, R. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502.12143, 2025. [33] Li, Y., Zuo, Y., He, B., Zhang, J., Xiao, C., Qian, C., Yu, T., Gao, H.-a., Yang, W., Liu, Z., et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016, 2026. [34] Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. [35] Lu, K. and Lab, T. M. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. [36] Luo, M., Tan, S., Wong, J., Shi, X., Tang, W. Y., Roongta, M., Cai, C., Luo, J., Zhang, T., Li, L. E., et al. Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl. Notion Blog, 2025. [37] Munos, R. Error bounds for approximate policy iteration. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, pp. 560–567, 2003. [38] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. [39] Pan, J., Zhang, J., Wang, X., Yuan, L., Peng, H., and Suhr, A. Tinyzero. https://github.com/JiayiPan/TinyZero, 2025. Accessed: 2025-01-24. [40] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PmLR, 2021. [41] Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. [42] Rajaraman, N., Yang, L., Jiao, J., and Ramchandran, K. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 33:2914–2924, 2020. [43] Rajaraman, N., Han, Y., Yang, L., Liu, J., Jiao, J., and Ramchandran, K. On the value of interaction and function approximation in imitation learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1325–1336. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/ paper/2021/file/09dbc1177211571ef3e1ca961cc39363-Paper.pdf. [44] Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34: 11702–11716, 2021. [45] Rohatgi, D., Block, A., Huang, A., Krishnamurthy, A., and Foster, D. J. Computational-statistical tradeoffs at the next-token prediction barrier: Autoregressive and imitation learning under misspecification. arXiv preprint arXiv:2502.12465, 2025. 15
[46] Ross, S. and Bagnell, D. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661–668. JMLR Workshop and Conference Proceedings, 2010. [47] Ross, S. and Bagnell, J. A. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979, 2014. [48] Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627–635. JMLR Workshop and Conference Proceedings, 2011. [49] Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv: 2402.03300, 2024. [50] Shenfeld, I., Pari, J., and Agrawal, P. Rl’s razor: Why online reinforcement learning forgets less. arXiv preprint arXiv:2509.04259, 2025. [51] Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024. [52] Simchi-Levi, D. and Xu, Y. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research, 47(3):1904–1931, 2022. [53] Song, Y., Rohatgi, D., Singh, A., and Bagnell, J. A. To distill or decide? understanding the algorithmic trade-off in partially observable reinforcement learning. arXiv preprint arXiv: 2510.03207, 2025. [54] Swaminathan, A. and Joachims, T. The self-normalized estimator for counterfactual learning. advances in neural information processing systems, 28, 2015. [55] Swamy, G., Rajaraman, N., Peng, M., Choudhury, S., Bagnell, J. A., Wu, S., Jiao, J., and Ramchandran, K. Minimax optimal online imitation learning via replay estimation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. [56] Syed, U. and Schapire, R. E. A game-theoretic approach to apprenticeship learning. Advances in neural information processing systems, 20, 2007. [57] Team, Q. Qwen2. 5: A party of foundation models, september 2024. URL https://qwenlm. github. io/blog/qwen2, 5(4), 2024. [58] von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. TRL: Transformers Reinforcement Learning, 2020. URL https://github.com/ huggingface/trl. [59] Wang, L., Krishnamurthy, A., and Slivkins, A. Oracle-efficient pessimism: Offline policy optimization in contextual bandits. In International Conference on Artificial Intelligence and Statistics, pp. 766–774. PMLR, 2024. [60] Xiao, B., Xia, B., Yang, B., Gao, B., Shen, B., Zhang, C., He, C., Lou, C., Luo, F., Wang, G., et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026. [61] Xie, T., Foster, D. J., Krishnamurthy, A., Rosset, C., Awadallah, A. H., and Rakhlin, A. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf. In The Thirteenth International Conference on Learning Representations, 2025. [62] Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprint arXiv:2312.11456, 2023. 16
[63] Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [64] Yang, W., Liu, W., Xie, R., Yang, K., Yang, S., and Lin, Y. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026. [65] Ye, T., Dong, L., Chi, Z., Wu, X., Huang, S., and Wei, F. Black-box on-policy distillation of large language models. CoRR, abs/2511.10643, 2025. doi: 10.48550/ARXIV.2511.10643. URL https: //doi.org/10.48550/arXiv.2511.10643. [66] Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L., et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. [67] Zeng, W., Huang, Y., Liu, Q., Liu, W., He, K., Ma, Z., and He, J. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892, 2025. [68] Zhao, S., Xie, Z., Liu, M., Huang, J., Pang, G., Chen, F., and Grover, A. Self-distilled reasoner: On-policy self-distillation for large language models. CoRR, abs/2601.18734, 2026. doi: 10.48550/ARXIV.2601.18734. URL https://doi.org/10.48550/arXiv.2601.18734. [69] Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., and Ma, Y. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http://arxiv.org/abs/2403.13372. [70] Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433–1438. Chicago, IL, USA, 2008.
17
A
Proofs and Additional Results
A.1
Proof of Theorem 1
Without loss of generality, assume that K := |Π| = 2l for some l ∈ N∗ . If not, let K ′ = 2⌊log2 K⌋ ≤ K. We construct the hard instance using only K ′ policies and augment the policy class with K − K ′ dummy policies. Since the original hard instance is embedded in this larger class, this augmentation cannot decrease the minimax risk. To prove the lower bound of the optimal suboptimality, we construct a joint distribution P over instances C and experts π e such that for any offline learner π̂(D) based on N i.i.d. expert demonstrations D, e C∞ log |Π| E(πe ,C)∼P [1 − E [V (π̂)]] ≳ min 1, . N Let X = {x1 , . . . , xl } and Y = {y1 , y2 , y3 }. Define the student policy class Π to consist of deterministic policies that, for each context xi , choose either y2 or y3 : ∀i ∈ [l], π(· | xi ) ∈ {δy2 , δy3 }. Thus |Π| = 2l = K. Let the context distribution Dx satisfy Dx (xi ) = ζ for i ≤ l − 1 and Dx (xl ) = 1 − (l − 1)ζ. Moreover, let S := {y2 , y3 }l , and index environments by s = (s1 , . . . , sl ) ∈ S. For each s ∈ S, define the expert policy πse by, for all i ∈ [l], πse (· | xi ) = 1 − C1e δy1 + C1e δsi . ∞
Define the reward function rs by
∞
rs (xi , y) = 1[y = y1 or y = si ] .
The optimal student policy is πs⋆ (· | xi ) = δsi for all i ∈ [l]. Let P denote the uniform prior over instances induced by S: draw S ∼ Unif(S) and set the instance to be (Dx , rS , πSe ). Let X (D) be the context for which the secret action si is covered by i.i.d. samples in dataset D. Then E(πe ,C)∼P E[1 − V (π̂)] = EE(πe ,C)∼P [1 − V (π̂)] " l−1 # ζX ≥ EE(πe ,C)∼P 1 [i ̸∈ X (D)] 2 i=1 " l−1 # X 1 ≥ E ζ Pr [i ̸∈ X (D)] . 2 i=1 e For each i ≤ l − 1, the probability that i becomes covered in one sample is Pr(x = xi , y ̸= y1 ) = ζ/C∞ . e /N } gives Setting ζ = min{ 1l , C∞
Pr(i ∈ / X (D)) =
ζ 1− e C∞
N ≳ 1,
and therefore E(πe ,C)∼P E[1 − V (π̂)] ≳ ζl. Plugging back yields
C e log |Π| E(πe ,C)∼P ED [1 − V (π̂(D))] ≳ ζl ≍ min 1, ∞ N
as was to be shown. 18
A.2
Proof of Theorem 2
Fix any π ∈ Π that is ϵ-suboptimal for the online IL objective Equation (2), i.e., Jon (π) ≥ max Jon (π ′ ) − ϵ ≥ Jon (π ⋆ ) − ϵ. ′ π ∈Π
By Assumption 1, there exist α > 0, h, and b such that for all (x, y), f (π e (y | x)) = α r(x, y) + h(x, y) + b(x). Taking expectation over x ∼ Dx and y ∼ π(· | x) yields Jon (π) = Ex∼Dx Ey∼π(·|x) f (π e (y | x)) = α Ex∼Dx Ey∼π(·|x) r(x, y) + Ex∼Dx Ey∼π(·|x) h(x, y) + Ex∼Dx b(x) = α V (π) + Ex∼Dx Ey∼π(·|x) h(x, y) + Ex∼Dx b(x). The same decomposition holds for π ⋆ : Jon (π ⋆ ) = α V (π ⋆ ) + Ex∼Dx Ey∼π⋆ (·|x) h(x, y) + Ex∼Dx b(x). Subtracting the two identities cancels the b(x) term and gives h i Jon (π) − Jon (π ⋆ ) = α V (π) − V (π ⋆ ) + Ex∼Dx Ey∼π(·|x) h(x, y) − Ey∼π⋆ (·|x) h(x, y) . Rearranging, h i α V (π ⋆ ) − V (π) = Jon (π ⋆ ) − Jon (π) + Ex∼Dx Ey∼π(·|x) h(x, y) − Ey∼π⋆ (·|x) h(x, y) . Now use Jon (π) ≥ Jon (π ⋆ ) − ϵ, which implies Jon (π ⋆ ) − Jon (π) ≤ ϵ. Moreover, since Jon (π) ≥ Jon (π ⋆ ) − ϵ, the second part of Assumption 1 applies and yields h i Ex∼Dx Ey∼π(·|x) h(x, y) − Ey∼π⋆ (·|x) h(x, y) ≤ δ(ϵ). Combining these two bounds,
α V (π ⋆ ) − V (π) ≤ ϵ + δ(ϵ),
and dividing by α > 0 finishes the proof.
A.3
Finite Sample Guarantee of Learning Objective Equation (2)
The online IL objective Equation (2) is typically optimized in an online and on-policy manner in LLM post-training, as in on-policy distillation [20, 35], where the learner adaptively generates responses according to the current student policy (similarly to reinforcement learning). In practice, it is often optimized in an RL framework. The sample complexity analysis of on-policy algorithms for a general policy class is challenging. Existing results typically require additional assumptions, such as reverse-KL regularization and realizability of the policy class, or study simplified policy classes [16, 28, 61]. Other online contextual-bandit results study indirect learning formulations, where policies are learned through reward/value-function estimation rather than direct policy learning [12, 13, 52]. These results do not directly apply to our setting, where the learner optimizes over a general policy class. Therefore, we instead present a simpler algorithm based on pessimistic policy optimization [17, 54, 59], which only samples from the base policy and queries expert feedback, yet already yields meaningful improvements over offline imitation learning.
19
Concretely, the learner draws N i.i.d. samples with xi ∼ Dx and yi ∼ π base (· | xi ), and queries the expert signal f (π e (yi | xi )). For any policy π ∈ Π, define the inverse-propensity weighted estimator N
1 X π(yi | xi ) Jbon (π) := f (π e (yi | xi )), N i=1 π base (yi | xi ) and the regularizer
r b(π) :=
base 2 log(2|Π|/δ) 2C∞ 2C2base (π)Vmax (π)Vmax log(2|Π|/δ) + , N 3N
where base C∞ (π) :=
π(y|x) , C2base (π) := Ex∼Dx Ey∼πbase base (y|x) x∈X ,y∈Y π
sup
π(y|x) base π (y|x)
2 .
The learned policy π̂ is defined as π̂ = arg max Jbon (π) − b(π).
(7)
π∈Π
Note that b(π) can be evaluated without querying the expert, and therefore does not contribute to the statistical complexity of expert feedback. We now prove Theorem 3. π(y|x) Proof of Theorem 3. For a policy π, denote wπ (x, y) = πbase , Ziπ = wπ (xi , yi )f (π e (yi | xi )). Then (y|x)
N 1 X π Jbon (π) = Z , N i=1 i
Jon (π) = E[Ziπ ]. So N 1 X π Jbon (π) − Jon (π) = (Z − EZiπ ). N i=1 i
Since base |Ziπ | ≤ wπ (x, y)|f (π e (yi | xi ))| ≤ Vmax C∞ (π), 2 2 Var(Ziπ ) ≤ E[(Ziπ )2 ] = Ex∼Dx Ey∼πbase (·|x) [wπ (x, y)2 Vmax ] ≤ Vmax C2base (π),
Bernstein inequality gives Pr
δ Jbon (π) − Jon (π) > b(π) ≤ . |Π|
By union bound, with probability at least 1 − δ, it holds for any π ∈ Π that Jbon (π) − Jon (π) ≤ b(π).
20
Under this event, Jbon (π̂) ≤ Jon (π̂) + b(π̂), Jon (π ⋆ ) ≤ Jbon (π ⋆ ) + b(π ⋆ ). Thus Jon (π̂) ≥ Jbon (π̂) − b(π̂) ≥ Jbon (π ⋆ ) − b(π ⋆ ) ≥ Jon (π ⋆ ) − 2b(π ⋆ ).
Remark 2 (Online–Offline Separation under the Structural Conditions). We note that the hard instances used in the offline lower bound (Theorem 1) can also be chosen to satisfy the additional conditions required by the online upper bound, including Assumption 1 and the Vmax -bounded signal condition, by taking a truncated identity shaping function and choosing α appropriately. Indeed, consider the hard instance indexed by s in the proof of Theorem 1. Recall that for each context xi , the expert policy is given by 1 1 − C e , y = y 1 , πse (y | xi ) =
∞
1 e , C∞
y = si ,
0,
otherwise.
Take the shaping function f : R → R to be the truncated identity 1 f (p) := min p, e . C∞ Applying this shaping function to the expert probability gives 1 C e , y = y1 , f (πse (y | xi )) =
∞
1 e , C∞
0,
y = si ,
otherwise.
Since student policies only choose between y2 and y3 , the relevant signal values are f (πse (si | xi )) =
1 , e C∞
f (πse (y | xi )) = 0
for y ∈ {y2 , y3 } \ {si }.
e Thus the signal gap between the optimal student action and the suboptimal student action is exactly 1/C∞ , while the corresponding reward gap is 1. e Thus f (πse (y | x)) = C1e r(x, y). Hence Assumption 1 holds with α = 1/C∞ and δ(ϵ) = 0. Moreover, the ∞ e shaped signal is Vmax -bounded with Vmax = 1/C∞ .
Therefore, converting the online estimation error ϵ into suboptimality through ϵ/α gives ! r base ⋆ 2 1 C2base (π ⋆ )Vmax log(2|Π|/δ) C∞ (π )Vmax log(2|Π|/δ) 1 − V (π̂on ) ≲ + α N N r base ⋆ C2base (π ⋆ ) log(2|Π|/δ) C∞ (π ) log(2|Π|/δ) = + . N N e In particular, after substituting α = Vmax = 1/C∞ , the online guarantee no longer scales with the experte coverage coefficient C∞ . By contrast, Theorem 1 shows that any offline IL algorithm must incur C e log |Π| 1 − E[V (π̂)] ≳ min 1, ∞ . N
21
Thus, even within the regime where Assumption 1 and the bounded-signal condition hold, offline IL must pay the expert-coverage dependence, whereas online IL depends instead on the typically more favorable base-policy coverage terms. This gives a clean offline–online separation under the structural conditions. p Although the online IL guarantee contains a slower term C2base (π ⋆ )/N , the coverage coefficient appearing base ⋆ there, C2base (π ⋆ ), can be much smaller than the worst-case coverage coefficient C∞ (π ).
22
B
Additional Experiments
Out-of-distribution generalization and catastrophic forgetting. We evaluate the model’s out-ofdistribution (OOD) generalization on the Countdown task, where test instances use a larger number range than those seen during training. We also assess catastrophic forgetting during GSM8K training by measuring general language capability on the MMLU benchmark [23]. Figure 5 shows that both SFT and on-policy distillation with a realizable expert achieve OOD performance matching that of the expert, whereas SFT with a non-realizable expert performs substantially worse and exhibits a larger gap between in-distribution and OOD performance. We observe a similar pattern for forgetting: under SFT with a realizable expert or on-policy distillation, the MMLU score decreases only slightly throughout training, with a final drop within 0.5 points; in contrast, under a non-realizable expert, it drops more substantially, by more than 2 points. These findings refine prior observations that SFT tends to generalize worse OOD [7] and forget more than RL [5, 50]. Instead, our results suggest this gap is not inherent to the algorithm, but arises primarily under non-realizability. It is the training data distribution, rather than the SFT algorithm itself, that drives inferior OOD generalization and severe forgetting. 0.7
Countdown (OOD)
0.6
Performance
Performance
0.5 0.4 0.3 0.2 0.1 0.0 0
MMLU
54.5 54.0 53.5 53.0 52.5 52.0 51.5 51.0
100
200
Steps
SFT (realizable expert) SFT (non-realizable expert)
300
0
100
On-policy distillation (realizable expert) RL
200
Steps
300
Base model performance Realizable expert performance
Figure 5: OOD generalization and catastrophic forgetting. Left: Models are evaluated on Countdown instances with larger number range. Right: Catastrophic forgetting during GSM8K training. Models are evaluated on MMLU benchmark. Both on-policy distillation and SFT from a realizable expert achieve strong OOD performance that matches the expert, whereas SFT from a non-realizable expert yields much worse OOD generalization and exhibits more severe forgetting.
C
Experimental Details
C.1
Countdown Task Countdown Task Example [INST] Using the numbers [5, 94, 9, 44], create an equation that equals 93. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer>(1 + 2) / 3</answer>. [/INST] Let me solve this step by step.
23
Experimental Setup. We utilize the dataset introduced by Pan et al. [39], which comprises 327,680 training instances and 1,024 test samples.5 To illustrate the task format, we provide a representative training prompt. Our implementation is built upon the official codebase of Pan et al. [39],6 incorporating specific adaptations for NVIDIA A100 execution from a community fork.7 RL Training Details. We follow the training setup of Pan et al. [39] and provide the details below for completeness. We fine-tune the Qwen2.5-3B-Instruct model [57] using Reinforcement Learning (RL) for 320 steps on 2 NVIDIA A100 GPUs. The training configuration employs a global batch size of 128 with 5 rollouts per prompt, and a mini-batch size of 64 for optimization. We set the learning rate to 1 × 10−6 and the KL penalty coefficient to βKL = 1 × 10−3 . The maximum generation length is capped at 1024 tokens. We use a reward function that assigns 1.0 to correct responses, 0.1 to incorrect but format-compliant responses, and 0 to invalid outputs. Table 1: RL training details for the Countdown task. Parameter
Value
Max prompt length Clip ratio Validation temperature Rollout temperature Validation top-p Validation top-k
2,048 0.2 1 1 0.7 50
SFT Training Details. For training with both realizable and non-realizable experts, we use the same SFT configuration. We perform one-pass supervised fine-tuning with LlamaFactory on Qwen2.5-3B using full-parameter training. The model is trained for 3,200 optimization steps with a maximum sequence length of 2,048 tokens. We use a per-device batch size of 16, a learning rate of 1 × 10−5 , a cosine learning rate schedule, and a warmup ratio of 0.1. For each question, we generate 5 different solutions, matching our RL setup, where 5 rollouts are generated for each question. For SFT with a realizable (RL-trained) expert, we roll out the expert with temperature 1.0 and disable both top-k and top-p filtering. For the GSM8K and DeepScaleR experiments, we use the same SFT configuration, except that the batch size and maximum sequence length are matched to the corresponding RL setting. We therefore do not repeat these details separately. One RL training step consists of multiple gradient updates. In Figure 1, we report progress in terms of RL steps. To align SFT with RL on the horizontal axis, we rescale SFT steps by a constant factor (/10 for Countdown and /16 for GSM8K and DeepScaleR). On-Policy Distillation Training Details. We use TRL’s MiniLLM trainer [20, 58]. We train for 3,200 optimization steps on 8 GPUs. We use a per-device batch size of 4 and a global batch size of 64, which corresponds to gradient accumulation over 2 steps. For each prompt, the student samples 4 completions on-policy. The maximum prompt length is 256 tokens and the maximum completion length is 1,024 tokens. We use a learning rate of 1 × 10−5 . Unless otherwise noted, other optimization settings follow the default TRL MiniLLM configuration, including AdamW, a linear learning-rate schedule with no warmup, and dropout disabled. 5 https://huggingface.co/datasets/Jiayi-Pan/Countdown-Tasks-3to4 6 https://github.com/Jiayi-Pan/TinyZero 7 https://github.com/JerryWu-code/TinyZero
24
For the GSM8K experiments, we use a learning rate of 5 × 10−6 , while for DeepScaleR we use 1 × 10−5 . The sequence length in each setup is matched to that used in the corresponding RL training. All other configurations follow the Countdown setup. Evaluation Details. During evaluation, we sample with temperature 1.0, top-p 0.7, and top-k 50. For each test example, we generate 8 responses and report the averaged accuracy. We set the maximum generation length to 1024 tokens. OOD Evaluation Task. In Section 4 and Appendix B, we evaluate both in-distribution (ID) and out-ofdistribution (OOD) performance for the Countdown task. For OOD evaluation, we use instances with four operands only, with larger input and answer ranges. Table 2: Detailed settings for Countdown tasks used in ID and OOD evaluation.
C.2
Setting
Number of Operands
Range of Input
Range of Answer
Train / ID OOD
3–4 4
100 125
100 300
GSM8K Task
Experimental Setup. We conduct experiments on the GSM8K benchmark [8], using the original train/test split, which contains approximately 7.47K training examples and 1.32K test examples. We use Llama-3.2-2B [19] as the base model. Warmup Supervised Fine-Tuning. To enable RL training, we first perform supervised fine-tuning on the OpenR1 dataset [11] to strengthen the model’s reasoning capabilities. We use the LlamaFactory [69] framework and train for 3,064 steps with batch size 64 and learning rate 10−5 using the AdamW optimizer. RL Training Details. We then perform RL training on the GSM8K training split for 300 steps using the DeepScaleR codebase [36], which is built upon verl [51]. A response receives reward 1 if its extracted final answer exactly matches the ground-truth answer, and 0 otherwise. Table 3: RL training details for GSM8K experiments. Parameter
Value
Train batch size PPO mini-batch size Max prompt length Max response length Rollout temperature Rollout top-k Rollout top-p Responses per prompt KL coefficient
128 64 512 4,096 0.6 50 0.95 8 0.001
Evaluation Details. During evaluation, we sample with temperature 0.6, top-p 0.95, and top-k 50. For MMLU evaluation, we use temperature 0.95, top-p 0.7, and top-k 50, and report the category-averaged MMLU accuracy. 25
Table 4: Empirical estimation of quantities in Assumption 1. Quantity
Empirical proxy PM 1 Vb (π) = M i=1 r(xi , yi ) PM b 1 b Jon (π) = f (xi , yi )
V (π) = Ex Ey∼π(·|x) r(x, y)s Jon (π) = Ex Ey∼π(·|x) f (π e (y | x))
M
policy class Π
b Collected 162 checkpoints Π n o b ϵ := π ∈ Π b : Jbon (π) ≥ max ′ b Jbon (π ′ ) − ϵ Π π ∈Π π̂ ⋆ = arg max b Vb (π)
ϵ-optimal policies w.r.t. Jon Reference policy π ⋆
C.3
i=1
π∈Π
DeepScaleR Task
Experimental Setup. We follow the setup of DeepScaleR [36] training with their 40k dataset, and evaluating with MATH [24] and AIME2024 and AIME 2025 benchmark. We use DeepSeek-R1-Distill-Qwen1.5B [9] as the base model. RL Training Details. We perform RL training using the DeepScaleR codebase [36], which is built on top of verl [51]. We follow the training configuration of the first stage (8k context length) of Luo et al. [36]. Specifically, for each prompt, we sample 8 responses with temperature 0.6. The maximum prompt length is 1,024 tokens and the maximum response length is 8,192 tokens. We use a train batch size of 128, a PPO mini-batch size of 64, an actor learning rate of 1 × 10−6 , and a KL coefficient of 0.001. We use binary reward. Evaluation Details.
C.4
During evaluation, we sample with temperature 0.6, top-p 0.95, and top-k 50.
Experimental Details for Section 5.1
For both models, we generate responses with temperature 0.6, top-p 0.95, and a maximum generation length of 100,000 tokens. We evaluate both models on the AIME 2024 and AIME 2025 benchmarks. For each benchmark, each model generates 14 responses per question, yielding 840 responses per model in total. The response length is measured using the Qwen3-8B tokenizer for both models.
C.5
Experimental Details for Section 5.3
Training Setup. We reproduce the DeepSeek-family setting in Figure 4 (left) of Li et al. [33]. For completeness, we repeat the experimental details below. We use DeepSeek-R1-Distill-Qwen-1.5B as the student model. We consider two expert policies: DeepSeek-R1-Distill-Qwen-7B and Skywork-OR1-Math-7B. We train on the DAPO-Math-17K dataset [66] for 270 steps and evaluate performance on AIME24 and AIME25. We use the OPD training pipeline implemented with verl, using the PPO trainer in the verl [51] codebase together with vLLM [31] for rollout generation. We disable KL regularization and use token-mean loss aggregation. Both runs share the same optimization hyperparameters and differ only in the expert model. We train with learning rate 10−6 for 270 steps on 8 GPUs. The rollout temperature is 1.0, and we sample 4 responses per prompt during training. We use top-k truncation with k = 16. We use a maximum prompt length of 1024, a maximum response length of 7168, and a maximum generation length of 31744 for validation. Estimating the suboptimality bound. We perform three distillation runs for each expert and collect b For each checkpoints every 10 steps, obtaining 162 checkpoints in total. We denote this checkpoint set by Π. b we sample M = 600 responses on AIME 2024 and AIME 2025. checkpoint policy Π, 26
Figure 6: The on-policy distillation results in Section 5.3. Starting from the same base model, DeepSeek-R1-Distill-Qwen-1.5B, we perform on-policy distillation using two 7B experts: R1-Distill-7B and Skywork-7B. Distillation from Skywork-7B improves average performance on AIME 2024 and 2025 from 25.0% to 32.8%, whereas distillation from R1-Distill-7B improves it to 27.1% (averaged over three runs). Each sample consists of a benchmark prompt xi and a response yi ∼ π(· | xi ). The reward value is estimated over these M responses, i.e., M 1 X r(xi , yi ), Vb (π) := M i=1 where r(xi , yi ) is the correctness reward. For the expert score, we use the expert log-probability, i.e. f (π e (y | x)) = log π e (y | x), as is commonly done in on-policy distillation for training stability. This gives M 1 X Jbon (π) := log π e (y | x). M i=1
The resulting estimates are summarized in Table 4. We estimate the suboptimality bound in Theorem 2 separately for each expert. To simplify notation, we b omit the expert index in the estimators below. Let π b⋆ ∈ arg maxπ∈Π b V (π) be the empirically best checkpoint policy in terms of reward value. Using the decomposition in Equation (6), for a fixed α > 0 we define the empirical residual b α (π) := Jbon (π) − αVb (π) H for some α to be decided later. The action-independent offset b(x) cancels when comparing two policies, and b ϵ: therefore does not need to be estimated. We then estimate the residual variation over Π h i b α (π) − H b α (b δbα := sup H π⋆ ) . bϵ π∈Π
b := minα>0 (ϵ + δbα )/α. Finally, we report the estimated suboptimality bound ∆(ϵ)
27