arXiv CSopen access

Self-Distilled Agentic Reinforcement Learning

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

Self-Distilled Agentic Reinforcement Learning Zhengxi Lu1,2 ,∗ Zhiyuan Yao1,2 , Zhuowen Han2 , Zi-Han Wang2,3 , Jinyang Wu3 Qi Gu2†, Xunliang Cai2 , Weiming Lu1 , Jun Xiao1 , Yueting Zhuang1 , Yongliang Shen1† 1 Zhejiang University

2 Meituan

{zhengxilu, syl}@zju.edu.cn

3 Tsinghua University

[email protected]

arXiv:2605.15155v1 [cs.LG] 14 May 2026

Abstract Reinforcement learning (RL) has emerged as a central paradigm for posttraining LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL–OPSD baselines across model scales. Code available: https://github.com/ZJU-REAL/SDAR.

Figure 1: (a) Comparison between GRPO+OPSD and SDAR; (b) Overall Performance. ∗ Work done during internship at Meituan. † Corresponding author

Introduction

Agentic post-training has become a central challenge for Large Language Models (LLMs) (Guo et al., 2025; Team et al., 2025; Yang et al., 2025; Comanici et al., 2025; Team et al., 2026b). Unlike static single-turn reasoning, multi-turn agents interact with environments over extended horizons, where each action changes future observations and each generated response becomes part of the context for subsequent decisions (Shen et al., 2023; Shi et al., 2025; Jimenez et al., 2023). Two paradigms naturally emerge as complementary forces: Reinforcement Learning (RL) (Shao et al., 2024; Dong et al., 2025; Feng et al., 2025) provides task-level optimization grounded in environment or verifier feedback, whereas On-Policy Distillation (OPD) (Ye et al., 2026; Yang et al., 2026b; Team et al., 2026a; GLM-5-Team et al., 2026) and On-Policy Self-Distillation (OPSD) (Zhao et al., 2026; He et al., 2026; Zhang et al., 2026) provide dense token-level guidance from a teacher branch. Yet, OPSD does not transfer cleanly to multiturn agent training. We attribute this to two observations: [1] Multi-turn OPSD Instability and [2] Asymmetric Trust in Privileged Guidance.

KL Divergence

KL Loss

Success Rate

[Observation-1] Multi-turn OPSD Instability Once the student agent inevitably drifts 0.35 from the teacher-supported traMulti-turn OPSD GRPO+OPSD 0.3 GRPO jectory, the once-helpful token0.30 RLSD level supervision becomes increas0.2 0.25 ingly unreliable. This compound0.20 0.1 ing error leads to surging perTraining Steps 3.0 0.15 turn KL divergence and catas2.5 0.10 trophic degradation in task per2.0 0.05 formance, as shown in Figure 2 1.5 Multi-turn OPSD 1.0 0.00 (Left). TCOD (Wang et al., 2026b) 0 5 10 15 20 0 50 100 150 attempts to address this through Training Steps Turn Step curriculum learning, but relies on rigid temporal schedules or Figure 2: Left: Multi-turn OPSD Instability, with performance and KL reported. Right: RLSD-Style Instability, with KL loss. trajectory-depth thresholds. [Observation-2] Asymmetric Trust in Privileged Guidance. In OPSD, the teacher branch is not an independently stronger model, but the same policy augmented with privileged training-only context, such as retrieved skills. This makes its token-level guidance inherently asymmetric. For a student-sampled token yt , if the privileged teacher assigns a higher probability than the student, the retrieved skill provides an endorsement signal: it supports an on-policy behavior that the student can already generate but has not fully internalized. Such positive guidance is particularly suitable for distillation. In contrast, if the privileged teacher assigns a lower probability to the sampled token, the signal should be interpreted more cautiously. A negative gap may indicate that the token should indeed be suppressed, but in skill-conditioned OPSD it may also arise from the instability of privileged context: (1) Skill Quality. Retrieved skills may be irrelevant, incomplete, or redundant. (2) Skill Utilization. The teacher may fail to ground even relevant skills into reliable token-level preferences (Chen et al., 2019). (3) Multi-turn Drift. As trajectories unfold, the teacher-student gap tends to widen across turns (Figure 3, Middle), amplifying early mismatches over successive decisions (Ross et al., 2011). Our preliminary study on Qwen2.5-3B-Instruct shows that negative-gap tokens exceed 50% of all tokens (Figure 3), making this issue pervasive. This motivates an asymmetric treatment of privileged guidance: trust positive teacher endorsements more strongly, while applying negative teacher rejections more conservatively. A stark realization emerges: for multi-turn agents, RL could reign as the primary optimization backbone, while OPSD is relegated to a carefully controlled auxiliary role. But how should this auxiliary role be controlled? RLSD (Yang et al., 2026a) directly uses selfdivergence to re-weight token-level RL advantages, but can substantially amplify updates especially early in training when teacher-student mismatch is large (see Figure 2, Right). 2

0.11

Mean=-0.1168

250000

0.04 0.06

150000 100000

Average Gaps

0.08

Average Gaps

Token Count

200000

0.12

50000 0

0.10 0.12 0.14 0.16

-1

-0.5

0.5

Teacher-Student Gap

0.13

Turn Step Interval

0.18

30%

55%

80%

Relative Position in Turn

Figure 3: Teacher-Student Gap Analysis. Left: Token count distribution partitioned by Teacher-Student gap value. Middle: Average teacher-student gap indexed by multi-turn step. Right: Average teacher-student gap indexed by relative position within a single turn.

We take a different path: the OPSD loss is treated as a direct, auxiliary optimization objective, leaving the verifier-driven RL policy loss untouched and thereby strictly preserving the semantics and unbiasedness of the RL advantage. To overcome instability of multi-turn OPSD and privileged guidance, distillation is not performed uniformly on every token. Instead, tokens are selectively distilled via an adaptive, smooth gating mechanism rather than a hand-crafted, rigid schedule (such as Skill-SD (Wang et al., 2026a) and HDPO (Ding, 2026)). Inspired by TIP (Xu et al., 2026), we use token-level signals (such as student entropy or teacher-student divergence) to control the gate’s activation. The core philosophy is simple: let each token decide the intensity of its own supervision. This yields a dynamic, selfpaced curriculum operating at the finest possible granularity: the individual token level. We validated our method across the Qwen2.5 and Qwen3 model families on three diverse benchmarks for llm-based agents: ALFWorld (Shridhar et al., 2020), WebShop (Yao et al., 2022), and Search-QA (Jin et al., 2025). SDAR achieves substantial improvements over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, and +10.2% on WebShop-Acc for 7B), entirely avoids the catastrophic instability of naı̈ve GRPO+OPSD, and consistently outperforms RL–OPSD hybrid methods such as Skill-SD and RLSD across all three model scales (Qwen3-1.7B included). Furthermore, robustness analysis shows that SDAR degrades gracefully with retrieval quality: even random retrieval outperforms the GRPO baseline, as our gating design filters out noise from low-quality skills and distills beneficial signals only.

Method

2.1

Problem Setup

We consider a multi-turn agent that interacts with an environment over a finite horizon. Given an initial prompt or task description x, at turn k the agent receives an observation ok , generates a response ak , and the environment returns the next observation ok+1 . Each response ak may contain both intermediate reasoning tokens and executable action tokens. For notational simplicity, we flatten all valid response tokens in one trajectory into a single token sequence y = (y1 , . . . , y T ) ∼ πθ (· | x ), where πθ denotes the student policy and T is the total number of valid response tokens. At token position t, we denote the self-student context by st = ( x, y<t ), and the self-teacher context by + s+ t = ( x, c , y<t ),

where c+ denotes privileged training-only context available only to the teacher branch, such as reference answers, skills (ours), or other auxiliary information not accessible at test time. 3

Agent Loop

OPSD

Turn 1

Env

Skills

Self-Teacher

Retrieval

Given the Current Situation ,

Observation

Action

GRPO

the

sid etable

-0.25 0.06

Agent

0.20 -1.7

Turn N

Environment

-0.07

0.00

0.02

0.0 -0.03

Task-Outcome Signal

seems

τ1

0 0.46 -0.43

τ2

τG

Traj

Reward

empty

Rollout

0.28

Gaps

Token-Level Advantages

Token-Level Gating

Self-Student

Tokens

λ* OPSD Loss + GRPO Loss

Figure 4: Illustrations of SDAR framework, which trains multi-turn agents using tokenlevel OPSD loss and verifier-driven RL loss.

Skills Retrieval We retrieve task-relevant skills—compact, structured demonstrations that encode domain-specific knowledge such as sub-goal decompositions or action templates. We implement four retrieval strategies of varying quality to evaluate the robustness of our framework to the fidelity of the retrieved context: (1) UCB Retrieval, (2) Keyword Matching (KM), (3) Full Retrieval, and (4) Random Retrieval. Skill retrieval is cast as a multi-armed bandit problem over the skill library E = {e1 , . . . , e M }. For each incoming task, UCB Retrieval selects the single highest-scoring skill file according to the Upper Confidence Bound (UCB) criterion: s ln Nucb , (1) score(e) = r̄ (e) + c n(e) where r̄ (e) is the running mean reward obtained when skill e was previously supplied as context, Nucb is the total number of retrieval queries issued for the same task type, n(e) is the number of times e has been selected, and c controls the exploration–exploitation trade-off. Keyword Matching bypasses the bandit formulation and instead identifies the task scenario by matching keywords in the task description against predefined category labels, directly retrieving the skill file associated with the matched category. 2.2

Optimization Goals

Our method is designed as an auxiliary objective on top of a standard policy optimization GRPO loss. The overall training objective is

L(θ ) = LGRPO (θ ) + λ SDAR · LSDAR (θ ), where LGRPO is the original policy loss and L SDAR is our on-policy self-distillation objective. Let mt ∈ {0, 1} be the response mask indicating whether token t is valid. We define masked token averaging as Agg(z1:T ) = RL Optimization

∑tT=1 mt zt . ∑tT=1 mt

For each input x, GRPO samples a group of responses

{y(i) }iG=1 ∼ πθ (· | x ), 4

and computes a sequence-level advantage A(i) from environment rewards. Using a reference policy πref , the GRPO objective can be written as

LGRPO (θ ) = −

1 G (i ) (i ) Agg min rt A(i) , clip(rt , 1 − ϵ, 1 + ϵ) A(i) ∑ G i =1

+ β· (i )

(i )

1 G (i ) (i ) π (· | s ) ∥ π (· | s )) , Agg D ( KL θ ref t t G i∑ =1

(i )

(2)

(i )

where rt = πθ (yt | st )/πθold (yt | st ) is the importance sampling ratio. OPSD Optimization At a fixed token position t, the teacher and student induce conditional token distributions π T (· | s+ t ) and πθ (· | st ), respectively. The per-token reverse KL divergence is defined as: (t)

DRKL = DKL πθ (· | st )

πθ (v | st ) π T (· | s+ . t ) = ∑ πθ ( v | st ) log π T (v | s+ t ) v∈V

To efficiently derive an importance signal without computing the expensive full-vocabulary summation, we take a single-sample estimate on the student-sampled token yt ∼ πθ (· | st ). The negation of this estimate directly yields the Teacher-Student log-probability gap ∆t : (t)

+ b ∆t = − D RKL = log π T ( yt | st ) − log πθ ( yt | st ).

2.3

Token-Level Gating

The key idea is to convert privileged teacher guidance into a token-level trust weight, while keeping the verifier-driven RL objective unchanged. We introduce a token-level gate gt ∈ [0, 1] that modulates the OPSD signal on each student-sampled token, and apply it to a sampled-token surrogate so that different gating strategies share the same optimization. Let

∆t = sg log πθ+ (yt | s+ t ) − log πθ ( yt | st )

denote the detached Teacher-Student log-probability gap on the student-sampled token, and ht = − ∑ πθ (v | st ) log πθ (v | st ) v∈V

denote the student entropy at position t. We compose each raw score with the logistic sigmoid σ so that every gate is smooth, differentiable, and naturally bounded in (0, 1). The sharpness parameter β > 0 controls the transition between conservative attenuation and strong activation. We instantiate three complementary gating strategies: 1. Entropy gating: gt = σ ( β ht ), targeting high-entropy positions where the student is most uncertain. 2. Gap gating: gt = σ ( β ∆t ), assigning larger weights to positive-gap tokens endorsed by the privileged teacher while attenuating negative-gap tokens. 3. Soft-OR gating: gt = σ( β[1 − (1 − ht )(1 − ∆t )]), combining student uncertainty and teacher-student gap as an alternative gating strategy. In all cases, the gate is detached via sg(·), so gradients flow exclusively through the student log-probability. The token-level loss is

ℓtSDAR = gt · (log πθ+ (yt | s+ t ) − log πθ ( yt | st )),

LSDAR = Agg(ℓtSDAR ).

With gap gating, the sigmoid gate implements asymmetric token-level modulation: positivegap tokens receive stronger auxiliary distillation, while negative-gap tokens are softly attenuated. We also provide theoretical analysis of our design in Appendix A. 5

Table 1: Performance on ALFWorld, Search-QA and WebShop tasks. We report the success rate (%) on ALFWorld, accuracy (%) on Search-QA, and Score/Acc (%) on WebShop (128 tasks). * means validation with skills. Best and second-best are highlighted. ALFWorld Method

Search-QA

Pick Look CleanHeat Cool Pick2 Avg NQ

WebShop

Triv Pop Hotp 2Wk MuS Bam Avg Score Acc

Qwen2.5-3B-Instruct Vanilla 44.4 Skill-Prompt* 51.7 OPSD 48.8 GRPO 91.2 Skill-GRPO 88.9 Skill-GRPO* 94.3 GRPO+OPSD 100 Skill-SD 88.2 RLSD 87.9 SDAR 97.1

11.1 66.7 41.7 62.5 71.4 57.1 82.4 50.0 75.0 62.5

6.2 48.4 16.7 96.2 58.8 100 85.7 96.2 90.9 100

15.4 0.0 0.0 61.9 70.6 66.7 75.0 52.4 75.0 61.9

28.6 4.3 15.8 65.0 40.7 73.1 70.0 65.0 73.1 75.0

12.5 10.0 16.7 47.4 29.2 57.1 60.0 57.9 68.4 84.2

21.9 28.9 28.1 75.0 60.2 80.5 81.2 73.4 79.7 84.4

24.6 23.7 0.1 39.3 43.5 44.3 44.9 44.4 41.5 44.8

48.1 46.2 0.1 60.6 58.8 59.6 61.2 60.4 58.6 58.1

31.0 30.6 0.1 41.1 43.0 44.3 45.2 44.0 42.3 44.3

26.3 24.4 0.0 37.4 36.8 39.0 40.4 39.5 40.4 38.6

25.3 22.1 0.0 34.6 32.2 36.1 38.5 40.4 40.2 36.2

7.2 7.5 0.0 15.4 11.7 14.5 16.0 15.4 16.8 15.7

59.7 12.5 0.0 26.4 12.5 14.9 66.1 64.9 66.9 66.1

31.7 23.9 0.0 36.4 34.1 36.1 44.6 44.1 43.8 43.4

6.7 0.2 11.3 79.8 77.3 76.3 77.8 75.9 84.4 85.0

0.8 0.8 3.1 63.3 60.9 66.4 66.4 64.0 66.4 68.0

Qwen2.5-7B-Instruct Vanilla 36.1 Skill-Prompt* 51.7 OPSD 50.0 GRPO 91.2 Skill-GRPO 88.5 Skill-GRPO* 100 GRPO+OPSD 91.4 Skill-SD 93.9 RLSD 100 SDAR 94.7

22.2 50.0 60.0 87.5 66.7 83.3 61.5 93.8 87.5 75.0

3.1 32.3 22.7 96.2 65.2 96.4 100 90.9 92.3 100

0.0 5.3 21.4 81.0 61.1 83.3 87.5 100 58.8 86.7

0.0 4.3 17.6 65.0 57.7 75.0 76.5 69.2 80.0 68.2

0.0 0.0 9.5 57.9 73.1 78.9 52.2 68.4 65.2 78.9

12.5 23.4 32.8 81.2 69.5 88.3 80.4 85.1 82.0 85.9

25.2 30.9 8.8 45.1 45.2 44.8 47.3 47.1 46.8 46.3

50.8 52.1 8.6 63.7 63.7 63.0 64.5 64.5 63.0 63.5

29.5 32.7 17.5 44.0 45.7 45.1 46.9 47.8 44.4 48.2

29.0 32.7 2.5 43.6 43.1 43.7 43.8 44.2 45.5 43.8

29.0 27.9 4.2 43.2 43.3 43.7 39.3 42.1 48.9 48.4

10.4 12.7 0.5 16.8 19.6 20.5 18.0 20.2 21.5 19.6

63.7 66.1 1.2 37.6 21.4 71.4 69.4 69.0 73.0 73.0

33.9 36.4 6.2 42.0 40.3 47.5 47.0 47.8 49.0 49.0

5.9 1.7 4.5 80.9 80.4 87.0 86.8 86.1 87.4 89.4

1.6 0.8 2.3 72.6 71.9 81.2 76.5 76.5 77.3 82.8

Qwen3-1.7B-Instruct Vanilla 25.0 Skill-Prompt* 10.3 26.3 OPSD GRPO 71.1 Skill-GRPO 27.6 Skill-GRPO* 31.4 GRPO+OPSD 38.2 Skill-SD 52.9 RLSD 50.0 SDAR 73.5

22.2 50.0 33.3 41.7 54.5 42.9 50.0 37.5 37.5 25.0

3.1 16.1 9.1 36.4 22.7 51.9 30.8 69.2 61.5 76.9

0.0 0.0 0.0 40.0 27.3 8.3 28.6 42.9 19.0 33.3

21.4 0.0 4.5 31.8 0.0 11.5 30.0 60.0 50.0 40.0

4.2 5.0 5.3 31.6 19.2 7.1 21.1 36.8 21.1 36.8

12.5 9.4 14.1 46.1 21.1 28.1 32.0 52.3 42.2 53.9

29.4 29.4 4.2 40.0 39.2 38.0 40.7 39.1 38.6 39.7

46.9 46.5 8.3 58.9 58.6 58.4 58.9 57.5 57.3 58.9

37.0 36.2 4.6 43.5 43.9 43.9 45.0 45.4 43.0 45.3

23.5 22.9 6.6 35.4 35.2 36.3 37.0 34.8 34.5 35.9

19.6 20.8 15.3 30.3 28.2 29.0 34.6 34.1 34.1 35.5

6.4 4.3 0.7 12.0 11.5 12.5 13.3 10.7 11.5 12.6

10.5 10.1 1.2 65.7 66.1 66.9 65.7 64.1 65.3 65.3

24.8 24.3 5.8 40.8 40.4 40.7 42.2 40.8 40.6 41.9

46.5 23.0 47.4 67.3 73.4 80.4 70.7 81.8 74.0 76.8

4.7 2.3 9.3 38.3 46.1 50.0 38.3 53.9 50.8 58.6

Experiment

Benchmarks We evaluate our methods on ALFWorld (Shridhar et al., 2020), Search-based QA (Jin et al., 2025), and Webshop (Yao et al., 2022). ALFWorld is a text-based game aligned with the ALFRED embodied AI benchmark, including 3,827 task instances across six categories of common household activities: Pick and Place (Pick), Look at Obj in Light (Look), Pick Clean then Place in Recep (Clean), Pick Heat then Place in Recep (Heat), Pick Cool then Place in Recep (Cool), and Pick Two Obj and Place (Pick2). Search-based QA contains several widely-used search-augmented QA benchmarks, including single-hop QA datasets (NQ (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2023)) and multi-hop QA datasets (HotpotQA (Yang et al., 2018), 2Wiki (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), and Bamboogle (Press et al., 2023)). WebShop is a complex, web-based interactive environment designed to test the LLM agents in realistic online shopping scenarios. Agents navigate a realistic web interface to find and purchase products matching user specifications. We select 128 fixed tasks in validation set, which aligns with Feng et al. (2025). Implementation Details. We train the Qwen2.5-Instruct and Qwen3-Instruct series using SDAR for at 150 steps on 8 H800 GPUs. For ALFWorld, we adopt the training data split from GiGPO (Feng et al., 2025), with each batch sampling 16 tasks and 8 rollouts per prompt, and a maximum prompt length of 2,048 tokens. For Search-QA, we follow the experimental setup of Search-R1 (Jin et al., 2025), using E5 (Wang et al., 2022) as the retriever. The training data are drawn from NQ and HotpotQA, making these two benchmarks in-domain, while the remaining datasets serve as out-of-domain evaluation. Each batch samples 128 tasks with a maximum prompt length of 4,096 tokens. For Webshop, 1000 tasks are selected for 6

training, with each batch sampling 16 tasks and 8 rollouts per prompt, and a maximum prompt length of 4,096 tokens. We set the SkillBank from SkillRL (Xia et al., 2026) for all three environments. We set λ SDAR = 0.01 and β = 5.0 in our experiments. Baselines We compare SDAR against three categories of methods on three base models. (1) Training-free methods. Skill-Prompt retrieves task-relevant skills from the SkillBank via keyword matching (KM) and prepends them to the input prompt at inference time. (2) Post-training methods, such as GRPO (Shao et al., 2024), OPSD (Zhao et al., 2026) and Skill-GRPO. Skill-GRPO augments GRPO by retrieving skills via KM and injecting them into the training prompt; at test time it can run with (Skill-GRPO*) or without retrieved skills. (3) Hybrid methods, that combine RL with privileged knowledge distillation, such as GRPO+OPSD, and Skill-SD (Wang et al., 2026a), RLSD (Yang et al., 2026a). GRPO+OPSD simply adds the OPSD distillation loss as an auxiliary objective on top of GRPO training. All the algorithms of SDAR and other baselines are detailed in Appendix A. 3.1

Main Results

Overall Performance. As summarized in Table 1, SDAR demonstrates exceptional performance, achieving the best or second-best results across almost all settings. Compared to GRPO, it delivers substantial gains: on Qwen2.5-3B, it improves ALFWorld by +9.4% (84.4 vs. 75.0), Search-QA by +7.0%, and WebShop-Acc by +4.7%, with similarly consistent improvements on the 7B model. While standalone OPSD collapses catastrophically (near-zero on Search-QA) and a naive GRPO+OPSD combination degrades severely on Qwen3-1.7B (32.0 vs. 46.1) due to unbounded distillation gradients overwhelming the RL signal, SDAR avoids the observed instability and maintains stable gains. Through its adaptive gating mechanism, it ensures stable optimization and consistent gains across all model scales. Skills Internalization. Beyond overall performance, SDAR successfully internalizes privileged knowledge rather than superficially relying on it at inference (Lu et al., 2026c). While Skill-GRPO shows a massive performance drop when tested without skills (e.g., 60.2 vs. 80.5 on ALFWorld-3B) and even underperforms vanilla GRPO due to harmful distributional dependencies, SDAR requires no external skills during inference. Yet, it surpasses even the skill-augmented Skill-GRPO* in most settings, achieving 84.4 on ALFWorld-3B and a striking 53.9 (vs. 28.1) on ALFWorld-1.7B. These consistent gains confirm that our token-level gated distillation genuinely transfers underlying knowledge into the policy’s parameters. Strong Generalization. SDAR also exhibits stronger generalization compared to hybrid baselines such as Skill-SD and RLSD. On Qwen2.5-3B, it outperforms both methods on ALFWorld (84.4 vs. 73.4 for Skill-SD and 79.7 for RLSD) and WebShop. This advantage is most pronounced on the challenging Qwen3-1.7B model, where smaller models may struggle to utilize retrieved skills effectively. In this regime, Skill-GRPO drops to 21.1% on ALFWorld, well below GRPO’s 46.1%, and RLSD reaches 42.2%. In contrast, SDAR achieves the highest score of 53.9%. By attenuating uncertain negative teacher guidance while preserving positive teacher endorsements, our gating mechanism provides a more robust way to incorporate privileged knowledge without sacrificing generalization. 3.2

Training Dynamics

To elucidate the adaptive behavior of SDAR throughout RL optimization, we monitor two key metrics for the Qwen2.5-7B backbone on ALFWorld in Figure 5. (a) shows that the mean ¯ = Et [∆t ]) remains consistently negative, indicating Teacher-Student log-probability gap (∆ that the privileged teacher assigns lower probability than the student to sampled tokens on average. This reveals partial asymmetric trust in privileged guidance regime where naı̈ve ¯ steadily converges toward distillation would actively degrade performance. Crucially, ∆ zero, confirming that the gating mechanism successfully identifies and up-weights the specific subset of tokens where the teacher does provide beneficial signals. To further validate this adaptive filtering, (b) tracks the gate activation ratio (the fraction of tokens where gt > 0.5). For the majority of early training, this ratio remains strictly below 0.5, correctly 7

0.55

0.04

0.50

Gate Active Ratio

Teacher Gap Mean

0.02 0.06 0.08 0.10 0.12

0.45 0.40 0.35 0.30 0.25

Step

100

120

140

Step

100

120

140

Figure 5: Training Dynamics. Average teacher-student gap (Left) and gate activation ratio (Right) during the training of Qwen2.5-7B-Instruct on ALFWorld. Table 2: Robust Testing of different skill retrieval methods. Method

ALFWorld

WebShop-Score

WebShop-Acc

UCB KM Full Random w/o OPSD

86.8+5.6 85.9+4.7 83.2+2.0 83.1+1.9 81.2

87.5+6.6 89.4+8.5 87.2+6.3 82.5+1.6 80.9

81.2+8.6 82.8+10.2 78.1+5.5 73.6+1.0 72.6

suppressing tokens that carry negative signals. However, as the student’s policy evolves, the ratio gradually increases, reflecting that more tokens enter a regime of constructive teacher guidance. 3.3

Robust Analysis

To address the practical concern of whether SDAR heavily relies on high-quality skill retrieval, we fix our optimal configuration (λ = 0.01, β = 5.0) and evaluate performance across four retrieval quality tiers (Table 2). All four strategies consistently outperform the pure GRPO baseline (w/o OPSD). Even Random Retrieval—which selects skills with zero task awareness—yields gains of +1.9/+1.6/+1.0 on ALFWorld/WebShop-Score/WebShopAcc. Higher-quality retrieval further amplifies these benefits: Keyword Matching achieves gains of +4.7/+8.5/+10.2 and even surpasses UCB on WebShop. These results echo our observation on asymmetric privileged guidance. Low-quality retrieval can introduce mismatched or unstable teacher signals, especially negative guidance from irrelevant skills. Rather than uniformly following such signals, SDAR uses token-level gating to retain positive teacher endorsements while softly attenuating uncertain negative rejections. Thus, the performance gains remain robust across retrieval qualities, suggesting that the uplift stems primarily from gated distillation rather than retrieval fidelity alone. 3.4

Ablation Studies

Token-Level Gating Strategy. As shown in Figure 6, Teacher-Student Gap gating consistently outperforms both the entropy and soft-OR gating strategies (introduced in Section 2.3), achieving a higher asymptotic success rate (∼0.84) and a steeper performance climb after the initial 100 steps. We attribute this superiority to the directness of the Teacher-Student gap (∆t ) as an importance signal, which precisely measures the teacher’s disagreement with the student’s chosen token. In contrast, entropy (ht ) acts as an indirect proxy that may erroneously activate on uncertain but already well-handled tokens, while soft-OR dilutes 8

Step

Success Rate

Entropy SOFT-OR Score Teacher-Student Gap

0.8 0.7 0.6 0.5 0.4 0.3 0.2

100

125

150

= 0.01 (Ours) = 0.001 = 0.1

Step

100

125

Step

100

125

150

Figure 7: Ablations of β on Qwen2.5-3BInstruct.

Success Rate

Figure 6: Ablations of Token-level Gating on Qwen2.5-3B-Instruct.

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

= 5.0 (Ours) = 1.0 = 10.0 = 0.0 (No Gate)

0.8 0.7 0.6 0.5 0.4 0.3 0.2

150

Reverse KL (Ours) Forward KL JSD

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0

Step

100

125

150

Figure 9: Ablations of L SDAR type on Qwen2.5-7B-Instruct.

Figure 8: Ablations of λ on Qwen2.5-3BInstruct.

the gating signal by triggering when only one score is moderately large, thereby reducing its selectivity. All remaining experiments default to gap gating. Sharpness β. Figure 7 evaluates the impact of sigmoid sharpness across β ∈ {0, 1, 5, 10}, where β = 0 denotes the complete removal of the gating mechanism (i.e., uniform distillation). The optimal performance is achieved at β = 5, which effectively balances two distinct failure modes: an excessively small β (including the no-gate baseline) applies distillation indiscriminately, thereby inheriting the multi-turn instability of naı̈ve OPSD; conversely, an overly large β strictly binarizes the gate, stripping away the smooth modulation necessary for assigning partial credit on borderline tokens. Distillation Coefficient λ. Figure 8 sweeps the distillation weight λ SDAR ∈ {0.001, 0.01, 0.1}, revealing that λ = 0.01 provides an optimal, steady complementary signal without interfering with the primary RL objective. When λ is increased to 0.1, the distillation gradient overwhelmingly dominates the policy update; since the teacher is on average no confident than the student in multi-turn settings (as evidenced by the negative gap in Figure 5), this over-weighted term forces the student toward inferior behaviors, causing a severe performance decline that overshadows the GRPO reward signal. Conversely, λ = 0.001 exerts insufficient corrective pressure to meaningfully aid the RL process, confirming the necessity of a carefully calibrated, moderate coefficient. Distillation Objective. Figure 9 compares three token-level matching objectives on Qwen2.5-7B: reverse KL (our default), forward KL, and Jensen–Shannon divergence (JSD), where JSD is defined as the symmetrized average with respect to the mixture 9

Mt = 21 (πθ (· | st ) + π T (· | s+ t )): (t)

DJSD = 12 DKL(πθ (· | st ) ∥ Mt ) + 21 DKL(π T (· | s+ t ) ∥ Mt ) . Reverse KL clearly outperforms both alternatives, aligning perfectly with our design rationale in Section 2.2: the reverse direction DKL (πθ ∥π T ) is inherently mode-seeking (Murphy, 2012), encouraging the student to concentrate probability mass only on modes supported by the teacher. In our partial ”weak” teacher signals—where the teacher is frequently lost—this selectivity is paramount, as reverse KL naturally down-weights tokens with low teacher probability, thereby seamlessly complementing the explicit gating mechanism. In contrast, the mode-covering nature of forward KL forces the student to spread mass across all teacher-supported tokens, indiscriminately incorporating unreliable guidance, while JSD acts as a symmetric compromise that inherits this detrimental mode-covering tendency, ultimately yielding intermediate performance.

Related Work

4.1

Agentic RL

Recent advances in reinforcement learning for LLMs have demonstrated strong effectiveness on verifiable reasoning tasksn (Shao et al., 2024; Yu et al., 2025; Guo et al., 2025; Yao et al., 2026; Chen et al., 2026). Building on this progress, LLMs are increasingly extended from static reasoning problems to autonomous agents that operate in dynamic, open-world environments, including GUI automation (Ye et al., 2025), gameplay (Shridhar et al., 2020), and embodied control (Wang et al., 2023). In these settings, agents must make sequential decisions based on environment observations and feedback, making agentic RL a crucial post-training recipe for improving their decision-making capabilities (Lu et al., 2025; Dong et al., 2025; Feng et al., 2025; Lu et al., 2026a;b; Shi et al., 2026). 4.2

OPSD

On-policy distillation (OPD) supervises a student on its own generated sequences, avoiding offline distribution mismatch (Agarwal et al., 2024; Gu et al., 2026). GKD-style methods (Agarwal et al., 2024; Wen et al., 2023) minimize token-level divergences but require full-vocabulary teacher distributions, while PG-style methods (Yang et al., 2026a; Xu et al., 2026) convert discrepancy into token-level rewards but risk high-variance updates. For multi-turn agents, TCOD (Wang et al., 2026b) applies a turn-level curriculum to mitigate compounding drift, but relies on rigid schedules. On-Policy Self-Distillation (OPSD) (Zhao et al., 2026; He et al., 2026) further removes the need for a separate teacher by conditioning only on privileged context. Hybrid Methods Recent works have explored combining RL with distillation to leverage their complementary strengths (Wang et al., 2026a; Yang et al., 2026a; Ding, 2026), but suffer from rigid hand-crafted scheduling or substantially unstable updates. In contrast, our method treats distillation as a strictly separate auxiliary objective with adaptive, bounded, token-level gating, preserving the unbiasedness of the RL advantage while selectively injecting only beneficial teacher signals.

Conclusion

We presented SDAR, which reconciles RL and OPSD for multi-turn agent training through a sigmoid gate that lets each token autonomously regulate its distillation intensity. This preserves RL as the unbiased optimization backbone while selectively extracting beneficial teacher signals. Experiments across three benchmarks and three model scales confirm consistent gains over both pure RL and hybrid baselines.

References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes, 2024. URL https://arxiv.org/abs/2306.13649. Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating, 2019. URL https://arxiv.org/abs/1912.12294. Yuxin Chen, Yu Wang, Yi Zhang, Ziang Ye, Zhengzhou Cai, Yaorui Shi, Qi Gu, Hui Su, Xunliang Cai, Xiang Wang, et al. Learning to self-verify makes language models better reasoners. arXiv preprint arXiv:2602.07594, 2026. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. Ken Ding. Hdpo: Hybrid distillation policy optimization via privileged self-distillation, 2026. URL https://arxiv.org/abs/2603.23871. Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849, 2025. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. arXiv preprint arXiv:2505.10978, 2025. GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, Jiajie Zhang, Jian Jiao, Jiaqi Guo, Jingsen Wang, Jingzhao Du, Jinzhu Wu, Kedong Wang, Lei Li, Lin Fan, Lucen Zhong, Mingdao Liu, Mingming Zhao, Pengfan Du, Qian Dong, Rui Lu, Shuang-Li, Shulin Cao, Song Liu, Ting Jiang, Xiaodong Chen, Xiaohan Zhang, Xuancheng Huang, Xuezhen Dong, Yabo Xu, Yao Wei, Yifan An, Yilin Niu, Yitong Zhu, Yuanhao Wen, Yukuo Cen, Yushi Bai, Zhongpei Qiao, Zihan Wang, Zikang Wang, Zilin Zhu, Ziqiang Liu, Zixuan Li, Bojie Wang, Bosi Wen, Can Huang, Changpeng Cai, Chao Yu, Chen Li, Chengwei Hu, Chenhui Zhang, Dan Zhang, Daoyan Lin, Dayong Yang, Di Wang, Ding Ai, Erle Zhu, Fangzhou Yi, Feiyu Chen, Guohong Wen, Hailong Sun, Haisha Zhao, Haiyi Hu, Hanchen Zhang, Hanrui Liu, Hanyu Zhang, Hao Peng, Hao Tai, Haobo Zhang, He Liu, Hongwei Wang, Hongxi Yan, Hongyu Ge, Huan Liu, Huanpeng Chu, Jia’ni Zhao, Jiachen Wang, Jiajing Zhao, Jiamin Ren, Jiapeng Wang, Jiaxin Zhang, Jiayi Gui, Jiayue Zhao, Jijie Li, Jing An, Jing Li, Jingwei Yuan, Jinhua Du, Jinxin Liu, Junkai Zhi, Junwen Duan, Kaiyue Zhou, Kangjian Wei, Ke Wang, Keyun Luo, Laiqiang Zhang, Leigang Sha, Liang Xu, Lindong Wu, Lintao Ding, Lu Chen, Minghao Li, Nianyi Lin, Pan Ta, Qiang Zou, Rongjun Song, Ruiqi Yang, Shangqing Tu, Shangtong Yang, Shaoxiang Wu, Shengyan Zhang, Shijie Li, Shuang Li, Shuyi Fan, Wei Qin, Wei Tian, Weining Zhang, Wenbo Yu, Wenjie Liang, Xiang Kuang, Xiangmeng Cheng, Xiangyang Li, Xiaoquan Yan, Xiaowei Hu, Xiaoying Ling, Xing Fan, Xingye Xia, Xinyuan Zhang, Xinze Zhang, Xirui Pan, Xu Zou, Xunkai Zhang, Yadi Liu, Yandong Wu, Yanfu Li, Yidong Wang, Yifan Zhu, Yijun Tan, Yilin Zhou, Yiming Pan, Ying Zhang, Yinpei Su, Yipeng Geng, Yong Yan, Yonglin Tan, Yuean Bi, Yuhan Shen, Yuhao Yang, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yurong Wu, Yutao Zhang, Yuxi Duan, Yuxuan Zhang, Zezhen Liu, Zhengtao Jiang, Zhenhe Yan, Zheyu Zhang, Zhixiang Wei, Zhuo Chen, Zhuoer Feng, Zijun Yao, Ziwei Chai, Ziyuan Wang, Zuzhou Zhang, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang. Glm-5: from vibe coding to agentic engineering, 2026. URL https://arxiv.org/abs/2602.15763. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: On-policy distillation of large language models, 2026. URL https://arxiv.org/abs/2306.08543. 11

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, and Sanjeev Arora. Self-distillation zero: Selfrevision turns binary rewards into dense supervision, 2026. URL https://arxiv.org/ abs/2604.12002. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609–6625, 2020. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611, 2017. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019. Zhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen, Haiyang Xu, Ziwei Zheng, Weiming Lu, Ming Yan, Fei Huang, Jun Xiao, et al. Ui-s1: Advancing gui automation via semi-online reinforcement learning. arXiv preprint arXiv:2509.11543, 2025. Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Pengxiang Zhao, Guangyi Liu, et al. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 17608–17616, 2026a. Zhengxi Lu, Fei Tang, Guangyi Liu, Kaitao Song, Xu Tan, Jin Ma, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Ui-copilot: Advancing long-horizon gui automation via tool-integrated policy optimization, 2026b. URL https://arxiv.org/ abs/2604.13822. Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Skill0: In-context agentic reinforcement learning for skill internalization, 2026c. URL https://arxiv.org/abs/2604.02268. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 9802–9822, 2023. Kevin P Murphy. Machine learning: a probabilistic perspective. 2012. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687–5711, 2023. Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2011. URL https://arxiv.org/ abs/1011.0686. 12

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36:38154–38180, 2023. Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao, Shugui Liu, Qi GU, Xunliang Cai, Xiang Wang, and An Zhang. Skill1: Unified evolution of skill-augmented agents via reinforcement learning, 2026. URL https://arxiv.org/abs/2605.06130. Zhengliang Shi, Shen Gao, Lingyong Yan, Yue Feng, Xiuyi Chen, Zhumin Chen, Dawei Yin, Suzan Verberne, and Zhaochun Ren. Tool learning in the wild: Empowering language models as automatic tool agents. In Proceedings of the ACM on Web Conference 2025, pp. 2222–2237, 2025. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. Core Team, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, Gang Xie, Hailin Zhang, Hanglong Lv, Hanyu Li, Heyu Chen, Hongshen Xu, Houbin Zhang, Huaqiu Liu, Jiangshan Duo, Jianyu Wei, Jiebao Xiao, Jinhao Dong, Jun Shi, Junhao Hu, Kainan Bao, Kang Zhou, Lei Li, Liang Zhao, Linghao Zhang, Peidian Li, Qianli Chen, Shaohui Liu, Shihua Yu, Shijie Cao, Shimao Chen, Shouqiu Yu, Shuo Liu, Tianling Zhou, Weijiang Su, Weikun Wang, Wenhan Ma, Xiangwei Deng, Bohan Mao, Bowen Ye, Can Cai, Chenghua Wang, Chengxuan Zhu, Chong Ma, Chun Chen, Chunan Li, Dawei Zhu, Deshan Xiao, Dong Zhang, Duo Zhang, Fangyue Liu, Feiyu Yang, Fengyuan Shi, Guoan Wang, Hao Tian, Hao Wu, Heng Qu, Hongfei Yi, Hongxu An, Hongyi Guan, Xing Zhang, Yifan Song, Yihan Yan, Yihao Zhao, Yingchun Lai, Yizhao Gao, Yu Cheng, Yuanyuan Tian, Yudong Wang, Zhen Tang, Zhengju Tang, Zhengtao Wen, Zhichao Song, Zhixian Zheng, Zihan Jiang, Jian Wen, Jiarui Sun, Jiawei Li, Jinlong Xue, Jun Xia, Kai Fang, Menghang Zhu, Nuo Chen, Qian Tu, Qihao Zhang, Qiying Wang, Rang Li, Rui Ma, Shaolei Zhang, Shengfan Wang, Shicheng Li, Shuhao Gu, Shuhuai Ren, Sirui Deng, Tao Guo, Tianyang Lu, Weiji Zhuang, Weikang Zhang, Weimin Xiong, Wenshan Huang, Wenyu Yang, Xin Zhang, Xing Yong, Xu Wang, Xueyang Xie, Yilin Jiang, Yixin Yang, Yongzhe He, Yu Tu, Yuanliang Dong, Yuchen Liu, Yue Ma, Yue Yu, Yuxing Xiang, Zhaojun Huang, Zhenru Lin, Zhipeng Xu, Zhiyang Chen, Zhonghua Deng, Zihan Zhang, and Zihao Yue. Mimo-v2-flash technical report, 2026a. URL https://arxiv.org/abs/2601.02780. Kimi Team, Yifan Bai, Yiping Bao, Y Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. Meituan LongCat Team, Anchun Gui, Bei Li, Bingyang Tao, Bole Zhou, Borun Chen, Chao Zhang, Chen Gao, Chen Zhang, Chengcheng Han, et al. Longcat-flash-thinking-2601 technical report. arXiv preprint arXiv:2601.16725, 2026b. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, Xiaohu Ruan, Xiaoxin Chen, and Honggang Qi. Skill-sd: Skill-conditioned self-distillation for multi-turn llm agents, 2026a. URL https://arxiv.org/abs/2604.10674. 13

Jiaqi Wang, Wenhao Zhang, Weijie Shi, Yaliang Li, and James Cheng. Tcod: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents, 2026b. URL https://arxiv.org/abs/2604.24005. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022. Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequencelevel knowledge distillation, 2023. URL https://arxiv.org/abs/2307.15190. Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. Skillrl: Evolving agents via recursive skill-augmented reinforcement learning, 2026. URL https://arxiv.org/abs/2602.08234. Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation, 2026. URL https://arxiv.org/abs/2604. 14084. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr, 2026a. URL https: //arxiv.org/abs/2604.03128. Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation, 2026b. URL https://arxiv.org/abs/2602.12125. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2369–2380, 2018. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022. Zhiyuan Yao, Yi-Kai Zhang, Yuxin Chen, Yueqing Sun, Zishan Xu, Yu Yang, Tianhao Hu, Qi Gu, Hui Su, and Xunliang Cai. Coba-rl: Capability-oriented budget allocation for reinforcement learning in llms. arXiv preprint arXiv:2602.03048, 2026. Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, et al. Mobile-agent-v3: Fundamental agents for gui automation. arXiv preprint arXiv:2508.15144, 2025. Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models, 2026. URL https://arxiv.org/abs/2602.12275. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Ruixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, and Yizhe Zhang. Embarrassingly simple self-distillation improves code generation, 2026. URL https://arxiv.org/abs/2604.01193. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models, 2026. URL https://arxiv.org/abs/2601.18734. 14

Table of Contents 1

Introduction

Method

2.1

Problem Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.2

Optimization Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.3

Token-Level Gating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Experiment

3.1

Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.2

Training Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.3

Robust Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.4

Ablation Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Related Work

4.1

Agentic RL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.2

OPSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Conclusion

A Theoretical Analysis

A.1 Design Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.2 Theoretical Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B Algorithm

C Hyperparameters

D Training Dynamics

E Prompt

Theoretical Analysis

A.1

Design Rationale

The central design question is how the divergence signal should enter optimization. We adopt the reverse-KL-aligned gap ∆t = log π T (yt | s+ t ) − log πθ ( yt | st ) rather than forward KL, because it naturally evaluates on student-sampled tokens and avoids the computationally expensive full-vocabulary matching. However, using this raw gap directly as a coefficient would create overly strong, unbounded token-level gradients during early training or under severe teacher-student mismatch. To resolve this, we wrap the gap in a sigmoid function gt = σ ( β∆t ), which transforms the raw discrepancy into a bounded and monotone importance weight ∂gt > 0. ∂∆t

gt ∈ (0, 1),

This preserves the ordering of token importance while strictly preventing gradient explosion. Finally, we apply a stop-gradient operator to the gate. Detaching gt ensures it acts purely as a confidence weight rather than creating an additional, self-referential optimization pathway, yielding a stable, first-order weighted likelihood update. A.2

Theoretical Properties

We formalize the stability and curriculum properties of SDAR through the following propositions. Proposition 1 (Equivalent Weighted Likelihood Form). Assume that both log π T (yt | s+ t ) and gt are detached from gradient computation. Minimizing L SDAR is equivalent, up to an additive constant, to maximizing a token-weighted log-likelihood objective on student-sampled tokens:

LSDAR = C − Agg( gt log πθ (yt | st )) , where

C = Agg gt log π T (yt | s+ t )

is constant with respect to θ. Proof. By definition, LSDAR = Agg gt (log π T (yt | s+ t ) − log πθ ( yt | st )) = Agg gt log π T (yt | s+ t ) − Agg( gt log πθ ( yt | st ))

= C − Agg( gt log πθ (yt | st )) , where the first term C = Agg gt log π T (yt | s+ t ) is constant w.r.t. θ since both gt and log π T (yt | s+ t ) are detached. Proposition 2 (Gradient Form). Under the same assumptions, the gradient of L SDAR is strictly modulated by the bounded scalar gate:

∇θ LSDAR = − Agg( gt ∇θ log πθ (yt | st )) . Proof. From Proposition 1,

∇θ LSDAR = ∇θ [C − Agg( gt log πθ (yt | st ))] = 0 − Agg( gt ∇θ log πθ (yt | st )) = − Agg( gt ∇θ log πθ (yt | st )) .

Proposition 3 (Monotonicity and Smoothness of the Gate). The gate gt = σ ( β∆t ) is strictly increasing in ∆t , inducing an online token-level curriculum where larger discrepancies receive stronger weights. Its derivative satisfies ∂gt = β σ( β∆t )(1 − σ( β∆t )) ∈ (0, β/4]. ∂∆t Proof. By the chain rule, ∂gt = β σ′ ( β∆t ). ∂∆t Since the logistic sigmoid satisfies σ′ (z) = σ (z)(1 − σ(z)) > 0

∀ z ∈ R,

we obtain ∂gt = β σ( β∆t )(1 − σ( β∆t )) > 0. ∂∆t Let u = σ ( β∆t ) ∈ (0, 1): u (1 − u ) ≤

u + (1 − u ) 2

1 4

∂gt β = β u (1 − u ) ≤ . ∂∆t 4 Proposition 4 (Bounded Auxiliary Gradient). Assume that ∥∇θ log πθ (yt | st )∥ ≤ Bt for each valid token. Then the gate cannot amplify the auxiliary gradient beyond the unweighted likelihood gradient: ∥∇θ LSDAR ∥ ≤ Agg( Bt ). Proof. By Proposition 2,

∥∇θ LSDAR ∥ = ∥Agg( gt ∇θ log πθ (yt | st ))∥ ≤ Agg( gt ∥∇θ log πθ (yt | st )∥) ≤ Agg(1 · Bt ) = Agg( Bt ), where the first inequality is the triangle inequality and the second uses 0 < gt < 1 and ∥∇θ log πθ (yt | st )∥ ≤ Bt . Proposition 5 (Effect of Not Detaching the Gate). Without stop-gradient on the gate, the nondetached token loss ℓ̃t = σ( β∆t ) ∆t introduces an unstable self-referential coupling term into the gradient:

∇θ ℓ̃t = − gt + β∆t gt (1 − gt ) ∇θ log πθ (yt | st ).

Proof. Write ℓ̃t = gt ∆t . Since log π T (yt | s+ t ) is constant w.r.t. θ,

∇θ ∆t = ∇θ [log π T (yt | s+ t ) − log πθ ( yt | st )] = −∇θ log πθ ( yt | st ). By the chain rule on gt = σ ( β∆t ),

∇θ gt = β σ′ ( β∆t ) ∇θ ∆t = β gt (1 − gt ) ∇θ ∆t = − β gt (1 − gt ) ∇θ log πθ (yt | st ). Applying the product rule,

Algorithm

The full procedure of SDAR is presented in Algorithm 1. We compare against five baselines listed below: • GRPO (Shao et al., 2024) (Algorithm 2): RL baseline that optimizes the policy via a clipped surrogate objective with group-relative advantages. • OPSD (Zhao et al., 2026) (Algorithm 3): an on-policy self-distillation method that distills token-level knowledge from a frozen reference policy πref into the student. • Skill-SD (Wang et al., 2026a) (Algorithm 4): a hybrid method that augments GRPO with an importance-weighted K3 -divergence distillation loss, using retrieved skills as privileged context to construct the teacher signal. • GRPO+OPSD (Algorithm 5): a hybrid method that simply adds the OPSD distillation loss from πref as an auxiliary objective on top of GRPO training. • RLSD (Yang et al., 2026a) (Algorithm 6): a hybrid method that re-weights GRPO’s advantages with self-teacher’s gap. Algorithm 1 SDAR Require: Policy πθ , task set S , skill library E = {e1 , . . . , e M }, group size G, mixing coefficient λ, sharpness β, clip bound ϵ 1: for each training iteration do 2: Sample a batch of tasks { x } from S 3: for each task x do 4: Retrieve skill c+ from E ▷ UCB / KM / Full / Random 5: // Step 1: On-policy rollout 6: Sample G responses {y(1) , . . . , y(G) } ∼ πθ (· | x ) 7: // Step 2: Sequence-level advantage from environment 8: for i = 1, . . . , G do 9: Obtain reward R( x, y(i) ) from environment interaction 10: end for R( x,y(i) )−µ G 11: Compute A(i) = ▷ Group-relative advantage σG 12: // Step 3: GRPO policy loss 13: for i = 1, . . . , G do 14: for t = 1, . . . , |y(i) | do 15: 16: 17: 18: 19: 20: 21: 22: 23: 24:

(i )

rt ← πθ (yt | st ) / πθold (yt | st ) end for end for (i ) Compute LGRPO via clipped surrogate with { A(i) , rt } // Step 4: Token-level gated distillation for i = 1, . . . , G do Compute teacher logits via forward pass with ( x, c+ , y(i) ) for t = 1, . . . , |y(i) | do (i )

(i )

∆t ← sg(log πθ (yt | s+ t ) − log πθ ( yt | st )) gt ← σ ( β · ∆ t )

25: ℓt ← gt · (log πθ (yt | s+ t ) − log πθ ( yt | st )) 26: end for 27: end for (i ) 28: LSDAR ← G1 ∑iG=1 Agg(ℓt ) 29: // Step 5: Joint policy update 30: Update θ by minimizing L(θ ) = LGRPO (θ ) + λ · L SDAR (θ ) 31: end for 32: end for

Algorithm 2 GRPO Require: Policy πθ , task set S , group size G, clip bounds ϵlo , ϵhi , dual-clip constant c 1: for each training iteration do 2: Sample a batch of tasks { x } from S 3: for each task x do 4: // Step 1: On-policy rollout 5: Sample G responses {y(1) , . . . , y(G) } ∼ πθ (· | x ) 6: // Step 2: Sequence-level advantage from environment 7: for i = 1, . . . , G do 8: Obtain reward R( x, y(i) ) from environment interaction 9: end for R( x,y(i) )−µ G 10: Compute A(i) = ▷ Group-relative advantage σG 11: // Step 3: Clipped surrogate policy loss 12: for i = 1, . . . , G do 13: for t = 1, . . . , |y(i) | do (i )

(i )

14:

rt ← πθ (yt | st ) / πθold (yt | st )

15:

L1 ← − A (i ) r t

16:

L2 ← −(A(i) clip(rt , 1−ϵlo , 1+ϵhi )

17:

ℓt ←

(i )

min(− A(i) c, max( L1 , L2 )) max( L1 , L2 )

if A(i) < 0 otherwise

18: end for 19: end for (i ) 20: LGRPO ← Agg({ℓt }) 21: // Step 4: Policy update 22: Update θ by minimizing L(θ ) = LGRPO (θ ) 23: end for 24: end for

Algorithm 3 OPSD Require: Policy πθ , frozen reference πref , task set S , group size G, KL coefficient α 1: for each training iteration do 2: Sample a batch of tasks { x } from S 3: for each task x do 4: // Step 1: On-policy rollout 5: Sample G responses {y(1) , . . . , y(G) } ∼ πθ (· | x ) 6: // Step 2: Token-level KL distillation from reference 7: for i = 1, . . . , G do 8: for t = 1, . . . , |y(i) | do (i )

(i )

9: dt ← log πθ (yt | st ) − log πref (yt | st ) 10: end for 11: end for (i ) 12: LOPSD ← α · Agg({dt }) 13: // Step 3: Policy update 14: Update θ by minimizing L(θ ) = LOPSD (θ ) 15: end for 16: end for

▷ DKL (πθ ∥πref )

Algorithm 4 Skill-SD Require: Policy πθ , task set S , skill library E = {e1 , . . . , e M }, group size G, distillation coefficient λ, clip bound ϵ 1: for each training iteration do 2: Sample a batch of tasks { x } from S 3: for each task x do 4: Retrieve skill c+ from E ▷ UCB 5: // Step 1: On-policy rollout 6: Sample G responses {y(1) , . . . , y(G) } ∼ πθ (· | x ) 7: // Step 2: Sequence-level advantage from environment 8: for i = 1, . . . , G do 9: Obtain reward R( x, y(i) ) from environment interaction 10: end for R( x,y(i) )−µ G 11: Compute A(i) = ▷ Group-relative advantage σG 12: // Step 3: GRPO policy loss (same as Algorithm 2) 13: 14: 15: 16: 17:

(i )

Compute LGRPO via clipped surrogate with { A(i) , rt } // Step 4: Importance-weighted K3 distillation for i = 1, . . . , G do Compute teacher log-probs via forward pass with ( x, c+ , y(i) ) for t = 1, . . . , |y(i) | do (i )

(i )

18: 19:

dt ← log πθ (yt | st ) − log πθ (yt | s+ t ) k t ← exp(−dt ) − 1 + dt

20:

ρt ← exp(log πθ (yt | st ) − log πθold (yt | st ))

(i )

▷ Student − Teacher ▷ K3 divergence ▷ On-policy IS ratio

21: ℓt ← ρt · k t 22: end for 23: end for (i ) 24: LSkill-SD ← Agg({ℓt }) 25: // Step 5: Joint policy update 26: Update θ by minimizing L(θ ) = LGRPO (θ ) + λ · LSkill-SD (θ ) 27: end for 28: end for

Algorithm 5 GRPO+OPSD Require: Policy πθ , frozen reference πref , task set S , group size G, KL coefficient α, clip bound ϵ 1: for each training iteration do 2: Sample a batch of tasks { x } from S 3: for each task x do 4: // Step 1: On-policy rollout 5: Sample G responses {y(1) , . . . , y(G) } ∼ πθ (· | x ) 6: // Step 2: Sequence-level advantage from environment 7: for i = 1, . . . , G do 8: Obtain reward R( x, y(i) ) from environment interaction 9: end for R( x,y(i) )−µ G 10: Compute A(i) = ▷ Group-relative advantage σG 11: // Step 3: GRPO policy loss (same as Algorithm 2) 12: 13: 14: 15:

(i )

Compute LGRPO via clipped surrogate with { A(i) , rt } // Step 4: Token-level KL penalty toward πref for i = 1, . . . , G do for t = 1, . . . , |y(i) | do (i )

(i )

16: dt ← log πθ (yt | st ) − log πref (yt | st ) 17: end for 18: end for (i ) 19: LOPSD ← α · Agg({dt }) 20: // Step 5: Joint policy update 21: Update θ by minimizing L(θ ) = LGRPO (θ ) + LOPSD (θ ) 22: end for 23: end for

Algorithm 6 RLSD Require: Policy πθ , task set S , skill library E = {e1 , . . . , e M }, group size G, mixing coefficient λ, weight clip bound ϵw , policy clip bound ϵ 1: for each training iteration do 2: Sample a batch of tasks { x } from S 3: for each task x do 4: Retrieve skill c+ from E ▷ UCB / KM / Full / Random 5: // Step 1: On-policy rollout 6: Sample G responses {y(1) , . . . , y(G) } ∼ πθ (· | x ) 7: // Step 2: Sequence-level advantage from environment 8: for i = 1, . . . , G do 9: Obtain reward R( x, y(i) ) from environment interaction 10: end for R( x,y(i) )−µ G 11: Compute A(i) = ▷ Group-relative advantage σG 12: // Step 3: Token-level advantage reweighting via teacher 13: for i = 1, . . . , G do 14: Compute teacher log-probs via forward pass with ( x, c+ , y(i) ) 15: for t = 1, . . . , |y(i) | do (i )

(i )

17:

δt ← log πθ (yt | s+ t ) − log πθold ( yt | st ) wt ← clip(exp(sign( A(i) ) · δt ), 1−ϵw , 1+ϵw )

18: 19: 20: 21:

Ât ← A(i) · [(1 − λ) + λ · wt ] end for end for // Step 4: Clipped surrogate with token-level advantages

16:

(i )

22: Compute LRLSD via clipped surrogate with { Ât , rt } 23: // Step 5: Policy update 24: Update θ by minimizing L(θ ) = LRLSD (θ ) 25: end for 26: end for

▷ Teacher − Student gap

Hyperparameters

Table 3 summarizes the method-specific hyperparameters used for all baselines and SDAR across our experiments. Table 3: Hyperparameters. η: learning rate; G: group size; ϵ: PPO clip ratio; λ: distillation loss coefficient; β: sigmoid gate sharpness; αKL : KL penalty coefficient toward the reference policy; SRS: skill retrieval strategy (KM = keyword matching). η

αKL

SRS

10−6 10−6 10−6 10−6 10−6 10−6 10−6

8 8 — 8 8 8 8

0.2 0.2 — 0.2 0.2 0.2 0.2

— — 0.01 0.001 0.01 0.5 0.01

— — 5.0 — 0.0 — 5.0

0.01 0.01 0.01 0.01 0.01 0.01 0.01

— KM KM KM KM KM KM

Method GRPO Skill-GRPO OPSD Skill-SD GRPO+OPSD RLSD SDAR (Ours)

Training Dynamics

We present the full training dynamics of SDAR across all model scales and environments in Figures 10–14, tracking five diagnostic metrics throughout training.

Qwen2.5-3B

Qwen2.5-7B

0.50

ALFWorld

0.32

0.50

0.45

0.30

0.45 0.40 0.35 0.30

0.28

0.40

0.26

0.35

0.24

0.30

0.22 0.20

0.25 0

100

120

140

Qwen3-1.7B

0.34

0.55

100

120

140

100

120

140

100

120

140

100

120

140

WebShop

0.40 0.45

0.50

0.40

0.45

0.35

0.40

0.30

0.35

0.25

0.30

0.35

0.20 0

100

120

140

0.25

100

120

140

0.375

0.325

0.350

0.40

0.325

0.300

0.300 0.275

0.35

0.250

0.275 0.250

0.225

0.30

0.200

0.225

0.175

0.25

0.200

0.150

0.175 0

Step

100

120

140

Step

100

120

140

0.125

Step

Figure 10: Gate Active Ratio when training Qwen2.5-3B, Qwen2.5-7B and Qwen3-1.7B on ALFWorld, WebShop and Search-QA.

Qwen2.5-3B

Qwen2.5-7B

Qwen3-1.7B

ALFWorld

0.47 0.47

0.49

0.46

0.48

0.45

0.47

0.44

0.46

0.43

0.45

0.46 0.45 0.44 0.43

100

120

140

100

120

140

0.49

0.475

WebShop

0.470

100

120

140

100

120

140

100

120

140

0.46

0.460

0.47

0.455

0.45

0.450

0.46

0.44

0.445 0.45

0.440 0

100

120

140

0.485 0.480 0.475

0.47

0.48

0.465

0 0.48

0.470 0.465 0.460 0.455

100

120

140

0.490

0.480

0.485

0.475

0.480

0.470

0.475

0.465

0.470

0.460

0.465

0.455 0.450

0.460

0.450 0

Step

100

120

140

0.43

Step

100

120

140

Step

Figure 11: Gate Mean when training Qwen2.5-3B, Qwen2.5-7B and Qwen3-1.7B on ALFWorld, WebShop and Search-QA.

Prompt

Figures 15–17 present the full prompt templates used by SDAR for the three evaluation environments, where {skill context} is populated with the retrieved skill during training and left empty at inference time.

Qwen2.5-3B

Qwen2.5-7B

Qwen3-1.7B 0.050

ALFWorld

0.070 0.065

0.06

0.060

0.05

0.045 0.040

0.055 0.050

0.04

0.045

0.03

0.035 0.030

0.040 0.02

0.035

0.025 0

100

120

140

0.060

100

120

140

100

120

140

100

120

140

100

120

140

0.040 0.035

0.045

0.04

0.040 0.035

0.030 0.025

0.03

0.020

0.030

0.015

0.02

0.025 0

0.045

0.05

0.050

0 0.050

0.06

0.055

WebShop

100

120

140

100

120

140

0.045

0.040

0.035

0.030

0.025

0.020

0.015

0.015 0

Step

100

120

140

0.030 0.025 0.020 0.015 0

Step

100

120

140

0.010

Step

Figure 12: OPSD Loss when training Qwen2.5-3B, Qwen2.5-7B and Qwen3-1.7B on ALFWorld, WebShop and Search-QA.

Qwen2.5-3B

Qwen2.5-7B 0.2

0.04

0.10

ALFWorld

Qwen3-1.7B

0.02

0.08

0.3

0.06

0.12

0.08

0.14

0.4

0.10

0.16

0.5

0.12

0.18 0

100

120

140

100

120

140

0.02

0.04

WebShop

0.20

0.08

0.25

0.10

0.30

100

120

140

0.14

100

120

140

0.15

0.06

0.20

0.08

0.25

0.10

0.30

0.10

0.12

0.35

0.12

0.14

0.08

Step

100

120

140

100

120

140

100

120

140

100

120

140

0.10

0.04

0.06

0.40 0

0.02 0.04

0.35

0.12 0

0.15

0.06

0.12

0.10

0.04 0.06

0.40 0

Step

100

120

140

Step

Figure 13: Teacher-Student Gap when training Qwen2.5-3B, Qwen2.5-7B and Qwen3-1.7B on ALFWorld, WebShop and Search-QA.

Qwen2.5-3B

ALFWorld

Qwen2.5-7B

Qwen3-1.7B

2.5

2.0

1.5

3 2 1

1.0

0.5

1 0

100

120

140

0.0 0

100

120

140

100

120

140

100

120

140

WebShop Search

0.55

0.45

0.50

0.40

0.45

0.40

0.35

0.40

0.35

Step

100

120

140

100

120

140

100

120

140

0.25

0.25 40

0.30

0.30 20

0.45

0.35

0 0.50

0.50

0.25

140

0 0

0.60

0.30

120

0 20

100

1 0

4 2

6 5

Step

100

120

140

Step

Figure 14: Reward Curve when training Qwen2.5-3B, Qwen2.5-7B and Qwen3-1.7B on ALFWorld, WebShop and Search-QA.

Prompt of SDAR on ALFWorld You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: {task description}. {skill context} Prior to this step, you have already taken {step count} step(s). Below are the most recent {history length} observations and the corresponding actions you took: {action history} You are now at step {current step} and your current observation is: {current observation} Your admissible actions of the current situation are: [{admissible actions}]. Now it’s your turn to take an action. You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.

Figure 15: Prompt template used by SDAR for the ALFWorld task environment.

Prompt of SDAR on Search-based QA You are an expert agent tasked with answering the given question step-by-step. {skill context} Your question: {task description}. Prior to this step, you have already taken {step count} step(s). Below is the interaction history where <search> </search> wrapped your past search queries and <information> </information> wrapped the corresponding search results returned by the external search engine. History: {memory context} Now it’s your turn to respond for the current step. You should first conduct a reasoning process. This process MUST be enclosed within <think> </think> tags. After completing your reasoning, choose only one of the following actions (do not perform both): 1. If you find you lack some knowledge, you MUST call a search engine to get more external information using format: <search> your query </search>. 2. If you have enough knowledge to answer the question confidently, provide your final answer within <answer> </answer> tags, without detailed illustrations. For example, <answer>Beijing</answer>.

Figure 16: Prompt template used by SDAR for the Search-based QA task environment.

Prompt of SDAR on WebShop You are an expert autonomous agent operating in the WebShop e-commerce environment. {skill context} Your task is to: {task description}. Prior to this step, you have already taken {step count} step(s). Below are the most recent {history length} observations and the corresponding actions you took: {action history} You are now at step {current step} and your current observation is: {current observation}. Your admissible actions of the current situation are: [ {available actions} ]. Now it’s your turn to take one action for the current step. You should first reason step-by-step about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.

Figure 17: Prompt template used by SDAR for the WebShop task environment.

Record · ID 187299 · SHA-256 853ce7318607682b
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.