ZEDA
2026-05-19
Post-Trained MoE Can Skip Half Experts via Self-Distillation Xingtai Lv*1 , Li Sheng*1 , Kaiyan Zhang*1,5 , Yichen You1 , Siyan Gao1,4 , Xueheng Luo1 , Yuxin Zuo1 , Yuchen Fan1 , Junlin Yang1,5 , Ganqu Cui2 , Bingning Wang3 , Fan Yang4 , Youbang Sun†1,2 , Ning Ding†1,2 and Bowen Zhou†1,2 1 Tsinghua University, 2 Shanghai AI Lab, 3 WeChat AI, 4 Kuaishou Technology, 5 Frontis.AI ∗ Equal Contributions, † Corresponding Authors
arXiv:2605.18643v1 [cs.LG] 18 May 2026
# [email protected], § TsinghuaC3I/ZEDA
Abstract | Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation, and its dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner. Existing dynamic MoE methods usually rely on pre-training from scratch or task-specific adaptation, leaving the practical conversion of fully trained MoE underexplored. Enabling such adaptation would directly alleviate the inference costs by allowing easy tokens to bypass unnecessary expert during serving. This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones. To stabilize this architectural conversion, ZEDA injects parameter-free zero-output experts into each MoE layer and adapts the augmented model through two-stage self-distillation, utilizing the original MoE as a frozen teacher and applying a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning math, code, and instruction following, ZEDA eliminates over 50% of expert FLOPs at marginal accuracy loss. It outperforms the strongest dynamic MoE baseline by 6.1 and 4.0 points on the two models, and delivers ~1.20× end-to-end inference speedup. Step1: Zero-Expert Injection Post-Trained MoE
Injecting NZ Zero Experts
E1
… EN
Router Input ℎ
𝑥!
E1
Post-Trained MoE Model
Step2.1: SFT ℒ = ℒ"#$ + ℒ%&
SFT Data
Dynamic MoE Model
𝑔2
𝑔N
E2
Dynamic MoE Output 𝑦#
Output 𝑦
𝑔2
Step2: Self-Distillation
Frozen
E2 … EN
Router Input ℎ
Z1 … ZNz Normal Expert Injected Zero Expert
Trainable
Post-Trained MoE Model Step2.2: OPD ℒ = ℒ'() + ℒ%& 𝑥!
Online Rollout
Reverse KL
Dynamic MoE Model
Figure 1 | Illustration of ZEDA. ZEDA leverages the post-trained MoE to initialize the dynamic MoE (with zero-expert injection) and further utilizes it as a teacher model for distillation.
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Contents 1 Introduction
3
2 Method
4
2.1 Adaptation Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4
2.2 Group Auxiliary Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
3 Experiments
6
3.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
3.2 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
3.3 Inference Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
4 Analysis
8
4.1 Zero Expert Activation Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
4.2 Effect of Adaptation Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
4.3 Ablation Studies on ZEDA Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
4.3.1 Effect of 𝑤 and 𝑟𝑍𝐸 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
4.3.2 L𝐺 𝐴 Coefficient 𝛼 Ablation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
4.3.3 Effect of Training Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
4.3.4 Impact of Router Probability Renormalization . . . . . . . . . . . . . . . . . .
12
4.4 Out-of-Distribution Generalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
5 Related Work
13
5.1 Dynamic Expert Activation in Mixture-of-Experts LLMs . . . . . . . . . . . . . . . . .
13
5.2 Self-Distillation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
6 Conclusion
14
A Limitations and Future Work
20
B Zero Experts versus Copy Experts
20
C Auxiliary-Loss Comparison
22
D Theoretical FLOPs Analysis
22
D.1 Shared MoE Cost Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
D.2 Prefill Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
D.3 Decode Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
D.4 Numerical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
2
Post-Trained MoE Can Skip Half Experts via Self-Distillation
1. Introduction Mixture-of-Experts (MoE) architectures have significantly advanced the scaling of large language models (LLMs) by increasing model capacity while keeping bounded per-token computation [Lepikhin et al., 2020; Fedus et al., 2022; Du et al., 2022; Dai et al., 2024; Jiang et al., 2024]. Building upon this foundation, a variant we refer to as dynamic MoE, further introduces token-level dynamism that adjusts the number of activated experts, enabling an input-dependent allocation of computation budgets [Jin et al., 2024; Team et al., 2025; Wu et al., 2025; Guo et al., 2024; Chaudhari et al., 2026; Zeng et al., 2024]. Many studies have demonstrated that easy tokens can be processed with substantially fewer experts without compromising output quality, making dynamic MoE a principled route to inference-time efficiency [Jin et al., 2024; Team et al., 2025; Zeng et al., 2024; Lu et al., 2024; Huang et al., 2024]. Most existing approaches to dynamic MoE concentrate on either pre-training dynamic MoE models from scratch [Jin et al., 2024; Team et al., 2025; Chaudhari et al., 2026] or adapting a pre-trained base model into a task-specific dynamic MoE [Zeng et al., 2024], leaving the migration of fully trained MoE models largely unexplored. Yet in practical deployment, MoE models have typically undergone an extensive training pipeline encompassing both pre-training and post-training such as supervised fine-tuning (SFT), reinforcement learning (RL), and on-policy distillation (OPD) [Team, 2026; Zeng et al., 2026; DeepSeek-AI, 2026]. We refer to such models as post-trained MoE throughout this paper. If such a post-trained static MoE model could be converted into a more efficient dynamic counterpart with the architecture and primary training already finalized, the resulting inference savings would be of tremendous practical value given the ever-growing serving costs and demand. However, directly applying existing dynamic MoE methods to such models risks disrupting the carefully calibrated routing and capability distributions established during the full training pipeline. In this paper, we focus on exploring whether a post-trained MoE model can be cost-effectively migrated into a more efficient dynamic MoE without sacrificing its established capabilities. We introduce Zero-Expert Self-Distillation Adaptation (ZEDA), transforming a post-trained MoE model into a dynamic one with faster inference at minimal adaptation cost. ZEDA injects parameterless zero experts [Jin et al., 2024; Team et al., 2025], whose outputs are identically zero, into the existing expert pool of a post-trained MoE model. This expands the router candidate pool with zero-computation experts while the activation number remains unchanged, naturally reducing active normal experts. The augmented model is then adapted through a two-stage self-distillation process comprising SFT [Ouyang et al., 2022] and OPD [Gu et al., 2023; Agarwal et al., 2024; Lu and Lab, 2025], using the original MoE as a fixed teacher, to recover performance under the new dynamic routing regime. To make this architectural conversion stable, ZEDA further introduces a Group Auxiliary Loss L𝐺 𝐴 that regulates the relative activation frequency between normal experts and zero experts while preserving the learned routing structures among normal experts. Experiments on Qwen3-30B-A3B [Yang et al., 2025] and GLM-4.7-Flash [Zeng et al., 2025] across 11 benchmarks spanning math, code, and instruction following demonstrate the effectiveness of ZEDA. Our method successfully migrates post-trained MoE models into dynamic ones in less than 31 hours for Qwen and 62 hours for GLM on 8 NVIDIA H200 GPUs. This adaptation eliminates over half of the expert computation and achieves an inference speedup around 20%, while incurring only a marginal accuracy loss compared with the original model. ZEDA outperforms the strongest baseline by an average of 6.1 points on Qwen and 4.0 points on GLM, and also achieves the best overall performance among our proposed variants. Through detailed illustrative visualizations and analysis, the dynamic characteristics of the zero expert activation and the operating mechanisms of ZEDA are clearly revealed. The following are several key takeaways:
3
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Takeaways
1. ZEDA cost-effectively converts post-trained MoE models into dynamic ones across diverse domains, reducing over half of the MoE computation while maintaining performance (§ 3). 2. Injecting zero experts and group-level balancing strategy minimize disruptions to original routing distribution, facilitating stable adaptation of post-trained MoE (§ 2). 3. Zero-expert activation (compute allocation) intrinsically correlates with the teacher-student distribution gap, model uncertainty, and response patterns, not overall task difficulty (§ 4.1).
2. Method We propose Zero-Expert Self-Distillation Adaptation (ZEDA), a method that transforms a post-trained MoE model into a dynamic one with faster inference at minimal adaptation cost, by augmenting each MoE module with zero experts and adapting the expanded model through self-distillation. In the following, we present the overall adaptation framework in Section 2.1, and then introduce the Group Auxiliary Loss L𝐺 𝐴 that regulates zero expert utilization in Section 2.2. 2.1. Adaptation Framework ZEDA first injects zero experts [Jin et al., 2024], whose outputs are identically zero, into a post-trained MoE, architecturally converting it into a dynamic one whose activated normal experts number varies across tokens. The augmented model is then adapted through the two-stage self-distillation with the original post-trained MoE as a fixed teacher, yielding a more efficient dynamic MoE with negligible performance loss. Zero-Expert Injection. Consider a post-trained MoE model where each MoE module contains 𝑁 normal experts E = { 𝐸1 , . . . , 𝐸 𝑁 } and activates 𝐾 of them per token. For an input hidden state ℎ, the Í router selects a top- 𝐾 subset S( ℎ) ⊆ E and produces 𝑦 ( ℎ) = 𝑖 ∈ S ( ℎ ) 𝑔𝑖 ( ℎ) 𝐸 𝑖 ( ℎ), where 𝑔𝑖 ( ℎ) is the normalized routing weight for expert 𝐸 𝑖 . ZEDA introduces 𝑁𝑍 additional experts Z = { 𝑍1 , . . . , 𝑍 𝑁𝑍 } that satisfy 𝑍 𝑗 ( ℎ) = 0 for all 𝑗, referred to as zero experts. The augmented expert pool E ′ = E ∪ Z expands the router from 𝑁 to 𝑁 + 𝑁𝑍 candidates while the top- 𝐾 budget remains unchanged. The dynamic MoE output becomes ∑︁ ˜𝑦 ( ℎ) = ˜𝑔𝑖 ( ℎ) 𝐸 𝑖 ( ℎ) , (1) 𝑖 ∈ S̃ ( ℎ )∩E
where S̃( ℎ) denotes the top- 𝐾 set selected from E ′ and ˜𝑔𝑖 ( ℎ) is the corresponding routing weight. Because zero experts contribute no computation, selecting them reduces the number of active normal experts, yielding token-dependent computation without modifying the normal expert parameters. We also compare the zero expert with another zero-computation alternative, copy expert, which outputs its input, in Appendix B, showing that copy experts induce both scale and direction mismatches. For router initialization, the original router parameters for the 𝑁 normal experts are kept unchanged. The new parameters for the 𝑁𝑍 zero experts are drawn from a Gaussian distribution matching the mean and variance of the original router parameters in the same module, preserving the post-trained scale of router logits while inserting new routing options. Two-Stage Self-Distillation. ZEDA then adapts the augmented model via self-distillation, using the original MoE as a fixed teacher. The adaptation proceeds in two stages, supervised fine-tuning (SFT) 4
Post-Trained MoE Can Skip Half Experts via Self-Distillation
followed by on-policy distillation (OPD). Let 𝜋𝑇 denote the teacher (original MoE) distribution, 𝜋𝜃 the student (augmented model) distribution, and D the prompt set used for adaptation. • The SFT stage trains 𝜋𝜃 on responses sampled from the teacher 𝜋𝑇 . The training loss is: L = LSFT + LGA = −𝔼𝑥 ∼D , 𝑦 ∼𝜋𝑇 (· | 𝑥 )
" |𝑦| ∑︁
# log 𝜋𝜃 ( 𝑦𝑡 | 𝑥, 𝑦<𝑡 ) + LGA ,
(2)
𝑡 =1
where 𝑥 is a prompt from D, 𝑦 = ( 𝑦1 , . . . , 𝑦 | 𝑦 | ) is a teacher-sampled response, and LGA is the group auxiliary loss introduced in Section 2.2. • The subsequent OPD stage [Gu et al., 2023; Agarwal et al., 2024] shifts to on-policy learning, where responses are sampled from the current student 𝜋𝜃 and the teacher evaluates the same trajectories to supply token-level targets. Following Thinking Machines [Lu and Lab, 2025], we cast the sampled-token reverse KL objective as a reward signal and optimize it within the policy optimization framework, yielding the training loss: L = LOPD + LGA = 𝔼𝑥 ∼D , 𝑦 ∼𝜋𝜃 (· | 𝑥 )
" |𝑦| ∑︁
# KL( 𝜋𝜃 (· | 𝑥, 𝑦<𝑡 ) ∥ 𝜋𝑇 (· | 𝑥, 𝑦<𝑡 )) + LGA .
(3)
𝑡 =1
The SFT stage stabilizes the initial transition from a static to a dynamic MoE, and the OPD stage further aligns the student with the teacher under the student’s own rollout distribution. 2.2. Group Auxiliary Loss ZEDA incorporate the Group Auxiliary Loss L𝐺 𝐴 to regulate the relative activation frequency between normal experts and zero experts, thereby controlling the zero expert activation ratio 𝑟𝑍𝐸 . Auxiliary Loss. L𝐺 𝐴 is derived from the vanilla auxiliary load balancing loss L 𝐴 [Lepikhin et al., 2020; Fedus et al., 2022], which encourages uniform routing across all experts. L 𝐴 is defined through the batch B: 𝑁 +𝑁
L𝐴 = 𝛼 ·
𝑁 + 𝑁𝑍 ∑︁𝑍 𝑓𝑖 · 𝑃 𝑖 , · 𝐾 𝑖=1
where
𝑓𝑖 =
o 1 ∑︁ n 1 ∑︁ 1 𝑖 ∈ S̃( ℎ) , 𝑃𝑖 = ˜𝑔𝑖 ( ℎ) . |B| ℎ ∈ B |B| ℎ ∈ B
(4)
Here, 𝑓𝑖 denotes the fraction of tokens in a batch B routed to expert 𝑖, 𝑃𝑖 is the mean routing probability assigned to expert 𝑖 over B, and 𝛼 is a scalar loss coefficient. However, applying L 𝐴 directly in ZEDA is problematic. A post-trained MoE model exhibits non-uniform, input-dependent routing patterns over normal experts, and enforcing expert-level uniformity would disrupt these learned distributions, degrading model performance. Appendix C presents a dedicated experiment comparing L 𝐴 and L𝐺 𝐴 . Group Load Balancing Loss. The objective of ZEDA is to regulate zero expert utilization while preserving the relative routing structure among normal experts. This motivates a group-level balancing strategy in which the 𝑁 normal experts form a group E and the 𝑁𝑍 zero experts form a group Z, with balancing applied only between the two groups. The Group Auxiliary Loss is defined as 𝑓Z · 𝑃 Z 𝑁 + 𝑁𝑍 · 𝑤 𝑓E · 𝑃E , (5) L𝐺 𝐴 = 𝛼 · · + 𝐾 𝑁 𝑁𝑍 · 𝑤
5
Post-Trained MoE Can Skip Half Experts via Self-Distillation
where
𝑓E =
∑︁ 𝑖∈ E
𝑓𝑖 ,
𝑃E =
∑︁ 𝑖∈ E
𝑃𝑖 ,
𝑓Z =
∑︁ 𝑖∈ Z
𝑓𝑖 ,
𝑃Z =
∑︁
𝑃𝑖 .
(6)
𝑖∈ Z
𝑤 > 0 is the relative weight of the zero-expert group, and a larger 𝑤 encourages higher 𝑟𝑍𝐸 .
Analogously to L 𝐴 , minimizing L𝐺 𝐴 drives the two groups toward an equilibrium in which the expected number of activated normal experts 𝐾 E and zero experts 𝐾 Z satisfy 𝐾 E : 𝐾 Z = 𝑁 : 𝑁𝑍 · 𝑤, yielding a target 𝑟𝑍𝐸 = ( 𝑁𝑍 · 𝑤)/( 𝑁 + 𝑁𝑍 · 𝑤) . Since the constraint is imposed only at the group level, it does not explicitly flatten the routing distribution within the normal-expert group, which makes it better aligned with post-trained MoE adaptation. L𝐺 𝐴 drives 𝑟𝑍𝐸 toward the target value, while the other loss component (LSFT or LOPD ) optimizes performance. Under the joint influence, the model reaches a trade-off, causing 𝑟𝑍𝐸 to converge to an appropriate value.
3. Experiments 3.1. Experimental Setup Models. To evaluate the generalizability of ZEDA across different backbone architectures, two post-trained MoE models are selected: Qwen3-30B-A3B [Yang et al., 2025] and GLM-4.7-Flash [Zeng et al., 2025]. Qwen3-30B-A3B is consistently used in Thinking mode throughout all experiments. The two models differ in scale and expert configuration. Qwen3-30B-A3B contains 𝑁 =128 normal experts with 𝐾 =8 activated per token, while GLM-4.7-Flash has 𝑁 =64 and 𝐾 =4. Following LongCat [Team et al., 2025], the number of injected zero experts 𝑁𝑍 is set to 64 and 32 for Qwen3-30B-A3B and GLM-4.7-Flash, respectively. Evaluation Setup. To comprehensively assess the post-adaptation performance, ZEDA is evaluated on 11 benchmarks spanning 3 categories. For math reasoning, the benchmarks include AIME 24, AIME 25, AIME 26 [Li et al., 2024], GSM8K [Cobbe et al., 2021], and MATH-500 [Lightman et al., 2023]. For code generation, the benchmarks include LiveCodeBench v5 (LCB v5), LiveCodeBench v6 (LCB v6) [Jain et al., 2024], HumanEval+ [Liu et al., 2023], and MBPP+ [Liu et al., 2023]. HumanEval+ and MBPP+ are two code generation benchmarks introduced by EvalPlus [Liu et al., 2023]. For instruction following, the benchmarks include IFEval [Zhou et al., 2023] and IFBench [Pyatkin et al., 2025]. All evaluations adopt a temperature of 0.6, a top- 𝑝 value of 0.95, and a top-𝑘 value of 20, with a maximum generation length of 38k tokens following the Qwen3 setting [Yang et al., 2025]. We report avg@32 for AIME24, AIME25, and AIME26 to reduce variance on these small-scale competition benchmarks, avg@8 for the 4 coding benchmarks, and avg@1 for all remaining benchmarks. Following the conventions of Qwen3 [Yang et al., 2025] and IFBench [Pyatkin et al., 2025], results on IFEval and IFBench are reported as strict prompt accuracy and loose prompt accuracy, respectively. Implementation Details. For the inference efficiency of the adapted dynamic MoE, the relative weight 𝑤 in L𝐺 𝐴 (Eq. 5) is set to 2, which drives the target 𝑟𝑍𝐸 toward 50%, and the loss coefficient 𝛼 is set to 0.1. Ablation studies on 𝑤 and 𝛼 are presented in Section 4.3.1 and Section 4.3.2, respectively. The self-distillation data consists of 60k prompts in total. It consists of 17k math prompts and 15k coding prompts randomly sampled from NVIDIA AceReason-1.1-SFT [Liu et al., 2025], together with 28k chat prompts randomly sampled from NVIDIA Llama-Nemotron-Post-Training-Dataset [Bercovich et al., 2025]. In the SFT stage, the learning rate is set to 2 × 10−5 . The subsequent stage employs Sampled-Token OPD with a learning rate of 5 × 10−6 for Qwen3-30B-A3B and 1 × 10−6 for GLM-4.7Flash, a batch size of 16 prompts × 2 sampled responses, a sampling temperature of 1.0, a maximum generation length of 32k tokens, and runs for 320 training steps. All experiments are conducted 6
Post-Trained MoE Can Skip Half Experts via Self-Distillation
on the slime [Zhu et al., 2025], SGLang [Zheng et al., 2024], and Megatron [Shoeybi et al., 2019] codebases, and on NVIDIA H200 and H20 GPUs. Baselines. AdaMoE [Zeng et al., 2024] and the Dynamic Skipping method in [Lu et al., 2024] serve as the dynamic routing baselines. We further propose three variants to evaluate the efficacy of ZEDA’s components. ZEDASFT , which applies only the SFT stage of ZEDA, is included to isolate the contribution of OPD. To validate the dynamic expert selection mechanism, we propose Naive Expert Truncation (NET), a straightforward variant of ZEDA that directly halves the number of activated experts in the original MoE model. NET is combined with SFT alone or SFT followed by OPD, yielding NETSFT and NETSFT→OPD , respectively. More experimental setup details are reported in Appendix ??. 3.2. Main Results Performance. Table 1 summarizes the performance of all methods on 11 benchmarks spanning mathematical reasoning, code generation, and instruction following. Compared with the original post-trained MoE, ZEDA incurs only a marginal average accuracy loss while eliminating over half of the expert computation, and even surpasses the original model on several individual benchmarks such as IFBench, demonstrating the practical utility of the dynamic MoE models produced by ZEDA. Among all baselines, ZEDA achieves the highest average evaluation scores on both Qwen3-30B-A3B and GLM-4.7-Flash, indicating its effectiveness and robustness across architectures. Furthermore, ZEDA achieves superior overall performance over all three variants, ZEDASFT , NETSFT and NETSFT→OPD , demonstrating the contributions of OPD and the dynamic expert selection mechanism. Moreover, the dynamic routing baselines exhibit severe capability imbalances, where AdaMoE collapses on hard reasoning like AIME 24 and Dynamic Skipping fails on code generation. ZEDA is the only method preserving competitive performance uniformly across all domains. Finally, ZEDA achieves average 𝑟𝑍𝐸 values of 51.2% on Qwen and 53.0% on GLM, exceeding or matching the baselines, indicating that ZEDA attains better performance with comparable or lower computation. Table 1 | Performance of ZEDA and baselines on Qwen3-30B-A3B and GLM-4.7-Flash. Math
Code
IF
Method
Avg Acc
Avg 𝑟𝑍𝐸
AIME 24
AIME 25
AIME 26
GSM8k
MATH-500
LCB v5
LCB v6
HumanEval+
MBPP+
IFBench
IFEval
Qwen3-30B-A3B AdaMoE Dynamic Skipping NETSFT NETSFT→OPD ZEDASFT ZEDA
74.9 54.8 68.1 72.3 73.0 73.3 74.2
0.0 51.9 43.8 50.0 50.0 51.5 51.2
80.9 25.0 78.1 76.8 79.5 78.1 79.0
71.0 24.8 67.9 65.7 67.6 66.2 69.1
72.3 36.7 72.5 72.1 70.6 71.2 72.5
95.4 92.4 95.2 94.7 95.4 94.8 95.5
94.4 79.8 94.4 94.0 94.6 94.4 95.2
61.5 36.1 57.3 56.5 57.0 58.2 58.2
57.1 34.3 51.9 50.9 52.9 52.8 53.2
85.6 80.3 59.1 86.7 87.4 86.8 88.5
79.2 72.8 70.0 78.2 77.5 78.6 78.2
39.7 38.7 32.0 37.7 38.7 39.7 42.3
86.3 82.4 70.4 82.4 81.7 85.2 84.3
GLM-4.7-Flash AdaMoE Dynamic Skipping NETSFT NETSFT→OPD ZEDASFT ZEDA
72.5 57.1 67.8 70.6 70.9 70.9 71.8
0.0 47.0 37.5 50.0 50.0 52.8 53.0
84.2 44.1 79.9 78.2 78.6 78.1 79.8
76.5 42.4 69.9 71.4 71.8 71.4 73.1
74.0 47.3 74.8 68.3 72.9 71.9 74.4
95.2 93.9 93.8 94.1 94.2 95.2 94.4
96.4 86.4 96.0 95.0 95.6 95.0 95.2
48.0 26.4 32.3 50.6 49.5 49.9 51.6
44.4 28.6 32.4 44.7 43.8 45.0 45.6
89.0 82.7 86.3 86.8 87.1 88.1 86.3
75.7 69.5 71.6 74.2 74.3 72.3 73.5
47.3 43.0 45.3 47.0 46.7 46.7 47.3
67.3 63.8 63.4 65.8 65.1 66.4 68.2
Adaptation Time. Table 2 reports the training time of the ZEDA pipeline. ZEDA requires less than 31 hours for Qwen3-30B-A3B and 62 hours for GLM-4.7-Flash on 8 H200 GPUs, which is negligible compared with prior MoE pre-training and post-training costs, demonstrating its cost-effectiveness.
7
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Table 2 | Adaptation time (hours) of Qwen3-30B-A3B and GLM-4.7-Flash on 8 NVIDIA H200 GPUs Qwen3-30B-A3B
All
SFT Data Rollout
SFT
OPD
30.12
8.16
1.97
19.99
GLM-4.7-Flash
All
SFT Data Rollout
SFT
OPD
61.37
14.56
4.51
42.30
3.3. Inference Efficiency ZEDA yields average zero-expert activation ratios (𝑟ZE ) of 51.2% and 53.0% on Qwen and GLM respectively, effectively halving expert-level computation. We further demonstrate the practical inference speedups achieved by the resulting dynamic MoE. Inference efficiency is evaluated by comparing the original model with its ZEDA-adapted counterpart at 8192 sequence length, using SGLang [Zheng et al., 2024] as the inference framework with the maximum concurrency set to 32. We randomly sample 256 examples from the training data to construct the test set. To ensure a fair comparison across models, for each target sequence length we control the total numbers of input and output tokens to be identical across compared models and to match the intended test sequence length. In addition, the input sequence content is kept exactly the same across models. We report the throughput results on 1 × H200 GPU. We measure both prefill and decode efficiency, and we also provide the theoretical analysis of inference efficiency in Appendix D. As shown in Figure 2, ZEDA delivers consistent inference gains across both backbone models, achieving approximately 20% speedup during the prefill and decode phases, demonstrating its effectiveness in improving model’s inference efficiency. Throughput (10 3 token¢s ¡1 )
Prefill
Decode +19%
+18% 50
2 +19% Original MoE
40 1.5
+26% 30
43.92
1.76
1.74 32.36
20
0
2.07
51.63
1
ZEDA
1.48
25.73
Qwen3-30B-A3B
GLM-4.7-Flash
0
Qwen3-30B-A3B
GLM-4.7-Flash
Figure 2 | Inference efficiency comparison between the original MoE and ZEDA at 8192 sequence length. Speedup is defined relative to the original MoE.
4. Analysis We provide a detailed analysis of the dynamic characteristics of zero expert activation (§ 4.1), the effects of different adaptation durations (§ 4.2), ablation studies on zero-expert group weight 𝑤(§ 4.3.1), L𝐺 𝐴 coefficient 𝛼(§ 4.3.2), training stages(§ 4.3.3), and router probability renormalization(§ 4.3.4), and ZEDA’s performance on OOD tasks (§ 4.4). 4.1. Zero Expert Activation Dynamics ZEDA transforms a static MoE model into a dynamic one in which different tokens exhibit different 𝑟𝑍𝐸 values, corresponding to varying computation amounts. This section provides a deeper investigation into this token-level dynamism, using Qwen3-30B-A3B. The analysis examines how 𝑟𝑍𝐸 relates to 8
Post-Trained MoE Can Skip Half Experts via Self-Distillation
distillation signals, response patterns, task difficulty, and layer-wise behavior, aiming to establish connections between computation allocation in the dynamic MoE and other interpretable metrics. Teacher-Student Logp-Diff and Entropy. To analyze factors affecting token-level 𝑟𝑍𝐸 , 110 prompts (10 per benchmark) are sampled and decoded with the ZEDA-adapted dynamic MoE model. For each generated token, we record the student log probability log 𝜋𝜃 ( 𝑦𝑡 | 𝑥, 𝑦<𝑡 ) and entropy, and compute the teacher log probability log 𝜋𝑇 ( 𝑦𝑡 | 𝑥, 𝑦<𝑡 ) on the same token to obtain the teacher-student logp-diff Δlogp . Figure 3 visualizes all tokens from the 110 prompts. Tokens with larger Δlogp or higher entropy tend to have lower 𝑟𝑍𝐸 , clustering in the upper-left. the dynamic MoE intrinsically allocates more computation, i.e., activates fewer zero experts, when the teacher-student distributional gap or model uncertainty is larger.
Token jlog ¼µ ¡ log ¼T j
30
5
Token Entropy
25 20 15 10
3 2 1
5 0 0.2
4
0 0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.2
Zero Expert Activation Ratio (rZE )
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Zero Expert Activation Ratio (rZE )
Figure 3 | Distribution of token-level 𝑟𝑍𝐸 versus teacher-student logp-diff (left) and entropy (right) for all generated tokens across 110 rollouts. Each point represents a single token.
Response Pattern. Aligning per-token 𝑟𝑍𝐸 of the 110 sampled responses with the decoded text reveals a clear relationship between 𝑟𝑍𝐸 and response pattern. Figure 4 presents 3 representative examples. Compared with natural text, code fragments and mathematical expression exhibit notably higher 𝑟𝑍𝐸 , indicating that the model intrinsically assigns less computation to these structured segments. Since math and code rollouts often contain many such segments after the thinking process, their average 𝑟𝑍𝐸 tends to increase toward the response end, while instruction-following responses show a more uniform 𝑟𝑍𝐸 distribution, as illustrated in Figure 5. Task Difficulty. The relationship between 𝑟𝑍𝐸 and task difficulty is further investigated. Table 3 reports the 𝑟𝑍𝐸 and performance of ZEDA on MATH-500, which provides human-annotated difficulty levels, and on AIME24, a generally considered more challenging task. ZEDA achieves comparable performance and 𝑟𝑍𝐸 across all five difficulty levels of MATH-500, and the corresponding 𝑟𝑍𝐸 values remain close to those observed on AIME24. This suggests that 𝑟𝑍𝐸 is largely independent of task difficulty. The model adjusts computation allocation based on the token-level characteristics within a single response rather than the overall difficulty of the task itself. Table 3 | Performance and 𝑟𝑍𝐸 of ZEDA on five difficulty level tasks of MATH-500 and AIME24. MATH-500
Performance (𝑟𝑍𝐸 )
AIME 24
Level 1
Level 2
Level 3
Level 4
Level 5
95.3 (51.1)
95.6 (51.8)
97.1 (52.2)
94.5 (52.5)
94.0 (52.5)
79.0 (52.1)
9
Post-Trained MoE Can Skip Half Experts via Self-Distillation Zero Expert Activation Ratio 0.0
1.0
<think>
<think>
<think>
Okay, so I need to find the distance from a point I in
Okay, I need to solve this problem where I have to fin
Okay, the user wants me to improve the academic tone o
side a tetrahedron ABCD to each of its faces , given th
d the maximum number of buildings that can be chosen s
f their text. Let me start by reading the original pas
e edge lengths . The tetrahedron has AB=CD=√41, AC=BD=√
uch that they have the same height and are equally spa
sage carefully . The main points are about the Anthrop o
80, and BC=AD=√89. The distance is supposed to be in t
ced. Let me think about how to approach this.
cene, its definition , the term's origin , and the impli
he form m√n/p, and
cations of human impact on Earth 's systems . ...
So the problem says that the chosen buildings must sat
sqrt{n}}{p} $, where :
isfy two conditions : same height and arranged at equal
First, I need to check for any informal language or se
- $ m = 20 $
intervals . The maximum number possible is what we nee
ntence structures that might not fit an academic tone.
- $ n = 21 $
d to find. And if there
Words like "ir ...
- $ p = 63 $
...
dp[j][d] = prev + 1
and Earth 's systems , as evidenced by the Anthrop ocene
---
else: dp[j][d] = prev + 1
ience, incorporating insights from geology, ecology , a
### Step 6: Final Answer
if dp[j][d] > current _max:
nd social sciences to foster a comprehensive understan
current _max = dp[j][d]
ding of global transformations . Ultimately , the recogn
The problem asks for $ m + n + p $, which is: $$
, necess itates a holistic approach to environmental sc
if current _max > max_answer :
ition of the Anthrop ocene marks a paradigm shift in ho
max_answer = current _max
w humanity perce ives its place within the Earth 's dyna
print (max_answer )
mic history , demanding a renewed commitment to respons
20 + 21 + 63 = \boxed{104}
ible environmental practices and sustainable developme
$$<|im_end|>
if __name__ == "__main__":
nt.<|im_end|>
main() ```<|im_end|>
Figure 4 | Visualization of per-token 𝑟𝑍𝐸 for decoded text, showing one sampled response from AIME24 (left), LiveCodeBench v5 (middle), and IFBench (right), respectively. Due to space constraints, only the first and last 80 tokens of each response are retained. LiveCodeBench v5
IFBench
1.0
0.8
Layer Index
12 0.6 24 0.4 36
47
0.2
2000
4000
6000
8000
Token Chunk
10000
2000
4000
6000
8000
Token Chunk
10000
200
400
600
800
1000
Zero Expert Activation Ratio
AIME24
0
0.0
Token Chunk
Figure 5 | Visualization of 𝑟𝑍𝐸 across layers and response positions, using the same data in Figure 4. Token-level 𝑟𝑍𝐸 values are averaged over chunks of size 1000 (for AIME24 and LiveCodeBench v5) and 100 (for IFBench), respectively. The last chunk is averaged over its actual token number. Layer. For each of the 110 responses, 𝑟𝑍𝐸 on the 48 MoE layers of the dynamic model is computed. Figure 5 presents the layer-wise 𝑟𝑍𝐸 distributions for 3 representative cases. Although minor variations exist across layers, the differences are relatively small and exhibit no systematic pattern. Connecting the Observations. The above analyses reveal that 𝑟𝑍𝐸 is uncorrelated with task difficulty yet strongly related to teacher-student logp-diff. This can be explained by the nature of the selfdistillation training data. Diverse sources of the training data make sample-level accuracy-based difficulty signals generally unavailable. In contrast, larger Δlogp directly implies larger LSFT (Eq. 2) and LOPD (Eq. 3). When these task losses dominate, the relative influence of LGA , which encourages higher 𝑟𝑍𝐸 , diminishes, leading to lower 𝑟𝑍𝐸 for such tokens. Furthermore, the correlations of 𝑟𝑍𝐸 with entropy and response patterns align with prior findings. Tokens with higher student-model entropy tend to exhibit larger Δlogp [Ko et al., 2026], and low-entropy tokens are often code or math expressions [Wang et al., 2025].
10
Post-Trained MoE Can Skip Half Experts via Self-Distillation
4.2. Effect of Adaptation Cost To study the scaling trend of zero expert adaptation, we track the average benchmark score and 𝑟𝑍𝐸 throughout the SFT stage. As illustrated in Figure 6 (left), both metrics exhibit similar evolution as the amount of SFT data and GPU hours increases, which is a rapid initial ascent followed by convergence at approximately 60k prompts. This trend indicates that the majority of useful adaptation happens relatively early, when the router is learning how to incorporate the injected zero experts while preserving the backbone’s original capabilities. After this point, additional supervised adaptation mainly provides incremental refinements. These empirical results justify our 60k self-distillation prompt setting, as this scale achieves a performance plateau and stable routing patterns. The observed "dual saturation" underscores the sample efficiency of ZEDA, demonstrating that the modified architecture can reach a stable, high-performance state with affordable post-training costs. 0.0 7.3 14.8
#Data (k)
29.8
44.8
59.9
74.1
68.0
AVG Score AVG rZE
66.0 0.0 9.9 20.1
40.3
60.5
81.0
100.2
Training Time (GPU Hour)
47.0
46.0
74.0 60.0 73.0
AVG Score AVG rZE
72.0 71.0
50.0
40.0
AVG rZE (%)
48.0
70.0
AVG Score (%)
49.0
72.0
AVG rZE (%)
AVG Score (%)
74.0
70.0 1
1.5
2
2.5
3
3.5
4
Relative Weight w of the Zero-Expert Group
Figure 6 | Scaling trend of average benchmark score and 𝑟𝑍𝐸 over different amounts of SFT data and adaptation time (left). Effect of the zero-expert group weight on average score and 𝑟𝑍𝐸 (right).
4.3. Ablation Studies on ZEDA Design 4.3.1. Effect of 𝑤 and 𝑟𝑍𝐸 To investigate the impact of group-level balancing strength and the zero expert activation ratio, we vary the zero-expert group weight 𝑤 and analyze its effect on model performance and routing. As shown in Figure 6 (right), increasing 𝑤 monotonically elevates 𝑟𝑍𝐸 but leads to a gradual decline in benchmark scores. This confirms that 𝑤 serves as an effective control knob for the quality-efficiency trade-off in ZEDA. Empirical results indicate that 𝑤=2 offers the optimal balance, yielding significantly higher zero-expert utilization than 𝑤∈{1, 1.5} while maintaining competitive performance. In contrast, pushing 𝑤 further to 3 or 4 causes a more pronounced accuracy drop. Consequently, we select 𝑤=2 as the preferred operating point. 4.3.2. L𝐺 𝐴 Coefficient 𝛼 Ablation The loss coefficient 𝛼 in Eq. 5 controls how strongly the Group Auxiliary Loss L𝐺 𝐴 influences the overall training objective. To investigate its effect, the SFT stage of ZEDA is conducted on Qwen3-30B-A3B with 𝛼 varied across {0.001, 0.01, 0.1, 1.0}, while the relative weight 𝑤 is fixed at 2, corresponding to a target 𝑟𝑍𝐸 of 50%. As shown in Table 4, at 𝛼=0.1, the observed 𝑟𝑍𝐸 is closest to the 50% target prescribed by L𝐺 𝐴 , while the average accuracy remains comparable to the original model. This suggests that 𝛼=0.1 achieves the best effect of enforcing the intended zero expert utilization. Based on these findings, 𝛼 is set to 0.1 in all subsequent experiments. 11
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Table 4 | ZEDASFT performance on Qwen3-30B-A3B with different 𝛼. 𝛼
0.001 0.01 0.1 1.0
Avg
Avg
Acc
𝑟𝐶𝐸
AIME 24
AIME 25
AIME 26
Math GSM8k
MATH-500
LCB v5
LCB v6
Code HumanEval+
MBPP+
IFBench
IF IFEval
74.3 73.6 73.3 73.3
31.8 44.6 51.5 50.9
79.2 78.3 78.1 77.4
68.5 67.8 66.2 67.3
73.1 71.7 71.3 69.8
95.2 95.2 94.8 94.9
94.8 94.4 94.4 94.8
59.7 59.0 58.2 59.0
55.6 52.5 52.8 54.4
88.3 87.1 86.8 87.0
79.0 78.4 78.6 78.7
38.0 41.7 39.7 39.7
86.3 84.1 85.2 83.6
4.3.3. Effect of Training Stages The full ZEDA pipeline employs a two-stage self-distillation strategy. To assess whether both stages are necessary, three variants are compared: (1) SFT only, (2) OPD only, and (3) the full SFT→OPD pipeline. To ensure a fair comparison, the SFT-only and OPD-only variants are trained with an increased number of steps so that their total computational cost matches or exceeds that of the full pipeline. As shown in Table 5, the full SFT→OPD pipeline consistently outperforms both single-stage alternatives, with OPD alone performing the worst. This may be because SFT first establishes stable zero-expert routing patterns, without which OPD must simultaneously learn routing decisions and generate coherent responses, compounding the adaptation difficulty. Once SFT has stabilized the router, OPD can focus on closing the remaining distribution gap under on-policy rollouts, yielding further gains that neither stage achieves in isolation. Table 5 | Performance of ZEDA with different self-distillation strategies on Qwen3-30B-A3B.
SFT OPD SFT → OPD
Math
Code
IF
Avg Acc
Avg 𝑟𝑍𝐸
AIME 24
AIME 25
AIME 26
GSM8k
MATH-500
LCB v5
LCB v6
HumanEval+
MBPP+
IFBench
IFEval
73.6 72.9 74.2
51.5 50.1 51.2
78.5 78.4 79.0
69.3 67.2 69.1
69.8 72.3 72.5
95.3 94.9 95.5
94.8 94.4 95.2
58.1 57.0 58.2
52.1 51.5 53.2
88.6 87.3 88.5
79.0 79.6 78.2
40.3 36.3 42.3
83.5 83.2 84.3
4.3.4. Impact of Router Probability Renormalization Í As defined in Eq. 1, the dynamic MoE output after zero-expert injection is ˜𝑦 ( ℎ) = 𝑖 ∈ S̃ ( ℎ )∩E ˜𝑔𝑖 ( ℎ) 𝐸 𝑖 ( ℎ). In this formulation, the routing weights ˜𝑔𝑖 ( ℎ) of the remaining normal experts are not renormalized after zero experts are removed from the top- 𝐾 selection. An alternative is to redistribute the router probability among the active normal experts through renormalization. Concretely, the renormalized output becomes ∑︁ ˜𝑔𝑖 ( ℎ) Í ˜𝑦renorm ( ℎ) = 𝐸 𝑖 ( ℎ) , (7) 𝑔 𝑗 ( ℎ) 𝑗 ∈ S̃ ( ℎ )∩E ˜ 𝑖 ∈ S̃ ( ℎ )∩E
where the routing weights are rescaled to sum to one over the active normal experts. To evaluate this design choice, SFT is conducted on Qwen3-30B-A3B with and without renormalization under identical hyperparameters. As reported in Table 6, renormalization leads to a consistent accuracy drop compared with the default formulation. A likely reason is that in the original model, the sum of routing weights over the top- 𝐾 experts is calibrated during pre-training to produce outputs at a Table 6 | Performance of ZEDASFT with and without renormalization on Qwen3-30B-A3B.
w/ Renorm w/o Renorm
Math
Code
IF
Avg Acc
Avg 𝑟𝑍𝐸
AIME 24
AIME 25
AIME 26
GSM8k
MATH-500
LCB v5
LCB v6
HumanEval+
MBPP+
IFBench
IFEval
71.6 73.3
51.0 51.5
76.6 78.1
64.8 66.2
67.9 71.3
94.8 94.8
94.2 94.4
55.0 58.2
48.6 52.8
87.4 86.8
77.7 78.6
38.3 39.7
82.1 85.2
12
Post-Trained MoE Can Skip Half Experts via Self-Distillation
certain magnitude. And renormalization artificially amplifies the routing weights of the active normal experts, inflating the effective scale of the MoE residual branch. 4.4. Out-of-Distribution Generalization To evaluate whether zero-expert adaptation preserves capability beyond the in-distribution evaluation suite, we further test all methods on two out-of-distribution (OOD) benchmarks, MMLU-Redux [Gema et al., 2025] and GPQA-Diamond [Rein et al., 2023]. These benchmarks primarily assess knowledgeintensive question answering and scientific reasoning, which are out of distribution with respect to the math, code, and instruction-following domains represented in the self-distillation training data. Unless otherwise specified, the evaluation setup follows Section 3.1. Deviations are limited to a maximum generation length of 32k tokens, avg@8 for GPQA-Diamond, and avg@1 for MMLU-Redux. Table 7 shows that ZEDA consistently preserves competitive OOD accuracy while maintaining high zero-expert utilization (47.2% and 50.0% average 𝑟𝑍𝐸 , respectively) on both Qwen3-30B-A3B and GLM-4.7-Flash, indicating a favorable quality-efficiency trade-off under distribution shift. These results further demonstrate the strong out-of-distribution generalization capability of ZEDA. Table 7 | OOD generalization results for Qwen3-30B-A3B (left) and GLM-4.7-Flash (right) on MMLURedux (MMLU) and GPQA-Diamond (GPQA). Method Qwen3-30B-A3B ZEDA
Avg Acc
Avg 𝑟𝑍𝐸
MMLU
GPQA
Method
76.7 76.2
0.0 47.2
90.1 89.2
63.3 63.2
GLM-4.7-Flash ZEDA
Avg Acc
Avg 𝑟𝑍𝐸
MMLU
GPQA
76.1 72.9
0.0 50.0
89.8 89.0
62.4 56.8
5. Related Work 5.1. Dynamic Expert Activation in Mixture-of-Experts LLMs Mixture-of-Experts (MoE) has emerged as an effective architecture for scaling large language models by increasing model capacity while keeping bounded per-token computation [Lepikhin et al., 2020; Fedus et al., 2022; Du et al., 2022; Shazeer et al., 2017]. Subsequent studies [Dai et al., 2024; Jiang et al., 2024; Zoph et al., 2022] further improved sparse expert training and specialization, making MoE a practical design for large-scale language modeling. Nevertheless, in standard MoE architectures, routing is typically constrained by a fixed top-𝑘 policy, meaning that while expert activation is input-dependent, the computation budget remains largely static across tokens. To address this limitation, prior work has mainly proceeded along two directions. One line of work improves efficiency by reducing expert redundancy or activated computation, and can be further categorized into experts pruning [Lu et al., 2024; Liu et al., 2024], merging [Li et al., 2023; Chen et al., 2024] and compression [Li et al., 2023; Chen et al., 2025; Zhang et al., 2025; Hao et al., 2026]. The other direction replaces the static top-𝑘 routing policy with dynamic expert activation, enabling token-level input-dependent allocation of computation budgets [Jin et al., 2024; Guo et al., 2024; Zeng et al., 2024; Lu et al., 2024; Huang et al., 2024; Zhou et al., 2022; Yue et al., 2024; Sun et al., 2026]. Early work relaxed the fixed-cardinality assumption by allowing the number of activated experts to vary across tokens, either through expert-selected token assignment [Zhou et al., 2022] or by allocating more experts to harder inputs [Huang et al., 2024]. More recent work adapts these methods to modern autoregressive MoE language models: AdaMoE [Zeng et al., 2024] introduces null experts so that the number of real activated experts can vary with minimal changes to standard routing, Ada-K Routing [Yue et al., 2024] explicitly learns a token-dependent 𝑘 for expert routing,
13
Post-Trained MoE Can Skip Half Experts via Self-Distillation
DynMoE [Guo et al., 2024] jointly auto-tunes both the total number of experts and the per-token activation budget, and Expert Threshold Routing [Sun et al., 2026] replaces fixed top-𝑘 selection with threshold-based activation to obtain causal variable-size expert sets with improved load balancing, MoE++ [Jin et al., 2024] extends dynamic routing into dynamic computation-path selection by introducing zero-computation experts, which allow some tokens to bypass expensive FFN computation. Dynamic activation can also be introduced from a deployment perspective via inference-time expert skipping [Lu et al., 2024], where selected experts are conditionally bypassed at inference time without fundamentally changing the underlying router. In contrast to these approaches, we study a lower-cost form of dynamic expert activation that begins at the post-training stage, instead of relying on expensive re-pretraining or substantial router redesign. Our method operates entirely in the post-training regime and follows the zero-computation expert paradigm of MoE++ [Jin et al., 2024], which has also been validated at industrial scale in Meituan’s LongCat-Flash [Team et al., 2025]. This design avoids substantive architectural modifications to the underlying MoE model, making it particularly appealing for practical adaptation and deployment. 5.2. Self-Distillation Knowledge distillation (KD) was originally introduced as a teacher–student framework in which a student network learns from the softened output distribution of a stronger teacher network [Hinton et al., 2015]. This paradigm had already been broadly extended to language modeling, from sequence-level distillation in neural text generation [Kim and Rush, 2016], supervised distillation in autoregressive language models [Sanh et al., 2019] to more recent rationale-augmented distillation with large language models [Hsieh et al., 2023]. More recently, on-policy distillation methods for language models argued that standard distillation is inherently off-policy, since the student is trained on teacher-generated trajectories but tested on its own generations. To reduce this mismatch, methods such as MiniLLM [Gu et al., 2023] and GKD [Agarwal et al., 2024] apply teacher supervision on student-sampled sequences, a perspective that has also been highlighted in recent practitioner discussions of language model post-training [Lu and Lab, 2025]. In parallel, self-distillation has been shown to improve performance even without an external teacher [Furlanello et al., 2018; Zhang et al., 2019]. More recent work has further explored on-policy self-distillation and demonstrated its potential in scenarios such as reasoning and continual learning [Zhao et al., 2026; Shenfeld et al., 2026; Hübotter et al., 2026]. Beyond the capability improvement and task-specific adaptation, recent work has also examined self-distillation in the context of architecture adaptation towards higher computational efficiency. RAD [Hoshino et al., 2025] and HALO [Chen et al., 2026] utilize self-distillation as a principled mechanism to transform standard full-attention layers into computationally efficient alternatives, thereby achieving substantial gains in inference efficiency while maintaining model performance. LaDiMo [Kim et al., 2024] employs layer-wise distillation to transform dense models into sparse MoE architectures, facilitating efficient sparse architecture adaptation. Nevertheless, existing efforts have primarily focused on static architecture conversion, with limited attention to using self-distillation for efficient dynamic MoE architectures. In particular, the introduction of on-policy self-distillation to reduce redundant expert activation in MoE models remains underexplored, representing a critical gap in the current literature.
6. Conclusion This study presents ZEDA, a lightweight and effective framework for migrating post-trained static MoE models to dynamic ones through zero-expert injection and two-stage self-distillation. With the 14
Post-Trained MoE Can Skip Half Experts via Self-Distillation
group auxiliary loss, ZEDA regulates computation allocation while preserving the delicate routing distributions of the original MoE. Empirical evaluations across multiple architectures and benchmarks demonstrate that ZEDA eliminates over half the expert computation and provides significant inference speedups with negligible impact on model performance. These findings validate that post-trained MoE models can be adapted to efficient dynamic ones via self-distillation, offering a practical solution for enhancing the deployment efficiency of large-scale MoE systems across diverse domains.
References Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International conference on machine learning, pages 5547–5569. PMLR, 2022. Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixtureof-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, 2024. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. Moe++: Accelerating mixture-of-experts methods with zero-computation experts. arXiv preprint arXiv:2410.07348, 2024. Meituan LongCat Team, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-flash technical report. arXiv preprint arXiv:2509.01322, 2025. Haoyuan Wu, Haoxing Chen, Xiaodong Chen, Zhanchao Zhou, Tieyuan Chen, Yihong Zhuang, Guoshan Lu, Zenan Huang, Junbo Zhao, Lin Liu, et al. Grove moe: Towards efficient and superior moe llms with adjugate experts. arXiv preprint arXiv:2508.07785, 2025. Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin. Dynamic mixture of experts: An auto-tuning approach for efficient transformer models. arXiv preprint arXiv:2405.14297, 2024. Marmik Chaudhari, Idhant Gulati, Nishkal Hundia, Pranav Karra, and Shivam Raval. Moe lens–an expert is all you need. arXiv preprint arXiv:2603.05806, 2026. Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. Adamoe: Token-adaptive routing with null experts for mixture-of-experts language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6223–6235, 2024.
15
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6159–6172, 2024. Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder task needs more experts: Dynamic routing in moe models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12883–12895, 2024. Qwen Team. Qwen3.5: Accelerating productivity with native multimodal agents, February 2026. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– 27744, 2022. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The twelfth international conference on learning representations, 2024. Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471, 2025. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The twelfth international conference on learning representations, 2023.
16
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems, 36:21558–21572, 2023. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following. arXiv preprint arXiv:2507.02833, 2025. Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotron 1.1: Advancing math and code reasoning through sft and rl synergy. arXiv preprint arXiv:2506.13284, 2025. Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949, 2025. Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583, 2024. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137, 2026. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939, 2025. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5069–5096, 2025. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.
17
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022. Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs. arXiv preprint arXiv:2407.00945, 2024. Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. arXiv preprint arXiv:2310.01334, 2023. I Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, Chun-Yi Lee, et al. Retrainingfree merging of sparse moe via hierarchical clustering. arXiv preprint arXiv:2410.08589, 2024. Yuanteng Chen, Yuantian Shao, Peisong Wang, and Jian Cheng. Eac-moe: Expert-selection aware compressor for mixture-of-experts large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12942–12963, 2025. Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts. In Findings of the Association for Computational Linguistics: ACL 2025, pages 86–102, 2025. Jiawei Hao, Zhiwei Hao, Jianyuan Guo, Li Shen, Yong Luo, Han Hu, and Dan Zeng. Lightmoe: Reducing mixture-of-experts redundancy through expert replacing. arXiv preprint arXiv:2603.12645, 2026. Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114, 2022. Tongtian Yue, Longteng Guo, Jie Cheng, Xuange Gao, Hua Huang, and Jing Liu. Ada-k routing: Boosting the efficiency of moe-based llms. In The Thirteenth International Conference on Learning Representations, 2024. Hanchi Sun, Yixin Liu, Yonghui Wu, and Lichao Sun. Expert threshold routing for autoregressive language modeling with dynamic computation allocation and load balancing. arXiv preprint arXiv:2603.11535, 2026. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327, 2016. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8003–8017, 2023. 18
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference on machine learning, pages 1607–1616. PMLR, 2018. Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3713–3722, 2019. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026. Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026. Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026. Yuichiro Hoshino, Hideyuki Tachibana, Muneyoshi Inahara, and Hiroto Takegawa. Rad: Redundancyaware distillation for hybrid models via self-speculative decoding. arXiv preprint arXiv:2505.22135, 2025. Yingfa Chen, Zhen Leng Thai, Zihan Zhou, Zhu Zhang, Xingyu Shen, Shuo Wang, Chaojun Xiao, Xu Han, and Zhiyuan Liu. Hybrid linear attention done right: Efficient distillation and effective architectures for extremely long contexts. arXiv preprint arXiv:2601.22156, 2026. Sungyoon Kim, Youngjun Kim, Kihyo Moon, and Minsung Jang. Ladimo: Layer-wise distillation inspired moefier. arXiv preprint arXiv:2408.04278, 2024. Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Jiashi Li, and Liang Zhao. Deepep: an efficient expert-parallel communication library. https://github.com/deepseek-ai/DeepEP, 2025. Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, 2023.
19
Post-Trained MoE Can Skip Half Experts via Self-Distillation
A. Limitations and Future Work Lack of Larger MoE Deployments. Although we demonstrate consistent improvements on 30Bscale MoE models, we do not yet evaluate substantially larger-scale MoE models due to computational resource constraints. Lack of Long-Horizon Agentic Tasks. Our experimental evaluation is confined to standard posttraining tasks and does not cover agentic workloads. A contributing factor is the limited availability of mature open-source agentic infrastructure and training recipes. Speedup Decay at Long Sequence Lengths. Beyond the 8𝑘 results reported in the main text, we also evaluate sequence lengths of {2𝑘, 4𝑘, 6𝑘, 8𝑘 }, as shown in Table 8. The speedup gradually diminishes as sequence length increases. Nevertheless, even at 8𝑘, a commonly used long-context setting, ZEDA still achieves approximately 20% speedup, demonstrating its practical usability. Furthermore, ZEDA exhibits greater potential for advanced communication frameworks like DeepEP [Zhao et al., 2025], which we aim to integrate in future work. Table 8 | Inference efficiency comparison between the original model and ZEDA from 2048 to 8192 sequence length. Speedup is defined relative to the original model and throughput (103 token·s−1 ) is shown in the format ZEDA/original. Model
Qwen3-30B-A3B GLM-4.7-Flash
Prefill
Metric Speedup Throughput Speedup Throughput
Decode
2048
4096
6144
8192
2048
4096
6144
8192
1.21x 71.43/58.86 1.36x 47.78/35.11
1.20x 62.53/51.97 1.31x 39.42/30.20
1.19x 58.09/48.75 1.27x 34.37/27.06
1.18x 51.63/43.92 1.26x 32.36/25.73
1.25x 2.49/1.99 1.22x 1.91/1.57
1.24x 2.39/1.93 1.20x 1.88/1.56
1.21x 2.22/1.82 1.19x 1.80/1.51
1.19x 2.07/1.74 1.19x 1.76/1.48
B. Zero Experts versus Copy Experts An alternative to the zero expert is the copy expert, whose output equals its input, carrying negligible computational cost. In this section, we establish that zero expert is the preferable design for adapting a post-trained MoE model to a dynamic architecture. Compared with zero experts, copy experts introduce stronger perturbations to the original model. Assuming the same router parameterization, the output of a dynamic MoE module with zero experts and that with copy experts is ∑︁ ∑︁ ∑︁ cp norm ˜𝑦zero = ˜𝑔𝑖 ( ℎ) 𝐸 𝑖 ( ℎ) , ˜𝑦copy = ˜𝑦copy + ˜𝑦copy = ˜𝑔𝑖 ( ℎ) 𝐸 𝑖 ( ℎ) + ˜𝑔 𝑗 ( ℎ) ℎ, (8) 𝑖 ∈ S̃∩E
𝑖 ∈ S̃∩E
𝑗 ∈ S̃∩Z
cp
norm and ˜ where ˜𝑦copy 𝑦copy are the normal expert component and copy component of the copy-expert model, respectively. Zero experts implement true expert omission, whereas copy experts incur an Í additional term 𝑗 ∈ S̃ ( ℎ )∩Z ˜𝑔 𝑗 ( ℎ) ℎ rather than a no-op.
To study the effect of the zero-compute-expert type, we compare two post-training adaptation variants on Qwen3-30B-A3B that are identical in training data, SFT recipe, and routing regularization, differing only in whether the inserted zero-compute experts are instantiated as copy experts or zero experts. For both variants, the routing regularizer is the group auxiliary loss L𝐺 𝐴 in Eq. (5) with coefficient 𝛼=0.1 and group weight 𝑤=2.0, matching the setting used in Section 3.1. We report the average accuracy, 20
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Table 9 | Comparison of two zero-compute-expert types, copy expert and zero expert, on mathematical reasoning benchmarks. Math
Method
Avg Acc
Avg 𝑟𝑍𝐶𝐸
AIME 24
AIME 25
AIME 26
GSM8K
MATH-500
Qwen3-30B-A3B Copy Expert + L𝐺 𝐴 Zero Expert + L𝐺 𝐴
82.8 20.7 81.0
0.0 53.2 52.7
80.9 1.0 78.1
71.0 2.9 66.2
72.3 0.8 71.3
95.4 58.8 94.8
94.4 40.0 94.4
Direction Mismatch 1.0
80
Cosine Similarity
Relative Norm Diff (%)
Scale Mismatch
60 40 20
0.8
y vs y~zero y vs y~copy y vs y~norm copy
0.6 0.4
y vs y~cp copy
0.2 0.0
0 0
10
20
30
Layer Index
40
0
10
20
30
40
Layer Index
Figure 7 | Layer-wise comparison between the original MoE output 𝑦 and four variants: ˜𝑦zero , ˜𝑦copy , cp norm and ˜ ˜𝑦copy 𝑦copy , including relative absolute L2-norm difference (left) and cosine similarity (right). the average activation ratio of the inserted zero-compute experts 𝑟𝑍𝐶𝐸 , and the performance on five mathematical reasoning benchmarks. Table 9 shows that the copy-expert type performs substantially worse than the zero-expert type despite nearly identical activation ratios (53.2% vs. 52.7%). While the zero-expert type largely preserves the mathematical reasoning ability of the original model, the copy-expert type leads to severe performance degradation across all five benchmarks. The gap is particularly pronounced on the more challenging AIME tasks, where the copy-expert type achieves only 1.0, 2.9, and 0.8 on AIME 24, AIME 25, and AIME 26, respectively. These empirical results suggest that, for post-training adaptation of a post-trained MoE model toward dynamic computation, the zero-expert type is substantially more suitable than the copy-expert type. We extract the hidden-state outputs of the MoE blocks and conduct the following experiments to further analyze why copy experts are harmful from two complementary perspectives: • Scale Mismatch. The left panel of Figure 7 shows that the full copy expert output has a much larger norm mismatch with the original output than the zero expert output does. Decomposition further shows that the normal expert component is less mismatched than the copy component, but still consistently more mismatched than the zero-expert output. This indicates that the copy component is the main source of scale mismatch, and also pulls the normal expert component away from the original scale. • Direction Mismatch. The right panel of Figure 7 shows the same trend in direction space. The zero expert output remains well aligned with the original output, whereas the full copy expert output shows a clear directional mismatch. The copy component stays strongly misaligned across layers, while the mismatch of the normal expert component grows from shallow to deep layers. This suggests that the copy component is the primary cause of directional mismatch and progressively drags the normal expert component away from the original direction as depth increases.
21
Post-Trained MoE Can Skip Half Experts via Self-Distillation
C. Auxiliary-Loss Comparison Experimental Setting. To isolate the effect of the routing regularizer in Section 2.2, we compare three zero-expert adaptation variants on Qwen3-30B-A3B that are identical in architecture and SFT adaptation recipe, differing only in the balancing objective: one uses the standard expert-level auxiliary loss L 𝐴 in Eq. (4), and the other two use the proposed group auxiliary loss L𝐺 𝐴 in Eq. (5). The experimental setting is otherwise the same as the SFT implementation details in Section 3.1. In particular, the auxiliary-loss coefficient 𝛼 is set to 0.1 for all variants. For the group auxiliary loss, the relative weight of the zero-expert group 𝑤 is set to 1.0 and 2.0, respectively. We report the results on five mathematical reasoning benchmarks. Table 10 | Comparison of auxiliary loss and group auxiliary loss under different 𝑤 values on mathematical reasoning benchmarks. Math
Method
Avg Acc
Avg 𝑟𝑍𝐸
AIME 24
AIME 25
AIME 26
GSM8K
MATH-500
Qwen3-30B-A3B Zero Expert + L 𝐴 Zero Expert + L𝐺 𝐴 ( 𝑤=1.0) Zero Expert + L𝐺 𝐴 ( 𝑤=2.0)
82.8 59.5 82.2 81.0
0.0 34.4 35.0 52.7
80.9 39.6 79.4 78.1
71.0 32.3 69.7 66.2
72.3 47.1 71.9 71.3
95.4 93.5 95.2 94.8
94.4 85.0 94.8 94.4
Conclusion. Compared with the original auxiliary loss, both group auxiliary loss variants deliver significant improvements, showing that the benefit of group-level balancing is robust across different 𝑤 settings. In particular, replacing L 𝐴 with L𝐺 𝐴 at 𝑤=1.0 improves average accuracy from 59.5 to 82.2, nearly recovering the original Qwen3-30B-A3B performance of 82.8, while maintaining a similar zero-expert activation ratio. Increasing the group weight to 𝑤=2.0 raises 𝑟𝑍𝐸 further to 52.7 while preserving a strong average accuracy of 81.0, indicating a clear quality-efficiency trade-off. These results are consistent with the design motivation in Section 2.2. A post-trained MoE model exhibits non-uniform, input-dependent routing patterns over normal experts, and enforcing expertlevel uniformity disrupts these learned routing distributions, which can severely degrade model performance. By contrast, the group auxiliary loss regulates only the competition between the normalexpert group and the zero-expert group, thereby preserving the relative routing structure among normal experts while enabling controllable zero-expert utilization through 𝑤.
D. Theoretical FLOPs Analysis In this section, we analyze the theoretical FLOPs of both the prefill and decode stages for the original MoE model and the model adapted by ZEDA. We focus on dominant matrix multiplication terms and omit lower-order operations such as normalization, residual connections, activation functions, routing top-𝑘 selection, and softmax overhead. For a matrix multiplication [ 𝑚, 𝑛] × [ 𝑛, 𝑝], we count its cost as 2𝑚𝑛𝑝 FLOPs. All expressions are reported per Transformer layer. Multiplying by the number of layers does not change the ZEDA/original FLOP ratios when all layers share the same configuration. The notation used throughout this section is summarized in Table 11. D.1. Shared MoE Cost Decomposition The MoE FFN and router costs have the same form in both stages; the only difference is the number of tokens processed in the current forward pass. Let 𝑛 denote that token count. For the original 22
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Table 11 | Notation used in the theoretical FLOP analysis. Symbol
Description
𝑙 𝐻 𝐻attn 𝑔kv 𝐻𝑒 𝑁 𝑁𝑍 𝐾 𝑟𝑍𝐸
Sequence length in the stage under analysis Hidden size Attention intermediate size Ratio of KV heads to query heads in GQA Expert intermediate size Number of normal experts Number of zero-computation experts Number of activated experts per token Fraction of activated zero experts
MoE model, each token activates 𝐾 normal experts, and each expert contains up, gate, and down projections. Hence, the expert FFN and router costs are 𝐹MoE,orig ( 𝑛) = 6 𝐾𝑛𝐻 𝐻𝑒 + 2 𝑁𝑛𝐻.
(9)
For the ZEDA model, only an (1 − 𝑟𝑍𝐸 ) fraction of the activated experts perform FFN computation, while the router scores both normal and zero-computation experts. Therefore, 𝐹MoE,ZEDA ( 𝑛) = 6(1 − 𝑟𝑍𝐸 ) 𝐾𝑛𝐻 𝐻𝑒 + 2( 𝑁 + 𝑁𝑍 ) 𝑛𝐻.
(10)
In the prefill stage, 𝑛 = 𝑙. In the decode stage with KV cache, each forward pass processes one newly generated token, and the total decode cost is obtained by summing over all decode steps. D.2. Prefill Stage For grouped-query attention (GQA) [Ainslie et al., 2023], the prefill stage processes all 𝑙 tokens in parallel. The attention cost therefore consists of five parts: the query projection, the key/value projections, the query-key score computation over all token pairs, the attention-value aggregation, and the output projection. These terms sum to pre
𝐹attn = 4𝑙 2 𝐻attn + 4(1 + 𝑔kv ) 𝑙𝐻 𝐻attn .
(11)
Substituting 𝑛 = 𝑙 into Equations (9) and (10), and then adding the prefill attention term in Equation (11), yields the total prefill FLOPs of the original model and the ZEDA model. In both expressions, the first two terms come from attention, the third term is the MoE FFN cost, and the last term is the router cost: pre 𝐹orig = 4𝑙 2 𝐻attn + 4(1 + 𝑔kv ) 𝑙𝐻 𝐻attn + 6 𝐾𝑙𝐻 𝐻𝑒 + 2 𝑁 𝑙𝐻. (12) For the ZEDA model, the attention term remains unchanged, while the expert FFN cost is reduced by the factor (1 − 𝑟𝑍𝐸 ) and the router cost increases because the router now scores 𝑁 + 𝑁𝑍 experts: pre
𝐹ZEDA = 4𝑙 2 𝐻attn + 4(1 + 𝑔kv ) 𝑙𝐻 𝐻attn + 6(1 − 𝑟𝑍𝐸 ) 𝐾𝑙𝐻 𝐻𝑒 + 2( 𝑁 + 𝑁𝑍 ) 𝑙𝐻.
(13)
The corresponding FLOP ratio, obtained from Equations (12) and (13), is pre
𝐹ZEDA pre 𝐹orig
=
2𝑙𝐻attn + 2(1 + 𝑔kv ) 𝐻 𝐻attn + 3(1 − 𝑟𝑍𝐸 ) 𝐾 𝐻 𝐻𝑒 + ( 𝑁 + 𝑁𝑍 ) 𝐻 2𝑙𝐻attn + 2(1 + 𝑔kv ) 𝐻 𝐻attn + 3 𝐾 𝐻 𝐻𝑒 + 𝑁 𝐻
(14)
23
Post-Trained MoE Can Skip Half Experts via Self-Distillation
D.3. Decode Stage In the decode stage, we assume standard KV caching and analyze a decode-only process that generates 𝑙 tokens. As in the prefill case, the attention cost consists of query projection, key/value projections, score computation, attention-value aggregation, and output projection. The difference is that at decode step 𝑡 , only one new token is processed, and the score computation and attention-value aggregation each involve 𝑡 − 1 cached tokens rather than all 𝑙 tokens. Summing these per-step costs over all 𝑙 decode steps gives dec 𝐹attn =
𝑙 ∑︁
[4( 𝑡 − 1) 𝐻attn + 4(1 + 𝑔kv ) 𝐻 𝐻attn ]
(15)
𝑡 =1
= 2𝑙 ( 𝑙 − 1) 𝐻attn + 4(1 + 𝑔kv ) 𝑙𝐻 𝐻attn . Substituting the per-token MoE costs from Equations (9) and (10) across all 𝑙 decode steps, and adding the accumulated attention cost in Equation (15), gives the total decode FLOPs. As in the prefill case, the first two terms correspond to attention, the third term is the MoE FFN cost, and the last term is the router cost: dec 𝐹orig = 2𝑙 ( 𝑙 − 1) 𝐻attn + 4(1 + 𝑔kv ) 𝑙𝐻 𝐻attn + 6 𝐾𝑙𝐻 𝐻𝑒 + 2 𝑁 𝑙𝐻.
(16)
For the ZEDA model, the decode attention term is again identical to that of the original model, whereas the MoE branch differs in exactly the same way as in prefill: only an (1 − 𝑟𝑍𝐸 ) fraction of activated experts incur FFN cost, and the router expands from 𝑁 to 𝑁 + 𝑁𝑍 outputs: dec 𝐹ZEDA = 2𝑙 ( 𝑙 − 1) 𝐻attn + 4(1 + 𝑔kv ) 𝑙𝐻 𝐻attn + 6(1 − 𝑟𝑍𝐸 ) 𝐾𝑙𝐻 𝐻𝑒 + 2( 𝑁 + 𝑁𝑍 ) 𝑙𝐻.
(17)
Therefore, the decode-stage FLOP ratio, obtained from Equations (16) and (17), is dec 𝐹ZEDA dec 𝐹orig
=
( 𝑙 − 1) 𝐻attn + 2(1 + 𝑔kv ) 𝐻 𝐻attn + 3(1 − 𝑟𝑍𝐸 ) 𝐾 𝐻 𝐻𝑒 + ( 𝑁 + 𝑁𝑍 ) 𝐻 ( 𝑙 − 1) 𝐻attn + 2(1 + 𝑔kv ) 𝐻 𝐻attn + 3 𝐾 𝐻 𝐻𝑒 + 𝑁 𝐻
(18)
D.4. Numerical Results We instantiate the prefill and decode ratios in Equations (14) and (18) using the Qwen3-30B-A3B configuration [Yang et al., 2025] in Table 12. Table 12 | Architectural parameters of Qwen3-30B-A3B used in the FLOP analysis. Symbol
𝐻
𝐻attn
𝑔kv
𝐻𝑒
𝑁
𝑁𝑍
𝐾
Value
2048
4096
1/8
768
128
64
8
To facilitate direct comparison with empirical measurements, we convert the FLOP ratios in Equations (14) and (18) into theoretical speedups by taking their reciprocals. Table 13 reports the resulting prefill and decode speedups for 𝑙 ∈ {1024, 2048, . . . , 8192} and 𝑟𝑍𝐸 = 0.5, together with the corresponding empirically measured results. Two trends are apparent from Table 13.
24
Post-Trained MoE Can Skip Half Experts via Self-Distillation
Table 13 | Comparison between theoretical speedups derived from the FLOP analysis and measured empirical speedups on Qwen3-30B-A3B across different sequence lengths. Length 1024 2048 3072 4096 5120 6144 7168 8192
Prefill Speedup
Decode Speedup
Theoretical
Empirical
Theoretical
Empirical
1.403x 1.341x 1.296x 1.261x 1.234x 1.212x 1.194x 1.178x
1.141x 1.214x 1.203x 1.203x 1.202x 1.192x 1.176x 1.175x
1.443x 1.403x 1.370x 1.341x 1.317x 1.296x 1.278x 1.261x
1.233x 1.252x 1.238x 1.236x 1.228x 1.215x 1.210x 1.185x
(i) The speedup decays with sequence length in both stages. Under this model, the prefill theoretical speedup drops from 1.403× at 𝑙 = 1024 to 1.178× at 𝑙 = 8192, while the decode theoretical speedup drops from 1.443× to 1.261× over the same range. The empirical results broadly match these theoretical predictions: in both stages, the measured speedups exhibit the same monotonic decay with sequence length, while remaining consistently below the theoretical values due to implementation overheads and computational costs not captured by the FLOP analysis. (ii) The decode speedup is consistently higher than the prefill speedup at the same length. For a fixed 𝑙, the unchanged attention cost in decode is smaller than that in prefill, so the reduction in MoE computation accounts for a larger fraction of the total FLOPs and translates into a larger overall speedup. This ordering is consistently reflected in the empirical results across all evaluated lengths.
25