ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs Zukang Xu∗ , Zhixiong Zhao∗ , Xing Hu, Jiangyong Yu, Houji Wen, Jun Li, Zhe Jiang, Dawei Yang† [email protected]
Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs), yet fixed top-k routing executes the same number of expert slots for every token and can retain substantial redundant computation. Existing expert-skipping methods often rely on router confidence, calibration data, or additional training,and therefore cannot reliably estimate the actual contribution of routed experts. To this end, we propose ACE, a training-free, calibrationfree, and checkpoint-preserving framework for token-adaptive expert skipping in MoE-based LLMs. ACE contains two complementary components: 1) Global Spectral Proxy (GSP), which estimates global transformation capacity from the coupled gate, up, and down projections together with RMSNorm scaling; and 2) Router-Conditioned Refinement (RCR),which constructs expert-specific direction prototypes from centered router weights and evaluates expert responses along routingpreferred directions. During inference, ACE combines both estimates with runtime router gates and skips an expert slot only when both views identify it as low-contribution, while always retaining the top-1 expert. All expert statistics are computed offline, leaving only table lookups and lightweight scalar operations online. Extensive experiments across three MoE-based LLMs and eight benchmarks demonstrate that ACE consistently outperforms existing static and dynamic baselines, with increasingly pronounced advantages under aggressive expert skipping. For instance, at a 50% skipping ratio on Qwen3.6-35B-A3B, ACE reduces WikiText-2 perplexity by 7.96% and improves average downstream accuracy by 4.15 percentage points over the strongest competing method. Optimized expert dispatch further yields up to 2.25× prefill and 1.41× decoding speedups in our measurements. The code is available at https://github.com/xzktx003/moe-prune.
1
Introduction
Mixture-of-Experts (MoE) has become a key approach for scaling large language models (LLMs) (Shazeer et al. 2017). Unlike dense Transformers that execute the full feed-forward network at every layer, MoE models replace each FFN sublayer with multiple experts and route each token to only a small subset, thereby expanding model capacity while controlling per-token computation. Recent models further demonstrate this advantage: Qwen3.6-35B-A3B (Qwen Team ∗ †
These authors contributed equally. Corresponding author.
Aggressive skipping regime
𝟖𝟎
𝟕𝟎
Avg. Accuracy (%)
arXiv:2609.05228v1 [cs.AI] 4 Sep 2026
Abstract
𝟔𝟎
𝟓𝟎
BF16 (0% skipping) SERE AIMER NAEE DiEP MoDES Ours
𝟒𝟎
𝟑𝟎 𝟏𝟎
𝟐𝟎
𝟑𝟎
𝟒𝟎
𝟓𝟎
𝟔𝟎
Expert skipping Ratio (%)
Figure 1: Accuracy–efficiency trade-off on Qwen3-30B-A3BInstruct-2507 under different expert skipping ratios. ACE consistently outperforms all baselines across the full range. 2026) activates only about 3B of its 35B parameters, while DeepSeek-V4-Pro and DeepSeek-V4-Flash (DeepSeek-AI et al. 2026) adopt total-to-active parameter configurations of 1.6T/49B and 284B/13B, respectively. However, sparse activation does not make MoE inference inherently efficient. Most MoE-based LLMs still use fixed top-k routing, activating the same number of expert slots for every token despite substantial variation in their contributions. As a result, low-contribution experts may still be fully executed, causing redundant computation and notable inference overhead, especially in resource-constrained or memory-bandwidth-limited deployments. To further improve MoE inference efficiency, existing studies mainly reduce expert redundancy through offline expert compression or online dynamic activation control. Offline pruning and merging methods (Liu et al. 2026b; Bai et al. 2025) modify the expert pool before deployment based on expert usage, activation responses, gradient/Fisher information, or search-based criteria, thereby reducing model size and resident memory. However, such static compression shares the same expert structure across all inputs and cannot adapt to token-specific demands for routed expert slots. Even after the expert pool is compressed, the remaining MoE layers typ-
ically retain fixed top-k routing and execute a preset number of experts, leaving dynamic redundancy within the routed set unaddressed. Dynamic expert skipping is more closely aligned with our goal, as it adaptively reduces executed expert slots during inference. Some methods modify or retrain the router (Huang et al. 2024), or introduce additional decision modules to learn token-wise expert budgets (Zhao et al. 2025), while post-training approaches determine skipping thresholds from router-score imbalance, calibration-based layer sensitivity, or activation statistics (Huang et al. 2026). Although these methods avoid permanently removing experts, their decisions still rely largely on router confidence or calibration-dependent signals. The former reflects relative routing preference rather than the effective transformation contribution of an executed expert, whereas the latter requires additional calibration-time forward passes and may be sensitive to calibration size, domain distribution, and task shift. As shown in Figure 1, existing methods remain stable under mild skipping but degrade rapidly as the skipping ratio increases, whereas ACE maintains a superior accuracy–efficiency tradeoff in the aggressive 40%–60% regime. This suggests that the key challenge of high-ratio skipping is not merely to reduce expert executions, but to reliably estimate the relative contributions of routed expert slots for token-adaptive skipping without calibration data or checkpoint modification. To understand the key challenges of calibration-free expertslot skipping, we examine the problem from three perspectives: routing preference, expert structure, and input directionality. First, router gates only reflect the relative assignment preference among the top-k candidate experts, rather than the actual contribution of each routed expert slot to the MoE output. Experts with similar gate values may produce substantially different responses due to variations in their FFN structures, RMSNorm scaling, and output mappings; conversely, an expert with a smaller gate may still contribute non-negligibly through stronger transformation capacity. Therefore, skipping experts solely based on router scores may incorrectly equate routing preference with expert contribution. Second, estimating expert transformation capacity directly from model parameters, without calibration activations or additional training, is non-trivial. The output of a SwiGLU expert arises from the nonlinear interaction among its gate, up, and down projections, which cannot be adequately characterized by a single matrix norm, usage frequency, or global routing statistic. Finally, expert contribution is inherently direction-dependent. An expert with weak global responses may still exhibit strong behavior within the local input region favored by the router. A reliable data-free contribution estimator should therefore capture both the global structural capacity of each expert and the directional specialization encoded by the router. Motivated by these observations, we propose ACE (Adaptive Calibration-Free Expert Skipping), a training-free, calibration-free, and checkpoint-preserving framework for dynamic expert skipping in MoE-based LLMs. ACE retains the pretrained router and its original top-k candidate set, while estimating the relative contribution of each routed expert slot by combining token-specific routing preference with expert-specific transformation capacity. Specifically, the Global Spectral Proxy (GSP) extracts a direction-agnostic
structural response from the gate, up, and down projections of each SwiGLU expert together with the preceding RMSNorm scaling, thereby characterizing its global transformation capacity. Router-Conditioned Refinement (RCR) further constructs an expert-specific direction prototype from the centered router weights and measures the expert response along its preferred routing direction, compensating for directional specialization overlooked by the global proxy. During inference, ACE combines the router gates with the precomputed GSP and RCR amplification tables, and progressively skips low-contribution slots under dual-view contribution constraints and a routermass safeguard, while always preserving the top-1 expert. Since all expert-level statistics are derived offline from model weights and router geometry, ACE requires no real samples, activation caching, or threshold profiling, and introduces only table lookups and lightweight top-k scalar operations at runtime. Our key contributions can be summarized as follows: • We identify key challenges in calibration-free expert-slot skipping: mismatch between routing preference and expert contribution, structural coupling within SwiGLU experts, and direction-dependent specialization of responses. • We propose ACE, a training-free and checkpointpreserving framework that combines calibration-free GSP and RCR contribution estimates for safeguarded tokenadaptive skipping. • Extensive experiments across MoE-based LLMs and tasks show that ACE consistently outperforms static and dynamic baselines, especially under aggressive skipping.
2
Related Work
Static Expert Compression in MoE-based LLMs. Static expert compression reduces model size and resident memory by pruning, merging, or restructuring experts before deployment. Existing methods identify redundant experts using routing frequency, task statistics, or structural importance. The expert-pruning branch of NAEE (Lu et al. 2024) selects fixed subsets for task-agnostic and task-specific compression, EACMoE (Chen et al. 2025) removes rarely selected experts based on selection frequency, and AIMER (Liu et al. 2026b) develops a calibration-free criterion for task-agnostic pruning. More fine-grained methods include MoE-Pruner (Xie et al. 2024), which combines router information with input activations for one-shot intra-expert pruning, and REAP (Lasby et al. 2026), which jointly considers router gates and expert activation norms. DiEP (Bai et al. 2025) learns non-uniform layer-wise sparsity through differentiable optimization, COMPEL (Yoon and Choi 2026) integrates Fisher-based importance, layerwise outlier distributions, and weight compensation, while STUN (Lee et al. 2025) applies structured expert pruning followed by unstructured compression within retained experts. Despite reducing parameter and memory costs, these methods determine a fixed structure before inference and apply it uniformly to all inputs. The remaining MoE layers generally retain fixed top-k routing and cannot adaptively eliminate low-contribution expert slots for individual tokens. Dynamic Expert Skipping in MoE-based LLMs. Dynamic expert skipping reduces expert activations according
Global Spectral Proxy (GSP)
Original Top-k MoE Routng Token 𝑥
Top-k expert slots {𝑒 , 𝑔 } Input x
Router (Linear + Softmax)
𝑒 ×
𝑒
𝒈𝒂𝒕𝒆
𝒂𝒍,𝒆
Multiply
𝑊
𝑊
×
𝑎,
=
{(𝑒 , 𝑔 , , )}
𝒖𝒑 𝒈𝒂𝒕𝒆
𝒂𝒍,𝒆 𝒂𝒍,𝒆
𝒈𝒂𝒕𝒆
= 𝜞𝒍 𝑾𝒍,𝒆
Normalize (per layer)
Γ (RMSNorm)
𝑎,
+
+
Token 𝑥 Routed top-k slots
𝒖𝒑
Symmetric
𝑊
⨀ Element-wise
⋯ 𝑒
×
𝒖𝒑
𝑊
SiLU
𝑒
𝑎,
𝑊
Online Adaptive Inference Geometric coupling
𝒂𝒍,𝒆 = 𝜞𝒍 𝑾𝒍,𝒆
𝑊
𝑻𝒐𝒑𝒌
⋯
Up-conditioned path
Frozen SwiGLU Expert weights
𝐴
Gate-conditioned path
(𝑒)
Offline GSP table
Router-Conditioned Refinement (RCR) Original Router Weights
Centered Roter Weights 𝑤, 𝑤 𝑤,
Direction Prototype 𝑞,
𝑤, 𝑞 , ||𝑤 ,
𝑤,
Fixed-RMS shell
Frozen Expert 𝒆 𝑞,
SiLU
𝑊
𝑤, =𝑤, −
1 𝑁
𝑤,
𝑊
+ 𝑞 , = 𝑅𝑀𝑆𝑁𝑜𝑟𝑚(𝑤 , ) 𝜌
𝑦,
𝐴
𝑐 , , = max( 𝑝 , , , 𝑝 , , ) 𝑠𝑘𝑖𝑝 𝑖𝑓 𝑐 , , < 𝜏(𝑞)
(𝑒)
𝑒
(𝑒 )
𝑠,, =𝑔,,𝐴 𝑒 𝑝 , , = 𝑛𝑜𝑟𝑚(𝑠 )
𝑠,, =𝑔,,𝐴 𝑒 𝑝 , , = 𝑛𝑜𝑟𝑚(𝑠 )
Offline RSR table Gain
𝑒
𝑒
⋯
Always Keep Router Top-1 expert
𝑒
𝑎,
⨀ 𝑊
𝑅𝑀𝑆 𝑞 = 𝜌 (1 𝑖𝑛 𝐴𝐶𝐸)
RCR branch Lookup 𝐴
(𝑒 )
ACE Joint Skipping Decision Conditional Response 𝑞,
𝑊
𝑤,
GSP branch Lookup 𝐴
𝑓,
= 𝑓, 𝑞 ,
Normalize (per layer)
𝑓 , (𝑞 , )
𝑎,
=
, (
, )
Renormalize surviving gates
Aggregate surviving experts ∈
𝑔 , , 𝑓,
(𝑥 )
,
Figure 2: Overview of ACE. Global Spectral Proxy (GSP) estimates each expert’s transformation capacity from frozen SwiGLU weights, while Router-Conditioned Refinement (RCR) evaluates responses along router-induced directions. During inference, ACE combines offline scores with token-wise router gates to skip low-contribution expert slots and aggregate surviving experts. to token-, layer-, or batch-specific states. Some methods replace fixed top-k routing with variable-cardinality policies. Top-P Routing (Huang et al. 2024) determines activated expert counts from cumulative routing probabilities, Ada-K Routing (Zhao et al. 2025) learns a lightweight allocator through reinforcement learning, and DynMoE (Guo et al. 2025) combines Top-Any gating with adaptive training to vary expert counts across tokens. For pretrained MoE-based LLMs, NAEE (Lu et al. 2024) skips secondary experts using relative router scores and calibration-derived layer thresholds. Alloc-MoE (Liu et al. 2026a) distributes a global activation budget through layer sensitivity profiling and dynamic programming, followed by routing-score-based token allocation, while MoDES (Huang et al. 2026) combines globally modulated local gating with modality-specific threshold search for training-free skipping in multimodal MoE models. Batchlevel methods include SERE (Wu et al. 2026), which reroutes tokens from secondary to similar primary experts, and XShare (Vankov et al. 2026), which selects shared expert subsets from batch routing demands. Despite their effectiveness, these methods generally require routing-policy training, additional modules, calibration or sensitivity profiling, router-only importance signals, or specific batch-serving assumptions.
3
Method
Overview. To address the three challenges of calibrationfree expert skipping identified in Sec. 1, we propose ACE, a parameter-driven framework for token-adaptive expert skipping in MoE-based LLMs (Fig. 2). First, Global Spectral Proxy (GSP) models the multiplicatively coupled gate, up, and down projections of each SwiGLU expert to estimate its transformation capacity without calibration data (Sec. 3.1). Second, Router-Conditioned Refinement (RCR) constructs expert-specific directional prototypes from centered router
weights and evaluates experts along their routing-preferred directions, compensating for directional specialization missed by GSP (Sec. 3.2). Finally, ACE combines runtime router gates with offline GSP and RCR scores, skipping an expert slot only when both views identify it as low-contribution. Detailed pseudocode is provided in Appendix Sec. A.1.
3.1
Global Spectral Proxy (GSP)
In standard MoE inference, router gates typically serve as the primary indicator of expert importance, determining a fixed top-k activation pattern. However, a gate reflects only the relative routing preference of the current token among candidate experts, rather than the intrinsic nonlinear transformation capacity of the expert networks. Thus, high routing affinity does not necessarily imply a large contribution to the MoE output. Gate-only skipping may therefore conflate routing preference with marginal contribution, while overlooking differences in the parameter structures and response capacities of individual experts. To mitigate this measurement bias, ACE introduces a static transformation-capacity proxy derived exclusively from intrinsic expert parameters. Without modifying the original top-k candidate set selected by the pretrained router, this proxy enables fine-grained, adaptive expert skipping. Specifically, for the l-th MoE layer and the t-th token, let {ei }ki=1 denote the original top-k experts selected by the pretrained router, with gate values {gl,t,i }ki=1 . Within this candidate set, we define the unnormalized GSP contribution score of the i-th routed expert slot as GSP sGSP (ei ), l,t,i = gl,t,i · Al
(1)
where gl,t,i represents the token-dependent routing preference for expert ei , while AGSP (ei ) is a static amplification l proxy derived solely from intrinsic model parameters. The latter characterizes the expert’s transformation capacity for
a normalized hidden representation. By combining dynamic routing preference with static transformation capacity, GSP distinguishes whether an expert is strongly preferred by the router from whether it is structurally capable of producing a strong response. Consequently, an expert slot receives a high GSP contribution score only when it exhibits both strong routing compatibility and high intrinsic transformation capacity. To construct AGSP (e) without calibration data, we analyze l the parameterized input–output transformation of a SwiGLU expert. Since the RMSNorm preceding each MoE module maintains hidden states at a relatively stable scale, the expert input can be written as x = x̄Γl , where x̄ denotes the RMSnormalized input direction and Γl is the diagonal matrix formed by the learned RMSNorm scaling vector at layer l. Consider a SwiGLU expert with gate, up, and down projection gate up down weights Wl,e , Wl,e , and Wl,e , respectively. Ignoring bias terms, its forward transformation is gate up down fl,e (x) = SiLU x̄Γl Wl,e ⊙ x̄Γl Wl,e Wl,e . (2) Since | SiLU(z)| ≤ |z| for any scalar z, the expert output admits the upper bound is 2 ∥fl,e (x)∥2 ≤ ∥x̄∥2
gate Γl Wl,e
2
up Γl Wl,e
2
agate l,e =
gate Γl Wl,e
F
gate down · Γl Wl,e Wl,e
· F
AGSP (e) = l
up down Γl Wl,e Wl,e
, F
pGSP l,t,i = Pk
sGSP l,t,i
GSP j=1 sl,t,j + ϵ
(7)
.
(8)
Here, pGSP l,t,i measures the relative contribution of the i-th expert slot under the joint effect of token-dependent routing preference and global structural response capacity. Given a global threshold τ (q) for target skipping rate q, a slot satisfying pGSP l,t,i < τ (q) is identified as a low-contribution candidate from the GSP perspective. Because all static scores are computed before deployment, online GSP requires only expert-indexed table lookup, scalar multiplication, and normalization over the original top-k candidates. It therefore adds no high-dimensional expert computation and provides ACE with a calibration-free, parameter-intrinsic, and deploymentfriendly baseline for global expert contribution estimation.
(4)
3.2 .
F
e aGSP l,e , P Nl 1 aGSP j=1 e l,j + ϵ Nl
where Nl is the number of experts in the l-th MoE layer and ϵ ensures numerical stability. The normalized score AGSP (e) l measures the global structural response of expert e relative to the layer average: AGSP (e) > 1 indicates above-average l static amplification capacity, whereas AGSP (e) < 1 indicates l a relatively weak global response. During online inference, GSP requires neither additional expert forward passes nor activation-statistics collection. For each expert slot selected by the pretrained router, ACE retrieves the corresponding static amplification score AGSP (ei ), l precomputed once before deployment, and combines it with the current gate value. The resulting scores are normalized within the original top-k candidate set to obtain the relative GSP contribution distribution:
down Wl,e . 2
(3) This bound shows that the response capacity of an expert is jointly determined by the gate branch, up branch, and down projection through multiplicative coupling. Hence, the norm of any individual projection matrix is insufficient to characterize the transformation capacity of a SwiGLU expert. The complete derivation is provided in Appendix Sec. A.2. Motivated by this multiplicative structure, GSP constructs a branch-symmetric matrix-norm surrogate requiring neither input activations nor calibration samples. Specifically, we estimate each expert’s global response capacity through two complementary factorization paths: up aup l,e = Γl Wl,e
scaling factors can vary substantially across MoE layers, raw surrogate values are not directly comparable. We therefore normalize e aGSP l,e by the layer-wise mean:
(5)
Here, aup l,e combines the response scale of the up branch with the gate-to-down transmission path, while agate l,e symmetrically combines the gate-branch scale with the up-to-down path. Together, these quantities provide complementary factorizations of the multiplicative SwiGLU transformation, preserving structural information from both input branches and the shared output projection. We aggregate them using the geometric mean: q gate e aGSP = aup (6) l,e l,e al,e . The geometric mean aligns naturally with SwiGLU’s multiplicative structure and is more sensitive to imbalanced responses between the two complementary paths. Compared with additive aggregation, it limits the dominance of an exceptionally large single-path norm, assigning higher scores to experts with consistently strong responses along both paths. Since parameter scales, hidden dimensions, and RMSNorm
Router-Conditioned Refinement (RCR)
Although GSP provides a calibration-free baseline for estimating global expert contributions, it is inherently unconditional: it evaluates an expert’s overall response to generic normalized input directions based on its parameter structure, without distinguishing the input subspaces that the expert receives. In practice, however, an MoE router often exhibits pronounced directional selectivity, assigning different experts to distinct regions of the hidden-state space. Consequently, an expert with a weak global structural response may not be uniformly weak across all directions; it may still produce a strong response along directions favored by the router. Relying solely on GSP may therefore underestimate experts with pronounced directional specialization. To address this limitation, we introduce Router-Conditioned Refinement (RCR), which exploits directional preferences encoded by the pretrained router to restrict expert evaluation from the generic input space to the corresponding router-conditioned subspace, thereby providing a local directional correction to the global GSP estimate. To extract expert-specific routing directions without accessing real activations, RCR directly leverages the parameter
structure of the pretrained router. Consider the routing logit assigned to expert e at layer l: zl,e (x) = x⊤ wl,e + bl,e , where wl,e denotes the router weight vector associated with expert e. Since softmax routing depends only on relative differences among expert logits, the router weights are invariant to a shared translation: adding the same vector to all expert weight vectors leaves the resulting routing probabilities unchanged. We therefore center the router weights within each layer to obtain a direction representation with explicit relative discriminative meaning: N
w el,e = wl,e −
l 1 X wl,j . Nl j=1
(9)
The centered vector w el,e characterizes the routingdiscriminative direction of expert e relative to other experts in the same layer, while remaining invariant to any shared translation of the router weights (proof is provided in Appendix Sec. A.2). To interpret this centered direction from a conditional-distribution perspective, we adopt a Gaussian approximation to the centered hidden states. Let µxl = Ex∼pl [x] and assume x − µxl ∼ N (0, Σl ), where pl (x) denotes the input distribution at layer l and Σl is its covariance matrix. We then define an exponentially tilted distribution induced by the centered router direction: ⊤ exp w el,e x pl (x) h i . pl,e (x) = (10) ⊤x Ex∼pl exp w el,e Under the Gaussian approximation, the conditional mean shift is Ex∼pl,e [x] − Ex∼pl [x] = Σl w el,e . When Σl ≈ σl2 I, the mean-shift direction is approximately aligned with w el,e . Therefore, the centered router weight can be interpreted as a data-free approximation to the expert-conditioned input direction (complete derivation is provided in Appendix Sec. A.2). Based on the above analysis, RCR uses the centered router weight to approximate the expert-conditioned input direction and constructs an expert-specific direction prototype through RMS normalization: w el,e · ρl , (11) RMS(w el,e ) + ϵ q P d 1 2 where, for w ∈ Rd , RMS(w) = j=1 wj , d denotes d the hidden-state dimension, and ρl is a prior on the input scale of layer l. To preserve the fully calibration-free property of ACE, we set ρl = 1 throughout. RMS normalization removes variations in the overall magnitudes of router weight vectors across experts, allowing ql,e to primarily retain their relative routing directions rather than redundantly encode routing confidence, which is already captured by the runtime gate gl,t,i . Since inputs following RMSNorm exhibit a relatively stable root-mean-square scale, ql,e serves as a data-free directional prototype whose scale is compatible with actual expert inputs. Before deployment, RCR feeds ql,e into its corresponding expert and performs a complete SwiGLU forward pass to measure the response along its routing-preferred direction. Specifically, the raw router-conditioned amplification ql,e =
of expert e is defined as e aRCR = l,e
∥fl,e (ql,e )∥2 . ∥ql,e ∥2 + ϵ
(12)
The numerator measures the output magnitude produced along the router-conditioned direction, while the denominator normalizes the prototype input scale. This ratio therefore characterizes the expert’s directional amplification capacity and improves comparability across experts. Similar to GSP, raw router-conditioned responses may vary across MoE layers due to differences in parameter magnitudes and nonlinear response ranges. We therefore normalize the prototype amplification within each layer: ARCR (e) = l
e aRCR l,e . P Nl 1 aRCR +ϵ j=1 e l,j Nl
(13)
The normalized score ARCR (e) measures the routerl conditioned response strength of expert e relative to the average expert in the same layer. A value greater than one indicates an above-average response along its routing-preferred direction, even if its global structural response measured by GSP is weak. During online inference, RCR introduces no prototype evaluation or expert forward computation. ACE only retrieves the precomputed score ARCR (ei ) for each l routed expert slot and combines it with the current gate: RCR sRCR (ei ). l,t,i = gl,t,i Al
(14)
The resulting scores are normalized within the top-k candidate set to obtain the router-conditioned contribution distribution: pRCR l,t,i = Pk
sRCR l,t,i
RCR j=1 sl,t,j + ϵ
.
(15)
Here, pRCR l,t,i measures the relative contribution of the i-th expert slot by considering token-dependent routing preference and the expert response along its router-conditioned direction. GSP and RCR assess each expert slot from two complementary perspectives: GSP captures the expert’s global transformation capacity encoded by its parameter structure, whereas RCR measures its localized response along the routing-preferred direction. To combine these estimates without additional fusion hyperparameters, ACE adopts a conservative maximum criterion and defines the final contribution score as GSP RCR cACE . (16) l,t,i = max pl,t,i , pl,t,i Under this criterion, a routed expert slot is considered skippable only when both GSP and RCR assign it low contribution. Given the global threshold τ (q) associated with target skipping rate q, slot i is skipped only if cACE l,t,i < τ (q). This conservative aggregation prevents an expert from being removed when it is considered important from either the global structural or router-conditioned directional perspective. Equivalently, let the low-contribution candidate sets identified by GSP and RCR be GSP Sl,t = i pGSP (17) l,t,i < τ (q) , RCR RCR Sl,t = i pl,t,i < τ (q) . (18)
According to Eq. (16), the final skipping set is ACE GSP RCR Sl,t = Sl,t ∩ Sl,t .
(19)
Thus, an expert slot is skipped only if both perspectives identify it as low-contribution. This intersection rule provides a conservative safeguard against incorrectly skipping directionspecialized experts whose global response may be weak but whose router-conditioned response remains strong (analysis of its conservativeness and resulting output perturbation is provided in Appendix Sec. A.2). To preserve the routing structure of the pretrained model, ACE always retains the top-1 expert with the highest original router gate and optionally enforces a minimum number of active experts per token. After the skipping decision, the gate values of the remaining experts are renormalized before final MoE output aggregation. Through this complementary global–local verification, ACE reduces the risk of excessively skipping direction-specialized experts while enabling fine-grained, token-adaptive expert sparsification. It thereby preserves both global structural stability and router-conditioned directional fidelity without modifying the pretrained router or expert parameters.
4
Experiments
We evaluate ACE along three dimensions: quality preservation at matched expert-skipping budgets, the contribution of its two parameter-derived views, and realized inference efficiency. The main text presents one complete 10–60% budget table per model, component ablations, threshold-construction cost, and complete latency measurements. Appendix B contains all task-level results and threshold construction details.
4.1
Experimental Setup
Models and benchmarks. We study Qwen3-30B-A3BInstruct-2507, Qwen3.6-35B-A3B, and Gemma-4-26B-A4Bit (Yang et al. 2025; Qwen Team 2026; Google DeepMind 2026). We evaluate WikiText-2 PPL at sequence length 2048 (Merity et al. 2017) and 7 downstream tasks: ARCChallenge (ARC-C), ARC-Easy (ARC-E), PIQA, MATH-500, GPQA-Diamond, HumanEval, and LiveCodeBench (Clark et al. 2018; Bisk et al. 2020; Hendrycks et al. 2021; Rein et al. 2023; Chen et al. 2021; Jain et al. 2024). We report unweighted mean of seven task accuracies. We omit Gemma-4 Table 1: Complete 10–60% budget comparison on Qwen330B-A3B-Instruct-2507. W2 denotes WikiText-2 PPL and Acc. denotes average downstream accuracy. BF16 is excluded from ranking; bold marks the best skipping result per metric and budget. Method
10%
20%
30%
40%
50%
60%
W2↓
Acc.↑
W2↓
Acc.↑
W2↓
Acc.↑
W2↓
Acc.↑
W2↓
Acc.↑
W2↓
BF16
7.52
80.67
7.52
80.67
7.52
80.67
7.52
80.67
7.52
80.67
7.52
80.67
Score NAEE MoDES DiEP AIMER Top-P SERE XShare ExpertSparsity GSP RCR ACE
7.70 7.70 7.57 10.09 7.55 7.54 8.77 8.10 7.68 7.56 7.59 7.55
79.69 80.39 79.79 76.58 80.08 79.95 75.59 76.24 79.54 80.14 80.13 80.39
8.09 8.07 7.84 16.00 7.65 7.66 11.00 9.14 8.02 7.67 7.83 7.66
77.85 79.34 78.80 64.36 79.94 79.54 66.65 63.27 79.45 79.94 78.85 80.25
8.65 8.68 8.33 28.42 7.82 7.82 14.27 10.82 8.49 7.87 8.31 7.86
77.97 77.77 77.99 47.05 79.36 79.27 62.55 32.96 78.63 79.59 77.81 79.77
9.63 9.82 9.28 65.87 8.24 8.19 20.81 13.67 9.31 8.22 9.19 8.19
75.39 72.93 76.03 27.44 77.17 77.73 58.05 22.48 76.76 77.86 75.36 78.21
12.03 12.48 12.25 136.92 8.86 8.85 46.36 19.63 10.98 8.99 11.62 8.85
66.69 64.40 56.61 3.96 73.40 74.17 38.62 10.98 73.67 74.10 67.01 74.30
18.76 19.56 17.80 459.88 10.89 10.87 178.96 39.22 13.39 10.99 17.95 10.86
37.64 43.44 39.15 3.62 60.88 61.91 9.82 6.15 62.28 62.53 39.60 63.35
PPL because unusually high perplexity for the original model is poorly aligned with its downstream capability; consequently, downstream task accuracy provides a more informative measure of pruning-induced quality changes for this model. Evaluation protocol. All methods use BF16 and the same Hugging Face and EvalScope 1.4.1 evaluation pipeline. All downstream evaluations are zero-shot, use identical prompts and splits, and apply greedy decoding. Generation is capped at 1,024 tokens for ARC-C, ARC-E, and PIQA and at 2,048 tokens for reasoning and coding tasks. Results are single deterministic runs with seed 42; sub-point differences, especially on GPQA-Diamond, should therefore be interpreted cautiously. The skipping ratio is the fraction of routed top-k expert slots that are not executed. For every method and benchmark, a single unlabeled full-model pass collects runtime selection scores and searches for the threshold realizing each operating point in {10, 20, 30, 40, 50, 60}%. Accordingly, the ratios reported in all result tables are measured, realized skipping ratios rather than unconstrained target values. The top-1 routed expert is always retained, and threshold search accounts for this safeguard and the minimum-active-expert constraint. ACE’s contribution estimators remain entirely data-free: GSP and RCR are computed only from frozen model parameters and require neither training samples nor evaluation inputs. The unlabeled pass is used solely to translate a requested execution budget into the numerical scale of resulting scores; it never defines, trains, or updates either estimator. Moreover, thresholds associated with the same realized budget remain closely clustered across workloads and can therefore be transferred across datasets. Appendix B.1 formalizes the mapping procedure and reports its cross-workload stability. Baselines. We compare ACE with router-score skipping (Score), NAEE (Lu et al. 2024), MoDES (Huang et al. 2026), and DiEP (Bai et al. 2025). On Qwen3-30B, where compatible runs are available, we additionally report AIMER (Liu et al. 2026b), Top-P routing (Huang et al. 2024), SERE (Wu et al. 2026), XShare (Vankov et al. 2026), and ExpertSparsity (Lu et al. 2024). All methods preserve pretrained top-k candidate set and are evaluated with the same BF16 implementation, realized skipping budgets, prompts, splits, and decoding pipeline. This controlled protocol ensures performance differences primarily reflect the quality of expert-selection decisions rather than differences in evaluation conditions. Table 2: Complete 10–60% budget comparison on Qwen3.635B-A3B. W2 denotes WikiText-2 PPL and Acc. denotes average downstream accuracy. BF16 is excluded from ranking; bold marks the best skipping result per metric and budget.
Acc.↑
Method
10%
20%
30%
40%
50%
60%
W2↓
Acc.↑
W2↓
Acc.↑
W2↓
Acc.↑
W2↓
Acc.↑
W2↓
Acc.↑
W2↓
Acc.↑
BF16
7.01
81.17
7.01
81.17
7.01
81.17
7.01
81.17
7.01
81.17
7.01
81.17
Score NAEE MoDES DiEP GSP RCR ACE
7.38 7.37 7.38 8.83 7.18 7.31 7.16
81.66 81.27 80.86 79.18 81.12 80.97 81.41
7.79 7.80 7.41 12.09 7.38 7.65 7.34
80.51 80.99 80.59 76.91 80.50 81.05 81.46
8.36 8.40 7.72 17.62 7.66 8.17 7.57
78.41 79.01 78.19 72.73 79.46 78.36 79.89
9.21 9.30 8.10 28.45 8.08 8.95 7.88
74.61 74.84 75.74 66.16 78.25 75.35 78.79
10.58 10.89 9.42 45.43 8.87 10.39 8.67
67.45 67.50 71.42 56.42 75.01 66.42 75.57
13.38 14.17 10.87 85.12 10.48 13.48 9.98
54.66 53.42 69.04 32.27 70.54 45.38 70.70
4.2
Main Results
4.3
Ablation Study Table 4 isolates the two of-
Table 4: Component ablation. fline estimators. GSP proBF16: PPL 7.52 and Avg. 80.67; vides the stronger individual Drop is the Avg. decrease. signal, whereas RCR alone Tables 1–3 report the complete 10–60% curves for all three models. Under mild skipping, several methods remain close to the unpruned model because only the least influential routed slots are removed. For example, Score attains the highest average accuracy on Qwen3.6 at 10%, while ACE remains competitive. This result indicates that router confidence can be sufficient in the low-sparsity regime and that ACE’s structural estimates become most useful when increasingly consequential expert decisions must be made. As the skipping ratio increases, the difference between router-only or single-view criteria and ACE becomes substantially larger. ACE achieves the best average accuracy on Qwen3-30B (tied with NAEE at 10%) and from 20% onward on Qwen3.6, where it also obtains the lowest PPL throughout. At 50%, ACE improves Qwen3.6 from MoDES’s 9.42/71.42 to 8.67/75.57 (PPL/Acc.). At 60%, it leads the strongest external baseline by 1.07 and 1.66 accuracy points on Qwen3-30B and Qwen3.6, respectively. These widening margins support the central motivation of ACE: high-ratio skipping requires more than relative router preference because structurally weak and directionally specialized experts cannot be reliably distinguished by gate values alone.
Skip Method PPL↓ Avg.↑ Drop
10%
GSP RCR ACE
7.56 80.14 0.53 7.59 80.13 0.54 7.55 80.39 0.28
30%
GSP RCR ACE
7.87 79.59 1.08 8.31 77.81 2.86 7.86 79.77 0.90
50%
GSP RCR ACE
8.99 74.10 6.57 11.62 67.01 13.66 8.85 74.30 6.37
Table 5 shows that Max per-
Table 5: Fusion-strategy ablation forms best. It implements a on Qwen3-30B-A3B. Max is the conservative intersection: a ACE rule. slot is skipped only when
The results also reveal that language-modeling perplexity and downstream accuracy are complementary rather than interchangeable. A method with competitive WikiText-2 PPL does not necessarily preserve reasoning and coding accuracy, particularly at aggressive budgets. ACE is consistently strong on both measures, suggesting that its conservative maximum fusion avoids removing experts that appear unimportant under only one contribution view. Complete task-level results supporting these trends are provided in Appendix B.2.
30%
Fusion
50%
PPL↓ Avg.↑ PPL↓ Avg.↑ Min Mean Max (ACE)
4.4 Table 3 reports average Table 3: Gemma-4 average accuracy downstream accuracy; across 10–60% skipping. BF16 is excluded from ranking and bold marks Method Acc.↑ (%) the best skipping result. 10% 20% 30% 40% 50% 60% ACE leads at every budget. Its advantage over BF16 81.63 81.63 81.63 81.63 81.63 81.63 GSP grows from 0.44 Score 80.59 80.14 78.19 76.10 71.14 64.05 points at 30% to 1.38 NAEE 80.67 80.73 77.97 73.84 68.28 57.36 points at 50% and 2.46 MoDES 80.82 80.63 78.42 77.84 38.74 26.86 points at 60%. ComDiEP 79.61 78.66 75.90 72.14 30.83 18.13 peting criteria degrade GSP 81.14 81.16 79.71 78.65 75.73 69.65 sharply beyond 40%, RCR 80.74 79.84 78.00 77.49 68.96 60.03 whereas the conservaACE 81.59 81.28 80.15 79.51 77.11 72.11 tive maximum rule preserves experts that remain important under either structural or router-conditioned evidence.
degrades more rapidly because a single router-derived prototype cannot represent the complete conditional input distribution. Their complementary value emerges in ACE: at 50% skipping, the combined rule reduces PPL from 8.99 to 8.85 and improves average accuracy from 74.10 to 74.30 over GSP.
8.21 77.20 7.95 78.45 7.86 79.77
9.20 73.12 8.97 74.01 8.85 74.30
both views consider it dispensable. Min is overly aggressive, while averaging can suppress a strong signal from one view. Max preserves complementary evidence without an additional fusion hyperparameter.
Efficiency Analysis ACE precomputes GSP and
Table 6: Threshold selection on RCR once; online inference Qwen3-30B at 10% skipping. uses only expert-indexed lookups and top-k scalar operations. Table 6 shows that Quantile mapping 1.8 1.0× one quantile pass constructs Binary search 16.5 9.2× a threshold in 1.8 minutes, MoDES frontier search 18.9 10.5× versus 16.5 minutes for binary search and 18.9 minutes for MoDES frontier search. This 9.2–10.5× reduction concerns deployment preparation rather than token-level inference. On an NVIDIA A100, Table 7: Latency at 60% skipping. ACE provides 1.72–2.25× TTFT (ms) TPOT (ms/tok) TTFT and 1.31–1.41× Length Batch TPOT speedups. At length BF16 ACE BF16 ACE 1,024 and batch size 1, 256 1 144.7 84.1 34.9 26.7 TTFT falls from 270.9 512 1 183.7 102.3 57.2 44.9 to 120.3 ms and TPOT 1024 1 270.9 120.3 85.1 63.0 from 85.1 to 63.0 ms/token. 1024 2 290.5 156.7 95.3 67.8 Matched-budget methods 1024 4 344.9 182.6 127.3 91.1 share the same dispatch path; lightweight scoring keeps their latency within 1%. Thus, ACE improves quality at the same executed-expert cost. Procedure
Time (min) Slowdown
5
Conclusion
We introduced ACE, a training-free and checkpointpreserving method for token-adaptive expert skipping in MoE language models. GSP captures global SwiGLU transformation capacity, while RCR measures router-conditioned
directional response; their conservative fusion skips a slot only when both views find it low-contribution. Across three MoE models, ACE is particularly effective at aggressive budgets. At 50% skipping, it achieves the best average accuracy on all three models, reduces Qwen3.6 WikiText-2 perplexity by 7.96% versus the strongest competitor, and accelerates prefill and decoding. Current results use workload-matched threshold mapping and single deterministic runs; future work should examine threshold transfer, repeated-run uncertainty, and distributed expert dispatch.
References Bai, S.; Li, H.; Zhang, J.; Hong, Z.; and Guo, S. 2025. DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning. Advances in neural information processing systems, 38: 56090–56115. Bisk, Y.; Zellers, R.; Le Bras, R.; Gao, J.; and Choi, Y. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. In Proceedings of the AAAI Conference on Artificial Intelligence. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374. Chen, Y.; Shao, Y.; Wang, P.; and Cheng, J. 2025. EAC-MoE: Expert-selection aware compressor for mixture-of-experts large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12942–12963. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457. DeepSeek-AI; Xu, A.; Lin, B.; Xue, B.; Wang, B.; Xu, B.; Wu, B.; Zhang, B.; Lin, C.; Dong, C.; Ling, C.; et al. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348. Google DeepMind. 2026. Gemma-4-26B-A4B-it Model Card. Hugging Face model card. Guo, Y.; Cheng, Z.; Tang, X.; Tu, Z.; and Lin, T. 2025. Dynamic mixture of experts: An auto-tuning approach for efficient transformer models. In The Thirteenth International Conference on Learning Representations. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving with the MATH Dataset. In Advances in Neural Information Processing Systems. Huang, Q.; An, Z.; Zhuang, N.; Tao, M.; Zhang, C.; Jin, Y.; Xu, K.; Chen, L.; Huang, S.; and Feng, Y. 2024. Harder task needs more experts: Dynamic routing in MoE models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12883–12895. Huang, Y.; Wang, Z.; Yuan, Z.; Ding, Y.; Gong, R.; Guo, J.; Liu, X.; and Zhang, J. 2026. Modes: Accelerating mixtureof-experts multimodal large language models via dynamic expert skipping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 30205–30215. Jain, N.; Han, K.; Gu, A.; Li, W.-D.; Yan, F.; Zhang, T.; Wang, S.; Solar-Lezama, A.; Sen, K.; and Stoica, I. 2024. LiveCodeBench: Holistic and Contamination-Free Evaluation of Large Language Models for Code. arXiv preprint arXiv:2403.07974. Lasby, M.; Lazarevich, I.; Sinnadurai, N.; Lie, S.; Ioannou, Y.; and Thangarasa, V. 2026. Reap the experts: Why pruning prevails for one-shot moe compression. arXiv preprint arXiv:2510.13999.
Lee, J.; Hwang, S.-w.; Qiao, A.; Campos, D. F.; Yao, Z.; and He, Y. 2025. Stun: Structured-then-unstructured pruning for scalable moe pruning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 13660–13676. Liu, B.; Tian, K.; Wang, W.; Zhang, Z.; Qiao, L.; and Li, D. 2026a. Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9653–9667. Liu, Z.; Chen, G.; Tang, S.; Shen, Y.; Wang, H.; and Yuan, X. 2026b. AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning. arXiv preprint arXiv:2603.18492. Lu, X.; Liu, Q.; Xu, Y.; Zhou, A.; Huang, S.; Zhang, B.; Yan, J.; and Li, H. 2024. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 6159–6172. Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2017. Pointer Sentinel Mixture Models. In International Conference on Learning Representations. Qwen Team. 2026. Qwen3.6-35B-A3B Model Card. Hugging Face model card. Rein, D.; Hou, B. L.; Stickland, A. C.; Petty, J.; Pang, R. Y.; Dirani, J.; Michael, J.; and Bowman, S. R. 2023. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv preprint arXiv:2311.12022. Shazeer, N.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q. V.; Hinton, G. E.; and Dean, J. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. International Conference on Learning Representations. Vankov, D.; Ivkin, N.; Ulrich, K.; Song, X.; Khetan, A.; and Karypis, G. 2026. XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference. arXiv preprint arXiv:2602.07265. Wu, J.; Cheng, J.; Lv, F.; Dan, O.; and Yuan, L. 2026. SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models. arXiv preprint arXiv:2602.07616. Xie, Y.; Zhang, Z.; Zhou, D.; Xie, C.; Song, Z.; Liu, X.; Wang, Y.; Lin, X.; and Xu, A. 2024. Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router. arXiv preprint arXiv:2410.12013. Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. Yoon, S.; and Choi, Y. S. 2026. COMPEL: Compensated Mixture-of-Experts Pruning with Expert-Layer distribution. In Findings of the Association for Computational Linguistics: ACL 2026, 30408–30423. Zhao, Z.; Guo, L.; Cheng, J.; Gao, X.; Huang, H.; and Liu, J. 2025. Ada-K Routing: Boosting the Efficiency of MoE-based LLMs. In International Conference on Learning Representations, volume 2025, 89619–89635.
Appendix Overview •
A A.1
Appendix
Detailed Algorithms of ACE
This subsection presents the complete algorithmic workflow of ACE. We provide four pseudocode blocks corresponding to the overall framework, Global Spectral Proxy (GSP), Router-Conditioned Refinement (RCR), and online conservative expert skipping. All expert-level statistics used by GSP and RCR are computed once before deployment. During inference, ACE only performs expert-indexed table lookup and lightweight scalar operations before selectively executing the retained experts. Overall Workflow of ACE Pipeline description. ACE consists of an offline parameteranalysis stage and an online token-adaptive skipping stage. Given a pretrained MoE-based LLM, the offline stage first applies GSP to estimate the global transformation capacity of each expert from its frozen SwiGLU parameters. RCR then constructs expert-specific direction prototypes from the centered router weights and evaluates the corresponding experts along their routing-preferred directions. The resulting GSP and RCR amplification scores are stored as two expertindexed lookup tables. During inference, ACE combines these offline scores with runtime router gates and skips an expert slot only when both perspectives identify it as low-contribution. The overall workflow is summarized in Algorithm 1.
Algorithm 1: ACE: Overall Calibration-Free Expert Skipping Framework Input: Pretrained MoE-based LLM M ; target skipping control q; threshold mapping τ (·); minimum number of active experts mmin ; numerical constant ϵ Output: GSP table AGSP ; RCR table ARCR ; ACE c inference procedure M // Offline parameter analysis GSP 1 A ← GSP(M, ϵ); RCR 2 A ← RCR(M, ϵ); GSP 3 Store A and ARCR as expert-indexed lookup tables; // Online token-adaptive inference 4 τq ← τ (q); 5 for each input sequence do 6 for each token t do 7 for each MoE layer l do 8 Obtain the original top-k routed experts and gates {(ei , gl,t,i )}ki=1 ; 9 Il,t , {b gl,t,i }i∈Il,t ← ACESkip l, t, {(ei , gl,t,i )}ki=1 , AGSP , ARCR , τq , mmin , ϵ 10 11
12
Evaluate only the experts indexed by Il,t ; Aggregate their outputs using {b gl,t,i }i∈Il,t ; c; Construct the resulting inference procedure as M
Global Spectral Proxy Global structural response estimation. GSP estimates each expert’s intrinsic transformation capacity directly from its frozen parameters. For the l-th MoE layer, let Γl denote the diagonal matrix formed by the learned RMSNorm scaling vector. For each expert e, GSP constructs two branchsymmetric surrogates that jointly preserve the gate, up, and down projections. Their geometric mean provides the raw global amplification score, which is subsequently normalized by the average score of all experts in the same layer. Offline computation. The entire procedure requires neither calibration samples nor input activations. Once the normalized scores AGSP (e) have been computed for all experts, they are l stored as a static lookup table and reused for every token during inference. The complete procedure is given in Algorithm 2. Router-Conditioned Refinement Router-conditioned direction construction. RCR refines the unconditional GSP estimate by measuring each expert along the direction preferred by the pretrained router. For the l-th layer, the router weight associated with expert e is centered by subtracting the mean router weight over all experts. This removes the shared translation component and preserves the relative routing-discriminative direction. RCR then applies RMS normalization to obtain the expert-specific direction prototype ql,e .
Directional response evaluation. Each prototype is passed once through its corresponding frozen SwiGLU expert before deployment. The ratio between the expert output norm and prototype input norm defines the raw router-conditioned amplification. As in GSP, the raw responses are normalized within each layer and stored as an expert-indexed lookup table. This offline computation is summarized in Algorithm 3. Calibration-free prototype scale. ACE sets ρl = 1 for every MoE layer. This preserves the fully calibration-free setting and prevents the prototype magnitude from encoding unavailable activation statistics. Because each prototype is RMS-normalized, it retains the relative router direction while maintaining a stable input scale. Online Conservative Expert Skipping Dual-view contribution estimation. During inference, ACE retains the original top-k candidate set selected by the pretrained router. For each routed expert slot, it combines the current router gate with the offline GSP and RCR amplification scores. The two resulting score sets are independently normalized within the top-k candidate set, producing pGSP l,t,i and pRCR . l,t,i Conservative skipping rule. ACE defines the final contribution score as the maximum of the two normalized probabilities. Consequently, an expert slot is selected for skipping
Algorithm 2: GSP: Global Spectral Proxy Input: Pretrained MoE-based LLM M with L MoE layers; numerical constant ϵ Output: Global amplification table AGSP = {AGSP (e)} l GSP 1 Initialize A ← ∅; 2 for each MoE layer l = 1, . . . , L do 3 Extract the RMSNorm scaling vector γl ; 4 Γl ← Diag(γl ); 5 Let Nl denote the number of experts in layer l; 6 for each expert e = 1, . . . , Nl do gate up down 7 Extract Wl,e , Wl,e , and Wl,e ; gate 8 Gl,e ← Γl Wl,e ; up 9 Ul,e ← Γl Wl,e ; 10
down aup l,e ← ∥Ul,e ∥F · Gl,e Wl,e
11
agate l,e ← ∥Gl,e ∥F ·
; F
;
q gate aup l,e al,e ; PNl GSP µGSP ← N1l j=1 e al,j ; l for each expert e = 1, . . . , Nl do e aGSP ← l,e
12 13 14
e aGSP l,e AGSP (e) ← µGSP ; l +ϵ l GSP GSP Add Al (e) to A ;
15 16 17
down Ul,e Wl,e
F
return A
GSP
Algorithm 3: RCR: Router-Conditioned Refinement Input: Pretrained MoE-based LLM M with L MoE layers; numerical constant ϵ Output: Router-conditioned amplification table ARCR = {ARCR (e)} l RCR 1 Initialize A ← ∅; 2 for each MoE layer l = 1, . . . , L do 3 Let Nl denote the number of experts in layer l; 4 Let d denote the router input dimension; l 5 Extract router weight vectors {wl,e }N e=1 ; PNl 1 6 wl ← Nl j=1 wl,j ; 7 Set the calibration-free prototype scale ρl ← 1; 8 for each expert e = 1, . . . , Nl do 9 w el,e ← wl,e − wl ; q P d 2 10 rl,e ← d1 u=1 (w el,e,u ) ; 11 12 13 14 15
Sparsified MoE output. Let Il,t denote the final active expert-slot set returned by Algorithm 4. The sparsified MoE output is computed as X ACE yl,t = gbl,t,i fl,ei (xl,t ). (20) i∈Il,t
Because the skipping decision is completed before expert execution, only experts in Il,t require high-dimensional SwiGLU computation. The additional online operations introduced by ACE consist only of expert-indexed table lookup, scalar multiplication, normalization over the original top-k candidates, and small-set comparisons. ACE therefore preserves the pretrained router and expert parameters while introducing negligible control overhead relative to expert forward computation.
A.2
Detailed Proofs
This subsection provides the theoretical results supporting the design of ACE. We first derive the output-response upper
proto yl,e ← fl,e (ql,e ); proto ∥yl,e ∥2 e aRCR ← l,e ∥ql,e ∥2 +ϵ ; PNl RCR µRCR ← N1l j=1 e al,j ; l for each expert e = 1, . . . , Nl do e aRCR
16
l,e ARCR (e) ← µRCR ; l +ϵ
17
Add ARCR (e) to ARCR ; l
l
18
only when both views assign it a contribution below τq . The original top-1 expert is always retained. If the threshold rule would leave fewer than mmin experts active, ACE restores the skipped candidates with the largest contribution scores until the minimum active-expert constraint is satisfied. Finally, the gates of the retained experts are renormalized before expert execution and output aggregation. Algorithm 4 details this online procedure.
w e
ql,e ← ρl rl,el,e +ϵ ;
return ARCR
bound of a SwiGLU expert, which motivates the multiplicatively coupled construction of Global Spectral Proxy (GSP). We then prove the shared-translation invariance of router weights and derive the conditional mean shift induced by exponential tilting under a Gaussian approximation, providing the theoretical basis for Router-Conditioned Refinement (RCR). Finally, we establish the conservativeness of the dualview skipping criterion and analyze the output perturbation introduced by expert skipping and gate renormalization. Output-Response Bound of a SwiGLU Expert We derive the output-response bound used to motivate the branchsymmetric construction of GSP. Consider the e-th SwiGLU expert in the l-th MoE layer. After the preceding RMSNorm, its input is represented as x = x̄Γl , where x̄ is the RMSnormalized hidden direction and Γl is the diagonal matrix formed by the learned RMSNorm scaling vector. Ignoring bias terms, the expert output is gate up down fl,e (x) = SiLU x̄Γl Wl,e ⊙ x̄Γl Wl,e Wl,e . (21) For compactness, define gate u = x̄Γl Wl,e , up v = x̄Γl Wl,e .
For any scalar z, the SiLU activation satisfies
(22)
Algorithm 4: ACE: Online Conservative Expert Skipping Input: Layer index l; token index t; routed experts and gates {(ei , gl,t,i )}ki=1 ; lookup tables AGSP and ARCR ; threshold τq ; minimum number of active experts mmin ; numerical constant ϵ Output: Active expert-slot set Il,t ; renormalized gates {b gl,t,i }i∈Il,t 1 for each routed expert slot i = 1, . . . , k do GSP 2 sGSP (ei ); l,t,i ← gl,t,i Al RCR RCR 3 sl,t,i ← gl,t,i Al (ei ); Pk GSP GSP 4 Zl,t ← j=1 sl,t,j + ϵ; Pk RCR ← j=1 sRCR 5 Zl,t l,t,j + ϵ; 6 for each routed expert slot i = 1, . . . , k do 7
pGSP l,t,i ← pRCR l,t,i ←
sGSP l,t,i GSP Zl,t sRCR l,t,i RCR Zl,t
j=1
m m X X ≤ a2j b2j j=1
(25)
j=1
= ∥a∥22 ∥b∥22 , and therefore ∥a ⊙ b∥2 ≤ ∥a∥2 ∥b∥2 .
(26)
Using the submultiplicativity of the spectral norm, Eq. 21 satisfies down ∥fl,e (x)∥2 ≤ ∥SiLU(u) ⊙ v∥2 Wl,e 2
(27)
down Wl,e . 2
The two branch responses can be further bounded as gate ∥u∥2 = x̄Γl Wl,e
≤ ∥x̄∥2
2
gate Γl Wl,e
up ∥v∥2 = x̄Γl Wl,e
, 2
(28)
2
up ≤ ∥x̄∥2 Γl Wl,e
. 2
Substituting Eq. 28 into Eq. 27 yields
order; Restore the first mmin − |Il,t | indices to Il,t ; P gate 19 Zl,t ← j∈Il,t gl,t,j + ϵ; 20 for each retained expert slot i ∈ Il,t do g 21 gbl,t,i ← Zl,t,i gate ;
gate ∥fl,e (x)∥2 ≤ ∥x̄∥22 Γl Wl,e
18
2
up Γl Wl,e
2
down . Wl,e 2
(29) This result shows that the potential response of a SwiGLU expert depends jointly and multiplicatively on the gate branch, the up branch, and the down projection. Therefore, a proxy based on only one projection matrix cannot fully capture the structural response capacity of the expert. The branchsymmetric construction of GSP preserves all three components through two complementary factorizations of this multiplicative dependence.
l,t
return Il,t , {b gl,t,i }i∈Il,t
|SiLU(z)| = |zσ(z)| ≤ |z|,
a2j b2j
≤ ∥u∥2 ∥v∥2
ACE GSP RCR 12 Sl,t ← Sl,t ∩ Sl,t ; 13 itop1 ← arg maxi∈{1,...,k} gl,t,i ; ACE ACE 14 Sl,t ← Sl,t \ {itop1 }; ACE 15 Il,t ← {1, . . . , k} \ Sl,t ; 16 if |Il,t | < mmin then ACE 17 Sort the indices in Sl,t by cACE l,t,i in descending
22
m X
down ≤ ∥SiLU(u)∥2 ∥v∥2 Wl,e 2
;
; GSP RCR 9 cACE l,t,i ← max pl,t,i , pl,t,i ; n o GSP 10 Sl,t ← i pGSP < τ ; q l,t,i n o RCR 11 Sl,t ← i pRCR l,t,i < τq ; 8
∥a ⊙ b∥22 =
(23)
because the sigmoid function satisfies 0 < σ(z) < 1. Applying this inequality element-wise gives
Shared-Translation Invariance of Router Weights We prove that softmax routing probabilities are invariant to a shared translation of all router weight vectors. This property motivates centering the router weights before constructing expert-specific directional prototypes. For the l-th MoE layer, let the routing logit of expert e be zl,e (x) = x⊤ wl,e + bl,e .
(30)
The corresponding softmax routing probability is ∥SiLU(u)∥2 ≤ ∥u∥2 .
(24)
For any two vectors a, b ∈ Rm , their Hadamard product satisfies
exp (zl,e (x)) πl,e (x) = PNl . j=1 exp (zl,j (x))
(31)
Consider adding a common vector cl to every router weight vector:
′ wl,e = wl,e + cl ,
e = 1, . . . , Nl .
(32)
The translated routing logit becomes ′ ′ zl,e (x) = x⊤ wl,e + bl,e
= x⊤ wl,e + x⊤ cl + bl,e
(33)
Conditional Mean Shift Under Gaussian Exponential Tilting We derive the conditional mean shift induced by the centered router direction under a Gaussian approximation. This result provides the distributional interpretation of w el,e as an expert-conditioned input direction. Let the hidden-state distribution at the l-th MoE layer be approximated by x ∼ N (µxl , Σl ) .
⊤
= zl,e (x) + x cl . Because the additive term x⊤ cl is identical for all experts, the resulting routing probability is exp zl,e (x) + x⊤ cl
′ πl,e (x) = PNl
⊤ j=1 exp (zl,j (x) + x cl ) exp x⊤ cl exp (zl,e (x)) = PNl exp (x⊤ cl ) j=1 exp (zl,j (x))
(34)
Therefore, simultaneously translating all router weights by the same vector does not alter the routing probabilities. The same argument also applies to adding a shared scalar to all router biases. We next show that the centered router representation is invariant to this shared translation. Define the layer-wise mean router weight as N
l 1 X wl,j , Nl j=1
(35)
1 ⊤ −1 pl (x) = exp − (x − µ) Σ (x − µ) . 2 (2π)d/2 |Σ|1/2 (40) The exponentially tilted distribution induced by w is pl,e (x) =
= πl,e (x).
wl =
Gaussian density can be written as 1
(39)
For compactness, let w = w el,e , µ = µxl , and Σ = Σl . The
exp(w⊤ x)pl (x) . Ex∼pl [exp(w⊤ x)]
(41)
The exponent in the unnormalized density is 1 − (x − µ)⊤ Σ−1 (x − µ) + w⊤ x. 2 Expanding the quadratic term gives
(42)
1 1 − x⊤ Σ−1 x + x⊤ Σ−1 µ − µ⊤ Σ−1 µ + w⊤ x 2 2 (43) 1 ⊤ −1 1 ⊤ −1 = − x Σ x + x Σ (µ + Σw) − µ⊤ Σ−1 µ. 2 2 Completing the square with respect to x yields
and the centered weight of expert e as w el,e = wl,e − wl .
(36)
After the shared translation in Eq. 32, the layer-wise mean becomes N
w′l =
l 1 X (wl,j + cl ) Nl j=1
(37)
1 − (x − µ)⊤ Σ−1 (x − µ) + w⊤ x 2 1 ⊤ (44) = − (x − µ − Σw) Σ−1 (x − µ − Σw) 2 1 + w⊤ µ + w⊤ Σw. 2 The last two terms do not depend on x and are absorbed into the normalization constant. In particular, the momentgenerating function of the Gaussian distribution gives
= w l + cl . 1 ⊤ ⊤ ⊤ Ex∼pl exp(w x) = exp w µ + w Σw . 2
The centered translated weight is therefore ′ ′ w el,e = wl,e − w′l
= (wl,e + cl ) − (wl + cl ) = wl,e − wl =w el,e .
(45)
It follows that the tilted distribution is also Gaussian: (38)
Hence, w el,e removes the non-identifiable shared component of the router weights and preserves only the direction that discriminates expert e relative to the other experts in the same layer. This makes the centered router weight a more appropriate representation of expert-specific routing preference than the uncentered weight.
pl,e (x) = N (µ + Σw, Σ) .
(46)
Therefore, its conditional mean satisfies Ex∼pl,e [x] = µxl + Σl w el,e ,
(47)
and the mean shift relative to the original hidden-state distribution is Ex∼pl,e [x] − Ex∼pl [x] = Σl w el,e .
(48)
When the hidden-state covariance is approximately isotropic, i.e., Σl ≈ σl2 I, this result becomes Ex∼pl,e [x] − Ex∼pl [x] ≈ σl2 w el,e .
Conservativeness and Output Perturbation of Dual-View Skipping We prove that the maximum-based ACE criterion is equivalent to intersecting the low-contribution candidate sets identified by GSP and RCR. We further derive an outputperturbation bound for the resulting expert skipping and gate renormalization. Equivalence to candidate-set intersection. For a routed expert slot i, ACE defines (50)
Given a threshold τq = τ (q), the expert slot is initially considered skippable if cACE l,t,i < τq .
(51)
For any scalars a, b, and τ , the following equivalence holds: ⇐⇒
max(a, b) < τ
a < τ and b < τ.
k X
(49)
Thus, the conditional mean-shift direction is approximately aligned with the centered router weight. RCR consequently uses the RMS-normalized version of w el,e as a data-free approximation to the characteristic input direction associated with expert e.
GSP RCR cACE . l,t,i = max pl,t,i , pl,t,i
Skipped router-mass comparison. Assume that the original top-k gates are nonnegative and normalized such that
(52)
Applying Eq. 52 gives
gl,t,i = 1.
For any skipping set S, define its skipped router mass as X α(S) = gl,t,i . (59) i∈S
Since the gates are nonnegative and the ACE skipping set is a subset of either single-view set, ACE ACE α Sbl,t ≤ α Sl,t GSP RCR ≤ min α Sl,t , α Sl,t .
⇐⇒
pGSP l,t,i < τq and
pRCR l,t,i < τq .
(53)
Define the two low-contribution candidate sets as GSP Sl,t = i pGSP l,t,i < τq , RCR Sl,t = i pRCR l,t,i < τq .
Output perturbation after gate renormalization. For readability, omit the layer and token indices and let S and I denote the skipped and retained expert-slot sets, respectively. Define X X α= gi , 1−α= gi . (61) i∈S
i∈I
The original MoE output is X X y= gi fi (x) + gi fi (x).
(62)
i∈S
After skipping and renormalization, the retained gate is gbi = gi /(1 − α), and the ACE output becomes X gi yb = fi (x). (63) 1−α i∈I
(54)
Equation 53 directly implies
For 0 < α < 1, define the gate-weighted mean responses of the retained and skipped experts as 1 X gi fi (x), 1−α i∈I 1X µS = gi fi (x). α µI =
ACE GSP RCR Sl,t = Sl,t ∩ Sl,t .
(55)
Consequently, ACE GSP Sl,t ⊆ Sl,t ,
(60)
Thus, under the same threshold, the dual-view rule removes no more router probability mass than either individual-view rule.
i∈I
cACE l,t,i < τq
(58)
i=1
(64)
i∈S
ACE RCR Sl,t ⊆ Sl,t .
The original and sparsified outputs can then be written as (56)
The subsequent top-1 and minimum-active-expert safeguards can only remove indices from the initial ACE skipping ACE set. Hence, if Sbl,t denotes the actual skipping set after these safeguards, then
y = (1 − α)µI + αµS ,
yb = µI .
(65)
Their difference is therefore y − yb = α (µS − µI ) .
(66)
Taking the Euclidean norm gives ACE ACE GSP RCR Sbl,t ⊆ Sl,t ⊆ Sl,t ∩ Sl,t .
(57)
This proves that ACE is conservative relative to either individual view: an expert regarded as important by GSP or RCR cannot be skipped by the dual-view criterion.
∥y − yb∥2 = α ∥µS − µI ∥2 .
(67)
This identity shows that the output perturbation is jointly determined by the skipped router mass α and the discrepancy
between the average responses of the skipped and retained experts. Suppose that the expert outputs satisfy ∥fi (x)∥2 ≤ Fmax
for all i ∈ {1, . . . , k}.
(68)
Because µI and µS are convex combinations of expert outputs, ∥µI ∥2 ≤ Fmax ,
∥µS ∥2 ≤ Fmax .
(69)
Applying the triangle inequality to Eq. 67 yields ∥y − yb∥2 ≤ 2αFmax .
(70)
For the degenerate case α = 0, no expert is skipped and the perturbation is exactly zero. The top-1 retention rule guarantees that α < 1, so the gate-renormalized output remains well-defined. Combining Eq. 60 with Eq. 70 shows that the worst-case perturbation bound of ACE is no larger than that induced by either single-view candidate set under the same threshold: ACE ∥y − ybACE ∥2 ≤ 2Fmax α Sbl,t GSP RCR ≤ 2Fmax min α Sl,t , α Sl,t . (71) Therefore, the dual-view intersection, top-1 retention, and minimum-active-expert constraint jointly provide a conservative safeguard against excessive output perturbation, while still allowing token-adaptive expert sparsification.
B B.1
Additional Experimental Details
Budget-to-Threshold Mapping
ACE derives both expert-importance estimators exclusively from frozen model parameters. Neither GSP nor RCR uses calibration samples, evaluation inputs, labels, losses, or parameter updates. Deploying a requested execution budget nevertheless requires expressing that budget in the numerical scale of the data-free scores. We perform this separate budgetcontrol step with one unlabeled score-collection pass. Let Nall be the number of routed expert slots observed in the pass and Ncand the number remaining after excluding the slots protected by the top-1 and minimum-keep constraints. For a requested global skipping ratio q, the number of skipped slots is mq = clip(round(qNall ), 0, Ncand ) .
(72)
If the sorted candidate scores are c(1) ≤ · · · ≤ c(Ncand ) , we place τq between c(mq ) and c(mq +1) whenever the two values differ. Tied boundary values are handled by choosing the adjacent representable threshold whose realized count is closest to mq . One collected score distribution therefore yields the complete threshold table for all budgets. This procedure is deliberately separated from expert scoring and does not alter the GSP or RCR lookup tables. Its only role is budget control. After accounting for safeguarded slots and boundary ties, we report the measured fraction of skipped
routed slots; hence, the 10–60% operating points in the paper are realized rather than merely requested ratios. Table 8 reports the ACE thresholds obtained independently from every evaluation workload. For a fixed model and realized skipping ratio, the thresholds remain tightly clustered across datasets despite their different domains and output formats. This consistency shows that the score scale is primarily determined by the pretrained model and the requested execution budget rather than by a specific benchmark. Consequently, a threshold obtained from one unlabeled workload can be reused on another without recomputing GSP or RCR, supporting cross-dataset deployment of the data-free contribution estimators.
B.2
Complete Task-Level Results
Tables 9–26 report every task-level result underlying the budget curves in the main paper. Task metrics and Avg. are accuracies (%); PPL is evaluated on WikiText-2. Within each model and skipping budget, BF16 is an unpruned reference and is excluded from ranking. Boldface marks the best result among all pruning methods for each metric, with ties retained. Tables are ordered by model and then by target expert-skipping ratio from 10% to 60%.
Table 8: Cross-workload stability of ACE thresholds. Each cell reports the threshold independently obtained from the corresponding unlabeled evaluation workload at the specified realized skipping ratio. Model
Workload
10%
20%
30%
40%
50%
60%
Qwen3-30B-A3B
WikiText-2 ARC-C ARC-E PIQA MATH-500 GPQA-Diamond HumanEval LiveCodeBench
0.066076 0.058638 0.058372 0.059348 0.064817 0.062387 0.059915 0.063674
0.078141 0.071791 0.071509 0.073267 0.077139 0.074762 0.073406 0.076165
0.088436 0.082505 0.082139 0.084159 0.087528 0.085411 0.084152 0.086691
0.098972 0.093486 0.093135 0.095478 0.097363 0.095339 0.094608 0.096949
0.110965 0.105523 0.105159 0.106510 0.108013 0.106186 0.105755 0.107934
0.125460 0.121092 0.120699 0.121876 0.120972 0.120229 0.119866 0.121980
Qwen3.6-35B-A3B
WikiText-2 ARC-C ARC-E PIQA MATH-500 GPQA-Diamond HumanEval LiveCodeBench
0.068089 0.058495 0.058061 0.058142 0.060603 0.066227 0.060916 0.062504
0.080015 0.070524 0.070070 0.071141 0.073996 0.078559 0.075139 0.076356
0.089632 0.080678 0.080227 0.081325 0.084784 0.088131 0.086014 0.086684
0.098947 0.091051 0.090632 0.091348 0.095119 0.097670 0.096280 0.096633
0.109093 0.102505 0.102167 0.102849 0.105744 0.107836 0.106390 0.106850
0.121344 0.116258 0.116018 0.116473 0.117395 0.119644 0.117196 0.118335
WikiText-2 AIME 2025 ARC-C ARC-E Gemma-4-26B-A4B-it PIQA MATH-500 GPQA-Diamond HumanEval LiveCodeBench
0.051152 0.054991 0.052549 0.051525 0.054232 0.056684 0.055609 0.057597 0.056699
0.068149 0.070915 0.069626 0.068566 0.071812 0.073440 0.072302 0.073689 0.073054
0.081192 0.083559 0.082656 0.081766 0.084943 0.085783 0.084753 0.085818 0.085367
0.093195 0.095452 0.094467 0.093692 0.096504 0.097104 0.096559 0.097168 0.096766
0.105726 0.108153 0.107235 0.106557 0.108424 0.109174 0.108750 0.109297 0.108862
0.120364 0.122674 0.121681 0.121260 0.122556 0.122829 0.122434 0.122688 0.122271
Table 9: Qwen3-30B-A3B-Instruct-2507 at 10% expert skipping.
Table 11: Qwen3-30B-A3B-Instruct-2507 at 30% expert skipping.
Method
PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg.
Method
PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg.
BF16
7.52
93.17
94.23 56.57
93.90
55.26
80.00 91.57 80.67
BF16
7.52
93.17
94.23 56.57
93.90
55.26
80.00 91.57 80.67
Score 7.70 NAEE 7.70 MoDES 7.57 DiEP 10.09 AIMER 7.55 Top-P 7.54 SERE 8.77 XShare 8.10 ExpertSparsity 7.68 GSP 7.56 RCR 7.59 ACE 7.55
93.00 93.33 93.10 90.78 93.09 93.23 91.55 91.72 92.83 93.19 93.17 93.26
94.36 94.27 94.29 93.98 94.32 94.23 93.77 91.11 94.32 94.31 94.23 94.40
93.29 92.94 94.10 86.59 94.10 94.11 78.48 82.01 92.48 94.33 93.90 94.51
52.17 53.37 50.23 48.72 53.11 53.18 50.86 50.40 52.37 53.27 53.18 53.27
80.00 91.95 79.69 80.00 91.22 80.39 79.80 91.20 79.79 78.60 88.90 76.58 78.60 91.35 80.08 79.00 90.86 79.95 75.40 86.83 75.59 75.80 89.33 76.24 78.40 91.40 79.54 78.80 91.29 80.14 79.60 91.78 80.13 79.80 91.40 80.39
Score 8.65 NAEE 8.68 MoDES 8.33 DiEP 28.42 AIMER 7.82 Top-P 7.82 SERE 14.27 XShare 10.82 ExpertSparsity 8.49 GSP 7.87 RCR 8.31 ACE 7.86
92.58 91.89 91.19 78.41 92.66 92.43 83.87 26.62 92.49 92.01 91.64 92.66
94.28 94.11 94.89 86.78 94.23 93.40 91.33 36.83 91.71 94.22 94.19 94.36
92.68 91.29 93.01 47.56 93.14 93.11 62.09 60.82 92.86 93.39 92.07 93.29
48.01 47.20 49.88 37.86 51.77 52.48 40.78 30.78 52.77 53.20 49.67 52.66
78.20 89.55 77.97 79.00 90.91 77.77 75.40 89.60 77.99 3.80 70.40 47.05 79.00 88.66 79.36 79.00 90.10 79.27 56.20 77.15 62.55 20.20 22.21 32.96 77.40 88.90 78.63 78.60 89.28 79.59 75.40 89.66 77.81 79.40 89.45 79.77
53.03 57.57 55.78 48.48 55.97 55.05 52.22 53.31 55.01 55.77 55.05 56.06
50.51 50.00 51.98 4.55 56.09 54.39 26.44 33.28 54.29 56.42 52.02 56.56
Table 10: Qwen3-30B-A3B-Instruct-2507 at 20% expert skipping.
Table 12: Qwen3-30B-A3B-Instruct-2507 at 40% expert skipping.
Method
PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg.
Method
PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg.
BF16
7.52
93.17
94.23 56.57
93.90
55.26
80.00 91.57 80.67
BF16
7.52
93.17
94.23 56.57
93.90
55.26
80.00 91.57 80.67
Score 8.09 NAEE 8.07 MoDES 7.84 DiEP 16.00 AIMER 7.65 Top-P 7.66 SERE 11.00 XShare 9.14 ExpertSparsity 8.02 GSP 7.67 RCR 7.83 ACE 7.66
93.00 92.83 93.19 87.29 93.09 93.13 89.25 73.98 93.00 92.77 92.41 93.09
94.23 94.36 94.23 91.41 94.23 94.23 92.47 80.12 94.28 94.23 94.23 94.23
91.46 92.34 91.48 60.98 91.46 91.64 68.31 70.36 90.66 91.76 91.46 92.26
48.18 52.89 50.10 47.39 54.17 52.76 47.33 38.67 52.39 54.29 50.17 54.50
78.20 90.86 77.85 80.00 90.96 79.34 78.40 90.10 78.80 59.80 84.44 64.36 78.80 90.70 79.94 78.20 90.42 79.54 60.80 82.59 66.65 58.80 80.17 63.27 78.60 90.91 79.45 78.40 91.21 79.94 79.40 90.26 78.85 78.60 91.02 80.25
Score 9.63 NAEE 9.82 MoDES 9.28 DiEP 65.87 AIMER 8.24 Top-P 8.19 SERE 20.81 XShare 13.67 ExpertSparsity 9.31 GSP 8.22 RCR 9.19 ACE 8.19
89.27 89.51 89.17 54.52 91.66 91.47 75.26 20.22 91.47 91.88 90.70 92.15
93.64 93.35 93.30 61.11 93.56 93.01 84.86 24.41 93.52 94.11 93.98 94.07
83.93 83.15 85.95 0.00 87.47 88.49 57.49 58.83 86.29 88.93 86.59 90.85
46.92 37.82 49.91 22.17 48.98 50.49 38.91 4.81 50.10 50.29 49.10 51.37
76.60 87.87 75.39 79.00 88.82 72.93 76.40 88.48 76.03 3.80 44.89 27.44 78.20 88.12 77.17 78.40 89.61 77.73 56.40 73.23 58.05 5.80 13.02 22.48 76.40 87.54 76.76 78.60 88.14 77.86 76.80 88.41 75.36 78.60 88.41 78.21
48.99 52.02 54.11 19.19 57.10 56.42 25.83 40.77 56.28 56.90 54.04 58.08
49.49 38.89 48.99 5.56 52.17 52.67 20.17 30.30 52.00 53.10 41.92 52.02
Table 13: Qwen3-30B-A3B-Instruct-2507 at 50% expert skipping. Method
PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg.
BF16
7.52
93.17
94.23 56.57
93.90
55.26
80.00 91.57 80.67
Score 12.03 NAEE 12.48 MoDES 12.25 DiEP 136.92 AIMER 8.86 Top-P 8.85 SERE 46.36 XShare 19.63 ExpertSparsity 10.98 GSP 8.99 RCR 11.62 ACE 8.85
88.48 86.43 86.19 5.03 88.99 90.06 45.39 16.64 85.32 89.79 86.26 90.78
91.92 92.05 90.47 5.39 88.89 90.56 59.47 20.73 91.71 90.92 91.96 89.15
70.18 67.68 60.98 0.00 85.23 85.49 40.81 0.00 86.28 87.23 77.44 86.59
37.02 23.89 25.79 5.02 44.89 45.74 17.88 0.00 44.28 45.89 44.81 46.26
65.00 83.90 66.69 73.20 80.75 64.40 26.40 79.28 56.61 1.80 6.96 3.96 74.20 79.43 73.40 73.40 81.56 74.17 32.20 54.41 38.62 4.20 10.44 10.98 72.20 83.73 73.67 73.20 79.16 74.10 65.60 84.33 67.01 74.60 79.16 74.30
30.30 26.77 27.19 3.54 52.18 52.38 20.16 24.88 52.18 52.49 18.69 53.54
Table 18: Qwen3.6-35B-A3B at 40% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
7.01
95.14
94.82 47.47
95.12
57.35
84.60 93.69 81.17
Score 9.21 NAEE 9.30 MoDES 8.10 DiEP 28.45 GSP 8.08 RCR 8.95 ACE 7.88
94.88 94.62 94.22 80.84 95.05 93.94 95.10
94.44 94.40 94.12 82.07 94.61 94.15 94.67
84.15 84.15 89.38 80.11 95.73 88.41 95.38
39.05 37.82 44.28 32.98 45.21 44.74 47.88
82.00 93.42 74.61 81.20 92.82 74.84 79.40 90.11 75.74 71.20 80.16 66.16 82.40 93.36 78.25 78.00 90.32 75.35 82.60 93.33 78.79
34.34 38.89 38.67 35.75 41.41 37.88 42.56
Table 19: Qwen3.6-35B-A3B at 50% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
Table 14: Qwen3-30B-A3B-Instruct-2507 at 60% expert skipping. Method
PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg.
BF16
7.52
93.17
94.23 56.57
93.90
55.26
80.00 91.57 80.67
Score 18.76 NAEE 19.56 MoDES 17.80 DiEP 459.88 AIMER 10.89 Top-P 10.87 SERE 178.96 XShare 39.22 ExpertSparsity 13.39 GSP 10.99 RCR 17.95 ACE 10.86
75.34 56.25 68.81 4.44 79.82 79.97 10.48 4.86 80.42 79.61 66.81 80.97
81.86 56.25 79.96 6.40 82.87 83.21 19.76 10.84 83.22 83.98 80.56 85.61
10.50 27.44 10.29 0.00 70.29 74.11 0.00 0.00 74.80 74.59 17.68 75.00
5.95 5.88 13.82 0.00 31.17 31.27 0.00 0.00 32.92 33.59 18.80 32.61
13.20 66.54 37.64 60.40 80.21 43.44 19.80 70.98 39.15 2.60 7.89 3.62 57.10 79.64 60.88 58.70 80.53 61.91 3.80 29.83 9.82 2.20 8.81 6.15 59.40 80.02 62.28 59.40 80.58 62.53 23.00 68.82 39.60 60.20 82.32 63.35
10.10 17.68 10.39 4.04 25.29 25.56 4.86 16.32 25.18 25.93 1.52 26.77
7.01
95.14
94.82 47.47
95.12
57.35
84.60 93.69 81.17
Score 10.58 NAEE 10.89 MoDES 9.42 DiEP 45.43 GSP 8.87 RCR 10.39 ACE 8.67
93.77 94.20 93.18 73.87 94.28 91.13 94.49
94.15 93.98 92.65 77.98 94.49 91.88 94.40
60.98 67.68 80.49 56.91 90.85 71.34 91.22
25.21 23.89 34.42 20.88 39.72 26.16 41.98
75.80 91.46 67.45 75.20 90.75 67.50 76.00 90.20 71.42 68.40 76.11 56.42 79.60 91.78 75.01 65.40 88.25 66.42 79.40 91.69 75.57
30.81 26.77 33.01 20.76 34.34 30.81 35.78
Table 20: Qwen3.6-35B-A3B at 60% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
7.01
95.14
94.82 47.47
95.12
57.35
84.60 93.69 81.17
Score 13.38 NAEE 14.17 MoDES 10.87 DiEP 85.12 GSP 10.48 RCR 13.48 ACE 9.98
89.68 88.31 91.87 70.37 93.86 78.75 93.98
91.04 91.04 94.11 70.11 94.28 84.18 94.30
29.27 27.44 82.29 0.00 83.54 33.54 84.20
6.26 5.88 27.44 0.00 27.96 4.83 27.23
59.00 85.64 54.66 61.40 82.21 53.42 70.70 88.45 69.04 10.20 63.91 32.27 73.20 90.15 70.54 26.80 71.38 45.38 73.80 90.22 70.70
21.72 17.68 28.40 11.28 30.81 18.18 31.15
Table 15: Qwen3.6-35B-A3B at 10% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
7.01
95.14
94.82 47.47
95.12
57.35
84.60 93.69 81.17
Score NAEE MoDES DiEP GSP RCR ACE
7.38 7.37 7.38 8.83 7.18 7.31 7.16
95.31 94.97 95.22 93.28 95.05 95.31 95.34
94.78 94.87 94.39 94.19 94.82 94.74 94.82
97.56 96.95 95.31 95.10 96.34 95.12 95.78
59.72 58.39 56.21 52.87 56.30 56.59 56.44
84.80 94.02 81.66 84.00 94.23 81.27 83.20 93.80 80.86 80.20 93.11 79.18 84.40 94.45 81.12 82.80 93.74 80.97 84.60 94.40 81.41
45.45 45.45 47.89 45.48 46.46 48.48 48.48
Table 21: Gemma-4-26B-A4B-it at 10% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
–
95.82
94.91 51.52
95.73
59.81
86.60 87.05 81.63
Score NAEE MoDES DiEP GSP RCR ACE
– – – – – – –
94.11 94.45 94.07 94.10 94.11 94.03 94.87
94.28 94.40 93.88 93.19 93.90 93.98 93.98
92.68 93.90 93.90 92.45 93.90 94.51 95.12
58.10 56.21 57.34 55.98 57.91 56.55 58.29
85.60 87.32 80.59 88.20 87.54 80.67 86.20 87.05 80.82 84.20 87.05 79.61 87.40 87.27 81.14 87.80 87.27 80.74 88.00 87.21 81.59
52.02 50.00 53.29 50.33 53.48 51.01 53.65
Table 16: Qwen3.6-35B-A3B at 20% expert skipping. Table 22: Gemma-4-26B-A4B-it at 20% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
7.01
95.14
94.82 47.47
95.12
57.35
84.60 93.69 81.17
Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg.
Score 7.79 NAEE 7.80 MoDES 7.41 DiEP 12.09 GSP 7.38 RCR 7.65 ACE 7.34
94.71 94.88 94.11 91.22 93.86 94.80 94.88
94.82 94.99 94.20 91.17 94.82 94.78 94.82
93.90 96.34 95.29 93.67 95.73 95.73 95.73
54.88 52.89 56.19 49.34 51.85 58.39 58.11
85.60 93.20 80.51 84.40 93.96 80.99 84.60 93.08 80.59 79.80 91.11 76.91 84.80 94.45 80.50 84.60 93.58 81.05 84.80 94.40 81.46
BF16
–
95.82
94.91 51.52
95.73
59.81
86.60 87.05 81.63
Score NAEE MoDES DiEP GSP RCR ACE
– – – – – – –
94.28 94.45 93.78 92.98 94.11 93.26 94.93
94.11 94.23 94.02 92.46 94.15 93.64 94.17
92.68 93.90 93.90 92.17 95.73 93.90 95.73
55.26 55.83 56.27 53.91 56.94 54.08 57.09
85.60 87.05 80.14 85.60 87.05 80.73 85.60 86.84 80.63 84.00 84.10 78.66 86.40 87.27 81.16 85.40 86.56 79.84 86.80 86.89 81.28
46.46 49.49 46.66 42.09 47.98 45.45 47.49
Table 17: Qwen3.6-35B-A3B at 30% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
7.01
95.14
94.82 47.47
95.12
57.35
84.60 93.69 81.17
Score 8.36 NAEE 8.40 MoDES 7.72 DiEP 17.62 GSP 7.66 RCR 8.17 ACE 7.57
95.14 95.14 94.77 88.49 94.97 93.69 94.16
94.78 94.70 94.65 87.48 94.70 94.23 94.70
89.02 93.29 89.80 89.71 94.51 92.68 94.66
47.20 47.20 48.19 42.88 48.34 51.37 52.00
83.40 92.34 78.41 84.40 93.91 79.01 84.00 91.11 78.19 75.20 85.24 72.73 83.40 93.31 79.46 82.00 92.11 78.36 83.40 93.33 79.89
46.97 44.44 44.79 40.11 46.97 42.42 47.01
52.02 54.04 53.97 50.98 53.54 52.02 53.35
Table 23: Gemma-4-26B-A4B-it at 30% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
–
95.82
94.91 51.52
95.73
59.81
86.60 87.05 81.63
Score NAEE MoDES DiEP GSP RCR ACE
– – – – – – –
94.28 93.94 93.77 90.38 94.54 91.55 94.44
94.15 93.98 93.78 92.06 93.98 93.01 94.11
90.24 93.68 93.44 90.45 93.34 93.29 93.90
49.19 50.62 50.43 50.38 54.28 52.19 54.77
82.20 86.78 78.19 81.40 86.24 77.97 83.00 86.17 78.42 81.20 80.14 75.90 82.80 87.00 79.71 82.20 85.75 78.00 83.80 87.00 80.15
50.51 45.96 48.38 46.70 52.02 47.98 53.03
Table 24: Gemma-4-26B-A4B-it at 40% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
–
95.82
94.91 51.52
95.73
59.81
86.60 87.05 81.63
Score NAEE MoDES DiEP GSP RCR ACE
– – – – – – –
92.32 94.03 93.45 88.58 94.45 88.82 94.41
93.73 93.77 93.84 91.39 94.15 91.96 94.44
87.20 78.66 93.05 88.20 93.90 89.02 95.12
43.13 42.37 47.92 44.84 48.98 48.54 52.80
81.00 85.80 76.10 72.40 85.15 73.84 81.80 85.78 77.84 75.80 74.11 72.14 81.80 86.29 78.65 79.60 83.57 77.49 82.20 86.58 79.51
49.49 50.51 49.05 42.09 51.01 39.90 51.01
Table 25: Gemma-4-26B-A4B-it at 50% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
–
95.82
94.91 51.52
95.73
59.81
86.60 87.05 81.63
Score NAEE MoDES DiEP GSP RCR ACE
– – – – – – –
90.78 92.32 69.84 50.76 93.69 84.64 94.11
92.89 93.60 59.28 50.92 94.19 90.07 94.23
85.98 76.83 22.56 15.98 90.85 82.32 92.07
35.07 29.95 0.00 0.00 41.29 40.11 44.89
70.60 82.75 71.14 60.00 82.86 68.28 20.80 70.81 38.74 10.60 67.47 30.83 77.40 85.69 75.73 75.80 80.96 68.96 78.80 85.69 77.11
39.90 42.42 27.89 20.08 46.97 28.79 50.00
Table 26: Gemma-4-26B-A4B-it at 60% expert skipping. Method PPL↓ ARC-C ARC-E GPQA HumanEval LiveCodeBench MATH-500 PIQA Avg. BF16
–
95.82
94.91 51.52
95.73
59.81
86.60 87.05 81.63
Score NAEE MoDES DiEP GSP RCR ACE
– – – – – – –
85.07 86.18 50.01 29.87 90.53 75.68 91.89
90.53 91.71 48.18 38.97 93.35 82.58 93.73
73.78 58.54 6.10 0.00 87.80 67.07 91.46
26.07 16.21 0.00 0.00 30.98 26.10 33.87
59.40 80.69 64.05 36.00 78.56 57.36 6.80 48.92 26.86 0.00 39.17 18.13 61.60 82.86 69.65 65.80 74.21 60.03 67.60 84.82 72.11
32.83 34.34 28.01 18.92 40.40 28.79 41.38