DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
Chenyang Song 1 Weilin Zhao 1 Xu Han 1 Chaojun Xiao 1 Yingfa Chen 1 Zhiyuan Liu 1 [email protected], {han-xu,liuzy}@tsinghua.edu.cn
arXiv:2605.10933v1 [cs.LG] 11 May 2026
Abstract
High Performance
While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens. DECO utilizes the differentiable and flexible ReLU-based routing enhanced by learnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduce NormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing a more stable trend of routed-expert activation ratio and a higher intrinsic sparsity level. We also identify an empirical advantage in using non-gated MLP experts with ReLU-based routing, indicating the possibility of MoE architecture simplification. Experiments demonstrate that DECO, activating only 20% of experts, matches dense performance and outperforms established MoE baselines. Our specialized acceleration kernel delivers a 3.00× speedup on real hardware compared with dense inference. Codes and checkpoints will be released.
DECO DEnse COmparable Sparse MoE
Low Computation Cost (Low Active Parameter Scale)
Pruning Quantization …...
Small Storage Overhead (Low Total Parameter Scale)
Figure 1. The “ideal triangle” of end-side MoE. Beyond the high performance and reduced computational cost of sparse MoE, the model should maintain a minimal storage footprint, achieving high performance within dense-comparable total parameter budgets.
creasingly prominent model architecture. The key property of MoE is the sparse activation, namely, activating a small subset of expert modules from a large pool of parameters. Therefore, MoE retains high capacity and strong performance while substantially reducing computation costs. As a research hotspot, MoE has been extensively studied, from architecture design (Liu et al., 2024; Cai et al., 2025) to scaling laws and compute-optimal settings (Krajewski et al., 2024; Tian et al., 2025). Prior work primarily pursues two objectives: high performance and low computation cost. However, when it comes to end-side deployment, a third non-negligible objective emerges: small storage overhead. Concretely, MoE with a huge number of total parameters demands substantial storage space. More critically, large MoE models may incur high memory-access costs when transferring experts between GPU high-bandwidth memory and shared memory (Li et al., 2025), or when moving offloaded parameters from disk/flash storage to GPU/NPU memory of end-side devices. Such latency can erode the efficiency gains afforded by sparse computation.
1. Introduction The scale of large language models (LLMs) has grown rapidly to achieve consistent performance gains across diverse tasks. The rising training and deployment costs for massive LLMs have made mixture-of-experts (MoE) an in-
Therefore, as shown in Figure 1, an ideal MoE model for end-side deployment should satisfy the above three objectives. To pursue this “ideal triangle”, we pose the question: Can a sparse MoE model achieve performance comparable to a dense model, given the same total parameter budget and the same number of training tokens?
1
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China. Correspondence to: Xu Han <[email protected]>, Zhiyuan Liu <[email protected]>.
A closely related study by Li et al. (2025) identifies the
Preprint. May 12, 2026.
1
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices 0.5 0.2 -0.4 0.1
linear projection
………...
Token 1
Token T
other tokens
linear projection
-0.1 0.5 0.3 -0.6
① ReLU-based Routing 0.5 0.2
① ReLU-based Routing 0
② Learnable Expert-Wise Router Scaling
0.15 0.02
0
0.1
Expert 1
Up Proj ③ NormSiLU Down Proj
Expert 2
Up Proj ③ NormSiLU Down Proj
Expert 3
Up Proj ③ NormSiLU Down Proj
Expert 4
Up Proj ③ NormSiLU Down Proj
0
0.5 0.3
(learnable weights)
(learnable weights)
0.3 0.1 0.2 0.3
0.3 0.1 0.2 0.3
0.03
weighted sum
calculate router entropy
④ Non-Gated MLP Experts
Output 1
………...
Output T
0.731
⑤ Adaptive Sparsity Regularization
0.689
weighted sum
0
0
② Learnable Expert-Wise Router Scaling
0.05 0.06
0
calculate router entropy
Figure 2. The overall architecture of DECO. For router design, we adopt ReLU-based routing enhanced by learnable expert-wise router scaling. For expert design, we propose NormSiLU as a better routed-expert activation function and employ non-gated MLP experts. For precise sparsity control, we employ adaptive sparsity regularization.
optimal settings of DeepSeek-V3-style MoE architectures that enable them to surpass dense models under matched total parameters and computation budget. However, due to the low per-token computation of sparse MoE, in that work, MoE settings are trained on substantially more tokens under the same computation budget. We adopt a stricter setting that requires exactly the same number of training tokens.
As shown in Section 4, DECO demonstrates performance comparable to dense models when matched for total parameters and training tokens. DECO also surpasses established MoE baselines of the same scale and activation ratio. Naturally, DECO’s dense-comparability is conditional, since the performance of MoE is affected by many factors, including the activation ratio, expert granularity, and shared expert size. In Section 5, we analyze the effect of these factors.
To achieve this goal, we propose DECO (Figure 2), a sparse MoE architecture that achieves DEnse-COmparable performance through a fundamental revision of MoE design.
Finally, we implement a tailored acceleration kernel for DECO to test its practical inference acceleration value on real hardware. Based on CUTLASS (Thakkar et al., 2023), the kernel leverages tensor cores to improve computational throughput and reduces memory-access overhead by exploiting the sparse activation. Overall, the kernel achieves a speedup of 3.00× compared with vanilla dense inference.
For router design, conventional MoE models generally adopt TopK routing, which is non-differentiable and enforces a uniform activation ratio across all tokens. To overcome this issue, we adopt ReLU-based routing, a differentiable paradigm that enables flexible token-dependent activation ratios. Moreover, to mitigate output scale imbalances between shared and routed experts, while simultaneously accounting for potential expert heterogeneity, we introduce learnable expert-wise router scaling. This mechanism involves learnable scaling factors to calibrate the contribution of individual routed experts.
2. Preliminaries and Related Works To achieve high performance while curbing computational growth, MoE has recently risen as the mainstream architecture. An MoE typically comprises three components: a router, a set of experts, and an auxiliary training objective.
The expert design is similarly optimized for stability and efficiency. Empirical analysis reveals that coupling ReLUbased routing with vanilla SiLU-activated experts results in two critical problems: a surging routed-expert activation ratio (Figure 5) and vanishing SiLU output magnitudes (Figure 7). To resolve these issues, we propose NormSiLU, which applies dual-stage normalization prior to the SiLU operator. NormSiLU stabilizes activation trends and reduces the activation ratio, alleviating the need for aggressive sparsity regularization. It also produces more stable and significant SiLU output magnitudes, promoting better utilization of expert parameters. Beyond the activation function, we employ non-gated MLP experts rather than standard gated variants, as they exhibit superior empirical compatibility with ReLU-based routing. Finally, to precisely control the activation ratio, we design an adaptive sparsity regularization that auto-scales the regularization strength.
Router design. The router computes weights assigned to each expert and selects which experts to activate. It generally consists of a linear projection, an activation function, and post-processing of router scores. The activation function controls the expert selection pattern. Many MoE designs use TopK, which forces each token to activate a fixed number of experts (Jiang et al., 2024; Dai et al., 2024). However, TopK is criticized for its inflexibility (an input-invariant number of active experts) and nondifferentiability. TopP (Huang et al., 2024) selects experts by a threshold p, activating experts until their cumulative router score reaches at least p, thereby permitting token-dependent activation ratios. MoE++ (Jin et al., 2024) retains TopK but introduces zero-computation experts, which indirectly allows variable computation cost. To improve differentiability, ReMoE (Wang et al., 2024b) and BlockFFN (Song 2
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
et al., 2025b) adopt ReLU for expert selection. Since ReLU naturally produces considerable zero values while remaining differentiable, it enables smoothly learnable activation ratios and delivers performance advantages.
(Section 3.1), experts (Section 3.2), and adaptive sparsity regularization (Section 3.3).
Post-processing of router scores primarily normalizes expert weights to preserve a consistent output scale. Most designs use Softmax as the score normalizer. DeepSeek-V3 (Liu et al., 2024) instead applies element-wise Sigmoid followed by unit-sum normalization. Compared with Softmax, Sigmoid mitigates extremely skewed score distributions. Notably, DeepSeek-V3 also introduces a scalar scaling factor applied to router scores, which helps balance contributions between shared and routed experts.
ReLU-based routing. Distinct from conventional TopK routing, DECO incorporates ReLU in the router to determine expert activation. As demonstrated in prior studies (Yao et al., 2025; Song et al., 2025b), ReLU is fully differentiable, inherently induces sparsity, and supports tokendependent activation ratios. These attributes render ReLU a robust and flexible routing function.
3.1. Router Design
Learnable expert-wise router scaling. To balance the output scales of routed and shared experts, DECO applies a scaling operator to the routing scores before they are multiplied by the expert outputs. We extend the fixed scalar scaling factor of DeepSeek-V3 (Liu et al., 2024) to a learnable vectorized one. This modification accommodates the potential heterogeneity across routed experts by assigning them distinct, learnable coefficients.
In this work, DECO adopts ReLU-based routing, and replaces the fixed scalar scaling factor with learnable expertwise router scaling factors, providing flexibility and accommodating potential heterogeneity in expert output scales. Expert design. In most mainstream MoE models, each expert is a standard gated MLP with SiLU activation (SwiGLU) (Shazeer, 2020). DeepSeekMoE (Dai et al., 2024) shows the benefits of introducing fine-grained experts and shared experts but retains the SwiGLU backbone.
Formally, given the hidden dimension dh , the expert number Ne , and the input hidden state x ∈ Rdh , the router score p of DECO can be computed as follows:
DECO refines expert design by introducing NormSiLU as the expert activation, which resolves the issues of surging routed-expert activation ratio and vanishing SiLU output magnitudes. Moreover, we find that with ReLU-based routing, non-gated MLP experts empirically bring a more stable trend of activation ratio than the gated variant.
T p = α ⊙ ReLU(Wrouter x),
(1)
where W ∈ Rdh ×Ne and α ∈ RNe are learnable weights, and ⊙ represents element-wise multiplication. 3.2. Expert Design Non-gated MLP experts. While gated MLP is widely considered superior to the non-gated variant (Shazeer, 2020), we observe that non-gated experts exhibit more favorable properties within the specific context of ReLU-based routing. Concretely, in a ReLU-activated MoE, non-gated experts obtain a more stable trend of activation ratio, whereas gated variants exhibit a sharply increasing trend. This inherent stability implies that a significantly lower regularization penalty is required to achieve a target sparsity threshold, thereby alleviating the negative impact on performance.
Auxiliary training objective. Aside from the language modeling loss, MoE models generally introduce auxiliary training objectives. The most common one is for load balancing, typically implemented via the auxiliary loss proposed by Fedus et al. (2022). To alleviate auxiliary-loss interference with language modeling, DeepSeek-V3 adopts a loss-free load-balancing policy without a differentiable objective (Wang et al., 2024a). MoE models with variable activation ratios often incorporate a sparsification objective. For example, ReMoE applies adaptive L1-norm regularization, and BlockFFN employs chunk-wise sparsification (Wang et al., 2024b; Song et al., 2025b).
NormSiLU. We introduce NormSiLU (Algorithm 1) as an enhanced activation for MoE experts, prepending a dualstage normalization to the SiLU non-linearity.
Inspired by ReMoE, DECO uses an adaptive sparsity regularization, whose coefficient auto-scales to precisely control the sparsity level. We also replace the L1-norm with router entropy to improve numerical stability.
First, inter-expert mean normalization centers the expert upprojection weights around zero, ensuring the pre-activation input distribution is approximately zero-centered. This adjustment stabilizes the SiLU activation distribution within experts. Second, intra-expert RMS normalization is applied to maintain consistent activation magnitudes. We find that this dual-stage normalization not only prevents internal expert activations from vanishing, but also promotes a steady activation ratio at the router level. A theoretical demonstration for its rationality is presented in Appendix C.
3. Methodology of DECO We propose DECO, a sparse MoE architecture that achieves performance comparable to dense variants while maintaining the same total number of parameters and training tokens. We split our design into three components: the router 3
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
Algorithm 1 Pseudocode of NormSiLU. # Nt : num_tokens ; De : dim_expert ; Dh : dim_hidden # Ne : num_experts ; Na : num_active_experts rms_norm = RMSNorm ( dim = dim_expert ) # learnable RMSNorm layer , weight shape is [ De ] def NormSiLU ( input_hidden , up_proj_weight , intermediate_state ): # input_hidden : [Nt , Dh ], the input hidden state of MoE # up_proj_weight : [Ne , De , Dh ], the up - projection weights of experts # intermediate_state : [Nt , Na , De ], the expert intermediate state produced by a sparse linear operation between " input_hidden " and " up_proj_weight " # inter - expert mean normalization : conducted at the " num_experts " dimension up_proj_avg = mean ( up_proj_weight , dim =0) # [De , Dh ], fixed during inference intermediate_avg = matmul ( input_hidden , up_proj_avg .T) # [Nt , De ] intermediate_state -= intermediate_avg . unsqueeze (1) # intra - expert RMS normalization : conducted at the " dim_expert " dimension intermediate_state = rms_norm ( intermediate_state ) return SiLU ( intermediate_state ) # standard SiLU activation
4. Experiments
Given the expert intermediate dimension de , the structure of a DECO expert is formally defined as:
4.1. Main Results To demonstrate the architectural rationality of DECO, we compare it against the following baselines: Dense (i.e., a standard LLaMA-style Transformer using SwiGLU FFNs (Touvron et al., 2023)), TopP (Huang et al., 2024), DeepSeek-V3 (Liu et al., 2024), ReMoE (Wang et al., 2024b), and BlockFFN (Song et al., 2025b). Four total parameter scales are involved in experiments: Small (0.11B), Medium (0.24B), Large (0.53B), and XLarge (1.18B).
xup = SparseLinear(x, Wup ), x′up = NormSiLU(x, Wup , xup ),
(2)
y = SparseLinear(x′up , Wdown ), where Wup ∈ RNe ×de ×dh and Wdown ∈ RNe ×dh ×de are the up-projection and down-projection weights, respectively. SparseLinear operator facilitates sparse linear operations by involving only active experts at inference time.
All settings are trained on the same high-quality data mixture. The model performance is evaluated by two metrics: Perplexity (PPL) on the C4 English validation set (Raffel et al., 2020), and average accuracy across a suite of commonsense reasoning benchmarks. See Appendix A for more details about the experimental settings.
3.3. Adaptive Sparsity Regularization To effectively control the sparsity level, we adopt an adaptive sparsity regularization, based on the router entropy loss and a dynamic scaling algorithm for the coefficient.
To ensure a rigorous comparison, within each group of the same total parameter count, the number of training tokens is also held consistent at around 40 times the parameter count. All non-FFN components, including attention layers and embedding layers, remain identical within each group. All MoE settings within a group share the same routedexpert activation ratio (around 20% on the training data) and intermediate dimension of the shared expert. Furthermore, we ensure that all routed experts have close parameter counts to maintain consistent expert granularities.
Router entropy is a sparsification loss applied to a normalized router score. In DECO, it is calculated as: p1 = |p|/Sum(|p|),
Lent = −pT1 ln(p1 + ϵ)
(3)
where p is the router score defined in Equation 1. The router entropy loss, Lent , is then multiplied by a coefficient λ and added to the total training objective.
From the results shown in Figure 3 (evaluation results on individual benchmarks are shown in Appendix D), we derive the following conclusions:
Instead of using a static coefficient, inspired by ReMoE (Wang et al., 2024b), we adaptively scale λ according to the current sparsity level. Specifically, if the current sparsity falls below the target sparsity, λ is scaled by a η > 1 for the subsequent iteration; otherwise, λ is divided by η. In this way, DECO maintains a stable activation ratio centered precisely at the desired sparsity level.
(1) Dense comparability: With an average routed-expert activation ratio of 20%, DECO achieves performance parity with the Dense baseline. This holds true under the same total parameter budget and training token volume, demonstrating 4
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices Small, PPL ( )
Large, PPL ( )
Medium, PPL ( )
DECO
34.36
DECO
Dense
34.40
Dense
27.74 27.85
XLarge, PPL ( )
DECO
21.87
DECO
18.38
Dense
21.93
Dense
18.41
DeepSeek-V3
35.09
BlockFFN
28.16
DeepSeek-V3
ReMoE
35.13
DeepSeek-V3
28.19
ReMoE
22.19
18.69
ReMoE
22.49
18.99
DeepSeek-V3
BlockFFN
35.27
ReMoE
28.99
BlockFFN
22.94
BlockFFN
19.28
TopP
40.98
TopP
33.55
TopP
26.60
TopP
22.22
34.25 34.50 34.75 35.00 35.25 35.50
27.6 27.9 28.2 28.5 28.8 29.1 29.4
Small, Task ( )
36.90
DECO
36.53
BlockFFN DeepSeek-V3
36.31
TopP
34.71
42.48
ReMoE
38.57
ReMoE
42.80
Dense
38.82
DeepSeek-V3
42.81
DECO
39.01
Dense
36.60
ReMoE
BlockFFN
BlockFFN
42.34
TopP
36.34
TopP
38.67
38.4
38.6
38.8
39.0
39.2
39.4
42.30
42.45
XLarge, Task ( )
47.38
DECO 46.98
Dense
46.85
BlockFFN
42.38
DeepSeek-V3
38.44
36.30 36.45 36.60 36.75 36.90 37.05
18.25 18.50 18.75 19.00 19.25 19.50
Large, Task ( )
39.18
DECO
36.80
Dense
21.75 22.00 22.25 22.50 22.75 23.00 23.25
Medium, Task ( )
ReMoE
46.37
DeepSeek-V3
46.36
TopP 42.60
42.75
42.90
41.10 46.25 46.50 46.75 47.00 47.25 47.50 47.75
Figure 3. The evaluation results of DECO versus baseline settings. “PPL” and “Task” indicate the C4 validation perplexity and the average accuracy (%) on downstream benchmarks, respectively. DeepSeek-V3 uses gated MLP experts, and ReMoE uses non-gated ones. This is due to their better performance than the opposite settings, see Section 4.4 for detailed discussions.
DECO’s efficiency in maintaining dense-level representation power with reduced active computation.
To demonstrate the effect of DECO’s router scaling design, we experiment on two ablation settings: “Fixed” adopts a constant scaling factor for all routed experts, and “Scalar” involves a single learnable scalar scaling factor shared by experts. Both ablation settings are initialized with the same values as DECO. Evaluation results in Table 1 reveal the performance benefits of learnable vectorized scaling factors. We further explore the sensitivity of performance to initialization value in Appendix B.
(2) Performance superiority: Under the same routed-expert activation ratio, shared-expert dimensions, and expert granularity, DECO surpasses existing MoE baselines from perplexity to downstream task performance. 4.2. Effect of Learnable Expert-Wise Router Scaling
As empirical justification for this design, we analyze the distribution of expert output norms on the C4 validation set. As illustrated in Figure 4, the output scales of routed experts in DECO (Medium) exhibit clear heterogeneity. These results validate our hypothesis that applying expert-specific, learnable vectorized factors is essential to accommodate the varying output scale across different experts.
Table 1. The ablation results on the effect of learnable expert-wise router scaling. “Fixed” indicates one fixed scaling factor, while “Scalar” includes one shared learnable scalar scaling factor. Small PPL (↓) Task (↑)
Medium PPL (↓) Task (↑)
Fixed Scalar
34.43 35.16
36.74 36.63
27.94 27.92
39.08 38.54
DECO
34.36
36.90
27.74
39.18
4.3. Effect of NormSiLU Table 2. The ablation results on the effect of NormSiLU. The marker “w/o” means removing a specific normalization step, and “SiLU” is the vanilla SiLU operator without any normalization.
Average Output Norm
Sorted Routed-Expert Output Scales 1.0 0.8 0.6 0.4 0.2 0.0
mean level (0.23)
0
10
20
30 Expert Index
40
50
Figure 4. The distribution of routed-expert output norms in the first MoE layer of DECO (Medium) on the C4 validation set, which shows clear expert-wise heterogeneity.
Small PPL (↓) Task (↑)
Medium PPL (↓) Task (↑)
w/o Mean w/o RMS SiLU
34.57 35.77 35.69
36.85 36.78 36.46
27.77 27.97 28.05
38.88 39.04 38.85
NormSiLU
34.36
36.90
27.74
39.18
To validate the effect of NormSiLU, which incorporates both inter-expert mean normalization and intra-expert RMS 5
Activation Ratio (%)
45 40 35 30 25 20 0
3000
6000 9000 12000 15000 Training Step
Figure 5. The trend of activation ratio of DECO (Small) and ablation settings without different steps of NormSiLU. The baseline “SiLU” and “w/o RMS” settings show surging routed-expert activation ratio before being pulled back by regularization.
SiLU w/o RMS w/o Mean NormSiLU
0.06 0.05
Absolute SiLU Output Magnitudes
SiLU w/o RMS w/o Mean NormSiLU
50
Sparsity Regularization Coefficient
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
0.04 0.03 0.02 0.01 0.00
0
3000
6000 9000 12000 15000 Training Step
1.4 1.2 1.0
SiLU w/o RMS w/o Mean NormSiLU
0.8 0.6 0.4 0.2 0
3000
6000 9000 12000 15000 Training Step
Figure 6. The trend of the regularization co- Figure 7. The average absolute output magefficient of DECO (Small) and ablation set- nitudes of SiLU within routed experts of tings without different steps of NormSiLU. DECO (Small) and ablation settings without The baseline “SiLU” and “w/o RMS” set- different steps of NormSiLU. The baseline tings show significantly higher coefficients, “SiLU” and “w/o Mean” settings show vanwhich potentially harm performance. ishing SiLU output magnitudes. Table 3. The ablation results on the effect of expert gating. “GA” and “NG” indicate gated MLP experts and non-gated MLP experts, respectively. “DS-V3” indicates DeepSeek-V3.
normalization, we evaluate three ablation settings: “w/o Mean” removes inter-expert mean normalization, “w/o RMS” removes intra-expert RMS normalization, and “SiLU” is the standard SiLU operator without any normalization.
Setting
As shown in Table 2, both normalization steps contribute positively, with intra-expert RMS normalization providing a more substantial gain. To investigate the underlying mechanisms of NormSiLU, we track three critical variables throughout the training process: the routed-expert activation ratio, the sparsity regularization coefficient, and the average absolute output magnitudes of SiLU within experts.
DS-V3 (GA) DS-V3 (NG)
35.09 35.16
36.31 36.73
28.19 28.28
38.82 38.47
22.19 22.33
42.38 42.02
ReMoE (GA) ReMoE (NG)
36.02 35.13
36.61 36.60
29.55 28.99
38.15 38.57
22.78 22.49
42.27 42.48
DECO (GA) DECO (NG)
39.77 34.36
35.78 36.90
32.46 27.74
37.29 39.18
22.13 21.87
42.29 42.81
Activation Ratio (%)
As illustrated in Figure 5, the activation ratios of “SiLU” and “w/o RMS” surge rapidly during the initial training phase. While the adaptive regularization eventually pulls back this surge, Figure 6 reveals that these settings require a significantly higher regularization coefficient, which typically degrades overall performance. Conversely, NormSiLU and “w/o Mean” maintain stable activation trends, with NormSiLU achieving the lowest activation ratio. We conclude that intra-expert RMS normalization mitigates the uncontrolled growth of activation ratios, while inter-expert mean normalization further promotes sparsity level.
Small Medium Large PPL (↓) Task PPL (↓) Task PPL (↓) Task
DECO (GA) DECO (NG)
50 45 40 35 30 25 20 0
3000
6000 9000 Training Step
12000
15000
Figure 8. The trend of routed-expert activation ratio of DECO (Small) using different expert gating policies.
Moreover, analysis of the internal SiLU output magnitudes (Figure 7) reveals that “SiLU” and “w/o Mean” exhibit considerably lower magnitudes. This suggests that expert neurons (i.e., parameter columns/rows) in these settings are potentially under-utilized and less significantly activated. In contrast, inter-expert mean normalization effectively addresses this issue, ensuring more robust activation and utilization of expert neurons.
tectures: DeepSeek-V3, ReMoE, and DECO. DeepSeek-V3 is a well-performing MoE architecture using a fixed pertoken activation ratio, while ReMoE and DECO use ReLUbased routing to implement a flexible activation ratio. For each architecture, we compare non-gated MLP experts (NG) against gated MLP experts (GA). As demonstrated by Table 3, for ReLU-based routing, nongated MLP experts generally surpass gated counterparts. Conversely, for standard TopK routing (e.g., DeepSeek-V3), gated experts provide a marginal performance gain, though the difference is not significant.
4.4. Effect of Expert Gating To investigate whether expert gating significantly influences MoE performance, we conduct experiments on three archi6
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
34 32 30
34 33 DECO (Small) Dense (Small) DECO (Medium) Dense (Medium)
32 31 30 29
28
28 5.0 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 Activation Ratio (%)
32
64 96 128 160 192 224 256 Intermediate Dimension of Shared Expert
C4 Validation PPL
C4 Validation PPL
36
35
C4 Validation PPL
DECO (Small) Dense (Small) DECO (Medium) Dense (Medium)
38
35 34 33 32 31 30 29 28 27
DECO (Small) Dense (Small) DECO (Medium) Dense (Medium)
50
100
150 Granularity
200
250
Figure 9. The impact of the routed-expert ac- Figure 10. The impact of shared expert sizes Figure 11. The impact of the expert granutivation ratio on the performance of DECO on the performance of DECO (Small and larity (g = 4dh /de ) on the performance of (Small and Medium). Medium). Routed expert dimension is 64. DECO (Small and Medium).
We attribute this disparity to the training dynamics that emerge when gated experts are paired with flexible, threshold-based routing. As illustrated in Figure 8, DECO (GA) exhibits highly unstable activation trends, characterized by a drastic surge in the routed-expert activation ratio that must be aggressively counteracted by sparsity regularization. In contrast, DECO (NG) maintains a stable activation trajectory, requiring substantially less regularization and thereby preserving model performance.
Therefore, reaching target ratios beyond this intrinsic threshold is infeasible. The experimental results, illustrated in Figure 9, yield two primary observations: (1) Positive correlation with performance: DECO’s performance scales positively with the activation ratio. This aligns with findings in standard TopK MoE, where increased activation ratios correspond to higher per-token computational investment, typically resulting in superior model quality. (2) Scale-dependent comparability thresholds: The activation ratio required for DECO to achieve parity with dense models varies by scale. Specifically, the “Small” setting reaches dense-level performance at a 15% activation ratio, whereas “Medium” requires only 10%. This suggests that as DECO scales up, it may attain dense-comparable parameter efficiency at lower activation ratios. This observation is consistent with recent literature suggesting that the optimal activation ratio decreases as total parameter count increases (Zhao et al., 2025a). Further investigation on larger-scale models is required to validate this scaling trend.
Mechanistically, this divergence may stem from gradient behavior. Compared with non-gated variants, gated experts (e.g., SwiGLU) contain more multiplicative interactions that produce highly dynamic output scales, sending massive gradient signals back to the router. Because ReLU-based routing couples activation directly to the logit threshold, this gradient surge drastically destabilizes the activation ratio. Conversely, in TopK-routed architectures like DeepSeek-V3, the hard constraint of activating a fixed number of experts per token effectively masks this logit-induced instability, rendering the overall performance largely insensitive to the choice of expert gating.
5.2. Shared Expert Size We evaluate the performance of DECO across various shared-expert sizes, proportional to the intermediate dimension of the shared expert. For rigorous comparison, we hold the total parameter count approximately constant across all settings. This necessitates a trade-off between the capacity of the shared expert and the number of routed experts.
5. Effect of Key MoE Hyperparameters Compared to dense architectures, MoE models introduce unique hyperparameters that significantly influence performance, among which the activation ratio, expert granularity, and shared expert size are often considered the most important ones (Tian et al., 2025). Similarly, the performance of DECO is also sensitive to these factors, and the densecomparability of DECO has its conditions. In this section, we conduct an empirical study to characterize the impact of these three MoE-specific factors on DECO.
As illustrated in Figure 10, when the intermediate dimension of the shared expert is around 1∼2 times that of a routed expert (fixed at 64), DECO achieves comparability with the dense baseline and exhibits relative insensitivity to the size of the shared expert. However, as the shared-expert dimension increases to 3∼4 times that of the routed experts, the perplexity (PPL) degrades significantly. This performance drop is attributed to the reduced number of routed experts necessitated by the fixed parameter budget. These findings corroborates the observations in Tian et al. (2025), suggesting that an oversized shared expert is unnecessary. Instead,
5.1. Activation Ratio We evaluate DECO across a wide range of routed-expert activation ratios, from 5% to 25%. Notably, according to the red line in Figure 8, DECO exhibits an intrinsic activation ratio of approximately 28% if no regularization is applied. 7
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices Table 4. Single-GPU decoding speeds on Spec-Bench (token/sec) and average speedup ratios relative to “Baseline AR” on NVIDIA RTX 4090 (24GB) and Jetson AGX (64GB). “Ours” denotes the setting with our acceleration kernel. Device
Setting
MT.
Trans.
Summ.
QA
Math
RAG
Average
RTX 4090 24GB
Baseline AR Ours Speedup
86.63 224.28 2.59×
87.81 222.65 2.54×
86.20 218.24 2.53×
87.88 228.57 2.60×
87.83 231.88 2.64×
86.24 222.19 2.58×
87.10 224.63 2.58×
Jetson AGX 64GB
Baseline AR Ours Speedup
14.68 44.15 3.01×
14.89 44.08 2.96×
14.61 42.78 2.93×
14.91 45.29 3.04×
14.89 45.99 3.09×
14.62 43.64 2.98×
14.77 44.32 3.00×
dense-comparable. The property of activation sparsity, indicating that only a small fraction of parameters contribute largely to final outputs, also exists in dense models. Recent studies (Song et al., 2025a; Luo et al., 2024; Zhang et al., 2024) indicate that for each token, only about 30%∼40% neurons within a standard SwiGLU FFN provide non-negligible contributions. The remaining 60%∼70% of neurons “seem” to occupy computation, but actually do not have much contribution and are not considerably updated by the optimizer due to near-zero SiLU activation values. Actually, dense models can be regarded as a special form of sparse MoE, where the SiLU-activated gating projection functions as a “router”, and neurons within up-down projections serve as “experts”. Therefore, given an optimized architecture and a sufficient activation ratio, it is possible for a sparse MoE model to match dense performance.
a single shared expert with a scale comparable to that of a routed expert appears to be optimal. 5.3. Expert Granularity As illustrated in Figure 11, for “Medium” setting, we observe a monotonic improvement in perplexity (PPL) as granularity increases when g > 120. However, the “Small” setting shows reduced sensitivity to granularity, as its lowercapacity experts are potentially more difficult to capture the “long-tail” distribution of the data or develop fine-grained specialization. For both settings, when granularity is below 60, performance fluctuates near the dense baseline. By contrast, for g > 60, DECO consistently achieves dense parity. These results suggest that finer expert granularity is essential for stable and competitive performance of DECO.
6. Practical Inference Acceleration Table 5. The experimental results on FineWeb, a dataset with less heterogeneous composition.
To demonstrate the practical utility of DECO for real-world deployment, we implement an acceleration kernel tailored for DECO, which leverages sparse activation to reduce both computational overhead and memory access latency incurred by inactive routed experts. We evaluate the kernel’s efficacy on Spec-Bench (Xia et al., 2024), a comprehensive acceleration benchmark, using a single NVIDIA RTX 4090 (24GB) and Jetson AGX (64GB) as the inference devices. We establish a baseline, called “Baseline AR”, using standard autoregressive decoding without sparsitybased optimizations. Both the baseline and our kernel are implemented within FR-Spec (Zhao et al., 2025b), a highperformance CUDA-optimized inference framework, to ensure a controlled experimental environment. As detailed in Table 4, our acceleration kernel significantly outperforms “Baseline AR”, achieving an average speedup of 2.58× on RTX 4090 and 3.00× on Jetson AGX. These results confirm that DECO’s sparse activation patterns, produced by ReLUbased routing, can be effectively translated into tangible throughput gains in practical inference scenarios.
Dense DECO
Small PPL (↓) Task (↑)
Medium PPL (↓) Task (↑)
3.393 3.406
3.213 3.211
38.03 38.65
41.49 41.58
Dense-comparability may be easier to achieve under more heterogeneous training data. In previous experiments, we adopt a diverse data mixture for training, including web texts, codes, math, and many other data categories. On such heterogeneous data, DECO matches or exceeds the dense baselines across various parameter scales (Figure 3). However, this "dense-comparability" may be sensitive to the underlying data composition. As shown in Table 5, when trained on FineWeb (Penedo et al., 2024), a less heterogeneous dataset, the dense-comparability of DECO is less significant, and DECO (Small) slightly lags behind Dense (Small) in PPL. A reasonable explanation is that high-entropy, multi-domain datasets are inherently better suited for sparse MoE. In such settings, the model can effectively process domain-specific tasks by activating only a specialized subset of its parameters. More studies are needed to verify this explanation.
7. Discussion Considering the intrinsic activation sparsity of “dense” language models, it is reasonable to expect MoE to be 8
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
8. Conclusion
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. Deepseek LLM: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. URL https: //arxiv.org/pdf/2401.02954.
In this work, we propose DECO, a novel MoE architecture designed to minimize computational and storage overhead while maintaining dense-comparable performance. Under an activation ratio of 20%, DECO not only matches the performance of dense FFNs with an equivalent parameter count and training token budget, but also consistently outperforms existing MoE baselines. Our analysis demonstrates that DECO’s success stems from a series of architectural refinements, including ReLU-based routing with learnable expert-wise scaling, as well as the integration of non-gated MLP experts utilizing the NormSiLU activation. Furthermore, practical acceleration is achieved on real hardware, confirming that the sparsity of DECO can be translated into tangible efficiency gains for real-world deployment.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432–7439, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/ view/6239/6095. Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, J. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, 2025. URL https://arxiv.org/pdf/2507.11181.
Limitations
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. URL https://arxiv.org/pdf/1803.05457.pdf.
One potential limitation of this work is that we do not conduct experiments involving the supervised fine-tuning (SFT) or reinforcement learning (RL) stage. Recent studies have indicated that MoE architectures may encounter unique challenges during post-training, such as RL instability resulting from fluctuating router activations (Zheng et al., 2025). Therefore, it is reasonable to assume that DECO may have similar issues. To address this limitation, we are currently training a larger, product-level DECO model optimized for edge-device deployment. This process will involve finding potential issues during the stages of SFT and RL, and developing corresponding mitigation strategies.
Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., et al. DeepSeekMoE: Towards ultimate expert specialization in mixtureof-experts language models. CoRR, 2024. URL http: //arxiv.org/pdf/2401.06066. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/pd f/2407.21783.
Moreover, as already stated in this article, several hypotheses and empirical observations deserve more extensive verification. Specifically, it remains to be determined how the activation ratio threshold required for dense-comparability scales with model size, and how DECO’s intrinsic sparsity and performance fluctuate across diverse data distributions or inference tasks. Investigating these factors is essential to further establish the robustness and theoretical foundation of our proposed architecture.
Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. URL https://www.jmlr.org/p apers/volume23/21-0998/21-0998.pdf. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. URL https://arxiv.org/pdf/2101.00027.p df.
Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential social consequences of our work, none of which we feel must be specifically highlighted here.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. The language model evaluation harness, 07 2024. URL https://zenodo.org/records/12608602.
References Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. URL https: //arxiv.org/pdf/1607.06450.
9
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024.
datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024. URL https://proceedi ngs.neurips.cc/paper_files/paper/2024/file/3 70df50ccfdf8bde18f8f9c2d9151bda-Paper-Datas ets_and_Benchmarks_Track.pdf.
Huang, Q., An, Z., Zhuang, N., Tao, M., Zhang, C., Jin, Y., Xu, K., Chen, L., Huang, S., and Feng, Y. Harder tasks need more experts: Dynamic routing in MoE models. arXiv preprint arXiv:2403.07652, 2024. URL https: //arxiv.org/pdf/2403.07652.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text Transformer. Journal of machine learning research, 21 (140):1–67, 2020. URL https://www.jmlr.org/pap ers/volume21/20-074/20-074.pdf.
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. URL https://arxiv. org/pdf/2401.04088.
Sakaguchi, K., Le Bras, R., Bhagavatula, C., and Choi, Y. WinoGrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8732–8740, 2020. URL https://cdn.aaai.org/ojs/6399/6399-13-9 624-1-10-20200517.pdf.
Jin, P., Zhu, B., Yuan, L., and Yan, S. MoE++: Accelerating mixture-of-experts methods with zero-computation experts. arXiv preprint arXiv:2410.07348, 2024. URL https://arxiv.org/pdf/2410.07348?
Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. SocialIQA: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4463–4473, 2019. URL https://aclanthology.org/D19-1454. pdf.
Krajewski, J., Ludziejewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygóźdź, T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024. URL https://arxiv.org/pd f/2402.07871. Li, H., Lo, K. M., Wang, Z., Wang, Z., Zheng, W., Zhou, S., Zhang, X., and Jiang, D. Can mixture-of-experts surpass dense llms under strictly equal resources? arXiv preprint arXiv:2506.12119, 2025. URL https://arxiv.org/pd f/2506.12119.
Shazeer, N. GLU variants improve Transformer. arXiv preprint arXiv:2002.05202, 2020. URL https://arxiv. org/pdf/2002.05202.pdf. Song, C., Han, X., Zhang, Z., Hu, S., Shi, X., Li, K., Chen, C., Liu, Z., Li, G., Yang, T., and Sun, M. ProSparse: Introducing and enhancing intrinsic activation sparsity within large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 2626–2644, January 2025a. URL https://aclant hology.org/2025.coling-main.180.pdf.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024. URL https://arxiv.org/pdf/2412.19437. Luo, Y., Song, C., Han, X., Chen, Y., Xiao, C., Liu, Z., and Sun, M. Sparsing Law: Towards large language models with greater activation sparsity. arXiv preprint arXiv:2411.02335, 2024. URL https://arxiv.org/pd f/2411.02335.
Song, C., Zhao, W., Han, X., Xiao, C., Chen, Y., Li, Y., Liu, Z., and Sun, M. BlockFFN: Towards end-side acceleration-friendly mixture-of-experts with chunk-level activation sparsity. arXiv preprint arXiv:2507.08771, 2025b. URL https://arxiv.org/pdf/2507.08771.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.-Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525– 1534, 2016. URL https://aclanthology.org/P16-1 144.pdf.
Su, D., Kong, K., Lin, Y., Jennings, J., Norick, B., Kliegl, M., Patwary, M., Shoeybi, M., and Catanzaro, B. Nemotron-CC: Transforming common crawl into a refined long-horizon pretraining dataset. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2459– 2475, 2025. URL https://aclanthology.org/2025. acl-long.123.pdf.
Penedo, G., Kydlíček, H., Lozhkov, A., Mitchell, M., Raffel, C. A., Von Werra, L., Wolf, T., et al. The FineWeb 10
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Qiao, F., Majcher, P., Springer, P., Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS, Jan 2023. URL https://github.com/NVIDIA/cutlas s.
Zhao, G., Fu, Y., Li, S., Sun, X., Xie, R., Wang, A., Han, W., Yang, Z., Sun, W., Zhang, Y., et al. Towards a comprehensive scaling law of mixture-of-experts. arXiv preprint arXiv:2509.23678, 2025a. URL https: //arxiv.org/pdf/2509.23678. Zhao, W., Pan, T., Han, X., Zhang, Y., Sun, A., Huang, Y., Zhang, K., Zhao, W., Li, Y., Wang, J., et al. FRSpec: Accelerating large-vocabulary language models via frequency-ranked speculative sampling. arXiv preprint arXiv:2502.14856, 2025b. URL https://arxiv.org/ pdf/2502.14856.
Tian, C., Chen, K., Liu, J., Liu, Z., Zhang, Z., and Zhou, J. Towards greater leverage: Scaling laws for efficient mixture-of-experts language models. arXiv preprint arXiv:2507.17702, 2025. URL https://arxiv.org/pd f/2507.17702.
Zheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., Gao, C., Dang, K., Liu, Y., Men, R., Yang, A., et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. URL https://arxiv.org/pd f/2507.18071.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. LLaMA 2: Open foundation and finetuned chat models. arXiv preprint arXiv:2307.09288, 2023. URL https://arxiv.org/pdf/2307.09288.p df. Wang, L., Gao, H., Zhao, C., Sun, X., and Dai, D. Auxiliaryloss-free load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408.15664, 2024a. URL https: //arxiv.org/pdf/2408.15664. Wang, Z., Chen, J., and Zhu, J. ReMoE: Fully differentiable mixture-of-experts with ReLU routing. arXiv preprint arXiv:2412.14711, 2024b. URL https://arxiv.org/ pdf/2412.14711. Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Findings of the Association for Computational Linguistics ACL 2024, pp. 7655–7671, 2024. URL https://aclanthology.org/2024.find ings-acl.456.pdf. Yao, F., Cui, J., Zhang, R., Liu, L., Hao, S., Zhang, L., Dong, C., Wang, S., Gao, J., Shang, J., et al. DenseMixer: Improving moe post-training with precise router gradient. In NeurIPS 2025 Workshop on Structured Probabilistic Inference {\&} Generative Modeling, 2025. URL https: //openreview.net/pdf?id=88PI0SpVL3. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019. URL https://aclanthology.org/P19-1472.pdf. Zhang, Z., Song, Y., Yu, G., Han, X., Lin, Y., Xiao, C., Song, C., Liu, Z., Mi, Z., and Sun, M. ReLU2 wins: Discovering efficient activation functions for sparse LLMs. arXiv preprint arXiv:2402.03804, 2024. URL https://arxiv. org/pdf/2402.03804.pdf. 11
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
A. Detailed Experimental Settings Training datasets For all settings, we use the same pretraining dataset, which consists of a mixture of various data, including FineWeb (Penedo et al., 2024), Nemotron-CC (Su et al., 2025), Pile (Gao et al., 2020), Wikipedia, and many other collected raw corpora. The training data covers a wide range of categories, such as raw web texts, math, code, and articles. The data mixing ratio is carefully tuned through experiments on small-scale models. Evaluation benchmarks We evaluate the accuracy of models on various commonsense reasoning benchmarks with LMEval (Gao et al., 2024), including PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), ARC-C, ARC-E (Clark et al., 2018), WinoGrande (Sakaguchi et al., 2020), and LAMBADA (Paperno et al., 2016). Baseline adjustments Since BlockFFN is originally designed to combine activation sparsity and speculative decoding for better acceleration, it introduces a chunk-level sparsity regularization to increase the union sparsity level of consecutive tokens (Song et al., 2025b). However, we do not consider such a chunk-level issue in this work. Therefore, we replace its original regularization designs with our own adaptive sparsity regularization for fair comparison. Hyperparameters To find the optimal hyperparameters, we first conduct a grid search on small-scale models. Then, following DeepSeek LLM (Bi et al., 2024), we assume a power-law relationship of both the optimal learning rate and batch size with respect to the computation budget. Thereby, we can extrapolate the hyperparameters for large-scale models using those for small-scale models. The detailed hyperparameters of each experimental setting are shown in Table 6. All MoE baseline settings and DECO use a dense FFN at the first layer, and adopt sparse MoE for the remaining Nlayer − 1 layers. We use the WSD learning rate scheduler along the training process (Hu et al., 2024; Dubey et al., 2024), with 100 warmup steps at the beginning and 1,000 decay steps at the last stage. Table 6. The hyperparameter settings of models. ds , df f , Nlayer , and nstep denote the intermediate dimension of the shared expert, the intermediate dimension of the dense FFN layer, the total number of hidden layers, and the number of training steps, respectively. η and λinit are the coefficient multiplier and the initial value of the regularization coefficient as used in the adaptive sparsity regularization. Scale 0.11B (Small) 0.24B (Medium) 0.53B (Large) 1.18B (XLarge)
dh 768 1,024 1,280 1,792
de 64 64 64 64
ds 128 128 128 128
Ne 42 57 77 102
df f 1,920 2,560 3,360 4,480
Nlayer
nstep
16 20 27 32
lr
15,000 15,000 20,000 23,000
1.175e − 3 9.3e − 4 7.8e − 4 6.54e − 4
batch_size 5
2.62 × 10 5.24 × 105 1.05 × 106 2.10 × 106
η
λinit
1.002 1.002 1.001 1.001
1e − 8 1e − 8 1e − 8 1e − 8
B. Impact of the Scaling Factor Initialization Value In this section, we conduct experiments on DECO when its learnable expert-wise scaling factors are initialized with different values. The results are shown in Table 7. The performance of DECO does not change monotonically with the initialization value. Empirically, the best performance is achieved when the scaling factors are initialized around 0.1∼0.25. In our experiments in Section 4, we choose the initialization value of 0.1 for all DECO settings. Table 7. The performance of DECO (Small) when the learnable expert-wise scaling factors are initialized with different values. Init Value
0.01
0.025
0.05
0.1
0.25
0.5
1.0
PPL (↓) Task (↑)
34.71 36.86
35.66 36.57
34.61 36.64
34.36 36.90
34.26 36.72
35.20 36.57
34.74 36.54
C. Theoretical Support for NormSiLU To theoretically demonstrate the rationality of NormSiLU, we consider the post-activation output of all experts. Let Wup ∈ Rdh ×(Ne de ) denote the concatenated up-projection weights of Ne experts, and let x ∈ Rdh be the input hidden state. The post-activation intermediate state is computed as: y = SiLU(Norm(z)), 12
T z = Wup x ∈ RNe de .
(4)
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
For simplicity, we first assume that the operator “Norm” is implemented as a vanilla layer normalization (Ba et al., 2016) across the entire concatenated expert dimension. Let g = ∇y L be the upstream gradient from the language modeling loss L. The gradient with respect to the weights Wup is formulated as: ∇Wup L = x(∇z L)T = x(JTnorm Dsilu g)T , z0 u = Norm(z) = , z0 = z − z̄, ||z0 ||rms
(5)
where Dsilu = diag(σ(u) + u ⊙ σ(u) ⊙ (1 − σ(u))) is the diagonal Jacobian matrix of the SiLU activation, σ is the sigmoid function, and z̄ denotes the mean of z. Letting n = Ne de be the total dimension of z and 1 be the all-ones vector, the Jacobian matrix of the layer normalization, Jnorm = ∂u ∂z , is expanded as: Jnorm =
1 ||z0 ||rms
1 z0 zT0 . In − 11T − n n||z0 ||2rms
(6)
Since the matrix inside the parentheses is an affine shift of a projection matrix, its spectral norm ||Jnorm ||2 is strictly bounded by 1/||z0 ||rms . Furthermore, assuming the elements of Wup follow a zero-centered i.i.d. normal distribution, we can statistically approximate the RMS norm as ||z0 ||rms ≈ √d1 n ||Wup ||F · ||x||2 . h
Applying the sub-multiplicative property of matrix norms, the Frobenius norm of the gradient is bounded by: ||∇Wup L||F ≤ ||x||2 · ||Jnorm ||2 · ||Dsilu ||2 · ||g||2 ≤ ||g||2 · O(||Wup ||−1 F ).
(7)
Therefore, as long as Wup is initialized with a proper Frobenius norm, this normalization theoretically guarantees that the gradient scale remains bounded and invariant to the input magnitude, preventing gradient explosion. However, the above paradigm possesses a critical systems-level flaw: computing a global layer normalization requires the explicit materialization of z across all experts. This inherently violates the core sparsity principle of MoE, as it forces the computation of inactive experts. To resolve this bottleneck, our final implementation of NormSiLU decouples the operation into a dual-stage mechanism: an inter-expert mean normalization and an intra-expert RMS normalization. During inference, the global inter-expert averaging T T operator mathematically reduces to: Avg(Wup x) = w̄up x, where w̄up ∈ Rdh ×de is the fixed average up-projection weight. By contrast, the RMS normalization is strategically restricted to the intra-expert dimension (de ) and applied only to activated experts. This dual-stage design preserves the theoretical gradient-bounding stability while strictly adhering to the computational constraints of sparse inference.
D. Evaluation Results on individual benchmarks In this section, we provide the evaluation results of DECO and baselines on each individual benchmark. The results of “Small”, “Medium”, “Large”, and “XLarge” settings are shown in Table 8, Table 9, Table 10, and Table 11, respectively. Table 8. The evaluation scores of “Small” settings on individual benchmarks. PIQA
SIQA
HellaSwag
ARC-E
ARC-C
WinoGrande
LAMBADA
Avg.
TopP DeepSeek-V3 (GA) DeepSeek-V3 (NG) ReMoE (GA) ReMoE (NG) BlockFFN DECO (GA)
60.17 60.12 60.66 60.34 59.74 60.83 58.22
35.21 35.93 36.63 36.59 35.16 36.28 35.93
27.67 27.85 27.76 27.72 27.96 27.62 27.65
42.38 44.15 45.76 46.34 46.09 44.87 42.51
18.86 19.45 20.05 18.43 19.54 19.45 19.03
50.91 50.75 50.43 51.22 49.96 50.12 51.30
7.74 15.89 15.85 15.60 17.76 16.55 15.84
34.71 36.31 36.73 36.61 36.60 36.53 35.78
Dense DECO (NG)
60.77 60.94
35.57 36.85
28.03 27.96
45.88 45.84
20.22 20.22
50.59 50.67
16.51 15.80
36.80 36.90
13
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
Table 9. The evaluation scores of “Medium” settings on individual benchmarks. PIQA
SIQA
HellaSwag
ARC-E
ARC-C
WinoGrande
LAMBADA
Avg.
TopP DeepSeek-V3 (GA) DeepSeek-V3 (NG) ReMoE (GA) ReMoE (NG) BlockFFN DECO (GA)
61.37 62.79 63.28 62.62 63.22 63.38 61.86
36.49 37.87 37.67 37.67 36.64 35.82 37.10
28.57 29.61 29.64 29.10 29.40 29.61 28.44
45.75 48.53 49.07 48.65 49.71 48.82 47.18
19.03 20.90 20.65 20.05 19.71 20.99 19.37
50.51 52.17 49.25 50.67 50.83 50.51 49.64
12.63 19.84 19.76 18.28 20.45 19.95 17.43
36.34 38.82 38.47 38.15 38.57 38.44 37.29
Dense DECO (NG)
63.00 64.36
36.90 36.18
29.57 29.47
51.47 51.98
22.01 20.73
50.51 52.09
19.62 19.48
39.01 39.18
Table 10. The evaluation scores of “Large” settings on individual benchmarks. PIQA
SIQA
HellaSwag
ARC-E
ARC-C
WinoGrande
LAMBADA
Avg.
TopP DeepSeek-V3 (GA) DeepSeek-V3 (NG) ReMoE (GA) ReMoE (NG) BlockFFN DECO (GA)
64.53 66.32 65.94 66.05 66.92 66.10 66.14
36.44 38.38 37.87 38.69 38.79 38.59 38.74
31.42 32.71 32.75 32.08 32.66 32.51 32.69
50.55 55.18 54.67 55.26 55.93 56.36 56.41
22.53 24.91 23.38 24.32 24.57 24.15 23.72
51.07 51.07 52.49 52.33 51.41 51.38 51.78
14.13 28.06 27.03 27.19 27.10 27.30 26.57
38.67 42.38 42.02 42.27 42.48 42.34 42.29
Dense DECO (NG)
66.87 65.94
37.77 39.61
32.92 32.78
56.27 57.11
24.74 25.43
51.70 53.28
29.34 25.50
42.80 42.81
Table 11. The evaluation scores of “XLarge” settings on individual benchmarks. PIQA
SIQA
HellaSwag
ARC-E
ARC-C
WinoGrande
LAMBADA
Avg.
TopP DeepSeek-V3 (GA) ReMoE (NG) BlockFFN
66.16 69.53 68.93 70.02
37.36 39.92 40.07 40.28
34.10 36.65 37.43 36.81
57.32 62.21 61.20 61.91
22.87 28.41 28.07 28.50
49.41 53.28 53.43 55.80
20.49 34.52 35.46 34.60
41.10 46.36 46.37 46.85
Dense DECO (NG)
70.46 70.24
40.69 40.89
37.33 37.42
63.38 62.96
28.58 29.69
52.72 54.93
35.73 35.53
46.98 47.38
14