Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
arXiv:2605.23893v1 [cs.LG] 22 May 2026
Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong, Yifan Gong, Jianming Zhang, Yan Kang Adobe Research {hongwup, dibua, yxiong, yifang, jianmzha, yankang}@adobe.com
Abstract We propose Complete-muE, a framework which targets hyperparameter transfer across dense FFN and any Mixture-of-Experts (MoE) setups in transformer blocks. Existing tools such as µP (requires fixed architectue) or SDE (requires fixed perstep token count) cannot directly solve the hyperparameter transfer problem in MoE setups because Dense to MoE transfer or MoE total experts scaling changes both architecture and tokens per expert. Complete-muE solves this challenge with a two-bridge system: Bridge I maps between dense FFN and Dense MoE by activewidth µP with a normalized router scale. Bridge II maps between Dense MoE and sparse MoE by activated-expert scaling, where the first-order SDE LR/WD correction cancels while a bounded residual σ0 shift remains. The resulting transfer rule, which we term as Complete muE, covers changes in activated experts, total capacity, granularity, and shared/group-balanced hybrids for MoE models as well as network width/depth, batch size, and duration changes for general Transformer models. Extensive language model and diffusion model pretraining experiments confirm that complete-muE yields relatively stable hyperparameter optima across model architectures and parameter counts—with only minor drift consistent with the non-strict SDE behavior of Bridge II. In practice this drift is small enough that hyperparameters tuned on a single dense reference transfer near-optimally to all MoE configurations—tune dense once, transfer to all is the practical recipe at the core of Complete-muE. This enables MoE models to achieve accelerated convergence speedup over dense models when scaling model capacity without costly hyperparameter search.
1
Introduction
The goal of this paper is to transfer tuned initialization and AdamW hyperparameters across dense FFN and Mixture-of-Experts (MoE) transformer blocks. MoE exposes scaling axes absent from dense FFNs—per-expert width h, activated experts a, total experts N , shared experts, and group-balanced routing [1, 2, 3, 4, 5, 6, 7, 8]. Changing these axes also changes expert data exposure: with balanced routing, each expert processes roughly Ba/N tokens (B is per-iteration batch size) per step and T Ba/N (T is total training iterations) tokens over training. Dense-to-sparse transfer and total-expert scaling therefore couple architecture and workload transfer. Existing tools cover only one side. µP, originally developed for dense models, connects dense FFNs to Dense MoE through active width but cannot deal with changes in per-expert token batch size [9, 10]; SDE rules transfer across token batch size for a fixed model architecture but cannot handle architecture changes [11, 12, 13]. Complete-muE solves parameter transfer of MoE, which needs to simultaneously deal with both batch token count and architecture changes, with two bridges: (i) dense FFN ↔ Dense MoE via active-width µP and route scale ra = a; and (ii) Dense MoE ↔ sparse MoE via activated-expert scaling, where expert-side batch and duration ratios match and the first-order SDE LR/WD correction cancels, while a residual σ0 shift remains and produces bounded, minor Preprint.
hyperparameter drift in practice. Capacity, granularity, shared experts, group-balanced routing, and standard width/depth/batch/duration changes then follow by composition. We extensively evaluated Complete muE on language modeling (LM)/diffusion (DF) tasks, and find Complete muE enables relatively stable hyperparameter transfer across all MoE architecture/scale/training duration combinations, with only minor drift consistent with Bridge II’s non-strict SDE behavior. In practice this drift is small enough that hyperparameters tuned on a single dense reference transfer near-optimally to all MoE configurations—tune dense once, transfer to all MoE settings is the practical recipe at the core of Complete-muE. Both controlled small-scale axis sweeps and large-scale multimodal/LM runs directly verify this recipe: a single dense calibration delivers consistent MoE gains across MoE axes and across modalities (256P/512P images, 240P key frames, 240P 5s videos, LM). We also benchmark MoE granularity vs capacity to show the real scaling trade-offs, and observe that capacity scaling under moderate granularity scaling is more beneficial. Our large scale MoE runs with Complete-muE enabled reach roughly 4.5× speedup for 240P 5s video diffusion model and 5.3×–5.5× LLM convergence speedups with 100k training iterations. Our contributions are as follows: (1) we identify the key bottlenecks of dense-sparse MoE and MoE capacity scaling transfer cases; (2) to bridge the gap, we derive the two-bridge activated experts transfer rule through dense-dense MoE, and dense MoE-sparse MoE transfers; (3) we further compose those transfer rules to Complete muE, a framework which generalize to transfer across any MoE setups; and (4) we validate Complete muE in controlled small scale sweep and larger-scale LLM/multimodal experiments, and find Complete muE yields relatively stable hyperparameter optima across all MoE setups, with mild drift consistent with the non-strict SDE behavior of Bridge II. With Complete muE enabled, large scale MoE training can achieve significant convergence speedup with minimum tuning overhead.
2
Related Work and Theoretical Motivation
Complete-muE is motivated by a mismatch between two lines of scaling work. MoE layers add active-width and routing-structure choices, while sparse routing changes the token batch size seen by each expert. Existing parameterizations handle the former for dense model-size changes, and SDE rules handle the latter for fixed parameterized models; neither alone covers changes that alter both the routed layer and expert-side data exposure. Sparse MoE architectures and stability. Sparse MoE has progressed from sparsely gated layers to switch/top-k transformers with improved routing, load balancing, specialization, and system support [1, 2, 3, 14]. Recent large systems add fine-grained experts, shared experts, group-balanced routing, and optimized sparse training [4, 5, 6, 7, 8]; related diffusion/image MoE systems show similar benefits for generative models [15, 16]. These works motivate our target design space but do not provide deterministic transfer rules for initialization, learning rate, weight decay, and parameter groups across h, a, N , and hybrid layouts. Hyperparameter transfer and µ-parameterization. Tensor Programs and µP provide widthwise zero-shot hyperparameter transfer by preserving feature-learning dynamics [9]. Later work extends dense transfer to depth, practical transformer training, batch/duration effects, and diffusion transformers [17, 18, 12, 13, 19]. We use this layer-level matching as the first ingredient: dense FFN and Dense MoE can be related through active width. However, plain µP has no explicit variable for the routed-token workload of a sparsely used expert, so it does not explain optimizer stability when a or N changes expert data exposure. MoE-specific transfer. Recent MoE parameterization studies preserve learning-rate ranges under expert-count sweeps, study sparse MoE transformers across width/depth/expert axes, or analyze granularity and RMS-preserving routing [10, 20, 21]. Complete-muE is complementary: it stays in the standard AdamW regime and seeks one compositional rule for activated experts, total capacity, fixeddensity granularity, shared experts, and group-balanced routing. The empirical stability observed in expert-count sweeps suggests that capacity scaling is not a separate primitive hyperparameter-transfer rule in N , but can be explained by composing active-width transfer with a sparsification step. Expert workload and route scale. SDE-based optimizer transfer emphasizes the joint dependence on effective batch size and training horizon [11, 13]. For sparse MoE, the expert-side quantities 2
NOT DIRECTLY TRANSFERRABLE
× N blocks
× N blocks
FFN
TRANSFERRABLE (Complete-μE)
× N blocks
Sparse MoE
× N blocks
FFN
FFN
Tokens
Tensor program (μP)
Router
Self-attention
Self-attention
Tokens
Tokens
transfer across dense model architecture when training duration and batch size are fixed.
Router
Transferrable
Training as SDE
Sparse MoE
SDE
Router Not directly transferrable
μP
μP
FFN
Self-attention
× N blocks
Dense MoE
Transferrable Self-attention
Self-attention
Tokens
Tokens
transfer across batch size and training duration when the model architecture is fixed.
Figure 1: Complete-muE transfers across dense FFN, Dense MoE, and sparse MoE Table 1: How Eq. (1) specializes to dense FFN, Dense MoE, and sparse MoE. Case
Choice of gi (x) in Eq. (1)
Dense FFN
One always-active dense block (M = 1), g1 (x) = 1 All experts are active (a = N ), PN gi (x) = rN πi (x), i=1 πi (x) = 1 Top-a routed experts, gi (x) = ra πi (x)1{i ∈ A(x)}, |A(x)| = a
Dense MoE Sparse MoE
Active width Hact
Route scale
H
1
Nh
rN = N
ah
ra = a
scale as Bexp ∝ Ba/N and Dexp ∝ T Ba/N , so both must be tracked. At fixed (N, h, B, T ), changing a scales these ratios together; after active-width matching, the first-order raw LR/WD SDE multiplier cancels. Separately, normalized routing requires an explicit route scale proportional to the activated-expert count to preserve the routed-branch update, consistent with routed/shared-branch scaling used in recent systems [5, 7, 8]. These two observations define the bridges developed next.
3
Method
Complete-muE turns FFN/MoE tuning into a dense-proxy calibration problem: scan a reference dense FFN once, then transfer the output multiplier, down-projection initialization, normalized route scale, and optimizer multipliers to the target FFN/MoE layout. The method first writes dense, routed, sparse, and hybrid blocks in one notation, then uses two bridges—active-width matching and expert-workload bookkeeping—to cover the full MoE design space. 3.1
Problem statement and unified FFN/MoE formulation
Let d be the residual width. A dense FFN has hidden width H; an MoE layer has N experts, per-expert width h, and a activated experts, with routed activePwidth Ha = ah. PFor hybrid blocks with dense/shared branches D and routed groups G, Htot = m∈D Hm + a g∈G hg . We write ρd = d/d⋆ , ρL = L/L⋆ , ρB = B/B⋆ , and ρD = D/D⋆ with D = T B; Appendix A gives the full notation. All FFN/MoE cases are represented as y(x) = A(Hact )
M X
gi (x) oi (x),
(i)
oi (x) = Wdown ui (x),
(1)
i=1
where Hact is the hidden width active on Ptoken x. In MoE blocks, routing weights are normalized over the active experts, πi (x) ≥ 0 and i∈A(x) πi (x) = 1. Table 1 instantiates Eq. (1) for dense FFN, Dense MoE, and sparse MoE. Throughout this work we employ token-choice (top-k) routing, in which each token independently selects its a highest-scoring experts. Compared with expert-choice routing—which assigns a fixed quota of tokens per expert but requires access to the full token sequence and thereby leaks future-token information in autoregressive language models, causing a train–inference mismatch [22]—tokenchoice routing is causally consistent at inference and applies uniformly across both language and 3
visual modalities. Importantly, token-choice routing fixes the active width Ha = ah deterministically for every token, which makes the µP update-size matching in the bridges below exact: the per-token functional update scales are controlled by the fixed active set size a, rather than by a stochastic token-assignment process. The transfer problem is to reuse a dense-FFN scan rather than retune every MoE target. Tensors whose fan-in is controlled by the residual width d—the FFN up/gate projections and router readout—follow ordinary backbone-width µP. Complete-muE adds rules only for the FFN/MoE output branch and normalized routed sums, using the AdamW gradient-magnitude-normalized regime in which positive scalar rescalings of raw gradients do not change the leading preconditioned update direction. 3.2
Existing gaps in hyperparameter transfer for MoE
MoE design varies along h, a, and N , and modern layers may mix routed groups with shared experts. A useful dense-to-MoE rule must therefore work compositionally: the same dense-FFN scan should transfer when one axis changes or when several are combined. Neither existing tool provides this guarantee alone. µP matches parameter updates for model-size changes and gives the correct dense FFN ↔ Dense MoE transfer, but sparse routing changes expertside token batch size: each expert processes roughly Ba/N tokens per step and T Ba/N tokens over training. Standard SDE transfer can adjust optimizer hyperparameters for batch and horizon changes, but it assumes a fixed parameterized model and therefore does not fit into transfer problem of dense FFN, Dense MoE, and sparse MoE architectures directly [11, 12, 13]. Total-expert scaling has the same issue: changing N at fixed (a, h) changes both routed capacity and per-expert workload. Prior µP-matched expert-count sweeps nevertheless show only mild optimizer drift [10, 20], suggesting that MoE capacity transfer might be feasible. 3.3
Two bridges for compositional FFN/MoE transfer
The missing step is to avoid treating dense-to-sparse MoE transfer, or any other MoE transfer problem, as one direct jump. We instead connect a dense-FFN reference to any target FFN/MoE layout through two better-grounded bridge cases. Bridge I matches a dense FFN to a Dense MoE at the same active width, introducing the output multiplier, output initialization rule, and normalized-route correction. Bridge II then moves from Dense MoE to sparse MoE by changing the number of activated experts while keeps the overall parameterized architecture unchanged; after the layer update size are matched by µP rule, the remaining change is an expert-side token batch size changes that can be analyzed with SDE bookkeeping. These two bridges are the only primitive MoE-specific ingredients; all capacity, granularity, shared-expert, and group-balanced-routing rules in the next subsection are compositions of them. 3.3.1
Bridge I: dense FFN ↔ Dense MoE via active-width µP
Bridge I matches a dense FFN with active width H to a Dense MoE with the same active width. At fixed backbone width d, matching forward variance and the AdamW one-step functional update to the unit-expansion dense companion gives 1/2 d H (1) (1) A(H) = , σdown (d, H) = σdown (d), ηdown (d, H) = ηdown (d). (2) H d Backbone-width changes compose with ordinary µP factors; Appendix A.1 gives the derivation. Dense-MoE factorization and normalized route scale. When the same active width is factored as H = ah with a = N active experts, Eq. (2) still applies. Normalized routing, however, averages expert outputs and would shrink the update by 1/a, so we use ra = a,
(3)
with rN = N for Dense MoE. The optional forward-variance correction Fa,N is derived to Appendix A.2, and normally Fa,N can be set to 1 for sigmoid routing. Thus dense FFN ↔ Dense MoE transfer reduces to active-width µP plus route-scale correction; sparse MoE still needs Bridge II because it changes each expert’s token batch size. 4
3.3.2
Bridge II: Dense MoE ↔ sparse MoE via active width and expert-side SDE
The second bridge handles transfer across the number of activated experts. A Dense MoE is the special case a = N ; a sparse MoE uses a < N . Complete-muE resolves this sparse transfer in two steps. First, it matches the routed-FFN layer scales through the active width Ha = ah. Second, it analyzes how changing a perturbs the expert-side stochastic training trajectory. Layer-level transfer across activated experts. With total experts N and per-expert width h fixed, Complete-muE applies the dense active-width rule to Ha and retains the normalized-router correction: Aa =
d , ah
ra = a,
σdown (d, a) =
ah d
1/2
(1)
(1)
σdown (d),
ηdown (d, a) = ηdown (d).
(4) Thus a sparse MoE is matched to the dense companion through its active width, not through its total expert count. If d also changes, the same ordinary backbone-width factors are multiplied on top. Appendix A.2 shows that Eq. (4) matches the one-step MoE layer update size and the forward scale up to the bounded factor Fa,N , where Fa,N can be set to 1 for most cases. Expert-side SDE and cancellation across activated experts. The remaining effect is stochastic training process. Under approximate load balancing, one expert receives Bexp (a) ≈ B
a , N
Dexp (a) ≈ T B
a , N
(5)
and therefore σexp (a) ∝ Bexp (a)−1/2 . For a change a → a′ at fixed global batch B and fixed exp ′ optimizer steps T , the expert-side batch and duration ratios are identical, ρexp B = ρD = a /a— exp unlike a pure p global batch increase, where only ρB would change and the exact square-root rule η ′ = ρexp B η applies. Following the approximation philosophy of Complete(d)P [13], we hypothesize that after the layer-level matching of Eq. (4), the combined expert-side correction p exp exp exp η ′ ≈ η ρexp /ρ B D captures the dominant LR/WD effect. Since ρB = ρD , the dense-style SDE correction cancels: s s η′ ≈ η
ρexp B = η, ρexp D
λ′ ≈ λ
ρexp B = λ. ρexp D
(6)
p What does shift is the expert-side signal-to-noise parameter: σ0 (a′ ) = σ0 (a)/ ρexp B . Reducing sparsity (larger a′ ) lowers σ0 and thereby improves expert-side SNR at fixed optimization horizon HSDE = T η 2 —this is why denser MoEs can reach lower attainable loss even when raw hyperparameters are unchanged (see Appendix A.3). This is the bridge result underlying Complete-muE. Under the approximate load-balancing assumptions above, after the layer-level reparameterization in Eq. (4), changing the number of activated experts does not require an additional first-order raw LR/WD correction. p Activated-expert transfer is therefore not a strict SDE invariance: the residual σ0 (a′ ) = σ0 (a)/ ρexp B shift is not absorbed by the η, λ correction, so some mild hyperparameter drift across a is expected. Figure 2 confirms this picture empirically: the loss curves versus learning rate, weight decay, and initialization standard deviation remain relatively stable across activated-expert counts—with only minor drift, consistent with the non-strict SDE transfer rather than with exact invariance—while larger a consistently reaches lower attainable loss, as predicted by the improved expert-side SNR. This gives the second bridge, which we compose with the Dense FFN ↔ Dense MoE bridge to handle the remaining MoE variants. Appendix A.3 gives the full SDE derivation, including the imbalanced-routing extension. 3.4
Applications of Complete-muE: scaling across MoE settings
The two bridges connect the dense reference to the remaining MoE settings by composition. The organizing insight is that total-expert and granularity changes are not new primitive rules—they are derivable by composing the dense-width and activated-expert transfer rules established above, with the active width Ha = ah as the single governing quantity in every case. In each case, the main-text rule is to match the target active width, apply the route scale on normalized routed groups, and rely on Appendix A.4 for the detailed cancellation arguments. 5
0.470
1e 3e 6 1e 6 3e 5 1e 5 3e 4 1e 4 3e 3 1e 3 3e 2 1e 2 3e 1 1 1e 0
1
0.480
1e 0
2
0.490
DF weight-decay scan (25k, 256) Model ffn_1 64e2a 64e4a 64e8a 64e16a 64e32a 64e64a
weight_decay
(b) Weight decay sweep 11.0 LM Activated Experts Initial Loss (25k, 128) 10.6 10.4 10.2 10.0
init_std
9.800
Model ffn_1 64e8a 64e16a ffn_8 64e32a 64e2a 64e64a 64e4a 20 40 60 80 training step
(c) Init std sweep
Train loss
10.8 Train loss
DF init-std scan (25k, 256) Model ffn_1 64e2a 64e4a 64e8a 64e16a 64e32a 64e64a
1e 3e 6 6 1e 3e 5 5 1e 3e 4 4 1e 3e 3 3 1e 3e 2 2 1e 3e 1 1
Train loss
1
4e
init_std
4e 2 1e 1
2
4e 3 1e
3
4
4e 4 1e
Val loss
1e
3.7
0.505 0.500 0.495 0.490 0.485 0.480 0.475 0.470
0.500
w_decay
(a) Learning rate sweep LM MoE activated experts init-std sweep (25K, 128) 4.2 Model ffn_1 4.1 64e2a 64e4a 4 64e8a 3.9 64e16a 64e32a 3.8 64e64a
1e 1 2e 1 4e
2
4e
3
1e
3
Train loss
3.80
4e
Val loss
0.510
1e
e 3
e 3
LM MoE activated experts w_decay sweep (25K, 128) 4.10 Model 4.05 ffn_1 64e2a 4.00 64e4a 64e8a 3.95 64e16a 3.90 64e32a 64e64a 3.85
6.4
4.5
e 3
e 3
3.2
e 3
8e
lr
2.3
0.460
4
0.465
1.6
0.470
e 3
Model ffn_1 ffn_8 64e2a 64e4a 64e8a 64e16a 64e32a 64e64a
0.475
1.1
Train loss
0.480
1
2
0.485DF Sigmoid MoE activated experts (100k, 256)
1e
2
4e
3
lr
1e
3
4e
4
1e
4
4e
5
4e
1e
Val loss
4.8 LM MoE activated experts LR sweep (25K, 128) Model 4.6 ffn_1 ffn_8 4.4 64e2a 4.2 64e4a 64e8a 4 64e16a 64e32a 3.8 64e64a 3.6
1e2
1.90 DF Activated Experts Initial Loss (100k, 256) 1.85 1.80 1.75 1.70 ffn_1 64e8a 1.65 64e16a ffn_8 64e32a 64e2a 1.60 64e64a 64e4a 1.550 1e2 20 40 60 80 training step
(d) Initial training loss
Figure 2: Complete muE transfer across activated experts for both language-model (LM) and diffusion-model (DF) Total expert count (capacity scaling). For (N, a, h) → (N ′ , a, h), introduce Dense-MoE companions of widths N h and N ′ h, then sparsify the target companion back to a activated experts. This is an explicit two-step composition: (1) Dense-width step—scale the Dense-MoE companion from total width N h to N ′ h via Bridge I; (2) Reverse-sparsity step—apply Bridge II to restore a activated experts. The dense-width and sparsification factors cancel, leaving the same sparse-layer rule d Aa = , ah
ra = a,
σdown (d, a) =
ah d
1/2
(1)
σdown (d),
up to the bounded routing factor Fa,N . Because the reverse-sparsity step inherits Bridge II’s nonstrict SDE behavior—σ0 shifts with the per-expert workload and is not absorbed by the first-order η, λ correction—some mild hyperparameter drift across N is expected. Figure 3a confirms this empirically: the optimal loss region remains relatively stable across capacity settings, with only minor drift consistent with the non-strict SDE transfer. Increasing N still changes the attainable loss through a capacity–noise trade-off: total specialization capacity grows, p while each expert receives Bexp , Dexp ∝ a/N tokens, worsening expert-side SNR (σexp ∝ N/(Ba)) even as the raw transferred AdamW hyperparameters remain unchanged. Granularity and fixed-density scaling. If the routing density s = a/N is fixed, the sparse active width Ha = ah = sN h has the same ratio as the Dense-MoE companion width. Thus fixed-density granularity reduces to Dense-MoE width transfer followed by the same sparse output rule, with route scale ra = a on the source and ra′ = a′ on the target. Since each expert sees Bs tokens per step and exp T Bs tokens over training—independent of the specific partition (N, h), so ρexp B = ρD = 1 across source and target—no extra first-order SDE correction is introduced. 3.4.1
Shared, group-balanced-routing, and hybrid blocks
The rule follows three steps: (1) always-active dense/shared branches contribute active width with no route scale; (2) routed MoE groups are selected P using group-balanced routing, sharing one global route scale r = a and global normalization g,e πg,e (x) = 1; (3) one common FFN-output ABC P P parametrization is applied jointly to the total active width Htot = m∈D Hm + a g∈G hg . For hybrid blocks, Complete-muE uses one active-width multiplier for the total active width and applies route scale only to routed groups: Ng XX d X y(x) = om (x) + a πg,e (x) o(e) (7) g (x) . Htot e=1 g∈G
m∈D
6
Table 2: Complete-muE layer-level rules. Case
Multiplier A
What is transferred
Backbone width Backbone width
Dense FFN / gated MLP Dense MoE Activated experts Capacity / total experts Granularity (fixed s = a/N ) Shared / group-balanced / hybrid Shared / group-balanced / hybrid
Route R
Init std
LR
−1/2 ρd −1/2 ρd
ρ−1 d ρ−1 d
−1/2 1/2 ρH
ρ−1 d
−1/2 1/2
ρ−1 d ρ−1 d
Router readout Wgate , ρd = d/d⋆ FFN/MoE up & gate projections Wup , Wv , Wg (including per-expert copies in dense and sparse MoE), ρd = d/d⋆ Output projection Wdown , ρH = H/d
– –
– –
ρ−1 H
1
ρd
All experts active, a = N , ρH = N h/d Fixed (N, h), active width ρHa = ah/d
ρ−1 H ρ−1 Ha
N a
ρ−1 Ha
ρd ρH −1/2 1/2 ρd ρHa
a
ρd
ρ−1 Ha
a
ρd
−1/2 1/2 ρHa
ρ−1 d
ρ−1 Htot
1
ρd
−1/2 1/2 ρHtot
ρ−1 d
ρ−1 Htot
a
ρd
−1/2 1/2 ρHtot
ρ−1 d
Fixed (a, h); compose Dense-MoE companion transfer and sparsification back to a Same ratio as Dense-MoE companion transfer, but final sparse active width ρHa = ah/d Dense/shared branch, ρHtot = Htot /d
Routed experts (group-balanced selection), ρHtot = Htot /d
For one shared branch and one routed group, " # N X d (e) y(x) = osh (x) + a πe (x) o (x) , Htot e=1
−1/2 1/2 ρHa
Htot = Hsh + ah.
ρ−1 d
(8)
Equivalently, shared, group-balanced-routing, and general hybrid blocks are treated as one expanded FFN of width Htot , with one global route scale r = a on all selected routed experts; the special cases of shared experts (Eq. (8)), grouped MoE without shared branches (D = ∅, multiple routed groups), and shared-plus-grouped hybrids all follow from this general rule. The derivation is in Appendix A.5. 3.5
Operational recipe and global optimizer composition
Complete-muE’s workflow begins from a single dense-FFN reference scan—tune once at the chosen backbone width, batch size, and training duration—and then maps to any target FFN/MoE layout through two composable tables applied in sequence. Table 2: layer-level rule. For any targetPFFN/MoE block, P (1) identify the active width: H for dense FFN, ah for sparse MoE, or Htot = m∈D Hm + a g∈G hg for hybrid; (2) read the output multiplier A, route scale R, and down-projection initialization σdown from the corresponding row; (3) apply route scale only to normalized routed sums—tensors controlled by backbone width d (FFN up/gate, router readout) follow ordinary µP and need no additional rule. All MoE variants—activatedexpert, total-expert, fixed-density granularity, shared-expert, and group-balanced-routing—map to the same active-width row because their layer-level and expert-side SDE effects cancel compositionally, as established in the preceding subsections. Table 3: global optimizer factors. Two cases govern how the global AdamW multipliers interact with the layer-level rule. Case A—global schedule change at fixed architecture: when global batch or exp exp duration pchanges, every routed expert sees the same proportional change (ρB = ρB , ρD = ρD ), so the ρB /ρD factor applies uniformly to all parameters. Within Case A: when total tokens are e HSDE are simultaneously fixed (ρD = 1, batch√ up / steps down), all three global SDE objects σ0 , λ, preserved and η ′ = ρB η is exact; when training steps are fixed instead (ρD =√κB = ρB , batch up / token budget grows), the horizon is preserved with η ′ ≈ η but σ0 shifts by 1/ κB , making the transfer approximate—empirically (Figure 5a) the optimal LR region remains relatively stable across 7
batch sizes, with only minor drift consistent with the σ0 shift. Case B—MoE architectural change at fixed schedule: when expert count, routing density, or granularity changes at fixed global batch and exp duration, the expert-side ratios cancel (ρexp B = ρD ), so Table 2 already absorbs all such changes—no additional LR/WD multiplier is needed. Appendix A.6 gives the full SDE derivation of both cases. Across all three σ0 -shift situations (Bridge II activated experts, capacity composition, and Case A fixed-iteration batch transfer), the empirical drift is small enough that dense-tuned hyperparameters give near-optimal performance throughout, validating the tune dense once, transfer to all MoE settings recipe that this section operationalizes. Table 3: Remaining global AdamW transfer used together with Table 2. Axis
Ratio
Layer multiplier
η, λ
AdamW ϵ
1 − β1,2
Depth
ρL
residual branch ρ−1 L
1
1
1
Batch size Token budget / duration
ρB ρD
– –
1/2
ρB 1/2 ρD
Combined batch+duration rule: η, λ ∝
4
ρB −1/2 ρD p
ρB /ρD , AdamW ϵ ∝
−1/2
p
ρB ρ−1 D
Setting / comment use standard CompleteP depth-sensitive parameter groups [12] fixed D; T 7→ T /ρB fixed B; T 7→ ρD T
ρD /ρB , 1 − β1,2 ∝ ρB /ρD .
Experiments
We evaluate Complete-muE along three axes: controlled LM and diffusion-transformer proxy sweeps, a single-H100 latency benchmark for MoE capacity/granularity scaling, and larger multimodal diffusion and language-model runs. Because many minima are broad, we focus on alignment of the low-loss region rather than the exact best learning rate in each sweep. 4.1
Experimental setup and stable proxy-training recipe
Model families. The LM proxy uses the GPT-NeoX-20B tokenizer [23] with sequence length 2048; the default configuration has width d = 128, 32 layers, batch size 128, and 25k optimizer steps, with 12.5k/6.25k runs for duration and batch size studies. The diffusion proxy is a latent-diffusion transformer at 512 × 512 resolution with 16× VAE compression, Qwen3-VL text conditioning [24], width d = 128, batch size 256, and 100k steps, with 25k/50k/200k steps runs for duration and batch size studies. The diffusion model family uses flow-matching linear interpolant [25] and is trained for velocity prediction. Both families use gated attention [26], SwiGLU MLPs [27], and AdamW with β1 = β2 [28, 29]; diffusion also uses AdaLN timestep conditioning [30]. Schedules and reporting. We use a warmup-stable-decay schedule [31, 32, 33] with 1k warmup for most runs. Decay occupies 20% of 25k/50k runs, 15% of 100k runs, and 10% of 200k runs; the short LM runs use the adjusted warmup/decay points described by their sweep. We report the last-window average loss: 1k steps for 25k runs, 1.5k for 50k, and 2k for 100k/200k. Large-scale LM evaluations use 13 downstream benchmarks, detailed in Appendix B. Since all training runs process each sample at most once (single-epoch training), the training loss is an unbiased estimator of held-out loss and is interchangeable with validation loss in principle. In practice, the per-step diffusion training loss exhibits higher variance; we therefore report the last-window average training loss as described above to reduce this variance. For LLMs, the per-step validation loss variance is sufficiently low that we report it directly. MoE notation and sweep protocol. We write XeYa for X total experts and Y activated experts, with Zs for shared experts and Gg for groups; e.g., 128e8a4g1s. Unless stated otherwise, granularity, capacity, shared-expert, and group-balanced-routing studies match the dense unit-expansion active width. In activated-expert sweeps, 64e2a, 64e4a, 64e8a, and 64e16a correspond to active-width ratios 0.25, 0.5, 1, and 2. Each sweep applies Complete-muE first, then varies the base hyperparameter. 8
e 3
e 3
6.4
4.5
e 3
e 3
3.2
e 3
2.3
1.6
8e
2
1
1e
2
4e
3
1e
4e
1e
Val loss
e 3
e 3
6.4
4.5
e 3
e 3
3.2
e 3
lr
2.3
1.6
4
e 3 1.1
Train loss
8e
2
DF MoE (grouped) (100k, 256) Model ffn_1 64e8a 64e8a2g 64e8a4g 64e8a8g
1
4e
1e
2
3
1e
4e
3
4
4e
1e
4
1e
4e
5
Val loss
Train loss
0.500 DF MoE model width (16e4a, 100k, 256) Width 0.480 128 256 0.460 512 1024 0.440
e 3
e 3
6.4
4.5
e 3
e 3
3.2
e 3
lr
2.3
4
1.6
1.1
8e
2
1
4e
1e
2
3
1e
4e
3
4
1e
4
4e
4e
(e) MoE depth learning-rate sweep
1e
0.400
lr
e 3
0.420 5
e 3 4.5 e 3 6.4 e 3
e 3
3.2
e 3
2.3
8e
1.6
3 1.1
3
4
4e
5
e 3
e 3 4.5 e 3 6.4 e 3
e 3
Val loss
4 3.5 2.5
4
Train loss
4.5
0.460
lr
LM MoE 16e4a (25k, 128) Width 128 256 512 1024
5
0.480 1
2
4e
1e
2
1e
3
4e
3
4
4e
1e
4
5
1e
lr
lr
0.482 0.480 0.478 0.476 0.474 0.472 0.470
(d) Group-balanced-routing MoE learning-rate sweep
0.560 DF MoE layer (16e4a, 100k, 256, dim=128) Layers 0.540 4 8 0.520 16 32 0.500
e 3
LM MoE 16e4a (25k, 128) Layer 4 8 16 32
4e
Val loss
(c) Shared-expert MoE learning-rate sweep 4.8 4.6 4.4 4.2 4 3.8 3.6
lr
(b) MoE granularity learning-rate sweep
3.2
e 3
2.3
1.6
4
e 3
Train loss
8e
1.1
lr
lr
e 3
4
1e
LM MoE (grouped) LR sweep (25K, 128) Model ffn_1 64e8a 64e8a2g 64e8a4g 64e8a8g
DF MoE shared-expert (100k, 256) Model ffn_1 64e8a 64e2a6s 64e4a4s 64e6a2s 64e7a1s
1
2
4e
1e
2
1e
3
4e
3
4
4e
1e
4
5
1e
Val loss
4e
lr
0.482 0.480 0.478 0.476 0.474 0.472 0.470
0.484DF Sigmoid-routed MoE granularity (100k, 256) Model 0.482 0.480 ffn_1 0.478 ffn_4 0.476 32e2a 0.474 64e4a 0.472 128e8a 0.470 256e16a 0.468 512e32a 0.466 0.464
1.1
4
4e 4.8 4.6 4.4 4.2 4 3.8 3.6
(a) MoE capacity learning-rate sweep 4.8 LM MoE shared experts LR sweep (25K, 128) Model 4.6 ffn_1 64e8a 4.4 64e2a6s 64e4a4s 4.2 64e6a2s 4 64e7a1s 3.8 3.6
Train loss
e 3
6.4
4.5
e 3
e 3
lr
3.2
e 3
1.1
8e
2
1
1e
2
4e
3
1e
3
4e
4
1e
4
4e
5
4
e 3
0.465
lr
2.3
0.470
1.6
0.475
LM MoE granularity LR sweep (25K, 128) Model ffn_1 ffn_4 32e2a 64e4a 128e8a 256e16a 512e32a
4.8 4.6 4.4 4.2 4 3.8 3.6
ffn_1 ffn_4 4e4a 8e4a 16e4a 32e4a 64e4a 128e4a 256e4a 512e4a 1024e4a 2048e4a
0.480 Train loss
Model ffn_1 ffn_4 4e4a 8e4a 16e4a 32e4a 64e4a 128e4a 256e4a
1e
Val loss
MoE capacity (100k, 256) 0.485DF Sigmoid-routed Model
LM MoE capacity LR sweep (25K, 128)
4e
4.8 4.6 4.4 4.2 4 3.8 3.6
(f) MoE width learning-rate sweep
1
11.0 LM MoE Expert Width Initial Loss (25k, 128) 10.8
10.6
10.6
10.4 10.2 10.0
(a) FFN/MoE FFN width learning-rate sweep
LM Dense LR Initial Loss (25k, 128)
10.8 Train loss
Train loss
11.0
1e
2
2
4e
3
lr
1e
4e
3
4
4e
1e
4
5
1e
Val loss
5 LM MoE expert width LR sweep (25K, 128) Model 4.8 ffn_1 16e4a ffn_1 4.5 16e4a ffn_4 16e4a ffn_16 4.2 16e4a ffn_64 4 3.8 3.5 3.2
4e
1
2
4e
1e
2
lr
1e
3
4e
3
4
4e
1e
4
5
LM dense LR sweep (25K, 128) Model ffn_1 ffn_4 ffn_16 ffn_64
1e
5 4.8 4.5 4.2 4 3.8 3.5 3.2
4e
Val loss
Figure 3: Transfer across MoE architectural variants.
9.800
Model ffn_1 ffn_16 ffn_4 ffn_64 20 40 60 training step
10.4 10.2 10.0
80
1e2 9.800
Model ffn_1 16e4a ffn_16 16e4a ffn_1 16e4a ffn_64 16e4a ffn_4 1e2 20 40 60 80 training step
(b) FFN/MoE FFN width sweep initial training loss
Figure 4: Transfer across FFN/MoE FFN width expansion.
4.2
Transfer across all MoE scaling axes
Figure 2 verifies Bridge II at fixed total experts and per-expert width. Complete-muE keeps LR, WD, and init optima broad and aligned: representative ranges are LR 4 × 10−4 –4 × 10−3 ; WD 0.01–0.2 for LM and 10−4 –3 × 10−2 for diffusion; and init std 4 × 10−4 –4 × 10−2 for LM and 10−4 –3 × 10−2 for diffusion. The first 100 iterations nearly overlap, while larger a lowers attainable loss. Figure 3 and Figure 4 covers the remaining axes with normalized sigmoid routing. Capacity, fixeddensity granularity, shared experts, group-balanced routing, depth, width, and active FFN width expansion ratio both preserve similar LR optima. Capacity/granularity scaling both leads to lower loss. Increasing the number of shared experts consistently leads to higher loss, suggesting that shared experts are not necessary for performance unless they are used to overlap computation and communication in expert-parallel training/inference. Figure 5 shows that batch/duration transfer composes with the MoE rules. For batch sizes 128–1024, optima stay close at fixed steps (6.25k LM, 25k diffusion) and fixed samples (3.2M LM, 25.6M diffusion); √at fixed samples, the learning rate in the figure is aligned and scaled according to ρD = 1 and η ∝ ρB (an exact SDE transfer with all three SDE objects preserved), while at fixed steps the transfer is approximate—the optimal LR region remains relatively stable across batch sizes with only minor drift, consistent with the σ0 shift at fixed horizon. 9
e 3
e 3
6.4
4.5
e 3
e 3
3.2
e 3
lr
2.3
1.6
4
e 3
1.1
Train loss
8e
1
2
DF MoE equal-samples (32e4a, 25.6M, aligned LR) 0.477 Iteration / Batch 0.476 it=25k, 1024 it=50k, 512 0.475 it=100k, 256 0.474 it=200k, 128 0.473 0.472 0.471
1e
2
4e
3
lr
1e
3
4e
4
1e
4
4e
4e
(a) Batch-size study at fixed training iterations
1e
Val loss
5
e 2
e 3
4.8LM MoE equal sample aligned LR (3.2M samples) MoE 32e4a 4.6 6.25K, 512 12.5K, 256 4.4 25K, 128 4.2 4 3.8 3.6
1.3
e 3
9.1
e 3
6.4
lr
4.5
e 3
e 3
3.2
e 3
2
2.3
2
8e
Train loss
3
2e
lr
1.6
3
4
8e
8e
4
2e
2e
5
8e
Val loss
DF MoE batch-size (32e4a, 25k) 4.8LM MoE 32e4a batch-size LR sweep (6.25K train) 0.490 Batch size Batch size 128 128 4.6 0.485 256 256 512 512 4.4 0.480 1024 1024 4.2 0.475 4 0.470 3.8
(b) Batch-size study at fixed total trained tokens
Figure 5: Transfer across batch size. Activated Expert Scaling (64 Total Experts)
4.05
MoE FFN_1
2a
0.465
2a
6a 2a 4a 8a 64e 64e1 64e3 64e6 MoE setting - LLM (LR=1e-3)
4a
64e
MoE FFN_1
64e
(a) LM, activated experts
0.4800
3.8
Granularity Scaling (16x Capacity Ratio)
MoE FFN_1
0.4750 0.4725 0.4700
2a
a a 4a e8a e16 e32 64e 128 256 512 MoE setting - LLM (LR=1e-3)
(e) LM, granularity
2a 32e
Train loss
0.4750 0.4725 MoE FFN_1 0.4675 a a a a a a a 4e4 8e4 16e4 32e4 64e4 128e4 256e4 MoE setting - Diffusion (LR=1.6e-3) 0.4700
a a a a a a 8e4 16e4 32e4 64e4 128e4 256e4 MoE setting - LLM (LR=1e-3)
(c) LM, capacity
0.4775 Train loss
Val loss
4.0 MoE FFN_1
a
4e4
(b) DF, activated experts
Granularity Scaling (16x Capacity Ratio)
3.9
MoE FFN_1
3.8
6a 2a 4a 4a 8a 64e 64e 64e1 64e3 64e6 MoE setting - Diffusion (LR=1.6e-3)
a a 4a e8a e16 e32 64e 128 256 512 MoE setting - Diffusion (LR=1.6e-3)
(d) DF, capacity
Layer Scaling (16e4a MoE)
4.5 4.4 4.3 4.2 4.1 4.0 3.9
Layer Scaling (16e4a MoE)
0.54 0.52 0.50 0.48
4L
(f) DF, granularity
Capacity Scaling (4 Activated Experts)
0.4775
3.9
Train loss
3.85
0.470
Val loss
3.90
0.475
Val loss
Train loss
Val loss
3.95
32e
0.4800
4.0
4.00
64e
Capacity Scaling (4 Activated Experts)
Activated Expert Scaling (64 Total Experts)
0.480
8L 16L MoE setting - LLM (LR=1e-3)
32L
(g) LM, layers
4L
8L 16L 32L MoE setting - Diffusion (LR=1.6e-3)
(h) DF, layers
Figure 6: Fixed-hyperparameter loss scaling across MoE axes. We fix AdamW hyperparameters at the dense-tuned values (LM: LR=10−3 , init std=10−2 , WD=0.1; DF: LR=1.6 × 10−3 , init std=2 × 10−2 , WD=10−2 ) and vary the MoE architecture along four axes for both LM and DF. With Complete-muE, each axis produces consistent loss improvement without per-setting hyperparameter retuning—direct evidence for the tune-dense-once-and-transfer recipe.
Figure 6 provides the direct empirical evidence for the tune-dense-once-and-transfer recipe. Fixing AdamW hyperparameters at the dense-tuned values (LM: LR=10−3 , init std=10−2 , WD=0.1; DF: LR=1.6 × 10−3 , init std=2 × 10−2 , WD=10−2 ) and varying only the MoE architecture along four axes—activated experts, total capacity, granularity, and depth—yields consistent loss reduction for both LM and DF without per-setting retuning. Combined with the LR-sweep evidence above, this shows that Complete-muE’s bounded residual drift is small enough in practice for a single dense calibration to deliver consistent scaling gains across the entire MoE design space.
85
1.05x
80
1.00x
751
2 4 8 16 MoE capacity ratio (E/k)
0.95x 32
(a) Capacity scaling
1.5e2SwiGLU MoE Granularity (H100 80GB) 1.80x Sonic DCMoE 1.4e2 PyTorch Dense 1.3e2 1.60x 1.2e2 1.40x 1.1e2 1e2 1.20x 90 1.00x 80 702 4 8 16 32 64 Granularity
Relative latency
1e2 SwiGLU MoE Capacity (H100 80GB) Sonic DCMoE 1.20x PyTorch Dense 95 1.15x 90 1.10x
Relative latency
The preceding sweeps show two qualityimproving MoE axes—more total capacity at fixed activated experts, and finer granularity at fixed active width. Figure 7 compares their training latency on a single H100 80GB GPU using bfloat16, sequence length 40k, input width 4096, and Sonic DCMoE, our internal MoE implementation following SonicMoE-style design [34].
Latency (ms/step)
Systems benchmark for MoE scaling axes
Latency (ms/step)
4.3
(b) Granularity scaling
Capacity scaling fixes expert hidden size 2048 and k = 8 activated experts while Figure 7: SwiGLU MoE latency benchmark on a single varying E = 8–256; latency stays at 87.8– H100 80GB GPU. 97.0 ms/step, only 1.08×–1.20× slower than the 81.0 ms dense SwiGLU baseline. Granularity 10
DF Train Loss (1, 256, 256)
0.520 0.500
0.420 0.400
4e4 6e4 Training step
8e4
(a) 256P image training loss 0.600
Convergence speedup
Train loss
0.500 0.450 0.400 0.3500
2e4
4e4 6e4 Training step
8e4
Train loss
Val loss 8e4
(g) LLM training loss
0.400
8e4
0.3600
1e5
2 1.5 1 2e4
3e4 4e4 5e4 6e4 Dense training step
7e4
1e5
3.2 3.1 3 2.9 2.8 2.7 2.6 2.50
5
(h) LLM validation loss
8e4
1e5
4 3 2 1 02e3 1e4
8e4
2e4
3e4 4e4 5e4 6e4 Dense training step
7e4
8e4
(f) 240P 5s video convergence speedup LM Convergence Speedup vs Dense Step MoE (grouped) MoE Parity
6
8e4
4e4 6e4 Training step
DF Convergence Speedup vs Dense Step (57, 240, 432) convergence speedup Parity
6
LM Validation Loss
Model ffn_4p5 MoE (grouped) MoE 2e4 4e4 6e4 Training step
2e4
(c) 240P key-frame training loss
(e) 256P image convergence speedup
LM Train Loss
Model ffn_4p5 MoE (grouped) MoE 2e4 4e4 6e4 Training step
2.5
0.52e3 1e4
1e5
(d) 240P 5s video training loss 3 2.9 2.8 2.7 2.6 2.5 2.4 2.30
4e4 6e4 Training step
DF Convergence Speedup vs Dense Step (1, 256, 256) convergence speedup Parity
3
Model ffn_4p5 MoE
0.550
2e4
(b) 512P image training loss
DF Train Loss (57, 240, 432)
0.420
0.380
0.3800
1e5
Convergence speedup
2e4
Model ffn_4p5 MoE
0.440
Convergence speedup
0.4800
DF Train Loss (4, 240, 432)
0.460
Model ffn_4p5 MoE
0.440 Train loss
Train loss
0.540
DF Train Loss (1, 512, 512)
0.460
Model ffn_4p5 MoE
Train loss
0.560
1e5
5 4 3 2 1 01e3
2e4
4e4 6e4 Dense training step
8e4
(i) LLM convergence speedup
Figure 8: Complete-muE for larger-scale MoE training for multi-modal generation and LLMs.
scaling fixes active width hdim k = 16,384 and E/k = 8 while increasing k = 2–64; latency rises from 84.1 to 135.0 ms/step. Dense SwiGLU width scaling is far more expensive, from 81.0 to 667.6 ms/step before larger settings run out of memory. Thus capacity scaling is the cheaper axis when token batch size per device is large enough. 4.4
Larger-scale Complete-muE experiments
We apply the transferred recipe to larger runs. The multimodal diffusion setting trains for 100k iterations on 256P images (batch 3072), 512P images (batch 768), 240P key frames (batch 96), and 240P 5s videos (batch 48), using Knapformer workload balancing [35], per-layer AdaLN, and 3D RoPE. The large-scale LM uses 1D RoPE, and 512 batch size (2K seq length), trained for 100k iterations. Unless noted, models have 32 layers, width 1024, and MoE configuration 128e8a4g1s with per-expert width 512 and total active FFN width 4608. Dense and MoE models have about 0.62B active non-embedding parameters; MoE variants have 6.29B total non-embedding parameters. Both settings use a WSD schedule with 100k total iterations, 1,000 warmup steps, and a 15% decay phase. For diffusion training, the 3D RoPE uses channel splits [48, 48, 24]; AdamW is configured with learning rate 2.26 × 10−3 , β1 =β2 =0.95, and weight decay 0.01. For LLM training, AdamW uses learning rate 5 × 10−4 , β1 =β2 =0.95, and weight decay 0.05; LLM validation is performed on C4 [36, 37]. In both cases the target backbone width d=1024 is 8× the proxy width d⋆ =128, giving the Complete-muE width-expansion ratio ρd =8. Figure 8 shows that the recipe transfers beyond proxy sweeps. MoE models keep lower training loss across image, key-frame, video, and LM settings. We measure convergence speedup as dense steps divided by MoE steps needed to reach the same loss during the stable-LR phase of WSD. The MoE reaches roughly 2.5× speedup on 256P images and roughly 4.5× on 240P 5s videos; the LLM MoE variants reach roughly 5.3×–5.5×. The group-balanced variant is slightly behind the non-grouped routed variant, matching the smaller scale LM findings. Crucially, all four diffusion regimes (256P images, 512P images, 240P key frames, 240P 5s videos) share a single hyperparameter setting (LR=2.26×10−3 , WD=0.01) and the LM run uses one other (LR=5×10−4 , WD=0.05); every 11
setting still delivers consistent MoE-over-dense convergence speedup without any per-setting retuning. This is direct large-scale evidence for the tune-dense-once-and-transfer recipe: one dense calibration suffices to deliver consistent MoE gains across the entire multimodal/LM training landscape.
(a) Dense
(b) MoE
Figure 9: Caption 1: A cinematic head-and-shoulders video of a Black man in casual attire against a neutral seamless backdrop, softly lit with shallow focus and centered rule-of-thirds composition. The camera performs a handheld dolly-in and zoom-in toward his face as he bursts into a broad, joyful smile with visible teeth, creating a warm and expressive portrait. Table 4: Larger-scale dense versus MoE benchmark evaluation. Scores are percentages. Model Dense MoE 128e8a4g1s MoE 128e8a1s Model
SVAMP(5)
MMLU(5)
ARC-Easy(5)
ARC-Challenge(5)
COPA(5)
PIQA(5)
HellaSwag(5)
7.0 9.7 10.0
25.4 25.9 26.2
64.6 71.2 72.9
33.6 43.6 43.4
70.0 80.0 85.0
72.6 77.7 77.8
56.5 69.3 69.4
WinoGrande(5) LAMBADA(5)
Dense MoE 128e8a4g1s MoE 128e8a1s
59.0 65.2 66.5
50.5 61.9 63.4
BoolQ(5) 62.6 63.4 63.9
AGIEval-LSAT-RC(5) AGIEval-LSAT-LR(5) AGIEval-SAT-EN(5)
Average
27.3 27.1 24.3
44.3 49.1 50.6
20.9 22.8 27.2
26.2 20.9 27.2
The qualitative video example in Figure 9 is consistent with the loss trends: the MoE sample shows cleaner structure and less flicker for this prompt, while the quantitative evidence remains the loss, speedup, and benchmark evaluation. Table 4 reports the final 100k-step LM evaluation. Both MoE variants improve the dense average score: 44.3 → 49.1 for 128e8a4g1s and 44.3 → 50.6 for 128e8a1s. The non-grouped variant is best on average, while the group-balanced variant is strongest on ARC-Challenge.
5
Conclusion
We presented Complete-muE, a compositional AdamW transfer rule for FFN/MoE families. We identify the key bottleneck and uses dense FFN to dense MoE, and dense MoE to sparse MoE to bridget the gap of transfer problem. The resulting recipe covers activated experts, total capacity, fixed-density granularity, shared/group-balanced hybrids, and width/depth/batch/duration changes. Controlled LM and diffusion sweeps show relatively stable hyperparameter optima for LR, WD, and initialization across all MoE combinations, with mild drift consistent with Bridge II’s non-strict SDE behavior. In practice this drift is small enough that hyperparameters tuned on a single dense reference transfer near-optimally to all MoE configurations—tune dense once, transfer to all MoE settings is the practical recipe at the core of Complete-muE. Both controlled small-scale axis sweeps and large-scale multimodal/LM runs directly verify this recipe: a single dense calibration delivers consistent MoE gains across MoE axes and across modalities (256P/512P images, 240P key frames, 240P 5s videos, LM). The H100 benchmark shows that total-capacity scaling is much cheaper than fine granularity at fixed active width. Larger runs reach roughly convergence speedup up to 4.5× on 240P 5s videos, and 5.3×–5.5× on LLM training for only 100k training steps.
References [1] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. 12
[2] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. [3] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022. [4] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, 2024. [5] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. [6] Yehui Tang, Xiaosong Li, Fangcheng Liu, Wei Guo, Hang Zhou, Yaoyuan Wang, Kai Han, Xianzhi Yu, Jinpeng Li, Hui Zang, et al. Pangu pro moe: Mixture of grouped experts for efficient sparsity. arXiv preprint arXiv:2505.21411, 2025. [7] Yehui Tang, Yichun Yin, Yaoyuan Wang, Hang Zhou, Yu Pan, Wei Guo, Ziyang Zhang, Miao Rang, Fangcheng Liu, Naifu Zhang, et al. Pangu ultra moe: How to train your big moe on ascend npus. arXiv preprint arXiv:2505.04519, 2025. [8] Meituan LongCat Team, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-flash technical report. arXiv preprint arXiv:2509.01322, 2025. [9] Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: tuning large neural networks via zero-shot hyperparameter transfer. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pages 17084–17097, 2021. [10] Jan Małaśnicki, Kamil Ciebiera, Mateusz Boruń, Maciej Pióro, Jan Ludziejewski, Maciej Stefaniak, Michał Krutul, Sebastian Jaszczur, Marek Cygan, Kamil Adamczewski, et al. Muparametrization for mixture of experts. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models. [11] Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 35:7697–7711, 2022. [12] Nolan Simran Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don’t be lazy: Completep enables computeefficient deep transformers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [13] Bruno Mlodozeniec, Pierre Ablin, Louis Béthune, Dan Busbridge, Michal Klein, Jason Ramapuram, and Marco Cuturi. Completed hyperparameter transfer across modules, width, depth, batch and duration. arXiv preprint arXiv:2512.22382, 2025. [14] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020. [15] Haotian Sun, Tao Lei, Bowen Zhang, Yanghao Li, Haoshuo Huang, Ruoming Pang, Bo Dai, and Nan Du. Ec-dit: Scaling diffusion transformers with adaptive expert-choice routing. arXiv preprint arXiv:2410.02098, 2024. [16] Chandan Akiti, Ajay Modukuri, Murali Nandan Nagarapu, Gunavardhan Akiti, and Haozhe Liu. Nucleus-image: Sparse moe for image generation. arXiv preprint arXiv:2604.12163, 2026. 13
[17] Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Feature learning in infinite-depth neural networks. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023. [18] Blake Bordelon, Hamza Chaudhry, and Cengiz Pehlevan. Infinite limits of multi-head transformer dynamics. Advances in Neural Information Processing Systems, 37:35824–35878, 2024. [19] Chenyu Zheng, Xinyu Zhang, Rongzhen Wang, Wei Huang, Zhi Tian, Weilin Huang, Jun Zhu, and Chongxuan Li. Scaling diffusion transformers efficiently via µP. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [20] Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, and Boris Hanin. Hyperparameter transfer with mixture-of-expert layers. arXiv preprint arXiv:2601.20205, 2026. [21] Liliang Ren, Yang Liu, Yelong Shen, and Weizhu Chen. Rethinking language model scaling under transferable hypersphere optimization. arXiv preprint arXiv:2603.28743, 2026. [22] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114, 2022. [23] Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An opensource autoregressive language model. In Proceedings of BigScience Episode# 5–Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95–136, 2022. [24] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. [25] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. [26] Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, et al. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [27] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. [28] Antonio Orvieto and Robert M Gower. In search of adam’s secret sauce. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [29] Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I Mestre, Manuel F Dolz, and Enrique S Quintana-Ortí. Why adam works better with β1 = β2 : The missing gradient scale invariance principle. arXiv preprint arXiv:2601.21739, 2026. [30] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023. [31] Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling. [32] Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna B Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. Advances in Neural Information Processing Systems, 37:76232–76264, 2024. [33] Howe Tissue, Venus Wang, and Lu Wang. Scaling law with learning rate annealing. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [34] Wentao Guo, Mayank Mishra, Xinle Cheng, Ion Stoica, and Tri Dao. Sonicmoe: Accelerating moe with io and tile-aware optimizations. arXiv preprint arXiv:2512.14080, 2025. 14
[35] Kai Zhang, Peng Wang, Sai Bi, Jianming Zhang, and Yuanjun Xiong. Knapformer: An online load balancer for efficient diffusion transformers training. arXiv preprint arXiv:2508.06001, 2025. [36] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286–1305, 2021. [37] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. [38] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online, June 2021. Association for Computational Linguistics. [39] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. [40] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. [41] Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret, editors, *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394–398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. [42] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020. [43] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019. [44] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [45] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, Berlin, Germany, August 2016. Association for Computational Linguistics. [46] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, 2019. 15
[47] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299–2314, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
16
Table 5: Notation used in the method and appendix. Symbol
Meaning
ρd , ρH ρL , ρB , ρD ρHa , ρN exp ρexp B , ρD s d H Htot h D, G u(x) Wup , Wv , Wg vup , σup , ηup
residual/model-width ratio d/d⋆ and active-width ratio H/d. depth, global batch, and global token-budget ratios relative to the reference model. active-width ratio induced by activated experts, ρHa = ah/d, and total-expert ratio N/N⋆ . expert-side effective batch and expert-token-budget ratios used in the SDE analysis. routing density a/N . residual/model width. effective active FFN width; for Dense MoE, H = N h; for sparse MoE, H = Pah. P total active width of a shared/group-balanced-routing/hybrid block, Htot = m∈D Hm + a g∈G hg . per-expert hidden width. index sets of always-active dense/shared submodules and routed groups in a hybrid block. FFN hidden activation vector after the up/gate projections. FFN/MoE up and gate projections; in MoE the same notation refers to the corresponding per-expert copies. initialization variance, standard deviation, and AdamW learning rate for an FFN/MoE up or gate projection; these follow the ordinary backbone-width µP rule. second moment and mean absolute value of one hidden coordinate of u(x). FFN output / down projection. initialization variance and standard deviation of one entry of Wdown . AdamW learning rate applied to Wdown . total number of routed experts. number of activated experts per token; Dense MoE corresponds to a = N . selected expert set for token x, with |A(x)| = a.P normalized routing weight on expert e, satisfying e πe (x) = 1 over the active experts. Complete-muE output-branch multiplier for active width H, and its sparse-MoE specialization at Ha = ah. route scale multiplying the normalized routed sum. P 2 finite-width routing factor a E[ N e=1 πe (x) ]. router projection producing logits ℓ(x) = Wgate x + b. initialization variance, standard deviation, and learning rate of the router projection. global AdamW learning rate, weight decay, numerical stabilizer, and momentum coefficients. global token batch per optimizer step. global training token budget / total number of tokens seen during training. number of optimizer steps. mean routed-token batch seen by one expert when a experts are activated. mean expert-token budget across training, Dexp (a) = T Bexp (a). per-expert gradient-noise scale in the simplified SDE model; under balanced routing it scales as Bexp (a)−1/2 . effective SDE signal-to-noise parameter, σ0 (a) = η σexp (a). SDE integration horizon, HSDE = T η 2 . number of residual layers.
qu , µu Wdown vdown , σdown ηdown N a A(x) πe (x) A(H), Aa ra Fa,N Wgate vgate , σgate , ηgate η, λ, ϵ, β1 , β2 B D T Bexp (a) Dexp (a) σexp (a) σ0 (a) HSDE L
A
Additional derivations for Complete-muE
This appendix collects the derivations that were moved out of the main paper. The presentation mirrors the main method section: notation, dense FFN active-width transfer, Dense MoE factorization, activated-expert SDE analysis, compositional derivations for capacity and granularity, the hybrid/shared-expert/group-balanced-routing extension, and a brief note on how the global AdamW transfer composes with Complete-muE. A.1
Dense FFN active-width transfer derivation
Consider a dense FFN output branch y(x) = cH Wdown u(x),
(9)
where u(x) ∈ RH denotes the hidden activation after the up/gate projections. Let qu := E[uj (x)2 ],
µu := E[|uj (x)|],
and initialize Wdown,ij ∼ N (0, vdown ). Under the AdamW gradient-magnitude-normalized regime, ∆Wdown = −ηdown P(Gdown ),
P(cG) ≈ P(G)
for c > 0,
(10)
so matching one-step functional updates only depends on explicit forward multipliers. The forward variance and one-step functional update scale as Var[yi (x)] ≍ c2H Hvdown qu , E|∆yi (x)| ≍ cH ηdown Hµu . 17
(11) (12)
To match these quantities to the unit-expansion dense companion at the same backbone width d, we set d H (1) (1) 2 AH = cH = , BH = vdown (d, H) = vdown (d), CH = ηdown (d, H) = ηdown (d). H d (13) Equivalently, r H (1) σ (d). σdown (d, H) = d down Eq. (13) is the active-width ABC form used throughout the paper. The up/gate projections still follow ordinary backbone-width µP, −1/2
σup (d) = ρd σup,⋆ , ηup (d) = ρ−1 (14) d ηup,⋆ , and the same rule applies componentwise to gated MLP projections. The router readout also remains ordinary µP, −1/2 σgate (d) = ρd σgate,⋆ , ηgate (d) = ρ−1 (15) d ηgate,⋆ . A.2
Dense MoE factorization and normalized route scale
For Dense MoE or sparse MoE, let expert e produce (e)
oi (x) =
h X
(e)
(e)
Wdown,iℓ uℓ (x),
ℓ=1
and let the routed branch be yiMoE (x) = ra
N X
N X
(e)
πe (x) oi (x),
e=1
πe (x) = 1.
(16)
e=1
Since one expert has width h, its output variance satisfies (e)
Var[oi (x)] ≍ hvdown qu . Using independence and zero-mean initialization across experts, Var[yiMoE (x)] ≍ Hvdown qu ·
ra2 Fa,N , a2
(17)
Fa,N := a E
N hX
i πe (x)2 ,
(18)
e=1
where H = ah is the active width. For the one-step update, the expert gradient is (e)
(e)
Gdown,iℓ = ra πe (x)gi uℓ (x), so Eq. (10) removes the leading dependence on the positive scalar ra πe (x). Hence E|∆yiMoE (x)| ≍ ra ηdown hµu . (19) Matching this to the dense FFN update at the same active width H = ah yields the route scale ra = a, (20) which is Eq. (3) in the main paper. Exact forward matching would instead use a raexact−fwd = p . (21) Fa,N In practice we keep the simpler main rule ra = a because Fa,N is usually close to one at initialization. This bounded correction can be understood by Pexpanding the normalized router around equal active logits. Write ℓj = ℓ̄ + δj for j ∈ A(x) with j δj = 0. For normalized softmax, 1 δj 1 h X 2i πj = + + O(∥δ∥2 ), Fa,N = 1 + E δj + O E∥δ∥3 . (22) a a a j∈A(x)
′
For normalized sigmoid, letting κ(ℓ̄) := σ (ℓ̄)/σ(ℓ̄) with σ(·) the logistic sigmoid, 1 κ(ℓ̄) 1 h X 2i πj = + δj + O(∥δ∥2 ), Fa,N = 1 + κ(ℓ̄)2 E δj + O E∥δ∥3 . a a a
(23)
j∈A(x)
Thus Fa,N remains an order-one constant near one unless the selected logits are already highly separated. 18
A.3
Expert-side SDE analysis for activated experts
The layer rule in Eq. (4) already matches the routed-FFN layer scales. What remains is the expert-side stochastic effect induced by changing how many tokens each expert receives. Under approximate load balancing, a a Bexp (a) ≈ B , Dexp (a) = T Bexp (a) ≈ T B . (24) N N Therefore r N −1/2 σexp (a) ∝ Bexp (a) ∝ . Ba Changing the activated-expert count from a to a′ therefore changes the expert noise scale by s Bexp (a) σexp (a′ ) 1 Bexp (a′ ) a′ = p exp , = ρexp = . (25) B := ′ σexp (a) Bexp (a ) Bexp (a) a ρB Model one expert by the stochastic gradient decomposition (e)
(e)
gt (a) = ḡ (e) + σexp (a) ξt ,
(e)
ξt
∼ N (0, I).
(26)
Under a simplified normalized RMSPropW/AdamW proxy, (e)
(e) (e) ϑt+1 = ϑt − η
gt (a) (e) + λϑt σexp (a)
! .
(27)
HSDE := T η 2 .
(28)
Introducing the continuous-time step ∆τ = η 2 , define σ0 (a) := η σexp (a),
e := λ , λ η
Then Eq. (27) is the Euler–Maruyama discretization of dΘ(e) τ =−
1 e (e) dτ − dW (e) , ḡ (e) dτ − λΘ τ τ σ0 (a)
0 ≤ τ ≤ HSDE .
(29)
The Brownian coefficient is normalized to one, so the per-expert SDE is governed by three quantities: the signal-to-noise parameter σ0 (a) (smaller means less noise relative to gradient drift), the normalized e and the optimization horizon HSDE . weight decay λ, Reference: exact batch scaling at fixed expert-token budget. Before analyzing activated-expert transfer, consider a clean reference case. Keep the architecture and sparsity a fixed, increase the global batch from B to κB B, and reduce optimizer steps from T to T /κB so that total trained ′ tokens are unchanged. Under balanced routing, each expert then sees Bexp (a) = κB Bexp (a) and exp exp ′ Dexp (a) = Dexp (a), giving ρB = κB and ρD = 1. Preserving all three SDE quantities (σ0′ = σ0 , e′ = λ, e H′ λ SDE = HSDE ) requires q q η ′ = ρexp λ′ = ρexp (30) B η, B λ. This is the dense fixed-token batch-transfer rule applied to the expert process [11, 13]. Why activated-expert transfer is not an exact SDE invariance problem. In activated-expert transfer we instead change a to a′ at fixed global batch B and fixed steps T . Unlike the reference case (ρexp D = 1), this changes both expert batch and expert duration by the same factor: Bexp (a′ ) a′ Dexp (a′ ) a′ = , ρexp = . (31) D = Bexp (a) a Dexp (a) a p Applying rule (30) here would scale η by a′ /a, simultaneously shifting the SDE horizon to ′ HSDE = T (η ′ )2 = (a′ /a)HSDE and perturbing non-sparse parameters, which do not obey the same sparsity-dependent transformation. We therefore do not claim an exact full-model SDE invariance for activated-expert transfer. ρexp B =
19
Hypothesis: after layer-level matching, the SDE correction cancels. Following the approximation philosophy of Complete(d)P [13], we hypothesize that once Eq. (4) is in place, changing a primarily shifts the expert-side SNR σ0 (a) while the global AdamW terms and SDE horizon remain unchanged. Under this hypothesis, the combined correction applies the expert batch factor together with an iso-horizon duration factor: s q ρexp −1/2 B , (32) η′ ≈ ρexp · (ρexp ·η = η B D ) ρexp | {z } D | {z } expert batch iso-horizon duration exp ′ and similarly for λ . Since ρB = ρexp D = a /a, the correction is unity: Therefore the dense-style ′
batch/duration correction cancels,
s ′
η ≈η
s
ρexp B = η, ρexp D
′
λ ≈λ
ρexp B = λ, ρexp D
(33)
which is Eq. (6) in the main paper. What does change is the expert-side signal-to-noise parameter: σ0 (a) (34) σ0 (a′ ) = η σexp (a′ ) = p exp . ρB Making the MoE less sparse (larger a′ ) reduces σ0 (a′ ) and thereby improves expert-side SNR at fixed optimization horizon HSDE ; this is the direct analogue of the iso-horizon picture in Complete(d)P [13]. The interpretation is not that changing a leaves training unchanged. Rather, after the Complete-muE layer reparameterization, the remaining first-order effect of changing a is a shift in expert-side SNR at roughly fixed optimization horizon, not a new global LR/WD transfer rule. Because the σ0 (a) shift is not absorbed by the first-order η, λ correction, mild hyperparameter drift across a is expected—bounded in magnitude by the σ0 shift—and Figure 2 confirms this empirically: optima remain relatively stable across activated-expert counts with only minor drift. Generalization to imbalanced routing. The balanced analysis above is the clean closed-form proxy. In real MoE training the router is not perfectly load balanced, so different experts receive (e) different numbers of tokens at different steps. Let ℓt (a) denote the realized normalized load of P (e) expert e at step t, with e ℓt (a) = a, so that (e)
(e)
Bexp,t (a) = B ℓt (a),
(e) Dexp (a) =
T X
(e)
Bexp,t (a).
(35)
t=1
The expert-noise scale and SNR parameter become step-dependent, −1/2 (e) (e) (e) (e) σexp,t (a) ∝ Bexp,t (a) , σ0,t (a) := η σexp,t (a),
(36)
′
so there is generally no single expert-independent constant σ0 (a ) after changing a. The first-order cancellation nevertheless survives at the expert-averaged level. Define the time-averaged load T
ℓ̄(e) (a) :=
1 X (e) ℓ (a), T t=1 t
(e) B̄exp (a) = B ℓ̄(e) (a),
(e) (e) Dexp (a) = T B̄exp (a).
For every expert with nonzero average load under both settings, the effective batch and duration factors are equal: (e)
m̄exp B,e :=
B̄exp (a′ ) (e)
=
ℓ̄(e) (a′ ) , ℓ̄(e) (a)
(e)
m̄exp D,e :=
Dexp (a′ ) (e)
=
ℓ̄(e) (a′ ) = m̄exp B,e . ℓ̄(e) (a)
B̄exp (a) Dexp (a) Therefore the same expert-side approximation gives s exp s exp m̄B,e m̄B,e ′ ′ ηe ≈ η λe ≈ λ = λ, exp = η, m̄D,e m̄exp D,e
(37)
(38)
so the same cancellation holds expertwise to first order. What changes in the imbalanced case is not (e) the first-order cancellation, but the fact that σ0,t (a) fluctuates across experts and training steps; these mild deviations contribute to—but do not dominate—the bounded hyperparameter drift visible in Figure 2, where learning-rate optima remain relatively stable across activated-expert counts even without strict load balancing. 20
Summary. After the Complete-muE layer-level reparameterization, changing the activated-expert count is best viewed as keeping the deterministic routed-FFN update matched while shifting the expert-side SNR at roughly fixed optimization horizon. The first-order SDE correction cancels, so AdamW hyperparameter transfer remains effective to first order even though different sparsity levels can still reach different attainable losses. Equivalently, activated-expert transfer is relatively stable rather than a strict invariance: the residual σ0 (a) shift produces mild, bounded hyperparameter drift across a, consistent with Figure 2. A.4
Compositional derivations: total experts and granularity
The two bridges established above yield all remaining MoE scaling cases by composition. The key organizing principle is that total-expert and granularity changes are not new primitive rules—they are derivable by composing the dense-width and activated-expert transfer rules. In each case the final sparse layer recovers the same active-width rule Aa = d/(ah), route scale ra = a, and output initialization, confirming that the active width Ha = ah is the single governing quantity. Total expert count. Consider transfer from (N, a, h) to (N ′ , a, h) with fixed activated experts a and fixed per-expert width h. Introduce the Dense-MoE companions (N, N, h) and (N ′ , N ′ , h), obtained by activating all experts. The transfer decomposes into a dense-width step from total width N h to N ′ h followed by the reverse sparsity transfer from N ′ activated experts back to a. These two steps are explicitly: 1. Dense-width step. Transfer the auxiliary Dense MoE from total width N h to N ′ h via Bridge I. The output multiplier and initialization become r d N ′ h (1) (1) (1) σdown = σ , c = ′ , Nh d down,⋆ and the auxiliary route scale becomes rN ′ = N ′ . 2. Reverse-sparsity step. At fixed total experts N ′ , reduce the activated count from N ′ back to a via Bridge II. The active-width rule at Ha = ah gives r d ah (1) (2) (2) c = , σdown = σ , ah d down,⋆ and restores the route scale ra = a. The extra dense-width factor from Step 1 is exactly reversed by Step 2. Because the sparsity rule exactly reverses the extra active-width factor introduced by the dense step, the final sparse layer again uses r ah (1) d Aa = , σdown (d, a) = σ (d), ra = a, ah d down up to the bounded routing factor Fa,N . This is identical to the rule obtained by sparsifying the original companion (N, N, h) directly to a activated experts. Thus total-expert transfer is not a new primitive hyperparameter-transfer rule; it is a composition of dense-width transfer and activated-expert transfer. The SDE bookkeeping is consistent with the same conclusion. Under balanced routing, a a Bexp (N ) ≈ B , Dexp (N ) ≈ T B , N N
(39)
exp ′ so for N → N ′ we again have ρexp B = ρD = N/N , and the first-order SDE cancellation from ′ ′ Appendix A.3 gives η ≈ η, λ ≈ λ, consistent with the deterministic derivation above. The SDE perspective is best read as a consistency check after the layer-level cancellation has been established. What remains is the capacity–noise trade-off discussed in the main text: increasing N raises representational capacity but reduces the expert-wise token budget, worsening expert-side SNR: r r N N σexp (N ) ∝ , σ0 (N ) = η σexp (N ) ∝ η . (40) Ba Ba These two effects compete: over some range of N the extra capacity may dominate and loss decreases while the optimal LR stays nearly unchanged; at larger N , saturation or reversal is possible once
21
the SNR penalty becomes dominant. The transferable-hyperparameter claim is therefore only that the same raw AdamW hyperparameters remain appropriate after the two-step cancellation, not that increasing N must monotonically improve loss. As with activated-expert transfer, the reverse-sparsity step within this composition is not a strict SDE invariance—σ0 (N ) shifts with N and is not absorbed by the first-order η, λ correction—so mild hyperparameter drift across N is expected. Figure 3a confirms this empirically: the optimal loss region remains relatively stable across capacity settings, with only minor drift consistent with the non-strict SDE transfer. Granularity at fixed routing density. By granularity scaling we mean changing the expert partition (N, h) 7→ (N ′ , h′ ) while keeping the routing density fixed, s :=
a a′ = ′. N N
(41)
Each sparse layer can be viewed as its Dense-MoE companion followed by the same sparsification ratio s: (N, N, h) 7→ (N, a, h) and (N ′ , N ′ , h′ ) 7→ (N ′ , a′ , h′ ). The Dense-MoE companions have ′ total widths Hdense = N h and Hdense = N ′ h′ , while the sparse active widths are Ha = ah = sN h,
Ha′ = a′ h′ = sN ′ h′ .
(42)
Therefore
H′ Ha′ N ′ h′ = dense , = (43) Ha Nh Hdense which is exactly the Dense-MoE companion width ratio. Granularity transfer at fixed density therefore reduces to Dense-MoE width transfer, while the routed output continues to use ra = a and ra′ = a′ on the two sides. On the stochastic side, both expert batch size and expert trained duration remain unchanged because Bexp = Bs, Dexp = T Bs, (44) which are unchanged when s is fixed, so no additional SDE correction is introduced. A.5
Hybrid, shared, and group-balanced-routing MoE blocks
The previous sections already provide all ingredients needed for shared experts, grouped MoE, and their combination. The rule is simplest when stated in three steps. First, every always-active dense submodule contributes active width but no route scale. Second, P every routed MoE group is selected using group-balanced routing, while global normalization ( g,e πg,e (x) = 1) and global route scale r = a are applied jointly across all selected experts. Third, after those dense and routed pieces are identified, one adds their active widths and treats the whole block as one equivalent expanded dense FFN, so one common FFN-output ABC parametrization is applied jointly to all active submodules. Let D index always-active dense/shared branches and G index routed groups. Dense branch m ∈ D has hidden width Hm . Routed group g ∈ G has Ng total experts and per-expert width hg ; groups are used for balanced routing selectionP only. All selected experts share global routing weights πg,e (x) normalized jointly over all groups ( g,e πg,e (x) = 1) and global route scale r = a. The total active width of the whole block is therefore X X Htot = Hm + a hg . (45) m∈D
g∈G
All dense/shared up projections and all expert-local up/gate projections still follow the ordinary backbone-width µP rule; the hybrid-specific coupling appears only in the common output-layer ABC parametrization through the total expansion ratio ρHtot = Htot /d and in the global route scale r = a. Using this total active width, Complete-muE parameterizes the block as Ng X (m) XX y(x) = ctot πg,e (x) o(e) Wdown u(m) (x) + a g (x) , m∈D
g∈G e=1
ctot =
d . Htot
Equivalently, the whole block shares one common active-width ABC triple Atot = ctot ,
Btot = σdown (d, Htot ), 22
Ctot = ηdown (d, Htot ),
(46)
while the routed groups collectively carry one global route scale r = a; in a full model transfer, the actual output-layer init std and LR are obtained by further multiplying the standard backbone-width −1/2 factors ρd and ρ−1 d . A shared-expert block is the special case with one always-active shared branch of width Hsh and one routed branch (N, a, h), so " # N X (e) y(x) = ctot osh (x) + a πe (x) o (x) , Htot = Hsh + ah. e=1
This can be derived in three compositional steps. First split a dense FFN of width Htot into two dense coordinate blocks of widths Hsh and ah; as a pure dense decomposition, both blocks inherit the same output multiplier ctot and the same output initialization from the parent dense FFN, while the shared block needs no route scale. Next convert the routed-equivalent dense block of width ah into a DenseMoE companion with a experts of width h, which introduces only the normalized-routing factor a. Finally apply the total-expert transfer from (a, a, h) to (N, a, h). By the capacity-scaling result above, this changes total routed capacity without introducing any additional first-order hyperparameter change. Hence the shared branch keeps the dense FFN setting and the routed branch keeps the same dense split plus the route scale, yielding the shared-expert formulation used in the main text. The µP justification is straightforward. If the active hidden coordinates were concatenated into one vector of width Htot , Eq. (46) would be exactly the dense FFN rule from Eq. (13); keeping separate output projections is simply its block decomposition. Consequently all active output projections share the same initialization and optimizer scale, r Htot (1) (1) σdown (d, Htot ) = σdown (d), ηdown (d, Htot ) = ηdown (d), (47) d −1/2
with the ordinary ρd and ρ−1 d factors composed on top if the backbone width changes. The routed groups additionally require the global route scale r = a, while dense/shared branches do not. Assuming independent zero-mean initialization across active branches, the forward scale obeys X X Var[yi (x)] ≍ c2tot vdown (d, Htot )qu Hm + a hg Fg , (48) m∈D
g∈G
P PNg where F := a E g∈G e=1 πg,e (x)2 is a single global order-one factor (F ≈ 1 at initialization under uniform routing). P The dense/shared branches contribute exactly their active widths. The routed groups contribute a g hg up to the global factor F . In practice F ≈ 1, so Complete-muE sets F = 1 and uses the common initialization rule in Eq. (47) for all active submodules. Likewise the one-step functional update scales as X X E|∆yi (x)| ≍ ctot ηdown (d, Htot )µu Hm + a hg = ctot ηdown (d, Htot )Htot µu , (49) m∈D
g∈G
P
where we used r = a and g,e πg,e (x) = 1. Hence shared experts, MoE with group-balanced routing, and arbitrary hybrids all reduce to the same total-active-width rule used in the main text. A shared-expert MoE is the special case |D| = 1, |G| = 1: " # N X (e) y(x) = ctot Wdown,sh ush (x) + a πe (x) o (x) , Htot = Hsh + ah.
Shared experts.
(50)
e=1
The shared branch carries no route scale; the routed branch carries ra = a; and both sets of output weights use the same initialization rule in Eq. (47). For example, if Hsh = 0.5d and three routed experts are active each of width h = 0.5d, then Htot = 0.5d + 3 × 0.5d = 2d, hence ctot = 1/2 and √ −1/2 σdown (d, Htot ) = ρd 2 σdown,⋆ for both the shared-expert output weights and the routed-expert output weights. 23
Grouped MoE. For grouped MoE without shared experts, D = ∅ and G routed groups are simultaneously active: y(x) = ctot a
Ng G X X
πg,e (x) o(e) g (x),
Htot = a
g=1 e=1
G X
hg .
(51)
g=1
Groups are used for balanced routingP selection only; all selected experts share one global route scale r = a and global normalization ( g,e πg,e (x) = 1), and all output projections share the same initialization rule determined by Htot . This remains true whether the implementation concatenates the active group features before a single down-projection or uses separate group projections and sums the outputs. Shared experts plus grouped MoE. The combination is immediate: place always-active branches in D, routed groups in G, assign one global route scale r = a across all routed experts (not pergroup), and compute one common FFN multiplier from Htot in Eq. (45). This is the most general Complete-muE rule for hybrid MoE blocks. A.6
How global AdamW transfer composes with Complete-muE
Table 3 in the main text contains the remaining global AdamW transfer factors inherited from CompleteP / Complete(d)P [12, 13]. These factors are applied after the layer-level Complete-muE rule in Table 2. The connecting link between the SDE analysis in Appendix A.3 and the global optimizer table is the general expert-side correction formula: after the layer-level reparameterization, any change to the optimizer schedule or MoE architecture produces an expert-side LR and weightdecay multiplier of the form s s ρexp ρexp ′ ′ B B , λ ≈ λ (52) η ≈η exp , ρexp ρ D D exp where ρexp B and ρD denote the expert-side batch and duration ratios. Whether this correction introduces a new multiplier on top of Table 2 depends on which quantities change.
Global SDE framework for all-layer transfer. Let ϑt denote the rescaled parameter of any layer (dense or routed) after the Complete-muE layer-level reparameterization in Table 2. For a global batch of size B, the per-step stochastic gradient decomposes as gt = ḡ + σnoise ξt ,
ξt ∼ N (0, I),
σnoise ∝ B −1/2 ,
(53)
where ḡ is the mean gradient and σnoise captures finite-batch stochasticity. Under the simplified RMSPropW proxy, the preconditioned parameter update reads gt η ϑt+1 = ϑt − η + λϑt = ϑt − ḡ + λσnoise ϑt − ηξt . (54) σnoise σnoise Introducing the three SDE governing objects σ0 := η σnoise ,
e := λ , λ η
HSDE := T η 2 ,
(55)
and the rescaled time τ = tη 2 , Eq. (54) becomes the Euler–Maruyama discretization of dΘτ = −
1 e τ dτ − dWτ , ḡ dτ − λΘ σ0
0 ≤ τ ≤ HSDE .
(56)
In this normalized form the Brownian coefficient is fixed at one: the stochastic noise level is encoded e and entirely in σ0 (smaller σ0 means lower diffusion relative to drift), the weight-decay strength in λ, the training budget in HSDE . Two configurations yield statistically identical optimization trajectories e′ = λ, e and H ′ if and only if all three objects match: σ0′ = σ0 , λ SDE = HSDE . The two cases below differ precisely in which of these conditions can be satisfied simultaneously. 24
Case 1: Fixed total tokens, changing batch size (exact transfer). Scale the global batch from B to B ′ = κB B and reduce the optimizer-step budget proportionally, T′ =
T , κB
(57)
so that the total trained tokens BT = B ′ T ′ are unchanged. Every layer—dense and routed alike— then has B′ B′T ′ ρB := = κB , ρD := = 1. (58) B BT √ ′ From Eq. (53), the gradient-noise scale transforms as σnoise = σnoise / κB . We match all three √ SDE ′ ′ objects of Eq. (55) step by step. (i) Matching σ0 = σ0 , i.e. η ′ σnoise = ησnoise , gives η ′ = κB η. e′ = λ, e i.e. λ′ /η ′ = λ/η, then gives λ′ = √κB λ. (iii) The horizon is automatically (ii) Matching λ ′ preserved: HSDE = T ′ (η ′ )2 = (T /κB )(κB η 2 ) = T η 2 = HSDE . All three SDE objects of Eq. (56) are simultaneously preserved, so the transfer is exact: √ √ √ √ λ′ = κB λ = ρB λ, (ρD = 1). (59) η ′ = κB η = ρB η, This is the CompleteP fixed-token batch rule, applied uniformly to all layers—dense and routed—with no additional expert-specific multiplier. Case 2: Fixed training iterations, changing batch size (approximate transfer). Now keep the optimizer-step budget T fixed while scaling the batch from B to B ′ = κB B. Total trained tokens grow by κB , giving ρB = κB , ρD = κB (ρB = ρD ). (60) ′ In this case the two SDE invariance conditions σ0′ = σ0 and HSDE = √ HSDE become mutually ′ ′ ′ incompatible at fixed T : preserving σ0 = σ0 would require η = κB η, but then HSDE = ′ 2 ′ ′ T (η ) = κB HSDE√̸= HSDE ; conversely, preserving HSDE = HSDE forces η = η, leaving ′ σ0′ = ησnoise = σ0 / κB ̸= σ0 . The practical choice is to fix the horizon and accept the σ0 shift:
η ′ ≈ η,
σ0 σ0′ = √ , κB
′ HSDE = HSDE .
(61)
e′ = λ e at η ′ = η gives λ′ = λ automatically. In Eq. (56) this means the normalized SDE Matching λ e while the diffusion amplitude σ0 decreases: the retains the same horizon HSDE and decay rate λ, target run sees less stochastic noise per step. This transfer is approximate rather than exact; the error for σ0′ term is of order |κB − 1| in σ0 . Structurally this mirrors Bridge II’s activated-expert transfer: the residual σ0 shift is not absorbed by the first-order η, λ correction, so mild hyperparameter drift across batch sizes is expected. Figure 5a confirms this empirically: the optimal LR region remains relatively stable across batch sizes 128–1024 at fixed training iterations, with only minor drift consistent with the bounded σ0 shift. Global batch and duration change at fixed MoE architecture. When the global batch size B and training duration T change while the MoE architecture remains fixed, every routed expert sees exp the same p proportional change aspthe global model: ρexp B = ρB and ρD = ρD . Eq. (52) then reduces ′ ′ to η ≈ η ρB /ρD and λ ≈ λ ρB /ρD , recovering the familiar CompleteP rule. Cases 1–2 above make precise when this reduction is exact (fixed tokens, ρD = 1) versus approximate (fixed iterations, ρD = κB ). MoE-specific changes at fixed global batch and duration. When instead the MoE architecture changes—activated-expert count a, total expert count N , or granularity at fixed routing density s = a/N —while global batch B and total steps T are held fixed, both the expert-side batch and the expert-side duration change by the same factor. As derived in Appendix A.3, exp ρexp B = ρD ,
so the correction in Eq. (52) cancels to unity: η ′ ≈ η and λ′ ≈ λ. The layer-level reparameterization in Table 2 already absorbs all first-order architectural changes—no additional global LR or WD multiplier is required for MoE-specific scaling axes. 25
For an FFN/MoE output branch with active-width ratio ρH = H/d and backbone-width ratio ρd = d/d⋆ , the full transfer therefore takes the form r ρB −1/2 1/2 −1 σdown = ρd ρH σdown,⋆ , ηdown = ρd ηdown,⋆ , (62) ρD with the analogous batch/duration factors applied to weight decay, AdamW ϵ, and 1 − β1,2 according to Table 3. For tensors whose scaling is controlled only by the residual width d—the router readout and the FFN/MoE up/gate projections—the ρH factor is absent and only the standard backbone-width and global optimizer multipliers remain. Operationally, Complete-muE can be read as the following sequence. First choose the local FFN/MoE rule from Table 2: determine the active width, assign the route scale, and set the output-layer initialization. Then compose the global optimizer factors from Table 3: depth-sensitive residual groups use the p usual CompleteP parameter grouping, while batch and duration changes contribute the familiar ρB /ρD correction on η, λ, together with the corresponding AdamW ϵ and momentum adjustments. The two-case structure above makes the scope of these global factors precise: they cover global schedule changes (Case A) but not MoE architectural changes (Case B), which are handled entirely within Table 2.
B
LLM benchmark evaluation details
This appendix summarizes the post-training language benchmark protocol used for the results in Table 4. We do not include in-training benchmark probes in this paper. The final evaluation covers 13 datasets spanning symbolic problem solving, world knowledge, commonsense reasoning, language understanding, and reading comprehension. Multiple-choice benchmarks are reported with accuracy, while free-response arithmetic is reported with exact match. The benchmark scores in Table 4 use the 5-shot setting shown in that table. Table 6: Post-training NLP evaluation datasets and reporting settings. Area
Dataset
Split #Samples Shot Metric
Symbolic problem solving SVAMP [38] test World knowledge MMLU [39] test World knowledge ARC-Easy [40] test World knowledge ARC-Challenge [40] test Commonsense reasoning COPA [41] val Commonsense reasoning PIQA [42] val Language understanding HellaSwag [43] val Language understanding WinoGrande [44] val Language understanding LAMBADA [45] test Reading comprehension BoolQ [46] val Reading comprehension AGIEval-LSAT-RC [47] test Reading comprehension AGIEval-LSAT-LR [47] test Reading comprehension AGIEval-SAT-EN [47] test
300 14,042 2,376 1,172 100 1,838 10,042 1,267 5,153 3,270 268 510 206
5 5 5 5 5 5 5 5 5 5 5 5 5
Exact Match Multiple-choice Acc. Multiple-choice Acc. Multiple-choice Acc. Multiple-choice Acc. Multiple-choice Acc. Multiple-choice Acc. Cloze Acc. Language-modeling Acc. Multiple-choice Acc. Multiple-choice Acc. Multiple-choice Acc. Multiple-choice Acc.
Random Acc. 0% 25% 25% 25% 50% 50% 25% 50% 0% 50% 25% 25% 25%
Symbolic problem solving. SVAMP [38] contains 300 short arithmetic word problems with freeform numerical answers. We use a chain-of-thought prompt before the final answer and score the final prediction with exact match; the chance-level baseline is effectively 0%. World knowledge. MMLU [39] evaluates broad academic knowledge over 57 subjects and 14,042 four-choice questions. ARC-Easy and ARC-Challenge [40] contain grade-school science questions, with 2,376 and 1,172 examples respectively. ARC-Easy focuses on more direct science knowledge, while ARC-Challenge emphasizes harder questions that require additional reasoning. These three datasets are reported with multiple-choice accuracy, with a 25% random baseline. Commonsense reasoning. COPA [41] is a 100-example two-choice task in which the model selects the more plausible cause or effect for a premise. PIQA [42] contains 1,838 two-choice physical commonsense questions about everyday actions and affordances. Both are evaluated with accuracy, and the random baseline is 50%. 26
Language understanding. HellaSwag [43] asks the model to choose the most plausible ending for a partial narrative from four candidates, over 10,042 validation examples. WinoGrande [44] contains 1,267 two-choice coreference examples formulated as cloze questions. LAMBADA [45] evaluates long-context word prediction on 5,153 passages by asking the model to predict the final token. HellaSwag uses a 25% random baseline, WinoGrande uses a 50% random baseline, and LAMBADA is treated as an open-vocabulary language-modeling task. Reading comprehension. BoolQ [46] evaluates yes/no reading comprehension over 3,270 passagequestion pairs. AGIEval-LSAT-RC, AGIEval-LSAT-LR, and AGIEval-SAT-EN [47] evaluate reading comprehension, logical reasoning, and SAT-style English respectively, with 268, 510, and 206 fourchoice examples. BoolQ has a 50% random baseline, while the AGIEval subsets use a 25% random baseline.
27