arXiv CS open access

AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding

2026 · arxiv_cs

arXiv CS · Papers · 2026

cloud distributed-computing parallel-computing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding Shuang Liang1 , Hao (Mark) Chen1 , Zhiwen Mo1 , Qianzhou Wang1 , Guoyu Li1 , Lingxiao Ma2 , Wayne Luk1 1

Imperial College London, 2 Tile-AI shuang.liang@imperial.ac.uk

arXiv:2608.02989v1 [cs.LG] 4 Aug 2026

Abstract Speculative decoding verifies a tree of draft tokens in one target-model forward pass. For a mixture-of-experts (MoE) target, however, parallel verification can activate the union of the experts selected by all tree nodes, even though only a small subset of those nodes reaches the accepted output. Token count, activated-expert union size, and expert-weight traffic are therefore distinct cost measures: reducing the token workload need not shrink the expert union proportionally, and under offloading, transfer traffic also depends on cache residency. We introduce AcceptMoE, a verifier-side expert selector that combines target-router scores with offline-estimated commitment probabilities and automatically adjusts the number of eligible experts for each verification block, eliminating the need for a user-specified expert budget. Under offloading, AcceptMoE conditions expert eligibility on cache residency instead of predicting natural routes and prefetching the corresponding expert weights. Although constraining target-expert eligibility changes the model distribution, across 12 model– task pairs spanning three MoE targets and four benchmarks, AcceptMoE’s mean accuracy is 0.27 percentage points lower than that of EAGLE-3 speculative decoding with natural routing. Served with SGLang at batch size one, it reaches 1.290× the throughput of this baseline with all expert weights in GPU memory, and 2.06× under physical expert offloading, while reducing host-to-device traffic by 73.6% to 77.1%.

Introduction Autoregressive decoding remains a dominant cost in large language model (LLM) inference because every output token requires a full forward pass through the model (Yang et al. 2025; Guo et al. 2025; Agarwal et al. 2025). Speculative decoding (Cai et al. 2024; Li et al. 2024b; Chen et al. 2025a) amortizes this cost by using a draft mechanism to propose a block of tokens that the target verifies in one forward pass. The verifier then accepts a prefix according to a rule that preserves the target distribution. For an MoE target, however, parallel verification need not reduce memory traffic proportionally. Although each token activates only its top-k experts, different draft-tree nodes can select different experts (Dai et al. 2024). A verification block can therefore access the union of all per-node top-k expert sets, whose cardinality may substantially exceed k. The size of this union determines the number of distinct expert weights accessed during verification. Challenge 1: Natural routing

incurs expert-weight traffic for draft branches that are ultimately discarded, so reducing the number of verified tokens need not proportionally reduce the activated-expert union. Draft-side selection determines which draft nodes are retained for target verification, whereas verifier-side expert selection determines which target experts remain eligible for the resulting verification block. The two decisions operate on different quantities. Reducing the number of retained draft nodes need not shrink the activated-expert union proportionally. On Qwen3-30B-A3B, EVICT reports 74.7% fewer verified tokens than EAGLE-3 but only 32.5% fewer activated experts (Pan et al. 2026). A verifier-side expert selector determines the eligible expert set to which each token’s routing is restricted. MoE-Spec constructs this set by aggregating router mass over the verification block and retaining a prescribed number B of experts at each layer (McDanel et al. 2026). Its set size is therefore externally specified, while its aggregation weights every draft position equally despite their unequal probabilities of reaching the output. Challenge 2: A fixed expert budget must be supplied in advance, but the best measured budget varies across evaluated model–task pairs. AcceptMoE addresses both limitations by constructing commitment-weighted router demand for each verification block. For each draft position, we estimate offline the probability that a node proposed there is committed to the output and use this probability to weight target-router scores before summing them across the block. AcceptMoE ranks experts by this weighted demand and retains the highest-ranked nonanchor experts while always preserving the natural top-k experts of the root token. It determines how many non-anchor experts to retain using the effective rank of their demand distribution, defined as the exponential of its entropy. Concentrated demand yields a smaller set, whereas diffuse demand yields a larger one. The resulting expert set is therefore recomputed and self-sized for every verification block without a predefined expert budget. Across the 12 model–task pairs, the rounded per-pair mean set size selected by AcceptMoE ranges from 23 to 34 experts. Challenge 3: Under expert offloading, transfer cost depends on which selected experts are nonresident, not only on the set cardinality. AcceptMoE therefore applies residency-aware pruning: it orders selected nonresident ex-

perts by increasing commitment-weighted demand and removes the longest prefix allowed by the rerouting budget and minimum-set-size constraint. Prior offloading systems for speculative MoE decoding predict the target’s natural expert routes and prefetch the corresponding expert weights. AcceptMoE instead makes expert eligibility a function of the resident set, without requiring a learned expert predictor or a predefined expert budget. Our contributions are as follows. • Commitment-weighted expert demand. To resolve Challenge 1, we replace the uniform router-mass aggregation of prior verifier-side selectors with targetrouter scores weighted by position-dependent commitment probabilities estimated offline. This places greater weight on demand from draft positions that are more likely to be committed. At matched per-pair budgets B0 , AcceptMoE-B improves the mean accuracy across the 12 model–task pairs by 2.45 percentage points over MoESpec. • Self-sizing expert sets. To resolve Challenge 2, we derive the expert-set size from the effective rank of the same demand, removing the expert budget that prior selectors must be given in advance. Across five budget-swept pairs, the self-sized point is 0.97 percentage points below the best measured fixed-budget point on average and at most 1.83 points below it. • Residency-aware pruning. To resolve Challenge 3, we evaluate the same demand against the resident set and prune low-demand nonresident members, making expert eligibility a function of the expert cache rather than a prefetch schedule. Compared with the same selector with pruning disabled, this stage reduces expert-weight traffic by 38.6% to 48.6%. Its mean accuracy change over the 12 pairs is −0.27 percentage points, with per-pair changes from −1.83 to +1.22 points. Measured end to end in the SGLang serving stack across three MoE targets and four tasks at batch size one, AcceptMoE’s mean accuracy is 0.27 percentage points lower than that of Standard SD, EAGLE-3 speculative decoding with the target’s natural routing. It reaches 1.290× the throughput of Standard SD with all experts resident, decoding at up to 257 tokens per second on one RTX PRO 6000 Blackwell, and 2.06× under expert offloading on one RTX 5090, while reducing host-to-device traffic by 73.6% to 77.1%.

Related Work Speculative Decoding Speculative decoding uses a draft mechanism to propose a block of tokens that the target model verifies in one parallel forward pass, with an accept/reject rule that preserves the target distribution (Stern, Shazeer, and Uszkoreit 2018; Leviathan, Kalman, and Matias 2023). Many subsequent methods vary the drafter. Medusa and PPD let the target draft its own continuations through parallel decoding heads or trained prompt tokens (Cai et al. 2024; Chen et al. 2025a). EAGLE drafts autoregressively in feature space, with EAGLE-2 adding dynamic draft trees and EAGLE-3 using direct token drafting over fused multi-level features (Li et al. 2024b,a, 2025).

Mixture-of-Experts Optimization Sparse MoE layers decouple parameter count from per-token compute and are used by many large open-weight models (Jiang et al. 2024; Dai et al. 2024). Their memory footprint has motivated expert pruning, which removes experts offline from router or task statistics (Chen et al. 2022; Zhang et al. 2025), and expert skipping, which drops low-value experts per token at run time (Huang et al. 2026a; Lu et al. 2024; Huang et al. 2025; Aghdam, Jin, and Wu 2024). Expert sharing instead routes a parallel workload through a shared expert subset. Opportunistic activation reuses experts already selected in the batch (Oncescu et al. 2025), XShare maximizes aggregate gating mass over a per-batch subset (Vankov et al. 2026), and dynamic expert sharing selects a compact expert coreset for a parallel diffusion-decoding block (Chen et al. 2026). Expert sharing is the family closest to ours, since it also chooses one expert set for a parallel workload. AcceptMoE specifically uses the position-dependent commitment structure of a speculative verification tree and derives the expert-set cardinality for each block.

Speculative Decoding for MoE Models A recent line of work studies how speculation interacts with MoE sparsity. MoESD analyzes how batch size and sparsity affect target efficiency and speculative speedup (Huang et al. 2026b), whereas Cascade measures speculation utility online and selects a speculation length, disabling speculation altogether when its measured utility falls below one (Saxena et al. 2025). Neither method changes verifier expert routing. Under expert offloading, SP-MoE uses draft-model attention outputs with the target router to predict and prefetch experts needed for verification (Chen et al. 2025b). MoESpeQ uses a quantized MoE draft model for multi-step expert lookahead, then orchestrates prefetching, caching, and draft length (Wang et al. 2025). Both systems predict the target’s natural expert routes and schedule the resulting weight movement. Neither changes which experts remain eligible during verification. Draft-side selection determines which draft nodes enter target verification. EVICT selects an ancestor-closed prefix by combining drafter scores with offline-profiled verification cost (Pan et al. 2026). EcoSpec addresses expert scattering by fine-tuning a target-specific expert predictor and using a dynamic expert buffer to select a specified number of draft nodes based on acceptance likelihood and predicted marginal expert cost (Xie et al. 2026). Both methods preserve the target’s natural top-k routing and the distribution-preserving semantics of speculative verification while selecting the retained draft-node set before target-model verification begins. In contrast, AcceptMoE runs inside the serving engine after the verification block is fixed and requires no auxiliary expert predictor, routing supervision, or additional model inference. Verifier-side methods instead constrain expert eligibility after the verification block has been fixed. In its speculative mode, XShare performs hierarchical expert selection under per-request and batch capacities (Vankov et al. 2026). MoESpec aggregates router probabilities over the draft tree and retains the top B experts at each layer (McDanel et al. 2026).

Standard SD on MoE

AcceptMoE: Commitment-Weighted, Self-Sizing Expert Selection (a) Demand computation

E3 E4 E5 E6 E7 Draft tree

High Commitment Probability

(b) Self-sizing eligible expert set

E1 E2

Root ancher

Concentrated

E4 E5 E6 Low

S Eligible expert set

Diffuse

E7 Draft tree

S E1

E7 Same draft tree (top-k restricted to S)

Figure 1: AcceptMoE constructs a layer- and block-specific verifier expert set. Position-dependent commitment probabilities weight target-router demand, the root token’s natural top-k experts seed the set, and effective rank determines how many additional experts are retained. Figure 2 shows the residency-aware pruning applied when expert weights are offloaded. These methods select experts subject to externally specified capacity constraints. AcceptMoE instead derives both expertset membership and per-block cardinality from commitmentweighted demand and, under offloading, prunes nonresident members based on the current resident set.

Method AcceptMoE constructs one eligible expert set for each MoE layer and verification block. Figure 1 summarizes the three stages used under full residency. The first panel scores each expert by commitment-weighted demand, obtained by weighting the target router’s natural top-k probabilities by the marginal probability that a node at the same draft position is committed to the output. The second panel combines a root anchor with this demand. Demand ranking selects the nonanchor experts, and demand concentration determines how many are retained. The final panel restricts every token’s routing to the resulting set during verification. Under expert offloading, Figure 2 shows the additional residency-aware pruning stage, which removes low-demand nonresident experts subject to the rerouting budget and minimum set size.

Verifier-Side Expert Selection Consider one target MoE layer with E experts and per-token top-k routing. A draft-side selection policy has already produced a verification block V containing T tokens. For token t ∈ V, let zt ∈ RE denote the target-router logits, rt = softmax(zt ) the router distribution, and dt the draft position, given by block offset for a chain and tree depth for a tree. Natural routing selects Kt = TopKk (rt ), so the layer’s activated expert union is [ Unat (V) = Kt . (1) t∈V

A verifier-side expert selector constructs one layer- and block-specific eligible expert set S ⊆ [E], where [E] = {1, . . . , E}. The layer routes each token by e t (S) = TopKk (zt + MS ) , K

(2)

where (MS )e = 0 for e ∈ S and (MS )e = −∞ otherwise. The implementation applies this logit mask, or an equivalent zero-probability mask, in the corresponding backend

and preserves the target’s native post-top-k weighting. Dispatch under this constraint is stage (c) of Figure 1, and setting S = [E] recovers Standard SD. A fixed-budget selector takes |S| as a hyperparameter and determines only membership, whereas AcceptMoE derives both membership and cardinality from the current block. Because the mask changes the target router’s logits, any such selector is an approximation rather than a distribution-preserving optimization.

Commitment-Weighted Demand P Aggregate router mass, t rt,e , weights every token in the verification block equally. Two properties of the draft tree argue against this choice. Deeper positions are less likely to reach the accepted prefix, and sibling branches are mutually exclusive because at most one root-to-leaf path can be committed; consequently, their demands are never realized together. AcceptMoE therefore weights each position by how often a node proposed there is committed to the model’s output. For draft position d, we estimate pbd = cd /md from offline verification traces, where md is the number of proposed nodes observed at position d and cd is the number of those nodes that lie on the accepted output prefix. This is a marginal commitment probability, not the conditional acceptance rate usually reported for speculative decoders. The estimates are computed from training-split traces, fixed before evaluation, P and never use evaluation labels. Let Nd = t 1[dt = d] be the number of tokens at position d in the block. AcceptMoE assigns token t the importance −1/2 αt = pbdβt Ndt , (3) where β controls how sharply the commitment estimates are −1/2 applied (default β = 0.5). The Ndt factor discounts wide tree levels sublinearly, so their aggregate weight grows as √ Nd rather than Nd . Expert utility is ue =

T X

αt rt,e 1[e ∈ Kt ],

e ∈ [E].

(4)

t=1

An expert therefore receives router probability mass only through the natural top-k set of some token, with lower weights at positions that are less likely to be committed.

Because Eq. 4 uses only target-router outputs already available in the verifier and the offline position weights, it requires no additional model inference.

Self-Sizing the Eligible Expert Set Expert-set construction ranks experts by ue and always includes the root anchor [ Kt , (5) A=

Residency-Aware Pruning of the Eligible Expert Set max |D| s.t. ∑ ve ≤ a(S) ∧ |S| − |D| ≥ k

Eligible expert set

GPU memory

E4 Load

GPU memory CPU DRAM

No load

Demand high

Demand low

Expert Cache Update and Rerouting

t∈Tanc

where Tanc contains the designated anchor tokens. For EAGLE this is the depth-0 root. Before residency-aware pruning, its natural top-k set guarantees coverage for the one token that is certain to be committed, while every other eligible expert is selected by estimated demand. Let R = [E]\A denote the non-anchor experts and UR the utilities restricted to R. AcceptMoE retains the top-ranked experts of R and derives their number fromP the shape of UR . + Let u+ = max(u , 0) for e ∈ R and Z = e e j∈R uj . When + Z > 0, define qe = ue /Z and & !' X ner = exp − qe log qe . (6)

e∈D

LRU eviction

GPU memory (before)

GPU memory (after)

Figure 2: Residency-aware pruning of the AcceptMoE expert set. All nonresident members, including experts contributed by the anchor, are ordered by commitment-weighted demand; the rerouting budget and the minimum set size determine how long a prefix is removed.

e∈R

The eligible expert set is S = A ∪ Topner (UR ), of total size |A| + ner . The quantity ner is the effective-rank functional (Roy and Vetterli 2007) applied to the demand distribution, namely the exponential of its entropy. It approaches one when residual demand concentrates on a single expert and m when the demand spreads uniformly over m experts, so a block whose demand is concentrated is served by a smaller set. A minimum-size constraint enforces |S| ≥ k. If the residual utilities are all zero, AcceptMoE retains only as many non-anchor experts as that constraint requires.

Residency-Aware Pruning The size of S bounds the number of distinct experts that can be activated, but does not determine weight-transfer traffic under partial residency. A resident expert executes without a host-to-device transfer, whereas a nonresident expert triggers a weight load and can evict a cached expert. AcceptMoE therefore treats the resident set as a runtime signal of transfer cost and prunes S with the same utility, as illustrated in Figure 2. Each MoE layer has an X-slot expert pool in GPU memory and holds the full expert weights in pinned host memory. The pool caps residency, not eligibility. A nonresident expert is loaded synchronously when it receives tokens. If the block’s routed expert union after masking exceeds X, tokens are partitioned into several waves, each of whose routed expert union fits in the pool. For each layer, let H be the current LRU resident set and C = S \ H the nonresident members of S. AcceptMoE removes a subset D ⊆ C and routes the block through S \ D under the same constrained top-k rule as Eq. 2. Tokens whose natural experts were removed are rerouted to retained experts, which need not themselves be resident. Any such expert is loaded on demand when it first receives a token. At full residency C = ∅ and this step leaves S unchanged.

The removal is bounded by the P rerouting that forming S has already caused. Let ve = t 1[e ∈ Kt ] be the token S assignment count under natural top-k routing and K = P t Kt . Restricting the layer to S already displaces a(S) = e∈K\S ve natural assignments, and we allow pruning to displace at most that many more, which we call the rerouting budget. Let π order the experts in C by increasing utility ue , breaking ties by expert index. We remove the longest prefix that respects the rerouting budget and the minimum-size constraint: Pm m⋆ = max m : j=1 vπ(j) ≤ a(S), (7) |S| − m ≥ k , D⋆ = {π(1), . . . , π(m⋆ )}. The final eligible expert set is S ′ = S \ D⋆ . In the evaluated configuration, the implementation’s optional removal cap is set to E and is therefore nonbinding; removals are determined by the two constraints above, without a fitted transfer-cost threshold.

Experiments Models and tasks. We evaluate three target–draft model pairs: Qwen3-30B-A3B-Instruct-2507, Qwen3-Coder-30BA3B-Instruct, and GPT-OSS-120B, each with its corresponding EAGLE-3 draft model. We generate one greedy completion for every example in the full evaluation sets of GSM8K (Cobbe et al. 2021) (flexible exact match), MATH500 (Lightman et al. 2024) (math_verify), HumanEval (Chen et al. 2021), and MBPP (Austin et al. 2021) (pass@1 for both code benchmarks). For GPT-OSS on GSM8K and MATH500, we extract and score the final Harmony channel. We report full-residency accuracy and accuracy under residency-aware pruning with X = 48 slots per MoE layer.

Baselines. All speculative methods use the same EAGLE3 draft tree with five draft steps, an EAGLE top-k of 10, and 64 draft tokens, isolating verifier-side expert selection. The baselines are target-only autoregressive decoding (Vanilla AR), Standard SD with the target model’s native top-k routing, and two fixed-budget verifier-side selectors at the same B0 : Count-B, which keeps the experts receiving the most natural top-k votes in the block, and MoE-Spec, which ranks experts by router mass summed over the verification block and retains the top B0 experts (McDanel et al. 2026). For each model–task pair, we compute the mean AcceptMoE expert-set size on the first 50 evaluation prompts and round it to the nearest integer, with half values rounded up. If the bootstrap interval crosses a rounding boundary, we instead use the first 100 prompts. This budget-selection procedure uses only routing and selector traces, never correctness outcomes, and B0 is fixed before either fixed-budget method is evaluated. AcceptMoE-B is our matched-budget variant, which uses the same commitment-weighted utility and root anchor as AcceptMoE but holds the expert-set size at B0 . At the same budget, MoE-Spec and AcceptMoE-B directly compare expert-selection policies. Comparing AcceptMoEB with AcceptMoE contrasts fixed-budget and self-sizing selection, with B0 matched to the rounded mean capacity selected by AcceptMoE. Commitment probabilities are estimated from disjoint training traces and held fixed during evaluation. Throughput measurement. Full-residency throughput is measured at batch size one on one NVIDIA RTX PRO 6000 Blackwell using SGLang 0.5.12.post1 (Zheng et al. 2024), synchronous Spec V1, overlap scheduling disabled, and CUDA Graphs enabled. After one untimed warm-up request using evaluation example 50, we time examples 0–49 in fixed order, generating exactly 256 output tokens per prompt while ignoring EOS. We run three repetitions from fresh server instances, rotate the method order across repetitions, and sum output tokens and timed generation across the three repetitions to compute throughput for each model–task pair. Physical expert-offloading throughput is measured for all three targets on one NVIDIA RTX 5090. At X = 48, all runs fit within its 32 GB memory. GPT-OSS has the largest expert-pool footprint and uses 28.60 GiB at peak. Each configuration uses batch size one, 20 frozen prompts, one untimed warm-up request, greedy decoding with EOS ignored, and exactly 256 generated tokens per prompt. We report the median of three fresh-server repetitions. The offload runtime uses SGLang 0.5.12.post1 with overlap scheduling and CUDA Graphs disabled because the physical slot cache requires eager execution. Non-expert target weights, the draft model, the KV cache, and the per-layer slot pools remain on the GPU. Qwen expert weights are stored in pinned host memory in BF16, whereas GPT-OSS expert weights retain their native packed-MXFP4 representation in pinned host memory. Load-time redirection ensures that the complete checkpoint expert stacks are never simultaneously loaded into GPU memory. Host-to-device (H2D) traffic and cachehit statistics are collected in separate single-repetition instrumented runs over the same prompts and are excluded from

throughput aggregation.

Accuracy Table 1 separates the two decisions a verifier-side selector makes. The Fixed budget regime holds the number of selected experts at B0 and changes only the expert-selection policy. The Self-sizing regime lets AcceptMoE determine size per verification block. Count-B and MoE-Spec require the deployment to supply B. The pair-specific budget B0 spans 23 to 34 experts and equals the rounded mean cardinality selected by AcceptMoE on each pair. Supplying this value to the fixed-budget baselines matches their expert-set size to AcceptMoE for the same workload instead of forcing one global B across all pairs. Thus, B0 provides a size-matched comparison and is not tuned to maximize a baseline’s accuracy. At this matched budget, expert membership still affects accuracy. Ranking by vote count reaches 82.72% on average and MoE-Spec reaches 88.15%, compared with 90.60% for AcceptMoE-B. Aggregate router mass gives every draft node equal weight even though at most one root-to-leaf path is accepted. Within the B0 -expert shortlist, demand from discarded branches can therefore displace experts used by committed tokens. Commitment weighting reduces the contribution of positions that are less likely to reach the output. Because cardinality is fixed, the observed differences reflect the membership construction rule rather than the number of selected experts. The two selectors also have similar acceptance behavior. The largest absolute difference in τ between AcceptMoE and MoE-Spec is 0.17 tokens, and their signed median difference is below 0.01 tokens. Both selectors remain close to Standard SD, so the accuracy difference is not accompanied by a material change in mean accepted length. With self-sizing, AcceptMoE determines expert membership and cardinality per block. Its mean accuracy over the 12 pairs is 0.27 points below Standard SD, with a mean absolute per-pair difference of 0.43 points and a maximum of 1.22 points. Compared with self-sizing, fixing the size at the matched B0 yields a mean absolute per-pair difference of 0.74 points and a maximum of 2.44 points. The self-sizing and fixed-budget variants have nearly identical mean accuracy across the 12 pairs but differ on individual pairs. Under expert offloading, AcceptMoE uses an expert pool of X = 48 slots per MoE layer and modifies expert eligibility according to the current expert-cache state. To measure the accuracy effect of this cache-aware optimization, we compare it with the same selector with residency-aware pruning disabled on the same offload stack. Enabling pruning changes mean accuracy by −0.28 points across the eight Qwen pairs and −0.27 points across the four GPT-OSS pairs, with perpair changes from −1.83 to +1.22 points. Thus, cache-aware pruning incurs less than 0.3 percentage points of mean accuracy loss.

End-to-End Throughput Figure 3 reports end-to-end throughput. AcceptMoE outperforms Standard SD on all 12 model–task pairs, with a mean speedup of 1.290× and a per-pair range of 1.217–1.339×. AcceptMoE and Standard SD both have a mean accepted

Table 1: Task accuracy (%) for Qwen3-Instruct (Inst.), Qwen3-Coder (Coder), and GPT-OSS-120B (OSS). B0 is the rounded mean number of experts selected by AcceptMoE on each model–task pair and is supplied to the fixed-budget methods. τ is the mean accepted length per target pass. Bold marks the best result per column among the six full-residency rows. The final Acc. column is the unweighted mean over the 12 pairs, and vs. AR is its relative change from Vanilla AR. GSM8K

MATH500

HumanEval

MBPP

Average

Method

Inst.

Coder OSS

Inst.

Coder OSS

Inst.

Coder OSS

Inst.

Coder OSS

Acc.

vs. AR

Expert budget B0 τ , Standard SD τ , MoE-Spec τ , AcceptMoE

34 4.04 3.96 3.97

28 3.52 3.52 3.55

32 4.62 4.66 4.66

26 4.06 3.94 4.07

31 3.82 3.86 3.81

33 4.77 4.77 4.74

30 4.36 4.24 4.40

33 4.80 4.84 4.82

– – – –

Vanilla AR Standard SD

Natural routing 95.98 94.24 90.45 95.80 88.00 93.20 93.29 93.29 94.51 78.60 78.60 96.20 91.01 – 95.53 93.93 91.43 95.60 87.40 93.80 93.29 93.29 92.68 79.60 78.80 95.40 90.90 −0.1%

Count-B MoE-Spec AcceptMoE-B

Fixed budget B0 90.98 91.51 90.90 94.80 82.00 93.00 76.83 63.41 90.24 67.20 61.20 90.60 82.72 −9.1% 94.77 92.72 91.89 96.00 84.60 94.00 89.63 80.49 91.46 76.40 71.40 94.40 88.15 −3.1% 95.98 94.16 91.05 97.00 87.00 94.00 89.63 92.68 94.51 80.00 76.60 94.60 90.60 −0.5%

24 4.73 4.84 4.75

23 5.04 5.06 5.04

25 4.32 4.31 4.34

24 4.58 4.56 4.55

Self-sizing expert set AcceptMoE 95.53 94.47 91.28 96.00 86.80 93.80 92.07 92.68 92.68 79.40 78.20 94.60 90.63 −0.4% AcceptMoE (Offload) 95.38 93.78 91.28 95.60 87.40 93.80 93.29 93.29 90.24 79.00 76.60 93.40 90.25 –

Vanilla AR 300 200

Qwen3-Instruct 1.28×

Standard SD 1.34×

AcceptMoE 1.32×

Vanilla AR 60

1.33×

0 300 200

Qwen3-Coder 1.28×

1.28×

1.33×

1.32×

100 0 300 200

GPT-OSS-120B 1.22×

1.29×

1.25×

100 0

Output throughput (tok/s)

MATH500

HumanEval

MBPP

2.64×

2.22×

2.30×

2.99×

2.29×

1.95×

1.89×

1.63×

0 60 40 20

Qwen3-Coder

1.68×

1.54×

0 60

GPT-OSS-120B 2.02×

40 20

GSM8K

AcceptMoE

Qwen3-Instruct

100

Standard SD

1.59×

GSM8K

MATH500

HumanEval

MBPP

Figure 3: End-to-end decoding throughput with all expert weights resident in GPU memory, on one RTX PRO 6000 Blackwell.

Figure 4: End-to-end decoding throughput under physical expert offloading, on one RTX 5090 with 48 expert slots per MoE layer.

length of 4.39 tokens after rounding. Their largest per-pair difference is 0.07 tokens. The throughput improvement therefore does not come from longer accepted prefixes. The result is consistent with smaller eligible sets reducing per-pass expert computation and expert-weight accesses. Combining output tokens and timed generation across the four tasks for each model gives per-model speedups of 1.249–1.318×. The gain is smaller on GPT-OSS, whose lower per-token routing fan-out leaves a smaller natural expert union and less scope for reduction. Model architecture and execution kernels also differ across targets, so this comparison does not isolate the

effect of routing fan-out. Expert offloading exposes the cost of the verifier-side expert union. Standard SD applies natural routing to every node in the draft tree, so each verification pass may request the union of experts used across the entire tree. This expanded union lowers cache reuse and increases expert-weight transfers. The resulting transfer overhead can outweigh the benefit of verifying tokens in parallel. Standard SD spends 65% to 78% of wall time waiting for expert weights and is slower than Vanilla AR on seven of the eight Qwen pairs. AcceptMoE restricts the eligible expert union for each verification block

50 25

Qwen3 Instruct

Qwen3 GPT-OSS Coder 120B

(b) Self-sizing vs a tuned budget

1500

75 50

Inst./HumanEval Inst./MBPP Coder/HumanEval Coder/MBPP OSS/HumanEval Self-sized B0

25 0

Fixed expert budget B

pruning off pruning on

-38.6%

1000

-44.3% -48.6%

500

Qwen3 Instruct

(d) Expert cache hit rate Expert cache hit rate (%)

100

Vanilla AR Standard SD AcceptMoE

H2D (MiB / token)

(a) Time lost to expert fetches

Accuracy (%)

Fetch stall (% of wall)

100

90 85

+7.2 pp

+6.9 pp

pruning off pruning on

Qwen3 GPT-OSS Coder 120B

Qwen3 Instruct

+8.2 pp

Qwen3 GPT-OSS Coder 120B

Figure 5: Offload mechanism and ablations. (a) Fraction of wall time spent waiting for expert fetches. (b) Accuracy across a sweep of fixed expert budgets. (c) Expert-weight traffic and (d) expert-cache hit rate with residency-aware pruning disabled and enabled. Table 2: Expert-weight traffic and cache behavior under expert offloading. For each model, H2D traffic and cache hit rate are computed after summing their counters across four tasks. H2D is host-to-device expert traffic per output token. ∆ denotes the percentage change in H2D traffic and the percentage-point change in hit rate. Qwen expert weights are transferred in BF16, and GPT-OSS expert weights in packed MXFP4. H2D byte counts are therefore not directly comparable across targets.

Method

H2D (MiB/tok) ↓

Hit rate (%) ↑

Inst. Coder

Inst. Coder OSS

OSS

Vanilla AR 888 721 439 81.0 83.5 81.0 Standard SD 2394 2025 1243 73.0 74.4 74.3 AcceptMoE 633 464 298 83.8 87.0 88.5 ∆ −73.6% −77.1% −76.0% +10.8 +12.6 +14.2

and reduces the fetch-stall share to 27%–35%. It outperforms both Standard SD and Vanilla AR on all 12 model–task pairs and achieves a mean speedup of 2.06× over Standard SD. Figure 4 reports the throughput results, and Figure 5 reports the wait-time shares. Standard SD remains faster than Vanilla AR on all four GPT-OSS pairs. This difference is consistent with GPT-OSS expert weights using packed MXFP4 rather than BF16. Table 2 quantifies this mechanism. Standard SD transfers more expert-weight data than Vanilla AR on all three targets and has a lower cache hit rate. Compared with Standard SD, AcceptMoE reduces expert-weight traffic per output token by 73.6–77.1% and raises the expert-cache hit rate by 10.8 to 14.2 percentage points. Together, these measurements show that AcceptMoE reduces offload overhead by transferring fewer expert weights and reusing cached experts more effectively.

Ablation Study We ablate two components of AcceptMoE’s verifier-side expert selection: expert-set cardinality and residency-aware pruning.

Self-sizing versus measured fixed budgets. The fixedbudget sweep in Figure 5 evaluates five model–task pairs while holding the membership rule fixed. The self-sized point is 0.97 percentage points below the best measured fixed-budget point in its own sweep on average and 1.83 points below it in the worst case. At the smallest displayed budgets, losses compared with each sweep’s best measured point range from 23.2 to 92.1 points. A budget selected for one workload also need not transfer. On HumanEval, B=25 is the best measured budget for GPT-OSS-120B. Applying this budget to Qwen3-Coder on the same task yields accuracy 9.1 percentage points below Qwen3-Coder’s best measured fixed-budget result. Across these five sweeps, self-sizing avoids a per-pair budget sweep with an average measured accuracy gap of 0.97 percentage points. Residency-aware pruning. To isolate residency awareness, we compare AcceptMoE with a variant that disables residency-aware pruning. Both configurations construct the same commitment-weighted, self-sized expert set before pruning. The disabled variant leaves this set unchanged and does not use cache residency to modify expert eligibility. Figure 5 reports the traffic and cache results. Enabling residencyaware pruning reduces expert-weight traffic by 38.6% to 48.6% and raises cache hit rate by 6.9 to 8.2 percentage points. It also improves throughput by 4.6% to 15.1%. Enabling pruning changes mean accuracy by −0.27 points across the 12 pairs, with per-pair changes from −1.83 to +1.22 points. The largest throughput gain and traffic reduction both occur on GPT-OSS. These results show that configurations with similar mean accuracy across the 12 pairs can nevertheless differ substantially in transfer traffic.

Conclusion We presented AcceptMoE, a verifier-side expert selector for speculative decoding with MoE targets. Verification cost depends not only on the number of verified tokens but also on the union of experts they activate. Under expert offloading, it also depends on which experts are nonresident. AcceptMoE combines target-router scores with offline-estimated commitment probabilities and automatically adjusts the number of eligible experts for each verification block. This self-sizing

design removes the need for a predefined expert budget. Under partial residency, AcceptMoE further conditions expert eligibility on cache residency to reduce transfers without a learned expert predictor. Across three targets and four tasks, AcceptMoE’s mean accuracy is 0.27 percentage points lower than that of Standard SD. Compared with Standard SD, it reaches 1.290× the throughput with all experts resident and 2.06× under expert offloading. It also reduces host-to-device traffic by 73.6% to 77.1%. These results show that verifier-side expert selection benefits from adapting expert-set size to each verification block and expert eligibility to the current cache state.

References Agarwal, S.; Ahmad, L.; Ai, J.; Altman, S.; Applebaum, A.; Arbus, E.; Arora, R. K.; Bai, Y.; Baker, B.; Bao, H.; et al. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. Aghdam, M. A.; Jin, H.; and Wu, Y. 2024. Da-moe: Towards dynamic expert allocation for mixture-of-experts models. arXiv preprint arXiv:2409.06669. Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732. Cai, T.; Li, Y.; Geng, Z.; Peng, H.; Lee, J. D.; Chen, D.; and Dao, T. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774. Chen, H. M.; Luk, W.; Cedric, Y. K. F.; Li, R.; Mishchenko, K.; Venieris, S. I.; and Fan, H. 2025a. Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference. In EMNLP (Findings), 2221–2238. Chen, H. M.; Mo, Z.; Lee, R.; Wang, Q.; Li, D.; Hu, S. X.; Luk, W.; Hospedales, T.; and Fan, H. 2026. Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs. arXiv preprint arXiv:2602.00879. Chen, L.; Wen, Z.; Wu, T.; Zhang, X.; and Wu, C. 2025b. SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference. arXiv preprint arXiv:2510.10302. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. D. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Chen, T.; Huang, S.; Xie, Y.; Jiao, B.; Jiang, D.; Zhou, H.; Li, J.; and Wei, F. 2022. Task-specific expert pruning for sparse mixture-of-experts. arXiv preprint arXiv:2206.00277. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168. Dai, D.; Deng, C.; Zhao, C.; Xu, R.; Gao, H.; Chen, D.; Li, J.; Zeng, W.; Yu, X.; Wu, Y.; et al. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 1280–1297. Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Huang, W.; Liao, Y.; Liu, J.; He, R.; Tan, H.; Zhang, S.; Li, H.; Liu, S.; and Qi, X. 2025. Mixture compressor for mixtureof-experts llms gains more. In International Conference on Learning Representations, volume 2025, 68628–68648. Huang, Y.; Wang, Z.; Yuan, Z.; Ding, Y.; Gong, R.; Guo, J.; Liu, X.; and Zhang, J. 2026a. MoDES: Accelerating

Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 30205–30215. Huang, Z.; Zhu, L.; Zhan, Z.; Hu, T.; Mao, W.; Yu, X.; Liu, Y.; and Zhang, T. 2026b. MoESD: Unveil Speculative Decoding’s Potential for Accelerating Sparse MoE. Advances in Neural Information Processing Systems, 38: 125276–125311. Jiang, A. Q.; Sablayrolles, A.; Roux, A.; Mensch, A.; Savary, B.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Hanna, E. B.; Bressand, F.; et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. Leviathan, Y.; Kalman, M.; and Matias, Y. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, 19274–19286. PMLR. Li, Y.; Wei, F.; Zhang, C.; and Zhang, H. 2024a. Eagle2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 conference on empirical methods in natural language processing, 7421–7432. Li, Y.; Wei, F.; Zhang, C.; and Zhang, H. 2024b. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. In Proceedings of the 41st International Conference on Machine Learning, 28935–28948. PMLR. Li, Y.; Wei, F.; Zhang, C.; and Zhang, H. 2025. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. In Advances in Neural Information Processing Systems, volume 38, 136737–136756. Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2024. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, 39578– 39601. Lu, X.; Liu, Q.; Xu, Y.; Zhou, A.; Huang, S.; Zhang, B.; Yan, J.; and Li, H. 2024. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 6159–6172. McDanel, B.; Li, S.; Surineni, S.; and Khaitan, H. 2026. MoE-Spec: Expert Budgeting for Efficient Speculative Decoding. arXiv preprint arXiv:2602.16052. Oncescu, C.-A.; Wu, Q.; Chung, W. T.; Wu, R.; Gopal, B.; Wang, J.; Dao, T.; and Athiwaratkun, B. 2025. Opportunistic expert activation: Batch-aware expert routing for faster decode without retraining. arXiv preprint arXiv:2511.02237. Pan, L.; Tao, Z.; Pang, R.; Wang, X.; Zhao, J.; and Zhang, Y. 2026. Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding. arXiv preprint arXiv:2605.00342. Roy, O.; and Vetterli, M. 2007. The Effective Rank: A Measure of Effective Dimensionality. In Proceedings of the 15th European Signal Processing Conference (EUSIPCO), 606– 610.

Saxena, A.; Tsai, P.-A.; Taneja, H.; Jaleel, A.; and Qureshi, M. 2025. Utility-Driven Speculative Decoding for Mixtureof-Experts. arXiv preprint arXiv:2506.20675. Stern, M.; Shazeer, N.; and Uszkoreit, J. 2018. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31. Vankov, D.; Ivkin, N.; Ulrich, K.; Song, X.; Khetan, A.; and Karypis, G. 2026. XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference. arXiv preprint arXiv:2602.07265. Wang, W.; Liu, J.; Hou, X.; Xia, X.; Tang, P.; Zhang, M.; Li, C.; and Guo, M. 2025. MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts. arXiv preprint arXiv:2511.14102. Xie, J.; Liu, R.; Huang, H.; Ling, Y.; Dai, H.; Zheng, Y.; and Hu, W. 2026. Less Experts, Faster Decoding: CostAware Speculative Decoding for Mixture-of-Experts. arXiv preprint arXiv:2607.12696. Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Zhang, Z.; Liu, X.; Cheng, H.; Xu, C.; and Gao, J. 2025. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts. In Findings of the Association for Computational Linguistics: ACL 2025, 86–102. Zheng, L.; Yin, L.; Xie, Z.; Sun, C.; Huang, J.; Yu, C. H.; Cao, S.; Kozyrakis, C.; Stoica, I.; Gonzalez, J. E.; et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37: 62557–62583.

AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding

Analyze & explore related research in Conceptio

Related documents