Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
Jincheng Xie * 1 ‡ Runheng Liu * 2 ‡ Heyan Huang 2 Yawen Ling 3 Hanbin Dai 3 Yu Zheng 4 5 Wen Hu 3
arXiv:2607.12696v1 [cs.CL] 14 Jul 2026
Abstract
speed, achieving up to 1.62× speedup. These results show that accounting for expert activation cost is important for efficient speculative decoding in large-scale MoE models.
Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns. Speculative decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in parallel, yet existing draft selection strategies primarily optimize acceptance likelihood. In large-scale MoE models, however, selecting draft tokens also determines the union of experts activated during verification. We observe that confidence-driven SD can introduce expert scattering: high-probability draft tokens may route to disjoint experts, increasing expertweight memory traffic and reducing the speedup from speculation. Motivated by this observation, we revisit draft-tree selection under the nonuniform memory-cost structure of MoE inference. We propose E CO S PEC, a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection. With a lightweight expert predictor and a dynamic expert buffer, E CO S PEC favors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target-model verification rule. We evaluate E CO S PEC on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235BA22B, and GPT-OSS-120B, across reasoning, coding, question-answering, and dialogue benchmarks. E CO S PEC consistently reduces active expert footprints and improves end-to-end decoding
1. Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across logic, coding, and creative tasks (Brown et al., 2020; OpenAI et al., 2024; Grattafiori et al., 2024), yet their deployment is increasingly constrained by high inference latency and serving cost (Rajbhandari et al., 2020). The standard autoregressive decoding process generates tokens sequentially, where each step requires a full pass through the target model (Vaswani et al., 2017). As models scale to hundreds of billions of parameters, this serial dependency makes inference heavily memory-bound: the arithmetic intensity is low, and throughput is often limited by the bandwidth required to load model weights from High Bandwidth Memory (HBM) to on-chip compute units (Williams et al., 2009; Shazeer, 2019; Dao et al., 2022). Consequently, reducing decoding latency and memory traffic has become a critical priority for both real-time user experience and infrastructure efficiency. To mitigate the serial decoding bottleneck, Speculative Decoding (SD) has been widely studied as an inference acceleration paradigm (Leviathan et al., 2023; Chen et al., 2023; Li et al., 2024a;b; 2026; Cai et al., 2024). SD uses a lowcost draft mechanism to propose multiple candidate tokens, which are then verified in parallel by the target model. When the drafts are accepted, a single target-model forward pass can advance generation by multiple tokens, thereby amortizing parameter loading and improving hardware utilization. For dense Transformers, where the same weight matrices are reused across all verified positions, this parallel verification can substantially increase arithmetic intensity and reduce the effective cost per generated token (Miao et al., 2024).
* Equal contribution ‡ Work done while interning at JDT AI Infra. 1 Department of Mathematical Sciences, Tsinghua University, Beijing, China 2 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China 3 JDT AI Infra 4 School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, Sichuan, China 5 School of Cyber Engineering, Xidian University, Xi’an, Shaanxi, China. Correspondence to: Wen Hu <[email protected]>.
However, the dense-model assumption behind this amortization does not directly extend to sparse Mixture-of-Experts (MoE) architectures (Shazeer et al., 2017; Fedus et al., 2022), which are increasingly adopted in large-scale lan-
Preprint. July 15, 2026.
1
Submission and Formatting Instructions for ICML 2026 (a) Latency Scaling with Expert Load
0.8 0.6 0.4 0.2 4
8
12
16
20
24
Number of Active Experts ( )
32
(a) Latency vs. Expert Load
40
(c) Single Expert Memory Footprint 50
DeepSeek-V3.1 (Top-8) Qwen3-235B-A22B (Top-8) GPT-OSS-120B (Top-4)
35 30
44.04 MB
Memory per Expert (MB)
Cumulative Unique Experts ( total)
Forward Latency (s)
(b) Expert Activation Burden
DeepSeek-V3.1 Qwen3-235B-A22B GPT-OSS-120B
1.0
25 20 15 10 5
40
(FP8)
(BP16)
30 20
12.45 MB (MXFP4)
10 0
1
2
3
4
Number of Verification token ( )
(b) Expert Activation Burden
5
37.75 MB
.1
eek-V3
DeepS
22B
-235B-A
Qwen3
SS-120
GPT-O
B
(c) Single Expert Cost
Figure 1. The Bandwidth Bottleneck in MoE Speculative Decoding. (a) Verification latency scales linearly with the number of active experts (E), creating a strict latency penalty for retrieving extra experts. (b) Top-K means K activated experts in each layer. As the verification budget γ increases, standard baselines (e.g., Eagle) rapidly activate disjoint sets of experts, quickly saturating the memory bandwidth. (c) The physical memory footprint of a single expert is substantial. For DeepSeek-V3.1 (FP8), loading just one expert consumes 44.04 MB of HBM bandwidth, implying that every mispredicted expert path incurs a massive I/O overhead.
guage models (OpenAI et al., 2025; DeepSeek-AI et al., 2025; Yang et al., 2025). In MoE layers, the dense feedforward module is replaced by a pool of experts, and a router assigns each token to a small top-k subset of experts (Lepikhin et al., 2021; Du et al., 2022). Therefore, parallel verification no longer reuses a single fixed set of feed-forward weights across all verified positions. Its memory cost depends on the union of experts activated by the verified tokens: when different candidate tokens route to disjoint experts, the verifier must fetch additional expert weight blocks from HBM. As a result, verification latency in MoE speculative decoding becomes highly sensitive to expert overlap and reuse, rather than being determined mainly by the number of verified tokens.
with the marginal cost of introducing new experts. To address this issue, we propose E CO S PEC, a costaware speculative decoding framework for MoE models. E CO S PEC operates at the draft-tree selection stage, where it selects candidates for parallel verification under an acceptance–cost trade-off. Instead of ranking draft tokens primarily by acceptance likelihood, E CO S PEC also accounts for the marginal expert cost induced by each candidate. This allows the selection procedure to prefer draft paths that maintain high acceptance probability while reusing experts already covered by the current verification set. Importantly, E CO S PEC does not modify the target-model verification rule and therefore preserves the lossless semantics of standard speculative decoding (Leviathan et al., 2023). By aligning draft selection with the memory-cost structure of MoE inference, E CO S PEC reduces unnecessary expert-weight traffic during verification. Empirically, E CO S PEC achieves consistent speedups across multiple production-scale MoEs, including up to 1.62× on Qwen3-235B, 1.50× on GPTOSS-120B, and 1.47× on DeepSeek-V3.1.
This MoE-specific cost structure exposes a mismatch between the acceptance-driven selection objective of existing SD methods and the memory cost of MoE verification. Many recent SD methods organize draft candidates as a tree and select a verification subset primarily according to confidence or acceptance likelihood. This criterion is effective for dense models, where the main objective is to maximize the number of accepted tokens per target-model pass. In MoE verification, however, a high-probability candidate can still activate experts that are disjoint from those used by other verified tokens. Adding such a candidate may therefore enlarge the per-step expert union, increase expert-weight memory traffic, and reduce cache reuse (Huang et al., 2026; Xue et al., 2024). We refer to this expansion of the per-step expert footprint as expert scattering. As shown in Fig. 1, verification latency increases with the active expert footprint, while confidence-driven selection can rapidly increase the number of unique experts touched within one verification step. Consequently, expanding the verification set may erode end-to-end speedup even when acceptance rates remain comparable. This observation motivates a cost-aware draft selection objective that balances acceptance likelihood
Our contributions are summarized as follows: • We identify and analyze expert scattering in large-scale MoE speculative decoding: confidence-driven draft selection can expand the per-step union of activated experts, increasing expert-weight memory traffic during verification. • We propose E CO S PEC, a cost-aware speculative decoding framework that incorporates marginal expert activation cost into draft-tree selection. With a lightweight expert predictor and a dynamic expert buffer, E CO S PEC favors draft paths that preserve high acceptance likelihood while improving expert reuse, without changing the standard lossless verification procedure. 2
Submission and Formatting Instructions for ICML 2026
• We evaluate E CO S PEC on three large-scale MoE models—DeepSeek-V3.1 (671B), Qwen3-235BA22B, and GPT-OSS-120B—across diverse reasoning, coding, and dialogue benchmarks. E CO S PEC consistently reduces activated experts and improves end-to-end decoding speed, achieving up to 1.62× speedup over existing SD baselines.
time expert cost by imposing an expert budget and selecting only a subset of experts to load during speculative verification (McDanel et al., 2026). These methods either optimize runtime execution under predicted or routed expert demand, or change the verification-time expert budget. E CO S PEC addresses a different stage of the decoding pipeline: it preserves the standard target-model verifier and lossless speculative decoding semantics, and only changes which draft-tree nodes are selected before verification. Therefore, E CO S PEC is complementary to MoE runtime optimizations, since expert caching, prefetching, offloading, or optimized dispatch can still be applied after E CO S PEC reduces the expert working set induced by the selected verification nodes.
2. Related Work Speculative Decoding and Multi-Token Prediction. Speculative decoding (SD) accelerates LLM inference by using a lower-cost draft mechanism to propose multiple tokens, which are then verified in parallel by the target model (Leviathan et al., 2023; Chen et al., 2023). Early approaches commonly use a separate draft model to generate candidate continuations (Miao et al., 2024). Recent methods further improve draft efficiency and acceptance by adopting tree-structured drafting or auxiliary prediction heads, including Medusa and the EAGLE series (Cai et al., 2024; Li et al., 2024a;b; 2026). In parallel, Multi-Token Prediction (MTP) introduces an auxiliary training objective that enables in-model token predictors to serve as draft heads for speculative verification (DeepSeek-AI et al., 2025; Gloeckle et al., 2024). Despite architectural differences in how drafts are produced, these approaches typically prioritize draft tokens based on confidence (acceptance likelihood) and do not explicitly account for the hardware cost of activating additional experts in MoE verification. As a result, when applied to MoE models, high-confidence drafts may still trigger a rapidly expanding union of activated experts, corresponding to our Expert Scattering phenomenon and the associated memory inefficiency. EcoSpec makes this explicit by optimizing a cost-aware objective that balances acceptance likelihood against the incremental cost of activating new experts.
3. MoE Speculative Decode Bottleneck Speculative Decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in a single targetmodel forward pass. For dense Transformers, the memory cost of verifying γ tokens is close to that of verifying a single token, because all verified positions reuse the same dense weight matrices within the verification batch. Thus, increasing the verification budget mainly improves the amortization of target-model parameter loading and can reduce the effective cost per generated token. This premise becomes less reliable for sparse MoE models. For a MoE layer ℓ, let Sℓ (xt ) denote the set of experts activated by the t-th verified token. Verifying a draft sequence x1:γ requires accessing the union of experts activated by all verified tokens at that layer: ℓ Everify =
γ [
Sℓ (xt ).
(1)
t=1
The verification footprint is therefore determined by the size of this union, aggregated across MoE layers, rather than by the number of verified tokens alone. If selected draft tokens route to overlapping experts, verification can reuse expert weights. If they route to disjoint experts, the verifier must fetch additional expert weight blocks from HBM, increasing memory traffic within the same speculative step.
Efficient Inference for Mixture-of-Experts. Mixture-ofExperts (MoE) models increase model capacity while reducing per-token FLOPs, but their inference efficiency is often limited by expert-weight memory traffic and token– expert dispatch overheads (Shazeer et al., 2017; Liu et al., 2026). Prior work improves MoE execution under a given routing pattern through expert caching and prefetching (Xue et al., 2025; Huang et al., 2024), optimized token dispatch and fused MoE kernels (Gale et al., 2023), and routing or load-balancing strategies that reduce uneven expert utilization (Fedus et al., 2022; DeepSeek-AI et al., 2025). Recent systems further study MoE inference in the speculativedecoding setting. SP-MoE and MoE-SpeQ use speculative lookahead to support expert prefetching, offloading, and execution scheduling, aiming to hide or reduce expertmovement overhead during MoE serving (Chen et al., 2025; Wang et al., 2025). MoE-Spec instead reduces verification-
This creates an expert-scattering effect in MoE speculative decoding. Standard SD selection strategies are typically designed to maximize acceptance likelihood and are not aware of expert locality. As a result, adding more highconfidence draft tokens can enlarge the per-step expert union even when those tokens are likely to be accepted. In practice, ℓ |Everify | can grow rapidly with the verification budget γ, reducing expert reuse and creating an additional memorytraffic bottleneck. We profile DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B on GSM8K (Cobbe et al., 2021) under identical execution settings. Fig. 1(a) shows that verifica3
Submission and Formatting Instructions for ICML 2026
EcoSpec Bonus Token Token （Root） Embeddings
0.8
�3
�1
(0.6)
(0.48)
��
0.7
�7
(0.34)
0.6
0.1
(1.0)
0.2
0.7
�5
�4
0.6
(0.05)
(0.08)
(0.06) 0.1
�0
�8
�2
(0.2)
(0.14)
�9
0.2
Lightweight Expert Predictor (Πθ )
0.1
Global Expert Buffer �
Activated Expert Unctivated Expert
...
Layer1:{E1, ....}
...
Layer2:{E4, ....}
�6
(0.02)
...
�10
Token activate expert/ Update Buffer
(0.03)
Tree Struct & Probs
Draft Tree （Tokens & Probs）
Parallel Verification
... ...
Draft Generation
Cost-Aware Draft Selection
Selected Set �
��
...
The quick brown fox jumps over the lazy dog.
Verified Sequence
Reduce Activate Expert and Memory Tracffic
Figure 2. Overview of the E CO S PEC framework. The process begins with draft generation, where a draft model Md produces a tree of candidate tokens with associated probabilities. The E CO S PEC module then selects a set of draft tokens S for verification. It employs a lightweight expert predictor Πθ to estimate expert activations, maintains a global expert buffer B to track experts already covered by the selected tokens, and uses cost-aware draft selection to balance acceptance likelihood with expert activation cost. Finally, the target MoE model Mp performs parallel verification on the selected set S.
tion latency increases with the number of unique experts activated per step, indicating that fetching additional expert weights is a major source of verification cost. Fig. 1(b) shows that standard SD selection can rapidly increase the cumulative number of unique activated experts as more tokens are selected for verification. This means that a larger verification set provides more opportunities for token acceptance, but can also make each verification pass more expensive in MoE models.
experts each draft token is likely to activate in the target MoE model. It outputs per-layer expert distributions, and the top-K predicted experts are used as an approximation of the token’s activated expert set. (2) A Global Expert Buffer B that records experts already covered by the selected verification set. This buffer enables efficient marginal-cost estimation by counting only the newly introduced experts when a candidate is added. (3) A Cost-Aware Draft Selection algorithm that selects draft tokens from the draft tree under an acceptance–cost scoring rule. The score favors candidates with high cumulative draft probability and low marginal expert cost, encouraging selected paths to reuse experts already present in B.
The cost of this effect is substantial at large scale. As shown in Fig. 1(c), a single expert can occupy tens of megabytes of memory, so even a small increase in the per-layer expert union can translate into large HBM traffic. For DeepSeekV3.1, reducing the expected expert footprint by only 0.2 experts per layer per step decreases expert-weight traffic by approximately 0.5 GB per speculative step. These findings motivate E CO S PEC, which incorporates expert activation cost into draft selection to reduce the verification expert footprint while maintaining high acceptance likelihood.
Execution order. E CO S PEC runs after draft generation and before target-model verification. It first constructs a verification subset S of size γ using draft probabilities and predicted expert footprints, without querying the target router during this stage. During subset construction, the buffer B is updated from predicted path-level expert sets to estimate the marginal cost of the remaining candidates. Once S is constructed, the selected nodes are submitted to the standard speculative verifier and checked together in a single target-model forward pass.
4. Methodology: E CO S PEC To reduce verification memory traffic in MoE speculative decoding, E CO S PEC introduces a cost-aware selection mechanism between draft generation and target-model verification. Given a draft tree with candidate probabilities, E CO S PEC selects γ draft tokens for parallel verification by considering both acceptance likelihood and predicted expert activation cost. As shown in Figure 2, the framework consists of three components:
4.1. Lightweight Expert Predictor Cost-aware selection requires estimating the expert footprint of each draft token before target-model verification. Directly querying the target router for every draft candidate would
(1) A lightweight Expert Predictor Πθ that estimates the 4
Submission and Formatting Instructions for ICML 2026 draft token Buffer token Select token Buffer expert
�1
�3 �4 �7
2 token select
2 token select
�6
�8
�1 (0.53)
�2
�1
�4
�0 (1.0)
�3 �3 �5
�4
�4 �7
�6
� = {}, � = {}
�Cost(�0 |�) = {�1 , �2 , �3 , �4 } �Cost(�1 |�) = {�1 , �2 , �3 , �4 , �6 , �7 , �8 } �Cost(�2 |�) = {�1 , �2 , �3 , �4 , �5 , �6 } �(�0 ) = 0.25, �(�1 ) = 0.075, �(�2 ) = 0.066
�3
�4
�3 �5
�8
�1 (0.53)
�2 (0.4)
�1
�2
�0 (1.0)
�6
2 token select
�4
�4
�7
�6
�6
�2
�4
�0 (1.0)
�1 (0.53)
�2 (0.4)
�3 �5
�8
�4
�6
�2 (0.4)
� = {�1 , �2 , �3 , �4 , �5 , �6 } return:� = {�0 , �2 }
� = {�0 }, � = {�1 , �2 , �3 , �4 } �Cost(�1 |�) = {�6 , �7 , �8 } �Cost(�2 |�) = {�5 , �6 } �(�1 ) = 0.17, �(�2 ) = 0.2
Figure 3. Illustration of Pre-verification Cost-Aware Subset Construction. We select γ = 2 draft tokens before target-model verification. Left (Step 1): the root token t0 is selected according to the acceptance–cost score, and its predicted expert footprint {E1 , E2 , E3 , E4 } is added to the expert buffer B for subsequent scoring. Middle (Step 2): the algorithm re-evaluates the remaining candidates using the updated buffer. Although t1 has higher cumulative draft probability (P = 0.53), it introduces three new predicted experts (∆Cost = 3). In contrast, t2 reuses experts already covered by B and introduces only two new predicted experts (∆Cost = 2), so it obtains a higher score (S = 0.20 > 0.17) and is selected. Right (Final): the final selected set is S = {t0 , t2 }, illustrating how marginal expert cost can change the ranking induced by cumulative draft probability.
layer ℓ, let Egt (t, ℓ) denote the top-K experts selected by the target router. We define a normalized target distribution qt,ℓ ∈ [0, 1]E by assigning uniform probability mass to the activated experts:
require running the target MoE model, which would offset the benefit of speculative decoding. We therefore train a lightweight expert predictor Πθ to approximate target-model routing and provide a low-cost estimate of expert activation.
(
Architecture. We instantiate Πθ with a small decoderonly backbone. For compatibility with the target MoE model Mp , each input token is first mapped by the target embedding layer and then projected into the predictor hidden space. Given a draft token ti with context t<i , the predictor outputs routing logits zti = Πθ (ti | t<i ) ∈ RL×E ,
qt,ℓ,e =
p̂ti ,ℓ = Softmax(zti ,ℓ ) ∈ R ,
0,
e ∈ Egt (t, ℓ), otherwise.
(5)
We train Πθ with layer-wise cross-entropy: Lpred (θ) = −
(2)
L X E XX t
where L is the number of MoE layers and E is the number of experts per layer. We obtain a routing distribution for each MoE layer by applying softmax over experts: E
1 K,
qt,ℓ,e log p̂t,ℓ,e .
(6)
ℓ=1 e=1
4.2. Cost-Aware Draft Selection
ℓ ∈ {1, . . . , L}. (3)
The predicted expert set is then constructed as a set of layer– expert pairs: Epred (ti ) = {(ℓ, e) | e ∈ TopK(p̂ti ,ℓ , K) , ℓ ∈ {1, . . . , L}} . (4) Training Objective. During predictor training, we run the target MoE model offline and record the ground-truth activated experts for each token. For token t and MoE 5
Given the predicted expert sets from § 4.1, E CO S PEC selects draft-tree nodes for verification under an acceptance–cost trade-off. The selection is path-based: each candidate is scored using the cumulative draft probability and predicted expert footprint of its root-to-node path. Existing draft-tree methods primarily rank candidates by confidence or acceptance likelihood (Li et al., 2024b). For MoE verification, however, a high-confidence candidate can still introduce many new experts if its routing footprint has little overlap with the candidates already selected. Therefore, E CO S PEC augments confidence-based selection with the marginal expert cost induced by each candidate.
Submission and Formatting Instructions for ICML 2026
Selection State. Let T denote the draft tree, where each node ti corresponds to a candidate token. During selection, E CO S PEC maintains a selected set S and an expert buffer B. The selected set S contains tokens chosen for verification, while B contains the experts already covered by the current selected set. The buffer is used to measure how many new experts a candidate would introduce beyond those already covered.
to extend high-probability paths that also preserve expert locality, rather than expanding the draft tree solely by confidence. The full selection procedure is provided in Appendix Alg. 1.
5. Experiments 5.1. Setup Models and Benchmarks. We evaluate E CO S PEC on three large-scale MoE models with different sizes and routing configurations: DeepSeek-V3.1 (DeepSeek-AI et al., 2025) (671B total / 37B active, Top-8), Qwen3-235B-A22B (Yang et al., 2025) (235B total / 22B active, Top-8), and GPT-OSS-120B (OpenAI et al., 2025) (120B total / 5.1B active, Top-4). We use seven benchmarks covering mathematical reasoning, code generation, question answering, and dialogue: GSM8K (Cobbe et al., 2021), HumanEval (Chen et al., 2021), AIME-25 (MAA, 2025), Math500 (Hendrycks et al., 2021; Lightman et al., 2024), AMC22-24 (AI-MO), MTBench (Zheng et al., 2023), and MMStar (Chen et al., 2024).
Path-Dependent Expert Footprint. Because draft-tree verification is prefix-dependent, selecting a node requires covering its path from the root. For a candidate node ti , let Path(root → ti ) denote the sequence of nodes from the root to ti . The predicted expert footprint of this path is [ Etraj (ti ) = Epred (τ ). (7) τ ∈Path(root→ti )
The marginal expert cost of selecting ti under the current buffer is then defined as ∆Cost(ti | B) = |Etraj (ti ) \ B| .
(8)
This cost counts only the additional experts that are not already covered by previously selected candidates.
Baselines and Hardware. We compare E CO S PEC with autoregressive (AR) decoding and the corresponding speculative decoding baseline for each target model: MTP for DeepSeek-V3.1, and EAGLE-3 for Qwen3-235B-A22B and GPT-OSS-120B1 . All experiments are conducted on a server with 8 NVIDIA H200 GPUs.
Acceptance–Cost Scoring. Let P (ti ) be the cumulative draft probability along the path from the root to ti . E CO S PEC scores each candidate by S(ti ) =
P (ti ) , ∆Cost(ti | B) + ϵ
(9)
Speculative Decoding Configuration. To isolate the effect of cost-aware selection, we keep the draft-generation configuration fixed between each speculative baseline and its E CO S PEC variant. The draft process runs for 3 forward steps and keeps the top-2 tokens at each step to construct the draft tree. The verification budget γ is set to 4, where γ denotes the total number of tokens verified per speculative step, including the bonus token. This setting follows the operating regime of the large-scale MoE speculative baselines evaluated in this work, and the same configuration is used for the baseline and E CO S PEC to ensure a controlled comparison. Appendix C further evaluates different verification budgets and shows that γ = 4 gives the highest average end-to-end speedup in our setting. All main experiments use batch size 1 unless explicitly stated otherwise. Table 1 reports both greedy target decoding (T =0) and samplingbased target decoding (T =1), following common speculative decoding evaluation practice (Li et al., 2024a). For the latency breakdown and HBM-traffic analysis in § 5.3, we use the greedy target-decoding rows of Table 1, i.e.,
where ϵ avoids division by zero. This score favors candidates with high acceptance likelihood while penalizing those that introduce many new experts. Equivalently, it encourages the selected verification set to reuse experts already present in B, thereby limiting the growth of the per-step expert union. Prefix Consistency. The score is naturally compatible with the prefix structure of the draft tree. For a parent node tp and its child tc , the cumulative probability satisfies P (tc ) = P (tp )P (tc | tp ) ≤ P (tp ), while the path footprint satisfies Etraj (tp ) ⊆ Etraj (tc ). Therefore, ∆Cost(tp | B) ≤ ∆Cost(tc | B). Under the same buffer B, a child node therefore cannot receive a higher score than its parent except in tie cases. Thus, the scoring rule is aligned with prefix-closed draft-tree verification without requiring an additional structural penalty. Dynamic Buffer Effect. After a candidate path is selected, its predicted experts are added to B. This update reduces the marginal cost of later candidates that share experts with the selected path, increasing their scores without changing their draft probabilities. As a result, E CO S PEC tends
1 https://huggingface.co/lmsys/ Qwen3-235B-A22B-EAGLE3 https://huggingface.co/lmsys/ EAGLE3-gpt-oss-120b-bf16.
6
and
Submission and Formatting Instructions for ICML 2026 Table 1. Main results across target models, benchmarks, and decoding temperatures. Each entry reports end-to-end speedup relative to AR decoding / mean acceptance length α / average active experts E. Here, E denotes the average number of unique experts activated per MoE layer within one verification step. Rows are grouped by target MoE model and target-model decoding temperature. Method
GSM8K Spd↑ / Len↑ / Exp↓
HumanEval Spd↑ / Len↑ / Exp↓
MMStar Spd↑ / Len↑ / Exp↓
AIME-25 Spd↑ / Len↑ / Exp↓
Math500 Spd↑ / Len↑ / Exp↓
AMC22-24 Spd↑ / Len↑ / Exp↓
MTBench Spd↑ / Len↑ / Exp↓
Average Spd↑ / Len↑ / Exp↓
Qwen3-235B-A22B (Total 235B, Active 22B, Top-8) Temperature = 0 (Greedy) AR 1.00× / - / 8.0 1.31 / 2.54 / 22.3 Eagle-3 EcoSpec 1.39 / 2.54 / 21.0 Temperature = 1 (Sampling) AR 1.00× / - / 8.0 1.28 / 2.59 / 22.8 Eagle-3 EcoSpec 1.37 / 2.45 / 20.3
1.00× / - / 8.0 1.11 / 2.35 / 26.2 1.33 / 2.32 / 21.6
1.00× / - / 8.0 1.35 / 2.64 / 20.5 1.57 / 2.58 / 18.4
1.00× / - / 8.0 1.14 / 2.41 / 22.1 1.23 / 2.37 / 20.9
1.00× / - / 8.0 1.05 / 1.96 / 25.4 1.13 / 1.88 / 19.4
1.00× / - / 8.0 1.12 / 2.20 / 25.4 1.28 / 2.10 / 22.2
1.00× / - / 8.0 1.46 / 2.78 / 24.0 1.62 / 2.44 / 19.9
1.00× / - / 8.0 1.22 / 2.41 / 23.7 1.36 / 2.32 / 20.5
1.00× / - / 8.0 1.17 / 2.21 / 24.3 1.27 / 2.20 / 21.4
1.00× / - / 8.0 1.49 / 2.88 / 22.4 1.62 / 2.74 / 19.9
1.00× / - / 8.0 1.13 / 2.26 / 21.8 1.22 / 2.23 / 20.7
1.00× / - / 8.0 1.17 / 2.19 / 24.9 1.25 / 2.13 / 20.8
1.00× / - / 8.0 1.28 / 2.25 / 25.0 1.32 / 2.20 / 22.20
1.00× / - / 8.0 1.45 / 2.58 / 26.5 1.61 / 2.56 / 21.3
1.00× / - / 8.0 1.28 / 2.42 / 24.0 1.38 / 2.36 / 20.9
GPT-OSS-120B (Total 120B, Active 5.1B, Top-4) Temperature = 0 (Greedy) 1.00× / - / 4.0 AR 1.05 / 1.56 / 11.9 Eagle-3 EcoSpec 1.11 / 1.52 / 10.6 Temperature = 1 (Sampling) AR 1.00× / - / 4.0 1.10 / 1.90 / 11.8 Eagle-3 EcoSpec 1.15 / 1.85 / 10.3
1.00× / - / 4.0 1.05 / 1.72 / 11.8 1.26 / 1.72 / 10.7
1.00× / - / 4.0 1.28 / 1.91 / 11.7 1.45 / 1.90 / 10.5
1.00× / - / 4.0 1.17 / 2.02 / 12.2 1.33 / 2.02 / 10.8
1.00× / - / 4.0 1.14 / 2.02 / 12.0 1.27 / 2.00 / 10.6
1.00× / - / 4.0 1.14 / 2.02 / 11.9 1.45 / 2.02 / 10.9
1.00× / - / 4.0 1.18 / 1.91 / 10.3 1.30 / 1.81 / 9.6
1.00× / - / 4.0 1.14 / 1.88 / 11.6 1.31 / 1.86 / 10.6
1.00× / - / 4.0 1.27 / 2.13 / 11.8 1.34 / 2.06 / 10.8
1.00× / - / 4.0 1.31 / 2.20 / 11.3 1.50 / 2.15 / 10.5
1.00× / - / 4.0 1.19 / 2.13 / 12.1 1.27 / 2.11 / 11.5
1.00× / - / 4.0 1.17 / 2.08 / 12.1 1.27 / 2.06 / 10.8
1.00× / - / 4.0 1.17 / 2.22 / 12.1 1.36 / 2.08 / 11.4
1.00× / - / 4.0 1.07 / 2.17 / 12.4 1.23 / 2.06 / 11.1
1.00× / - / 4.0 1.18 / 2.12 / 11.9 1.30 / 2.05 / 10.9
DeepSeek-V3.1 (Total 671B, Active 37B, Top-8) Temperature = 0 (Greedy) AR 1.00× / - / 8.0 MTP 1.13 / 3.19 / 31.9 EcoSpec 1.19 / 3.13 / 31.7 Temperature = 1 (Sampling) AR 1.00× / - / 8.0 MTP 1.22 / 3.72 / 31.8 EcoSpec 1.35 / 3.53 / 31.6
1.00× / - / 8.0 1.06 / 2.82 / 31.8 1.10 / 2.80 / 31.6
1.00× / - / 8.0 1.09 / 2.61 / 31.5 1.13 / 2.55 / 31.2
1.00× / - / 8.0 1.05 / 3.13 / 31.3 1.08 / 3.03 / 31.2
1.00× / - / 8.0 1.12 / 2.73 / 31.1 1.19 / 2.64 / 30.9
1.00× / - / 8.0 1.20 / 3.05 / 31.5 1.28 / 3.00 / 31.3
1.00× / - / 8.0 1.04 / 2.41 / 31.1 1.09 / 2.38 / 30.9
1.00× / - / 8.0 1.10 / 2.85 / 31.4 1.15 / 2.79 / 31.2
1.00× / - / 8.0 1.29 / 3.39 / 31.9 1.31 / 3.32/ 30.9
1.00× / - / 8.0 1.28 / 3.37 / 31.7 1.35 / 3.36 / 31.4
1.00× / - / 8.0 1.26 / 3.49 / 31.6 1.30 / 3.43 / 31.3
1.00× / - / 8.0 1.24 / 3.61 / 31.1 1.29 / 3.61 / 30.8
1.00× / - / 8.0 1.44 / 3.68 / 31.1 1.47 / 3.59 / 30.1
1.00× / - / 8.0 1.20 / 3.15 / 30.8 1.23 / 3.04 / 30.5
1.00× / - / 8.0 1.28 / 3.49 / 31.4 1.33 / 3.41 / 30.9
Table 2. Estimated HBM read traffic during verification. We report the estimated total HBM read bytes during the verification phase of one speculative step.
batch size 1, γ = 4, and T =0. We further evaluate larger batch sizes, different verification budgets γ, and different predictor accuracy levels in Appendix A, Appendix C, and Appendix B.2, respectively.
Expert Predictors. E CO S PEC reuses the draft-generation infrastructure of the corresponding speculative baseline and only modifies the draft selection stage. For expert-cost estimation, we use DeepSeek-R1-Distill-Qwen-1.5B (Guo et al., 2025) as the predictor backbone for DeepSeek-V3.1, and Qwen3-0.6B (Yang et al., 2025) for Qwen3-235B-A22B and GPT-OSS-120B. Predictor training details are provided in Appendix B.1.
Model
Baseline
Baseline HBM (GB)
E CO S PEC HBM (GB)
Reduction (GB / %)
Qwen3-235B-A22B GPT-OSS-120B DeepSeek-V3.1
EAGLE-3 EAGLE-3 MTP
99.3 6.0 97.3
88.1 5.5 96.8
11.2 / 11.3% 0.5 / 8.0% 0.5 / 1.0%
5.2. Main Results Table 1 reports end-to-end speedup relative to autoregressive decoding, mean acceptance length α, and the average number of unique experts E activated per MoE layer within one verification step. Across three MoE backbones and seven benchmarks, E CO S PEC improves decoding speed over the corresponding speculative baseline while consistently reducing the expert footprint. The acceptance lengths remain close to the baselines, indicating that the speedup mainly comes from reducing verification cost rather than increasing the number of accepted tokens.
Inference Backend. The main experiments are conducted with a HuggingFace Transformers (Wolf et al., 2020) research prototype following the EAGLE-3 inference pipeline. We use this prototype to keep the draft-generation, treeverification, and KV-cache update workflow consistent with the released EAGLE-3 implementation, so that the comparison isolates the effect of E CO S PEC’s cost-aware draft selection. During decoding, the draft model constructs the draft tree, and the selected draft nodes are verified by the target MoE model in a single forward pass with a tree-structured attention mask. E CO S PEC only adds the expert predictor and cost-aware selection before this verification step; the target-model verification rule is unchanged.
Greedy decoding (T =0). Under greedy decoding, E CO S PEC consistently improves speedup across all three MoE models. For Qwen3-235B-A22B, E CO S PEC improves the average speedup from EAGLE-3’s 1.22× to 1.36×, while reducing E from 23.7 to 20.5. The largest gain appears on MTBench, where E CO S PEC reaches 1.62× speedup. For GPT-OSS-120B, E CO S PEC increases the average speedup from 1.14× to 1.31× and reduces E from 11.6 to 10.6. On 7
Submission and Formatting Instructions for ICML 2026 Table 3. Latency breakdown and throughput analysis. We report the average wall-clock time in seconds per speculative step. E denotes the average number of unique experts activated per MoE layer within one verification step. Tpred , Tdraft , Tverify , and Ttotal denote predictor overhead, draft-generation time, target verification time, and total speculative-step latency, respectively. α denotes the mean accepted tokens per step. The speedup column reports the corresponding end-to-end throughput speedup under the same T = 0 setting as Table 1. The latency-breakdown columns are reported to explain the sources of the end-to-end speedup.
Model
Qwen3-235B-A22B
GPT-OSS-120B
DeepSeek-V3.1
Method
Experts (E) ↓
Predictor (Tpred )
Drafting (Tdraft )
Verify (Tverify ) ↓
Total Step (Ttotal ) ↓
Accept Len (α) ↑
Throughput Speedup ↑
Baseline (AR) EAGLE-3 E CO S PEC
8.0 23.7 20.5
– – 0.004s
– 0.008s 0.008s
– 0.832s 0.730s
0.490s† 0.840s 0.742s
1.00 2.41 2.32
1.00× 1.22× 1.36×
∆ vs. EAGLE-3
-3.2
+0.004s
0.000s
-0.102s
-0.098s
-0.09
+0.14×
†
Baseline (AR) EAGLE-3 E CO S PEC
4.0 11.6 10.6
– – 0.004s
– 0.035s 0.035s
– 0.113s 0.090s
0.081s 0.148s 0.129s
1.00 1.88 1.86
1.00× 1.14× 1.31×
∆ vs. EAGLE-3
-1.0
+0.004s
0.000s
-0.023s
-0.019s
-0.02
+0.17×
Baseline (AR) MTP E CO S PEC
8.0 31.4 31.2
– – 0.004s
– 0.008s 0.008s
– 0.980s 0.930s
0.369s† 0.988s 0.942s
1.00 2.85 2.79
1.00× 1.10× 1.15×
∆ vs. MTP
-0.2
+0.004s
0.000s
-0.050s
-0.046s
-0.06
+0.05×
†
AR baseline time is per-token latency. Speculative methods report per-step latency and generate α accepted tokens per step. Delta rows compare E CO S PEC with the corresponding speculative baseline.
MMStar and AMC22-24, the speedup reaches 1.45×. For DeepSeek-V3.1, E CO S PEC improves the average speedup from MTP’s 1.10× to 1.15×, while reducing E from 31.4 to 31.2.
5.3. Latency Breakdown and Overhead Analysis To understand why E CO S PEC improves performance, we analyze the HBM traffic and latency composition of a speculative decoding step in Table 2 and Table 3. Following the evaluation scope defined in § 5.1, this subsection analyzes the latency and HBM-traffic behavior of the T =0 runs. We decompose the total speculative-step latency into three parts: (1) draft generation (Tdraft ), (2) expert prediction overhead introduced by E CO S PEC (Tpred ), and (3) target-model verification of the selected draft tokens (Tverify ). E CO S PEC does not reduce the computation required by each activated expert. Instead, its speedup comes from reducing expertweight memory traffic during verification.
Sampling decoding (T =1). Under sampling, draft candidates become more diverse, but E CO S PEC continues to reduce expert activation and improve speedup. For Qwen3235B-A22B, the average speedup increases from EAGLE3’s 1.28× to 1.38×, while E decreases from 24.0 to 20.9. For GPT-OSS-120B, the average speedup improves from 1.18× to 1.30×, with E reduced from 11.9 to 10.9. For DeepSeek-V3.1, E CO S PEC improves the average speedup from MTP’s 1.28× to 1.33× and reduces E from 31.4 to 30.9. These results show that the acceptance–cost trade-off remains effective under both greedy and sampling-based target decoding regimes.
HBM Read Traffic. We estimate HBM read traffic during verification from the activated expert footprint and the model-specific expert size. As shown in Table 2, E CO S PEC reduces the estimated HBM reads from 99.3 GB to 88.1 GB on Qwen3-235B-A22B, saving 11.2 GB per speculative step. For GPT-OSS-120B, the estimated reads decrease from 6.0 GB to 5.5 GB. For DeepSeek-V3.1, the reduction is smaller, from 97.3 GB to 96.8 GB, which is consistent with its smaller expert-reuse headroom. These reductions correspond to the lower verification latency reported in Table 3.
Backbone-dependent gains. The magnitude of the gain varies across MoE backbones. Qwen3-235B-A22B and GPT-OSS-120B show larger reductions in active experts, suggesting more opportunity for expert reuse during draft selection. DeepSeek-V3.1 shows smaller expert-footprint reductions, which is consistent with its more balanced routing pattern analyzed in Appendix D. Nevertheless, because DeepSeek-V3.1 has a large per-expert memory footprint, even a reduction of 0.2 experts per layer can translate into approximately 0.5 GB less expert-weight traffic per speculative step. Thus, E CO S PEC remains beneficial even when the available expert-reuse headroom is smaller.
Verification Latency and Predictor Overhead. Table 3 shows that E CO S PEC reduces verification latency across all three MoE models while adding only a small predic8
Submission and Formatting Instructions for ICML 2026 Table 4. Ablation study on GSM8K. We compare the static globalcost variant with E CO S PEC’s marginal-cost scoring. Speedup ↑
Len (α) ↑
Avg. E ↓
Global Cost E CO S PEC
1.28× 1.39×
2.21 2.54
21.5 21.0
GPT-OSS-120B
Global Cost E CO S PEC
1.06× 1.11×
1.35 1.52
10.8 10.6
DeepSeek-V3.1
Global Cost E CO S PEC
1.16× 1.19×
2.95 3.13
31.8 31.7
Model
Strategy
Qwen3-235B-A22B
Table 5. Additional baseline evaluation with GTO. We report end-to-end speedup and acceptance length for GTO and GTO+E CO S PEC. GSM8K HumanEval GSM8K HumanEval
GTO Spd.
GTO Len
GTO+EcoSpec Spd.
GTO+EcoSpec Len
1.28× 1.13× 1.03× 1.08×
2.50 2.41 1.53 1.77
1.34× 1.29× 1.08× 1.27×
2.50 2.37 1.50 1.76
For this experiment, we start from the released EAGLE-3 draft model and continue training it with the GTO procedure. The resulting GTO-trained drafter is used to generate candidate trees under the standard speculative verification workflow. We then evaluate GTO+E CO S PEC by applying the same cost-aware draft-selection strategy at the verification stage.
5.4. Ablation Study: Impact of Marginal Cost Scoring We study the effect of marginal-cost scoring on GSM8K across all three target models. We compare E CO S PEC with a static Global Cost variant, where each node is scored by the accumulated predicted expert footprint along its path: P (ti ) . |Etraj (ti )| + ϵ
Dataset
Qwen3-235B-A22B Qwen3-235B-A22B GPT-OSS-120B GPT-OSS-120B
model: MTP for DeepSeek-V3.1, and EAGLE-3 for Qwen3235B-A22B and GPT-OSS-120B. To provide an additional baseline comparison, we further include Group Tree Optimization (GTO) (Hu et al., 2026), which improves EAGLEstyle draft models by better aligning draft training with treebased decoding. GTO and E CO S PEC act on different stages of the speculative decoding pipeline: GTO improves draft generation, while E CO S PEC changes the pre-verification selection of draft nodes.
tor overhead. On Qwen3-235B-A22B, the average active expert count decreases from 23.7 to 20.5, and Tverify decreases from 0.832s to 0.730s. On GPT-OSS-120B, the active expert count decreases from 11.6 to 10.6, and Tverify decreases from 0.113s to 0.090s. On DeepSeek-V3.1, the expert-footprint reduction is smaller (31.4 → 31.2), but Tverify still decreases from 0.980s to 0.930s. Across all three models, the predictor overhead is about 4ms per speculative step, which is small relative to the reduction in verification latency. The acceptance length changes only slightly, so the speedup is mainly explained by reduced verification cost rather than higher acceptance length.
Sglobal (ti ) =
Model
As shown in Table 5, GTO+E CO S PEC achieves higher endto-end speedup than GTO on all evaluated settings. This additional baseline comparison further supports the effectiveness of E CO S PEC.
(10)
6. Conclusion
Unlike E CO S PEC, this variant does not update the expert buffer during selection and therefore cannot discount experts that are already covered by previously selected nodes. As a result, deeper nodes tend to receive larger accumulated costs even when they reuse experts from earlier selected paths.
We presented E CO S PEC, a cost-aware speculative decoding framework for large-scale MoE models. E CO S PEC addresses expert scattering during speculative verification by incorporating predicted marginal expert activation cost into draft-tree selection. With a lightweight expert predictor and a dynamic expert buffer, E CO S PEC selects draft tokens that preserve acceptance likelihood while reducing the growth of the verification expert footprint, without modifying the target-model verification rule. Experiments on DeepSeek-V3.1, Qwen3-235B-A22B, and GPT-OSS-120B show consistent speedups and reduced active experts across reasoning, coding, and dialogue benchmarks, with up to 1.62× speedup. These results highlight the importance of accounting for MoE-specific expert activation costs when applying speculative decoding to large-scale sparse models.
Table 4 shows that marginal-cost scoring consistently improves speedup over the static global-cost variant. On Qwen3-235B-A22B, E CO S PEC increases speedup from 1.28× to 1.39× and mean acceptance length from 2.21 to 2.54, while reducing the average active experts from 21.5 to 21.0. A similar pattern appears on GPT-OSS-120B, where speedup improves from 1.06× to 1.11× and acceptance length increases from 1.35 to 1.52. On DeepSeek-V3.1, the gain is smaller but still consistent. These results indicate that updating the buffer during selection helps E CO S PEC identify candidates that extend accepted paths while reusing already covered experts.
References AI-MO. Aimo validation amc dataset. URL https://huggingface.co/datasets/ AI-MO/aimo-validation-amc. Hugging Face.
5.5. Additional Baseline Evaluation The main experiments compare E CO S PEC with the corresponding speculative decoding baselines for each target 9
Submission and Formatting Instructions for ICML 2026
Brown, T., Mann, B., Ryder, N., Subbiah, M., Katechnical report, 2025. URL https://arxiv.org/ plan, J. D., Dhariwal, P., Neelakantan, A., Shyam, abs/2412.19437. P., Sastry, G., Askell, A., et al. Language models Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, are few-shot learners. In Larochelle, H., Ranzato, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Fedus, L., Bosma, M. P., Zhou, Z., Wang, T., Wang, E., Advances in Neural Information Processing Systems, Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, volume 33, pp. 1877–1901. Curran Associates, Inc., K., Duke, T., Dixon, L., Zhang, K., Le, Q., Wu, Y., Chen, 2020. URL https://proceedings.neurips. Z., and Cui, C. GLaM: Efficient scaling of language modcc/paper_files/paper/2020/file/ els with mixture-of-experts. In Chaudhuri, K., Jegelka, S., 1457c0d6bfcb4967418bfb8ac142f64a-Paper. Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), pdf. Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., Learning Research, pp. 5547–5569. PMLR, 17–23 Jul and Dao, T. Medusa: Simple LLM inference accelera2022. URL https://proceedings.mlr.press/ tion framework with multiple decoding heads. In Fortyv162/du22c.html. first International Conference on Machine Learning, 2024. URL https://openreview.net/forum? Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: id=PEpbUobfJv. scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), January 2022. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, ISSN 1532-4435. L., and Jumper, J. Accelerating large language model decoding with speculative sampling, 2023. URL https: //arxiv.org/abs/2302.01318.
Gale, T., Narayanan, D., Young, C., and Zaharia, M. MegaBlocks: Efficient Sparse Training with Mixture-ofExperts. Proceedings of Machine Learning and Systems, 5, 2023.
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., and Zhao, F. Are we on the right way for evaluating large vision-language models? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview.net/forum?id=evP9mxNNxJ.
Gloeckle, F., Idrissi, B. Y., Rozière, B., Lopez-Paz, D., and Synnaeve, G. Better & faster large language models via multi-token prediction. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.
Chen, L., Wen, Z., Wu, T., Zhang, X., and Wu, C. Sp-moe: Speculative decoding and prefetching for accelerating moe-based model inference, 2025. URL https:// arxiv.org/abs/2510.10302.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code, 2021. URL https://arxiv.org/ abs/2107.03374. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.14168. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. Flashattention: Fast and memory-efficient exact attention with IO-awareness. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: //openreview.net/forum?id=H4DqfPSibmx.
Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http://dx. doi.org/10.1038/s41586-025-09422-z. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum? id=7Bywt2mQsCe. Hu, S., Li, J., Lu, Z., and Zhou, P. Bridging draft policy misalignment: Group tree optimization for speculative decoding. In The Fourteenth International Conference on Learning Representations, 2026. URL https:// openreview.net/forum?id=dwPdYFqVWO.
DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. Deepseek-v3 10
Submission and Formatting Instructions for ICML 2026
Huang, H., Ardalani, N., Sun, A., Ke, L., Bhosale, S., Lee, H.-H. S., Wu, C.-J., and Lee, B. Toward efficient inference for mixture of experts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum? id=stXtBqyTWX.
Liu, J., Tang, P., Wang, W., Ren, Y., Hou, X., Heng, P. A., Guo, M., and Li, C. A survey on inference optimization techniques for mixture of experts models. ACM Comput. Surv., 58(10), March 2026. ISSN 0360-0300. doi: 10.1145/3794845. URL https://doi.org/10. 1145/3794845.
Huang, Z., Zhu, L., Zhan, Z., Hu, T., Mao, W., Yu, X., Liu, Y., and Zhang, T. MoESD: Unveil speculative decoding’s potential for accelerating sparse moe. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/ forum?id=FAeU7516MR.
MAA. American invitational mathematics examination AIME, 2025. URL https://huggingface.co/ datasets/opencompass/AIME2025. Hugging Face.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. {GS}hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL https://openreview. net/forum?id=qrwe7XHTmYb. Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023. Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Forty-first International Conference on Machine Learning, 2024a. URL https://openreview. net/forum?id=1NdN7eXyb4. Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE-2: Faster inference of language models with dynamic draft trees. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7421–7432, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main. 422. URL https://aclanthology.org/2024. emnlp-main.422/. Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum? id=4exx1hUffq.
McDanel, B., Li, S., Surineni, S., and Khaitan, H. Moe-spec: Expert budgeting for efficient speculative decoding, 2026. URL https://arxiv.org/abs/2602.16052. Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., Shi, C., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS ’24, pp. 932–949, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703867. doi: 10.1145/3620666.3651335. URL https://doi. org/10.1145/3620666.3651335. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774. OpenAI, :, Agarwal, S., Ahmad, L., Ai, J., Altman, S., Applebaum, A., Arbus, E., Arora, R. K., Bai, Y., et al. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’20. IEEE Press, 2020. ISBN 9781728199986. Shazeer, N. Fast transformer decoding: One write-head is all you need, 2019. URL https://arxiv.org/ abs/1911.02150.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=v8L0pN6EOi.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum? id=B1ckMDqlg. 11
Submission and Formatting Instructions for ICML 2026
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, Datasets and Benchmarks Track, 2023. URL https: L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. //openreview.net/forum?id=uccHPGDlao. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Algorithm 1 Cost-Aware Draft Selection in E CO S PEC and Garnett, R. (eds.), Advances in Neural Information Input: Draft Tree T , Budget γ, Predictor Πθ , Initial Processing Systems, volume 30. Curran Associates, Inc., Buffer B0 2017. URL https://proceedings.neurips. Output: Selected tokens S cc/paper_files/paper/2017/file/ S←∅ 3f5ee243547dee91fbd053c1c4a845aa-Paper. B ← B0 pdf. Construct the verification subset before target-model verWang, W., Liu, J., Hou, X., Xia, X., Tang, P., Zhang, M., ification. Li, C., and Guo, M. Moe-speq: Speculative quantized while |S| < γ do decoding with proactive expert prefetching and offloading for each unselected S node ti ∈ T do for mixture-of-experts, 2025. URL https://arxiv. Etraj (ti ) ← τ ∈Path(root→ti ) Epred (τ ) org/abs/2511.14102. Costi ← |Etraj (ti ) \ B| Scorei ← P (ti )/(Costi + ϵ) Williams, S., Waterman, A., and Patterson, D. Roofline: end for an insightful visual performance model for multicore t∗ ← arg maxti Scorei architectures. Commun. ACM, 52(4):65–76, April S ← S ∪ {t∗ } 2009. ISSN 0001-0782. doi: 10.1145/1498765. B ← B ∪ Etraj (t∗ ) {Update buffer for subsequent 1498785. URL https://doi.org/10.1145/ scoring} 1498765.1498785. end while Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., return S {Verify all selected nodes in one target forward Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., pass} Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State-ofthe-art natural language processing. In Proceedings of A. Impact of Batch Size on Scalability the 2020 Conference on Empirical Methods in Natural A.1. End-to-End Batch-Size Scaling Language Processing: System Demonstrations, pp. 38– 45, Online, October 2020. Association for Computational Batching is a common way to improve GPU utilization durLinguistics. URL https://aclanthology.org/ ing inference. To examine how batching affects E CO S PEC, 2020.emnlp-demos.6/. we compare it with EAGLE-3 under batch sizes B = {1, 2, 4} and report end-to-end speedup together with active Xue, F., Zheng, Z., Fu, Y., Ni, J., Zheng, Z., Zhou, W., and expert counts. You, Y. Openmoe: an early effort on open mixture-ofexperts language models. In Proceedings of the 41st InAs shown in Tables 6 and 7, both methods exhibit lower ternational Conference on Machine Learning, ICML’24. speedup as batch size increases. On Qwen3-235B-A22B, JMLR.org, 2024. EAGLE-3’s average speedup drops from 1.22× at B = 1 Xue, L., Fu, Y., Lu, Z., Mai, L., and Marina, M. Moeinfinity: Efficient moe inference on personal machines with sparsity-aware expert cache, 2025. URL https: //arxiv.org/abs/2401.14361.
to 1.00× at B = 4, while E CO S PEC maintains a higher average speedup of 1.09× at B = 4. A similar trend appears on GPT-OSS-120B, where EAGLE-3 drops from 1.14× to 1.00×, while E CO S PEC retains 1.06× at B = 4.
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report, 2025. URL https://arxiv.org/ abs/2505.09388.
Active expert counts help explain this speedup gap. As batch size increases, each verification step covers more candidate tokens in parallel, expanding the union of activated experts and increasing verification cost. At B = 4 on Qwen3-235BA22B, EAGLE-3 activates an average of 54.9 experts per step, while E CO S PEC reduces this number to 45.7. On GPTOSS-120B, the corresponding counts are 32.6 for EAGLE-3 and 28.1 for E CO S PEC. This smaller expert footprint lowers verification cost relative to EAGLE-3, allowing E CO S PEC to retain higher end-to-end speedup under batching.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems 12
Submission and Formatting Instructions for ICML 2026 Table 6. Batch-size scaling on Qwen3-235B-A22B. We report endto-end speedup relative to AR decoding and the average number of active experts. Dataset
Method
BS = 1
BS = 2
Table 7. Batch-size scaling on GPT-OSS-120B. We report end-toend speedup relative to AR decoding and the average number of active experts.
BS = 4
Spd ↑
Exp ↓
Spd ↑
Exp ↓
Spd ↑
Exp ↓
GSM8K
EAGLE-3 E CO S PEC
1.31 1.39
22.3 21.0
1.18 1.24
42.9 35.5
1.00 1.07
53.5 45.3
HumanEval
EAGLE-3 E CO S PEC
1.11 1.33
26.2 21.6
1.05 1.21
46.8 36.0
0.95 1.08
MMStar
EAGLE-3 E CO S PEC
1.35 1.57
20.5 18.4
1.22 1.40
41.0 34.1
AIME-25
EAGLE-3 E CO S PEC
1.14 1.23
22.1 20.9
1.08 1.15
Math500
EAGLE-3 E CO S PEC
1.05 1.13
25.4 19.4
AMC22-24
EAGLE-3 E CO S PEC
1.12 1.28
MTBench
EAGLE-3 E CO S PEC
Average
EAGLE-3 E CO S PEC
Dataset
BS = 1
Method
BS = 2
BS = 4
Spd ↑
Exp ↓
Spd ↑
Exp ↓
Spd ↑
Exp ↓
GSM8K
EAGLE-3 E CO S PEC
1.05 1.11
11.9 10.6
1.02 1.08
25.1 22.8
0.93 0.97
35.4 30.1
57.5 47.8
HumanEval
EAGLE-3 E CO S PEC
1.05 1.26
11.8 10.7
1.03 1.18
22.0 18.9
0.98 1.03
32.3 28.2
1.05 1.18
51.8 39.9
MMStar
EAGLE-3 E CO S PEC
1.28 1.45
11.7 10.5
1.15 1.30
21.9 18.7
1.04 1.15
32.2 26.0
42.6 31.3
0.98 1.06
53.0 41.9
AIME-25
EAGLE-3 E CO S PEC
1.17 1.33
12.2 10.8
1.10 1.21
22.5 20.1
1.02 1.08
31.8 27.4
1.02 1.09
46.0 39.8
0.98 1.03
56.5 47.5
Math500
EAGLE-3 E CO S PEC
1.14 1.27
12.0 10.6
1.08 1.15
24.2 20.8
0.97 1.05
34.5 29.0
25.4 22.2
1.07 1.14
47.8 38.6
1.02 1.04
56.5 49.3
AMC22-24
EAGLE-3 E CO S PEC
1.14 1.45
11.9 10.9
1.07 1.30
20.7 18.6
1.03 1.09
31.3 27.9
1.46 1.62
24.0 19.9
1.25 1.42
44.8 37.6
1.05 1.19
55.5 48.5
MTBench
EAGLE-3 E CO S PEC
1.18 1.30
10.3 9.6
1.10 1.20
19.9 16.6
1.03 1.08
30.4 27.9
1.22 1.36
23.7 20.5
1.12 1.24
44.6 36.1
1.00 1.09
54.9 45.7
Average
EAGLE-3 E CO S PEC
1.14 1.31
11.6 10.6
1.08 1.20
22.3 19.5
1.00 1.06
32.6 28.1
Table 8. Batch-size scaling of verification cost. We report ρ = TAR /Tverify , the ratio between average AR decoding time and average speculative verification time per step. BS=8 Spd reports throughput normalized by AR throughput at batch size 8.
A.2. Batch-Size Scaling of Verification Cost We further examine larger-batch behavior using a verification-cost-oriented metric on 100 randomly sampled instances from the seven evaluation datasets. In Table 8, we report
ρ=
TAR , Tverify
Model
Method
BS=1
BS=2
BS=4
BS=8
BS=8 Spd
Qwen3-235B-A22B
EAGLE-3 E CO S PEC
0.59× 0.67×
0.55× 0.62×
0.46× 0.52×
0.32× 0.33×
0.69× 0.69×
GPT-OSS-120B
EAGLE-3 E CO S PEC
0.72× 0.90×
0.69× 0.82×
0.63× 0.72×
0.43× 0.44×
0.65× 0.65×
B. Predictor Details and Analysis B.1. Predictor Training Details E CO S PEC trains a lightweight expert predictor for each target MoE model to estimate expert activation cost before target-model verification. The predictor is used only for cost-aware draft selection. It does not participate in token verification, does not query the target MoE model online, and does not modify the standard speculative verification rule.
where TAR denotes the average AR decoding time and Tverify denotes the average speculative verification time per step. A larger ρ indicates that speculative verification is cheaper relative to AR decoding. We also report BS=8 Spd, the throughput of each speculative method normalized by AR throughput at batch size 8. The ratio ρ decreases for both methods as batch size increases. On Qwen3-235B-A22B, ρ drops from 0.59× to 0.32× for EAGLE-3 and from 0.67× to 0.33× for E CO S PEC. On GPT-OSS-120B, it drops from 0.72× to 0.43× for EAGLE-3 and from 0.90× to 0.44× for E CO S PEC. At BS=8, both speculative methods fall below AR throughput in this prototype, with BS=8 speedup of 0.69× on Qwen3-235B-A22B and 0.65× on GPT-OSS120B.
Predictor backbones. We instantiate Πθ with small opensource LMs and fine-tune them to predict the target model’s per-layer expert activations from token-level inputs. For DeepSeek-V3.1, we use DeepSeek-R1-Distill-Qwen-1.5B as the predictor backbone. For Qwen3-235B-A22B, we use Qwen3-0.6B, leveraging architectural proximity within the Qwen family. For GPT-OSS-120B, since no lightweight model from the same series is publicly available, we also use Qwen3-0.6B. The predictor cost is small relative to target-model verification, as reflected in Table 3.
These results indicate that large-batch speculative decoding remains challenging in this prototype. Nevertheless, E CO S PEC consistently maintains a higher TAR /Tverify ratio than EAGLE-3 across the tested batch sizes, indicating lower verification cost under the same setting.
Training data and setup. We collect routing traces offline by running each target MoE model on seven datasets span13
Submission and Formatting Instructions for ICML 2026 Table 9. Robustness to Predictor Accuracy. Predictor-accuracy sensitivity. We vary the predictor checkpoint while keeping the decoding configuration fixed.
ning reasoning, coding, and dialogue: HumanEval, MMStar, MT-Bench, AMC22-24, GSM8K, AIME-25, and Math500. For each token and MoE layer, we record the target router’s Top-K selected experts as supervision, where K follows the target model’s routing configuration. We randomly split samples into 80% for training and 20% for testing. All evaluations reported in the main paper are conducted on the held-out test split. We fine-tune predictors for 100 epochs using Adam with learning rate 1 × 10−5 and batch size 16.
Model
Dataset
Acc ≈ 50% Spd / Len / Exp
Acc ≈ 60% Spd / Len / Exp
Converged Spd / Len / Exp
Qwen3-235B
GSM8K MT-Bench
1.17 / 2.20 / 24.5 1.20 / 2.45 / 25.2
1.29 / 2.50 / 22.5 1.45 / 2.50 / 23.8
1.39 / 2.54 / 21.0 1.62 / 2.44 / 19.9
GPT-OSS-120B
GSM8K MT-Bench
1.01 / 1.25 / 13.5 1.07 / 1.80 / 17.8
1.04 / 1.45 / 11.6 1.14 / 1.45 / 13.5
1.11 / 1.52 / 10.6 1.30 / 1.81 / 9.6
Table 10. Oracle expert-set analysis. Oracle uses ground-truth target-router activations collected offline, while Predictor uses the learned expert predictor. All entries report verification-oriented metrics under the same E CO S PEC selection rule.
Routing prediction accuracy. We report Top-K routing prediction accuracy on the test split, computed as the average overlap ratio between the predicted Top-K experts and the target router’s Top-K experts across tokens and MoE layers. The resulting accuracies are 80% for DeepSeek-V3.1, 82% for Qwen3-235B-A22B, and 93% for GPT-OSS-120B. The higher accuracy on GPT-OSS-120B is consistent with its Top-4-over-128 routing setting, which yields more concentrated activation patterns than the Top-8 routing used by DeepSeek-V3.1 and Qwen3-235B-A22B. Since EcoSpec uses the predictor only as a cost-estimation module, exact routing reconstruction is not required; the sensitivity to predictor quality is further evaluated in Appendix B.2.
Model
Selector
Time (s)
Accept Len
Active Experts
Qwen3-235B-A22B
Oracle Predictor
0.730 0.730
2.32 2.32
20.4 20.5
GPT-OSS-120B
Oracle Predictor
0.090 0.090
1.86 1.86
10.5 10.6
DeepSeek-V3.1
Oracle Predictor
0.929 0.930
2.79 2.79
31.2 31.2
selection. In this analysis, we keep the E CO S PEC selection rule unchanged and replace the predicted expert sets with ground-truth expert sets collected offline from the target MoE model. This isolates the effect of predictor error while keeping the selection rule fixed.
B.2. Impact of Predictor Accuracy To examine the effect of predictor quality, we evaluate E CO S PEC with three predictor checkpoints: two intermediate checkpoints with Top-K routing accuracy of approximately 50% and 60%, and the converged checkpoint used in the main experiments. The decoding configuration is fixed across all runs, including the default verification budget γ = 4; only the predictor checkpoint is changed.
The oracle expert sets are used only for analysis. In normal speculative decoding, the true expert sets of candidate draft nodes are unavailable before target-model verification unless additional target-model computation is performed. Therefore, this setting is reported with verification-oriented metrics rather than end-to-end throughput, and is not used as an online inference baseline.
Table 9 reports end-to-end speedup, acceptance length, and active expert count on Qwen3-235B-A22B and GPT-OSS120B. Lower-accuracy predictors lead to lower speedup and higher active expert counts. On Qwen3-235B-A22B, the converged predictor improves the speedup from 1.17× to 1.39× on GSM8K and from 1.20× to 1.62× on MT-Bench compared with the 50%-accuracy checkpoint. On GPT-OSS120B, the corresponding speedup improves from 1.01× to 1.11× on GSM8K and from 1.07× to 1.30× on MTBench. The active expert count also decreases consistently as predictor quality improves.
As shown in Table 10, replacing the learned predictor with oracle expert sets produces nearly unchanged acceptance lengths, verification times, and active expert counts. This indicates that the converged predictor provides a cost signal close to that obtained from oracle expert sets in the evaluated settings. This analysis complements the predictor-accuracy study. The accuracy study shows that lower-quality predictors can reduce the effectiveness of cost-aware selection, while the oracle expert-set analysis shows that replacing the converged predictor with ground-truth expert sets yields little additional change. Together, these results indicate that predictor quality matters, but the converged predictor is already sufficiently accurate for the selection rule used by E CO S PEC.
These results show that predictor accuracy affects the quality of cost-aware draft selection. The converged predictor gives the best speedup in all reported settings, while intermediate predictors still retain positive speedup under the same decoding configuration. B.3. Oracle Expert-Set Analysis
C. Evaluation under Different Verification Budgets (γ)
We further analyze whether the learned predictor provides a sufficiently accurate expert-cost signal for cost-aware draft
We vary the verification budget γ ∈ {3, 4, 5, 6} to support the common default setting used in the main experiments. 14
Submission and Formatting Instructions for ICML 2026 Table 11. Verification-budget sensitivity on Qwen3-235B-A22B. Each entry reports Spd / Len / Exp, corresponding to end-to-end speedup, acceptance length, and active expert count. γ = 4 (Default) Spd / Len / Exp
γ=5 Spd / Len / Exp
γ=6 Spd / Len / Exp
Dataset
Method
γ=3 Spd / Len / Exp
GSM8K
EAGLE-3 E CO S PEC
1.25 / 2.45 / 21.8 1.28 / 2.15 / 16.8
1.31 / 2.54 / 22.3 1.39 / 2.54 / 21.0
1.15 / 2.62 / 27.5 1.25 / 2.62 / 24.5
1.02 / 2.68 / 31.2 1.09 / 2.68 / 28.2
HumanEval
EAGLE-3 E CO S PEC
1.08 / 2.25 / 20.5 1.22 / 2.05 / 17.2
1.11 / 2.35 / 26.2 1.33 / 2.32 / 21.6
1.02 / 2.41 / 30.8 1.18 / 2.41 / 25.8
0.95 / 2.45 / 33.5 1.05 / 2.45 / 29.1
MMStar
EAGLE-3 E CO S PEC
1.30 / 2.55 / 19.8 1.41 / 2.25 / 15.1
1.35 / 2.64 / 20.5 1.57 / 2.58 / 18.4
1.22 / 2.70 / 25.2 1.38 / 2.65 / 22.5
1.08 / 2.75 / 29.5 1.15 / 2.70 / 26.2
AIME-25
EAGLE-3 E CO S PEC
1.10 / 2.30 / 21.5 1.15 / 2.08 / 16.5
1.14 / 2.41 / 22.1 1.23 / 2.37 / 20.9
1.05 / 2.45 / 27.8 1.12 / 2.44 / 24.2
0.96 / 2.50 / 30.5 1.02 / 2.48 / 27.8
Math500
EAGLE-3 E CO S PEC
1.02 / 1.85 / 20.5 1.08 / 1.75 / 15.9
1.05 / 1.96 / 25.4 1.13 / 1.88 / 19.4
0.98 / 2.02 / 29.5 1.05 / 1.95 / 23.8
0.90 / 2.05 / 32.8 0.98 / 2.01 / 27.5
AMC22-24
EAGLE-3 E CO S PEC
1.08 / 2.10 / 21.2 1.19 / 1.90 / 17.5
1.12 / 2.20 / 25.4 1.28 / 2.10 / 22.2
1.04 / 2.25 / 30.1 1.15 / 2.18 / 26.4
0.95 / 2.28 / 33.2 1.04 / 2.22 / 29.8
MTBench
EAGLE-3 E CO S PEC
1.40 / 2.65 / 21.5 1.48 / 2.40 / 16.2
1.46 / 2.78 / 24.0 1.62 / 2.44 / 19.9
1.32 / 2.85 / 28.5 1.45 / 2.55 / 24.1
1.15 / 2.90 / 32.5 1.20 / 2.60 / 28.5
Average
EAGLE-3 E CO S PEC
1.18 / 2.31 / 21.0 1.26 / 2.08 / 16.5
1.22 / 2.41 / 23.7 1.36 / 2.32 / 20.5
1.11 / 2.47 / 28.5 1.22 / 2.40 / 24.5
1.00 / 2.52 / 31.9 1.08 / 2.45 / 28.2
D. Expert Activation Pattern Analysis We analyze expert-activation patterns of the evaluated MoE backbones in Fig. 4. For each model, the left panel shows the expert-activation heatmap across layers, and the right panel shows the layer-wise expert-load distribution. This analysis characterizes the expert working sets encountered during speculative verification and provides context for why active expert count is reported alongside acceptance length.
DeepSeek-V3.1. DeepSeek-V3.1 shows a relatively diffuse activation pattern. The heatmap does not concentrate on a small subset of experts, and the layer-wise load distributions are relatively narrow across most layers. This indicates that expert usage is broadly balanced across routed experts. For speculative verification, such balanced routing still makes the expert working set an important cost component: verifying multiple draft tokens can involve many distinct experts even when no small group of experts dominates the routing pattern.
Table 12. Verification-budget sensitivity on GPT-OSS-120B. Dataset
Method
γ=3 Spd / Len / Exp
γ = 4 (Default) Spd / Len / Exp
γ=5 Spd / Len / Exp
γ=6 Spd / Len / Exp
GSM8K
EAGLE-3 E CO S PEC
1.02 / 1.45 / 11.2 1.20 / 1.30 / 8.5
1.05 / 1.56 / 11.9 1.11 / 1.52 / 10.6
1.01 / 1.62 / 13.5 1.08 / 1.60 / 11.5
0.96 / 1.65 / 15.2 1.02 / 1.65 / 13.2
HumanEval
EAGLE-3 E CO S PEC
1.02 / 1.62 / 11.0 1.15 / 1.45 / 8.8
1.05 / 1.72 / 11.8 1.26 / 1.72 / 10.7
1.00 / 1.78 / 13.2 1.18 / 1.78 / 11.9
0.94 / 1.82 / 15.0 1.09 / 1.82 / 13.5
MMStar
EAGLE-3 E CO S PEC
1.22 / 1.80 / 11.1 1.32 / 1.62 / 8.2
1.28 / 1.91 / 11.7 1.45 / 1.90 / 10.5
1.15 / 1.98 / 13.5 1.30 / 1.98 / 11.8
1.05 / 2.05 / 15.5 1.15 / 2.05 / 13.8
AIME-25
EAGLE-3 E CO S PEC
1.12 / 1.92 / 11.5 1.21 / 1.75 / 8.6
1.17 / 2.02 / 12.2 1.33 / 2.02 / 10.8
1.08 / 2.10 / 13.8 1.21 / 2.10 / 12.2
0.98 / 2.15 / 15.9 1.08 / 2.15 / 14.1
Math500
EAGLE-3 E CO S PEC
1.10 / 1.92 / 11.2 1.18 / 1.72 / 8.4
1.14 / 2.02 / 12.0 1.27 / 2.00 / 10.6
1.05 / 2.08 / 13.6 1.15 / 2.08 / 11.9
0.95 / 2.12 / 15.8 1.05 / 2.12 / 14.0
AMC22-24
EAGLE-3 E CO S PEC
1.10 / 1.92 / 11.1 1.32 / 1.75 / 8.9
1.14 / 2.02 / 11.9 1.45 / 2.02 / 10.9
1.04 / 2.10 / 13.8 1.30 / 2.10 / 12.4
0.96 / 2.15 / 15.7 1.09 / 2.15 / 14.5
MTBench
EAGLE-3 E CO S PEC
1.12 / 1.82 / 9.2 1.20 / 1.55 / 8.1
1.18 / 1.91 / 10.3 1.30 / 1.81 / 9.6
1.08 / 1.96 / 11.5 1.20 / 1.88 / 11.2
0.98 / 2.02 / 13.2 1.08 / 1.92 / 12.8
Average
EAGLE-3 E CO S PEC
1.10 / 1.72 / 10.9 1.23 / 1.59 / 8.5
1.14 / 1.88 / 11.6 1.31 / 1.86 / 10.6
1.06 / 1.96 / 13.3 1.20 / 1.93 / 11.8
0.97 / 2.02 / 15.1 1.08 / 1.98 / 13.7
Qwen3-235B-A22B and GPT-OSS-120B. Qwen3-235BA22B and GPT-OSS-120B show more concentrated expertactivation patterns. Their heatmaps contain clearer highusage regions, and their layer-wise load distributions have wider ranges with more visible high-load experts. These patterns indicate stronger locality in expert usage: different candidate tokens are more likely to share parts of their expert sets, making the verified expert working set sensitive to which draft tokens are selected.
Implication for cost-aware draft selection. These observations support treating the expert working set as an explicit cost component in MoE speculative decoding. Acceptance length measures how many drafted tokens are verified successfully, but it does not describe which experts are activated during verification. Two draft sets with similar acceptance length can induce different expert working sets and therefore different verification costs. E CO S PEC incorporates this cost dimension during draft selection by favoring candidates with lower predicted marginal expert cost under the same verification budget.
Tables 11 and 12 report end-to-end speedup, acceptance length, and active expert count for EAGLE-3 and E CO S PEC under each budget. We select the default verification budget according to average end-to-end speedup across the seven datasets. Under this criterion, γ = 4 gives the highest average end-to-end speedup for both EAGLE-3 and E CO S PEC on Qwen3-235BA22B and GPT-OSS-120B. Increasing γ beyond 4 further increases acceptance length in many cases, but does not improve the averaged end-to-end speedup. The sweep also shows that E CO S PEC maintains its advantage over EAGLE-3 across the tested budgets. On both target models, E CO S PEC consistently achieves higher average end-to-end speedup than EAGLE-3 under γ ∈ {3, 4, 5, 6}. It also uses fewer active experts on average under the same verification budget, showing that the benefit of cost-aware draft selection is not tied to a single budget choice.
E. Datasets and Evaluation Details We evaluate E CO S PEC on seven benchmarks covering mathematical reasoning, code generation, dialogue, and text-only inputs derived from a vision-language benchmark. All methods are evaluated with the same prompts and decoding settings on each dataset, so the reported speedup, acceptance length, and active expert count are computed under matched input conditions.
Therefore, we use γ = 4 as the default verification budget for both EAGLE-3 and E CO S PEC in the main experiments. 15
Submission and Formatting Instructions for ICML 2026
GSM8K GSM8K (Cobbe et al., 2021) contains gradeschool math word problems that require multi-step arithmetic reasoning. We use the standard 5-shot Chain-ofThought setting, where five exemplars are prepended to the input before the test question. AIME 2025 AIME 2025 (MAA, 2025) contains competition-level mathematical problems from the American Invitational Mathematics Examination. We use a zero-shot reasoning prompt and require the final answer to be placed in a boxed format: {question}\n Please reason step by step, and put your final answer within \boxed{}. MATH500 and AMC22-24 MATH500 (Hendrycks et al., 2021; Lightman et al., 2024) is a 500-problem subset of the MATH benchmark. AMC22-24 (AI-MO) contains problems from the American Mathematics Competitions from 2022 to 2024. For both datasets, we use the following zero-shot prompt: Problem:\n {problem}\n\n Solution: HumanEval HumanEval (Chen et al., 2021) contains 164 Python programming problems with function signatures, docstrings, and unit tests. We use the benchmark prompts as code-generation inputs.
Table 13. Architectural summary of evaluated MoE models. Total and active parameters follow the reporting convention of the corresponding model releases. Top-k denotes the number of selected experts per token.
MT-Bench MT-Bench (Zheng et al., 2023) is a multi-turn instruction-following benchmark.
Model Total / active parameters Transformer blocks MoE / dense layers Experts per MoE layer Top-k experts per token Maximum context length
MMStar MMStar (Chen et al., 2024) is a vision-language benchmark whose original evaluation involves both image and text inputs. Since this work focuses on text-only LLM inference, we remove the image inputs and provide only the textual questions to the model. This setting is used as a textonly generation workload and is not intended to measure vision-language grounding ability.
F. Model Details Table 13 summarizes the MoE configurations of the target models used in our experiments. We include the total and active parameter counts, the number of MoE layers, the number of experts per MoE layer, and the routing Top-k. These attributes determine the scale of expert activation during verification and are directly related to the cost-aware draft selection studied in this paper.
16
DeepSeek-V3.1
Qwen3-235B-A22B
GPT-OSS-120B
671B / 37B 61 58 / 3 256 routed + 1 shared 8 128K
235B / 22B 94 94 / 0 128 8 40,960
120B / 5.1B 36 36 / 0 128 4 131,072
Submission and Formatting Instructions for ICML 2026
(a) DeepSeek-V3.1: Activation Heatmap
(b) DeepSeek-V3.1: Load Distribution
(c) Qwen3-235B: Activation Heatmap
(d) Qwen3-235B: Load Distribution
(e) GPT-OSS-120B: Activation Heatmap
(f) GPT-OSS-120B: Load Distribution
Figure 4. Expert activation patterns across evaluated MoE models. Each row corresponds to one target model. The left panel shows the expert-activation heatmap, and the right panel shows the layer-wise expert-load distribution.
17