Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length

Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length Yuzheng Lan

arXiv:2610.34440v1 [cs.LG] 25 Oct 2026

ReClan AI Studio, LLC research@reclan.ai

Abstract Speculative decoding checks several drafted tokens in one verification pass. On a mixture-ofexperts (MoE) model, consecutive tokens often select the same experts. A deduplicating engine reads each selected expert once per pass, so it requests fewer bytes than a per-slot engine, which reads it once per token. We write a step’s time as the bytes each kernel requests over its effective bandwidth, plus fixed time. Under geometric acceptance, we derive from this step time when speculation is faster than plain decoding. The best draft length is a monotone staircase in the acceptance rate. We bound how a bandwidth error moves each result, and we check the main steps in Lean. Deduplication is faster exactly where its share ū(n) of the per-slot bytes is below its ratio r(n) of the per-slot bandwidth. On llama.cpp Metal, MLX-LM and llama.cpp CUDA, which we modified to deduplicate, the per-slot kernel has the higher effective bandwidth (r = 0.32 to 0.75). Thus deduplication is faster only from 5 to 8 tokens per pass, or never. We fit the bandwidths on one layer’s expert operation, run alone, in under an hour per model. Bytes from routing traces over these bandwidths predict the time that deduplication saves. The measured saving lies inside its error bound at every point on CUDA and MLX-LM and at 36 of 48 points on Metal. The crossover is the first pass size at which deduplication is faster. The prediction places it within one pass size in 10 of 12 configurations of engine, model and depth. We computed these predictions after the measurements. On DeepSeek-V2-Lite, operation times measured before its speculative runs give the best draft length on CUDA within one at all 22 acceptance values.

1

Introduction

A mixture-of-experts (MoE) layer replaces one feed-forward block by N experts and a router that sends each token to m of them (Fedus et al., 2022; Shazeer et al., 2017). At batch size one, memory reads limit decoding (Chordiya, 2026; Pope et al., 2023; Zhang, 2026). In speculative decoding (Chen et al., 2023; Leviathan et al., 2023), a small draft model proposes k tokens, and this number is the draft length. The target, the model being decoded, checks these tokens with the current token in one verification pass. The acceptance rate a is the probability that a drafted token survives. On an MoE model, the pass needs the experts of all k + 1 verified tokens. Published measurements disagree on whether speculation then speeds up decoding (Cao et al., 2026; Saxena et al., 2025). Consecutive tokens often choose the same experts (Suram, 2026; Zhang et al., 2026). An engine is the inference software that runs the model. A kernel is a GPU routine that the engine launches for one operation. The kernel that matters here runs the expert operation. This operation is the set of matrix products (gate, up and down projections) of each token of a verification pass with the weights of its selected experts. A slot is one of a token’s m expert selections, so a pass of n tokens has nm slots. A per-slot kernel multiplies each token by each of its selected experts separately, 1

with one expert read per slot. A deduplicating kernel groups the pass’s tokens by expert and reads each distinct expert once. We call an engine per-slot or deduplicating by the kernel that it runs on verification passes. So reuse lowers the bytes of a pass only on a deduplicating engine. Consider an affine step time 1 + χk, with χ the cost of a drafted token relative to a target step. For this step time, Leviathan et al. (2023) show that some draft length is faster than plain decoding exactly when a > χ. They also give the first step point, the acceptance at which the best draft length first exceeds zero. Section 3 extends both results to any step time. It also gives every step point when each further draft adds at least as much time as the one before. The cost of an MoE verification pass is not affine in k. Its bytes depend on how many distinct experts the verified tokens select. Its time depends on how fast each kernel moves those bytes. Prior systems decide whether to speculate either from measurement or from a model of the verification cost. Saxena et al. (2025) disables speculation when a utility measured online falls below one. Among cost models for MoE targets, MoESD (Huang et al., 2025) derived the expected number of distinct experts that n tokens select under uniform routing: N (1−q n ) with q = 1−m/N . Li et al. (2026) assumes deduplication and no fixed cost. Its speedup condition is on the ratio of the distinct experts that the tokens of a verification pass select to those of one token. So it is a condition on how often consecutive tokens repeat experts. S2-MoE (Huang et al., 2026) charges a verification the newly introduced experts times a per-expert latency. SpecFit (Cao et al., 2026) predicts cycle time as an intercept plus a coefficient times the per-layer union of experts. Both charge every expert read the same time at every pass size, which is a single bandwidth for every kernel. Neither derives the staircase, the best draft length as a function of acceptance. Neither derives a speedup from reuse together with the bandwidth of each kernel. Next-token expert reuse has been measured at 44 to 48% on Qwen3-30B-A3B (Zhang et al., 2026) and 0.444 in llama.cpp router traces (Suram, 2026). MonoMoE (Gong et al., 2026) finds that token-major kernels that group tokens by expert underuse memory bandwidth when few tokens reach each expert. It replaces them by a fused kernel that loops over expert weights and applies each to all of its tokens. Opportunistic Expert Activation (Oncescu et al., 2025) re-routes tokens to experts that a batch already loads. Neither models when a deduplicating kernel is faster than a per-slot one. Engine developers tune that switch by hand (pratiknarola-t, 2026). An MLX issue reports that the deduplicating product at one row runs near half its bandwidth at four rows (Haddad, 2026). Both are engineering reports without peer review. This paper separates the two inputs of a step’s time. A speedup is a ratio of step times. A step’s time is the bytes that its kernels request over the rate at which they move them, plus fixed time. That rate is the effective bandwidth of each kernel. The bytes follow from routing alone. On CUDA, they match a hardware count of DRAM bytes. The bandwidths depend on the kernel: on every engine and model that we measured, the per-slot kernel has the higher effective bandwidth. On a pass of n tokens, deduplication requests a share ū(n) of the per-slot bytes, and routing fixes this share. Deduplication moves these bytes at a ratio r(n) of the per-slot bandwidth, and the kernels fix this ratio. So deduplication is faster exactly where ū(n) < r(n). Its saving, the time it removes from a pass, is the per-slot expert time times 1 − ū(n)/r(n). Section 2 measures the bytes and the bandwidths. Section 3 works in that step time. It gives the condition under which a draft length is faster than plain decoding, and the staircase. On a deduplicating engine, it gives the reuse threshold, the least mean reuse at which a draft length is faster, and the speedup. It also gives bounds on how a bandwidth error moves each prediction. These results are proved in Lean 4 (de Moura and Ullrich, 2021) against Mathlib (The mathlib Community, 2020). Section 4 tests them on deduplicating versions of llama.cpp Metal, MLX-LM and llama.cpp CUDA. We measure the bandwidth on the expert operation, in under an hour per model. Bytes from routing over this bandwidth predict the saving within its stated error on CUDA and MLX-LM. They predict the 2

crossover within one pass size in 10 of 12 configurations. A configuration is an engine, a model and a context depth. On DeepSeek-V2-Lite, we measured operation times before any of its speculative runs. These times predict the best draft length on CUDA within one at every acceptance value. In contrast, the prediction from fits run afterwards misses its CUDA crossover. On Metal, the verify time needs one calibration of the time outside the expert operation. Routing, union and reuse. A decoder has L MoE layers of N experts. In layer l, the router selects a set Sl,t of m experts, 1 ≤ m < N , for the token at position t. We call a law on the array (Sl,t ) a routing process. It is stationary when a window of n tokens has the same law wherever it starts. A pass over n consecutive tokens needs their Xl,t (n) = |Sl,t ∪ · · · ∪ Sl,t+n−1 | distinct experts, with mean Ul (n), Ul (0) = 0. We define the consecutive-token reuse ρl (n) =

1 E Sl,t+n−1 ∩ (Sl,t ∪ · · · ∪ Sl,t+n−2 ) , m

ρl (1) = 0,

(1)

the expected share of a token’s experts that one of the n − 1 tokens before it also selected. At n = 2 it is the next-token reuse measured by Zhang et al. (2026). Each token therefore adds m(1 − ρl (n)) new experts on average. By linearity of expectation, for every stationary process, Ul (n) = m

n X

1 − ρl (j) .

(2)

j=1

Under independent uniform routing 1 − ρl (n) = q n−1 with q = 1 − m/N (Huang et al., 2025). P Bytes. Let wl be the bytes of one expert of layer l, W = l wl , and D the bytes read once per pass whatever the routing. A trace isPthe recorded expert sets of one token sequence. On every trace, a deduplicating kernel requests l wl Xl,t (n) expert bytes for a pass of n tokens. A per-slot P kernel requests mnW , one request per slot. In expectation, the union bytes are Ededup (n) = l wl Ul (n), and Eslot (n) = mnW . So a per-slot kernel requests Eslot (n) − Ededup (n) = m

X l

wl

n X

ρl (j) ≥ 0

(3)

j=1

more bytes. These are the bytes that reuse would save. They are requested bytes, and on-chip caches can serve some requests. Speculative decoding. A step drafts k ∈ {0, . . . , kmax } tokens, with kmax the largest draft length allowed. It verifies them with the current token in one pass of k + 1 tokens. The case k = 0 is plain decoding. With kernel p ∈ {slot, dedup} the step requests D + Ep (k + 1) + kv(k) bytes, with kv(k) ≥ 0 the key-value cache bytes it reads (Pope et al., 2023). Under geometric acceptance at rate a ∈ [0, 1) the drafts are accepted in order, each with probability a, until the first rejection. A step then emits Ga (k) = 1 + a + · · · + ak tokens on average (Leviathan et al., 2023). We generalize the walltime analysis of Leviathan et al. (2023). The throughput is Ta (k) = Ga (k)/V (k) tokens per second. Here V (k) is the step time, the wall-clock time of one step that drafts k tokens and verifies k + 1 (Section 3.1 gives its form). The best draft length k ∗ (a) is the smallest maximizer of Ta . So the staircase is a 7→ k ∗ (a). Speculation gives a speedup at a when k ∗ (a) ≥ 1. The j-th step point aj is the infimum of the acceptances a ∈ [0, 1) at which k ∗ (a) ≥ j. It is taken as 1 when there is none. Finally, ci = V (i) − V (i − 1) is the time that the i-th draft adds. Appendix A lists every symbol. 3

reuse ρl(n), layer mean

1.0 0.8 uniform, m/N = 1/8 (OLMoE, gpt-oss)

0.6

uniform, m/N = 1/16 (Qwen3)

0.4

measured (WikiText-103 fit documents)

0.2 0.0 1

2

3

4

5

6

7

8

9

OLMoE Qwen3 gpt-oss

consecutive tokens n

Figure 1: Consecutive-token reuse ρl (n), layer mean, on the WikiText-103 fit documents, against the number n of consecutive tokens. Dotted: independent uniform routing, 1 − q n−1 (Huang et al., 2025). In every figure, color marks the model: blue OLMoE, green Qwen3, orange gpt-oss and brown DeepSeek-V2-Lite.

2

Setup

This section measures the two inputs separately: the bytes and the effective bandwidths. The measurements use an Apple M4 Max (546 GB/s, Apple Inc., 2024) with llama.cpp on Metal and MLX-LM. They also use an NVIDIA RTX 4070 Ti SUPER (672 GB/s, 48 MiB L2 cache) with llama.cpp on CUDA. The models are OLMoE-1B-7B-0125 (Muennighoff et al., 2025) (N = 64, m = 8), Qwen3-30B-A3B (Yang et al., 2025) (N = 128, m = 8), gpt-oss-20b (OpenAI, 2025) (N = 32, m = 4) and DeepSeek-V2-Lite (Section 4.4). Appendix D gives the setup of the first three. Byte units are decimal (1 GB is 109 bytes), except MiB, which is 220 bytes. Expert reuse. We record the experts each token selects on the 60 articles of the WikiText-103 test set (Merity et al., 2017). A hash of the document identifier selects 32 of them for fitting. On these, ρ̂l (2) is the estimate of ρl (2) in (1), and Appendix D gives the estimator. Its layer mean is 0.363, 0.480 and 0.492 for OLMoE, Qwen3 and gpt-oss. Uniform routing gives m/N = 0.125, 0.0625 and 0.125. At n = 9 the layer mean is 0.762, 0.780 and 0.856 (Figure 1). Yet on the short passes of speculation (up to 8 tokens), llama.cpp Metal, MLX-LM and llama.cpp CUDA multiply each token’s experts separately (Appendix D.1). So each draft adds the full expert bytes of one more token. With a real draft model on llama.cpp Metal, plain decoding is fastest on 33 of 37 held-out MT-Bench prompts (Zheng et al., 2023) (Appendix D.3). Effective bandwidth. We write βp (n) for the effective bandwidth of kernel p on a pass of n tokens: its requested expert bytes over its time. We measure it in a separate benchmark of the expert operation of one MoE layer. The benchmark runs each engine’s own kernels, with no other part of the model. It takes the expert ids of each pass as input, and we give it two kinds of ids. In the first kind, we choose the ids ourselves. For each pass size n, we set how many distinct experts U the nm slots of the pass name. The range goes from heavy repetition to none. This lets a fit separate the time of the first read of an expert in a pass from the time of a repeated read. In the second kind, each pass copies its ids from a window of consecutive tokens of recorded routing, here the WikiText-103 traces. So the kernel sees the routing that the model produces. We write r(n) = βdedup (n)/βslot (n) for the bandwidth ratio. We write ū(n) = Ededup (n)/Eslot (n) for the share of the per-slot bytes that deduplication requests. A kernel whose repeated reads hit an on-chip cache has a higher effective bandwidth than one that reads every requested byte from memory. The per-slot kernel moves its requested bytes at a higher effective bandwidth than the deduplicating kernel moves its distinct ones (Figure 2). This holds on every engine and model 4

effective bandwidth (GB/s)

OLMoE

Qwen3

gpt-oss

llama.cpp CUDA

DRAM peak

600 400 6

tokens per pass n

MLX-LM

DRAM peak

800

4

deduplicating, on Ub

llama.cpp Metal

1000

2

per-slot, on nmb

DeepSeek-V2-Lite

8

500

500

400

400

300

300

200

200

100

100 2

4

6

8

tokens per pass n

2

4

DRAM peak

6

8

tokens per pass n

Figure 2: Effective bandwidth of each kernel on the expert operation alone. Per-slot kernel: on requested bytes nmb (solid line, filled markers). Deduplicating kernel: on union bytes U b (dashed line, open markers). Median and interquartile range over all MoE layers and windows of recorded routing; color: model; dotted: nominal DRAM bandwidth. CUDA was measured on OLMoE, gpt-oss and DeepSeek-V2-Lite, and MLX-LM on OLMoE. measured, at every n from 2 to 8. The ratio r(n) is 0.37 to 0.75 at n = 2 and 0.32 to 0.59 at n = 8. It is lower at n = 8 than at n = 2 in all eight engine and model pairs (Table 1). On the passes with chosen ids, U of the nm slots are the first read of their expert in the pass, and nm − U repeat one. We fit the time of the per-slot kernel as tslot = top + cu U b + cre (nm − U ) b,

(4)

where tslot is the median time, over repetitions, of the expert operation of one layer with the per-slot kernel. Here top is the part of that time that does not grow with the bytes, such as kernel launches. The bytes of one expert of the timed layer l are b = wl . The constant cu is the time per byte of a first read, and cre is the time per byte of a repeated slot. We fit the constants top , cu and cre per layer by least squares, over the passes with chosen ids at n = 2, . . . , 9. The re-read cost cre /cu is 1 when a repeated slot is read again from memory and near 0 when an on-chip cache serves it. On CUDA and MLX-LM, caching explains part of the gap. The re-read cost is 0.18 to 0.34 on CUDA and 0.27 to 0.40 on MLX-LM (Table 1). On CUDA, the median per-slot bandwidth also exceeds the DRAM bandwidth of the card, 672 GB/s, from n = 4 (Figure 2). On llama.cpp Metal the re-read cost is 0.88 to 1.04, so caching does not explain the gap there. There the gap lies in the deduplicating kernel. At n = 2 it moves 65 to 147 GB/s, against 131 to 250 GB/s for the per-slot kernel. Bytes against bandwidth. A deduplicating engine requests the share ū(n) of the per-slot bytes, at the lower bandwidth. So, with equal fixed work, it is faster exactly where ū(n) < r(n) (7). On the four models, ū(2) is 0.72 to 0.83 (Table 1), above r(2) in every engine and model pair. So at two tokens the deduplicating kernel is slower everywhere. The bytes that reuse saves become time saved only from a pass size that the bandwidth ratio sets (Section 4.2). Predicting the speedup needs both inputs: the bytes, which routing fixes, and the bandwidth of each kernel, which the kernels fix.

5

Table 1: Effective bandwidth of each kernel in GB/s, over all MoE layers and windows of recorded routing, including the fixed work of the operation. The values of r and ū come from bytes and times summed over layers and windows. Re-read cost: cre /cu of (4), as a range over two layers (−: no fit on chosen ids). The deduplicating kernel is faster where ū < r (7). re-read engine

model

llama.cpp CUDA llama.cpp CUDA llama.cpp CUDA llama.cpp Metal llama.cpp Metal llama.cpp Metal llama.cpp Metal MLX-LM

OLMoE gpt-oss DeepSeek-V2-Lite OLMoE Qwen3 gpt-oss DeepSeek-V2-Lite OLMoE

n=2

n=8

cost

βslot

βdedup

r

ū

βslot

βdedup

r

ū

0.18-0.22 0.34 − 0.88-0.90 0.92-0.98 0.98-1.04 − 0.27-0.40

618 654 523 250 131 213 196 129

464 346 355 147 65 79 91 75

0.75 0.53 0.68 0.59 0.50 0.37 0.47 0.58

0.82 0.72 0.83 0.82 0.77 0.72 0.82 0.82

933 1018 814 428 280 339 350 358

546 434 480 219 98 114 148 116

0.58 0.43 0.59 0.51 0.35 0.34 0.42 0.32

0.50 0.36 0.62 0.50 0.43 0.36 0.62 0.50

Testing the inputs. A bandwidth is bytes over time. So a wrong byte count would be absorbed into a bandwidth fitted on the same passes. For that reason, this paper uses no timing as evidence for a byte count. The fit (4) only locates the source of the higher bandwidth of the per-slot kernel. The bytes come from the routing and from the source code of the kernels. On CUDA, a hardware counter reads the DRAM bytes that each kernel moves, with no timing involved, and confirms them (Section 4.1). The Apple engines expose no such counter, so there the bytes rest on the source reading. The bandwidths come from the benchmark of the expert operation alone. Their stated error ε is a bound on the relative error of a measured bandwidth, and it comes from a split inside the benchmark. We make the linear fits of operation time to bytes on the passes with chosen ids. These fits are (4) for the per-slot kernel and its counterpart for the deduplicating kernel (Appendix E). We then score them on the windows of recorded routing, which the fits do not see. The test of the two inputs together is prediction. Bytes from routing over bandwidths from the benchmark give whole-model verify times τp (n), the time of one verification pass. They also give savings, best draft lengths and crossovers. A crossover is the smallest pass size from which the deduplicating kernel is faster. For a measurement, it is the smallest pass size from which the point estimate of the measured saving stays positive. We compare these predictions with whole-model runs that neither input used (Sections 4 and 4.4).

3

Theory

Section 3.1 writes the time of a step from the bytes each kernel requests and its effective bandwidth. Sections 3.2 and 3.3 state, in that step time, when a draft length is faster than plain decoding. They also state how the best draft length moves with acceptance. Section 3.4 specializes to a deduplicating engine. Section 3.5 bounds how a bandwidth error moves each prediction. Byte counts assume that the verified tokens are routed like consecutive tokens of a stationary process. Generated text that repeats its own earlier phrases violates this assumption (Section 4.1). Appendix B gives the proofs, with the Lean theorems that check them.

6

3.1

Step time

The verify time of a pass of n tokens on kernel p is τp (n) = t0 +

D(n) Ep (n) + + ϕp (n), βD βp (n)

(5)

where t0 ≥ 0 is time outside the memory reads and D(n) = D + kv(n − 1) counts the bytes read once per pass at the dense bandwidth βD . The kernel requests the expert bytes Ep (n): Eslot (n) for the per-slot kernel, and Ededup (n) times a byte factor fl ≥ 1 for the deduplicating kernel. The factor fl is the bytes that the deduplicating kernel reads per distinct expert of layer l, over wl . Finally, ϕp (n) ≥ 0 is fixed work, such as launches. The fit (4) describes the expert part of (5) for one layer of the per-slot kernel. Summed over layers, cu U b + cre (nm − U )b is Eslot (n)/βslot (n), and top is that layer’s share of ϕslot (n). The terms t0 and D(n)/βD lie outside the expert operation and have no counterpart in (4). The bandwidths of Table 1 divide the bytes by the whole operation time, so they include top . The predictions of Section 4 use those operation times directly. So the split between βp and ϕp does not enter them. A one-token pass runs the same kernel either way. So τ (1), β(1) and ϕ(1) carry no kernel index. The step time is V (k) = τp (k + 1) + h(k), with h(k) the time per step of the draft model and the host. Any measured time has some βp (n). So (5) says something only through where the bandwidth comes from (Section 2). Take fl = 1 and equal fixed work. Then t0 and the dense reads cancel in the saving S(n) = τslot (n) − τdedup (n). Because Ededup /βdedup = (Eslot /βslot ) ū/r, the saving is S(n) =

Eslot (n)  ū(n)  1− ; βslot (n) r(n)

(6)

unequal fixed work subtracts ϕdedup (n) − ϕslot (n). The deduplicating kernel is faster exactly when ū(n) < r(n).

(7)

The scale Eslot /βslot is the expert time of the per-slot kernel. One bandwidth for both kernels, r ≡ 1, makes the deduplicating kernel faster at every n ≥ 2.

3.2

Speedup condition

Take the step time V (k) and throughput Ta (k) of Section 1. A draft of length k ≥ 1 adds the time V (k) − V (0). It has higher throughput than plain decoding exactly when this time is less than the expected number of accepted drafts times the time of a plain step, Ta (k) > Ta (0) ⇐⇒ V (k) − V (0) < (a + a2 + · · · + ak ) V (0),

(8)

which with h = 0 reads τp (k + 1) − τ (1) < (a + · · · + ak ) τ (1) (Eq. (B.2)). The speedup is Ta (k) Ga (k) V (0) = . Ta (0) V (k)

(9)

At k = 1 the condition reads a > a0 = V (1)/V (0) − 1. So a0 is the acceptance above which one draft is faster than none. The condition holds for any positive step time and assumes no common bandwidth. By (5), V (k) carries the bandwidth βp (k + 1) of a pass of k + 1 tokens. Likewise, V (0) carries the bandwidth β(1) of a one-token pass. Each bandwidth is measured at its own pass size.

7

3.3

Staircase

The best draft length never decreases as acceptance rises, a ≤ a′ ⇒ k ∗ (a) ≤ k ∗ (a′ ). This holds because Ga (k)/Ga (i) is non-decreasing in a for i < k. The result does not assume a single peak of Ta in k, so the staircase can jump by more than one length. Each step point is exact: aj = maxi<j mink≥j αk,i , with αk,i the least acceptance above which draft length k has higher throughput than i (Eqs. (B.26) to (B.31)). If the step time is also convex and non-decreasing in k (0 ≤ c1 ≤ c2 ≤ . . . ), then j and j − 1 alone decide whether k ∗ (a) reaches j, k ∗ (a) ≥ j ⇐⇒ aj V (j − 1) > Ga (j − 1) cj ,

(10)

and a1 = min(c1 /V (0), 1) (Eqs. (B.35) and (B.36)). The algebra of this subsection and the previous one uses only V > 0. So these results hold for any positive step time, and the Lean statements are proved in that generality.

3.4

Deduplicating engines

The byte-weighted mean reuse of the k drafted tokens is k+1

ρ̄k =

1 X X wl ρl (j), kW

(11)

j=2

l

and by (2) the expert bytes of a pass of k + 1 tokens on a deduplicating kernel are Ededup (k + 1) = mW + mW k(1 − ρ̄k ),

(12)

a count that involves the routing alone. Take a deduplicating engine with fl = 1 and h = 0. Write Bk = βdedup (k + 1), ∆Dk = D(k + 1) − D(1) and ∆ϕk = ϕdedup (k + 1) − ϕ(1). By (5), a draft of length k adds the time V (k) − V (0) =

Ededup (k + 1) mW ∆Dk + ∆ϕk + − , βD Bk β(1)

(13)

and by (8) it is faster than plain decoding exactly when ρ̄k > ρ∗k (a), with ρ∗k (a) = 1 −

i 1 B  Bk h ∆Dk k (Ga (k) − 1) τ (1) − − ∆ϕk − −1 mW k βD k β(1)

(14)

(Eq. (B.14)). The threshold falls as a grows, since Ga (k) rises with a and τ (1), Bk > 0 (Eq. (B.19)). The bracket is the time that a draft may add outside the expert reads. The bandwidth Bk converts this time into union bytes. The last term asks for more reuse when the deduplicating kernel runs slower on k + 1 tokens than on one. Substituting (13) into the speedup (9) gives Ta (k) Ga (k) V (0) = , Ta (0) V (0) + ∆Dk /βD + ∆ϕk + Ededup (k + 1)/Bk − mW/β(1)

(15)

with the bytes Ededup (k + 1) fixed by the routing traces before any deduplicating kernel runs, and the bandwidths Bk and β(1) measured. With one bandwidth for every byte and t0 = ϕ = 0, (14) reduces to the byte form
(Ga (k) − 1)Vdedup (0) − kv(k) − kv(0) ∗ ρk (a) = 1 − , (16) mW k 8

Table 2: Error bounds for a relative bandwidth error |e| ≤ ε (first three rows) or a relative step-time error δ (last two). prediction

bound

verify time

ε ε |τ̂p (n) − τp (n)| ≤ 1−ε Mp (n) ≤ 1−ε τp (n)

saving

ε |Ŝ(n) − S(n)| ≤ 1−ε

faster kernel

1+ε (19) sides of (7) agree if | ln r(n) − ln ū(n)| > ln 1−ε 1−δ 2δ ∗ ∗ Ta (k̂ ) ≥ 1+δ Ta (k ): at most 1+δ of the throughput lost; 1+δ Ta (i) < Ta (k ∗ ) for every i ̸= k ∗ (20) k̂ ∗ = k ∗ if 1−δ T̂a (k̂∗ )/T̂a (0) 1−δ 1+δ (21) 1+δ ≤ Ta (k∗ )/Ta (0) ≤ 1−δ

best draft length best speedup

e e Mslot (n) + Mdedup (n)

(17)

(18)

with Vdedup (0) = D + mW + kv(0) the bytes of a plain step (Eq. (B.16)). This is the threshold that a pure byte count would give. When a > 0, the threshold (14) also falls as D grows, provided the bandwidths and the fixed work do not change with D (Eq. (B.19)). More reuse lowers every Ededup (k + 1). So a speedup persists under any routing with at least as much reuse. This holds provided that the bandwidth and the fixed work of the deduplicating kernel at fixed n do not depend on which experts repeat. Under the same proviso, routing enters only through (ρ̄1 , . . . , ρ̄kmax ). Also, no single rule “ρ̄k > c” decides for every routing (Appendix B.2).

3.5

Error bounds

Let β̂ = β(1 + e) for each bandwidth in (5), with |e| ≤ ε < 1 and the other terms exact. Here ε is the bandwidth error of Section 2. Write Mp (n) = D(n)/βD + Ep (n)/βp (n) for the memory part of τp (n) and Mpe (n) = Ep (n)/βp (n) for its expert part. A hat marks a quantity computed from the estimated bandwidths: M̂ , τ̂p , Ŝ, V̂ , T̂a = Ga /V̂ , and k̂ ∗ , the draft length that maximizes T̂a . For the best draft length and the best speedup, δ bounds the relative error of the full predicted step time, (1 − δ)V (k) ≤ V̂ (k) ≤ (1 + δ)V (k). With bandwidth errors alone, δ ≤ ε/(1 − ε). Then a 10% bandwidth error moves the predicted speedup by at most 25%. Table 2 states the five bounds, each proved in Lean (Appendix C). Relative to the saving, the saving bound (18) is ε/(1 − ε) times the sum of two expert times over their difference. This factor grows without limit near the crossover. A verify time carries the factor Mp /τp ≤ 1, and a common factor on every V (k) leaves k̂ ∗ = k ∗ . So the saving is the prediction most sensitive to a bandwidth error, and the best draft length is the least sensitive.

4

Experiments

Each engine gets one switch that sends small verification passes to its existing deduplicating expert kernel. With deduplication off, the engine keeps its default kernel choice: the per-slot kernel at n ≤ 9, or at n ≤ 8 on CUDA, which runs the deduplicating kernel at n = 9. With deduplication on, it runs the deduplicating kernel from n = 2 (Appendix E, Table 6). Appendix E gives the change to each engine and the check that both settings produce the same output. Each configuration has a context depth d ∈ {512, 4096}, the number of prompt tokens before decoding starts. OLMoE, Qwen3 and gpt-oss give twelve configurations. In each configuration we measure two quantities on the same six prompts. These prompts are articles of the WikiText-103 validation set truncated to d tokens, disjoint from the test-set documents that give the routing

9

traces. The first quantity is the verify time with deduplication off and on at n = 1, . . . , 9. The second is the throughput of speculative decoding at draft lengths k = 0, . . . , 8 and 11 acceptance values from 0.2 to 0.98. It uses the oracle draft, which proposes the target’s own greedy token and corrupts each draft with probability 1 − a. So acceptance is geometric at rate a (Appendix D). The prediction computes the time of each layer’s expert operation from its bytes. We made the fit (4) and its deduplicating counterpart on the passes with chosen ids of Section 2. We then evaluated each fit at the union bytes of every window of recorded routing from the WikiText-103 traces. So the prediction uses nothing from the whole-model runs. Then (22) below sums the layers into a whole-model verify time. Table 3 also gives the prediction with the recorded routing of the text those runs verified. The same sum with measured operation times is the measured-time version. It skips the fit and uses each layer’s measured median operation time on each window of recorded routing. It serves where a model has no fit. The gap between the measured-time version and the prediction is the fits’ error on recorded routing, which ε bounds. On OLMoE, Qwen3 and gpt-oss, the prediction and the measured-time version were computed after the whole-model data existed. As first computed, the prediction took the one-token time t̄l (1) from the deduplicating kernel’s fit. It then placed the crossover wrongly in ten of twelve configurations. We corrected it to the per-slot fit, because the engine runs the per-slot kernel at n = 1 with deduplication off and on. This was the only change made after a comparison with the measurements. DeepSeek-V2-Lite had no such fit. On it, we wrote down the measured-time version and its scoring rules, the tolerances it had to meet, before any whole-model speculative run (Section 4.4). The prediction and the measured-time version both take the whole-model verify time from the expert operation, L X
t̄p,l (n) − t̄l (1) , (22) τ̂p (n) = τ meas (1) + l=1

where τ meas (1) is the measured whole-model time of a one-token pass.

The term t̄p,l (n) is layer l’s operation time with kernel p, averaged over the windows of recorded routing. On the Apple engines, n = 9 is extrapolated. On OLMoE, Qwen3 and gpt-oss, the measured-time version uses the recorded routing of the text the whole-model runs verified. The term t̄l (1) is the same layer’s onetoken time, which the per-slot kernel runs with deduplication off and on. The predicted expert time P e is M̂p (n) = l t̄p,l (n). The saving, the crossover, the best draft length k̂ ∗ and the speedup follow, with V̂ (k) = τ̂dedup (k + 1) + h(k) and h(k) measured. The calibrated version is the measured-time version with one change. It takes the growth with n of the time outside the expert operation from the measured verify time with deduplication off. Besides caching and tokens per expert, fixed work separates the two kernels. With deduplication on, CUDA makes 160 kernel launches per OLMoE pass against 64, a difference of about 0.1 ms. We compute the bandwidth error ε from the fits of Section 2. Here εp is the median, over layers and models, of each fit’s median relative error on the windows of recorded routing from the WikiText-103 traces. The engine’s ε is the larger of its two kernels’ values: 0.054 on llama.cpp CUDA, 0.031 on llama.cpp Metal and 0.148 on MLX-LM. The measurement takes under an hour per model. For DeepSeek-V2-Lite it took 22 minutes on CUDA and 48 on Metal, against 3.8 and about 8 hours of whole-model speculative runs.

4.1

Bytes

We profiled the CUDA expert kernels with Nsight Compute (NVIDIA Corporation, 2026) on passes with chosen ids at n = 2, . . . , 8, on one layer of OLMoE and gpt-oss. In all 70 profiles, the deduplicating kernel reads the union bytes U b from DRAM within 0.2 to 2.5%. The per-slot kernel 10

0.9

nmb: no reuse served

OLMoE Qwen3

2.75 2.50 2.25 OLMoE gpt-oss per-slot kernel (filled) deduplicating kernel (open)

2.00 1.75 1.50 1.25

Ub: union

1.00

distinct share U/(nm), windows the runs verified

DRAM bytes read over Ub

3.00

gpt-oss DeepSeek-V2-Lite llama.cpp CUDA llama.cpp Metal

0.8 0.7 0.6 0.5 0.4 0.3

gpt-oss, d = 4096

0.2 2

4

6

8

0.2

requested over union bytes, nm/U

0.4

0.6

0.8

distinct share U/(nm), WikiText-103 windows

Figure 3: Bytes. Left: DRAM bytes read by the CUDA expert kernels over the union bytes U b, against nm/U (Nsight Compute, chosen ids, one layer of OLMoE and gpt-oss). Filled: per-slot kernel; open: deduplicating kernel; color: model. The deduplicating kernel reads U b (horizontal line). The per-slot kernel reads between U b and its requested nmb (dotted). Right: the share of a pass’s nm slots that name distinct experts, U/(nm), at n = 2, 4, 6, 8. It is shown on the windows the whole-model runs verified, against WikiText-103 windows of the same model. Both axes lie between 1/n (every token repeats the first token’s experts) and 1 (no expert repeats). A point above the dotted line means the verified text repeated experts less often than WikiText-103. Color: model; shape: the engine of the runs (circle: llama.cpp CUDA, triangle: llama.cpp Metal). The labeled gpt-oss points at d = 4096 below the line are its repeating continuations. requests nmb and reads between U b and 2.35 U b (Figure 3). Here the L2 cache serves part of the repeated slots, an effect that belongs to the per-slot kernel’s effective bandwidth. The union is as good as the routing it is given. On the windows the whole-model runs verified, the share of distinct slots U/(nm) agrees with that of WikiText-103 windows within 0.06. This holds at every n on every model except gpt-oss at d = 4096. There the greedy continuations repeat, and the share lies 0.08 to 0.16 below: 81% of their 4-grams recur, against 48% at d = 512.

4.2

Saving and crossover

The prediction’s largest saving error is 0.23 to 0.52 ms on CUDA and 0.43 to 3.7 ms on the Apple engines (Table 3). The exception is Metal gpt-oss at d = 4096, with 9.7 ms, whose verified text repeats and routes unlike WikiText-103 (Figure 3). Its crossover is within one of the measured one in 10 of 12 configurations. It equals the measured crossover in all six other Apple configurations and is one pass size early in three of the four CUDA configurations. It misses both Metal gpt-oss configurations. With the recorded routing of the verified text, the saving error is 0.25 to 0.51 ms on CUDA and 0.42 to 3.8 ms on the Apple engines. The crossover is then within one in 11 of 12 configurations. The measured-time version gives saving errors of 0.17 to 0.20 ms on CUDA and 0.8 to 3.9 ms on the Apple engines. Its crossover is within one in 11 of 12 configurations and equal in all four CUDA configurations. The remaining miss is Metal gpt-oss at d = 512. Its measured saving at n = 8 is +0.2 ms, with an interval containing 0, and there the margin of (19) fails. Figure 4 shows the saving itself at every n. The saving is the difference of two expert times of 2.8 to 40.0 ms each. So the saving bound (18) amplifies a bandwidth error 3.2 to 3.8 times on MLX-LM, 6 to 220 times on CUDA and 3.2 to 340 times on Metal. The amplification is largest at the crossover (Appendix E, Table 7). With each engine’s ε, the measured saving lies inside the band 11

Table 3: The prediction with WikiText-103 routing (wiki) and with the routing of the verified text (text), and the measured-time version (op. time). All three are compared with the same measurements, one row per configuration. Saving error: largest |Ŝ(n) − S(n)| over n = 2, . . . , 9 (n ≤ 8 on CUDA), in ms. Crossover as defined in Section 2 (meas.: measured; −: none). k ∗ : acceptance values, of 11, at which the predicted best draft length is within one of the measured one. Metal and CUDA are llama.cpp. saving error (ms)

k ∗ within one

crossover

engine

model

d

wiki text op. time meas. wiki text op. time wiki text op. time

Metal Metal Metal Metal Metal Metal MLX-LM MLX-LM CUDA CUDA CUDA CUDA

OLMoE OLMoE Qwen3 Qwen3 gpt-oss gpt-oss OLMoE OLMoE OLMoE OLMoE gpt-oss gpt-oss

512 4096 512 4096 512 4096 512 4096 512 4096 512 4096

0.43 0.98 1.51 3.70 2.23 9.72 2.60 2.51 0.23 0.26 0.35 0.52

0.42 0.91 0.92 1.83 3.77 3.46 2.86 2.64 0.25 0.26 0.49 0.51

0.77 2.28 2.24 1.25 3.89 3.89 3.29 2.37 0.20 0.17 0.18 0.19

7 7 − − 8 6 − − 5 5 6 5

7 7 − − − − − − 4 4 5 5

7 7 − − − 7 − − 4 4 5 4

8 8 − − − 7 − − 5 5 6 5

9 8 7 8 10 10 8 7 11 11 10 10

9 8 8 8 9 9 7 7 11 11 10 11

9 8 7 8 10 10 8 9 11 11 11 11

around the prediction at all 28 CUDA points, all 16 MLX-LM points and 36 of 48 Metal points. Consider the bandwidth error at which the band just covers a configuration’s largest saving error. It is 0.022 to 0.033 on CUDA and 0.107 to 0.108 on MLX-LM, below the engines’ ε. On Metal it is 0.014 to 0.126. That is above Metal’s 0.031 in three of six configurations, and most above it on gpt-oss at d = 4096. At n = 1, an Apple engine runs the same kernel with deduplication off and on. There the engine with deduplication on measures slower in 6 of 8 configurations. So part of every Apple saving is a run-to-run offset.

4.3

Best draft length and speedup

On CUDA, the predicted k̂ ∗ is within one of the measured best at 10 or 11 of 11 acceptance values in every configuration. The measured-time version places it within one at all 11 (Table 3). Drafting the measured-time version’s k̂ ∗ loses at most 1.6 to 7.7% of the best measured throughput. On the Apple engines both place k̂ ∗ within one at 7 to 10 of 11 values. An error common to every step time leaves k̂ ∗ unchanged (Section 3.5). On the Apple engines, an error that grows with n moves it. Neither the prediction nor the measured-time version contains the growth with n of attention and the dense parts. This growth cancels in the saving and stays in the verify time. With deduplication on, the prediction’s verify time is within 3 to 21% on CUDA, 6 to 25% on Metal and 38 to 41% on MLX-LM. The measured-time version’s is within 4 to 16% on CUDA and 15 to 31% on the Apple engines. The calibrated version’s is within 1 to 6% in 8 of 12 configurations (Appendix E, Table 7). We set δ of (20) to the largest relative error of each configuration’s step time, under the measured-time version and under the calibrated version. This gives 24 pairs of a configuration and a version, plus 6 DeepSeek-V2-Lite pairs. At all 330 acceptance values of these pairs, the throughput lost lies inside its bound (20) and the best speedup inside its bound (21) (Figure 5). Measured speculative runs on CUDA differ from the speedup computed from verify times by a 12

saving bound, ε per engine

measured S(n), 95% interval

measured-time version

prediction

llama.cpp Metal OLMoE, d = 512 llama.cpp Metal OLMoE, d = 4096 llama.cpp Metal Qwen3, d = 512 llama.cpp Metal Qwen3, d = 4096 0.0

saving (ms)

2

2

−5.0

0

0

−2

−10.0

−2

2

4

6

8/8

8

−5

−7.5

8/8 2

4

6

−10 8/8

−12.5

8

2

llama.cpp Metal gpt-oss, d = 512 llama.cpp Metal gpt-oss, d = 4096

saving (ms)

0

−2.5

4

6

4/8

8

2

MLX-LM OLMoE, d = 512

10

0.0

0.0

0

5

−2.5

−2.5

0

−5.0

−5.0

−5

−7.5

−10

7/8 2

4

6

1/8

−10

8

2

4

6

8

−7.5

−10.0

8

6

MLX-LM OLMoE, d = 4096

5

−5

4

8/8 2

4

6

−10.0

8

8/8 2

4

6

8

llama.cpp CUDA OLMoE, d = 512 llama.cpp CUDA OLMoE, d = 4096 llama.cpp CUDA gpt-oss, d = 512 llama.cpp CUDA gpt-oss, d = 4096

saving (ms)

1.5

1.5

1.0

1.0

0.5

0.5

0.0

0.0

−0.5

7/7 2

4

6

8

tokens per pass n

7/7

−0.5 2

4

6

3

3

2

2

1

1

0

0

−1

−1 7/7

−2

8

2

tokens per pass n

4

6

tokens per pass n

8

7/7

−2 2

4

6

8

tokens per pass n

Figure 4: Saving S(n). Black and grey: measured on the six validation prompts, with its 95% bootstrap interval. Red: the prediction, dotted to an open circle where n = 9 is extrapolated. Shaded red: the band of the saving bound (18) for the engine’s ε. Grey dashed: the measured-time version. The number in each panel counts the measured points inside the band. Positive: the deduplicating kernel is faster. CUDA is shown to n = 8, since it runs the deduplicating kernel at n = 9 with deduplication off and on. median factor of 0.93 to 1.04 per configuration. Part of the reason is that the oracle draft emits 2 to 5% fewer tokens per step than Ga (k) (Appendix E).

4.4

DeepSeek-V2-Lite

We use DeepSeek-V2-Lite-Chat in the 4-bit llama.cpp format Q4 K M (DeepSeek-AI, 2024). It has 64 routed experts (6 per token), two shared experts that every token uses, multi-head latent attention and a dense first layer. The other three models have none of these. On llama.cpp CUDA and Metal, the measured-time version follows from (22). Its inputs are runs of the expert operation of all 26 MoE layers on WikiText-103 windows, and three whole-model numbers at k = 0. We wrote the measured-time version, six scoring rules and the scoring script before the first speculative run on each engine. The script then scored the runs unchanged. On CUDA, the deduplicating kernel is predicted and measured slower at every n from 2 to 8. The saving lies inside its band at all 14 points, and the best draft length is within one at all 22 acceptance values. The verify time is within 10.1% (Table 4). On Metal, the sign of the saving is right at every n, but the verify time is predicted up to 34% too low. The reason is the growth with n of attention, the shared experts and the dense layer. This growth is large on this model and lies outside the expert operation. The error grows with n, so drafting k̂ ∗ loses up to 22.5% of the best throughput. The calibrated version was written down beforehand as a second prediction. 13

gpt-oss

DeepSeek-V2-Lite

largest throughput lost at k̂

*

8

predicted k ̂

llama.cpp CUDA

llama.cpp Metal

6 4 2 0 0

2

4

6

0.6 ro er

r=

bo

MLX-LM

measured-time version

largest speedup error

Qwen3

*

OLMoE

r=

d un

0.4

0.2 inside the bound

0.0

8

0.0

0.2

0.4

0.6

draft-length bound 2δ/(1 + δ)

measured k *

calibrated version

1.5

ro er

bo

d un

1.0 0.5 inside the bound

0.0 0.0

0.5

1.0

1.5

speedup bound (1 + δ)/(1 − δ) − 1

Figure 5: Best draft length and the bounds (20) and (21). Each configuration has one point under the measured-time version (filled) and one under the calibrated version (open), with δ as in Section 4.3. Color: model; shape: engine (circle: llama.cpp CUDA, triangle: llama.cpp Metal, square: MLX-LM). Left: predicted against measured k ∗ , measured-time version; area grows with the number of acceptance values; dotted: exact prediction; shaded: within one. Middle: largest share of throughput lost by drafting k̂ ∗ , against its bound (20). Right: largest relative error of the predicted best speedup, against its bound (21). In the middle and right panels the solid line is where the error equals its bound. The theory requires every point to lie in the shaded region on or below it, and every point does. Table 4: DeepSeek-V2-Lite-Chat: the measured-time version and the calibrated version, written before its speculative runs. Crossover: predicted / measured (−: none), over n ≤ 8 on CUDA. Sign: values of n with the measured sign of S(n), of those whose interval excludes 0. Saving error in ms. k ∗ : acceptance values, of 11, with k̂ ∗ within one of the measured best. Loss: largest share of the best throughput lost at k̂ ∗ . τ error: largest relative error of the predicted verify time, over both kernels and n = 2, . . . , 9. δ as in Section 4.3. The best speedup lies inside its bound (21) in every row. engine

version

llama.cpp CUDA measured-time measured-time llama.cpp Metal measured-time measured-time calibrated calibrated

d

crossover

sign

512 4096 512 4096 512 4096

−/− −/− −/− −/− −/− −/−

7 of 7 7 of 7 8 of 8 8 of 8 8 of 8 8 of 8

saving error k ∗ within 1 0.23 0.28 1.6 4.4 1.5 4.4

11 11 9 6 11 9

loss

τ error / δ

2.9% 9.9% / 0.072 4.1% 10.1% / 0.077 20.1% 34.0% / 0.27 22.5% 28.4% / 0.25 6.4% 4.3% / 0.036 8.3% 10.5% / 0.098

It brings the verify time within 4.3% and 10.5%, and the best draft length within one at 20 of 22 values. The Metal saving requires a bandwidth error of up to 0.075, 2.4 times Metal’s ε of 0.031. Of the six scoring rules (Appendix E, Table 8), the CUDA measured-time version meets all six at both depths. The Metal measured-time version meets the crossover, sign and speedup rules. It fails the rules on the best draft length, the throughput lost and the verify time. On CUDA, the measured throughput speedup lies 1 to 21% below the predicted one. This is the gap of the oracle draft (Section 4.3), which (21) does not cover. The prediction needs fits on chosen ids. We ran them on DeepSeek-V2-Lite afterwards, on layers 1 and 13, with the protocol of the other models. We then applied the chain unchanged to 14

saving bound, ε per engine

llama.cpp CUDA, d = 512

measured S(n), 95% interval

llama.cpp CUDA, d = 4096

measured-time version

llama.cpp Metal, d = 512 0.0

1

saving (ms)

1

prediction

llama.cpp Metal, d = 4096 0

−2.5 −5

−5.0 0

0

−7.5 −1

7/7 2

4

6

tokens per pass n

8

−10

−10.0

−1

6/7 2

4

6

7/8

−12.5

8

2

tokens per pass n

4

6

8

tokens per pass n

4/8

−15 2

4

6

8

tokens per pass n

Figure 6: Saving S(n) on DeepSeek-V2-Lite, as in Figure 4. Black and grey: measured, with its 95% interval. Red: the prediction from fits run after the whole-model runs, with the band of the saving bound (18) for the engine’s ε. Grey dashed: the measured-time version, written before its speculative runs. The number in each panel counts the measured points inside the band. the same WikiText-103 windows. The whole-model results were known by then, and we adjusted nothing in the chain. The fits’ errors on recorded routing are 0.021 to 0.049 on Metal and 0.027 to 0.129 on CUDA. On Metal, the prediction matches or improves on the measured-time version at both depths. Its saving error is 1.1 and 3.7 ms, against 1.6 and 4.4. No crossover is predicted or measured. Both place the best draft length within one at 9 and 6 of 11 values. The prediction’s verify time is within 31.6% and 26.4%, against 34.0% and 28.4%, over both kernels and n = 2, . . . , 9 as in Table 4. On CUDA, the prediction keeps the verify time within 8.0% and 8.3%, and the best draft length within one at all 22 values. But the per-slot fit predicts the windows of recorded routing 5 to 13% slower than they run. That adds about 0.9 ms of per-slot time at n = 8, against a measured saving of about −0.5 ms. So the prediction places a crossover at n = 7 where none is measured. Its saving errors are 0.95 and 1.1 ms, against 0.23 and 0.28 ms for the measured-time version (Figure 6). With the engines’ ε from Section 4, the measured saving lies inside the band around the prediction at 13 of 14 CUDA points and 11 of 16 Metal points.

5

Limits

The theory assumes geometric acceptance, verified tokens routed like consecutive tokens of a stationary process, and no cost for the draft model. The theory does not predict the bandwidth of any kernel. We measure each kernel’s bandwidth on each model’s expert shapes. Bandwidth fitted on the other models of an engine misses a held-out model by 9 to 206% (Appendix E). The prediction on OLMoE, Qwen3 and gpt-oss was computed after their whole-model data existed. On DeepSeek-V2-Lite, the prediction used fits run after the whole-model runs. Only the measuredtime and calibrated versions on DeepSeek-V2-Lite were written before its speculative runs. On OLMoE, Qwen3 and gpt-oss, the measured-time version uses the recorded routing of the verified continuations, which approximates that of verify windows containing drafts. On Metal, the times of the expert operation alone miss part of each kernel’s growth inside the whole model. The sign of this miss depends on the model. From n = 7, the per-slot kernel exceeds its growth on the expert operation alone on gpt-oss and OLMoE. On DeepSeek-V2-Lite the deduplicating kernel does, by

15

1.1 to 4.4 ms. The Metal runs of DeepSeek-V2-Lite at d = 512 show a recurring slow GPU state with no thermal warning logged. The scoring rules include these runs. The measurements use one machine per platform, at batch one, with four models of 7B to 30B total parameters. DRAM byte counts exist only on CUDA.

6

Conclusion

A speedup is a ratio of step times, and a step’s time is its bytes over the effective bandwidth of the kernel that reads them. In that step time, the theory states exactly when speculation is faster, at which draft length and by how much. These results are proved in Lean. On a deduplicating engine the bytes follow from routing, and the CUDA deduplicating kernel reads them within 2.5%. The per-slot kernel has the higher effective bandwidth on every engine and model measured. So the deduplicating kernel is faster only where ū(n) < r(n). Its saving is the per-slot expert time times 1 − ū(n)/r(n) (6). The bandwidth’s error carries through the proved bounds as stated on CUDA and MLX-LM. It moves the best draft length and the speedup little on CUDA, and the saving amplifies it. On DeepSeek-V2-Lite, operation times measured before any of its speculative runs give the CUDA best draft length within one at every acceptance value. The prediction from fits run afterwards misses its CUDA crossover. The results change how a cost model for MoE speculation should charge expert reads. Models with one bandwidth for every byte, such as S2-MoE (Huang et al., 2026) and SpecFit (Cao et al., 2026), imply that deduplication with equal fixed work is faster wherever ū(n) < 1. That is every pass size with reuse. On every engine and model that we measured, the deduplicating kernel is slower at two tokens (Section 2). A cost model must therefore compare the byte share ū(n) with the bandwidth ratio r(n). The bytes alone do not decide the sign of the saving. For engine developers, the switch between the two kernels can follow from measurement. Engines currently tune it by hand (pratiknarola-t, 2026). Routing traces give ū(n), and the benchmark of the expert operation gives r(n) in under an hour per model. Together with the fixed work, they give the predicted crossover, and the deduplicating kernel should run from there. The ratio r(n) is also the lever for kernel work. With equal fixed work, a deduplicating kernel at r(n) = 1 is faster at every pass size with reuse. A higher bandwidth at small n can move the crossover to fewer tokens. Four extensions follow from the limits above. First, the bandwidth needs a model from kernel structure, since a bandwidth fitted on other models misses a held-out model. Second, a real draft model and acceptance that is not geometric would test the staircase under the conditions of deployment. Third, batches larger than one change both the routing union and the kernels’ bandwidths. Fourth, hardware with lower bandwidth or weaker compute, such as edge devices, would test whether the step time stays limited by memory reads.

Code and data availability The modified engines, the benchmark of the expert operation, the chosen ids, the routing traces, the timing files and the Lean formalization are available from the author on request.

Use of AI AI agents directed by the author produced the code that ran the experiments, generated the figures and tables and consolidated the statistics. They also produced the Lean formalization, the initial

16

literature review and the initial draft of the manuscript. The author chose the research direction, designed the experiments, reviewed the results and findings, and is responsible for the paper.

References Apple Inc. (2024). MacBook Pro (14-inch, 2024) technical specifications. https://support.apple. com. Memory bandwidth of the M4 Max with 40-core GPU: 546 GB/s. Cao, H., Li, X., Wang, S., Wang, Z., Zhu, J., & Yang, H. (2026). SpecFit: Not all draft tokens cost the same in offloaded MoE inference. arXiv preprint arXiv:2610.15417. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., & Jumper, J. (2023). Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Chordiya, P. (2026). Lossless but not free: An empirical anatomy of speculative decoding on consumer hardware. arXiv preprint arXiv:2607.17283. de Moura, L. & Ullrich, S. (2021). The Lean 4 theorem prover and programming language. In Automated Deduction – CADE 28, volume 12699 of Lecture Notes in Computer Science, pages 625–635. Springer. DeepSeek-AI (2024). DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434. Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. Gerganov, G. & llama.cpp contributors (2026). llama.cpp. Release b11617.

llama.cpp.

https://github.com/ggml-org/

Gong, Y., Budhathoki, K., Kim, T., Li, H., & Khetan, A. (2026). MonoMoE: An efficient fused mega-kernel for quantized MoE decoding. arXiv preprint arXiv:2609.04244. Haddad, F. (2026). [Performance] gather qmm at M=1 streams 4-bit expert weights at about half the bandwidth of M=4 (MoE decode). https://github.com/ml-explore/mlx/issues/4509. MLX issue 4509; not peer reviewed. Huang, H., Qiu, S., & Li, M. (2026). S2-MoE: Enabling efficient self-speculative decoding for mixture-of-experts on edge devices. arXiv preprint arXiv:2608.15018. Huang, Z., Zhu, L., Zhan, Z., Hu, T., Mao, W., Yu, X., et al. (2025). MoESD: Unveil speculative decoding’s potential for accelerating sparse MoE. arXiv preprint arXiv:2505.19645. Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast inference from Transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. Li, S., Lin, J., Ge, D., & Ye, Y. (2026). MoE-SpAc: Efficient MoE inference based on speculative activation utility in heterogeneous edge scenarios. arXiv preprint arXiv:2603.09983. Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2017). Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR). 17

Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Morrison, J., Min, S., et al. (2025). OLMoE: Open mixture-of-experts language models. In International Conference on Learning Representations (ICLR). NVIDIA Corporation nsight-compute.

(2026).

Nsight

Compute.

https://developer.nvidia.com/

Oncescu, C.-A., Wu, Q., Chung, W. T., Wu, R., Gopal, B., Wang, J., et al. (2025). Opportunistic expert activation: Batch-aware expert routing for faster decode without retraining. arXiv preprint arXiv:2511.02237. OpenAI (2025). gpt-oss-120b and gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2023). Efficiently scaling Transformer inference. In Proceedings of Machine Learning and Systems (MLSys), volume 5. pratiknarola-t (2026). metal : few-row MMA kernels for MUL MAT ID and more src0 types. https://github.com/ggml-org/llama.cpp/pull/30047. llama.cpp pull request 30047, open as of October 2026; not peer reviewed. Saxena, A., Tsai, P.-A., Taneja, H., Jaleel, A., & Qureshi, M. (2025). Utility-driven speculative decoding for mixture-of-experts. arXiv preprint arXiv:2506.20675. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR). Suram, S. R. (2026). Cacheable by design? Training mixture-of-experts routers for locality against the edge memory-bandwidth wall: A pre-registered negative result with a systems measurement study. arXiv preprint arXiv:2608.18261. The mathlib Community (2020). The Lean mathematical library. In Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs (CPP), pages 367–381. ACM. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., et al. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388. Zhang, H. (2026). Budgeting bytes: A windowed storage roofline and dual-budget architecture ablations for storage-bound LLM decoding. arXiv preprint arXiv:2609.04238. Zhang, J., Wu, L., Darbari, M., & Hong, L. (2026). WiSP: A working-set view of mixture-of-experts serving on extremely low-resource hardware. arXiv preprint arXiv:2606.21868. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., et al. (2023). Judging LLMas-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS), Datasets and Benchmarks Track.

18

A

Notations Symbol

Meaning

a a0

acceptance rate (geometric acceptance) V (1)/V (0) − 1, the acceptance above which one draft is faster than none j-th step point bytes of one expert of the timed layer, b = wl βdedup (k + 1) V (i) − V (i − 1), the time the i-th draft adds per-slot fit (4): time per byte of a first read and of a repeated slot deduplicating fit (E.1): time per union byte and per token context depth in tokens bytes read once per forward pass whatever the routing bytes read once per pass of n tokens, D + kv(n − 1) relative bandwidth error (Section 3.5) expectation expert bytes kernel p requests on a pass of n tokens expected expert bytes of a pass of n tokens with the per-slot and the deduplicating kernel; Ededup are the union bytes bytes the deduplicating kernel reads per distinct expert of layer l, over wl (1 − ak+1 )/(1 − a), expected tokens emitted per step draft model’s and host’s time per step aj /Ga (j − 1) (Appendix B) draft length; its maximum best draft length, the smallest maximizer of Ta key-value cache bytes read by a step verifying k + 1 tokens number of MoE layers; layer index experts selected per token memory part of τp (n) expert part of τp (n), Ep (n)/βp (n) number of consecutive tokens, or tokens verified in one pass experts per layer expert kernel of a pass: slot (per-slot) or dedup (deduplicating) 1 + x + · · · + xk (Appendix B) 1 − m/N bandwidth ratio βdedup (n)/βslot (n) saving of deduplication, τslot (n) − τdedup (n) (6) routing set: the m experts layer l selects for the token at position t token position time outside the pass’s memory reads fixed time of a fit on chosen ids operation time of layer l with kernel p, averaged over windows of recorded routing: the fit at each window’s bytes in the prediction, the measured median in the measured-time version (22)

aj b Bk ci cu , cre cg , ct d D D(n) e E Ep (n) Eslot (n), Ededup (n) fl Ga (k) h(k) hj (a) k, kmax k ∗ (a) kv(k) L, l m Mp (n) Mpe (n) n N p Pk (x) q r(n) S(n) Sl,t t t0 top t̄p,l (n)

19

Symbol

Meaning

Ta (k) Ttr U Ul (n)

throughput, Ga (k)/V (k) tokens per second trace length distinct experts of a pass in the benchmark mean number of distinct experts that n consecutive tokens select in layer l Ededup (n)/Eslot (n), the share of the per-slot bytes that deduplication requests step time of a step drafting k tokens bytes of a step with each kernel, the one-bandwidth case (Appendix B) P bytes of one expert of layer l; W = l wl number of distinct experts that n consecutive tokens select in layer l, from position t

ū(n) V (k) Vslot , Vdedup wl , W Xl,t (n) αk,i βD βp (n) δ ∆Dk , ∆ϕk ε ρl (n), ρ̂l (n) ρ̄k ρ∗k (a) τp (n) ϕp (n) χ x̂

infimum of the acceptances at which draft length k is faster than i, 1 if none (Appendix B) bandwidth of the bytes read once per pass effective bandwidth of kernel p: its requested expert bytes over its time bound on the relative error of the predicted step time D(k + 1) − D(1); ϕdedup (k + 1) − ϕ(1) bound on the relative bandwidth error, measured per engine (Section 4) reuse of n consecutive tokens (1); its estimate on traces byte-weighted mean reuse of the k drafted tokens (11) reuse threshold: the least ρ̄k at which draft length k is faster than plain decoding (14) verify time of a pass of n tokens with kernel p (5) fixed work per pass with kernel p cost of a drafted token relative to a target step, in the affine cost of Leviathan et al. (2023) a quantity computed from the estimated bandwidths, such as M̂pe , V̂ , T̂a and k̂ ∗

20

B

Proofs

This appendix derives each result of Section 3 under its exact conditions. Each paragraph starts from its assumptions, and its heading names the equation of Section 3 it derives. A result with no equation there gets a numbered display here. Acceptance is geometric at rate a ∈ [0, 1), and a step with draft length k costs V (k) > 0. A result that names no kernel holds for every positive cost, measured times included. The results for a deduplicating engine (Appendix B.2) assume that the k + 1 verified tokens are routed like k + 1 consecutive tokens of a stationary routing process. They are derived in the step time (5), with the bandwidth of each kernel at each pass size. The byte cost Vdedup (k) = D + Ededup (k + 1) + kv(k), with one bandwidth for every byte, is their special case of Appendix B.2. Notation is that of Section 1 and Section 3. Throughout, Pk (x) = 1+x+· · ·+xk , so Ga (k) = Pk (a) for a ∈ [0, 1), and every cost is positive. For each result proved in Lean, we name the module of the accompanying Lean package MoESpec and its theorems at the step they prove. We also say which steps, if any, Lean does not cover. The package builds with Lean 4.34.1 against a pinned Mathlib. Every cited theorem uses only the axioms propext, Classical.choice and Quot.sound. The probability models fix the start of the window and live on a finite probability space. The other Lean statements have been compared line by line with the paper’s statements. Two groups have not yet been compared in this way: the reuse threshold in time (module MoESpec.TimeThreshold) and the error bounds (17) to (21) (module MoESpec.ErrBounds). Where the only Lean proof of a result for a deduplicating engine is stated for the byte cost, we say so. In that case the step from the byte cost to the step time is by hand. Exact-arithmetic brute-force checks on small cases, which we do not list, also cover four groups of results. These are the speedup condition, the one-bandwidth (byte) forms of the results for a deduplicating engine, the staircase, and the step points under an increment condition. The time forms of the results for a deduplicating engine and the error bounds have no such checks. The Lean models fix the start of the window of positions, and stationarity enters only to drop it. They work on a finite probability space of traces. This space covers every routing process on traces of length Ttr , since the set of such traces is finite.

B.1

Speedup condition

Speedup condition (8). Here kmax ≥ 1, V : {0, . . . , kmax } → (0, ∞) is any cost, a ∈ [0, 1) and k ∈ {1, . . . , kmax }. No property of V beyond positivity is used. The geometric sum gives Ga (0) = 1,

Ga (k) − 1 = a + a2 + · · · + ak .

(B.1)

Since V (0), V (k) > 0, multiplying by V (0)V (k) > 0 turns the comparison of throughputs into one of costs,
Ga (k) 1 > ⇐⇒ Ga (k) V (0) > V (k) ⇐⇒ V (k)−V (0) < Ga (k)−1 V (0), V (k) V (0) (B.2) which by (B.1) is (8) (Lean: MoESpec.SpeedupCond, faster at k). Speculation gives a speedup at a when k ∗ (a) ≥ 1, and Ta (k) > Ta (0) ⇐⇒

k ∗ (a) ≥ 1 ⇐⇒ Ta (k) > Ta (0) for some k ∈ {1, . . . , kmax }.

(B.3)

If k ∗ (a) ≥ 1, then k ∗ (a) is a maximizer and 0 is not, because the smallest maximizer is positive. So Ta (k ∗ (a)) > Ta (0). Conversely, if some k ≥ 1 has Ta (k) > Ta (0), then 0 is not a maximizer and k ∗ (a) ≥ 1. With (B.2), speculation therefore gives a speedup at a if and only if V (k) − V (0) < (Ga (k)−1)V (0) for some k ∈ {1, . . . , kmax }. In Lean this is MoESpec.SpeedupCondIff, faster iff, 21

whose “faster” is the right side of (B.3). The equivalence (B.3) is by hand. For k = 1, Ga (1)−1 = a, so (B.2) reads V (1) − V (0) < aV (0). We divide by V (0) > 0 and repeat the computation with equality for the second case. This gives Ta (1) > Ta (0) ⇐⇒ a > a0 ,

Ta (1) = Ta (0) ⇐⇒ a = a0 ,

a0 =

V (1) −1 V (0)

(B.4)

(Lean: MoESpec.FirstTokenThreshold, thresh and thresh eq).

B.2

Deduplicating engines

Union bytes (2), (12). Here the routing process is stationary, with sets of m ≥ 1 experts and wl > 0. Nothing about a kernel or a bandwidth enters. Fix l, t and n ≥ 1, and let R = Sl,t ∪ · · · ∪ Sl,t+n−2 , which is empty for n = 1. Since |Sl,t+n−1 | = m, Xl,t (n) = |R ∪ Sl,t+n−1 | = |R| + |Sl,t+n−1 | − |R ∩ Sl,t+n−1 | = Xl,t (n − 1) + m − |Sl,t+n−1 ∩ R|. (B.5) We take expectations with both windows starting at t. Stationarity makes the result independent of t. Using E|Sl,t+n−1 ∩ R| = mρl (n) by (1), we get
Ul (n) − Ul (n − 1) = m 1 − ρl (n) (B.6) (Lean: MoESpec.ReuseUnion, increment). Summing from Ul (0) = 0 gives (2) (telescope). Multiplying by wl and summing over l gives (3) and X X
Ededup (n) = m wl 1 − ρl (j) (B.7) j≤n

l

(MoESpec.ClassGap, union eq and gap). At n = k + 1 the term j = 1 is m ρl (1) = 0. By the definition (11) of ρ̄k , the remaining terms give Ededup (k + 1) = mW + m

X l

wl

k+1 X

P

l wl = mW , since

1 − ρl (j) = mW + mW k − mW k ρ̄k ,

(B.8)

j=2

which is (12). This step is by hand. In Lean it appears only inside the byte cost (MoESpec.SpeedupCond, lemma Vu diff). The union of k +1 sets contains the first set’s m experts, so Xl,t (k + 1) ≥ m. With wl > 0, this gives Ededup (k + 1) ≥ mW > 0

(B.9)

(Lean: MoESpec.Positivity, positivity, whose second and third parts are (B.9)). Step time (13). Here the engine deduplicates with fl = 1, the draft model’s and the host’s time h is zero, and the routing process is as in the previous paragraph. By (5), t0 ≥ 0, βD > 0, D ≥ 0 and kv ≥ 0. The fixed work ϕ(1) and ϕdedup (k + 1) is non-negative. The bandwidths β(1) and Bk = βdedup (k + 1) are positive, and each is a function of the pass size alone. A one-token pass reads Ededup (1) = mW expert bytes. So with (12), the step times are, for k ∈ {1, . . . , kmax }, D(1) mW + + ϕ(1), βD β(1) D(k + 1) mW + mW k(1 − ρ̄k ) V (k) = τdedup (k + 1) = t0 + + + ϕdedup (k + 1). βD Bk V (0) = τ (1) = t0 +

22

(B.10)

When we subtract, t0 cancels. With ∆Dk and ∆ϕk as in Section 3.4, the difference is (13). Every term of (B.10) is non-negative. By (B.9), the expert term of V (k) is at least mW/Bk , so V (0) ≥

mW > 0, β(1)

V (k) ≥

Ededup (k + 1) mW ≥ > 0. Bk Bk

(B.11)

The byte inequality (B.9) is in Lean. The step from it to (B.11) is by hand, and the Lean theorems in time take positivity of V as a hypothesis. For the byte cost, MoESpec.Positivity, Vu pos and U1 holds prove Vdedup (k) ≥ Ededup (k + 1) ≥ mW > 0 from the setting and discharge that hypothesis. A non-negative h would enter (B.10) as extra fixed work, h(0) in ϕ(1) and h(k) in ϕdedup (k + 1). It would leave every step below unchanged. Reuse threshold (14). Here the step times are (B.10) under the assumptions of the previous paragraph. Also a ∈ [0, 1), k ∈ {1, . . . , kmax } and ρ∗k (a) is (14). Write
∆Dk Y = Ga (k) − 1 τ (1) − − ∆ϕk βD

(B.12)

for the bracket of (14). By (13),

mW k ∗ mW + mW k(1 − ρ̄k ) mW V (k) − V (0) − Ga (k) − 1 V (0) = − −Y = ρk (a) − ρ̄k , (B.13) Bk β(1) Bk k ∗ mW k mW where the last equality expands mW = − Y − mW (Lean: Bk ρk (a) Bk β(1) + Bk MoESpec.TimeThreshold, gap identity). That theorem needs only m, W, β(1), Bk > 0 and k ≥ 1. Since V > 0 by (B.11), (B.2) applies. So Ta (k) > Ta (0) if and only if the left side of (B.13) is negative. Since mW k/Bk > 0, this gives

Ta (k) > Ta (0) ⇐⇒ ρ̄k > ρ∗k (a)

(B.14)

(faster time, with positivity of V as a hypothesis). By (B.3), speculation gives a speedup at a if and only if ρ̄k > ρ∗k (a) for some k ∈ {1, . . . , kmax }. Each k has its own Bk , ∆Dk and ∆ϕk . This union over k is by hand in time form, and in Lean only for the byte cost (MoESpec.SpeedupCondIff, faster iff reuse). Now fix t0 , βD , D(k + 1), ϕdedup (k + 1) and Bk . Then only the expert term of V (k) in (B.10) depends on ρ̄k . Its numerator mW + mW k(1 − ρ̄k ) falls as ρ̄k rises. Since Bk > 0, V (k) is non-increasing in ρ̄k , and V (0) does not depend on it: ρ̄k ≤ ρ̄′k =⇒ V (k) ρ̄′ ≤ V (k) ρ̄ k

k

(B.15)

(tauk mono). Suppose moreover that βD = Bk = β(1) = β > 0 and t0 = ϕ(1) = ϕdedup (k + 1) = 0. Then τ (1) = (D(1) + mW )/β, Bk /β(1) = 1 and Bk Y /(mW k) = [(Ga (k) − 1)(D(1) + mW ) − (D(k + 1) − D(1))]/(mW k). Substituting into (14) gives


Ga (k) − 1 D(1) + mW − D(k + 1) − D(1) ∗ ρk (a) = 1 − (B.16) mW k (byte form). This is the one-bandwidth case of Appendix B.2. Finally, for any positive cost, Ta (0) = Ga (0)/V (0) = 1/V (0) and V (k)/V (0) = 1 + (V (k) − V (0))/V (0), so Ta (k) Ga (k) V (0) Ga (k)
= = Ta (0) V (k) 1 + V (k) − V (0) /V (0)

(B.17)

(speedup ratio). Substituting (13) for V (k) − V (0) gives (15). The substitution is by hand. 23

One draft. P Here the assumptions are those of the previous paragraph with k = 1. Then Ga (1) − 1 = a, ρ̄1 = l wl ρl (2)/W by (11), and Ededup (2) = mW (2 − ρ̄1 ) by (12). We substitute these into (14). We also substitute them, with (13) and V (0) = τ (1), into a0 = (V (1) − V (0))/V (0) of (B.4). This gives i  B  B1 h ∆D1 mW (2 − ρ̄1 ) mW i 1 h ∆D1 1 ρ∗1 (a) = 1− −∆ϕ1 − +∆ϕ1 + − a τ (1)− −1 , a0 = . mW βD β(1) τ (1) βD B1 β(1) (B.18) ∗ By (B.14) and (B.4), one draft is faster than none exactly when ρ̄1 > ρ1 (a), equivalently when a > a0 . Both equivalences are in Lean in time form. The first is MoESpec.TimeThreshold, faster time at k = 1. The second is MoESpec.FirstTokenThreshold, thresh, for any positive cost. The explicit forms (B.18) are substitutions by hand. Lean states the explicit a0 only for the byte cost (MoESpec.SpeedupCond, a0 union). Threshold movement. Here the assumptions are those of the reuse threshold in time, with k ≥ 1 fixed. Moving a changes only Ga (k) in (14), because no other quantity in it depends on a. The bandwidths and the fixed work are functions of the pass size alone. Moving D to D′ changes only τ (1), by (D′ − D)/βD , since ∆Dk = kv(k) − kv(0) does not involve D. This holds provided the bandwidths and the fixed work do not change with D. So, for a, a′ ∈ [0, 1),

Bk Ga (k) − 1 Bk τ (1) ∗ ′ ∗ ∗ ∗ ρk (a) − ρk (a ) = Ga′ (k) − Ga (k) , ρk (a) D − ρk (a) D′ = (D′ − D). mW k mW k βD (B.19) For 0 ≤ a < a′ < 1, every term of Ga (k) − 1 = a + · · · + ak is at most the matching term at a′ , and the first is smaller. So Ga′ (k) > Ga (k). With Bk , τ (1) > 0 by (B.11), ρ∗k (a) is strictly decreasing in a on [0, 1). For a > 0, Ga (k) − 1 > 0 and βD > 0, so ρ∗k (a) is strictly decreasing in D. At a = 0 it does not depend on D. These steps are by hand. Lean proves the same three statements only for the byte form (16) (MoESpec.ThresholdMonotone, rhoStar lt and U2 holds). More reuse. Here two stationary routing processes, a prime marking the second, have reuses ρl (j) ≤ ρ′l (j) for every l and j = 2, . . . , kmax + 1. They have the same L, N , m, wl , D, kv and a. Their step times are (B.10) under the assumptions of the step-time paragraph. They have the same t0 , βD , ϕ(1) and β(1) and, for every k, the same Bk and ϕdedup (k + 1). That is, the deduplicating bandwidth and fixed work at a fixed pass size do not depend on which experts repeat. Since wl > 0, the definition (11) gives the second relation below. The first holds because τ (1) involves no reuse: V ′ (0) = V (0),

ρ̄k ≤ ρ̄′k

(k = 1, . . . , kmax ),

(B.20)

so by (B.15), V ′ (k) ≤ V (k) for every k (MoESpec.TimeThreshold, tauk mono). The comparison of ρ̄k is by hand. Every input of ρ∗k (a) in (14) is shared by the two processes. Suppose that speculation gives a speedup at a under the first process. Then by (B.14), some k ∈ {1, . . . , kmax } has ρ̄k > ρ∗k (a), and by (B.20) ρ̄′k ≥ ρ̄k > ρ∗k (a), (B.21) and by (B.14) for the second process (V ′ > 0 by (B.11)), speculation gives a speedup at a under the second. Each use of (B.14) is faster time. The chaining is by hand. With V ′ (0) = V (0) and V ′ (k) ≤ V (k), (B.17) also shows that the speedup Ta (k)/Ta (0) does not fall. This step is by hand. The one-bandwidth case is in Lean for the byte cost (MoESpec.MoreReuse, cost zero, cost le and faster mono). If Bk or ϕdedup (k + 1) changes with the routing at fixed pass size, then ρ∗k (a) may change too. In that case the argument above does not apply. 24

One bandwidth (16). Here βD = Bk = β(1) = β > 0 for every k and t0 = ϕ(1) = ϕdedup (k+1) = 0. This is an admissible instance of (B.10). With D(n) = D + kv(n − 1), (B.10) becomes V (k) = Vdedup (k)/β with Vdedup (k) = D+Ededup (k+1)+kv(k). So every throughput is multiplied by β, and no comparison of throughputs changes. With D(1) = D+kv(0) and D(k+1)−D(1) = kv(k)−kv(0), (B.16) is (16), with Vdedup (0) = D +mW +kv(0) (MoESpec.TimeThreshold, byte form). At k = 1, (B.18) becomes

P aV (0) − kv(1) − kv(0) m w 1 − ρ (2) + kv(1) − kv(0) dedup l l l ρ∗1 (a) = 1 − , a0 = , (B.22) mW Vdedup (0) P using mW (1 − ρ̄1 ) = m l wl (1 − ρl (2)). In this case, Lean also proves the results of the previous paragraphs directly from the routing process, in the byte cost. These results are the threshold and the speedup at some k (MoESpec.SpeedupCond, faster reuse; MoESpec.SpeedupCondIff, faster iff reuse; MoESpec.Positivity, U1 holds, which also discharges positivity). They also include the explicit a0 (MoESpec.SpeedupCond, a0 union), the movement in a and D (MoESpec.ThresholdMonotone) and more reuse (MoESpec.MoreReuse). No single threshold. Here two stationary routing processes share every quantity that the paragraph on more reuse holds fixed, with no order between their reuses. Then ρ∗k (a) is the same for both. So by (B.14), two routing processes with equal (ρ̄1 , . . . , ρ̄kmax ) both give a speedup or neither does. This step is by hand from faster time. Lean proves it for the byte cost (MoESpec.NoSingleThreshold, U3a holds). With kmax ≥ 2, no single ρ̄k suffices. The example below uses one bandwidth with t0 = ϕ = 0. This is an admissible instance of the time model (Appendix B.2), so the example shows the same for the step time. Take N = 3, L = 1, m = 1, w1 = 1, D = 0, kv ≡ 0, kmax = 2 and a = 13/20. Then ρ∗1 = 7/20 and ρ∗2 = 371/800 by (16). Each process below repeats one pattern of experts, started at a uniformly random phase, so it is stationary. Pattern

Period

ρ̄1

ρ̄2

Speedup

01 00012 00102

2 5 5

0 2/5 1/5

1/2 2/5 2/5

yes (k = 2) yes (k = 1) no

For k = 1, pattern 01 gives a speedup with a lower ρ̄1 than 00102, which does not. For k = 2, 00012 gives a speedup with the same ρ̄2 as 00102. So no rule “ρ̄k > c” or “ρ̄k ≥ c” decides for either k. Lean: MoESpec.NoSingleThreshold, U3b holds and U3 holds, for the byte cost. The example has D = 0; we built none with D > 0.

B.3

Staircase

Monotonicity and step points. Here kmax ≥ 1, V : {0, . . . , kmax } → (0, ∞) is any cost and a ∈ [0, 1). For 0 ≤ i < k ≤ kmax , let Jk,i = {a ∈ [0, 1) : Ta (k) > Ta (i)}, and let αk,i be its infimum, taken as 1 when the set is empty. The ratio Ga (k)/Ga (i) rises with a. For i < k, Pk (x) = Pi (x) + xi+1 Pk−i−1 (x).

(B.23)

For 0 ≤ x ≤ y, termwise, since xi+1 y s = (xy)s xi+1−s ≤ (xy)s y i+1−s for s ≤ i, xi+1 Pi (y) ≤ y i+1 Pi (x), 25

(B.24)

and Pk−i−1 (x) ≤ Pk−i−1 (y). Hence, by (B.23) at x and at y, Pk (x)Pi (y) = Pi (x)Pi (y) + xi+1 Pk−i−1 (x)Pi (y) ≤ Pi (x)Pi (y) + y i+1 Pk−i−1 (y)Pi (x) = Pk (y)Pi (x), (B.25) that is, for 0 ≤ i < k ≤ kmax and 0 ≤ x ≤ y < 1, Gy (k) Gx (k) ≤ Gx (i) Gy (i)

(B.26)

(Lean: MoESpec.Staircase, ratio mono). Since Ta (k) > Ta (i) ⇐⇒

V (k) Ga (k) > , Ga (i) V (i)

(B.27)

the set Jk,i contains, with any a, every a′ ∈ [a, 1). That is, it is closed upward in [0, 1) (upset). So it contains (αk,i , 1) and misses [0, αk,i ). The best draft length rises with a. Let a ≤ a′ in [0, 1) and k ∗ (a) ≥ j, so no i < j maximizes Ta . Take i < j and a maximizer k0 of Ta . Then k0 ≥ j > i and Ta (i) < Ta (k0 ), so a ∈ Jk0 ,i . Hence a′ ∈ Jk0 ,i , that is, Ta′ (k0 ) > Ta′ (i), and i does not maximize Ta′ . So k ∗ (a′ ) ≥ j. As this holds for every j, 0 ≤ a ≤ a′ < 1 =⇒ k ∗ (a) ≤ k ∗ (a′ ) (B.28) (kstar mono). The step points are exact. For j ∈ {1, . . . , kmax } and a ∈ [0, 1), k ∗ (a) ≥ j ⇐⇒ every i ∈ {0, . . . , j − 1} has Ta (i) < Ta (k) for some k ∈ {j, . . . , kmax }.

(B.29)

“If”: each i < j has lower throughput than some k ≥ j. So no i < j is a maximizer, and the smallest maximizer is at least j. “Only if”: the maximizer k ∗ (a) ≥ j has higher throughput than each i < j, since none of them is a maximizer. In Lean this is ge iff, which states k ∗ (a) ≥ j as “no i < j maximizes Ta ”. The equivalence of that statement with k ∗ (a) ≥ j for the smallest maximizer is by hand. By (B.29), \ [ {a ∈ [0, 1) : k ∗ (a) ≥ j} = Jk,i . (B.30) 0≤i<j j≤k≤kmax

A union of upward-closed sets is upward closed, with infimum the least infimum. A finite intersection of them is upward closed, with infimum the largest. So the set (B.30) contains every a above maxi<j mink≥j αk,i and none below it. This number is its infimum, the step point: aj = max

min

0≤i<j j≤k≤kmax

αk,i ,

(B.31)

that is, k ∗ (a) ≥ j for every a ∈ (aj , 1) and k ∗ (a) < j for every a ∈ [0, aj ) (Lean: MoESpec.StepPoints, step maxmin, ge of gt maxmin and not ge of lt maxmin). For j = 1 only i = 0 occurs, so a1 = mink αk,0 (step one). So speculation gives a speedup at every a ∈ (a1 , 1) and at no a ∈ [0, a1 ). Jumps. The statement of Section 3.3 that the staircase can jump by more than one length rests on a two-expert example. Take L = 1, N = 2, m = 1, w1 = 1, D = 0, kv ≡ 0 and kmax = 2. Tokens that alternate between the two experts with a uniformly random phase then give Vdedup = (1, 2, 2). This is the step time (5) with one bandwidth β = 1 and t0 = ϕ = 0, an admissible instance of the time model. Then k ∗ (1/2) = 0 and k ∗ (9/10) = 2, so this staircase jumps from 0 to 2 (MoESpec.UnionJump, union half; MoESpec.UnionJump2, union half U and alt stair U). That k ∗ (a) ̸= 1 for every a is checked by hand. 26

Convex step time (10). j ∈ {1, . . . , kmax }, and

Here kmax ≥ 1, V : {0, . . . , kmax } → (0, ∞) is any cost, a ∈ [0, 1) and hj (a) =

aj . Ga (j − 1)

(B.32)

The first part assumes ci+1 ≥ a ci for every i ∈ {1, . . . , kmax − 1}. The second part assumes 0 ≤ c1 ≤ · · · ≤ ckmax , that is, V convex and non-decreasing. Once Ta stops rising it never rises again. Write Pi = Pi (a). Since V (i + 1) = V (i) + ci+1 and Pi+1 = Pi + ai+1 ,
Ta (i + 1) ≤ Ta (i) ⇐⇒ Pi + ai+1 V (i) ≤ Pi V (i + 1) ⇐⇒ ai+1 V (i) ≤ Pi ci+1 . (B.33) If Ta (i + 1) ≤ Ta (i), then by (B.33), a ≥ 0, Pi+1 > 0 and the hypothesis ci+2 ≥ a ci+1 ,

ai+2 V (i + 1) = a ai+1 V (i) + ai+1 ci+1 ≤ a Pi + ai+1 ci+1 = Pi+1 a ci+1 ≤ Pi+1 ci+2 ,

(B.34)

which is Ta (i + 2) ≤ Ta (i + 1) by (B.33) at i + 1 (Lean: MoESpec.Staircase, propagate). “If”: suppose Ta (j) > Ta (j − 1). Then Ta rose at every step before j, because a fall would propagate to the step from j − 1 to j. So every i < j has Ta (i) < Ta (j), and k ∗ (a) ≥ j. “Only if”: suppose Ta (j) ≤ Ta (j − 1). Then Ta (k) ≤ Ta (j − 1) for every k ≥ j, and a maximizer over 0, . . . , j − 1 is a global maximizer below j (uni iff). Negating (B.33) at i = j − 1 shows that Ta (j) > Ta (j − 1) if and only if (Pj−1 + aj )V (j − 1) > Pj−1 V (j), that is, aj V (j − 1) > Ga (j − 1) cj (step form). So under ci+1 ≥ a ci for every i ∈ {1, . . . , kmax − 1}, k ∗ (a) ≥ j ⇐⇒ Ta (j) > Ta (j − 1) ⇐⇒ aj V (j − 1) > Ga (j − 1) cj .

(B.35)

Under 0 ≤ c1 ≤ · · · ≤ ckmax , 0 ≤ ci ≤ ci+1 gives ci+1 ≥ ci ≥ a ci . So the hypothesis of (B.35) holds at every a, and (B.35) is (10). Dividing its last inequality by Ga (j − 1)V (j − 1) > 0 gives the first statement below. For 0 ≤ x < y < 1, xj Pj−1 (y) < y j Pj−1 (x) termwise, since each term satisfies xj y s ≤ y j xs for s ≤ j − 1, strictly at s = 0. So hj is strictly increasing on [0, 1) (h strict), and {a : hj (a) > cj /V (j − 1)} is upward closed with infimum aj . For j = 1, h1 (a) = a, and the infimum of {a ∈ [0, 1) : a > c1 /V (0)} is min(c1 /V (0), 1) (Lean: MoESpec.StepPoints, a1 uni). So n o  c  cj cj 1 k ∗ (a) ≥ j ⇐⇒ hj (a) > , aj = inf a ∈ [0, 1) : hj (a) > , a1 = min ,1 , V (j − 1) V (j − 1) V (0) (B.36) and a1 is a0 when a0 < 1. The theorem MoESpec.StaircaseInfimum, t5, proves (B.35) and (B.36) whole. This includes that the hypothesis of (B.36) implies that of (B.35), and the reading of aj as the infimum of an upward-closed set.

27

C

Error-bound proofs

This appendix derives the bounds of Table 2. Each paragraph starts from its assumptions, and its heading names the bound it derives. Throughout, the step time is (5). Each bandwidth in it is estimated as β̂ = β(1 + e) with |e| ≤ ε < 1, one e per bandwidth. Every other term of (5) is exact and non-negative. Each bound is proved in the module MoESpec.ErrBounds of the accompanying Lean package. The proofs use Mathlib at the pinned version and only the standard axioms. We name the theorems at the step they prove. The other Lean results of Appendix B have been compared line by line with the statements they formalize. These Lean statements have not yet been compared in this way with the statements below. The same holds for the reuse threshold in time (Eqs. (B.14) to (B.17)). Verify time (17). Here M ≥ 0 is the bytes of one memory term of (5) and β > 0 its bandwidth. The estimate moves the term by M M M e − =− . β(1 + e) β β 1+e

(C.1)

Since 1 + e ≥ 1 − ε > 0, cross-multiplying turns |e|/(1 + e) ≤ ε/(1 − ε) into |e|(1 − ε) ≤ ε(1 + e). This inequality follows from |e| ≤ ε and −|e| ≤ e. So each memory term moves by at most ε/(1 − ε) of itself (time err). By the triangle inequality, the memory part Mp (n) = D(n)/βD + Ep (n)/βp (n) also moves by at most ε/(1 − ε) of itself (time err2). The other terms of τp (n) are exact, so τ̂p − τp = M̂p − Mp . They are also non-negative, so Mp (n) ≤ τp (n). Together these give (17). Saving (18). Here both kernels read the same bytes D(n) once per pass at the same dense bandwidth βD , with the same estimate. By the previous paragraph, τ̂p − τp = M̂p − Mp for each kernel. The dense term is the same for both kernels, so Ŝ(n) − S(n) =

E

slot (n)

β̂slot (n)

−

Eslot (n)   Ededup (n) Ededup (n)  − − . βslot (n) βdedup (n) β̂dedup (n)

(C.2)

Each bracket is one memory term of (C.1). So the triangle inequality and the bound of the previous paragraph on each bracket give (18) (saving err). Faster kernel (19). Here x, y ≥ 0 are the true expert times Mpe (n) of the two kernels and ϕx , ϕy ≥ 0 their fixed work. The estimates multiply x and y by factors u, v = 1/(1 + e). Each factor lies in [ℓ− , ℓ+ ] = [1/(1 + ε), 1/(1 − ε)], with ℓ− ≤ 1 ≤ ℓ+ . If xℓ+ + ϕx < yℓ− + ϕy ,

(C.3)

then xu + ϕx ≤ xℓ+ + ϕx < yℓ− + ϕy ≤ yv + ϕy . Also x + ϕx ≤ xℓ+ + ϕx < yℓ− + ϕy ≤ y + ϕy . So the predicted and the true times order the kernels alike (decision agree f). The theorem decision agree proves the case ϕx = ϕy = 0 separately, for x, y > 0. Now take fl = 1 and equal fixed work, so that ϕx = ϕy . This fixed work cancels from (C.3). Take x = Ededup (n)/βdedup (n) and y = Eslot (n)/βslot (n), so that x/y = ū(n)/r(n). The margin (C.3) is then ū/r < ℓ− /ℓ+ = 1+ε . The same argument with the kernels exchanged gives (1 − ε)/(1 + ε), that is, ln r − ln ū > ln 1−ε the other sign. The two cases together are (19).

28

Best draft length (20). Here a ∈ [0, 1), the true step time is V (k) > 0 and the predicted one is V̂ (k). They satisfy (1 − δ)V (k) ≤ V̂ (k) ≤ (1 + δ)V (k) for every k, with 0 ≤ δ < 1. Over {0, . . . , kmax }, the draft length k ∗ maximizes Ta = Ga /V and k̂ ∗ maximizes T̂a = Ga /V̂ . Dividing Ga (k) by the bounds on V̂ (k) gives Ta (k) Ta (k) ≤ T̂a (k) ≤ 1+δ 1−δ

for every k

(C.4)

(T bounds). Since k̂ ∗ maximizes T̂a , 1−δ Ta (k ∗ ) ≤ (1 − δ) T̂a (k ∗ ) ≤ (1 − δ) T̂a (k̂ ∗ ) ≤ Ta (k̂ ∗ ) 1+δ

(C.5)

(regret). So the throughput lost, 1 − Ta (k̂ ∗ )/Ta (k ∗ ), is at most 1 − (1 − δ)/(1 + δ) = 2δ/(1 + δ). 1+δ ∗ ∗ If k̂ ∗ ̸= k ∗ , (C.5) gives Ta (k ∗ ) ≤ 1+δ 1−δ Ta (k̂ ). The separation condition of (20) is 1−δ Ta (i) < Ta (k ) for every i ̸= k ∗ . For i = k̂ ∗ , this condition forbids the previous inequality. So under the separation condition, k̂ ∗ = k ∗ (argmax agree, whose hypothesis is this condition). A common factor c > 0 on every step time leaves the maximizer unchanged. The reason is that Ga (i)/(cV (i)) ≤ Ga (j)/(cV (j)) exactly when Ga (i)/V (i) ≤ Ga (j)/V (j) (common factor). Best speedup (21). Here the assumptions are those of the previous paragraph. By (C.4), T̂a (k̂ ∗ ) ≥ T̂a (k ∗ ) ≥ Ta (k ∗ )/(1 + δ). Also T̂a (k̂ ∗ ) ≤ Ta (k̂ ∗ )/(1 − δ) ≤ Ta (k ∗ )/(1 − δ), so Ta (k ∗ ) Ta (k ∗ ) ≤ T̂a (k̂ ∗ ) ≤ , 1+δ 1−δ

Ta (0) Ta (0) ≤ T̂a (0) ≤ . 1+δ 1−δ

(C.6)

Dividing the first pair by the second, in the direction that keeps each inequality, gives (21) (speedup bounds). From ε to δ. Here the bandwidths are the only errors of the predicted step time, and ε < 1/2. By (17), the verify part of V̂ (k) is off by at most ε/(1 − ε) of itself. An exact draft and host time h(k) only lowers the relative error of the sum. So δ = ε/(1 − ε) satisfies the assumption of (20). Then 1 + δ = 1/(1 − ε) and 1 − δ = (1 − 2ε)/(1 − ε), so 1+δ 1 = , 1−δ 1 − 2ε which for ε = 0.1 is the factor 1.25 of Section 3.5. This step is not in Lean.

29

(C.7)

D

Engines and runs

D.1

Kernel dispatch

We follow each expert’s matrix product, for each model and pass size n, to the GPU kernel that runs it, at pinned releases. We then count the requested bytes. The kernel fixes which bandwidth βp (n) of (5) a pass of n tokens runs at. On llama.cpp Metal, an expert product goes to a tiled matrix-matrix kernel only at 32 rows or more. Below that, a matrix-vector kernel multiplies each token’s selected experts separately. MLX-LM switches to a tiled kernel only on passes of at least 4N/m tokens. The llama.cpp CUDA tiled kernel loads each selected expert once per tile on passes of 9 to 128 tokens. For Metal and MLX-LM, the step from each request pattern to its byte count is proved in Lean (MoESpec.MetalClass and MoESpec.MlxClass). The reading of the kernels from source is by hand. Table 5 gives the result.

D.2

Models and prompts

On llama.cpp, the models run as OLMoE-1B-7B-0125 in Q8 0, Qwen3-30B-A3B in Q4 K M and gpt-oss-20b in MXFP4 (Gerganov and llama.cpp contributors, 2026). On MLX-LM, OLMoE-1B7B-0125 runs in 4-bit quantization. The oracle draft of Section 4 marks the positions it corrupts by a seeded draw, and it costs essentially nothing. On the Apple engines, the realized acceptance is within 0.04 of a. On CUDA, where it was not logged per run, steps emit 2 to 5% fewer tokens than Ga (k) (Appendix E). Each draft length k = 0, . . . , 8 is run with each acceptance a in an 11-value grid from 0.2 to 0.98. Each run generates 256 tokens on the six validation-set prompts of Section 4, at context depth d = 512 and 4096. Each combination is run three times on the Apple engines and four times on CUDA. A run was discarded and redone if the machine was throttled, swapping, off AC power or running another inference process. Session load was not a discard reason. Other work on the M4 Max overlapped some whole-model runs. The run logs list these runs and are available on request. The DeepSeek-V2-Lite runs have no discard rule (Appendix E). Reuse ρ̂l (n) is computed over every window of n consecutive tokens in a fit document. For each window, it counts the last token’s experts that an earlier token of the window also selected. The sum over windows is divided by m times the number of windows. Its intervals come from 2000 bootstrap resamples of fit documents.

D.3

Real draft model

Qwen3-30B-A3B (Q4 K M) runs on llama.cpp Metal, drafted by Qwen3-0.6B (Q8 0), with both models greedy. This engine runs the per-slot kernel on every pass of speculation (Table 5). So each draft adds the expert bytes mW of one more token. The runs use the 80 first-turn MT-Bench prompts (Zheng et al., 2023) and generate 256 tokens at k = 0, . . . , 8. The run at k = 0 uses the same program with a drafter that never drafts. The prompts were split into 42 for fitting an earlier time model and 38 held out from it. That earlier model is linear in the per-slot bytes, and this paper does not use it. We report the held-out prompts, of which 37 have a run at every draft length. Plain decoding is fastest on 33 of the 37. The median speedup over plain decoding is 0.877 at k = 1 and 0.517 at k = 8.

30

Table 5: Kernel that runs the expert products, read from source. n is the number of tokens verified in one pass, and the tests verify n ≤ 9 tokens per step. “Neither”: the requested bytes equal neither the union nor the per-slot count on every trace. Bytes are requested bytes. The source does not determine how many requests on-chip caches serve. The CUDA row at 2 ≤ n ≤ 8 is the source reading of the matrix-vector kernel, which requests each slot’s expert once. Profiled L2 requests of this kernel fall below nmb when repeated experts sit in the same slot of every token. They fall as far as 0.14 of nmb, at n = 8 and U = 8. Two causes would both produce this: L1 hits under per-slot requests, and a pattern that requests each expert once per slot position. The hardware counters do not decide between them, so the row rests on the source. For gpt-oss, the factor 16/15 counts the loads of the tiled kernel in the source reading. The factor fl = 500/459 of Table 6 instead counts the padded rows of each weight tensor, the byte unit of Appendix E. engine

tested sizes

other sizes

where the source decides

llama.cpp Metal b11617

per-slot

MLX-LM 0.32.0

per-slot

per-slot for n ≤ 31; tiled (32 tokens per tile) for 32 ≤ n ≤ 512, deduplicating only at n = 32 for two models per-slot for n < 4N/m (32 or 64); tiled (16 per tile), neither, above

llama.cpp CUDA b11617

upstream; run in Sec. 4

dispatch test “batch ≥ 32 rows” in ggml-metal-common.cpp; matrixvector kernel in mul mv.metal, one grid slice per token-slot pair dispatch branch on rows per expert in mlx/backend/metal/quantized.cpp; MoE call in mlx lm/models/switch layers.py batch limit 8 for the matrix-vector kernel in ggml-cuda/mmvq.cuh; dispatch in ggml cuda mul mat id (ggml-cuda.cu); tiles in ggml-cuda/mmq.cu

both at n = 1; per-slot for 2 ≤ n ≤ 8; deduplicating for 9 ≤ n ≤ 128 (OLMoE, Qwen3), 16/15 times union (gpt-oss)

31

E

Deduplication evidence

Engine changes. The kernel each setting runs, its byte count and its factor fl are read from source at pinned commits. These are commits of the modified llama.cpp Metal and MLX-LM and of the unmodified CUDA build. On llama.cpp Metal, the minimum batch of the deduplicating kernel drops from 32 to 2. MLX-LM sorts rows by expert. On llama.cpp CUDA, an environment variable lowers the per-slot batch cap from 8 to 1. We checked each modified engine for equal output with deduplication off and on. On Metal, both settings give the same greedy tokens on OLMoE and gpt-oss, and Qwen3 diverges on one of three prompts at position 12. On CUDA, both settings give identical greedy tokens on both models and three prompts. There the verify argmax agrees on 132 of 132 rows (OLMoE) and 128 of 132 (gpt-oss, four near ties). On CUDA, the bytes read once per pass are the same with deduplication off and on, D(n) = 403062784 + 532480 n on OLMoE. Routing. We measured reuse by teacher forcing on each modified engine’s own greedy sequence. On OLMoE and Qwen3, it differs from the WikiText-103 values by −0.018 to +0.004 in ρ̄. For gpt-oss, the RTX 4070 Ti SUPER routes the same expert set as the M4 Max on 0.947 of 11845 (row, layer) pairs. Its layer-mean ρ̂(n) is within 0.00083 at n = 2, . . . , 9. Neither measurement covers the greedy continuations at d = 4096 (Section 4.1). Deduplicating fit. Here, as for (4), U is the number of distinct experts that a pass with chosen ids names. As there, b is the bytes of one expert of the timed layer. We fit the median operation time of the deduplicating kernel per layer, by least squares over the passes with chosen ids at n = 2, . . . , 9, as tdedup = top + cg U b + ct n, (E.1) with its own fixed time top , cg the time per union byte and ct the time per token. The benchmark of Section 2 times one layer’s expert operation through each engine’s library (llama.cpp’s ggml with the Metal or CUDA backend; MLX). It runs on the same model files, after the same 256 MiB cache flush. Its output was checked against a CPU reference on DeepSeek-V2-Lite (layers 1, 13 and 26, n = 1, 4 and 9). The relative L2 error is 1.0 × 10−2 to 2.3 × 10−2 on CUDA, where the GPU kernels quantize activations differently from the CPU. It is 9.5 × 10−3 to 1.08 × 10−2 on Metal. The chosen ids, the windows of recorded routing and every timing file are available on request. The prediction and the measured-time version read them through one script. This script also makes Figures 2 to 6 and 7 and Tables 1, 3, 4 and 7. It also computes the bandwidth error ε of Section 4 from the fits on chosen ids. Speculative runs. The measured speculative runs on CUDA differ from Ga (k)/(τdedup (k + 1) + h(k)) with the measured τdedup and h. Per configuration, the median factor between them is 0.93 to 1.04, over a range of 0.74 to 1.18. Two terms make up this difference. First, steps emit 2 to 5% fewer tokens than Ga (k). On steps that run to their full length, the shortfall is 1.6 to 2.7%. This part comes from the sampling of the seeded flags that mark drafts wrong. The rest comes from steps cut short where the argmax of the batched verify pass differs from the batch-one greedy token that the oracle draft proposes. This happens on up to 25.8% of steps, at k = 8, a = 0.98 on gpt-oss at d = 4096. That term belongs to the oracle draft, and a real draft model checked against the batched pass has none. Second, the time per step beyond the verify pass is 0.58 to 0.61 of the measured h(k). The verify time departs from its fit by −7 to +9%. Against the speedups of the measured speculative runs, the speedup bound (21) therefore holds at 9 of 11 acceptance values on 32

Table 6: The deduplicating engines. The kernel each setting runs is read from source at n = 1, . . . , 9 verified tokens. At n = 1, an engine runs the same kernel with deduplication off and on. fl : bytes the deduplicating kernel reads per distinct expert, over the expert’s bytes. On Qwen3 Q4 K M, the tiled kernel reads each K-quant expert more than once. MMVQ and MMQ: llama.cpp CUDA’s quantized matrix-vector and matrix-matrix kernels. engine

switch

deduplication off

deduplication on

fl , deduplicating

llama.cpp Metal, modified

deduplicating kernel from 2 rows (32 upstream)

mul mv id, per-slot, n≤9

mul mm id, deduplicating, 2 ≤ n ≤ 9

MLX-LM, modified

sort rows by expert, gather qmm rhs per-slot batch cap 8 to 1

gather qmv, slot, n ≤ 9

per-

sorted rows, deduplicating, 2 ≤ n ≤ 9

1 (OLMoE Q8 0, gpt-oss MXFP4); 2.14 or 2.33 (Qwen3 Q4 K M, K-quant reload) 10/9 (16-row tile)

MMVQ, per-slot, n ≤ 8; MMQ at n=9

MMQ, deduplicating, 2 ≤ n ≤ 9

llama.cpp CUDA, modified

1 (OLMoE); 500/459 (gpt-oss, padded unit)

e + M̂ e Table 7: Per configuration, three quantities. Amplification: the factor (M̂slot dedup )/|S(n)| by which the saving bound (18) turns a relative bandwidth error into a relative saving error. It is given as its range over n = 2, . . . , 9 (n ≤ 8 on CUDA; |S(n)| floored at 0.05 ms), and it is largest where S(n) crosses 0. Implied ε: the bandwidth error that the prediction’s largest saving error requires under (18), against the engine’s ε. Verify-time error: the largest relative error of the verify time with deduplication on over n = 2, . . . , 8, for the prediction, the measured-time version and the calibrated version. Metal and CUDA are llama.cpp.

ε

verify-time error

engine

model

d

amplification implied engine prediction measured-time calibrated

Metal Metal Metal Metal Metal Metal MLX-LM MLX-LM CUDA CUDA CUDA CUDA

OLMoE OLMoE Qwen3 Qwen3 gpt-oss gpt-oss OLMoE OLMoE OLMoE OLMoE gpt-oss gpt-oss

512 4096 512 4096 512 4096 512 4096 512 4096 512 4096

6.0 to 79 6.2 to 86 3.9 to 7.8 4.2 to 19 3.3 to 343 3.8 to 87 3.2 to 3.8 3.3 to 3.8 15.4 to 139 15.2 to 146 7.5 to 223 6.0 to 43

0.014 0.028 0.020 0.047 0.033 0.126 0.108 0.107 0.030 0.032 0.022 0.033

0.031 0.031 0.031 0.031 0.031 0.031 0.148 0.148 0.054 0.054 0.054 0.054

19% 24% 21% 25% 20% 6% 38% 41% 3% 5% 9% 21%

20% 24% 22% 31% 15% 18% 30% 31% 9% 5% 16% 4%

6% 6% 6% 4% 9% 8% 26% 17% 3% 2% 6% 1%

CUDA gpt-oss at d = 4096. It holds at all values elsewhere. The two misses come from these two terms. Scoring rules. Table 8 lists the six scoring rules written before the whole-model runs of Section 4.4, with the value each measurement gives. The scoring script decides each rule at each depth. 33

Table 8: Scoring rules for DeepSeek-V2-Lite and the measured values, measured-time version. The speedup rule uses δ, the largest relative error of the predicted step time. The crossover rule is scored over n = 2, . . . , 9, and a measured crossover at n = 9 counts as within one of none. On CUDA, the deduplicating kernel runs at n = 9 with deduplication off and on. So Table 4, like Table 3, reads the CUDA crossover over n ≤ 8. rule

CUDA 512

CUDA 4096

Metal 512

Metal 4096

crossover within one k̂ ∗ within one at ≥ 9 of 11 values loss at most 6% at every value sign of S(n) where the interval excludes 0 verify time within 15%, both kernels best speedup inside (21)

−/−, holds 11, holds 2.9%, holds 7 of 7, holds

−/9, holds 11, holds 4.1%, holds 7 of 7, holds

−/−, holds 9, holds 20.1%, fails 8 of 8, holds

−/−, holds 6, fails 22.5%, fails 8 of 8, holds

9.9%, holds δ = 0.072, holds

10.1%, holds 0.077, holds

34.0%, fails 0.27, holds

28.4%, fails 0.25, holds

The measured-time version meets the rules only if all six hold at both depths. The same script, byte for byte, scored both engines. On Metal, the whole-model runs at d = 512 show a slow GPU state in some runs. In each of the four whole-model runs at k = 0, the first prompt runs at 6.7 to 7.1 ms per step and the others at 7.5 to 9.4 ms. With a 90 s rest after every 10 minutes of runs, the state persists. The system reports no thermal warning at any of the 37 rests. At d = 4096, the ratio of the slowest to the fastest prompt’s median step time has a median of 1.03. It exceeds 1.08 in 2 of 94 runs at acceptance 1. The rules exclude no runs, so these runs are scored. Transfer. Characterizing the bandwidth once per engine, instead of measuring it per model, fails (Figure 7). We hold out one model at a time on CUDA (three models) and Metal (four). We fit log βp (n) on the other models as linear in log n and the log of the expert bytes, a form fixed before any held-out prediction. This fit misses the held-out bandwidth by 9 to 206% (median 23% on CUDA, 47% on Metal). It misses the ratio r by 2 to 30% on OLMoE and DeepSeek-V2-Lite and by 33 to 86% on gpt-oss and Qwen3. The expert sizes of gpt-oss and Qwen3 lie outside the range of the other models. Near the crossover, this error exceeds the margin of (19). The transferred bandwidth places the crossover within one in 1 of 6 CUDA configurations and 2 of 8 Metal configurations. With each model’s own measurement, the counts are 6 of 6 and 7 of 8. The quantization format (Q8 0, Q4 K M, MXFP4) and the model are confounded, and MLX-LM has one model.

34

Qwen3

gpt-oss

DeepSeek-V2-Lite

llama.cpp CUDA

103

102

llama.cpp Metal

per-slot

deduplicating

ratio r(n)

bandwidth βp(n), n = 2, 4, 6, 8

1.0

predicted r(n)

predicted from other models (GB/s)

OLMoE

0.8 0.6 0.4 0.2

102

0.2

103

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

measured r(n)

measured on the model (GB/s)

Figure 7: Bandwidth of a held-out model, predicted from the other models of the same engine. It is plotted against the bandwidth measured on the expert operation of the held-out model, at n = 2, 4, 6, 8. Left: each kernel’s bandwidth (filled: per-slot; open: deduplicating), log scale. Right: the ratio r(n). Color: the held-out model; shape: engine (circle: llama.cpp CUDA, triangle: llama.cpp Metal). Dotted: equal.

35

Related documents

• Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length#1457594
• The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts#1036839
• Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding#1412323
• Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding#1394261
• SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference#1326667