Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

C HANNEL S ELECTION FOR M IXTURE - OF -E XPERTS D ECODING N EEDS N O C ALIBRATION DATA Geunhyoung Kim1 Sung Ju Hwang1,2 1 KAIST 2 DeepAuto.ai {skybluejacket, sungju.hwang}@kaist.ac.kr

Mixture-of-Experts (MoE) models are bottlenecked by loading routed-expert weights from memory during interactive decoding at small batch sizes. Because most intermediate channels in a routed expert contribute little to any single token, channel selectors predict which channels matter and read only their weights, typically keeping channels above a threshold calibrated on offline data. We show that channel selection needs no calibration data. First, a calibrated threshold does not fix how much a selector reads, because the number of channels passing it varies across inputs; Prox with thresholds calibrated for an 18% read budget actually reads only 12.9% on average across our benchmarks. Second, top-k keeps each token’s k highest-scoring channels and needs no calibration data. At the calibrated budget, a selector’s top-k version matches or beats its threshold version on the same score in 40 of 43 comparisons at 13% and 18% budgets, including all 23 on generation tasks. The top-k versions also avoid the repetition-loop collapse that calibrated thresholds cause on Gemma 4. Calibrating a threshold thus provides neither of the two things selectors use it for: a known read budget and a consistent accuracy gain at that budget. We furthermore introduce a reader built entirely from the model’s weights, which screens channels with a ternary sketch and recovers the skipped ones with a low-rank map. Across three post-trained MoE models, our reader retains 98.1% of full-model accuracy at 17.8% of routed-expert reads, with Triton kernels running MoE blocks 1.60 to 1.90× faster than optimized vLLM at batch one and up to 2.69× at larger batches on the same three models. (a) Accuracy at heavy sparsity

(b) Speed of our reader

Our reader Prox score, top-k

WINA score, calibrated WINA as published, top-k

Prox as published, calibrated full model

Qwen3-Thinking Gemma 4 Qwen3.5

TEAL as published, calibrated 2.0×

Whole-model speedup

100

Worst-pair accuracy (%)

arXiv:2610.37386v1 [cs.LG] 27 Oct 2026

A BSTRACT

75

50

25

1.5×

1.0×

full model

0 10

15

20

25

30

35

40

Reads or calibrated target (%)

1

4

16

64

Decode batch size

Figure 1: At heavy sparsity, calibrated selectors lose most of their accuracy on at least one task, whereas data-free top-k selectors stay accurate and our reader performs best and decodes faster. (a) Worst accuracy over three MoE models and three benchmarks; dashed lines use calibrated thresholds, placed at the budget they were calibrated for. See Appendix Figure 11 for the same curves with calibrated selectors placed at the reads they actually used. (b) Whole-model decoding speedup of our reader at 17.8% of reads over optimized vLLM. 1

1

I NTRODUCTION

Mixture-of-Experts (MoE) models scale capacity by routing each token to only a few experts (Shazeer et al., 2017; Fedus et al., 2022). At the small batch sizes of interactive decoding, latency is dominated by reading routed-expert weights from GPU memory (DRAM) to on-chip SRAM. Within each routed expert’s gated MLP, however, most intermediate channels contribute little to any single token and can be skipped with minimal loss (Park et al., 2026). A channel selector exploits this by predicting which channels matter and reading only their weights. Every channel selector couples two components: a channel score that estimates each channel’s contribution to the output, and a keep rule that decides which channels to compute. To balance quality and sparsity without retraining, most selectors use a calibrated threshold, setting a fixed per-layer cutoff on offline calibration data so that the model reads a target budget on that data (Lee et al., 2024; Liu et al., 2025a; Zhang et al., 2025; Liu et al., 2026). The alternative keep rule is top-k, which keeps each token’s k highest-scoring channels and needs no calibration data itself (Liu et al., 2025b; Chen et al., 2026b). Prior studies disagree on which rule is preferable. Prox (Liu et al., 2026) argues that a threshold’s variable channel count adapts computation to each token, and both Prox and the Sparsing Law (Luo et al., 2025) find their threshold versions ahead of top-k on dense models. Conversely, other studies find top-k ahead (Liu et al., 2025b), while calibration introduces sensitivity to the choice of calibration data (Williams & Aletras, 2024; Ji et al., 2025). We show that channel selection for frozen MoE decoding needs no calibration data. Selectors calibrate a threshold for two reasons, and it delivers neither. First, a calibrated threshold does not fix how much a selector reads (Section 4.1). The number of channels above a fixed cutoff changes with the input, so selectors with calibrated thresholds read less than the budget they were calibrated for in all 72 benchmark runs we made, and calibrating on the models’ own reasoning does not fix this. On Qwen3-Thinking, the count follows the answer written so far more than how much each token needs. Second, calibration gives no consistent accuracy gain over top-k (Section 4.2). We run each selector twice on its own score, once with its calibrated threshold and once with top-k, at the budget the threshold was calibrated for. The top-k version matches or beats the threshold version in 40 of 43 comparisons across four MoE models at 13% and 18% budgets. Under heavy sparsity, Gemma 4 with calibrated thresholds falls into repetition loops during long reasoning, which the same selectors with top-k avoid (Figure 1). With top-k as the keep rule, performance depends on how well a selector ranks channels before reading them and handles skipped channels. Our reader is built entirely from the model’s weights (Section 5). For each token, it screens all candidate channels with a rotated ternary sketch of the weights (about 7% of the BF16 checkpoint), computes the top-k exactly, and recovers the skipped channels through a low-rank map of the down projections. We evaluate our reader in vLLM (Kwon et al., 2023) on Qwen3-30B-A3B-Thinking-2507 (Yang et al., 2025), Gemma-4-26B-A4B-it (Gemma Team, 2026), and Qwen3.5-35B-A3B (Qwen Team, 2026), charging every weight and parameter read in 32-byte hardware sectors and checking counts with a GPU profiler (Section 2). At both 12.8% and 17.8% of routed-expert reads, it is the most accurate tested selector, on average and in its worst model-benchmark pair, which keeps 95.0% of full-model accuracy at 17.8%. On OLMoE (Muennighoff et al., 2025), it matches the full model more closely than nine prior selectors at the budgets at which each was tested (Appendix I). Our Triton kernels make whole models decode up to 1.88× faster than optimized vLLM, with the gain peaking at batch size eight or sixteen (Section 6.2).

2

BACKGROUND

In an MoE layer, a router sends each token to a few experts, each a gated MLP (Shazeer, 2020). We call each intermediate dimension of a routed expert a channel: channel j of expert e owns row j of the gate and up projections Ge and Ve and column j of the down projection De (Figure 2a). The routed output sums all channel outputs, each weighted by its expert’s router weight. In Qwen3-Thinking (Yang et al., 2025), a token goes to 8 experts of 768 channels, so a selector chooses among 6,144 channels per layer. Under a calibrated threshold, the token’s scores decide how many channels pass the cutoff; under top-k, the deployer fixes this number in advance (Figure 2b).

2

(b) Keep rules

(a) Channel Ge , Ve

x

y

De

a

Threshold τ fitted on calibration data

I

Top-k

H

k=4 chosen by the deployer

channel j: rows gj , vj and column dj

y=

P

j aj dj ,

aj = ϕ(gj⊤ x) (vj⊤ x)

(c) Reads 32 bytes

needs 80 bytes, reads 128 (4 sectors)

token A

token B

τ keeps 6

keeps 2

keeps 4

keeps 4

bytes the selector needs rest of a touched sector, also read untouched sector, not read

Figure 2: A channel selector computes only a subset of each routed expert’s channels, and hardware charges every 32-byte sector it touches. (a) Channel j owns row j of the gate and up projections (Ge , Ve ) and column j of the down projection (De ); H is the hidden width, I the number of channels per expert, ϕ the gate activation. (b) On the same scores, a calibrated threshold τ keeps six channels on token A and two on token B, while top-k keeps four on both. (c) A selector that needs 80 bytes spread over four sectors is charged 128 bytes. Trained predictors x

predictor

Ge , Ve

Full-projection selectors x

Ge

a

Input-sparsity selectors x

Ve , D e

Low-bit-copy selectors

a

x copy score kept rows

Ge runs a small network first

reads a full projection first

skips small-input columns

scores a low-bit copy first

Deja Vu PowerInfer

CATS CHESS

TEAL R-Sparse WINA

HiRE FloE Prox

ShadowLLM

M-COUNTDOWN

La RoSA TopK-Guided

Our reader

Park et al. MoC

ActTail

read to score

read after choosing

calibrated threshold

top-k

other parts fitted on data

Figure 3: Prior channel selectors, grouped by what they read before choosing channels. Dashed rows or columns are skipped; chips show each keep rule, and a dot marks budgets or rotations fitted on data. HiRE and Mixture-of-Channels (MoC) train the model for top-k; TopK-Guided bounds a threshold by a top-k window. Our reader (bold) fits nothing to the model. We measure cost in reads: the bytes loaded per token as a percentage of the full model’s routed-expert reads. Reads include kept-channel weights, sketch or proxy weights, and auxiliary parameters. Modern NVIDIA GPUs fetch memory in 32-byte sectors (NVIDIA, 2025), so we charge every touched sector in full (Figure 2c). A GPU profiler confirms the count: at batch one on Qwen3Thinking, it measures cold-cache DRAM reads of 18.3% against our counted 17.8% (Appendix C). A calibrated threshold is not a selector by itself but a keep rule that selectors such as Prox and TEAL use. To compare keep rules, we run each such selector twice on the same score: its threshold version keeps the calibrated threshold, and its top-k version replaces the threshold with top-k. We compare the two at the budget the threshold was calibrated for, because a deployer sets this budget in advance, while the reads the threshold version actually uses, its realized reads, are known only afterward (Section 4.1). As a check, we also run the top-k version at these realized reads. Threshold versions read less than their budget, so this check favors them (Section 4.2).

3

R ELATED WORK

Four families of channel selectors. Prior selectors differ in what they read before choosing channels (Figure 3). Trained predictors use an auxiliary network (Liu et al., 2023b; Song et al., 3

2024; Akhauri et al., 2024), and SparseInfer uses sign bits without training (Shin et al., 2025). Full-projection selectors compute a whole gate or up projection first, so they read at least a third of each routed expert (Lee et al., 2024; He et al., 2024; Park et al., 2026), as does M-COUNTDOWN (Cheon & Kang, 2025). Input-sparsity selectors skip the weight columns that multiply small inputs (Liu et al., 2025a; Zhang et al., 2025; Liu et al., 2025b; Chen et al., 2026b; Hou et al., 2026; Chen et al., 2026a). Low-bit-copy selectors, including our reader, score channels with a compressed copy of the weights and then load the kept rows (Samaga B L et al., 2024; Zhou et al., 2025; Liu et al., 2026); FloE computes its whole up projection from an INT2 copy. Calibration and prior keep-rule comparisons. Most selectors for frozen models calibrate on sample data, while HiRE and Mixture-of-Channels (Wu et al., 2025) train the model for top-k. Six earlier studies compare the two keep rules, and they disagree (Appendix Table 2). La RoSA (Liu et al., 2025b), Sparser-MoE (Anonymous, 2026d), and batch-wise pruning (Kim et al., 2026c) favor top-k. The Sparsing Law (Luo et al., 2025) and Prox (Liu et al., 2026) favor thresholds, Prox by 1.8 points on dense Qwen3 models. TopK-Guided (Agarwalla & Lin, 2026) finds a hybrid best. However, five of the six studies test dense models, mostly on short tasks, and two change the score along with the keep rule. Several works note that reads under a calibrated threshold drift at test time (Liu et al., 2025b; Hou et al., 2026; Anonymous, 2026c), yet none of the six runs top-k at the reads a threshold actually used. Prox’s result does not carry over to OLMoE: on its likelihood tasks at 13% and 18% budgets, its two versions tie in 7 of 10 comparisons, and top-k leads in 2 (Appendix F.1). Scope. Our conclusions concern calibrated thresholds on the channels of frozen MoE experts. Taskdependent sparsity also motivates data-free selection (Szatkowski et al., 2026), and SparseDecoding finds that calibration and decoding activations differ (Wang et al., 2026). Thresholds can work elsewhere, as in sparse attention (Anonymous, 2026a) or trained MoE routers (Anonymous, 2026b). Appendix A covers recovery, batching, and other levels of sparsity.

4

C HANNEL SELECTION NEEDS NO CALIBRATION DATA

Selectors calibrate a threshold for two reasons: to read a known budget, and to gain accuracy at that budget by giving each token the channels it needs. 4.1

A CALIBRATED THRESHOLD DOES NOT FIX HOW MUCH A SELECTOR READS

Channel scores change across tokens, tasks, and answer length, so a different number of channels passes a fixed cutoff τ each time, while top-k always keeps k. Reads vary across tokens and fall during generation. Figures 4a–b isolate the keep rule from the errors of an approximate score: channels are scored by their exact contribution to the output, and τ is calibrated on UltraChat to keep about 12% (12.3% on Qwen3-Thinking). On held-out chat, the mean kept count matches the target, consistent with Prox’s small drift on five short tasks (Liu et al., 2026). Yet even on chat, the middle 90% of Qwen3-Thinking tokens keep 9.0 to 16.2% of channels. On benchmark answers, scores are smaller and the median falls to 10.0% (Figure 4a, orange). Counts also fall as the answer grows (Figure 4b): around 6,000 tokens, Qwen3-Thinking keeps 71% of its calibrated count, Gemma 4 92%, Qwen3.5 88%. Calibrated budgets are missed at deployment. Selectors with calibrated thresholds read less than their budget in all 72 benchmark runs we made: Prox and our reader, each at targets of about 13%, 18%, 28%, and 38% (Figure 4c; Appendix D). Prox with its threshold calibrated for 18% reads 12.9% on average and as little as 9.9%. About 1.7 percentage points of this gap are built into Prox’s published rule for splitting the budget between inputs and channels, once reads are counted in sectors (Appendix B). Our reader with calibrated thresholds hits its target on calibration data but reads 0.4 to 5.8 percentage points less on benchmarks. Calibrating on the models’ own reasoning does not stop this drift (Appendix D.1). Kept channels follow the earlier answer, not token need. Why do counts fall? We take a 256token window from position 6,000 of a Qwen3-Thinking GPQA-Diamond answer and place it directly 4

(a) Channels kept per token

threshold, held-out chat

Tokens (%)

Qwen3-Thinking

top-k, every token

threshold, benchmark answers

Gemma 4

Qwen3.5

10 5 0 5

10

15

20

5

Channels kept per token (%)

10

15

20

5

Channels kept per token (%)

100 90 80 70 Qwen3-Thinking

15

20

(c) Whole benchmark runs 40

Reads used (%)

Kept count (% of calibrated)

(b) Median along the answer

60

10

Channels kept per token (%)

Gemma 4

Qwen3.5

Prox as published, calibrated Our reader, calibrated top-k

30 20 10

top-k 0

0

2000

4000

6000

8000

Tokens into the answer

0

20

40

Read budget (%)

Figure 4: Under a calibrated threshold, the number of kept channels changes from token to token, and selectors read less than their budget on benchmarks. (a) Share of channels each token keeps under an exact-score threshold calibrated on UltraChat for about 12%, on held-out chat and on the model’s answers to GPQA-Diamond (Rein et al., 2024), IFEval (Zhou et al., 2023), and MMLU-Pro (Wang et al., 2024); top-k keeps the same share on every token (line). (b) Median kept count per 500-token window, relative to the calibrated count (about 94,000 answer tokens per model). (c) In all 72 benchmark runs, Prox and our reader with calibrated thresholds read below their budget (the diagonal). after the question. Under a Prox threshold calibrated for 13%, the same tokens then keep 57% more channels, close to the count at the start of the answer. Removing earlier context raises the count even for the same tokens. This fall barely helps: keeping each layer’s count the same early and late in the answer, at equal total reads, changes KL divergence by at most 1.0 ± 0.5% on all three models. Nor do harder tokens get more channels: kept counts correlate with the full model’s next-token loss at −0.29 to +0.14 (Spearman). Does the remaining variation from token to token help? Giving every token in a layer the same count, at equal reads, lowers KL divergence on reasoning by 3.3 ± 1.0% on Qwen3-Thinking and 12.5 ± 1.5% on Gemma 4, so there the variation hurts; on Qwen3.5 it raises KL by 10.5 ± 1.1%, so there it helps (Appendix E). 4.2

C ALIBRATION GIVES NO CONSISTENT ACCURACY GAIN OVER TOP -k

We compare the threshold and top-k versions of existing selectors (Section 2) at the budget each threshold was calibrated for. Prox also selects inputs first; Appendix F gives the input rules. Top-k matches or beats thresholds across models and scores. We make 43 paired comparisons at 13% and 18% budgets, on generation tasks on four MoE models and likelihood tasks on OLMoE, with three channel scores: Prox, TEAL, and Prox scaled to output units (Appendix F). The top-k version ties the threshold version in 30 (within two paired standard errors), leads in 10, and trails in 3, each time by at most 2.8 points (Figure 5). We exclude two TEAL comparisons where both versions collapse. Pooled per model, no model favors the threshold version (Appendix Table 14). On the three post-trained models alone, the top-k version leads in 5 of 11 comparisons and the threshold version in none.

5

Threshold − top-k (points)

within 2 SE: 30

threshold ahead: 3

0 −20 −40

top-k ahead: 10

Gemma 4: repetition loops

43 comparisons, ordered by accuracy difference

Figure 5: On the same score, a selector’s top-k version matches or beats its threshold version in 40 of 43 comparisons and avoids Gemma 4’s collapse. Each dot is the accuracy of a selector’s threshold version minus that of its top-k version, at the budget the threshold was calibrated for, with ±2 paired SE. Gray marks ties, blue top-k leads (the four on Gemma 4 by 38 to 45 points), and orange threshold leads (at most 2.8 points). See Appendix Figure 8 for the same comparisons with each top-k version run at the reads its threshold version actually used. Robustness at realized reads. We also run each top-k version at the reads its threshold version actually used, a check that favors the threshold version (Section 2). The top-k version still matches or beats it in 37 of 41 comparisons (Appendix F). The largest lead of a threshold version, +11.5 ± 1.9 points for Prox on Qwen3-Thinking IFEval, disappears at the calibrated budget, where Prox scores 85.4 with top-k and 84.3 with its threshold. At similar reads, controls show roughly equal gains from per-layer budgets and from per-token counts (Appendix E). Per-token variation helps most on Qwen3.5 GPQA-Diamond, by +7.1 ± 2.7 points. But this gain comes at a budget the deployer did not choose: with its threshold calibrated for 13%, Prox reads only 9.1% there and scores 80.6, which ties the 79.9 it scores with top-k at 13.5% of reads. Other scores and calibrated parts. Our reader behaves the same way. Replacing its top-k with calibrated per-layer thresholds gives 34 ties and 2 comparisons where the top-k version leads, both on Gemma 4, out of 36. CATS (Lee et al., 2024), which thresholds exact gate activations, ties in 16 of 18 comparisons at 53% and 67% of reads, and its threshold version leads in the other 2, both on Gemma 4 (Appendix F.3). Some selectors also calibrate layer budgets (TEAL, WINA) or recovery terms (RSparse, GRAIL). Calibrated versions of these parts lower KL divergence, calibrated layer budgets even on reasoning text unlike their calibration data, but improve no benchmark beyond two standard errors; a calibrated recovery map beats our weight-only map in 0 of 18 comparisons (Appendix F.4). 4.3

L ONG REASONING CAN COLLAPSE UNDER CALIBRATED THRESHOLDS

Channel thresholds trigger repetition loops on Gemma 4. The four largest gaps in Figure 5 share one cause: on Gemma 4 at the 13% budget, the top-k versions of Prox and its output-scaled variant beat their threshold versions by 38 to 45 points. With a calibrated threshold, Gemma 4 repeats phrases until the 32,768-token limit. Reading too little does not explain this. Over four samples per question, Prox with its threshold calibrated for 18% reads 13.3% and scores 11.7. With top-k at nearly the same reads, 13.5%, Prox scores 56.3 (Appendix Figure 10), and 145 rather than 674 of 792 answers hit the limit. Replacing only Prox’s channel threshold with top-k avoids the collapse; replacing only its input threshold does not (Appendix F.2). Neither a 4% floor on channels nor calibration on Gemma 4’s own reasoning prevents it (Appendices D.1 and F.2). Similar loops appear in other compressed models: pruning amplifies loops at stable perplexity (Lee et al., 2026b), 2-bit quantization often makes reasoning loop or exhaust its token budget (Alimaskina et al., 2026), and batch-shared TEAL masks collapse reasoning (Kim et al., 2026c). Collapse behaves like a cliff. We suspect that once a sparse model starts a repetition, it rarely stops on its own. If so, accuracy drops sharply, not gradually, once the error sparsity adds to each token passes a level that depends on the model, and a less accurate selector needs more reads to stay below it. On Gemma 4 IFEval, where the full model scores 93.5, WINA as published scores 40.9 at 20% of reads, and 53% of its answers stop at the length limit. Prox’s score with top-k reaches 57.1 at 13.5% of reads, and our reader 88.0 at 12.8%. Prox still collapses when each layer’s count is fixed to its 6

average under the threshold (Appendix E). With a calibrated threshold, a deployer therefore cannot see this level coming: the selector’s reads are not known in advance (Section 4.1), and even at equal reads its accuracy can fall far below that of its top-k version (11.7 against 56.3 above).

5

A READER BUILT FROM THE WEIGHTS ALONE

With top-k as the keep rule, a selector’s quality depends on two things: how well it scores channels before reading them, and how it handles the channels it skips. We build our reader around both. Scoring matters more than the keep rule. Switching Prox from its calibrated threshold to top-k changes next-token KL divergence by only −17% to −3% on held-out OLMoE documents and by −6% to +24% on chat. Changing the score moves KL much more. On the same 4-bit weight copy, scaling Prox’s score to output units, with down-projection norms and router weights, divides KL by 1.3 to 1.5. On OLMoE development data at 17.8% of reads, with no recovery, selectors that score with an INT4 copy of the gate projection or with TEAL’s input magnitudes reach 2.8 and 5.1 times the KL divergence of our reader’s full-sketch screen (Appendix Table 17). Across four scores and four budgets on OLMoE, KL divergence falls as the kept channels carry a larger share of the routed output’s energy (Spearman −0.96; Appendix G). Our reader therefore puts its effort into the score and into recovering skipped channels. It is built once from the model’s weights, with nothing fitted to the target model. Three settings were chosen once on OLMoE development data and reused on every model: 128-coordinate rotation blocks, 80-coordinate partial screens, and rank-32 recovery (Appendix G). For each token and layer, it screens all candidate channels, computes the top-k exactly, and recovers the rest (Figure 6). Full model: every channel Ge , V e

a

Our reader: only the top-k channels score

y

De

x

Ge , V e

not read darker = larger share of y

ternary sketch

reads all 3IH weights

sketch estimates of the rest

low-rank map from De

reads the sketch, 3kH weights and a small map

1 Screen every channel Rx

ŷ

De +

I

H

a k

x

score

2 Compute the top-k native G, V rows

D

yK

e1

e1

e2

e2

A ternary sketch of the weights, under half the size of an INT4 copy, estimates every candidate channel from the token’s largest coordinates and scores it by how much it adds to the output.

Each token keeps the same number of channels, chosen across all of its experts, so there is no threshold to tune. Their rows and columns are read from the native weights and computed exactly.

a

3 Recover the rest p

U

U z yK

bj z

+

ŷ

The skipped channels already have sketch estimates pj . A low-rank map built from the down projections estimates the output they would have added.

Each token reads 17.8% of the full model’s expert weights (Qwen3-Thinking) 82% is not read sketch 4.9%

exact channels 12.3%

recovery map 0.6%

The sketch, the ranking and the recovery map all come from the checkpoint’s weights alone.

Figure 6: Our reader screens candidate channels with a ternary sketch, computes the top-k channels exactly, and recovers the rest using only the full model’s weights. Top: the full model reads every channel of a routed expert; our reader reads a ternary sketch, the k kept channels, and a low-rank recovery map. Middle: the three stages. Bottom: reads at 17.8% on Qwen3-Thinking.

7

5.1

S CREENING CHANNELS WITH A ROTATED TERNARY SKETCH

The screen estimates every candidate channel’s gate and up activations from a ternary sketch of the weights, built once at load time. Weights and hidden states contain outliers, which make low-bit codes inaccurate. We therefore rotate each gate row gj and up row vj with a randomized Hadamard transform R (Ashkboos et al., 2024; Tseng et al., 2024), applied in blocks of 128 coordinates, which spreads large values across each block. Because R is orthogonal, the rotation leaves activations unchanged: gj⊤ x = (Rgj )⊤ (Rx). Each block of Rgj and Rvj is stored as ternary codes {−1, 0, 1} (Liu et al., 2023a), together with one FP16 scale and the block’s mean-squared quantization error ρb . This shrinks a block from 256 bytes in BF16 to about 30 bytes, so the sketch adds about 7% to resident memory. At decode time, the reader rotates x once and multiplies it by the sketch, giving estimates ĝj ≈ gj⊤ x and v̂j ≈ vj⊤ x. At our two lowest budgets, 12.8% and 17.8% of reads, reading the full sketch would take 44 to 61% of all reads on OLMoE. At these two budgets the reader runs a partial screen: in each block, it reads ternary codes only at the 80 coordinates where |Rx| is largest. This buys 20 to 50% more exact channels. 5.2

O UTPUT- SCALED SCORING AND EXACT TOP -k COMPUTATION

The reader scores each candidate channel by the size of what it would add to the MoE output. Channel j adds wj aj dj , where wj is the router weight of channel j’s expert, aj = ϕ(gj⊤ x)(vj⊤ x) its activation, and dj its down-projection column (Figure 2a). The size of this contribution is wj ∥dj ∥ |aj |, and replacing aj with its sketch estimate gives the score sj = wj ∥dj ∥ ϕ(ĝj ) v̂j ,

(1)

where the norms ∥dj ∥ are computed once at load time. Because the score is in output units, channels from different experts are directly comparable. The reader therefore picks the k best channels across all of the token’s experts, and one expert may receive many channels while another receives none. From 28.7% of reads upward, the reader reads the full sketch and also uses its block errors ρb to rank channels by the expected squared output error left after recovery if skipped (Appendix G). For the k kept channels, the reader loads the full weights and computes the exact activations aj . To make each kept channel’s down weights contiguous, we transpose every down projection De once at load time, so that column j becomes a row. Prefill and decode both use this one layout. 5.3

W EIGHT- ONLY LOW- RANK RECOVERY OF SKIPPED CHANNELS

Each skipped channel contributes little, but thousands of them per layer together matter. The screen has already estimated the activation pj = ϕ(ĝj )v̂j of every skipped channel j ∈ S, so the reader can reuse these estimates without rereading any gate or up weights. It still needs each skipped channel’s down column dj to turn the estimate into output. To avoid loading these full-width columns, we approximate them with a small map built once before deployment. We stack the down columns of all experts in a layer, compute their top 32 left singular vectors U ∈ RH×32 , and store each channel’s coefficients bj = U ⊤ dj . With kept channels K, the reader then outputs X X ŷ = wj dj aj + U wj bj pj . (2) j∈K

j∈S

The first sum is exact; the second adds the skipped channels’ estimated output through the rank-32 map. The map adds about 0.5% to resident memory and 0.5 to 0.8 percentage points of reads. Each component earns its reads. On OLMoE at equal reads, spending 0.6 to 0.7 percentage points on recovery lowers KL divergence more than spending them on more exact channels. The partial screen lowers KL relative to the full screen, from 0.488 to 0.310 at 13.5% of reads and from 0.245 to 0.187 at 18.5%. The same components help benchmark accuracy on Gemma 4. With the full screen at 18% of reads, recovery raises IFEval accuracy by 11.6 ± 1.6 points and reduces repetition loops. At 13%, the partial screen beats the full screen by 13.7 ± 1.3 points on MMLU-Pro.

8

6

ACCURACY AND SPEED ON POST- TRAINED M O E MODELS

6.1

ACCURACY

Experimental setup and baselines. We evaluate our reader in vLLM (Kwon et al., 2023) on three post-trained MoE models, Qwen3-Thinking, Gemma 4, and Qwen3.5, and three benchmarks, GPQA-Diamond, IFEval, and MMLU-Pro (Appendix B). The full model processes each prompt, and the selector runs on every decoded token. We compare against three published selectors. Prox (Liu et al., 2026) uses calibrated thresholds, and we also run its score with top-k. TEAL (Liu et al., 2025a) uses calibrated thresholds with greedy per-layer budgets. WINA (Chen et al., 2026b) as published keeps the top-k with calibrated per-projection budgets, and we also run its score with calibrated thresholds. Every calibrated part is fitted on the same UltraChat assistant turns (Ding et al., 2023). We report accuracy as a percentage of the full model’s accuracy, over nine model-benchmark pairs (Appendix Table 18), and count a difference only beyond two question-clustered paired standard errors. Table 1: Our reader is the most accurate tested selector at the two lowest budgets. Mean and worst accuracy over nine model-benchmark pairs, as a percentage of the full model’s accuracy. Selectors with calibrated thresholds list the budget they were calibrated for (Target) and the reads they used (Realized); WINA as published uses top-k and meets its budget to the precision shown. About 13% of reads Selector Our reader Prox score, top-k Prox as published, calibrated WINA as published, top-k WINA score, calibrated TEAL as published, calibrated

About 18% of reads

Target Realized Mean Worst Target Realized Mean Worst – – 13 – 15 15

94.5 87.0 51.5 2.4 0.2 0.2

12.8 13.5 8.5 15 7.8 8.9

86.9 61.1 0.0 0.0 0.0 0.0

– – 18 – 20 20

17.8 18.5 12.9 20 11.2 12.6

98.1 96.1 68.5 28.4 5.0 9.9

95.0 90.4 12.1 1.1 0.0 0.0

Accuracy at heavy sparsity. Our reader has the highest mean accuracy at both low budgets, and its weakest pair beats every other selector’s weakest pair (Table 1; Appendix Figure 9 shows every pair). Prox, with its threshold calibrated for 18%, reads 12.9%, about as much as our reader at 12.8%. Yet it reaches only 68.5% of full-model accuracy on average, against 94.5% for our reader. Most of this gap comes from Gemma 4, where its threshold version falls into the repetition loops of Section 4.3; on the two Qwen models, Prox is about as accurate as our reader at similar reads. Top-k cannot rescue a weak score: WINA as published uses top-k and reaches only 28.4% of full-model accuracy at 20% of reads. From 28% of reads upward, our reader and Prox, with either keep rule, stay within 10% of the full model. 6.2

S PEED

Our reader accelerates MoE blocks by up to 2.69× in optimized vLLM. Our Triton kernels select exactly the channels of the reader evaluated above, at 17.8% of reads on one RTX PRO 6000 Blackwell GPU. We compare against unmodified vLLM in its fastest configuration, default or tuned, at each batch size, with the same router (Figure 7). On Qwen3-Thinking, Gemma 4, and Qwen3.5, MoE blocks run 1.90×, 1.71×, and 1.60× faster at batch one, and 2.39×, 2.34×, and 2.51× at four. The whole model gains less, because attention runs as before. Whole models decode 1.15× to 1.36× faster at batch one and up to 1.75×, 1.74×, and 1.88× at batch eight or sixteen, in the same model order, and 1.08× to 1.54× with 8k to 32k tokens of context. At 100 tokens per second per user, one GPU serves 3.3 to 4.3 times as many tokens (Figure 7b; Appendix J). At batch one, the speedup is smaller than the saved reads suggest. On Qwen3-Thinking, the MoE block moves only 18.3% of the full model’s memory traffic, yet it runs 1.90× faster, not about 5×. Part of each step is a fixed cost, such as launching kernels, that fewer reads do not shrink. Larger batches share these fixed costs, so the block speedup grows up to batch four or eight (Table 20).

9

Qwen3-Thinking

2.12 1.90 1.71

1

Full model

(b) Whole model, batch sizes 1 to 128 2.51

2.69 2.33

2.62

2.22 1.98 2.01

2.00 1.71

1.60

1×

0

Our reader

(a) MoE block 2.39 2.34

2×

Qwen3.5

2

4

8

16

Decode batch size

Output tokens/s per GPU

MoE-block speedup over optimized vLLM

3×

Gemma 4

3,200 1,600 800 400 200 25

50

100

200

Output tokens/s per user

Figure 7: Our reader accelerates MoE blocks by 1.60 to 2.69× over optimized vLLM across three post-trained models. At 17.8% of routed-expert reads, median of three processes. (a) MoEblock decode speedup; both arms share the unmodified router, and Gemma 4’s dense MLP and Qwen3.5’s shared expert are timed inside the block. (b) Whole model, batch sizes 1 to 128: right is faster per user, up is more output per GPU. In dense models, the channels a batch needs together grow with batch size and approach the dense set in deeper layers, so channel selection helps mainly at small batches (Shrestha et al., 2025). MoE models with many small experts push this limit to larger batches: at batch 16, each expert serves about one token or fewer on average. Our reader still decodes 1.72× to 1.88× faster as a whole model at this batch size; Park et al. (2026) likewise find their MoE-layer speedup peaks at batch 16 to 128. At batch 128, the whole-model speedup falls to 1.00× to 1.24×.

7

L IMITATIONS AND CONCLUSION

Limitations. Our kernels turn only part of the saved reads into speed (Section 6.2), and they were tuned for one budget on one GPU type (Appendix J). Selecting channels for a whole batch at once, so the gain lasts to larger batches, is future work. The share of coordinates the partial screen reads was chosen on OLMoE and checked only on Gemma 4 (Table 17). Our reader changes only routed experts in decoding. Its sketch and recovery map add about 7% and 0.5% of a BF16 checkpoint, a larger share of a quantized one (Table 3). We tested models of up to 35B parameters. What our results suggest. We suspect two patterns hold beyond our models. First, short likelihood evaluations hide failures of long generation. On likelihood tasks, the two versions of a selector differ little and in both directions. On generation tasks at a 13% or 18% budget, no threshold version beats its top-k version, and Gemma 4 falls into repetition loops under thresholds. REAP (Lasby et al., 2026) saw the same contrast when merging experts, and pruning can add loops at largely unchanged perplexity (Lee et al., 2026b). Second, lower KL divergence need not mean higher accuracy: calibrated layer budgets lower KL divergence even on reasoning text unlike their calibration data, yet neither they nor calibrated recovery maps raise a benchmark score. Conclusion. Channel selection for MoE decoding needs no calibration data. With a calibrated threshold, a selector’s reads follow the input rather than the budget. At the calibrated budget, its threshold version gains no consistent accuracy over its top-k version on the same score. Built from the weights alone, our reader retains 98.1% of full-model accuracy at 17.8% of routed-expert reads on three models and runs MoE blocks up to 2.69× faster and whole models up to 1.88× faster than optimized vLLM.

10

AI USE STATEMENT We used LLM-based coding assistants to write the experiment code and kernels, run experiments, and write the scripts that summarize results, and to help draft and edit the text. The authors designed the study, checked every reported number against the experiment outputs, revised the manuscript, and take full responsibility for its content. R EPRODUCIBILITY STATEMENT All models, benchmarks and calibration data we use are publicly available. Sections 2 and 5 define the selectors and our reader. Appendix B gives the models, benchmarks, generation settings, statistics, calibration protocols, baseline adaptations and hardware, and Appendix C defines how reads are counted. Our reader, its Triton kernels, the vLLM patches, the baselines, the evaluation scripts and the scripts that compute the paper’s numbers from the evaluation results are in https: //github.com/AidenGeunGeun/calibration-free-moe.

R EFERENCES Ali Abbasi, Justin Shi, and Soheil Kolouri. BASE: Batch-aware selection of experts using predicted removal error for efficient MoE decoding. arXiv preprint arXiv:2609.36222, 2026. Mukund Agarwalla and Chih-Jen Lin. TopK-Guided: Adaptive, budget-aware activation sparsity for efficient LLM inference. arXiv preprint arXiv:2610.01763, 2026. Yash Akhauri, Ahmed F. AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M. Rush, Safeen Huda, and Mohamed S. Abdelfattah. ShadowLLM: Predictor-based contextual sparsity for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19154–19167, 2024. Ekaterina Alimaskina, Darya Rudas, Denis Shveykin, Gleb Molodtsov, Pavel Vasiliev, and Aleksandr Beznosikov. Extreme low-bit inference in reasoning models: Failure modes and targeted recovery. arXiv preprint arXiv:2606.02011, 2026. Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865–10873, 2024. Anonymous. Where does attention mass live? A matched-budget study of query-adaptive attention sparsification for decoder LMs. Submitted to International Conference on Learning Representations 2027, 2026a. URL https://openreview.net/forum?id=A9VqLttv6t. Anonymous. QuaL-MoE: Practical dynamic latent mixture-of-experts with quantile balancing. Submitted to International Conference on Learning Representations 2027, 2026b. URL https: //openreview.net/forum?id=5r5dBeWjfX. Anonymous. ReComp: Low-rank residual compensation for contextual sparsity in language models. Submitted to International Conference on Learning Representations 2027, 2026c. URL https: //openreview.net/forum?id=o6YIvAjXBd. Anonymous. Sparser-MoE: Beyond expert sparsity with heterogeneous neuron-level sparsity. Submitted to International Conference on Learning Representations 2027, 2026d. URL https: //openreview.net/forum?id=GRUZkymchi. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems, volume 37, pp. 100213– 100240, 2024. Sankar Behera, Dhruv Singh, Anshika Agnihotri, Raj Kumar Choudhary, Satyadev Ahlawat, and Yamuna Prasad. LILA: Calibration-free structured pruning of large language models via latent spectral geometry. arXiv preprint arXiv:2609.11163, 2026.

11

Lei Chen, Yuan Meng, Xiaoyu Zhan, Zhi Wang, and Wenwu Zhu. WiSparse: Boosting LLM inference efficiency with weight-aware mixed activation sparsity. arXiv preprint arXiv:2602.14452, 2026a. Sihan Chen, Dan Zhao, Jongwoo Ko, Colby Banbury, Huiping Zhuang, Luming Liang, Pashmina Cameron, and Tianyi Chen. WINA: Weight informed neuron activation for accelerating large language model inference. In International Conference on Learning Representations, pp. 122858– 122880, 2026b. Xing Chen and Hengshuai Yao. Training-free halving of activated experts in fine-grained mixture-ofexperts models. arXiv preprint arXiv:2609.04575, 2026. Runxi Cheng, Yuchen Guan, Yucheng Ding, Qingguo Hu, Yongxian Wei, Chun Yuan, Yelong Shen, Weizhu Chen, and Yeyun Gong. Mixture of neuron experts. arXiv preprint arXiv:2510.05781, 2025. Jaewon Cheon and Pilsung Kang. COUNTDOWN: Contextually sparse activation filtering out unnecessary weights in down projection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 28393–28409, 2025. Krishna Teja Chitty-Venkata, Sandeep Madireddy, Murali Emani, and Venkatram Vishwanath. LExI: Layer-adaptive active experts for efficient MoE model inference. arXiv preprint arXiv:2509.02753, 2025. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3029–3051, 2023. Harry Dong, Beidi Chen, and Yuejie Chi. Prompt-prompted adaptive structured pruning for efficient LLM generation. In Conference on Language Modeling, 2024. William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10323–10337. PMLR, 2023. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate post-training quantization for generative pre-trained Transformers. In International Conference on Learning Representations, 2023. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 2024. URL https://zenodo.org/records/12608602. Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026. Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. OLMES: A standard for language model evaluations. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 5020–5048, 2025. 12

Junhui He, Shangyu Wu, Weidong Wen, Chun Jason Xue, and Qingan Li. CHESS: Optimizing LLM inference via channel-wise thresholding and selective sparsification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18658–18668, 2024. Wenwen Hou, Xinyuan Song, Wenqian Weng, and Shiwei Liu. ActTail: Global activation sparsity in large language models. arXiv preprint arXiv:2603.12272, 2026. Yushi Huang, Zining Wang, Zhihang Yuan, Yifu Ding, Ruihao Gong, Jinyang Guo, Xianglong Liu, and Jun Zhang. MoDES: Accelerating mixture-of-experts multimodal large language models via dynamic expert skipping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 30205–30215, 2026. Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, and Min Zhang. Beware of calibration data for pruning large language models. In International Conference on Learning Representations, 2025. Dohyeon Kim, Bedionita Soro, and Sung Ju Hwang. Distribution-consistent inference for dynamic sparse mixture-of-experts. In Findings of the Association for Computational Linguistics: EMNLP 2026, 2026a. JuneHyung Kim, Sankeerth Durvasula, and Nandita Vijaykumar. Activation sparsity with weight approximation for faster LLM decoding on offloaded weights. arXiv preprint arXiv:2610.02598, 2026b. Yongmin Kim, Shota Takashiro, Yusuke Iwasawa, Takeshi Kojima, and Yutaka Matsuo. Batch-wise adaptive pruning: Periodic neuron activation-aware weight pruning for language reasoning model. In Conference on Language Modeling, 2026c. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023. Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, and Vithursan Thangarasa. REAP the experts: Why pruning prevails for one-shot MoE compression. In International Conference on Learning Representations, pp. 146883–146912, 2026. Donghyun Lee, Jaeyong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. CATS: Contextaware thresholding for sparsity in large language models. In Conference on Language Modeling, 2024. Jungseob Lee, Seungyoon Lee, Seongtae Hong, Sugyeong Eo, and Heuiseok Lim. Where activation sparsity and KV-cache sparsity cross in LLM decoding. arXiv preprint arXiv:2609.33889, 2026a. Junyoung Lee, Sehyeon Park, Shinhyoung Jang, Seonha Ryu, Hojeong Kim, Hyunsei Lee, Il Hong Suh, and Yeseong Kim. FOCUS & RePAIR: Mitigating text degeneration via token-level guidance for pruned large language models. In International Conference on Machine Learning, 2026b. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, pp. 87–100, 2024. Bin Liu, Fengfu Li, Xiaoxing Wang, Bo Zhang, and Junchi Yan. Ternary weight networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, 2023a. James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Trainingfree activation sparsity in large language models. In International Conference on Learning Representations, pp. 98302–98322, 2025a. Jinyi Liu, Wei Chen, Pengyu Chen, Xinyi Yuan, Minghe Bai, Guoquan Wu, and Jun Wei. Prox: Training-free FFN activation sparsity via approximate intermediate-channel salience in LLMs. arXiv preprint arXiv:2607.27591, 2026. 13

Kai Liu, Bowen Xu, Shaoyu Wu, Xin Chen, Hao Zhou, Yongliang Tao, and Lulu Hu. La RoSA: Enhancing LLM efficiency via layerwise rotated sparse activation. In International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 39968–39986. PMLR, 2025b. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quantization with learned rotations. In International Conference on Learning Representations, 2025c. Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja Vu: Contextual sparsity for efficient LLMs at inference time. In International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 22137–22176. PMLR, 2023b. Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6159–6172, 2024. Yuqi Luo, Chenyang Song, Xu Han, Yingfa Chen, Chaojun Xiao, Xiaojun Meng, Liqun Deng, Jiansheng Wei, Zhiyuan Liu, and Maosong Sun. Sparsing law: Towards large language models with greater activation sparsity. In International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 41311–41330. PMLR, 2025. Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Huaqing Zheng, Xindian Ma, and Peng Zhang. SharQ: Bridging activation sparsity and FP4 quantization for LLM inference. arXiv preprint arXiv:2606.26587, 2026. Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. OLMoE: Open mixture-of-experts language models. In International Conference on Learning Representations, pp. 62061–62121, 2025. NVIDIA. CUDA C++ best practices guide, 2025. URL https://docs.nvidia.com/cuda /archive/13.0.0/cuda-c-best-practices-guide/index.html. Version 13.0, August 4, 2025. Section “Coalesced Access to Global Memory”. Jongseok Park, Sunga Kim, Zhenyu Gu, Ion Stoica, and Alvin Cheung. Uncovering intra-expert activation sparsity for efficient mixture-of-expert model execution. arXiv preprint arXiv:2605.08575, 2026. Qwen Team. Qwen3.5-35B-A3B. Official model card; accessed 2026-09-24, 2026. URL https: //huggingface.co/Qwen/Qwen3.5-35B-A3B. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In Conference on Language Modeling, 2024. Yashas Samaga B L, Varun Yerram, Chong You, Srinadh Bhojanapalli, Sanjiv Kumar, Prateek Jain, and Praneeth Netrapalli. HiRE: High recall approximate top-k estimation for efficient LLM inference. arXiv preprint arXiv:2402.09360, 2024. Noam Shazeer. GLU variants improve Transformer. arXiv preprint arXiv:2002.05202, 2020. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. Jiho Shin, Hoeseok Yang, and Youngmin Yi. SparseInfer: Training-free prediction of activation sparsity for fast LLM inference. In 2025 Design, Automation & Test in Europe Conference (DATE), pp. 1–7, 2025. 14

Susav Shrestha, Bradley Settlemyer, Nikoli Dryden, and Narasimha Reddy. Polar Sparsity: High throughput batched LLM inferencing with scalable contextual sparsity. In Advances in Neural Information Processing Systems, volume 38, pp. 84842–84870, 2025. Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. PowerInfer: Fast large language model serving with a consumer-grade GPU. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pp. 590–606, 2024. Luoyang Sun, Guangyan Li, Cheng Deng, Haifeng Zhang, Jian Zhao, Yongqiang Tang, Wensheng Zhang, and Jun Wang. Dual activation-weight sparsity: A training-free framework for efficient large language model compression. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8350–8366, 2026. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. In International Conference on Learning Representations, 2024. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIGBench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051, 2023. Filip Szatkowski, Patryk B˛edkowski, Alessio Devoto, Jan Dubiński, Pasquale Minervini, Mikołaj Piórczyński, Simone Scardapane, and Bartosz Wójcik. Universal properties of activation sparsity in modern large language models. In International Conference on Learning Representations, pp. 146247–146269, 2026. Wenwu Tang, Dong Wang, Lothar Thiele, and Olga Saukh. GRAIL: Post-hoc compensation by linear reconstruction for compressed networks. In Conference on Parsimony and Learning, volume 328 of Proceedings of Machine Learning Research, pp. 881–895. PMLR, 2026. Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QuIP#: Even better LLM quantization with Hadamard incoherence and lattice codebooks. In International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 48630–48656. PMLR, 2024. Abhishek Tyagi, Yunuo Cen, Shrey Dhorajiya, Bharadwaj Veeravalli, and Xuanyao Fong. DART-ing through the drift: Dynamic tracing of knowledge neurons for adaptive inference-time pruning. arXiv preprint arXiv:2601.22632, 2026. Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao, Shiai Zhu, and Huan Wang. SparseDecoding: Decoding-aware pruning for accurate and efficient LLM inference. arXiv preprint arXiv:2610.12327, 2026. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems, volume 37, pp. 95266–95290, 2024. Miles Williams and Nikolaos Aletras. On the impact of calibration data in post-training quantization and pruning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10100–10118, 2024. Juntong Wu, Jialiang Cheng, Fuyu Lv, Dan Ou, and Li Yuan. SERE: Similarity-based expert rerouting for efficient batch decoding in MoE models. In International Conference on Learning Representations, 2026a. Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen, and Li Yuan. ExFold: Unified expert folding for training-free MoE prefill-decode acceleration. arXiv preprint arXiv:2608.24938, 2026b. Tong Wu, Yutong He, Bin Wang, and Kun Yuan. Mixture-of-Channels: Exploiting sparse FFNs for efficient LLMs pre-training and inference. arXiv preprint arXiv:2511.09323, 2025. 15

Zukang Xu, Zhixiong Zhao, Xing Hu, Jiangyong Yu, Houji Wen, Jun Li, Zhe Jiang, and Dawei Yang. ACE: Adaptive calibration-free expert skipping for MoE-based LLMs. arXiv preprint arXiv:2609.05228, 2026. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Zhenyu Zhang, Zechun Liu, Yuandong Tian, Harshit Khaitan, Zhangyang Wang, and Steven Li. R-Sparse: Rank-aware activation sparsity for efficient LLM inference. In International Conference on Learning Representations, pp. 77452–77466, 2025. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Yuxin Zhou, Zheng Li, Jun Zhang, Jue Wang, Yiping Wang, Zhongle Xie, Ke Chen, and Lidan Shou. FloE: On-the-fly MoE inference on memory-constrained GPU. In International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 78859–78882. PMLR, 2025.

16

A PPENDIX CONTENTS A Extended related work

18

B Experimental setup

19

C Read accounting and profiler check

21

D Calibrated thresholds read less than their budget

22

D.1 Calibrating on the model’s own generations . . . . . . . . . . . . . . . . . . . . .

22

E What a calibrated threshold’s channel count follows

23

F Every keep-rule comparison

25

F.1

Applying Prox to OLMoE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

F.2

Repetition loops on Gemma 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

F.3

Our reader with a calibrated threshold, and CATS . . . . . . . . . . . . . . . . . .

27

F.4

Calibrated layer budgets and calibrated recovery . . . . . . . . . . . . . . . . . . .

28

G What each part of the reader contributes

30

H Every model and task

31

I

Baseline comparison on OLMoE

35

J

Speed details

35

17

Table 2: Prior keep-rule comparisons reach opposite conclusions, focus mostly on short tasks on dense models, and none runs top-k at the reads a threshold actually uses. Unlike most prior comparisons, ours holds the channel score fixed (Same score), tests long chain-of-thought reasoning on MoE models (Regime), and runs top-k both at the target budget and at realized reads (Matched). Study

Regime

Models

Same scoreMatchedAhead

La RoSA (2025b) dense, PPL LLaMA2-7B yes Batch-wise pruning (2026c)dense, CoT 2 distilled 7–8B yes Sparsing Law (2025) dense, short tasks 0.1B, 0.8B no Prox (2026) dense, short tasks Qwen3-8B, 14B yes TopK-Guided (2026) dense, short tasks Llama-2/3, 7–8B no Sparser-MoE (2026d) MoE, short tasks OLMoE unclear

target target target target target target

top-k (PPL) top-k (batch 4) threshold threshold (+1.8) hybrid; top-k vs. τ mixed top-k (<1 pt)

This paper (§4.2)

both

tie; top-k on Gemma 4

A

MoE, short + CoT4 MoE models

yes

E XTENDED RELATED WORK

Prior comparisons of calibrated thresholds and top-k. Earlier studies disagree on whether a calibrated threshold or top-k selects channels better (Table 2). La RoSA, Sparser-MoE, and batch-wise adaptive pruning find top-k ahead; batch-wise adaptive pruning is the only prior study in the table that tests long chain-of-thought reasoning. The Sparsing Law and Prox find calibrated thresholds ahead. TopK-Guided finds top-k (WINA) ahead of thresholded TEAL on Llama-2-7B but not consistently on Llama-3-8B, and its best rule is a hybrid: a calibrated threshold whose per-token count stays within ±δ of k, combined with calibrated block budgets. In our MoE experiments, calibrated layer budgets give no held-out benchmark gain (Appendix F.4), and a per-token floor on the count does not prevent Gemma 4’s collapse (Appendix F.2). We do not test a two-sided ±δ band. Approximating skipped channels. Instead of simply dropping the channels they skip, several methods approximate what those channels would have added. R-Sparse adds low-rank weight factors from an SVD and chooses how to split the budget between them and sparse inputs on calibration data. GRAIL (Tang et al., 2026) fits linear compensation maps on calibration data to approximate omitted activations. ReComp (Anonymous, 2026c) distills a calibrated low-rank estimate of the skipped MLP output. Other methods send medium-importance columns through a compressed copy, as in SpAx (Kim et al., 2026b), or through a pruned or low-precision path, as in DAWS (Sun et al., 2026) and SharQ (Meng et al., 2026). Our recovery map needs no calibration data. It reuses each skipped channel’s activation estimate from the ternary screen and passes it through the top singular vectors of the down projections. Speed under batching. In batched decoding, the GPU must load every channel that any token in the batch selects, so per-token selection saves fewer reads as the batch grows (Shrestha et al., 2025). SERE (Wu et al., 2026a) addresses this by shrinking the union of routed experts across a batch. Weak dense baselines can also inflate reported speedups (Lee et al., 2026a). We therefore charge reads in hardware sectors, check them with a GPU profiler, time against optimized vLLM, and report speedups at batch sizes one to sixteen. Other levels of sparsity. Other methods act at a different granularity and can be combined with our reader. Expert-skipping and expert-merging methods drop, prune, or fold whole experts (Lu et al., 2024; Huang et al., 2026; Chitty-Venkata et al., 2025; Lasby et al., 2026; Chen & Yao, 2026; Xu et al., 2026; Wu et al., 2026b; Abbasi et al., 2026; Kim et al., 2026a). Static compression methods prune or quantize weights ahead of time (Frantar & Alistarh, 2023; Frantar et al., 2023; Sun et al., 2024; An et al., 2024; Liu et al., 2025c; Behera et al., 2026). Finally, prior work has studied per-token channel selection (Samaga B L et al., 2024), adaptive pruning with little or no calibration data (Dong et al., 2024; Tyagi et al., 2026; Kim et al., 2026c), and sparsity inside MoE experts (Zhou et al., 2025; Park et al., 2026), including top-k neuron selection trained into experts during pretraining (Cheng et al., 2025).

18

Table 3: Evaluated MoE models and the resident memory our reader adds. Each row gives the number of routed experts and how many each token uses, the channels per expert and the hidden width, the number of MoE layers, any module that always runs in full, and the resident memory of our reader’s ternary sketch and recovery map as a percentage of the BF16 checkpoint. OLMoE’s sketch size is computed analytically. Routed experts

Width

MoE

Always

Added memory (%)

Model

Total

Active

Expert

Hidden

layers

full

Sketch

Recovery

Qwen3-Thinking Gemma 4 Qwen3.5 OLMoE

128 128 256 64

8 8 8 8

768 704 512 1024

2048 2816 2048 2048

48 30 40 16

– dense MLP shared expert –

7.6 7.0 7.0 7.3

0.50 0.35 0.47 0.50

Table 4: Evaluation protocol on the post-trained models. Each model runs in thinking mode with its model card’s sampling parameters and random seed 20260923. GPQA-Diamond reports accuracy over four samples per question, IFEval reports prompt-level strict accuracy, and MMLU-Pro reports accuracy on a fixed 1,000-question subset. Sampling Model

Benchmark

Items

Samples

Temperature

Top-p

Top-k

Max tokens

Qwen3-Thinking

GPQA-D IFEval MMLU-Pro

198 541 1,000

4 1 1

0.6 0.6 0.6

0.95 0.95 0.95

20 20 20

32,768 32,768 32,768

Gemma 4

GPQA-D IFEval MMLU-Pro

198 541 1,000

4 1 1

1.0 1.0 1.0

0.95 0.95 0.95

64 64 64

32,768 32,768 32,768

Qwen3.5†

GPQA-D IFEval MMLU-Pro

198 541 1,000

4 1 1

1.0 1.0 1.0

0.95 0.95 0.95

20 20 20

81,920 32,768 32,768

†

B

Presence penalty 1.5.

E XPERIMENTAL SETUP

Models and runtime. We evaluate Qwen3-30B-A3B-Thinking-2507 (Qwen3-Thinking), Gemma4-26B-A4B-it (Gemma 4), and Qwen3.5-35B-A3B (Qwen3.5) in their official BF16 releases in vLLM 0.29.0. For every prompt, the full model processes the prompt and generates the first output token, and the channel selector runs on every later decoded token. Shared experts and parallel dense MLPs always run in full and are not counted in routed-expert reads. Benchmarks. GPQA-Diamond uses all 198 questions with four sampled answers per question, in each model card’s prompt format and with its sampling parameters. IFEval uses all 541 prompts and reports prompt-level strict accuracy. MMLU-Pro uses a fixed stratified subset of 1,000 questions. Answers may be up to 32,768 tokens long, except on Qwen3.5 GPQA-Diamond, where its model card recommends 81,920 tokens. The full Qwen3-Thinking model also ran GPQA-Diamond with an 81,920-token cap, but none of its answers exceeds 23,500 tokens. On OLMoE, GSM8K (Cobbe et al., 2021) (1,319 questions, eight-shot chain of thought) and BBH (Suzgun et al., 2023) (6,511 questions, three-shot chain of thought) use greedy decoding up to 512 tokens under the task definitions of the OLMES evaluation suite (Gu et al., 2025). Answers that hit the length limit. On the post-trained models, an answer that reaches the token limit without finishing counts as wrong, for every method. On IFEval, most such answers contain no final answer at all. Of the 10,872 IFEval answers that hit the limit in our main runs, 10,649 never left their reasoning trace. Of the other 223, 202 ended in repetition loops and the rest stopped mid-sentence. IFEval’s format checkers can still pass such fragments on rules such as lowercase-only

19

or no commas, and crediting them would change single model-benchmark results by up to 3.3 points. Counting them as wrong avoids crediting collapsed outputs. Statistics. Benchmark comparisons use paired standard errors clustered by question. We count two methods as different when their paired difference exceeds two standard errors, without correcting for the number of comparisons. A tie therefore means no difference was detected, not that the methods are equivalent. Unless stated otherwise, ± denotes one standard error. Next-token KL divergence is KL(P ∥ P̂ ), from the full model’s next-token distribution P to the sparse model’s P̂ , averaged over scored positions; top-1 agreement is the share of positions where both put the same token first. Calibration on the post-trained models. Every calibrated method is fitted on the same UltraChat conversations (Ding et al., 2023). We split them with a fixed seed into 448 fitting, 64 tuning, and 128 development conversations; the tuning set only selects the checkpoint of the calibrated recovery map. Each conversation is rendered with the model’s chat template and contributes one 128-token window from an assistant turn. Prox fits its thresholds on 20,480 token positions per layer, sampled from the full text of 30 fitting conversations. TEAL’s greedy budget allocation and both WINA variants use 512 fitting states per layer. UltraChat turns contain no reasoning traces, so these thresholds are fitted outside thinking mode and evaluated in thinking mode. Appendix D.1 repeats the calibration on the models’ own reasoning instead. How Prox splits its budget. Prox splits each target budget between input and channel sparsity with its published rule, which prices its 4-bit weight copy with Prox’s latency model. Counted in sectors, this rule plans 1.6 to 1.7 percentage points of reads below the target at budgets from 13 to 38%, and 5.6 points below at 58 and 78%. Qwen3.5’s 13% thresholds were fitted for a 12.5% target. At test time, every Prox run reads even less than this plan. Calibration on OLMoE. OLMoE is a base model, so its thresholds are fitted on Pile text (Gao et al., 2020). Prox uses the first 30 fitting documents (20,480 positions per layer), TEAL uses 177 documents (512 states per layer), and 40 further documents serve as tuning data. Our reader’s constants and the baselines’ input fractions were chosen on a separate set of 192 development documents, and the held-out OLMoE evaluation uses another 192. The development and held-out sets are disjoint from each other and from all fitting and tuning documents. Fitting Prox takes 3 to 19 minutes on a single GPU. Baselines. Prox (Liu et al., 2026) was designed for dense models. We apply it directly to each routed expert and store its 4-bit copy of the gate and up weights in the memory layout that needs the fewest sector reads. Prox’s top-k version keeps each token’s 30% largest input coordinates (50% from 28% of reads upward) and then its top-k channels; these input fractions were also chosen on OLMoE development data. TEAL (Liu et al., 2025a) uses its greedy search to divide sparsity among the projections of each layer; we apply it to the gate, up, and down projections of the routed experts. WINA’s score with a calibrated threshold uses one threshold per projection. WINA as published (Chen et al., 2026b) scores each input coordinate by its magnitude times the matching column norm of the weight matrix that consumes it (the gate projection in a gated MLP), keeps the top-k per token, and fits a sparsity level for each projection with TEAL’s greedy search. We adapt this search to routed experts. It minimizes the reconstruction error of the routed-expert output, uses empirical quantile thresholds measured on the fitting data, steps in increments of 0.01, and stops at the first sparsity level that reaches the target budget. Neither WINA variant applies the orthogonal weight transform that WINA adopts to maximize its gains. CATS (Lee et al., 2024) computes the full gate projection and keeps channels by exact gate magnitude with a calibrated threshold; we also run a per-token top-k version. Published HiRE trains the model with a group-sparse FFN; on frozen OLMoE we test only its screen, as a HiRE-style control that screens channels with an INT4 copy of the gate projection under our top-k. Hardware. Benchmark runs used a single RTX PRO 6000 Blackwell GPU, two A6000 GPUs, or four A5000 or RTX 3090 GPUs. In the multi-GPU setups, attention is split by tensor parallelism and routed experts by expert parallelism. Gemma 4 ran only on the first two setups. Before pooling results across these layouts, we ran a consistency test fixed in advance. Every layout replayed 192 20

Table 5: Most of our reader’s reads go to exact channels, and the sketch takes most of the rest. Routed-expert reads per token at batch one, as a percentage of the full model, by component at the four main budgets. At the two smallest budgets, the partial screen reads part of the ternary sketch. Routed-expert reads (%) Model

Sketch read

Sketch

Exact channels

Recovery

Metadata

Total

Qwen3-Thinking

partial partial full full

4.86 4.86 7.99 7.99

7.29 12.30 20.00 30.00

0.66 0.63 0.59 0.54

0.04 0.04 0.10 0.10

12.85 17.83 28.68 38.62

Gemma 4

partial partial full full

4.78 4.78 7.86 7.86

7.49 12.50 19.99 30.01

0.54 0.52 0.49 0.45

0.03 0.03 0.08 0.07

12.84 17.83 28.42 38.40

Qwen3.5

partial partial full full

5.08 5.08 8.33 8.33

6.98 12.04 20.00 30.00

0.74 0.72 0.68 0.62

0.04 0.04 0.11 0.10

12.84 17.87 29.11 39.06

Table 6: Our reader reads part of the sketch at the two smallest budgets and all of it above. Each column is one budget. It gives the share of candidate channels computed exactly across each token’s routed experts and the resulting total reads, which include the ternary sketch, exact channels, recovery map, and metadata, averaged over the three post-trained models and three benchmarks. Sketch read Channels computed exactly (%) Reads (% of full model)

partial 7.0–7.5 12.8

partial 12.0–12.5 17.8

full 20 28.7

full 30 38.7

full 50 58.6

full 70 78.5

held-out chat documents of 128 tokens each for the full model, our reader at 17.8% of reads, and Prox calibrated for 18%. For each sparse method, the 90% confidence interval of the change in its next-token KL divergence from the full model lay within 10% of that divergence, and its realized reads changed by less than 0.5 percentage points. For the full model, its KL divergence from the single-GPU RTX PRO 6000 run stayed below 10% of our reader’s divergence. So the layout does not materially change the numerics.

C

R EAD ACCOUNTING AND PROFILER CHECK

For each token and MoE layer, our reader is charged for every weight and parameter byte it loads from memory. This includes the ternary sketch codes and scales read during screening, the full-precision gate, up, and down rows of the k kept channels, the recovery map’s coefficient vectors for the skipped channels and its shared basis U , the precomputed per-channel norms, and the Hadamard rotation signs. Every read is counted in whole 32-byte hardware sectors at batch one, without modeling cache reuse across tokens. Every baseline is charged under the same sector model, in the weight layout that needs the fewest reads, including its thresholds, low-bit copies, and other metadata. For selectors with thresholds, we report the reads they realize on each benchmark, counted on the GPU. Table 5 breaks down our reader’s reads by component at the four main budgets. To check that the sector count matches what the hardware actually transfers, we profiled 12 of Qwen3-Thinking’s 48 MoE layers with NVIDIA Nsight Compute. The profiled kernels ran on weights, recovery maps, and hidden states captured from a live decoding step, with the cache flushed before each kernel launch. Table 7 reports the measured DRAM reads of the MoE block without the router. At batch one, our reader’s DRAM traffic is 18.3% of the full model’s, close to the 17.8% our sector count gives. At larger batches the share rises, to 21.8% at batch four and 25.7% at batch eight, because tokens in a batch select partly different channels.

21

Table 7: At batch one, the profiler measures 18.3% of the full model’s DRAM reads, close to the 17.8% we count. Measured DRAM reads per MoE layer on Qwen3-Thinking in MB, averaged over 12 layers, without the router, with a cold cache. Batch size

1

4

8

Full model Our reader Share

75.6 13.9 18.3%

229.1 49.9 21.8%

359.3 92.3 25.7%

Table 8: Every calibrated threshold reads less than the budget it was calibrated for. Each row is one selector at one calibrated budget. The columns give the lowest, mean, and highest realized routed-expert reads (%) over the nine model-benchmark pairs. Realized reads (%)

D

Method

Calibrated for (%)

Min

Mean

Max

CATS

53.3 66.7

51.1 63.8

52.1 65.1

53.0 66.0

Our reader, calibrated

12.8 17.8 28.7 38.7

11.2 15.2 24.5 32.9

11.8 16.5 26.5 35.8

12.4 17.1 27.7 37.2

Prox as published, calibrated

13.0 18.0 28.0 38.0 58.0 78.0

5.6 9.9 18.9 29.6 47.3 67.1

8.5 12.9 22.5 32.6 49.8 70.0

10.1 14.7 24.6 34.1 51.0 71.4

TEAL as published, calibrated

15.0 20.0 30.0 40.0 50.0 60.0

6.4 9.8 19.9 32.6 43.7 54.7

8.9 12.6 22.6 35.0 46.2 56.9

11.0 15.3 24.9 36.2 47.4 58.0

WINA score, calibrated

15.0 20.0 30.0 40.0 50.0 60.0

5.4 8.2 17.2 32.0 43.5 54.8

7.8 11.2 20.7 34.3 46.1 57.1

9.6 13.9 23.5 35.8 47.7 58.2

C ALIBRATED THRESHOLDS READ LESS THAN THEIR BUDGET

Section 4.1 counts 72 benchmark runs: Prox and our reader’s threshold version, each at four budgets. Table 8 extends this count to every selector we ran with a calibrated threshold on the post-trained models, 24 threshold settings on all nine model-benchmark pairs. All 216 runs read less than the budget they were calibrated for. For Prox and our reader’s threshold version, this budget is the read target of the calibration fit. For TEAL and WINA’s score with a threshold, it is the target share of inputs kept. For CATS, it is the reads of its top-k version at the same target sparsity. D.1

C ALIBRATING ON THE MODEL’ S OWN GENERATIONS

UltraChat assistant turns contain no reasoning traces, so we also fitted Prox’s thresholds on each full model’s own reasoning, for targets of 13.4, 15, 18.4, and 20%. The full model wrote one thinking-mode trace, up to 32,768 tokens, for each of 1,024 UltraChat first user turns. These turns lie outside every fitting, tuning, and evaluation split, and none shares a 13-gram with a benchmark prompt. The traces average 1,669 generated tokens on Gemma 4 and 2,302 on Qwen3-Thinking. As 22

in our chat calibration, Prox’s recipe uses 30 of these sequences and 20,480 evenly spaced positions. Each sequence is a prompt plus its trace, cut at a random point and capped at 8,192 tokens. Prox with these trace-calibrated thresholds still reads less than its targets on Gemma 4 GPQADiamond (four samples per question): 10.5, 11.6, 14.0, and 15.2% of reads. It scores 0.0, 0.3, 23.9, and 45.1, and 100, 100, 65, and 35% of its answers hit the 32,768-token limit. For comparison, Prox with its threshold calibrated on chat for 18% scores 11.7 at 13.3% of reads. On Gemma 4 IFEval, Prox with a trace-calibrated threshold scores 31.2 at 14.9% of reads, against 11.3 with chat calibration. On Qwen3-Thinking GPQA-Diamond, Prox with trace-calibrated thresholds scores 57.1 at 9.6% of reads and 62.8 at 11.2%, against 59.7 at 9.9% with chat calibration. On IFEval it scores 82.1, against 82.8. Calibrating on the model’s own reasoning therefore neither stops the reads from drifting nor prevents Gemma 4’s collapse into repetition loops.

E

W HAT A CALIBRATED THRESHOLD ’ S CHANNEL COUNT FOLLOWS

With a calibrated threshold, the number of channels kept changes from token to token. This appendix asks what that number follows, and whether the variation spends reads where tokens need them. Unless stated otherwise, we use Prox’s INT4 score on Qwen3-Thinking with the thresholds from our main evaluation, calibrated on UltraChat for 13% and 18% of reads. Where stated, we also use thresholds calibrated on the model’s own reasoning (Appendix D.1). Gemma 4 repeats the context test, and Gemma 4 and Qwen3.5 repeat the equal-read and generation tests, with thresholds calibrated on UltraChat. Every variant ranks channels by the same score; only the rule that sets each token’s count changes. Setup. We use 48 GPQA-Diamond reasoning traces written by the full Qwen3-Thinking model and teacher-force 256-token windows starting at answer positions 0, 2,000, and 6,000. The full model processes the question and the earlier answer. Inside the window, each variant runs on hidden states from its own sparse layers, and we measure its next-token KL divergence from the full model. We compare the threshold with two controls. Per-layer fixed counts give every token in a layer the same count, set so that reads in the window match the threshold’s; a two-pass procedure matches them to a relative residual below 0.05%. Shuffled counts randomly permute the threshold’s per-token counts across positions. To measure how much each token needs extra channels, a separate single-step experiment scores 576 tokens, four positions in each of the three windows of 48 traces. Each token is scored after a full-model prefix at 0.5, 0.7, 1, 1.4, and 2 times the top-k count, and its need is how much its KL divergence falls from 0.5 to 2 times the count. The earlier answer sets the count. Table 9 takes the window at position 6,000 and changes only what comes before it. Removing the earlier answer, so that the window follows the question directly, raises the kept count by 56.6 ± 2.7%. This brings it to 89% of the count at the start of the answer, where the tokens themselves differ. Among identical token IDs, Prox keeps 41% fewer channels at position 6,000 than at the start, and the count falls equally in every band of the full model’s next-token loss. On Gemma 4, whose reads fall less (from 10.5% to 9.7%), removing the earlier answer raises the kept count by 12.5 ± 0.5%. With thresholds calibrated on Qwen3-Thinking’s own traces, reads fall similarly, from 11.3% to 7.7% over 6,000 tokens. Both the length and the content of the earlier answer matter. With the exact scores and chat-calibrated threshold of Figure 4a, the window keeps 30.1 ± 1.3% more channels after the question alone and 10.4 ± 0.6% more after the last 1,000 answer tokens. Placing an answer to a different question, also 6,000 tokens long, before the window instead of the model’s own answer raises the count by 19.3 ± 1.3%. What the count follows on Qwen3-Thinking. In the single-step experiment, the threshold’s pertoken count correlates with answer position at −0.57 (Spearman, at both budgets), with need at only −0.13 (13%) and −0.19 (18%), and with the full model’s next-token loss at −0.29 and −0.26. Need itself follows loss (+0.58, +0.68) rather than position (+0.09, +0.14). Even among tokens at the same position, the count correlates slightly negatively with need (−0.11 ± 0.04 and −0.12 ± 0.05 over the 12 positions) and with loss (−0.19 ± 0.05 and −0.16 ± 0.06). On this model, removing the dependence on answer position would therefore leave variation that slightly favors tokens with lower need. On each model’s own sparse states in the three windows, the count correlates with loss at 23

Table 9: The same tokens keep more channels when the earlier answer is removed. The first three rows change only the context before the 256-token windows at answer position 6,000 of Qwen3Thinking GPQA-Diamond traces, under Prox’s threshold calibrated for 13%. Kept channels are per token, summed over 48 layers (of 294,912 candidates) and averaged over 48 traces; ± is one paired SE. The last row shows the start of the answer, which has different tokens. Context before the window

Kept channels

Question and all 6,000 earlier answer tokens Question and the last 1,000 answer tokens Question only

9,983 11,423 15,422

Start of the answer (different tokens)

17,283

Change – +15.0±1.2% +56.6±2.7%

−0.16 to −0.23 on Qwen3-Thinking, −0.21 to −0.28 on Gemma 4, and 0.00 to +0.14 on Qwen3.5. Yet on Qwen3.5, variable counts still improve quality at equal reads (below), so loss alone does not decide whether the variation helps. Equal reads. Does the variation help at equal reads? Pooled over the three windows, per-layer fixed counts lower KL divergence relative to the chat-calibrated threshold by 3.3 ± 1.0% at 13% and 3.4 ± 1.2% at 18%, and relative to the trace-calibrated threshold by 4.0 ± 0.9% and 2.9 ± 1.2%. On Gemma 4 at 13%, they lower it by 12.5 ± 1.5%. On Qwen3.5 they raise it, by 10.5 ± 1.1% at 13% and 14.8 ± 1.2% at 18%, and the threshold is ahead in all three windows at both budgets. So on Qwen3-Thinking and Gemma 4 reasoning the variation hurts, and on Qwen3.5 it helps. The threshold’s counts still carry information on every model: shuffling them across positions raises pooled KL on each. On held-out chat, the distribution the threshold was calibrated on, fixed counts instead raise KL by 6.5 ± 1.9% and 9.2 ± 2.2%. Moving reads across positions. Making each layer’s count constant across the three windows barely changes KL at equal pooled reads. We compare per-window fixed counts, matched to the threshold version’s reads in each window, with counts held at their three-window mean. At equal pooled reads, the constant counts change KL by −0.3 ± 0.5% and −0.2 ± 0.6% on Qwen3-Thinking, +0.1 ± 0.8% on Gemma 4, and +1.0 ± 0.5% (just beyond two SE) and +0.1 ± 0.7% on Qwen3.5. A reversed schedule, which reflects each layer’s counts about their three-window mean, is worse: +6.8 ± 0.7% and +7.1 ± 1.1% on Qwen3-Thinking, and +4.2 ± 0.7% and +3.5 ± 0.8% on Qwen3.5. Predicting need. An oracle that knows every token’s loss could halve KL divergence at the same total reads, but a predictor of need does not get close. A per-layer ridge regression from the token’s position and summary statistics of its screened channel scores (total and largest score mass, share in the top channels, entropy, and count above the threshold) explains about 20% of the variance in log KL. Our offline estimate of the allocation it implies is worse than fixed counts. Generation. Qwen3-Thinking IFEval is the benchmark where a threshold beats top-k at its realized reads. There, Prox keeps its top-k on inputs and uses a channel threshold calibrated for 13%. With this threshold, it scores 82.4 at 10.0% of reads. With per-layer fixed counts matched to the threshold’s per-layer budgets, it scores 77.6 at 10.3% (−4.8 ± 1.8), and with the same count in every layer, 72.8 at 10.3%. On GPQA-Diamond (one sample), it scores 59.1 at 9.9% with the threshold and 56.6 at 9.9% with fixed counts (−2.5 ± 2.8, a tie). Per-token counts thus help on IFEval but show no detectable gain on GPQA-Diamond. IFEval answers are shorter than GPQA-Diamond reasoning and fail on a single violated instruction, which may make a few tokens decisive; we did not test this. On Qwen3.5, with its threshold calibrated for 13%, Prox scores 89.5 at 9.7% of reads on IFEval, against 88.5 at 9.7% with fixed counts (−0.9 ± 1.3, a tie). On GPQA-Diamond (one sample), it scores 79.8 at 9.4%, against 72.7 at 9.1% (−7.1 ± 2.7). Without per-token counts, Qwen3.5’s answers grow longer, 16.2k against 11.3k tokens on average, and 5.1% of them hit the length limit, against 1.0%. On Gemma 4, removing the per-token variation does not prevent the collapse. On IFEval, with per-layer fixed counts matched to the threshold’s per-layer budgets, Prox scores 0.0 at 10.1% of reads, and 99.8% of its answers stop at the length limit. Prox’s threshold version also scores 0.0 at 10.1%;

24

Threshold − top-k (points)

10

within 2 SE: 30

threshold ahead: 4

0 −10 −20

top-k ahead: 7

Gemma 4: repetition loops

41 comparisons, ordered by accuracy difference

Figure 8: Figure 5 with each top-k version run at the reads its threshold version actually used. Each dot is one comparison, as in Figure 5. When a threshold version read more than one percentage point away from its calibrated budget, the top-k version is rerun at those realized reads. The figure has 41 comparisons at 13% and 18% budgets; four TEAL comparisons at heavy sparsity, where both versions collapse, are left out. Threshold versions read less than their budget, so this rerun lowers the top-k version’s reads too, yet the top-k version matches or beats the threshold version in 37 of 41. replacing only its input threshold with top-k gives 3.5 (Appendix F.2). With the same count in every layer, Prox’s top-k version scores 16.5 at 11.7%. On GPQA-Diamond, Prox with fixed counts also scores 0.0 at 9.8% of reads with every answer stopped at the limit, as does its threshold version (0.0 at 9.9%). Gemma 4’s collapse therefore comes with the threshold’s calibrated per-layer budgets or the low reads they settle at, rather than with per-token variation; this test cannot separate the two.

F

E VERY KEEP - RULE COMPARISON

Tables 10 and 12 list every paired comparison of a selector’s threshold and top-k versions, and Tables 11 and 13 repeat them at realized reads; Figure 5 summarizes the 43 at 13% and 18% budgets. Each row fixes one channel score and compares the two versions at the budget the threshold was calibrated for. When the threshold version’s realized reads differ from that budget by more than one percentage point, the top-k version is also rerun at those reads (Figure 8). In the 23 generation-task comparisons, the top-k version leads beyond two standard errors in 8 and the threshold version in none. Two TEAL comparisons at heavy sparsity, where both versions fall below 10% of the full model, are marked and not counted. We fixed this set of comparisons in advance, not by outcome: Prox’s score and its output-scaled variant on IFEval at 13% and on GPQA-Diamond at 13% for Gemma 4 and Qwen3-Thinking, TEAL’s score on IFEval at 18%, and GSM8K and BBH on OLMoE at both budgets. Table 18 covers the remaining pairs for Prox: over all nine model-benchmark pairs at 13% and 18%, its threshold version scores higher than its top-k version in 2 of 18, both on Qwen3.5 GPQA-Diamond and by at most 1.5 points. In the 20 likelihood-task comparisons on OLMoE at 13% and 18%, the top-k version leads in 2 and the threshold version in 3. At the 38% budget, Prox’s top-k version keeps 50% of inputs and its threshold version 30%, so the two differ in more than the keep rule. These 10 comparisons (top-k version ahead in 3, threshold version in none) appear in parentheses and are left out of every count. Table 14 pools each model’s counted comparisons. In the generation-task comparisons with Prox’s score and its output-scaled variant, both versions select inputs by top-k, so only the channel threshold is replaced. In the comparisons with TEAL’s score, the threshold on inputs is replaced by top-k. F.1

A PPLYING P ROX TO OLM O E

Prox (Liu et al., 2026) was evaluated on dense models, and five of its six tasks are multiple-choice likelihood tasks. We apply Prox to the routed experts of OLMoE-1B-7B-0924 and evaluate the same five tasks in lm-eval 0.4.3 (Gao et al., 2024) (Table 15). Each task runs under two prompt settings. In the first, as in Prox, the whole sequence is sparse. In the second, the full model computes the prompt except its last token, so only the positions that predict an answer token are sparse.

25

Table 10: Keep-rule comparisons on generation tasks. Each row fixes one model, task, score, and budget. It gives the accuracy of the threshold version (τ ) and the top-k version (k), their paired difference ± one standard error, and which version is ahead (= within two standard errors, – both collapsed). Model

Task

Score

Budget (%)

Gemma 4

GPQA-D

Prox, output-scaled Prox Prox, output-scaled Prox TEAL

IFEval

OLMoE

BBH

GSM8K

τ

k

τ −k

13 13 13 13 18

16.2 6.1 23.1 3.5 0.0

54.0 51.0 63.6 45.8 7.0

−37.9±3.8 −44.9±3.5 −40.5±2.3 −42.3±2.2 −7.0±1.1

k k k k –

Prox, output-scaled Prox TEAL Prox, output-scaled Prox TEAL Prox, output-scaled Prox TEAL Prox, output-scaled Prox TEAL

13 13 13 18 18 18 13 13 13 18 18 18

25.4 26.1 3.0 28.5 28.1 10.8 9.6 8.3 1.8 12.1 9.6 2.1

26.1 26.2 5.4 30.1 28.5 17.6 10.4 9.9 2.2 11.0 11.0 2.1

−0.7±0.6 −0.1±0.6 −2.4±0.3 −1.6±0.6 −0.4±0.5 −6.8±0.5 −0.8±1.0 −1.5±0.9 −0.4±0.4 +1.1±1.0 −1.4±0.9 +0.0±0.5

= = k k = k = = = = = =

Qwen3.5

IFEval

Prox, output-scaled Prox TEAL

13 13 18

91.1 91.5 3.0

91.9 90.9 10.4

−0.7±1.0 +0.6±1.1 −7.4±1.5

= = k

Qwen3-Thinking

GPQA-D

Prox, output-scaled Prox Prox, output-scaled Prox TEAL

13 13 13 13 18

62.6 57.1 85.4 84.3 0.0

59.6 58.1 85.6 85.4 0.0

+3.0±3.0 −1.0±3.0 −0.2±1.4 −1.1±1.6 +0.0±0.0

= = = = –

IFEval

We fit the thresholds with Prox’s sequential procedure. Layers are calibrated in order, with all earlier layers already sparse, and each layer gets one input threshold and one channel threshold from empirical quantiles of up to 200,000 activation samples pooled across its routed experts. Three settings differ from Prox’s dense setup. First, OLMoE is a base model, so we calibrate on 20,480 token positions from 30 Pile documents instead of Alpaca text. Second, we set the budgets (13, 18, and 38%) in charged sector reads instead of through Prox’s latency model. Third, the threshold version’s input threshold is fitted to keep 30% of inputs at all three budgets, while the top-k version keeps 30% of inputs at 13% and 18% and 50% at 38%. At realized reads and the 13% and 18% budgets, with the whole sequence sparse as in Prox, the threshold version leads in 1 of 10 comparisons and the top-k version in 2. With the prompt computed by the full model, the threshold version leads in 2 and the top-k version in none (Tables 12 and 13). F.2

R EPETITION LOOPS ON G EMMA 4

Prox selects in two stages: a threshold on input coordinates, then a threshold on expert channels. Section 4.3 showed that Gemma 4 falls into repetition loops under Prox’s thresholds even at nearly equal reads. To find which stage causes the collapse, we replaced one threshold at a time with top-k, at the 13% budget, on the first GPQA-Diamond sample of each question (realized reads in parentheses). With both thresholds, Prox scores 0.0 (10.9% of reads), and every answer hits the length limit. With top-k on inputs and the threshold on channels, it scores 6.1 (11.4%). With the threshold on inputs and top-k on channels, it scores 43.9 (12.4%), and with top-k at both stages, 51.0 (13.0%). On IFEval, the same four settings score 0.0, 3.5, 43.1, and 45.8. Replacing the channel threshold with top-k therefore raises GPQA-Diamond accuracy by 43.9 to 44.9 points while adding only 1.5 to 1.6 percentage points of reads, and raises IFEval accuracy by 42.3 to 43.1 points. Replacing the input threshold changes accuracy by at most 7.1 points on either 26

Table 11: The generation-task comparisons with each top-k version run at its threshold version’s realized reads. Columns follow Table 10. The top-k version is rerun only where the threshold version’s realized reads differ from the budget by more than one percentage point. † The threshold version read within one point of its budget, so the top-k run at the budget is reused. Model

Task

Score

Budget (%)

Gemma 4

GPQA-D

Prox, output-scaled Prox Prox, output-scaled Prox TEAL

IFEval

OLMoE

BBH

GSM8K

τ

k

τ −k

13 13 13 13 18

16.2 6.1 23.1 3.5 0.0

30.3 24.2 34.0 16.5 0.2

−14.1±3.5 −18.2±2.9 −10.9±2.1 −12.9±1.6 −0.2±0.2

k k k k –

Prox, output-scaled Prox TEAL Prox, output-scaled Prox TEAL Prox, output-scaled Prox TEAL Prox, output-scaled Prox TEAL

13 13 13 18 18 18 13 13 13 18 18 18

25.4 26.1 3.0 28.5 28.1 10.8 9.6 8.3 1.8 12.1 9.6 2.1

25.3 26.2† 3.0 28.0 28.5† 12.9 10.4† 9.9† 2.2† 11.8 11.0† 1.5

+0.1±0.6 −0.1±0.6 +0.1±0.2 +0.6±0.6 −0.4±0.5 −2.1±0.4 −0.8±1.0 −1.5±0.9 −0.4±0.4 +0.2±1.0 −1.4±0.9 +0.6±0.4

= = – = = k = = = = = =

Qwen3.5

IFEval

Prox, output-scaled Prox TEAL

13 13 18

91.1 91.5 3.0

91.9† 90.9† 0.0

−0.7±1.0 +0.6±1.1 +3.0±0.7

= = –

Qwen3-Thinking

GPQA-D

Prox, output-scaled Prox Prox, output-scaled Prox TEAL

13 13 13 13 18

62.6 57.1 85.4 84.3 0.0

57.1 +5.6±3.0 53.5 +3.5±3.3 85.2 +0.2±1.5 72.8 +11.5±1.9 0.0† +0.0±0.0

= = = τ –

IFEval

benchmark. So replacing the channel threshold, not the input threshold, is what avoids the collapse. Giving every token in a layer the same count, set to the average the threshold kept in that layer, does not avoid it either (Appendix E); that test cannot tell whether the calibrated layer budgets or the low reads they settle at drive the collapse. The collapse is not limited to 13%: with its threshold calibrated for 18%, Prox finishes its reasoning in only 132 of 792 GPQA-Diamond answers, against 787 of 792 for the full model. At the 13% budget, a per-token floor on channels does not prevent the collapse either. When every token keeps at least 4% of its candidate channels on top of the calibrated threshold, Prox still scores only 2.0 on GPQA-Diamond (11.1% of reads). Finally, when set to keep all channels, our implementations of Prox and TEAL match the full model on held-out chat, which rules out a gross implementation error in these baselines. F.3

O UR READER WITH A CALIBRATED THRESHOLD , AND CATS

Our reader. To test the keep-rule swap on our own reader, we replace its per-token top-k with one threshold per layer on the same score. The threshold is fitted on UltraChat with Prox’s recipe, so that it keeps k channels per token on average. On the benchmarks, this threshold version reads 0.4 to 5.8 percentage points less than its budget. Of 36 paired comparisons, across three post-trained models, three benchmarks, and four budgets, 34 are ties and none favors the threshold version. Both remaining comparisons favor the top-k version, on Gemma 4 at 12.8% of reads. There the threshold version reads 11.7% and scores 62.4 against 69.4 on GPQA-Diamond (−7.1 ± 1.7) and 73.2 against 80.4 on MMLU-Pro (−7.2 ± 1.2). It also hits the length limit on 15 to 17% of answers, against 4 to 6% for the top-k version.

27

Table 12: Keep-rule comparisons on Prox’s five likelihood tasks on OLMoE. Prox’s threshold and top-k versions, with the prompt either sparse or computed by the full model. Columns follow Table 10. At 38%, the top-k version keeps 50% of inputs and the threshold version 30%; these verdicts are in parentheses and not counted. Prompt

Task

Budget (%)

full model

ARC-C ARC-E HellaSwag MMLU PIQA

sparse

τ

k

τ −k

13 13 13 13 13

49.7 73.9 75.4 50.3 77.7

50.0 73.2 74.9 50.3 76.4

−0.3±0.9 +0.7±0.6 +0.6±0.3 −0.0±0.3 +1.3±0.7

= = τ = =

ARC-C ARC-E HellaSwag MMLU PIQA

18 18 18 18 18

51.9 75.8 77.2 50.4 78.6

51.4 75.8 76.6 50.6 77.5

+0.5±0.9 +0.0±0.6 +0.6±0.2 −0.2±0.3 +1.1±0.6

= = τ = =

ARC-C ARC-E HellaSwag MMLU PIQA

38 38 38 38 38

55.1 77.8 78.7 51.6 79.2

55.3 78.6 78.9 52.1 80.1

−0.2±0.7 −0.8±0.4 −0.2±0.2 −0.5±0.3 −0.9±0.5

(=) (=) (=) (=) (=)

ARC-C ARC-E HellaSwag MMLU PIQA

13 13 13 13 13

44.6 71.4 73.6 35.7 76.0

43.0 70.3 73.3 39.5 76.1

+1.6±1.0 +1.1±0.7 +0.3±0.3 −3.8±0.4 −0.2±0.8

= = = k =

ARC-C ARC-E HellaSwag MMLU PIQA

18 18 18 18 18

49.6 74.9 76.4 42.1 77.8

46.8 73.7 76.0 43.1 78.4

+2.8±1.0 +1.2±0.6 +0.4±0.3 −1.0±0.4 −0.6±0.7

τ = = k =

ARC-C ARC-E HellaSwag MMLU PIQA

38 38 38 38 38

55.2 77.4 78.8 49.5 79.0

54.4 77.9 79.4 50.7 80.1

+0.8±0.8 −0.5±0.5 −0.6±0.2 −1.1±0.3 −1.1±0.5

(=) (=) (k) (k) (k)

CATS. CATS computes the gate projection exactly and keeps channels by gate magnitude. We compare its threshold and top-k versions at two budgets, about 53% and 67% of reads, on all nine model-benchmark pairs. Of the 18 comparisons, 16 are ties. The threshold version leads in the other two, both on Gemma 4 at 53%: by 5.8 ± 1.7 points on GPQA-Diamond and 5.5 ± 1.1 points on MMLU-Pro, while reading slightly less (52.7% and 52.8%, against 53.3%). F.4

C ALIBRATED LAYER BUDGETS AND CALIBRATED RECOVERY

Layer budgets. We compare our reader’s uniform layer budget with non-uniform per-layer budgets, derived either from the weights or from calibration data, at the same total reads of 17.8%. Calibrated minus uniform, accuracy changes by −0.76 ± 1.10 on GPQA-Diamond (four samples) and +1.11 ± 1.01 on IFEval on Gemma 4, and by −1.77 ± 1.18 and −1.85 ± 1.31 on Qwen3-Thinking. On OLMoE, calibrated layer budgets lower next-token KL divergence by about 0.02 on held-out Pile text from the calibration distribution. Yet they change GSM8K accuracy by at most 0.1 points and BBH accuracy by −1.23 ± 0.50 and −0.28 ± 0.50 at the two smallest budgets. The KL gain is not confined to the calibration data. On Qwen3-Thinking with our reader at 17.8% of reads, we measure teacher-forced KL divergence on 144 windows from the full model’s GPQA-Diamond answers, with every layer sparse inside each window, fitting budgets on half of the answers and evaluating on the other half. Relative to the uniform budget, the chat-calibrated budgets lower KL divergence by 15.7 ± 1.9%, close to the 18.7 ± 2.1% of budgets fitted on GPQA-Diamond answers, while budgets

28

Table 13: The likelihood-task comparisons with each top-k version run at its threshold version’s realized reads. Columns follow Table 10. † The threshold version read within one point of its budget, so the top-k run at the budget is reused. Prompt full model

sparse

Task

Budget (%)

τ

k

τ −k †

ARC-C ARC-E HellaSwag MMLU PIQA

13 13 13 13 13

49.7 73.9 75.4 50.3 77.7

50.0 73.2† 74.9† 49.7 76.4†

−0.3±0.9 +0.7±0.6 +0.6±0.3 +0.5±0.3 +1.3±0.7

= = τ = =

ARC-C ARC-E HellaSwag MMLU PIQA

18 18 18 18 18

51.9 75.8 77.2 50.4 78.6

52.8 75.7 76.6† 50.6 78.0

−0.9±0.9 +0.1±0.5 +0.6±0.2 −0.2±0.3 +0.6±0.6

= = τ = =

ARC-C ARC-E HellaSwag MMLU PIQA

38 38 38 38 38

55.1 77.8 78.7 51.6 79.2

54.9 78.7 79.0 51.8 79.7

+0.2±0.8 −1.0±0.4 −0.2±0.2 −0.2±0.3 −0.4±0.5

(=) (k) (=) (=) (=)

ARC-C ARC-E HellaSwag MMLU PIQA

13 13 13 13 13

44.6 71.4 73.6 35.7 76.0

43.0† 70.3† 73.3† 39.5† 76.1†

+1.6±1.0 +1.1±0.7 +0.3±0.3 −3.8±0.4 −0.2±0.8

= = = k =

ARC-C ARC-E HellaSwag MMLU PIQA

18 18 18 18 18

49.6 74.9 76.4 42.1 77.8

46.8† 73.7† 76.0† 43.1† 78.4†

+2.8±1.0 +1.2±0.6 +0.4±0.3 −1.0±0.4 −0.6±0.7

τ = = k =

ARC-C ARC-E HellaSwag MMLU PIQA

38 38 38 38 38

55.2 77.4 78.8 49.5 79.0

54.9 77.9† 79.1 50.7† 80.3

+0.3±0.8 −0.5±0.5 −0.3±0.2 −1.1±0.3 −1.3±0.5

(=) (=) (=) (k) (k)

Table 14: No model favors the threshold version at the calibrated budget. Threshold-version minus top-k-version accuracy, pooled over each model’s counted comparisons with inverse-variance weights, at the calibrated budget and at realized reads. The ± is one SE that treats comparisons as independent; accounting for shared questions changes each SE by at most 0.1. The realized-read columns favor the threshold version (Section 2). Only Qwen3-Thinking leads beyond two SE there, a lead that comes from Prox on IFEval and reverses at the calibrated budget. At the calibrated budget Model Qwen3-Thinking Qwen3.5 OLMoE Gemma 4

Comparisons 4 3 32 4

τ −k −0.3±1.0 −1.7±0.7 −0.3±0.1 −41.5±1.3

At realized reads Comparisons 4 2 31 4

τ −k +4.5±1.1 −0.2±0.8 +0.0±0.1 −13.3±1.1

derived from the weights change it by +2.4 ± 1.7%. Lower KL divergence on reasoning text thus did not turn into a detected benchmark gain. Recovery. We also fit the low-rank recovery map on calibration text by minimizing next-token KL divergence from the full model, at the same reads as our weight-only map. We make 18 comparisons 29

Table 15: Prox’s five likelihood tasks on OLMoE. Few-shot examples, metric, number of questions, and full-model accuracy for each task. Task

Shots

Metric

Questions

Full model (%)

MMLU HellaSwag ARC-E ARC-C PIQA

5 10 0 25 0

accuracy normalized accuracy accuracy normalized accuracy accuracy

14,042 10,042 2,376 1,172 1,838

52.71 79.40 78.11 55.20 79.98

on the three post-trained models: GPQA-Diamond, IFEval, MMLU-Pro, and GSM8K at 17.8% of reads, plus IFEval and MMLU-Pro at 12.8%. The calibrated map never beats the weight-only map beyond two standard errors, and the weight-only map leads on Qwen3-Thinking MMLU-Pro at 12.8% (+2.1 ± 1.0). On OLMoE, the calibrated map lowers next-token KL divergence on held-out Pile text from its calibration distribution (0.158 against 0.200 at 17.8% of reads) but gives no benchmark gain beyond two standard errors.

G

W HAT EACH PART OF THE READER CONTRIBUTES

Score with the full sketch. When the reader reads the full ternary sketch (from 28.7% of reads upward), each 128-coordinate block also comes with the mean-squared quantization error ρb of its P ternary code. This gives each screened inner product an error variance σ 2 = b ρb ∥(Rx)b ∥2 (all norms are Euclidean). We model the true gate and up pre-activations as independent Gaussians centered at ĝj and v̂j , with variance σ 2 computed from each row’s own residuals. Five-node Gauss–Hermite quadrature then gives the mean µj and second moment mj of the gated activation aj = ϕ(gj⊤ x)(vj⊤ x). Let nj = ∥dj ∥, and let qj = U bj be channel j’s low-rank recovery vector, with ∥

∥

⊥ component βj = d⊤ j qj /nj along dj and orthogonal remainder of norm βj = ∥qj − βj dj /nj ∥. We score channel j by the expected squared output error of skipping it after recovery, h i ∥ sj = wj2 n2j (mj − µ2j ) + (nj µj − βj pj )2 + (βj⊥ pj )2 .

Output energy and KL divergence. Next-token KL divergence closely follows how much of the routed output’s energy the kept channels preserve, E = 1 − ∥y − yK ∥2 /∥y∥2 . Here y is the full routed-expert output and yK the output of the kept channels alone. We measure E on all 16 OLMoE layers, at 3,000 positions per layer, for four scores under per-token top-k, each at four read budgets without recovery. The four scores are our output-scaled ternary score, two versions of Prox’s score using each token’s largest 30% and 50% of input coordinates, and an INT4 screen of the gate projection. Across these 16 settings, E and held-out next-token KL divergence have a rank correlation of −0.96, and KL divergence scales roughly as (1 − E)1.2–1.9 . Ablations. Table 16 compares variants of our reader on 192 held-out OLMoE documents, by next-token KL divergence from the full model at equal total reads. Our reader uses the partial sketch at the two smallest budgets and the full sketch at higher budgets. We also replace our ternary sketch (1.0 GB next to the OLMoE checkpoint) with Prox’s 4-bit weight Table 16: Partial screening helps at small budgets, and recovery helps at every budget. Nexttoken KL divergence from the full model on held-out OLMoE documents at equal reads (lower is better). Each row is one variant of our reader. Reads (%)

13.5

18.5

28.5

38.4

Full sketch, no recovery Full sketch with recovery Partial sketch with recovery Prox’s 4-bit copy, output-scaled top-k, recovery

0.553 0.488 0.310 0.305

0.277 0.245 0.187 0.187

0.107 0.096 0.090 0.070

0.051 0.047 0.053 0.035

30

Table 17: Rotation and ranking channels across all routed experts lower KL divergence, and reading 80 of 128 coordinates is at or near the best. Next-token KL divergence from the full model on development data (lower is better). Top: at 17.8% of reads on OLMoE without recovery, the full sketch, two of its design variants, and two other scores under top-k. Bottom: coordinates the partial screen reads per 128-coordinate block, with recovery, at equal reads; bold marks the lowest in each row. Variant at 17.8% of reads (OLMoE, development)

KL

Full sketch, no recovery (rotated, top-k over all routed experts) Without rotation Fixed share of channels per expert

0.286 0.348 0.491

Score from an INT4 copy of the gate projection, same top-k Score from TEAL’s input magnitudes, top-k inputs

0.811 1.471

Coordinates read per 128-coordinate block Model

Budget (% of reads)

128

96

80

64

48

32

OLMoE

13 18 28

0.443 0.224 0.102

0.336 0.198 0.094

0.316 0.194 0.098

0.317 0.198 0.106

0.343 0.229 0.127

0.424 0.297 0.173

Gemma 4

13 18 28

0.235 0.124 0.056

0.188 0.110 0.052

0.176 0.106 0.052

0.177 0.111 0.057

0.197 0.129 0.070

0.260 0.179 0.103

copy (2.16 GB), keeping our output-scaled top-k and weight-only recovery. This variant also needs no calibration data. It matches our reader at the two smallest budgets and has lower KL divergence from 28.5% of reads upward, at about twice the resident auxiliary memory. Quantized checkpoints. Our reader also works on quantized checkpoints. On OLMoE held-out documents, with reads charged in BF16-equivalent routed bytes including quantization scales, our reader on a GPTQ (Frantar et al., 2023) 4-bit checkpoint reaches a next-token KL divergence of 0.045 at 18.9% of reads. AWQ (Lin et al., 2024) 3-bit weights alone reach 0.075 at 19.7% (paired difference −0.030 ± 0.003). At 21.5% of reads our reader reaches 0.033, against 0.019 for GPTQ 4-bit weights alone at 26.0%. On a quantized checkpoint, the ternary sketch takes a larger share of resident memory: a 4-bit checkpoint plus the sketch occupies about 34% of the BF16 routed bytes, against 20% for 3-bit weights. Design of the screen. Table 17 tests the screen’s design choices on development data. Removing the Hadamard rotation, or forcing each routed expert to keep a fixed share of channels, raises KL divergence on OLMoE. For the partial screen, we chose to read 80 of every 128 rotated coordinates per block on OLMoE development data. There, 80 gives the lowest KL divergence, or lies within about 0.004 of it, at each of three budgets. On Gemma 4 development data, 80 is also lowest at two budgets and within 0.0003 of the lowest at the third.

H

E VERY MODEL AND TASK

Figures 9 and 10 show accuracy against reads for every model and benchmark, with calibrated selectors at their calibrated budgets and at their realized reads. Table 18 lists the scores at heavy sparsity. In a rerun of GPQA-Diamond with eight samples per question, our reader at 17.8% of reads trails the full model by 3.9 ± 1.0 points on Gemma 4 and 4.6 ± 1.4 points on Qwen3-Thinking. Answer length. Our reader does not trade speed per token for longer answers. At 17.8% of reads, its mean answer length stays within about 15% of the full model’s on every model-benchmark pair. On Qwen3-Thinking, Gemma 4, and Qwen3.5, it changes by −3, +3, and +1% on GPQA-Diamond, +15, +9, and −12% on IFEval, and −14, +14, and −10% on MMLU-Pro. At most 1.3% of its answers stop at the length limit on any pair, and the same holds for the full model. 31

Our reader Prox score, top-k

Prox as published, calibrated WINA score, calibrated

GPQA-Diamond

IFEval

WINA as published, top-k TEAL as published, calibrated

MMLU-Pro

Qwen3-Thinking % of full model

100

50

0

Gemma 4 % of full model

100

50

0

Qwen3.5 % of full model

100

50

0 0

20

40

Reads or calibrated target (%)

60 0

20

40

Reads or calibrated target (%)

60 0

20

40

60

Reads or calibrated target (%)

Figure 9: Accuracy against reads on every model and task. Each panel is one model-benchmark pair, and its vertical axis runs from zero to above the full model’s accuracy (dotted line). Blue solid lines use no calibration data. Warm dashed lines are selectors with thresholds fitted on calibration data, placed at the budget they were calibrated for. WINA as published (red solid) keeps a top-k per token, with a sparsity level for each projection fitted on calibration data as in TEAL. GPQA-Diamond averages four samples per question, IFEval reports prompt-level strict accuracy, and MMLU-Pro uses a fixed subset of 1,000 questions.

32

Our reader Prox score, top-k

Prox as published, calibrated WINA score, calibrated

GPQA-Diamond

IFEval

WINA as published, top-k TEAL as published, calibrated

MMLU-Pro

Qwen3-Thinking % of full model

100

50

0

Gemma 4 % of full model

100

50

0

Qwen3.5 % of full model

100

50

0 0

20

40

60 0

Realized reads (%)

20

40

60 0

20

Realized reads (%)

40

60

Realized reads (%)

Figure 10: Figure 9 with calibrated selectors placed at the reads they actually used. Selectors with calibrated thresholds move left, because they read less at test time than the budget they were calibrated for.

Worst-pair accuracy (%)

Our reader Prox score, top-k

WINA as published, top-k

Prox as published, calibrated WINA score, calibrated

TEAL as published, calibrated

full model

100 75 50 25 0 0

5

10

15

20

25

30

35

40

Realized reads (%)

Figure 11: Figure 1 with calibrated selectors placed at the reads they actually used. Each curve is the worst accuracy over the nine model-benchmark pairs, as in Figure 1, but each calibrated selector sits at the mean reads it used on the pair that gives its worst accuracy, instead of the budget it was calibrated for. Selectors with calibrated thresholds read less than that budget (Section 4.1), so their points move left.

33

Table 18: Accuracy of every selector on every model and task. Scores in percent at heavy sparsity. Budget is the mean reads for selectors without calibration data and the calibrated budget for the others. GPQA-Diamond (GPQA-D) averages four samples per question. Method Qwen3-Thinking Full model Our reader

Prox score, top-k Prox as published, calibrated WINA as published, top-k TEAL as published, calibrated WINA score, calibrated Gemma 4 Full model Our reader

Prox score, top-k Prox as published, calibrated WINA as published, top-k TEAL as published, calibrated WINA score, calibrated Qwen3.5 Full model Our reader

Prox score, top-k Prox as published, calibrated WINA as published, top-k TEAL as published, calibrated WINA score, calibrated

Budget (%)

GPQA-D

IFEval

MMLU-Pro

100 12.8 17.8 28.7 38.7 13.5 18.5 28.5 13 18 28 20 30 20 30 20 30

70.8 63.8 67.3 67.9 70.8 63.8 64.0 69.3 30.8 59.7 67.2 8.6 58.6 0.0 35.5 0.0 17.8

87.8 85.0 87.2 85.4 86.1 85.8 86.3 86.3 43.3 82.8 86.5 0.9 79.5 0.0 50.5 0.0 33.3

80.5 76.9 78.2 78.6 80.3 74.9 76.3 79.3 64.9 76.3 76.9 28.4 73.8 0.2 61.7 0.0 40.0

100 12.8 17.8 28.7 38.7 13.5 18.5 28.5 13 18 28 20 30 20 30 20 30

79.9 69.4 76.6 77.9 80.2 56.3 73.7 77.5 0.0 11.7 72.3 14.6 69.6 0.0 22.5 0.0 25.8

93.5 88.0 92.6 94.5 94.5 57.1 90.8 93.9 0.0 11.3 91.7 40.9 94.3 0.0 38.3 0.0 27.2

85.1 80.4 82.5 84.2 84.4 67.3 82.0 83.9 0.0 16.7 81.9 21.8 78.8 0.0 28.6 0.0 27.9

100 12.8 17.8 28.7 38.7 13.5 18.5 28.5 13 18 28 20 30 20 30 20 30

83.6 81.3 83.3 82.7 84.5 79.9 81.6 83.3 80.6 83.1 83.3 15.4 70.1 10.7 72.7 4.8 58.2

92.1 90.0 91.9 92.4 91.9 91.3 91.5 91.7 89.3 91.1 91.7 37.0 88.5 17.4 88.7 4.6 81.7

85.5 83.7 85.2 84.9 86.2 83.0 84.4 85.5 82.8 84.1 85.7 52.2 83.3 49.1 83.6 29.0 81.0

34

I

BASELINE COMPARISON ON OLM O E

Table 19 compares our reader with nine prior channel selectors on OLMoE, by next-token KL divergence from the full model and top-1 token agreement on 192 held-out Pile documents (12,096 decoded positions), with the prompt processed by the full model. Calibrated baselines are fitted on disjoint Pile text for budgets near our reader’s and reported at the reads they realize. Prox divides each read budget between its input screen and the channels it computes; we report both Prox’s published division and the division with the lowest KL divergence on development data. At each budget, our reader has lower KL divergence than every baseline evaluated there, and it reads less than Prox’s nearest point at the 13% and 18% budgets. Among baselines evaluated at 13% to 18% of reads, every one except Prox has at least four times our reader’s KL divergence. Table 19: On OLMoE, our reader is closer to the full model than every baseline evaluated at each budget. Next-token KL divergence from the full model (lower is better) and top-1 agreement on 192 held-out Pile documents, at the reads each method realizes. HiRE-style screens channels with an INT4 copy of the gate projection under our top-k. R-Sparse adds a low-rank term for skipped inputs. CHESS, CATS, M-COUNTDOWN (the up-projection variant of COUNTDOWN (Cheon & Kang, 2025)), and Park et al. (2026) compute one projection in full before thresholding, so they read at least a third of each routed expert.

J

Budget

Method

Reads (%)

KL (↓)

Top-1 (↑)

∼13%

Our reader Prox, calibrated, division tuned on development data Prox as published, calibrated HiRE-style gate screen

12.8 13.2 13.7 12.8

0.334 0.488 0.462 1.709

0.751 0.709 0.723 0.469

∼18%

Our reader Prox, calibrated, division tuned on development data TEAL as published, calibrated WINA score, calibrated R-Sparse HiRE-style gate screen

17.8 18.3 18.5 17.6 18.1 17.8

0.199 0.298 1.109 1.295 1.132 0.860

0.808 0.774 0.613 0.583 0.577 0.602

∼28%

Our reader Prox as published, calibrated TEAL as published, calibrated WINA score, calibrated R-Sparse HiRE-style gate screen

28.5 29.1 29.1 28.5 28.8 27.8

0.096 0.136 0.378 0.461 0.445 0.376

0.859 0.842 0.760 0.731 0.717 0.720

∼38%

Our reader Prox, calibrated, division tuned on development data Prox as published, calibrated HiRE-style gate screen CHESS CATS M-COUNTDOWN Park et al. (2026)

38.4 38.7 38.3 37.8 40.4 40.5 40.9 40.4

0.047 0.057 0.060 0.203 0.643 1.113 1.123 1.120

0.900 0.891 0.892 0.783 0.663 0.555 0.612 0.557

S PEED DETAILS

Setup. We measure decoding speed for Qwen3-Thinking, Gemma 4, and Qwen3.5 at 17.8% of routed-expert reads, on a single RTX PRO 6000 Blackwell Max-Q GPU (300 W) with vLLM 0.29.0, PyTorch 2.13, and Triton 3.7.1. The full model runs vLLM’s fused-MoE Triton kernels. At each batch size, we take the faster of vLLM’s default kernel configuration and one tuned for this GPU and model. Our reader and the full model use the same unmodified router. Always-active modules. Gemma 4’s dense MLP and Qwen3.5’s shared expert are part of the timed block. Our implementation runs them on a second CUDA stream, alongside the router and the reader. Unmodified vLLM does not overlap Gemma 4’s dense MLP this way. If it did, Gemma 4’s MoE-block speedup would be 1.61 to 2.19× instead of 1.71 to 2.34×. 35

Table 20: Speedups peak at batch sizes four to sixteen, and the whole-model gain fades by batch 128. Decode times at 17.8% of routed-expert reads, in milliseconds per decoding step, median of three processes. The MoE block covers all MoE layers with the router, Gemma 4’s dense MLP, and Qwen3.5’s shared expert. Whole-model times use 128-token prompts. Batch size Qwen3-Thinking MoE block (ms)

Full model Our reader Speedup (×) Per output token (ms) Full model Our reader Speedup (×)

Gemma 4 MoE block (ms)

Full model Our reader Speedup (×) Per output token (ms) Full model Our reader Speedup (×)

Qwen3.5 MoE block (ms)

Full model Our reader Speedup (×) Per output token (ms) Full model Our reader Speedup (×)

1

2

4

8

16

32

64

128

3.084 1.619 1.90 6.253 4.605 1.36

5.525 8.517 11.969 18.370 2.604 3.562 5.131 9.260 not measured 2.12 2.39 2.33 1.98 9.454 12.450 15.666 22.044 27.366 34.107 38.196 6.609 7.453 8.957 12.850 19.831 28.925 38.363 1.43 1.67 1.75 1.72 1.38 1.18 1.00

3.646 2.136 1.71 6.459 4.915 1.31

5.399 8.010 10.750 15.244 2.704 3.428 4.846 7.591 not measured 2.00 2.34 2.22 2.01 8.260 11.060 14.005 19.122 23.250 29.890 36.345 5.691 6.515 8.058 11.109 16.385 24.408 33.937 1.45 1.70 1.74 1.72 1.42 1.22 1.07

2.163 1.350 1.60 5.862 5.092 1.15

3.511 5.503 9.132 15.085 2.049 2.193 3.394 5.760 not measured 1.71 2.51 2.69 2.62 8.141 10.174 14.127 20.397 29.029 40.668 53.605 6.680 6.943 8.540 10.837 16.680 30.812 43.353 1.22 1.47 1.65 1.88 1.74 1.32 1.24

Strength of the baseline. Unmodified vLLM is a strong baseline. At batch one, its fused-MoE expert kernels stream weights at 1.35 to 1.44 TB/s, which is 83% to 88% of the 1.63 TB/s read bandwidth we measured as achievable on this GPU. MoE block. To time the MoE blocks alone, we capture the layer inputs from seven real decoding steps on 128-token chat prompts and replay all MoE layers, routing included, in one CUDA graph. After four warmup replays, each process times seven repetitions with CUDA events and keeps the median. We report the median over three fresh processes. Whole model. For whole-model timing, vLLM generates 64 output tokens from 128-token prompts at batch sizes 1 to 128, with full CUDA graphs, one warmup run, and three timed repetitions in each of three processes. Time per output token is the wall-clock time between the first and last generated tokens divided by 63, and we report the median across processes. This time includes attention, the language-model head, sampling, and vLLM’s scheduler, but not prompt prefill. Same outputs. The timed Triton kernels compute the same function as the evaluated reader, up to the order of floating-point operations. On held-out chat documents at equal reads on Qwen3-Thinking, the two implementations’ next-token KL divergences from the full model differ by −0.0011 ± 0.0013 (90% confidence interval). On all three models, the kernels keep the same channels as the evaluated reader at equal reads: 0 of 25,606 checked selection decisions differ. Long contexts. In single-process runs at 8,192 to 32,768 context tokens and batch sizes one and four, whole-model decoding is 1.20 to 1.54× faster on Qwen3-Thinking, 1.16 to 1.48× on Gemma 4, and 1.08 to 1.40× on Qwen3.5. We leave out batch four at 32,768 tokens on Gemma 4 and Qwen3.5, where neither version fits in memory. The gain shrinks at longer contexts, because attention takes longer while the MoE work per token stays the same. Where the time goes. Traces of whole decoding steps at batch one show that each Qwen3-Thinking MoE layer takes 64.7 µs in the full model and 34.4 µs with our reader, router included. Within our

36

Full model, optimized vLLM

Our reader (17.8% of reads)

Output tokens/s per GPU

Qwen3-Thinking

Gemma 4

2,000

Qwen3.5

128

128 64

64 32

2,000

16 8

1,000

1

200 100

Tokens/s per user

200

64

2,000

16 8

32

16 8

1,000 4

500

2

50

32

1,000 4

500

128

4 500

2

2 1

200 50

100

Tokens/s per user

200

1

200 20

50

100

200

Tokens/s per user

Figure 12: Our reader raises per-GPU throughput at the same per-user speed. Whole-model decoding of three post-trained models at 17.8% of routed-expert reads, against optimized vLLM, from Table 20 (128-token prompts, one RTX PRO 6000 Blackwell GPU). Each point is one batch size, labeled for our reader. Per-user speed is the inverse of the time per output token, and per-GPU throughput is the batch’s total output rate. reader, screening takes 9.5 µs, top-k selection 5.0 µs, and computing the kept gate, up, and down rows 16.4 µs. Projecting the skipped channels onto the rank-32 recovery map takes 8.0 µs, partly overlapped on a second CUDA stream. Attention takes 48 µs per layer in both, so the whole model speeds up less than the MoE block. Larger batches. Tokens in a batch together select more channels, so the gain shrinks at large batches. Whole-model speedup peaks at batch 8 or 16 (1.75× on Qwen3-Thinking, 1.74× on Gemma 4, and 1.88× on Qwen3.5) and falls to 1.00×, 1.07×, and 1.24× at batch 128 (Table 20). Serving trade-off. Figure 12 shows the trade-off between each user’s speed and the GPU’s total throughput. At batch one, our reader generates 196 to 217 tokens per second per user. The full model reaches this speed at no batch size; at batch one it generates 155 to 171 tokens per second. At the same per-user speed, between 80 and 120 tokens per second, one GPU generates 3.3 to 4.4 times as many tokens per second in total with our reader (3.3 to 4.3 at 100), interpolating log-linearly between measured batch sizes. The full model must run smaller batches to stay that fast. At longer contexts, attention takes a larger share of each step, and this advantage shrinks.

37

Related documents

• Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data#1468990
• BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding#1131064
• AFORE: Attention-FFN Disaggregation with Overlapped Reconfiguration of Experts#1187881
• AFORE: Attention-FFN Disaggregation with Overlapped Reconfiguration of Experts#1181606
• Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing#414116