FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink?

FASTAFD: W HEN D OES ATTENTION -FFN D ISAGGREGATION PAY ON R ACK -S CALE NVL INK ?

arXiv:2611.07149v1 [cs.DC] 4 Nov 2026

Yichao Fu * 1 Yuxuan Zhang * 1 Minshen Zhang * 1 Ruitian Wang 1 Junda Chen 1 Hao Zhang 1

A BSTRACT Attention–FFN disaggregation (AFD) separates attention from the experts of a mixture-of-experts (MoE) model and has been reported to raise decode throughput over colocated serving. Whether this advantage holds on rackscale NVLink systems such as GB200 NVL72 remains unclear: prior results were obtained on other hardware, against narrower colocated baselines, or with analytical models, whereas production engines serve large MoE models on these systems with wide expert parallelism (wide EP). We build and carefully tune FastAFD, an AFD inference engine for GB200 NVL72 that overlaps the MoE of one microbatch with the attention of the other, and compare it against the best vLLM wide EP setting (with the EP width tuned per workload). Our experiments show that AFD has a throughput advantage of up to 1.17× over wide EP on Qwen3 and 1.16× on Qwen3.5, and matches it on MiniMax-M2.5. We derive an analytical model that predicts AFD throughput within 3% using only profiles of colocated vLLM, which tells whether switching to AFD would improve throughput before AFD is deployed. Beyond prediction, we turn the model into a concise insight: AFD’s speedup factors into a batch gain (larger batches, since attention GPUs hold no expert weights), an overlap gain (shorter steps, since the MoE runs on the expert GPUs behind attention), and a disaggregation cost (the GPUs given to experts). AFD has the advantage when the two gains outweigh the cost. This view explains which models benefit from AFD, and shows that speculative decoding helps light-expert models most, where the enlarged overlap gain outweighs the larger disaggregation cost and adds a further 8% to AFD’s advantage over wide EP. Looking ahead, serving the experts on accelerators that keep their weights in on-chip SRAM, such as the Groq LPX accelerators that NVIDIA pairs with Vera Rubin NVL72, would remove most of the disaggregation cost, raising the ideal speedup to 2.48× with speculative decoding. FastAFD is open source at https://github.com/hao-ai-lab/FastAFD. 1

I NTRODUCTION

Attention–FFN disaggregation (AFD) has drawn sharply divided conclusions as a design for large-scale mixtureof-experts (MoE) (Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2022) LLM serving: some works report substantial decode-throughput gains over colocated deployment (Zhu et al., 2025; StepFun et al., 2025), while others argue that it cannot match a well-tuned wide expert-parallel (wide EP) baseline (Qiu et al., 2026). Which design serves MoE models better on rack-scale NVLink systems remains unclear. On one hand, AFD separates attention from experts: attention GPUs hold the KV caches and they run larger decode batches, while expert GPUs gather tokens from attention GPUs into one batch, therefore the MoE GEMMs run closer to their roofline. MegaScale-Infer reports up to 1.90× higher per-GPU decode throughput than colocated serving baselines (Zhu et al., *

Equal contribution 1 University of California, San Diego. Correspondence to: Hao Zhang <haozhang@ucsd.edu>. Preprint.

2025), Step-3 co-designs its model around AFD (StepFun et al., 2025), NPU superpods deploy it at scale (Xiao et al., 2025), NVIDIA builds AFD into its Vera Rubin platform by running MoE experts on Groq LPX accelerators (Aubrey & Ghodsian, 2026), and vLLM ships an experimental AFD plugin (AFD Plugin Contributors, 2026). On the other hand, wide EP (Rajbhandari et al., 2022; DeepSeek-AI et al., 2024) keeps attention and experts on every GPU but shards the experts across a rack-scale NVLink domain such as GB200 NVL72 (NVIDIA, 2026a), which frees much of the same HBM for KV caches; vLLM and SGLang serve large MoE models this way (vLLM Team, 2025; Meta and NVIDIA Team, 2026; SGLang Team, 2025b), and analytical studies of modern hardware lean the same way: a communicationroofline analysis finds AFD promising only for specific hardware-model combinations (Liu et al., 2026a), a designspace exploration on a 128-GPU B200 cluster finds aggregated serving keeps the throughput lead (Wu et al., 2026), and a provisioning model that sweeps all EP widths finds AFD faster in only 7 of 36 settings (Qiu et al., 2026). The measured evidence for AFD, however, comes from older hardware against narrow colocated baselines, and the evi-

(1) Batch gain (memory) Wide EP

Others

Experts

KV cache

AFD

Others

+ KV cache

KV cache

(2) Overlap gain (latency) Wide EP

attention

wide-EP step MoE

attention

MoE

AFD step

AFD attn attn mb0 attn mb1 attn mb0 attn mb1 AFD FFN

MoE mb0 MoE mb1 MoE mb0

saved (MoE hidden)

In summary, this paper mainly makes three contributions:

(3) Disaggregation cost Wide EP AFD

Looking ahead, heterogeneous FFN placement on dedicated SRAM accelerators such as Groq LPX paired with Vera Rubin NVL72 (Aubrey & Ghodsian, 2026) (Section 6) removes the disaggregation cost from the GPU budget, raising AFD’s best speedup over wide EP from 1.17× with homogeneous AFD to an ideal 2.48× with heterogeneous placement and speculative decoding.

every node serves requests

• We build and carefully tune FastAFD, the first AFD inference engine for GB200 NVL72, and benchmark it against the best wide EP baselines, which it consistently matches or outperforms with up to a 1.17× throughput advantage (Sections 3 and 5).

cost: the FFN node serves none

Figure 1. AFD’s speedup over wide EP balances two gains against one cost (Eq. 1). (1) Batch gain: moving expert weights off attention GPUs frees HBM for larger KV caches and decode batches. (2) Overlap gain: two-microbatch ping-pong hides each microbatch’s MoE behind the other’s attention stage. (3) Disaggregation cost: attention nodes hold KV caches and serve requests; the FFN node does not, so one node in every attention:FFN group produces no tokens.

• We derive an analytical model that predicts FastAFD’s throughput within 3% mean error. It needs only a profile of vLLM on the target cluster to tell how much switching to AFD would speed up a deployment, without actually deploying AFD (Section 4).

dence against it is analytical and unvalidated on a rack-scale NVLink system. As a result, there is no measured basis for choosing between the two designs on GB200 NVL72, a platform on which large MoE models are now served (SGLang Team, 2025b; Meta and NVIDIA Team, 2026). To close this gap, we build FastAFD, an AFD inference engine for GB200 NVL72 whose kernels, microbatching, and coordination are each tuned for the rack (Section 5), and benchmark it against vLLM wide EP with the EP width tuned per workload, on three MoE models. We show empirically that AFD outperforms this baseline by up to 1.17× (Qwen3-235B at 16K context). Based on this empirical study, we further derive an analytical model that predicts AFD throughput using only profiles of the colocated vLLM deployment; its predictions stay within 3% of the measured throughput on average. Beyond prediction, we turn the analytical model into a concise key insight: AFD’s speedup over wide EP is the product of two gains and one cost (Figure 1): Sall = batch gain × overlap gain × disaggregation cost. (1) (1) The batch gain comes from attention GPUs that hold no expert weights and thus fit a larger KV cache; (2) the overlap gain comes from taking the MoE off the attention critical path; and (3) the disaggregation cost comes from the expert GPUs, which serve no requests. AFD has the advantage when the two gains outweigh the cost. This insight also predicts when speculative decoding helps AFD. On MiniMax-M2.5, whose light experts yield a small overlap gain, AFD only ties wide EP. Speculative decoding multiplies the MoE work on this model, which enlarges the overlap gain, while the disaggregation cost remains small and therefore lifts advantage by 8%.

• We turn the model into a concise insight with two gains (batch and overlap) and one disaggregation cost. The insight guides AFD tuning decisions: which models benefit from AFD, when speculative decoding helps, and how much headroom heterogeneous FFN placement unlocks (Sections 4 and 6).

2

BACKGROUND AND M OTIVATION

2.1

Colocated Wide EP and Attention–FFN Disaggregation

MoE decoding couples stateful attention with stateless expert feed-forward networks (FFNs). During each decode step, attention reads every active request’s growing key– value (KV) cache from fixed per-GPU HBM, whereas expert FFNs hold no per-request state and stream shared expert weights across requests. In colocated serving (Kwon et al., 2023; Zheng et al., 2024), every GPU stores a full copy of the attention weights alongside a shard of the routed experts. Single-node expert parallelism (e.g., EP4) keeps one quarter of all experts on each GPU, leaving limited HBM for KV caches and capping the per-GPU decode batch. On a 72-GPU GB200 NVL72 rack (NVIDIA, 2026a), colocated wide EP (e.g., EP8–EP64) (Rajbhandari et al., 2022; DeepSeek-AI et al., 2024; vLLM Team, 2025; SGLang Team, 2025b) shards experts across 8–64 GPUs while keeping attention data-parallel. Shrinking each GPU’s expert shard frees HBM for larger decode batches, while each step still executes attention, NVLink all-to-all routing, and expert FFN serially on the same GPU. Rather than colocating both modules, AFD (Zhu et al., 2025; StepFun et al., 2025) separates MoE inference into an atten-

(b) Intra-step ping-pong pipeline ( = microbatches)

(a) FastAFD architecture on GB200 NVL72 Attention pool 4 GPUs / node paged KV cache

each attention GPU RMSNorm + QKV proj

Attention kernel

attention stage toprouter

O proj + add & norm

Attention stage

Attention pool 4 GPUs / node

M2N

return

SwiGLU/down

each FFN GPU

mb0 attn

mb1 attn

N2M

M2N

Layer ℓ+

Layer ℓ+

mb0 attn

mb0 attn

mb1 attn

mb1 attn

N2M

fused MegaMoE loop

FFN pool 4 GPUs / node all FP8 experts

NVLink · 1 round trip / layer

FFN stage

Layer ℓ

FFN pool

mb0

mb1

mb0

mb1

mb0

mb1

recv/pack

4 GPUs / node

MoE

MoE

MoE

MoE

MoE

MoE

FFN stage

Arrow color = microbatch (orange mb0, slate mb1)

FP8 gate/up

Layer ℓ

Layer ℓ+

Layer ℓ+ time

Figure 2. FastAFD architecture and intra-step ping-pong overlap on GB200 NVL72. (a) Attention nodes (blue) hold paged KV caches and run the attention stage (norms, QKV/O projections, TensorRT-LLM FMHA attention, and top-K routing), while one NVLink round trip per layer (N2M/M2N) dispatches routed tokens to a dedicated FFN node (coral) running a single persistent fused MegaMoE kernel. (b) Splitting each attention GPU’s batch into two ping-pong microbatches (mb0, mb1) overlaps the attention stage on the attention pool with the FFN stage (MegaMoE) on the FFN pool across consecutive layers.

tion pool and a dedicated FFN pool provisioned at a chosen A:F node ratio (Figure 2a). Because attention GPUs store no expert weights, they dedicate all free HBM to KV caches and maximize their per-GPU decode batch. Within a single batch, however, layer ℓ+1’s attention depends sequentially on the return of layer ℓ’s remote experts (N2M dispatch → experts → M2N combine), which would serialize the two pools. To overlap them, AFD splits each attention GPU’s batch into two ping-pong microbatches (mb0 and mb1 in Figure 2b): in each layer, while the FFN pool runs mb0’s FFN stage, the attention pool concurrently runs mb1’s attention stage (norms, projections, core attention, and routing), and the two microbatches then swap roles. 2.2

Motivation: Why Re-evaluate AFD on Rack-Scale NVLink?

NVL72 strengthens the baseline as much as AFD. With 1.8 TB/s NVLink to each of 72 GPUs, remote expert dispatch costs no more than an intra-node hop, which enables rack-scale AFD and the heterogeneous racks built around it (Aubrey & Ghodsian, 2026). The same fabric lets colocated serving shard experts across the rack (EP8–EP64) and reclaim most of the HBM that AFD frees, so the TP and EP4 baselines of prior AFD studies (Shoeybi et al., 2019; Zhu et al., 2025) no longer represent the best colocated deployment. The comparison must also charge AFD for its FFN nodes, which hold no KV cache and serve no requests. The existing answers for NVL72 are models, and they disagree. The two studies that address rack-scale NVLink directly are analytical: one finds AFD likely to benefit only on superpod-class interconnects and models with coarsegrained experts (Liu et al., 2026a), the other finds homogeneous AFD ahead in only 7 of 36 settings (Qiu et al., 2026). Neither is checked against a rack-scale measurement, and MegaScale-Infer’s Ta ≈ Te rule picks AFD’s ratio but has no wide-EP counterpart (Zhu et al., 2025). Speculative decoding is standard in production and

changes the question. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023; Li et al., 2024) is supported by every major serving engine and accelerates decoding across production workloads (Liu et al., 2026b); frontier MoE models ship multi-token-prediction heads for it (DeepSeek-AI et al., 2024). Verifying q draft tokens per step reuses each request’s KV-cache read but multiplies routed expert work by q: under wide EP that work lands on every GPU’s critical path, under AFD on an FFN pool that standard decoding leaves partly idle. No study has measured AFD under speculation. We answer all three on hardware (Section 3.2), with an engine built to add no slack of its own (Section 2.3) and a model that covers both designs (Section 4). 2.3

FastAFD Overview

Such a runtime is harder to build on GB200 NVL72 than on earlier clusters. Because 7.9 TB/s HBM3e finishes a layer’s KV reads and expert-weight streaming in a few hundred microseconds, anything that touches the host inside a decode step is visible in the measurement. Prior AFD systems were built for multi-node clusters connected over InfiniBand or RoCE, where a layer takes on the order of a millisecond, and they keep the CPU in the per-layer communication loop: MegaScale-Infer’s M2N library issues RDMA writes and polls completions from host threads (Zhu et al., 2025), and Step-3’s StepMesh drives GPUDirect RDMA from CPU network threads so that communication consumes no GPU SMs (StepFun et al., 2025). On NVL72 the same host round trip is a visible fraction of the step. The runtime must also be comparable: any difference in attention kernels or numeric precision between the two engines would be mistaken for an architectural difference. Five design choices follow (Figure 2a). (1) A MoEboundary split with matched kernels makes the comparison isolate architecture rather than kernels: built on Mini-SGLang (SGLang Team, 2025a), FastAFD keeps

norms, QKV/O projections, and top-K routing on attention GPUs and uses colocated vLLM’s exact attention kernels (TensorRT-LLM FMHA (NVIDIA, 2026b)) and block-FP8 expert precision (DeepSeek-AI et al., 2024). The other four remove the four places where a decode step would otherwise wait: on the host within a layer, on a launch between kernels, on communication between pools, and on the host between steps. (2) Within a layer, GPU-initiated, one-sided communication replaces host posts and polls: attention GPUs publish FP8 tokens and routing metadata into pre-allocated symmetric NVLink buffers, at the cost of 24 SMs on each attention GPU for the fused send/receive kernel. (3) Between dispatch, GEMM, and combine, a persistent fused MegaMoE kernel on FFN GPUs replaces the separate launches of modular pipelines (Zhao et al., 2025b;a), fusing the NVLink TMA pull (N2M), block-FP8 grouped GEMMs, activation, and push-back (M2N) into one launch per layer. (4) Across the two pools, two ping-pong microbatches (k = 2) hide remote MegaMoE execution behind local attention: NVL72’s bandwidth covers the round trip with two microbatches, whereas slower interconnects require k ≥ 3 (Zhu et al., 2025) and pay for it in extra dense-projection passes and smaller GEMM tiles. (5) Between steps, whole-step CUDA graph capture with a one-step-ahead CPU coordinator replaces synchronous host bookkeeping: because (2) leaves no host synchronization inside the step, the whole step replays as one graph, and the host prepares Step t+1 (batch composition, padding, buffer slots, KV-cache frees) while Step t runs. Each choice is ablated, alongside alternative stage boundaries (Zhuang et al., 2026), in Section 5.2. Because only attention GPUs hold KV caches and emit tokens, our evaluation reports both all-GPU speedup (charging all allocated attention and FFN GPUs) and attention-GPU speedup (counting attention GPUs only, which reflects per-GPU gain when the FFN pool is placed off-GPU).

3

E ND - TO -E ND P ERFORMANCE ON NVL72

3.1

Experiment Setup

We evaluate inference decoding on a single GB200 NVL72 rack comprising 18 four-GPU nodes within one 72-GPU NVLink domain (NVIDIA, 2026a). We use three production open-weight FP8 MoE models (MiniMax, 2026; Qwen Team, 2025; 2026) chosen to cover the three architectural axes that govern AFD’s trade-off: light-expert full attention (MiniMax-M2.5, 10B active of 229B across 256 experts), heavy-expert full attention (Qwen3-235B (Yang et al., 2025), 22B active of 235B across 128 experts), and ultra-sparse hybrid linear-plus-full attention (Qwen3.5-397B, 17B active of 397B across 512 experts; full specifications in Appendix A). Workloads and protocol. Each request carries a fixed prompt of 4K, 8K, or 16K tokens (Ctx in Tables 1 and 2). We prefill once, then measure steady-state decode through-

put in closed loop with every slot occupied, over a window short enough that per-request KV length stays close to the nominal context; prefill is excluded, since AFD is a decode-phase design and prefill is served by prefill–decode disaggregation. Both engines reserve 2 GB of HBM for activations and communication buffers and give all remaining capacity to the BF16 paged KV cache, running the largest decode batch that fits it; per-GPU batch is therefore set by KV capacity, and because a colocated vLLM GPU holds 1/EP of the expert weights beside its KV cache while a FastAFD attention GPU holds none, the difference in free HBM is the batch gain bA /bV of Section 4.1. Configurations. For each workload, colocated vLLM (Kwon et al., 2023) scales expert parallelism across nodes from single-node EP4 through EP8, EP16, EP32, and EP64 (TP1, DP=EP), while FastAFD dedicates one four-GPU node to the FFN pool and scales the attention-to-FFN node ratio r = NA /NF over ratios from 1:1 to 17:1 until per-GPU decode throughput peaks and the next wider step turns down, or until reaching the 18-node rack limit. Each engine is reported at its highest-throughput configuration per workload. Speculative decoding. We evaluate draft depth d = 3 (q = 4 verified tokens per step) on MiniMax-M2.5 and Qwen3-235B. Draft tokens are simulated with acceptance fixed at ā ≈ 1.8 accepted tokens per step for both engines; since both engines verify the same q tokens with the same acceptance, the comparison depends on how each serves the multiplied expert work, not on draft quality. Metrics. Our primary metric, all-GPU speedup (Sall ), divides total decode throughput by every allocated GPU (EP GPUs for vLLM, 4(r+1) for FastAFD), charging FastAFD for its dedicated FFN nodes; we also report SEP4 against single-node EP4. We pair Sall with attention-GPU speedup (Satt = r+1 r Sall ), which counts only attention GPUs and measures the per-GPU gain when the FFN pool is placed on non-GPU accelerators. To rule out kernel discrepancies, both engines use identical TensorRT-LLM FMHA attention kernels (NVIDIA, 2026b) and DeepSeek-V3 block-FP8 expert precision (DeepSeek-AI et al., 2024). 3.2

Results

In standard decoding across 4K–16K contexts, FastAFD achieves 1.30–2.36× higher all-GPU throughput than single-node EP4 (Figure 3a, Table 1). However, scaling colocated vLLM across the rack boosts baseline throughput by 22–114%, peaking at interior widths EP16–EP32 rather than EP64 on every workload (Figure 3c) and compressing FastAFD’s per-GPU batch advantage to 1.06–1.16× on the full-attention models (Figure 3b). Against the best wide-EP configuration at each context, FastAFD’s allGPU speedup Sall narrows to 0.99–1.17× (Figure 3a; Ap-

3×

SEP4 (vs. vLLM EP4)

2×

Sall (vs. best wide EP, all GPUs)

(c) Colocated vLLM scaling

Satt (attention GPUs)

1.40× → 1.00×

Throughput / EP4 (×)

Speedup

(a) FastAFD speedup over vLLM EP4 and over best wide EP

1× 0×

Batch ratio

(b) Per-GPU decode batch ratio 6× 4× 2× 0×

bA /bEP4 (vs. vLLM EP4)

bA /bV (vs. best wide EP)

1.06–1.16× vs. wide EP

4K

8K

16K

4K

MiniMax-M2.5

8K

16K

Qwen3-235B

4K

8K

16K

4K 8K 16K 1.6 MiniMax-M2.5 1.4 1.2 1.0 1.4 1.2 1.0 0.8 2.5 2.0 1.5 1.0

Qwen3-235B 4K EP64: −25.4% vs. peak

Qwen3.5-397B

EP4

Qwen3.5-397B

peak

EP8

EP16

EP32

EP64

vLLM expert-parallel width

Figure 3. Context-tuned wide EP narrows homogeneous AFD gains from 1.30–2.36× to 0.99–1.17× across 4K–16K contexts. (a) FastAFD speedups over EP4 (SEP4 , slate) and over best vLLM wide EP counting all GPUs (Sall , coral) or attention GPUs only (Satt , teal; stems show the disaggregation cost). (b) Per-GPU batch ratios (bA /bEP4 vs. bA /bV ). (c) Colocated vLLM scaling across EP4–EP64 normalized to EP4 (outlined markers: per-curve optima). Table 1. Standard decoding (d = 0): context-tuned wide EP erases most of FastAFD’s gain over EP4. Each engine runs at its measured best configuration: the vLLM EP width (Wide EP) and the FastAFD attention:FFN node ratio (A:F). SEP4 and Sall compare per-GPU decode throughput over EP4 and the best wide EP, counting every allocated GPU; Satt counts attention GPUs only. ▲/▼: largest/smallest in each column; †: still rising at the 18-node rack limit (a lower bound). Best configuration Model

Table 2. Speculative decoding (d = 3) at least halves FastAFD’s best A:F ratio and raises Satt on every workload. Acceptance is held near ā ≈ 1.8 tokens per step. A:F shows the best ratio at d = 0 → d = 3; Wide EP is the best vLLM width at d = 3. Speedups as in Table 1. Best configuration Ctx

Wide EP

A:F

Sall

Satt

MiniMax-M2.5

4K 8K 16K

EP16 EP16 EP16

9:1 → 4:1 15:1 → 5:1 17:1 → 8:1

1.08 1.02▼ 1.08

1.34 1.22 1.22▼

Qwen3

4K 8K 16K

EP8 EP16 EP8

5:1 → 2:1 7:1 → 3:1 11:1 → 5:1

1.13▲ 1.10 1.12

1.70▲ 1.46 1.34

Speedup (×)

A:F SEP4

Sall

Satt

Ctx

Wide EP

MiniMax-M2.5 4K 8K 16K

EP32 EP32 EP32

9:1 15:1 17:1†

Qwen3

4K 8K 16K

EP16 EP16 EP32

5:1 1.30▼ 1.06 7:1 1.41 1.10 11:1 1.42 1.17▲

Qwen3.5

4K 8K 16K

EP32 EP32 EP32

5:1 2.25 1.16 1.39▲ 5:1 2.36▲ 1.10 1.33 5:1 2.11 0.99▼ 1.19

1.40 1.52 1.56

1.00 1.12 1.02 1.09 1.02 1.08▼ 1.28 1.26 1.28

pendix A). Specifically, MiniMax-M2.5 sits at parity with 1.00–1.02× speedup, whereas Qwen3 retains 1.06–1.17× all-GPU gains and Qwen3.5 falls from 1.16× at 4K to parity (0.99×) at 16K. Before charging dedicated FFN GPUs, however, attention-GPU speedup Satt reaches 1.08–1.39× across all workloads. Under speculative decoding at d = 3 with acceptance controlled near ā ≈ 1.8 tokens per step, multi-token verification accelerates both engines across 4K–16K contexts (Figure 4a; Appendix D). At the same time, verifying four tokens per step contracts FastAFD’s optimal A:F node ratio by more than half on every workload, from 9:1–17:1 down to 4:1–8:1 on MiniMax-M2.5 and from 5:1–11:1 down to 2:1–5:1 on Qwen3 (Figure 4c, Table 2). Despite dedicating a larger GPU share to FFN nodes, FastAFD lifts Sall on MiniMax-M2.5 from standard-decoding parity up to 1.08× while holding Qwen3 at 1.10–1.13×, and raises Satt across all workloads to 1.22–1.70× (Figure 4b).

Speedup (×)

Model

4

U NDERSTANDING THE P ERFORMANCE

4.1

Speedup Decomposition

FastAFD’s all-GPU speedup over colocated wide EP is the product of two gains and one cost (Eq. 1). Let r = NA /NF be the number of attention nodes per FFN node, bA and bV the per-GPU decode batches, and TA and TV the decode step times of FastAFD and colocated wide EP. Per-GPU throughput is batch over step time, and only r of every r+1 nodes emit tokens, so Sall =

bA TV r × × . bV TA r+1

(2)

The batch gain bA /bV comes from removing expert weights from attention GPUs, the overlap gain TV /TA from taking the MoE off the critical path, and the disaggregation cost r/(r+1) from FFN GPUs that hold no KV cache and serve no requests. The two gains together form the attentionGPU speedup of Section 3.1, Satt = (bA /bV )(TV /TA ), r so Sall = r+1 Satt : the disaggregation cost is exactly the gap between what each attention GPU gains and what the rack gains. Equation 2 holds in both standard and speculative decoding when the two engines reach the same mean acceptance.

1.8

FastAFD

1.54× (16K)

1.6 1.4 1.2 1.0 4K

8K

16K

MiniMax-M2.5

4K

8K

Qwen3

16K

(b) FastAFD vs. wide EP (d:0 → 3) Satt (d = 3)

Sall (d = 3)

Satt: 1.28 → 1.70×

1.7 1.5 1.3 1.1 0.9

Sall: 1.00 → 1.08×

4K

8K

16K

4K

MiniMax-M2.5

(c) Optimal A:F contraction (d:0 → 3) d=0

d=0

8K

Optimal A:F node ratio

Wide EP

FastAFD / wide EP speedup (×)

Self-speedup: d=3 / d=0 (×)

(a) Self-speedup (d=3 vs. d=0)

d=3 (MiniMax-M2.5)

d=3 (Qwen3)

17:1

18:1

15:1

14:1 10:1 6:1

11:1 9:1 8:1 4:1

5:1

16K

5:1

2:1

2:1

Qwen3

7:1 5:1

4K

8K

16K

MiniMax-M2.5

4K

3:1

8K

16K

Qwen3

Figure 4. Speculative decoding (d = 3, q = 4, ā ≈ 1.8) raises Satt on every workload and keeps Sall at 1.02–1.13× while contracting optimal A:F ratios (4K–16K). (a) Same-engine speculative self-speedup over d = 0 (FastAFD up to 1.54×). (b) FastAFD speedup over tuned wide EP (d = 0 circles vs. d = 3 diamonds): hiding 4× MoE work on the FFN pool raises Satt on all six workloads (up to 1.70×) and lifts MiniMax-M2.5 Sall to 1.08× at 4K and 16K. (c) Optimal A:F ratios contract (d: 0 → 3) as the FFN pool takes rnq routed tokens.

Figure 5a shows how each term plays out on the three models. Batch gain. Removing expert weights from the attention GPUs helps only where wide EP has not already freed that memory. On MiniMax-M2.5 and Qwen3, wide EP has done most of the work, so the batch grows only modestly. Qwen3.5 is the exception: its batch grows by 1.4–1.6×, because a FastAFD attention GPU keeps more HBM for KV cache and linear-attention state than a vLLM GPU, which also holds expert weights and reserves more memory for its runtime, and stores each request more compactly than vLLM’s shared hybrid-model pages. Overlap gain. Taking the MoE off the critical path pays only when the MoE is heavy. Qwen3, with heavy experts in all 94 layers, gains the most. MiniMax-M2.5’s light experts leave little to hide, so its step barely changes. Qwen3.5’s step is even slower than wide EP’s, because its attention-side projections run twice per step and its short linear-attention layers cannot always hide the FFN. Disaggregation cost. The cost shrinks as one FFN node serves more attention nodes. On the full-attention models, longer contexts allow this, so their advantage grows with context. Qwen3.5’s linear-attention layers keep its ratio low at every context, so its cost stays high, and its advantage fades as its gains shrink. Together, the three terms give each model its regime. MiniMax-M2.5 sits near parity because neither gain repays even a small cost. Qwen3 has a batch gain similar to MiniMax-M2.5’s but adds a real overlap gain, which is enough to win. Qwen3.5 leads on batch alone, despite a slower step, until its advantage shrinks to parity at 16K. 4.2

Per-Layer Overlap Model

While HBM capacity fixes the batch gain, the overlap gain depends on how two-microbatch pipelining overlaps the

attention stage and the FFN stage across the L model layers. Splitting each attention GPU’s batch into two ping-pong microbatches overlaps one microbatch’s attention stage Ta,ℓ in layer ℓ with the peer microbatch’s FFN stage Te,ℓ (r) across all r attention nodes, giving the predicted step time TbA (r) = Tdraft + 2

L X

 max Ta,ℓ , Te,ℓ (r) .

(3)

ℓ=1

Here Tdraft is the draft-and-accept time outside the layer loop, zero in standard decoding; the per-layer terms take the same form in both modes (closed-form hardware rooflines in Appendix C). In the attention stage, Ta,ℓ combines core attention KV-cache reads with layer norms, QKV/O projections, and top-K routing on the attention GPU, which run once per microbatch and therefore twice per step even when the FFN stage is hidden (Figure 5b). In the FFN stage, Te,ℓ (r) combines per-layer NVLink dispatch and expertweight reads with grouped-GEMM compute over the routed tokens aggregated from r attention nodes, growing monotonically with r. Under speculative decoding at draft depth d, verifying q = d + 1 candidate tokens per request reuses the same prefix KV-cache read on attention GPUs—leaving Ta,ℓ nearly unchanged—while routing q times as many tokens to the FFN pool and steepening Te,ℓ (r). Taking the maximum per layer before summing is essential on hybrid models such as Qwen3.5, where idle slack on full-attention layers cannot cancel FFN exposure on sequential linear-attention layers. 4.3

Coverage Boundary and Win Condition

In Equation 3, the FFN stage in layer ℓ stays hidden behind the peer microbatch’s attention stage whenever Te,ℓ (r) ≤ Ta,ℓ . Below this coverage boundary, remote MoE adds no exposed latency to the critical path, so adding attention nodes per FFN node reduces the disaggregation cost without stretching the step; once the FFN stage exceeds the attention stage (crossing 100% in Figure 5c), uncovered FFN

Pool r/(r+1)

Satt

Sall

2.0× 1.5× 1.0× 0.5× MiniMax-M2.5

Qwen3-235B

Qwen3.5-397B

(b) Step time (MoE hidden, 2× dense) Attention Dense Ops (2× in AFD)

Coloc. MoE Exposed FFN

100% 75% 50% 25% 0% MiniMax-M2.5 Qwen3 EP4 | AFD EP4 | AFD

Qwen3.5 EP4 | AFD

FFN / attention stage (%)

Ratio vs. best wide EP

Step TV /TA

Share of step time at 8K (%)

(a) Sources of speedup vs. wide EP Batch bA /bV

(c) Hiding boundary across models MiniMax-M2.5

125%

Qwen3

Qwen3.5 (GD)

Exposed FFN (>100%)

100% 75% Covered (<100%)

50% 1:1

5:1

9:1

13:1

17:1

A:F node ratio r (8K)

Figure 5. Three-factor throughput decomposition, step breakdown, and coverage boundary. (a) Equation 1 factors across 4K–16K standard decoding: batch gain bA /bV , overlap gain TV /TA , disaggregation cost r/(r+1), attention-GPU speedup Satt , and all-GPU speedup Sall . (b) Normalized 8K step share: FastAFD hides colocated MoE while running attention-stage projections twice per step. (c) Ratio of remote FFN latency to the attention stage at 8K, crossing 100% at each model’s measured peak (15:1, 7:1, and 5:1).

execution lengthens the step directly. Combining this overlap bound with Equation 1 defines the win condition for FastAFD to beat colocated wide EP on all-GPU throughput: bA TV 1 · > 1+ , bV TA (r) r

TV 1 > 1 + for bA ≈ bV . TA (r) r (4) In the general form, the batch gain and overlap gain together must outweigh the disaggregation cost. When rack-scale wide EP already reclaims most of the KV-cache memory gap so per-GPU batches nearly match, the second form applies: the overlap gain alone must exceed the disaggregation cost, requiring the colocated MoE time removed from the critical path to pay for both the duplicated second pass of attentionstage projections and any uncovered FFN time. or

Together, the coverage boundary and the two forms of Equation 4 explain the behavior across models and decoding modes (Section 6 analyzes the resulting provisioning trajectories). In standard decoding, longer contexts shrink the per-GPU batch and reduce the tokens routed per attention node while KV-cache scans keep the attention stage long on full-attention models, allowing one FFN node to cover progressively more attention nodes. Because wide EP already narrows their batch gap, these full-attention models follow the matched-batch condition: heavy-expert Qwen3 achieves a large enough overlap gain to clear the disaggregation cost, whereas light-expert MiniMax-M2.5 barely breaks even. On hybrid Qwen3.5, by contrast, linear-attention layers shrink their attention stage alongside the batch and cap the covered ratio early; although duplicated projections make its step slower than wide EP, its 1.4–1.6× larger batch satisfies the general form of Equation 4 through batch gain at 4K and 8K and reaches only parity at 16K. Under speculative decoding (Leviathan et al., 2023; Chen et al., 2023; Miao et al., 2024; Cai et al., 2024; Li et al., 2024; Liu et al., 2026b), verifying multiple draft tokens per request reuses prefix KV reads on attention GPUs while multiplying routed tokens on FFN GPUs, so the FFN stage saturates the attention stage at less than half the node ratio (Figure 4c). At the same time,

hiding this multiplied expert work behind the attention stage amplifies the overlap gain, lifting light-expert MiniMaxM2.5 from parity to a clear all-GPU win at 4K and 16K while trading a higher disaggregation cost for a much larger attention-GPU speedup on heavy-expert Qwen3. 4.4

Model Validation and Ratio Selection

We validate Equation 3 against all 88 measured FastAFD deployments across standard and speculative decoding (complete residuals in Appendix B). All model inputs come from hardware specifications and a single colocated wide-EP profile per workload, with zero parameters fitted to FastAFD runs (Appendix C). Across all 65 deployments up to each workload’s optimal ratio, the model predicts FastAFD throughput with a mean error of 2.9%—specifically 3.0% in standard decoding and 2.6% in speculative decoding— and never exceeds 7.4% on any run (Figure 6c). At 16K (Figures 6a and 6b), the model accurately reproduces both the standard-decoding throughput curves across all three architectures and the leftward shift of the optimal ratio under speculative decoding. Because the model predicts throughput across candidate ratios without running FastAFD, an operator can pick the deployment ratio directly from Equation 3 instead of running a multi-node grid search. Deploying the model-selected ratio retains 95–100% of the measured peak throughput in standard decoding and 97–100% in speculative decoding, matching the exact measured optimum on 7 of 9 standard workloads and landing one step away on the rest. The coverage condition Te,ℓ (r) ≤ Ta,ℓ explains why one-step misses at high ratios are benign: while the ratio stays within the coverage bound, step latency remains flat and the disaggregation cost barely changes between adjacent ratios such as 13:1, 15:1, and 17:1. At low ratios or under speculative decoding where fourfold expert load sharpens the peak, testing the model-selected ratio and the candidates within two steps of it finds the exact optimum.

Qwen3

4 3 2 1 1:1

5:1

9:1

13:1

(c) Error across 4K–16K (65 runs)

(b) Speculative decoding at 16K (d = 3) Qwen3.5

k tok/s per GPU

k tok/s per GPU

MiniMax-M2.5

Model

Peak r *

4 3 2 1

17:1

A:F node ratio (r:1)

1:1

4:1

8:1

12:1

16:1

A:F node ratio (r:1)

̂ Prediction error |X/X − 1| (%)

(a) Standard decoding at 16K (d = 0)

IQR & median

Run

Mean (2.9%)

8 6 4 2 0 MM-M2.5 Qwen3 Qwen3.5 MM-M2.5 Qwen3

Standard (d = 0)

Speculative (d = 3)

Figure 6. Zero-refit model validation across candidate A:F ratios. (a) At 16K standard decoding (d = 0), predicted throughput (dashed) against measured throughput (solid) across candidate ratios. (b) At 16K speculative decoding (d = 3), the model captures the b leftward shift of the optimal ratio. (c) Relative error |X/X − 1| across all 65 deployments up to each workload’s optimal ratio (2.9% mean error, dashed line).

To achieve the theoretical overlap of Equation 3 without software or communication bubbles, FastAFD co-designs its operator boundary and remote expert kernel around GB200 NVL72’s high HBM3e and NVLink bandwidth. In an r:1 deployment, the attention pool comprises r four-GPU nodes running data-parallel attention ranks, and the FFN pool comprises one four-GPU node running expert parallelism across all routed experts. FastAFD instead adapts DeepGEMM into a split MegaMoE architecture: attention GPUs dispatch topK indices and token counts, while each FFN GPU runs a single persistent MegaMoE kernel per layer that pulls routed FP8 activations over NVLink TMA, executes blockFP8 grouped GEMMs and SwiGLU activation, and pushes BF16 outputs back without host synchronization. FastAFD pairs this fused kernel with two-level overlap within and across decode steps. Within a single microbatch, each layer’s post-MoE residual and the next layer’s QKV projection depend sequentially on the expert output returning from the FFN pool. To overlap the two pools, FastAFD splits each attention GPU’s decode batch into k = 2 ping-pong microbatches captured inside a single whole-step CUDA graph: while the FFN pool executes one microbatch’s MegaMoE kernel, the attention pool concurrently executes the peer microbatch’s attention stage (attention and projections) (Figure 2b). Across decode steps, a ∼ 30 ms step would stall at step boundaries if GPUs waited synchronously for the CPU to process sampled tokens and update paged KV-cache tables. FastAFD eliminates this inter-step gap with an asynchronous one-step-ahead coordinator: while GPUs execute the current step’s CUDA graph, the CPU processes the previous step’s sampled tokens, prepares the next step’s KV-cache slots and metadata descriptors, and enqueues the next launch before the current step finishes on the device.

−50%

−44%

MiniMax-M2.5

(b) Microbatch count Step (ms)

75 50 25 0

DeepEP+DeepGEMM MegaMoE (fused)

+13.9 ms 29.9 ms

20 0

k=2

3:1 (4 nodes) 7:1 (8 nodes)

1

(c) Cross-step overlap 40

40 35 30 25

Qwen3

2 3 4 Microbatches (k)

(d) FastAFD vs. tuned CAD tok/s/GPU

System Implementation

(a) MoE backend (8K, 7:1) Step (ms)

5.1

S YSTEM I MPLEMENTATION AND A BLATION

Step (ms)

5

4500

FastAFD Tuned CAD 1.28–1.48×

3000 1500 0

Overlap on

Overlap off

4K

8K

16K

Figure 7. Runtime ablations at 8K and MoE-boundary vs. tuned CAD across 4K–16K. (a) Fused MegaMoE vs. DeepEP+DeepGEMM at 7:1. (b) Microbatch count k on MiniMaxM2.5 at 3:1 and 7:1. (c) One-step-ahead coordinator on vs. off at 7:1. (d) FastAFD vs. peak-tuned CAD on MiniMax-M2.5 across 4K–16K.

5.2

Ablation Study

Figure 7 evaluates the contribution of each runtime mechanism at 8K context (complete wall-clock measurements in Appendix E). First, replacing modular DeepEP+DeepGEMM with the fused MegaMoE kernel cuts step latency by 50% on Qwen3 and by 44% on MiniMaxM2.5 at a 7:1 ratio (Figure 7a), while avoiding over 12 GiB of extra buffer memory per attention GPU that causes DeepEP+DeepGEMM to run out of memory at full batch on Qwen3. Second, sweeping the microbatch count from k = 1 to k = 4 on MiniMax-M2.5 confirms that k = 2 achieves the lowest step latency at both 3:1 and 7:1 (Figure 7b): k = 1 serializes attention and FFN execution, whereas k = 3 and k = 4 repeat the attention GPUs’ dense projections k times per step and shrink per-microbatch GEMM dimensions. Third, disabling the one-step-ahead coordinator at 7:1 adds 14 ms of synchronous host overhead per step, reducing end-to-end throughput by 32% (Figure 7c). Finally, we compare FastAFD’s MoE-boundary split against an inference variant of Core Attention Disaggre-

MiniMax-M2.5

Qwen3

18-node NVL72 limit (17:1) 17:1

17:1

15:1

13:1 9:1 5:1

Qwen3.5

11:1

9:1 7:1 5:1 5:1

1:1 4K

8K

16K

Speedup vs. best wide EP (×)

Optimal A:F node ratio (r * )

(a) Homogeneous provisioning (d = 0)

(b) Standard decoding (d = 0) Sall: all GPUs (measured)

2.5 1.76×

1.69×

2.48×

2.0

1.75×

1.5

1.26×

1.0

Context length

Ideal: no FFN cost (modeled)

2.5

2.0 1.5

(c) Speculative decoding (d = 3)

Satt: FFN off the GPU budget (measured)

1.0 4K 8K 16K MiniMax-M2.5

4K

8K 16K Qwen3

4K

8K 16K Qwen3.5

4K 8K 16K MiniMax-M2.5

4K

8K 16K Qwen3

Figure 8. Rack-scale A:F provisioning and heterogeneous FFN placement across 4K–16K. (a) Optimal homogeneous node ratios r∗ = NA /NF in standard decoding (d = 0): full-attention MoEs fan out from 9:1 → 17:1 (MiniMax-M2.5) and 5:1 → 11:1 (Qwen3), whereas hybrid Qwen3.5 holds 5:1 across 4K–16K. (b, c) Measured speedup over best wide EP in standard (d = 0) and speculative (d = 3) decoding when counting all GPUs (Sall , coral) or attention GPUs only (Satt , solid teal; up to 1.39× and 1.70×), plus the modeled zero-FFN-cost ceiling (hatched; up to 1.76× and 2.48×).

gation (CAD), which DistCA proposed for long-context training (Zhuang et al., 2026); the variant moves QKV and output projections to the FFN pool while keeping only core attention and KV-cache writes on the attention GPUs (Appendix E). On HBM3e, bare core attention is too brief to cover remote execution, and moving the projections to the FFN pool places an NVLink round trip on the attention critical path while burdening the shared FFN node with dense projections from all r attention nodes. Across peak-tuned deployments on MiniMax-M2.5 from 4K to 16K (Figure 7d), CAD saturates its FFN node early at 2:1–4:1 ratios—far below FastAFD’s 9:1–17:1—and achieves only 0.68–0.78× of FastAFD’s per-GPU throughput, giving FastAFD a 1.28–1.48× advantage.

Enabling speculative decoding fundamentally shifts rackscale provisioning toward a larger FFN pool. Because tree verification processes q = d+1 candidate tokens per request while amortizing KV-cache reads across the draft tokens, it multiplies the routed FFN token volume per step by q without proportionally lengthening attention. As predicted by the coverage condition Te,ℓ (r) ≤ Ta,ℓ in Equation 3 and confirmed in Figure 4c, enabling speculative decoding (d = 3, q = 4) contracts the optimal A:F ratio by more than half across 4K–16K—shifting r∗ from 9:1–17:1 down to 4:1–8:1 on MiniMax-M2.5 and from 5:1–11:1 down to 2:1–5:1 on Qwen3—so operators must allocate 11–33% of the rack to FFN nodes rather than 6–17% across 4K–16K in standard decoding.

6

P ROVISIONING AND O UTLOOK

6.2

6.1

Rack-Scale Provisioning

Our speedup decomposition in Equation 1 points to a natural architectural evolution for next-generation rack-scale systems such as NVIDIA Vera Rubin NVL72 paired with Groq 3 LPX accelerators (Aubrey & Ghodsian, 2026): placing the stateless FFN pool on dedicated high-bandwidth SRAM accelerators while keeping stateful attention and KV caches on HBM-rich GPUs. In a homogeneous GPU deployment, every node assigned to the FFN pool imposes a disaggregation cost of r/(r + 1) on the GPU budget that holds KV cache and generates tokens. Placing the FFN pool on non-GPU accelerators removes this disaggregation cost, so per-attention-GPU throughput (Satt , solid teal bars in Figures 8b and 8c) directly becomes the per-GPU system speedup. Even with today’s NVLink communication and overlap schedules unchanged, eliminating the disaggregation cost lifts end-to-end GPU speedups from 0.99–1.17× to 1.08–1.39× under standard decoding (Figure 8b) and from 1.02–1.13× to 1.22–1.70× under speculative decoding (Figure 8c). Notably, heterogeneous FFN placement yields a much larger jump under speculative decoding precisely because verification requires a heavier FFN pool (11–33% of homogeneous GPUs at d = 3), so moving FFN

Figure 8a tracks how the optimal Attention-to-FFN node ratio r∗ = NA /NF evolves across context lengths from 4K to 16K under standard decoding. On full-attention MoEs (MiniMax-M2.5 and Qwen3), growing KV-cache footprints steadily increase attention time per token while shrinking the per-node HBM batch ceiling, driving r∗ upward from 9:1 to 17:1 on MiniMax-M2.5 and from 5:1 to 11:1 on Qwen3. At 8K–16K on these two models, a single FFN node serves 7–17 attention nodes—consuming only 6–13% of the GPU fleet—and sits on a broad plateau where neighboring integer ratios remain within 0.5% of peak throughput, allowing a single static split to cover wide workload shifts. In contrast, the hybrid Qwen3.5 model maintains a flat 5:1 ratio across 4K–16K: its 45 Gated-Delta linear-attention layers replace KV-cache scans with fixed-size recurrent state updates that scale with batch size rather than sequence length, keeping total attention time nearly constant across 4K–16K and delaying ratio fan-out until contexts well beyond 16K where its 15 full-attention layers eventually dominate HBM capacity and latency.

Heterogeneous FFN Placement

off the GPU budget recovers a far larger disaggregation cost. A second gain arises when dedicated FFN accelerators execute expert weights from high-bandwidth on-chip SRAM, compressing remote FFN latency toward zero. In that regime, attention nodes no longer need two-microbatch ping-pong pipelining to hide remote FFN execution, eliminating both the second pass of dense projections on attention GPUs and any uncovered FFN tail latency (closed-form bound in Appendix C). Under this ideal zero-FFN-cost ceiling (hatched bars in Figures 8b and 8c), per-GPU speedups reach 1.20–1.76× in standard decoding and 1.38–2.48× in speculative decoding across 4K–16K, turning even near-parity MiniMax-M2.5 into a 1.20–1.26× standard-decoding and 1.38–1.75× speculative-decoding win. Together, these bounds show that while homogeneous GPU disaggregation is bounded by the disaggregation cost and two-microbatch pipelining overhead, heterogeneous Attention-GPU + FFN-SRAM disaggregation unlocks both halves of that headroom simultaneously.

7

R ELATED W ORK

Disaggregated LLM serving operates at several orthogonal granularities. Request-phase disaggregation and chunked-prefill schedulers (Zhong et al., 2024; Patel et al., 2024; Qin et al., 2024; Agrawal et al., 2024) separate compute-bound prefill from memory-bound decoding across distinct node pools; FastAFD operates strictly within the decode phase and can be deployed inside the decode pool of a prefill–decode disaggregated cluster. Within the decode phase, attention offloading places KV caches and attention on memory-optimized devices (Chen et al., 2024) or across a cluster’s pooled GPU memory (Lin et al., 2024) while keeping dense and MoE projections together, whereas CAD, proposed for long-context training (Zhuang et al., 2026), runs only core attention on a separate device pool. In contrast, MoE-boundary AFD systems (Zhu et al., 2025; StepFun et al., 2025; Liu et al., 2025; Xiao et al., 2025; AFD Plugin Contributors, 2026) split each layer specifically at the routed expert interface; our ablation in Section 5 demonstrates that keeping dense projections in the attention stage is essential on HBM3e racks to cover remote FFN execution and avoid extra NVLink round trips. Finally, while recent rack-scale AFD studies evaluate homogeneous and heterogeneous trade-offs using analytical communication rooflines or offline simulators (Liu et al., 2026a; Qiu et al., 2026), we provide the first end-to-end hardware audit on GB200 NVL72 against context-tuned wide EP, paired with a zero-refit per-layer overlap model validated across 65 multi-node deployments. Our analysis also connects heterogeneous FFN placement with speculative decoding. Earlier heterogeneous MoE systems such as Fiddler (Kamahori et al., 2025) and

MoE-Lightning (Cao et al., 2025) run large MoE models on memory-constrained GPUs by keeping expert weights in CPU memory and coordinating CPU and GPU execution. Emerging heterogeneous racks such as NVIDIA Vera Rubin NVL72 paired with Groq 3 LPX accelerators (Aubrey & Ghodsian, 2026) instead run latency-sensitive FFN and MoE execution on the accelerators while GPUs keep attention and KV caches, and SQD (Tschand et al., 2026) splits decode by attention type, placing dense attention layers apart from subquadratic attention layers plus FFN. Similarly, prior speculative decoding studies (Leviathan et al., 2023; Chen et al., 2023; Miao et al., 2024; Cai et al., 2024; Li et al., 2024; Liu et al., 2026b) analyze multi-token verification under colocated execution, where shared prefix KV reads and multiplied FFN tokens execute on the same GPUs. We show that in a disaggregated rack, verifying q candidate tokens per request directly contracts the coverage boundary Te,ℓ (r) ≤ Ta,ℓ in Equation 3 and shifts cluster provisioning toward a heavier FFN pool—which limits homogeneous all-GPU gains on heavy-expert models, but makes heterogeneous FFN placement substantially more rewarding under speculative decoding than under standard decoding.

8

C ONCLUSION

We asked when attention–FFN disaggregation pays on rackscale NVLink and answered it by measurement. With FastAFD, an AFD engine for GB200 NVL72 that leaves no runtime slack between the measured gap and the architecture, we find that AFD ranges from parity to 1.17× over per-workload-tuned wide EP with every FFN GPU counted: a provisioning decision, not an architectural win. We trace the outcome to a batch gain, an overlap gain, and a disaggregation cost, and show that a per-layer model calibrated from colocated profiles alone predicts it within 3%, so an operator can decide before deploying AFD. Speculative decoding and off-GPU expert accelerators are the two levers that tilt the balance toward AFD.

ACKNOWLEDGEMENTS We thank NVIDIA for supporting our development on GB200 NVL72 and for providing access to the rack used for all experiments in this paper. We thank the Qwen team and MiniMax for open-sourcing Qwen3, Qwen3.5, and MiniMax-M2.5, and Yibo Zhu and Yonghao Zhuang for insightful feedback. FastAFD builds on the open-source SGLang, vLLM, DeepEP, and DeepGEMM projects.

R EFERENCES AFD Plugin Contributors. Announcing vLLM AFD Plugin: Disaggregating Attention and FFN for Flexible MoE Serving, 2026. URL https://vllm.ai/blog/2 026-07-23-vllm-afd-plugin. vLLM Blog, July 23, 2026. Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B., Tumanov, A., and Ramjee, R. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024. URL https://www.usenix.org/conference/ osdi24/presentation/agrawal. Aubrey, K. and Ghodsian, F. Inside NVIDIA Groq 3 LPX: The Low-Latency Inference Accelerator for the NVIDIA Vera Rubin Platform, 2026. URL https://develo per.nvidia.com/blog/inside-nvidia-g roq-3-lpx-the-low-latency-inference -accelerator-for-the-nvidia-vera-rub in-platform/. NVIDIA Technical Blog, March 16, 2026. Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. In Proceedings of the 41st International Conference on Machine Learning (ICML), pp. 5209–5235, 2024. URL https://proceedings.mlr.press/v235/c ai24b.html. Cao, S., Liu, S., Griggs, T., Schafhalter, P., Liu, X., Sheng, Y., Gonzalez, J. E., Zaharia, M., and Stoica, I. MoELightning: High-Throughput MoE Inference on Memoryconstrained GPUs. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pp. 715–730, 2025. doi: 10.1145/3669940.3707267. URL https://doi.org/10.1145/3669940. 3707267. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv preprint arXiv:2302.01318, 2023. URL https://arxiv.or g/abs/2302.01318.

Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Wang, J., Chen, J., Chen, J., Yuan, J., Qiu, J., Li, J., Song, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L., Wang, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Wang, Q., Zhu, Q., Chen, Q., Du, Q., Chen, R. J., Jin, R. L., Ge, R., Zhang, R., Pan, R., Wang, R., Xu, R., Zhang, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Pan, S., Wang, T., Yun, T., Pei, T., Sun, T., Xiao, W. L., Zeng, W., Zhao, W., An, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Zhang, X., Chen, X., Nie, X., Sun, X., Wang, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yu, X., Song, X., Shan, X., Zhou, X., Yang, X., Li, X., Su, X., Lin, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhu, Y. X., Zhang, Y., Xu, Y., Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Yu, Y., Zheng, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Tang, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Wu, Y., Ou, Y., Zhu, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Zha, Y., Xiong, Y., Ma, Y., Yan, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Wu, Z. F., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Huang, Z., Zhang, Z., Xie, Z., Zhang, Z., Hao, Z., Gou, Z., Ma, Z., Yan, Z., Shao, Z., Xu, Z., Wu, Z., Zhang, Z., Li, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Gao, Z., and Pan, Z. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437, 2024. URL https://arxiv.org/abs/2412.19437. Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 23, 2022. URL https://jmlr.org/papers/v2 3/21-0998.html. Kamahori, K., Tang, T., Gu, Y., Zhu, K., and Kasikci, B. Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models. In International Conference on Learning Representations (ICLR), 2025. URL http s://arxiv.org/abs/2402.07033.

Chen, S., Xiao, W., Lin, Y., Zhang, M., Shan, Y., Jiang, J., Chen, K., and Wu, Y. Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation. arXiv preprint arXiv:2405.01814, 2024. URL https://arxiv.org/abs/2405.01814.

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23), pp. 611–626, 2023. URL https://arxiv.org/ab s/2309.06180.

DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D.,

Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scal-

ing Giant Models with Conditional Computation and Automatic Sharding. In International Conference on Learning Representations (ICLR), 2021. URL https: //arxiv.org/abs/2006.16668. Leviathan, Y., Kalman, M., and Matias, Y. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML), pp. 19274–19286, 2023. URL https://arxiv.org/abs/2211.17192. Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. In Proceedings of the 41st International Conference on Machine Learning (ICML), pp. 28935–28948, 2024. URL https://proceedings.mlr.press/v235/l i24bt.html. Lin, B., Zhang, C., Peng, T., Zhao, H., Xiao, W., Sun, M., Liu, A., Zhang, Z., Li, L., Qiu, X., Li, S., Ji, Z., Xie, T., Li, Y., and Lin, W. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache. arXiv preprint arXiv:2401.02669, 2024. URL https://arxiv.org/abs/2401.02669. Liu, G., Li, H., Guo, Y., Lyu, Y., Zhou, M., Liu, Y., Li, Z., and Wang, Y. Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems. arXiv preprint arXiv:2602.09721, 2026a. URL https://arxiv.org/abs/2602.09721. Liu, X., Yu, J., Park, J., Stoica, I., and Cheung, A. Speculative Decoding: Performance or Illusion? In Proceedings of Machine Learning and Systems, volume 8, 2026b. URL https://proceedings.mlsys.org/paper_ files/paper/2026/hash/554e056fe2b6d9 fd27ffcd3367ae1267-Abstract-Conferenc e.html. Liu, Z., Tian, B., Wang, G., Jiang, Z., Sun, P., Han, Z., Tang, T., Hu, X., Jia, Y., Zhang, Y., Liu, H., Zhang, M., Zhang, Y., Chen, Q., Cheng, S., Gao, M., You, Y., and Feng, S. Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving. arXiv preprint arXiv:2509.17863, 2025. URL https://ar xiv.org/abs/2509.17863. Meta and NVIDIA Team. Driving vLLM WideEP and Large-Scale Serving Toward Maturity on Blackwell (Part I), 2026. URL https://vllm.ai/blog/2026-0 2-03-dsr1-gb200-part1. vLLM Blog, February 3, 2026. Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., Shi, C., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. SpecInfer: Accelerating Large Language Model Serving

with Tree-based Speculative Inference and Verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pp. 932–949, 2024. doi: 10.1145/3620666.3651335. URL https: //doi.org/10.1145/3620666.3651335. MiniMax. MiniMax-M2.5 model card, 2026. URL https: //huggingface.co/MiniMaxAI/MiniMax-M2. 5. NVIDIA. NVIDIA GB200 NVL72, 2026a. URL https: //www.nvidia.com/en-us/data-center/gb 200-nvl72/. Product page, accessed October 2026. NVIDIA. TensorRT-LLM, 2026b. URL https://gi thub.com/NVIDIA/TensorRT-LLM. Repository, accessed October 2026. Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., and Bianchini, R. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118–132, 2024. doi: 10.1109/ISCA59077.2024.00019. URL https://do i.org/10.1109/ISCA59077.2024.00019. Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. arXiv preprint arXiv:2407.00079, 2024. URL https://arxiv.or g/abs/2407.00079. Qiu, C., Fu, X., Li, F., Qian, Y., Xie, Y., Cai, X., Shan, Y., Wu, Y., and Zhang, M. AFD-Ledger: Deployment Provisioning for Attention–FFN Disaggregation. arXiv preprint arXiv:2608.04502, 2026. URL https://ar xiv.org/abs/2608.04502. Qwen Team. Qwen3-235B-A22B-Instruct-2507-FP8 model card, 2025. URL https://huggingface.co/Qwe n/Qwen3-235B-A22B-Instruct-2507-FP8. Qwen Team. Qwen3.5-397B-A17B-FP8 model card, 2026. URL https://huggingface.co/Qwen/Qwen3. 5-397B-A17B-FP8. Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Yazdani Aminabadi, R., Awan, A. A., Rasley, J., and He, Y. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. In Proceedings of the 39th International Conference on Machine Learning (ICML), pp. 18332–18346, 2022. URL https://proceedings.mlr.press/v162/r ajbhandari22a.html.

SGLang Team. Mini-SGLang: A Compact Implementation of SGLang, 2025a. URL https://github.com/s gl-project/mini-sglang. Repository, accessed October 2026. SGLang Team. Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I): 2.7x Higher Decoding Throughput, 2025b. URL https://lmsys.org/ blog/2025-06-16-gb200-part-1/. LMSYS Org Blog, June 16, 2025. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv preprint arXiv:1701.06538, 2017. URL https: //arxiv.org/abs/1701.06538. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053, 2019. URL https: //arxiv.org/abs/1909.08053. StepFun, Wang, B., Wang, B., Wan, C., Huang, G., Hu, H., Jia, H., Nie, H., Li, M., Chen, N., Chen, S., Yuan, S., Xie, W., Song, X., Chen, X., Yang, X., Zhang, X., Yu, Y., Wang, Y., Zhu, Y., Jiang, Y., Zhou, Y., Lu, Y., Li, H., Hu, J., Lo, K. M., Huang, A., Jiao, B., Li, B., Chen, B., Miao, C., Lou, C., Hu, C., Xu, C., Yu, C., Yao, C., Lv, D., Shi, D., Sun, D., Huang, D., Hu, D., Pang, D., Liu, E., Zhang, F., Wan, F., Yan, G., Zhang, H., Zhou, H., Wu, H., Guo, H., Chen, H., Zhang, H., Wu, H., Zhang, H., Yan, H., Lv, H., Wei, H., Zhou, H., Wang, H., Wang, H., Li, H., Zhou, H., Wang, H., Guo, H., Wang, J., Gong, J., Xie, J., Zhou, J., Sun, J., Wu, J., Zhang, J., Liu, J., Cheng, J., Luo, J., Yan, J., Yang, J., Hou, J., Zhang, J., Cao, J., Yin, J., Liu, J., Huang, J., Lin, J., Tan, K., Li, K., An, K., Lin, K., Liu, K., Yang, L., Zhao, L., Chen, L., Shi, L., Tan, L., Lin, L., Zhang, L., Chen, L., Huang, L., Shi, L., Gu, L., Chen, M., Ren, M., Li, M., Chen, M., Wang, N., Wu, N., Han, Q., Zhao, Q., Zhang, Q., Liu, Q., Chen, Q., Wu, Q., He, Q., Tan, Q., Wang, Q., Wu, Q., Liang, Q., Sun, Q., Li, R., Miao, R., Wan, R., Guo, R., Zhong, S., Pang, S., Fan, S., Shang, S., Jiang, S., Yang, S., Hao, S., Gao, S., Huang, S., Liu, S., Cao, T., Cheng, T., Peng, T., You, W., Ji, W., Sun, W., Deng, W., He, W., Zheng, W., Chen, X., Kong, X., Luo, X., Yang, X., Liu, X., Ren, X., Han, X., Li, X., Wu, X., Zhao, X., Wei, Y., Li, Y., Li, Y., Xu, Y., Xu, Y., Shi, Y., Shen, Y., Yang, Y., Yang, Y., Gong, Y., Chen, Y., Yang, Y., Zhang, Y., Zhou, Y., Ding, Y., Fan, Y., Yang, Y., Luo, Y., Peng, Y., Lu, Y., Deng, Y., Yin, Y., Liu, Y., Chen, Y., Zhao, Y., Mou, Y., Li, Y., Ju, Y., Li, Y., Yang, Y., Zhang, Y., Chen, Y., Weng, Z., Xie, Z., Ge, Z., Gong, Z., Lu, Z., Huang, Z., Chang, Z., Huang, Z., Wang, Z., Yang, Z., Wang, Z., Wang, Z., Zhang, Z., Jiang, D., Shum, H.-Y., and Zhang, X. Step-3 is Large yet

Affordable: Model-system Co-design for Cost-effective Decoding. arXiv preprint arXiv:2507.19427, 2025. URL https://arxiv.org/abs/2507.19427. Tschand, A., Fu, Y., Mailthody, V. S., Oswald, N., Tsai, P.-A., Zhao, R., Villa, O., Reddi, V. J., and Sankaralingam, K. Rethinking Heterogeneous System Disaggregation for Subquadratic Attention. arXiv preprint arXiv:2609.13134, 2026. URL https://arxiv.or g/abs/2609.13134. vLLM Team. vLLM Large Scale Serving: DeepSeek @ 2.2k tok/s/H200 with Wide-EP, 2025. URL https: //blog.vllm.ai/2025/12/17/large-scale -serving.html. vLLM Blog, December 17, 2025. Wu, H., Bambhaniya, A. R., Banerjee, S., Khare, T., Srinivasan, S., Subramanian, S., Kundu, S., Kumar, M., Elavazhagan, M., Won, W., Yazdanbakhsh, A., and Krishna, T. How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving. arXiv preprint arXiv:2605.28302, 2026. URL https://arxiv.or g/abs/2605.28302. Xiao, A., He, B., Zhang, B., Huai, B., Wang, B., Wang, B., Xu, B., Hou, B., Yang, C., Liu, C., Cui, C., Zhu, C., Feng, C., Wang, D., Lin, D., Zhao, D., Zou, F., Wang, F., Zhang, G., Dan, G., Chen, G., Guan, G., Yang, G., Li, H., Zhu, H., Li, H., Feng, H., Huang, H., Xu, H., Ma, H., Fan, H., Liu, H., Li, J., Liu, J., Xu, J., Meng, J., Xin, J., Hu, J., Chen, J., Yu, L., Miao, L., Liu, L., Jing, L., Zhou, L., Han, M., Deng, M., Deng, M., Deng, N., Lin, N., Zhao, P., Pan, P., Shen, P., Li, P., Zhang, Q., Wang, Q., Zhang, Q., Xia, Q., Zhang, Q., Fu, Q., Guo, R., Gao, R., Li, S., Long, S., Li, S., Wan, S., Shen, S., Zeng, S., Jing, S., Yang, S., Zhang, S., Xu, T., Du, T., Chen, T., Wu, W., Jiang, W., Tong, W., Chen, W., Peng, W., Zhou, W., Yang, W., Liang, W., Liu, X., Zhou, X., Jin, X., Duan, X., Li, X., Zhang, X., Chen, X., Shan, Y., Gan, Y., Lu, Y., Deng, Y., Zheng, Y., Xiong, Y., Zheng, Y., Zheng, Y., Shan, Y., Gao, Y., Zhang, Y., Yang, Y., Gong, Y., Yu, Y., Chen, Y., Zhu, Y., He, Y., Zhao, Y., Wu, Y., Zhang, Z., Zhuo, Z., Ji, Z., Wang, Z., Wang, Z., Fan, Z., Yang, Z., Sheng, Z., Yu, Z., Ji, Z., Ren, Z., Bian, Z., Liu, Z., Dong, Z., Li, Z., Yu, Z., Shen, Z., Peng, Z., Ye, Z., Xiang, Z., Fu, Z., and Zhang, Z. Huawei Cloud Model-as-a-Service on the CloudMatrix384 SuperPod. arXiv preprint arXiv:2508.02520, 2025. URL https: //arxiv.org/abs/2508.02520. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L.,

Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388. Zhao, C., Xu, Z., Zhao, L., Li, J., Xu, C., Xu, A., Liu, S., Zhou, K., and Yu, K. DeepGEMM: Clean and Efficient BLAS Kernel Library on GPU, 2025a. URL https: //github.com/deepseek-ai/DeepGEMM. Zhao, C., Zhou, S., Zhang, L., Deng, C., Xu, Z., Liu, Y., Yu, K., Li, J., and Zhao, L. DeepEP: An Efficient ExpertParallel Communication Library, 2025b. URL https: //github.com/deepseek-ai/DeepEP. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024. URL https://arxiv.org/abs/2312.0 7104. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 193–210, 2024. URL https://arxiv.org/abs/2401.0 9670. Zhu, R., Jiang, Z., Jin, C., Wu, P., Stuardo, C. A., Wang, D., Zhang, X., Zhou, H., Wei, H., Cheng, Y., Xiao, J., Zhang, X., Liu, L., Lin, H., Chang, L.-W., Ye, J., Yu, X., Liu, X., Jin, X., and Liu, X. MegaScale-Infer: Serving Mixture-ofExperts at Scale with Disaggregated Expert Parallelism. arXiv preprint arXiv:2504.02263, 2025. URL https: //arxiv.org/abs/2504.02263. Zhuang, Y., Chen, J., Pang, B., Gu, Y., Zhu, Y., Jiang, Y., Stoica, I., Zhang, H., and Xing, E. P. Efficient LongContext Language Model Training by Core Attention Disaggregation. In Proceedings of Machine Learning and Systems, volume 8, 2026. URL https://procee dings.mlsys.org/paper_files/paper/20 26/hash/423b59ae02381f27862c21d1c41a 5603-Abstract-Conference.html.

AI U SAGE D ISCLOSURE The authors used Claude (via Claude Code) and Gemini (via Antigravity) as assistive tools for code development in

portions of the serving runtime and verification and plotting scripts, manuscript polishing, literature search, and author-directed analysis. The tools also supported benchmark execution and logging under fixed, author-defined configurations. The authors retained control over the research questions, experimental design, and interpretation of results, reviewed the text and code, verified references, and recomputed reported results from raw hardware logs. The authors take full responsibility for the work.

These appendices give the complete data and model details behind the main paper. Every value is measured on the GB200 NVL72 rack unless it is marked as predicted or modeled. We write Qwen3 for Qwen3-235B-A22B and Qwen3.5 for Qwen3.5-397B-A17B. Appendix A defines the measurements and lists all standard-decode results (Section 3). Appendix B lists every model candidate with its prediction error (Section 4.4). Appendix C defines the model, its calibration, and the notation (Section 4). Appendix D gives the complete speculative comparisons (Sections 3 and 6). Appendix E reports the runtime ablations and the CAD sweep (Section 5). Appendix F lists the run presets and software settings.

A

M ODEL S PECIFICATIONS AND C OMPLETE S TANDARD -D ECODE R ESULTS

Table A1 summarizes the architectural parameters of the three evaluated FP8 MoE checkpoints. Table A2 gives the best measured configuration of each engine for the nine standard-decode workloads (4K–16K), and Tables A3–A5 give every measured configuration (rules in Appendix A.1). Throughput counts all allocated GPUs, including FFN GPUs, unless it is marked as attention-only; dashes mark configurations without a valid run. Widths past each wideEP peak were also measured: Qwen3 EP64 at 4K falls to 2986 tokens/s/GPU after peaking at EP16. Every comparison uses the best measured width. Table A1. Evaluated FP8 MoE models and architectural parameters. Model

Attention

MiniMax-M2.5 Full (GQA) Qwen3-235B-A22B Full (GQA) Qwen3.5-397B-A17B 15 Full + 45 Linear

Total / Active L Experts Top-K 229B / 10B 62 235B / 22B 94 397B / 17B 60

256 128 512

8 8 10

The throughput tables and the model-error tables (Appendix B) come from the same runs and can differ in the last digit through rounding or the choice among repeated readings. A.1

How Each Number Is Measured

FastAFD cells. Each request decodes exactly 1024 output tokens at a fixed per-attention-GPU batch bA : the KV-full batch, or a slightly smaller batch aligned to whole attentionkernel waves when that runs faster. After the last prefill step, the period TA is the mean decode step over the following steady decode window, and throughput counts every allor cated GPU, Xall = r+1 bA /TA . Cells with several readings report their mean.

vLLM cells. Every GPU runs one engine (TP1; EP equals the number of GPUs), and each engine holds the KV-full batch bV = ⌊KV tokens per GPU/(ISL + OSL)⌋, stepping down when a batch yields no valid window. Eight streaming probe requests timestamp every output token on the client; because every step is an all-to-all over all engines, a probe’s inter-token interval is the global step time. The window starts at the first metrics sample where all requests are running and the prompt-token counter has reached its total, and ends at the first completion; TV is the pooled mean of the probe intervals inside it, and throughput is bV /TV per GPU. A valid window needs at least 200 intervals. When the strict window is shorter or never starts (the prompt counter can stop just below its total), a fallback window is accepted only if it starts after prefill with margin, ends before the first completion, contains no prefill, queueing, or preemption, and its largest interval is at most 6.85× the median; probes that deliver tokens in bursts (over 1.15 tokens per interval) are excluded. Each request decodes OSL = 1024 output tokens (Table A18). Ablations. Ablation periods (Table A15) are wall-clock: the inverse slope of the decode-step index against log timestamps over the window, so host gaps between steps count.

B

A LL M ODEL C ANDIDATES AND R ESIDUALS

Tables A8–A11 list every measured candidate, 53 standard and 76 speculative (d ∈ {3, 7}), with its predicted period. Signed errors are e = 100(T /Tb − 1), which equals the relative throughput error at fixed batch and useful-token output. “Scope” marks ratios at or below the measured peak of the same model, context, and draft length; candidates past the peak are kept. Table A6 summarizes accuracy and zero-sweep ratio selection across the 88 primary 4K–16K runs evaluated in Section 4.4, and Table A7 extends the error summary to 32K and d = 7. Selection loss compares the measured throughput at the predicted best ratio with the measured best of the same workload; candidate ratios are discrete and bounded by the 18-node rack. In standard decoding at r ≤ r∗ , the median error is 2.69% and the IQR is 2.24–3.80%. Under d = 3 at r ≤ r∗ , the mean is 2.64% with a 5.66% maximum across 4K–16K, and the maximum is 8.77% at 32K. The largest standard error (43.38%) is Qwen3.5 4K at 7:1, past the peak; the largest speculative error (16.81%) is Qwen3 16K at d = 7 and 2:1, within scope.

Table A2. Best measured standard-decode configuration of each engine per workload (tokens/s/GPU, all GPUs counted). Optima are over the measured A:F ratios and EP widths. Model Context AFD Throughput Baseline Throughput / EP4 / best EP MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 Qwen3 Qwen3 Qwen3 Qwen3.5 Qwen3.5 Qwen3.5

4K 8K 16K 4K 8K 16K 4K 8K 16K

AFD 9:1 AFD 15:1 AFD 17:1 AFD 5:1 AFD 7:1 AFD 11:1 AFD 5:1 AFD 5:1 AFD 5:1

4118.0 2372.0 1234.0 4256.0 2624.0 1452.0 6968.0 5627.0 3955.0

Table A3. MiniMax-M2.5 standard-decode throughput (tokens/s/GPU, counting FFN GPUs). Dashes mark configurations without a valid run. Configuration 4K 8K 16K AFD 1:1 AFD 3:1 AFD 5:1 AFD 7:1 AFD 9:1 AFD 11:1 AFD 13:1 AFD 15:1 AFD 17:1 vLLM EP4 vLLM EP8 vLLM EP16 vLLM EP32 vLLM EP64

2341.0 3500.0 3883.0 4042.0 4118.0 3553.0 — — — 2936.0 3664.0 4076.0 4098.0 3812.0

1281.0 1920.0 2128.0 2230.0 2294.0 2331.0 2364.0 2372.0 2249.0 1563.0 1942.0 2251.0 2317.0 2029.0

657.0 987.0 1038.0 1148.0 1180.0 1201.0 1216.0 1229.0 1234.0 793.0 1028.0 1159.0 1208.0 1119.0

Table A4. Qwen3 standard-decode throughput (tokens/s/GPU, counting FFN GPUs). Dashes mark configurations without a valid run. Configuration 4K 8K 16K AFD 1:1 AFD 3:1 AFD 5:1 AFD 7:1 AFD 9:1 AFD 11:1 AFD 13:1 AFD 17:1 vLLM EP4 vLLM EP8 vLLM EP16 vLLM EP32 vLLM EP64

2721.0 4055.0 4256.0 3907.0 — — — — 3270.0 3795.0 4004.0 3728.0 2986.0

1542.0 2289.0 2542.0 2624.0 2530.0 — — — 1859.0 2173.0 2389.0 2310.0 2073.0

810.0 1195.0 1329.0 1396.0 1434.0 1452.0 1444.0 1365.0 1019.0 1157.0 1227.0 1240.0 1148.0

vLLM EP32 vLLM EP32 vLLM EP32 vLLM EP16 vLLM EP16 vLLM EP32 vLLM EP32 vLLM EP32 vLLM EP32

4098.0 2317.0 1208.0 4004.0 2389.0 1240.0 6012.0 5095.0 3982.0

1.403 1.518 1.556 1.302 1.412 1.425 2.254 2.362 2.114

1.005 1.024 1.022 1.063 1.098 1.171 1.159 1.104 0.993

Table A5. Qwen3.5 standard-decode throughput (tokens/s/GPU, counting FFN GPUs). Dashes mark configurations without a valid run. Configuration 4K 8K 16K AFD 1:1 AFD 3:1 AFD 5:1 AFD 7:1 vLLM EP4 vLLM EP8 vLLM EP16 vLLM EP32 vLLM EP64

4366.0 6494.0 6968.0 4716.0 3092.0 5186.0 5950.0 6012.0 5605.0

3532.0 5220.0 5627.0 5054.0 2382.0 4188.0 5037.0 5095.0 4895.0

2543.0 3711.0 3955.0 3854.0 1871.0 3226.0 3762.0 3982.0 3770.0

Table A6. Model accuracy and ratio-choice summary against the 88 measured FastAFD runs at 4K–16K (Section 4.4). Error is b |X/X − 1|. Standard Speculative (4K–16K) (d = 3, 4K–16K) Workloads 9 Mean error at the peak 3.41% Mean error 3.04% Max error at the peak 7.36% Max error 7.36% Model picks the best ratio 7 of 9 Peak throughput kept by its pick 95.3–100%

6 4.07% 2.64% 5.66% 5.66% 2 of 6 96.5–100%

Mean and max error cover every A:F ratio up to each workload’s measured best ratio r ∗ (44 standard and 21 speculative runs); “at the peak” is the run at r ∗ alone. Past the peak (9 and 14 runs), errors average 10.2% and 6.6%. “Its pick” is the ratio with the highest predicted throughput.

Table A9. Standard-decode model candidates 28–53: measured period T , predicted period Tb, and signed error e = 100(T /Tb − 1). Scope Y marks r ≤ r∗ . Table A7. Extended model error summary including 32K and d = 7: mean and maximum |e| (%) over the listed runs, and the worst selection loss. Decoding Runs n Mean Max Sel. loss Standard (d = 0) all 53 r ≤ r∗ 44 peaks 9

4.25 43.38 3.04 7.36 3.41 7.36

4.72

d=3

all 42 r ≤ r∗ 26 peaks 8

4.32 11.05 3.19 8.77 4.32 8.30

3.49

d ∈ {3, 7}

all 76 r ≤ r∗ 50 peaks 16

5.26 16.81 4.91 16.81 4.22 8.30

10.90

Model

Context d A:F bA T (ms) Tb (ms)

Qwen3.5 Qwen3.5 Qwen3.5 Qwen3.5 Qwen3.5 Qwen3.5 Qwen3.5 Qwen3.5 Qwen3.5 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3

4K 8K 8K 8K 8K 16K 16K 16K 16K 4K 4K 4K 4K 8K 8K 8K 8K 8K 16K 16K 16K 16K 16K 16K 16K 16K

0 7:1 0 1:1 0 3:1 0 5:1 0 7:1 0 1:1 0 3:1 0 5:1 0 7:1 0 1:1 0 3:1 0 5:1 0 7:1 0 1:1 0 3:1 0 5:1 0 7:1 0 9:1 0 1:1 0 3:1 0 5:1 0 7:1 0 9:1 0 11:1 0 13:1 0 17:1

491 383 373 365 357 247 241 235 231 202 200 200 198 108 106 106 106 106 56 54 54 54 54 54 54 52

91.092 54.220 53.590 54.050 61.804 48.570 48.710 49.513 52.452 37.120 36.990 39.160 44.340 35.010 34.730 34.740 35.340 37.700 34.560 33.880 33.870 33.850 33.900 34.100 34.710 35.990

e (%) Scope

63.532 +43.38 52.110 +4.05 51.127 +4.82 53.966 +0.16 58.526 +5.60 47.466 +2.33 50.252 -3.07 52.875 -6.36 55.576 -5.62 35.557 +4.40 35.320 +4.73 41.102 -4.72 47.649 -6.94 36.139 -3.12 36.139 -3.90 36.139 -3.87 36.726 -3.77 40.322 -6.50 35.862 -3.63 34.954 -3.07 34.954 -3.10 34.954 -3.16 34.954 -3.02 34.954 -2.44 36.048 -3.71 39.135 -8.04

N Y Y Y N Y Y Y N Y Y Y N Y Y Y Y N Y Y Y Y Y Y N N

Table A10. Speculative model candidates 1–38: measured period T , predicted period Tb, and signed error e = 100(T /Tb −1). Scope Y marks r ≤ r∗ . Table A8. Standard-decode model candidates 1–27: measured period T , predicted period Tb, and signed error e = 100(T /Tb − 1). Scope Y marks r ≤ r∗ . Model

Context d A:F bA T (ms) Tb (ms)

MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 Qwen3.5 Qwen3.5 Qwen3.5

4K 4K 4K 4K 4K 4K 8K 8K 8K 8K 8K 8K 8K 8K 8K 16K 16K 16K 16K 16K 16K 16K 16K 16K 4K 4K 4K

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1:1 3:1 5:1 7:1 9:1 11:1 1:1 3:1 5:1 7:1 9:1 11:1 13:1 15:1 17:1 1:1 3:1 5:1 7:1 9:1 11:1 13:1 15:1 17:1 1:1 3:1 5:1

156 156 156 154 154 152 76 76 76 76 76 76 76 76 76 38 38 38 38 38 38 38 38 38 527 515 503

33.320 33.420 33.480 33.340 33.660 39.210 29.660 29.690 29.770 29.830 29.820 29.880 29.860 30.040 31.920 28.920 28.870 30.500 28.960 28.970 29.010 29.020 28.980 29.080 60.350 59.480 60.155

e (%) Scope

34.047 -2.13 34.047 -1.84 34.047 -1.66 34.047 -2.08 34.047 -1.14 35.257 +11.21 30.382 -2.38 30.382 -2.28 30.382 -2.02 30.382 -1.82 30.382 -1.85 30.382 -1.65 30.382 -1.72 30.794 -2.45 32.069 -0.46 29.766 -2.84 29.766 -3.01 29.766 +2.46 29.766 -2.71 29.766 -2.68 29.766 -2.54 29.766 -2.51 29.766 -2.64 29.766 -2.31 57.457 +5.03 56.623 +5.05 56.032 +7.36

Y Y Y Y Y N Y Y Y Y Y Y Y Y N Y Y Y Y Y Y Y Y Y Y Y Y

Model

Context d A:F bA T (ms) Tb (ms)

MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5

4K 4K 4K 4K 4K 4K 4K 4K 4K 8K 8K 8K 8K 8K 8K 8K 8K 8K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 32K 32K 32K 32K 32K 32K 32K

3 3 3 3 3 7 7 7 7 3 3 3 3 3 7 7 7 7 3 3 3 3 3 3 3 3 7 7 7 7 7 3 3 3 3 7 7 7

1:1 3:1 4:1 5:1 7:1 1:1 2:1 3:1 4:1 1:1 3:1 5:1 6:1 7:1 2:1 3:1 4:1 5:1 1:1 3:1 5:1 7:1 8:1 9:1 12:1 16:1 4:1 5:1 6:1 7:1 8:1 12:1 13:1 14:1 15:1 5:1 6:1 7:1

148 150 150 150 148 150 150 150 148 76 76 76 76 76 76 76 76 76 38 38 38 38 38 38 38 38 38 38 38 38 38 20 20 20 20 20 20 20

36.596 38.874 41.367 44.305 54.269 46.196 48.868 53.635 60.772 32.721 33.112 34.465 36.540 38.484 40.052 40.309 42.949 46.374 30.789 30.683 30.945 31.709 31.973 32.740 35.412 40.173 40.521 39.579 38.683 39.645 41.937 32.945 33.027 33.039 33.203 37.332 37.495 37.440

e (%) Scope

36.438 +0.43 38.711 +0.42 43.745 -5.44 48.779 -9.17 58.362 -7.01 42.815 +7.90 46.399 +5.32 56.467 -5.02 65.963 -7.87 34.242 -4.44 34.242 -3.30 36.024 -4.33 38.574 -5.27 41.125 -6.42 38.462 +4.13 40.991 -1.66 46.092 -6.82 51.194 -9.41 32.244 -4.51 32.244 -4.84 32.244 -4.03 32.244 -1.66 33.282 -3.93 34.558 -5.26 38.383 -7.74 43.485 -7.62 35.409 +14.44 37.960 +4.27 40.510 -4.51 43.061 -7.93 45.612 -8.06 31.472 +4.68 31.762 +3.98 32.434 +1.87 33.105 +0.30 34.249 +9.00 34.249 +9.48 34.492 +8.55

Y Y Y N N Y Y Y N Y Y Y N N Y Y Y N Y Y Y Y Y N N N Y Y Y N N Y Y Y N Y Y Y

Table A11. Speculative model candidates 39–76: measured period T , predicted period Tb, and signed error e = 100(T /Tb −1). Scope Y marks r ≤ r∗ . Model

Context d A:F bA

MiniMax-M2.5 MiniMax-M2.5 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3

32K 32K 4K 4K 4K 4K 4K 4K 4K 4K 8K 8K 8K 8K 8K 8K 8K 8K 8K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 32K 32K 32K 32K 32K 32K 32K 32K 32K

7 7 3 3 3 3 3 7 7 7 3 3 3 3 3 3 7 7 7 3 3 3 3 3 3 7 7 7 7 3 3 3 7 7 7 7 7 7

8:1 9:1 1:1 2:1 3:1 5:1 7:1 1:1 2:1 3:1 1:1 2:1 3:1 4:1 5:1 7:1 1:1 2:1 3:1 1:1 2:1 3:1 4:1 5:1 7:1 1:1 2:1 3:1 4:1 7:1 8:1 9:1 3:1 4:1 5:1 6:1 7:1 8:1

T (ms)

Tb (ms)

e (%) Scope

20 37.605 35.835 +4.94 20 38.077 37.177 +2.42 194 46.352 45.227 +2.49 194 52.397 53.096 -1.32 192 62.060 65.843 -5.75 192 98.167 91.895 +6.83 190 129.910 116.989 +11.05 192 64.375 55.820 +15.33 192 85.266 81.546 +4.56 192 116.935 107.598 +8.68 104 41.419 42.101 -1.62 104 42.159 42.101 +0.14 104 45.775 47.559 -3.75 102 50.510 54.057 -6.56 102 58.734 60.976 -3.68 102 79.190 74.816 +5.85 104 55.688 50.667 +9.91 104 58.732 57.008 +3.02 102 70.467 70.271 +0.28 54 37.778 37.907 -0.34 54 37.887 37.907 -0.05 54 38.633 37.907 +1.91 52 39.781 40.099 -0.79 52 41.157 43.627 -5.66 52 48.428 50.683 -4.45 54 49.070 43.649 +12.42 54 50.986 43.649 +16.81 54 52.210 50.258 +3.88 52 53.971 56.432 -4.36 26 38.686 42.407 -8.77 26 38.886 42.407 -8.30 26 40.007 42.407 -5.66 26 47.365 49.722 -4.74 26 47.725 49.722 -4.02 26 48.857 49.722 -1.74 26 49.840 49.722 +0.24 26 51.209 52.694 -2.82 26 54.288 56.222 -3.44

Y N Y Y N N N Y Y N Y Y Y N N N Y Y N Y Y Y Y Y N Y Y Y N Y Y N Y Y Y Y N N

C

M ODEL D EFINITIONS , C ALIBRATION , AND P ROFILING

Table A12 defines every symbol of the main paper. The rest of this appendix gives the per-layer model, its calibration, and the speculative extension. C.1

Auxiliary Latency and Calibration

DV aggregates the auxiliary kernels of the colocated baseline: projections, normalization, routing, and the other attention-side work outside attention and the remote experts. The model fixes it at the profile value, and each of the two microbatches incurs it once per step. This approximation holds for the decode operating points studied here. Figure 5b uses this model for the FastAFD bars: attention from the wave model, 2DV from the measured vLLM EP4 profile, and FFN exposure as the remainder of the predicted step, which is within 0.2–3.8% of the measured step at the three 8K peaks. The attention estimate is a wave model with effective bandwidth utilization u = 0.89, HBM bandwidth 7.93 × 1012 bytes/s, per-SM bandwidth 107 × 109 bytes/s, and 152 SMs. The expert-compute model uses an FP8 rate scale P = 5 × 1015 operations/s, GEMM utilization uc = 0.3347407189, and a fixed service term τ = 42.834448717 µs. Both uc and τ are fitted once across the twelve vLLM EP4 profiles (τ : intercept of dispatch-plus-combine time against routed bytes; uc : slope of expert GEMM time against FLOPs). They are model inputs, not measured hardware maxima or isolated communication costs. The attention-wave function uses a block count N , bytes per block Bblk , SSM = 152, w = ⌊N/SSM ⌋, and k = N − wSSM :   kBblk Bblk wSSM Bblk + 1k>0 max , . Awave = u BW u BW BWSM (A1) For MiniMax and Qwen3, N = nhKV and Bblk = 4dh (c + 256), with context c, head dimension dh = 128, and KVhead counts eight and four. The Qwen3.5 full-attention estimate uses N = 2n and Bblk = 1024(c + 261); its gateddelta estimate is 1.3677n µs. Remote FFN service over m routed tokens (rn standard; rnq speculative) decomposes into dispatch intercept, HBM3e expert-weight reads, and grouped-GEMM compute: Fℓ (m) = τℓ +

Wℓ m 6Hℓ Iℓ Kℓ + . u BW uc P

(A2)

For hybrid Qwen3.5, the model sums 15 full-attention and 45 gated-delta layer terms and takes the maximum with FFN service within each layer type before summing; averaging the two attention types first would hide a bottleneck in one of them. The measured full-model DV is spread evenly

across layers, and the residuals include the effect of that simplification. Table A13 lists the profile inputs. C.2

Closed-Form Throughput and Stage-Level Win Condition

Decomposing the colocated wide-EP step into attention, auxiliary, and MoE components as TV = AV (bV ) + DV + MV (bV ), and writing the two-microbatch FastAFD step identically as Tb A = 2AA (bA /2) + 2DV + Texp (r) where P Texp (r) = 2 ℓ max{0, Fℓ (rnq) − Ta,ℓ }, Equation 2 yields Sall > 1 if and only if     1 bA AV (bV ) + DV + MV (bV ) >1+ . × bV 2AA (bA /2) + 2DV + Texp (r) r | {z } | {z } Batch gain

Overlap gain

(A3) Under a fully covered FFN stage (Texp (r) = 0) and linear attention scaling, let the baseline period be TV = AV + DV + MV , and let B = bA /bV . Approximating TA = BAV + 2DV gives B r , r + 1 B(1 − dV − mV ) + 2dV   1 2 S > 1 ⇐⇒ mV > + − 1 dV . r+1 B S=

(A4) (A5)

This closed form holds under the stated approximation; attention-wave effects, exposed FFN service, hybrid layers, and speculative auxiliary work require the full per-layer model. C.3

Speculative Kernel Partition and Two-Side Decomposition

The speculative profiles cover 16 model/context/draft combinations. The extraction identifies the main target CUDA graph and partitions its attention, expert, and auxiliary kernels (DV,spec ) from graph-external draft-and-accept activity (Tdraft ). Draft attention is scaled by bA /bp (Tdraft ≈ (bA /bp )Tdraft,att,V + Tdraft,rest,V ), where bp is the profiling run’s batch, distinct from the comparison baseline’s bV ; the remaining external activity is fixed at its profile value. Target verification already processes q = d + 1 positions, so its measured auxiliary work DV,spec is not multiplied by q again. Separating the two sides of Equation 3 in the main paper, verification attends q positions per request against the same prefix on the attention GPUs, reading each KV cache once: Taspec = κℓ Aℓ (n) + DV,spec /L.

(A6)

On the expert side, the FFN pool serves q× more token rows (rnq) while reading each expert’s weights once per layer. Writing Fℓ (m) = Fℓ,0 + m cℓ with fixed per-layer

Table A12. Notation used in the main paper. Source: config = chosen setting; measured = run records; profile = colocated vLLM EP4 profiles; modeled = this appendix’s model. Symbol

Meaning

Source

Configuration and measured quantities NA , N F attention / FFN nodes (four GPUs each) r A:F node ratio NA /NF bA , b V per-GPU decode batch: FastAFD attention GPU / colocated vLLM n requests per microbatch, bA /2 TA , TV decode step period: FastAFD / vLLM TbA predicted FastAFD period (Equation 3) Xall decode tokens/s per allocated GPU SEP4 , Sall FastAFD speedup over EP4 / best wide EP, all GPUs Sall Satt speedup per attention GPU, r+1 r ⋆ S zero-FFN-cost ceiling, (bA /bV ) TV /(TbA (rmin ) − DV ) L, ℓ number of layers, layer index

config config measured config measured modeled measured measured measured modeled model config

Step components AV , DV , MV vLLM attention, auxiliary (projections, norms, router), and MoE (expert GEMMs plus profile dispatch/combine) time per step Aℓ (n) attention time of layer ℓ for n requests modeled P AA (n) modeled ℓ Aℓ (n) Fℓ (m) remote FFN service of layer ℓ for m routed tokens (Equation A2) modeled m routed tokens per layer: rn standard, rnq speculative config Ta,ℓ , Te,ℓ (r) per-layer attention stage modeled P Aℓ (n) + DV /L / FFN stage Fℓ (rnq) (Section 4.2) exposed FFN time, 2 ℓ max{0, Fℓ (rn) − Aℓ (n) − DV /L}; since max{x, y} = x + modeled Texp (r) max{0, y − x}, in standard decoding (Tdraft = 0) Equation 3 equals 2AA (n) + 2DV + Texp (r) exactly FFN service constants (Equation A2) τℓ per-layer dispatch intercept (42.83 µs) Wℓ expert weight bytes per FFN GPU per layer BW, u HBM bandwidth (7.93 TB/s) and its efficiency (0.89) P, uc FP8 peak (5 PFLOPS) and GEMM efficiency (0.335) Hℓ , I ℓ , K ℓ hidden width, expert intermediate width, active experts per token Fℓ,0 fixed per-layer FFN overhead τℓ + Wℓ /(u BW), the intercept of Equation A2 cℓ per-token expert cost 6Hℓ Iℓ Kℓ /(uc P ), the slope of Equation A2

profile fit model config spec spec / profile fit model config modeled modeled

Speculative decoding d, q draft length; verified positions q = d + 1 ā mean accepted tokens per request per step κℓ , DV,spec speculative attention factor and target per-microbatch auxiliary time (Table A13) Tdraft graph-external draft-and-accept latency per step, (bA /bp )Tdraft,att,V + Tdraft,rest,V Taspec , Tespec speculative attention stage κℓ Aℓ (n) + DV,spec /L / FFN stage Fℓ (rnq) (Section 4.2)

config measured profile profile modeled

dispatch and weight-read overhead Fℓ,0 = τℓ +Wℓ /(u BW) and per-token expert GEMM cost cℓ = 6Hℓ Iℓ Kℓ /(uc P ) in Equation A2, only the linear expert-GEMM term grows: Tespec = Fℓ (rnq) = Te + ∆Fℓ , ∆Fℓ = (q − 1) rn cℓ = (q − 1)

rn 6Hℓ Iℓ Kℓ . uc P

(A7)

The split is reproducible at the kernel level but does not label every draft, sampling, acceptance, or synchronization kernel individually, and summed CUDA time need not equal the critical-path time of concurrent streams; the speculative errors in Appendix B measure how well this compact split works.

D

C OMPLETE S PECULATIVE C OMPARISONS

Table A14 lists the best measured configurations for standard decoding and for draft lengths three and seven on MiniMax-M2.5 and Qwen3 at 4K–32K. Draft and target models both run, and acceptance is held near 1.8 useful tokens per request per step; run-level calculations use the measured averages. Batch and topology may change across modes, and each throughput is normalized by the GPUs that configuration allocates. All-GPU and attention-only speedups differ by the FFNnode factor r/(r + 1) of the AFD configuration; the attention-only value describes the attention pool and does not treat FFN GPUs as free. Self-speedup compares each engine with its own standard-decode reference; speedups

Table A13. Colocated profile inputs of the model. Standard rows use the vLLM EP4 profile; speculative rows use the profile with the same draft length. A dash marks no speculative attention factor. Model

Context

d

bp

DV (ms)

κ

MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 Qwen3.5 Qwen3.5 Qwen3.5 Qwen3 Qwen3 Qwen3 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3

4K 8K 16K 4K 8K 16K 4K 8K 16K 4K 4K 8K 8K 16K 16K 32K 32K 4K 4K 8K 8K 16K 16K 32K 32K

0 0 0 0 0 0 0 0 0 3 7 3 7 3 7 3 7 3 7 3 7 3 7 3 7

94 49 25 83 62 45 125 66 34 126 109 67 67 34 34 18 17 151 119 88 76 38 38 23 23

3.8310 3.6400 3.5070 7.5140 7.2230 7.0110 5.7890 5.5130 5.2220 5.0788 6.7159 4.3770 5.1355 3.9178 4.3139 3.3866 3.7036 9.2576 12.1423 6.8859 9.3263 5.6180 6.7442 5.1682 5.8431

— — — — — — — — — 1.0320 1.0366 1.0149 1.0273 0.9914 0.9958 0.9632 0.9668 1.0456 1.1510 1.0384 1.0899 1.0117 1.0618 1.0187 1.1504

over wide EP compare the two engines in the same mode. MiniMax-M2.5 32K at d = 3 has no valid EP64 run, so its wide-EP peak is not bracketed (EP32 beats EP16); an asterisk marks this cell.

E

A BLATIONS AND CAD

Runtime ablations. The MiniMax-M2.5 8K, 7:1, batch76 ablation uses preset afd-cell-v3.4.3 (Table A18). Base, single-microbatch, and no-cross-step-overlap variants have wall-clock periods 29.893, 39.880, and 43.820 ms, respectively, and throughput of 2224.6, 1667.5, and 1517.6 tokens/s/GPU. The estimator is the inverse slope of decodestep number against the log timestamps. Internal cell means are 29.906, 39.874, and 40.483 ms; the last is not interchangeable with the wall-clock number because the disabled-overlap loop changes the timing scope. Table A15 lists every ablation leg behind Figure 7a and c, and Table A16 the microbatch sweep behind Figure 7b. On Qwen3, the non-fused DeepEP+DeepGEMM variant needs more than 12 GiB of extra memory per attention GPU: at memory ratio 0.88 (bA = 102), CUDA-graph capture of its dispatch runs out of memory on all 28 attention GPUs, so both legs of the pair use ratio 0.80 (bA = 92); the MegaMoE run at the Table A2 configuration (bA = 106, ratio 0.91) reproduces the table within 0.06%. Non-fused wall-clock pe-

riods exceed the internal step means by about 6% (53.21 vs. 50.02 ms on MiniMax-M2.5, 67.70 vs. 63.79 ms on Qwen3) because its steps serialize host work; the figure uses wallclock periods for every leg. CAD sweep. CAD moves the QKV/output projections and expert execution to the FFN node while keeping the attention core, Q/K normalization, and KV writes on the attention nodes. A token crosses this boundary before and after attention, so Equation 3 does not model CAD latency. Table A17 reports the complete 15-cell CAD sweep on MiniMax-M2.5 across 4K–16K. CAD runs use a small-tile MegaMoE setting that favors CAD relative to the AFD tables, so the comparison is conservative for FastAFD. The CAD optimum is bracketed on both sides at 2:1–4:1 and reaches 0.675–0.782× of FastAFD’s best per-GPU throughput. All 15 CAD cells produce output token-identical to their FastAFD reference runs (the first 1, 16, and 64 tokens and the full output of every sampled prompt); speculative modes were not compared.

Table A14. Best measured configurations for standard (d = 0) and speculative (d = 3, 7) decoding (tokens/s/GPU, all GPUs counted). Acceptance is held near 1.8 useful tokens per step. ∗ MiniMax-M2.5 32K, d = 3: EP64 has no valid run, so the wide-EP peak is not bracketed. Model Context d AFD tok/s/GPU EP tok/s/GPU Sall Satt MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 MiniMax-M2.5 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3

4K 4K 4K 8K 8K 8K 16K 16K 16K 32K 32K 32K 4K 4K 4K 8K 8K 8K 16K 16K 16K 32K 32K 32K

0 3 7 0 3 7 0 3 7 0 3 7 0 3 7 0 3 7 0 3 7 0 3 7

9:1 4:1 3:1 15:1 5:1 4:1 17:1 8:1 6:1 17:1 14:1 8:1 5:1 2:1 2:1 7:1 3:1 2:1 11:1 5:1 3:1 16:1 8:1 6:1

4118.1 5213.2 3770.2 2372.1 3302.6 2544.2 1234.2 1903.9 1517.5 603.6 1021.8 855.0 4256.4 4439.1 2699.4 2624.3 3064.2 2122.8 1451.6 1892.8 1393.2 735.8 1072.0 806.5

EP32 EP16 EP8 EP32 EP16 EP16 EP32 EP16 EP16 EP32 EP32∗ EP16 EP16 EP8 EP4 EP16 EP16 EP8 EP32 EP8 EP8 EP32 EP16 EP16

4098.0 4849.4 3551.7 2317.0 3238.4 2497.4 1208.0 1758.2 1520.9 600.7 999.2 862.7 4004.0 3912.5 2443.7 2389.0 2794.4 1899.5 1240.0 1696.8 1306.6 713.0 1006.3 740.9

1.005 1.075 1.062 1.024 1.020 1.019 1.022 1.083 0.998 1.005 1.023∗ 0.991 1.063 1.135 1.105 1.098 1.097 1.118 1.171 1.116 1.066 1.032 1.065 1.089

1.117 1.344 1.415 1.092 1.224 1.273 1.082 1.218 1.164 1.064 1.096∗ 1.115 1.276 1.702 1.657 1.255 1.462 1.676 1.277 1.339 1.422 1.096 1.198 1.270

Table A15. Runtime ablation legs behind Figure 7a and c: measured wall-clock periods, one run per row. MR is the run’s KV memory ratio. Config Variant bA T (ms) tok/s/GPU MiniMax-M2.5 8K 7:1 MegaMoE (full FastAFD) DeepEP+DeepGEMM full FastAFD (second session) one microbatch no cross-step overlap Qwen3 8K 7:1

76 76 76 76 76

29.813 53.209 29.893 39.880 43.820

2230.6 1249.8 2224.6 1667.5 1517.6

MegaMoE, reported batch (MR 0.91) 106 35.322 2625.8 DeepEP+DeepGEMM, MR 0.88 102 out of memory MegaMoE, MR 0.80 92 33.542 2400.0 DeepEP+DeepGEMM, MR 0.80 92 67.701 1189.1

(a) FastAFD on GB200 NVL72 Attention pool

(b) CAD Aℓ(n) + DV /L

each attention GPU

NA = r nodes 4r GPUs paged KV cache

RMSNorm + QKV proj

rn tokens per FFN GPU

FlashInfer attention

O proj + add & norm

top-K router

M2N NVLink · one round trip per layer N2M

FFN pool

return

SwiGLU/down

each FFN GPU

FP8 gate/up

FlashInfer attention Q/K/V

fused MegaMoE loop

NF = 1 node 4 GPUs all FP8 experts Wℓ

recv/pack

Fℓ(rn)

Aℓ(n)

each attention GPU

attn out

RMSNorm + QKV proj, O proj + residual, router each FFN GPU

MegaMoE

Figure A1. Operator split boundary comparison. (a) FastAFD keeps projections and routing on the attention pool around an off-path N2M/M2N round trip to fused MegaMoE. (b) Core Attention Disaggregation (CAD) moves projections to the FFN pool, placing its round trip on the attention critical path and shrinking the attention stage.

Table A16. Microbatch-count sweep behind Figure 7b on MiniMax-M2.5 8K: measured wall-clock periods (ms) at bA = 76 (bA = 75 with three microbatches, so every microbatch keeps a captured decode graph). The repeat is an independent 3:1 session on other nodes; the last column uses bA = 72. The bottom row is the reported configuration with two microbatches. Microbatches 1 2 3 4

3:1 3:1 repeat 37.770 29.607 35.061 36.699

Reported cell 29.688

37.720 29.605 35.033 36.634

7:1 7:1, bA = 72 39.902 29.873 35.139 38.797

38.214 28.825 34.774 38.649

29.688 29.827

—

Table A17. Complete 15-cell CAD measurements on MiniMaxM2.5 across 4K–16K. Each context brackets its throughput peak on both sides (kvfull maximizes per-GPU KV batch; aligned matches the selected AFD batch bA ). Throughput counts all GPUs. b T (ms) tok/s/GPU

Ctx

Scope

A:F

4K 4K 4K 4K

kvfull kvfull kvfull kvfull

1:1 2:1 3:1 4:1

154 154 154 154

31.315 36.918 42.285 45.889

2458.9 2781.0 2731.4 2684.7

8K 8K 8K 8K 8K

kvfull kvfull aligned kvfull kvfull

2:1 3:1 3:1 4:1 5:1

82 82 76 82 82

32.796 35.839 35.618 39.677 42.157

1666.9 1716.0 1600.3 1653.3 1620.9

16K 16K 16K 16K 16K 16K

kvfull kvfull aligned kvfull kvfull kvfull

3:1 4:1 4:1 5:1 6:1 7:1

42 42 38 42 42 42

32.668 34.827 34.436 37.187 39.180 41.317

964.2 964.8 882.8 941.2 918.8 889.5

F

R EPRODUCIBILITY

Table A18 lists the run presets behind every measured standard-decode cell. Each run records its preset, rendered environment, and code version: all MiniMax-M2.5 and Qwen3 table runs use one FastAFD version and all Qwen3.5 runs another. vLLM runs use v0.31.0 with the DeepEP low-latency all-to-all, --max-num-batched-tokens 1024, and --block-size 64; on Qwen3.5, a startup hook enables the DeepGEMM block-FP8 linears for which stock v0.31.0 falls back to CUTLASS on Blackwell. The Qwen3.5 vLLM rows use v0.33.0 with the same settings plus --no-enable-prefix-caching and the Triton linear-attention prefill: with prefix caching on, vLLM keeps extra linear-attention state blocks for every request, which FastAFD never allocates. Without prefix caching the client cannot warm the prompts in advance, so the scheduler prefills every request before the first decode step, and all requests then decode together inside the measured window. We use this configuration because it gives vLLM its best measured performance on Qwen3.5, so the comparison and analysis are against the strongest vLLM baseline. Both engines decode attention with the same TRT-LLM-gen FMHA kernels. Both also run the MoE experts as FP8 grouped GEMMs with E8M0 scales: vLLM through its DeepGEMM FP8 MoE backend, and FastAFD by rounding each checkpoint’s 128 × 128 block scales up to the per-32 UE8M0 format of the Mega MoE kernel once at load. Table A18. Run presets behind the measured standard-decode cells (62 FastAFD, 60 vLLM). OSL: output tokens per request. Preset

Cells OSL

FastAFD, MiniMax-M2.5 and Qwen3 afd-cell-v3.4 10 afd-cell-v3.4.1 25 afd-cell-v3.4.2 3 afd-cell-v3.4.3 9 FastAFD, Qwen3.5 afd-cell-q5balign-v2 afd-cell-q5balign-v2.1

1024 1024 1024 1024

13 1024 2 1024

vLLM v0.31.0, all models (Qwen3.5: v0.33.0) vllm-ep-v1.6 9 1024 vllm-ep-v1.7.1 35 1024 vllm-ep-v1.7.2 4 1024 vllm-ep-v1.7.4 4 1024 vllm-ep-v1.7.5 4 1024 vllm-ep-v1.7.8 2 1024 vllm-ep-v1.7.11 2 1024

Related documents

• LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention#1200286
• Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data#1473621
• Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data#1468990
• Diagnosing Zero-Shot Forgetting in CLIP Fine-Tuning: Attention Drift Characterizes Adaptation, Representation Drift Predicts Transfer Degradation#1535597
• You Only Index Once: Cross-Layer Sparse Attention with Shared Routing#259462