One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

O NE -S IDED E XPERT-PARALLEL A LL - TO -A LL FOR M IXTURE - OF -E XPERTS I NFERENCE OVER NVL INK

arXiv:2611.13188v1 [cs.DC] 9 Nov 2026

Bo Li 1 Chulian Zhang 1 Or Biri 1 Anthony Chang 1 Anton Korzh 1 Noam Duvdevani 1 XingFei Xi 1 Daniel Stokes 1 Zhenhuan Chen 1 Ludwig Schneider 1 Artem Y. Polyakov 1 Julien Demouth 1 Chandler Zhou 1

A BSTRACT Expert parallelism (EP) shards the experts of a Mixture-of-Experts (MoE) layer across GPUs, so every MoE layer requires two all-to-all communications, namely dispatch and combine. Compared with training, inference processes relatively few tokens per GPU at a time, and at small batch sizes fixed costs that do not grow with the number of tokens, rather than data movement, dominate the latency. We present NVLinkOneSided, a one-sided EP all-to-all for GPUs within a single NVLink domain. Each source rank pushes its tokens, without first exchanging metadata, into rank-major buffers on the target GPUs, and the grouped GEMMs read them in place instead of from a per-expert copy. We provide two protocols for the synchronization that the communication requires, namely F ENCE and S ENTINEL, for high-throughput and low-latency scenarios respectively. On Blackwell GPUs, NVLinkOneSided completes dispatch and combine 1.2–3.0× faster than DeepEP, with the largest speedups at small batch sizes. The resulting MoE layer is up to 2.4× faster than DeepEP with DeepGEMM. It even outperforms the fused DeepGEMM MegaMoE at up to 128 tokens per GPU, making it still competitive for decoding. We expose NVLinkOneSided through a modular interface, so that it is agnostic to the activation quantization format, decoupled from the MoE compute backend and easy to integrate into inference frameworks. 1

I NTRODUCTION

Mixture-of-Experts (MoE) models activate only a few of their many experts for each token (Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2022). They are commonly served with data-parallel (DP) attention and expert-parallel (EP) MoE layers, a deployment strategy popularized by DeepSeek-V3 (DeepSeek-AI, 2024). In the attention layers, every rank serves its own requests and processes only their tokens. In the MoE layers, every rank holds a subset of the experts, so every MoE layer contains two all-to-all communications. Dispatch sends each token to the ranks that own its experts, and combine returns the expert outputs to the source rank of the token. Compared with training, inference processes few tokens per GPU at a time. Prefill is usually split into chunks of thousands to tens of thousands of tokens (Agrawal et al., 2024; DeepSeek-AI, 2026a), and decoding must meet a servicelevel objective on the time per output token, which bounds its batch to a few hundred tokens per rank. DeepSeek’s DeepInfer (DeepSeek-AI, 2026a), for example, decodes DeepSeek-V4.1-Flash at about 250 tokens per second per user while serving 64 requests per rank; each request adds 1

NVIDIA. Correspondence to: Bo Li <lbo@nvidia.com>, Chulian Zhang <chulianz@nvidia.com>. Preprint.

three draft tokens, so a step processes 256 tokens per rank. At large batches, dispatch and combine are bound by the communication bandwidth, so the goal is to use as much of it as possible and to eliminate unnecessary data movement. At small batches, however, every microsecond matters (Shen et al., 2026). At a few tokens per rank, the bytes a rank sends would take less than a microsecond at the theoretical NVLink bandwidth, yet measured round trips of dispatch and combine take tens of microseconds (§5) in every MoE layer: the time goes to fixed costs that do not grow with the number of tokens, such as synchronization between producers and consumers on different GPUs. In common designs such as NCCL (NVIDIA, a) and DeepEP (Zhao et al., 2025), the sender writes data into an intermediate buffer, from which the receiver copies or regroups it into the layout that the next kernel needs; this extra pass over the data adds latency at small batches and consumes memory bandwidth at large ones. Within one NVLink domain, symmetric memory lets a rank load from and store to a peer’s memory directly, so data can be placed exactly where its consumer reads it. We present NVLinkOneSided, a one-sided EP all-to-all built on this direct access. The routing decisions of a rank alone determine where each of its tokens goes, so dispatch can push the tokens into the memory of the receivers without first sending routing metadata such as token counts or expert

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

ids. The dispatch workspace of each rank, a buffer in symmetric memory, is divided into one region per source rank, so that different ranks do not interfere when writing to the same target rank; within its region, a source rank places each token in one slot, however many experts of the target rank the token selects. We call this layout rank-major. The grouped GEMMs of the MoE layer read the tokens of each expert directly from this workspace through slot indices, without materializing the tokens into per-expert order. Dispatch and combine still need synchronization between producers and consumers, for which we provide two protocols targeting high-throughput and low-latency scenarios respectively. F ENCE publishes per-peer flags after a memory fence. S ENTINEL uses the data itself as the flag. We expose NVLinkOneSided through the general EP communication interface of FlashInfer, MoEEpCommunication, which makes it agnostic to the activation quantization format and easy to integrate with different inference frameworks and MoE backends. The rest of the paper is organized as follows. Section 3 presents the one-sided communication and the rank-major layout that MoE compute reads. Section 4 explains the synchronization protocols and implementation details. Section 5 evaluates NVLinkOneSided and the MoE layer built on it on Blackwell GPUs, analyzes the two protocols and compares both with existing implementations.

2

BACKGROUND

2.1

Mixture-of-Experts and expert parallelism

An MoE layer replaces the dense feed-forward network with E experts and a router (Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2022), which selects k experts for each token and weights their outputs. Under EP the experts are split over P GPUs (ranks), Eℓ = E/P per rank, while attention runs in DP, so each rank holds its own T tokens. The layer then takes three steps. Dispatch sends each token to the ranks that own its experts, each rank runs its local experts on the tokens it received, and combine returns the weighted expert outputs to the source rank of each token, which sums them. Dispatch is a multicast whose destinations depend on the router’s choices, and combine is the reverse, a gather with a reduction. Every rank may send tokens to every other rank, and the amounts depend on the routing, so both are all-to-all exchanges with variable sizes. Figure 1 shows the example used throughout the paper. 2.2

NVLink domains and symmetric memory

A Blackwell GPU has 18 fifth-generation NVLink links of 100 GB/s each, 900 GB/s per direction or 1.8 TB/s in total. NVLink switches fully connect the eight GPUs of an HGX B300 board, and a GB200 or GB300 NVL72 rack forms one

expert experts tokens ids E0

R0 E1

E2

R1 E3

T0 T1 T2 T3 T4 T5 T6

target rank

E0 E2 E0 E1 E1 E3 E2 E3

R0 R1 R0 R0 R0 R1 R1 R1

E1 E2 E2 E3 E0 E3

R0 R1 R1 R1 R0 R1

T =3

T^ = 4

Figure 1. Example used throughout the paper: two ranks, experts E0, E1 on R0 and E2, E3 on R1, and top-2 routing. R0 holds tokens T0–T3 and R1 holds T4–T6. Beside each token are its two expert ids and, through the routing (arrow), the target rank of each: both experts of T1 are on R0, both experts of T3 and T5 are on R1, and every other token reaches both ranks. The braces on R1 mark its number of tokens, T = 3, and the largest number of tokens of any rank, T̂ = 4 (Table 2).

72-GPU NVLink domain. Beyond the domain, each GPU communicates through one RDMA NIC, a ConnectX-8 of 800 Gb/s, that is, 100 GB/s per direction, roughly an order of magnitude less (Aubrey & Stam, 2025). Within the domain, GPUs address each other’s memory directly. With CUDA’s Virtual Memory Management, each rank allocates physical memory, exports a handle, and maps every peer’s allocation into its own address space; the result is a symmetric memory region in which a kernel accesses peer memory with ordinary loads, stores and atomics (NVIDIA, 2025; PyTorch Contributors). RDMA offers remote access through a different interface: the initiator, a CPU thread or, with GPU-initiated networking, a GPU thread (Markthub et al., 2022), posts work requests to the NIC, names registered regions by key, and is notified of completion through completion queues or a write with immediate data (InfiniBand Trade Association, 2020); transfers are efficient for messages of kilobytes. Table 1 lists the differences that matter for an all-to-all of small, data-dependent messages. NVLinkOneSided moves data into and out of peer memory with 16-byte stores and loads from many threads and detects completion through the GPU memory model (§4). A remote store does not wait for the peer: the thread moves on at once, and the data reach the peer later, possibly out of order with other stores. GPU memory is weakly ordered (Lustig et al., 2019): a consumer on another GPU that sees a producer’s write is guaranteed to see the producer’s earlier writes only if the producer ordered them with

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink Table 1. Remote access within an NVLink domain and over RDMA (Blackwell, per GPU). Initiator Transfer size Ordering Arrival in a message

NVLink domain

RDMA NIC

any thread, load/store store (≤32 B) or TMA copy PTX fences, system scope whole 32-bit words

work request to the NIC message, KB scale per queue pair unspecified

Table 2. Notation and the corresponding parameter names in the implementation. Symbol

Meaning

Name in the implementation

P k E Eℓ T T̂

EP ranks experts per token experts of the layer experts per rank tokens of a rank in the step largest T across ranks

ep_size top_k num_experts num_local_experts local_num_tokens runtime_max_tokens_per_rank

a system-scope release and the consumer reads with acquire semantics. In addition, each aligned 32-bit word stored to peer memory is observed whole or not at all (NVIDIA, 2026d). NCCL’s LL protocol relies on a stronger guarantee: an 8-byte store of 4 bytes of data and a 4-byte flag arrives whole (NVIDIA, a; Hu et al., 2025). RDMA offers no counterpart: the order of placement within an RDMA Write is undefined, and the responder may rely on the data only after a later completion, such as a write with immediate data (InfiniBand Trade Association, 2020; Recio et al., 2007).

3

O NE -S IDED C OMMUNICATION WITH R ANK -M AJOR L AYOUT

This section describes one-sided communication, the rankmajor layout and how MoE compute reads it. Table 2 summarizes the notation. 3.1

One-sided communication

Each rank routes its own tokens, so it knows locally which target ranks each token must reach. In dispatch, the source rank pushes each token directly, at a slot that it chooses itself, into the dispatch workspace of every target rank, which lies in symmetric memory, and records the target rank and slot of each selection (Figure 2). No metadata, such as token counts or expert ids, travels ahead of the data. MoE compute leaves the partial results in a second workspace, the combine workspace, from which combine returns them. 3.2

Rank-major layout

Source ranks push into a target rank concurrently, so their stores must not interfere. Slots drawn from one shared pool would need atomics on a remote counter, or offsets

precomputed from the counts of all source ranks, which would require a metadata exchange. Instead, each dispatch workspace has one region per source rank, which only that source rank writes: it takes the next slot by atomically incrementing its slot counter for that target rank, and its slots form a dense prefix of the region. We call this layout rankmajor. A region has T̂ slots, which every rank knows because inference engines with DP attention share the batch size information every step; the dispatch workspace is sized at initialization for the maximum number of tokens a rank can process. A token goes to each target rank once, however many experts of that rank it selects, with all k expert ids and router weights in its slot. Some grouped GEMM implementations, such as DeepGEMM (DeepSeek-AI, 2025), read the tokens of each expert contiguously, so the corresponding dispatch delivers an expert-major layout instead, with one slot per selected expert.1 The rank-major layout has two benefits over it: De-duplication. A token is sent at most min(k, P ) times instead of k times as in a per-expert transfer; for example, with k = 16 on eight GPUs, it sends at most half the bytes, whatever the routing. The saving is larger when P is small relative to k, or when a token chooses several experts located on the same rank. Smaller workspaces. The rank-major dispatch and combine workspaces each hold P T̂ slots. An expert-major buffer must hold every selection that a rank may receive. If every local expert may receive every token, as in the lowlatency kernels of DeepEP V1, it needs Eℓ times as many slots. Since a token selects at most min(k, Eℓ ) experts of one rank, DeepEP V2.5 still needs min(k, Eℓ ) times as many. For DeepSeek-V4.1-Flash on eight GPUs with up to 4096 tokens per rank, the hidden states take 160 MiB in our dispatch workspace (8 × 4096 slots of 5120 bytes in MXFP8) and 320 MiB in our combine workspace (BF16), besides a few MiB for scaling factors, expert ids and router weights. DeepEP V2.5 with DeepGEMM needs a 242 MiB communication buffer, shared by dispatch and combine, and an expert-major GEMM input whose hidden states alone take 1020 MiB, around k = 6 times the 160 MiB of our dispatch workspace (the excess is due to alignment padding). All these buffers grow linearly with the EP size, so memory becomes more pressing at large EP. 1

The low-latency kernels of DeepEP V1 push each token once per selected expert into an expert-major buffer. DeepEP V2.5 pushes each token once per target rank into a rank-major communication buffer; with do_expand, which DeepGEMM requires, it then copies each token once per selected local expert into an expert-major GEMM input.

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink target rank slot T0

R0

T1 T2 T3 T4

R1

T5 T6

R0 0 R1 0 R0 1 −1 −1 R0 2 R1 1 R1 2 −1 −1

Dispatch

MoE Compute dispatch combine workspace workspace T0 T0 T1 from to T1 R0 R0 T2 T2

R0 0 R1 0 R1 1 −1 −1 R0 1 R1 2

T4 T6

T4 T6

from R1

to R1

T0 T2 T3

from R0

to R0

T0 T2 T3

T4 T5 T6

from R1

to R1

T4 T5 T6

Combine

Σ

T0

Σ

T1

Σ

T2

Σ

T3

Σ

T4

Σ

T5

Σ

T6

Figure 2. Data path for the routing of Figure 1, with the F ENCE synchronization protocol. Dispatch: each source rank stores its tokens into its own region of the dispatch workspaces of the target ranks and records the target rank and slot of each selection; T1 selects E0 and E1, both on R0, so it takes one slot there and its second selection records no slot (−1). MoE compute turns the tokens from each source rank into partial results at the same slots of the combine workspace. Combine: each source rank loads the partial results from the target ranks and slots recorded in dispatch and adds them. White slots are empty.

3.3

MoE compute on the rank-major layout

An MoE layer on the rank-major layout runs dispatch, the MoE compute kernels (an indexing kernel, the grouped GEMMs FC1 and FC2, and finalize) and combine (Figure 3). The tokens stay at their slots of the dispatch workspace, and the grouping by expert is expressed as slot indices. Indexing. A small indexing kernel selects the ids of the local experts, skips the slots marked invalid (details in §4), and builds per-expert lists of slot indices, plus the inverse map, from each slot to the FC2 output rows of its token, for finalize. FC1 gathers its input through these indices, so no separate pass permutes the received tokens before FC1. FC1 and FC2. FC1 applies the up and gate projections and the activation of each expert to its tokens, and FC2 applies the down projection of the expert to the result. FC1 loads its input by index directly from the dispatch workspace: the Prims-TS grouped GEMM of FlashInfer (FlashInfer Contributors, 2026) gathers it with TMA gather4 instructions, each of which loads 128-byte slices of four non-contiguous slots, and never loads empty slots. FC1 writes its output grouped by expert, and FC2 reads it in that order. Finalize and combine. Finalize gathers the FC2 outputs of each token through the inverse map, scales them by the router weights, sums them over the local experts of the token, and writes the result into the combine workspace that the communication layer exposes, at the slot of the token. The finalized partial results are thus back in the rank-major layout of dispatch, so combine needs no metadata either: the source rank pulls the partial results of each token from the

combine workspaces of its target ranks, at the slots recorded in dispatch, and adds them.

4

I MPLEMENTATION AND S YNCHRONIZATION P ROTOCOLS

Kernels in one stream are ordered: a kernel sees every write to local memory made by the kernels before it. Remote memory accesses have no such guarantee: the stores of a source rank reach a peer while the kernels of the peer run, in no particular order (§2.2). One-sided dispatch and combine therefore need two producer–consumer synchronizations to prevent races: • End of dispatch. Before a target rank starts MoE compute, every source rank must have finished storing its tokens into the dispatch workspace of the target rank. • Start of combine. Before a source rank reads the partial results of its tokens, every target rank must have finished MoE compute and written the results into its combine workspace. NVLinkOneSided provides two synchronization protocols: F ENCE for high-throughput and S ENTINEL for lowlatency scenarios. They differ in how a source rank assigns slots and how a target rank learns that its slots are complete (Figure 4). 4.1

F ENCE synchronization

Each CUDA thread block handles the dispatch and the combine of one token, and the blocks of a rank run in paral-

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink MoE Compute Dispatch dispatch workspace

R0

Indexing

FC1 (Grouped GEMM)

FC2 (Grouped GEMM)

FC1 output

FC2 output

index lists

0 from 1 R0 2 3

T0 T1 T2

E0 E2 E0 E1 E1 E3 −1 −1

E0 0 1 5

E0 T0 up/gate proj T1 act T6

4

T4 T6

E1 E2 E0 E3 −1 −1 −1 −1

E1 1 2 4

E1 T1 up/gate proj T2 act T4

from 5 R1 6 7

down proj

down proj

Finalize combine workspace

E0 T0 T1 T6

Σ

E1 T1 T2 T4

Σ

Σ Σ

Σ

T0 T1 T2

0 1 2 3

T4 T6

4 5 6 7

Combine

to R0

to R1

Figure 3. The MoE layer on R0, detailing Figure 2; MoE compute comprises indexing, FC1, FC2 and finalize, and slots are numbered 0–7 across the two regions. The indexing kernel turns the expert ids of the received slots into per-expert lists of slot indices (empty slots 3, 6 and 7 carry invalid ids); FC1 gathers tokens through these lists and multiplies them by the expert weights, FC2 multiplies the FC1 output by its own weights, and finalize adds the outputs of each token into its slot of the combine workspace. (a) Fence

R1

(b) Sentinel

0 from 1 R0 2 3

T0 T2 T3

E0 E2 E1 E3 C = 3 E2 E3 −1 −1

0 1 2 3

T0

0

T4 T5 T6

E1 E2 E2 E3 C = 3 E0 E3 −1 −1

0 1 2 3

T4 T5 T6

from 1 R1 2 3

Critical path of Dispatch

T2 T3

E0 E2 T1 not sent −1 −1 to R1 E1 E3 E2 E3 E1 E2 E2 E3 E0 E3 past R1's −1 −1 batch

Fence

push tokens write counts fence sync flags MoE Compute push tokens saved Sentinel poll slots MoE Compute time

Figure 4. The dispatch workspace of R1 at the end of dispatch. (a) F ENCE gathers the tokens of each source rank into a prefix of its region, so each source rank sends its slot count C; the slots past the count get invalid expert ids (−1). (b) S ENTINEL pushes token i of each source rank into slot i of its region, so every slot signals its own arrival; a slot without a token keeps the sentinel payload (hatched) and gets invalid expert ids. Bottom: the critical path of dispatch until MoE compute starts; S ENTINEL shortens it, which pays off at small batches.

lel (Algorithm 1). A block takes the slot of its token in each target rank by atomically incrementing the slot counter for that rank. When a block has sent its token, it increments another local counter of finished blocks, and the last block to finish synchronizes for the whole rank, so the rank executes one system-scope fence per step. This block publishes the slot counts to their target ranks, executes a system-scope release fence (fence.release.sys), and writes a flag to every peer rank. It spins until the flags of all peer ranks have arrived, which ends the dispatch kernel. A small kernel marks the expert ids past each published count invalid, so that the slots past the prefix take no part in MoE compute. Combine synchronizes in the same way, except that the first block alone writes the flags while ev-

ery block spins on the flags of all peers and executes an acquire fence (fence.acquire.sys) before it pulls the finalized partial results of its token from the target ranks and slots recorded in dispatch. 4.2

S ENTINEL synchronization

F ENCE ends every dispatch with a serial chain: atomic counting for the slot indices, a system-scope fence, and the publishing of and spinning on completion flags. S ENTINEL shortens the critical path by letting the data signal their own arrival. Source ranks push the tokens into landing slots, each 32-bit word of which is initialized with a sentinel value, a bit pattern that the data do not take or that the producer avoids: BF16 payload words, for example, use 0xFFFFFFFF, two BF16 NaNs, and the producer stores such a word as two 0x7FFF NaNs instead. A target rank polls the words of a landing slot; since each word arrives whole (§2.2), a slot whose words all differ from the sentinel is complete without a fence. The target rank copies it into the same slot of its dispatch workspace, which MoE compute reads as with F ENCE, and resets it to the sentinel. Unlike NCCL’s LL protocol (NVIDIA, a), which interleaves flag words with the data, S ENTINEL adds no flag words: the data themselves are the flags.2 The routing decides which slots receive a token, so a target rank must tell a slot still in flight from one that will never be written. Token i of source rank s therefore always lands in slot i of region s, and for each of its T̂ slots every source rank writes expert ids to every rank, invalid ones (−1) where no token goes (Figure 4b): every expert-id slot arrives and tells whether a payload follows. Expert-id slots use the sentinel 0x88888888, which differs from −1. 2

Several code bases call this Lamport synchronization, after Lamport’s happened-before relation (Lamport, 1978): the arrival of the data itself orders the producer’s stores before the consumer’s loads.

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

Algorithm 1 NVLinkOneSided F ENCE protocol

Algorithm 2 NVLinkOneSided S ENTINEL protocol

Thread block: token i on source rank s Wd [r], Wc [r]: dispatch and combine workspaces of rank r C[r][s], F [r][s]: slot count and flag of source rank s on rank r cnt[t]: local slot counter for target rank t, initially 0 done: local counter of finished blocks, initially 0 f : flag value of the step 1: Dispatch 2: D ← target ranks of token i 3: for all t ∈ D do 4: jt ← atomicAdd(cnt[t], 1) ▷ pick slot on t 5: Wd [t][s][jt ] ← token i ▷ push token 6: end for 7: if atomicAdd(done, 1) = T − 1 then 8: C[r][s] ← cnt[r] for all r ▷ write slot counts 9: fence.release.sys 10: F [r][s] ← f for all r ▷ sync via completion flags 11: wait until all F [s][r] = f 12: end if 13: for all r, j ≥ C[s][r] do ▷ sanitize expert ids 14: Wd [s][r][j].expert_ids ← −1 15: end for

Thread block: token slot i < T̂ on source rank s L[r], I[r], M [r]: landing, expert-id and combine slots of rank r Wd [r], Wc [r]: dispatch and combine workspaces of rank r σ: sentinel, initial value of L, I and M 1: Dispatch 2: D ← target ranks of token i, ∅ if i ≥ T 3: L[t][s][i] ← token i for all t ∈ D ▷ push token 4: I[r][s][i] ← expert ids of token i (−1 if r ∈ / D) for all r 5: for all source ranks r do 6: wait until no word of I[s][r][i] equals σ ▷ poll expert ids 7: if I[s][r][i] ̸= −1 then 8: wait until no word of L[s][r][i] equals σ ▷ poll token 9: Wd [s][r][i] ← L[s][r][i]; L[s][r][i] ← σ ▷ copy, reset 10: end if 11: end for

16: Combine 17: if first block then 18: F [r][s] ← f + 1 for all r ▷ sync via completion flags 19: end if 20: wait until all F [s][r] = f + 1 21: fence.acquire.sys 22: pt ← Wc [t][s][jt ] forPall t ∈ D ▷ pull tokens in parallel ▷ reduction 23: output of token i ← t∈D pt

Combine pushes too: once the MoE compute of a target rank has finished, it stores the partial result of slot i of region s into a combine slot on source rank s without waiting for its peers, and the block of token i polls these slots like landing slots and adds the partial results. The kernel that polls a slot consumes it and resets it to the sentinel. A rank writes a landing or combine slot on a peer again only after receiving data that the peer sends in a later kernel, the partial result of the landing slot or the expert ids of the next step, so one set of these slots suffices. Expertid slots also reach ranks that send nothing back, so they alternate between two sets: a rank finishes dispatch only after the expert ids of every rank have arrived, so no rank starts a step before every rank has finished the step before last. S ENTINEL’s costs grow with the batch: each rank writes, polls and resets P T̂ expert-id slots per step, and every received word is checked, copied and reset, which shrinks the advantage of S ENTINEL (§5.2). 4.3

Kernel implementation details

NVLinkOneSided is implemented as a set of CUDA kernels behind FlashInfer’s MoEEpCommunication interface. Dispatch moves tokens after quantization: MoE com-

12: Combine 13: M [r][i][s] ← Wc [s][r][i] for all r with I[s][r][i] ̸= −1 ▷ push partial results 14: I[s][r][i] ← σ for all r ▷ reset expert-id slots 15: for all t ∈ D do ▷ poll partial results 16: wait until no word of M [s][i][t] equals σ 17: pt ← M [s][i][t]; M [s][i][t] ← σ ▷ load, reset 18: end for P 19: output of token i ← t∈D pt ▷ reduction

pute follows dispatch, so the quantized tokens naturally take part in the computation. Dispatch therefore supports every quantization format of the MoE compute backends (e.g., BF16, FP8, MXFP8 and NVFP4) and sends the payloads of each token: the quantized hidden states, their scaling factors, and the expert ids and router weights from the routing. MoE compute outputs BF16, so combine usually transfers BF16. The kernels with the F ENCE protocol can also quantize the partial results to FP8 before combine and dequantize them before the reduction, which saves combine bandwidth at a potential loss of precision. Both kernels keep as many bytes in flight over NVLink as they can, through two kinds of parallelism: • Data-level parallelism (DLP). Each thread uses the widest vectorized loads and stores that the payload alignment allows, 16 bytes. • Instruction-level parallelism (ILP). In dispatch, a thread loads each vector from local memory once and stores it to all target ranks of the token in an unrolled loop, so the remote stores of the token are in flight together. In combine, a thread first loads the vectors of all target ranks of the token into registers and then adds them pairwise in a tree, which keeps the dependency chain short.

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink NVLinkOneSided Sentinel

Table 3. Configuration of the MoE layers of the tested models, with the bytes per token and target rank.

Hidden size 5120 Experts, top-k 384, 6 Expert intermediate size 2304 Dispatch bytes (MXFP8) 5328 Combine bytes (BF16) 10240 a Latent MoE; the original hidden size is 7168.

a

3584 896, 16 3072 3824 7168

round trip (µs)

Kimi-K3

DeepSeek-V4.1-Flash 500

speedup

DeepSeek-V4.1-Flash

3×

5

E VALUATION

We measure the all-to-all alone (§5.2), the whole MoE layer with it (§5.3), and both in a larger NVLink domain (§5.4). 5.1

Setup

Sections 5.2–5.3 run on eight NVLink-connected B300 GPUs and §5.4 on a GB200 NVL72 rack to evaluate large EP. We test the routed-expert layers of DeepSeek-V4.1Flash (DeepSeek-AI, 2026b) and Kimi-K3 (Moonshot AI, 2026) (Table 3) at 1 to 4096 tokens per rank, with perfect routing: the k selections of each token go to consecutive ranks and spread evenly over their experts. Our MoE layer pairs NVLinkOneSided with the PrimsTS grouped GEMMs of §3.3, with MXFP4 weights and MXFP8 activations; where a single number stands for NVLinkOneSided, it is that of the faster protocol. The baselines are DeepEP V2.5 (Zhao et al., 2025) with the settings of its own tests; DeepGEMM MegaMoE (DeepSeekAI, 2025), one persistent kernel that overlaps NVLink transfers with computation; and DeepEP + DeepGEMM (DE+DG), the unfused baseline of MegaMoE’s test. Appendix B describes the measurement. 5.2

All-to-all performance

NVLinkOneSided is faster than DeepEP at every batch, most where fixed costs dominate (Figure 5): by 1.75–3.00× up to 128 tokens per rank and by 1.17–1.63× from 256. At one token per rank, the round trip on DeepSeek-V4.1Flash takes 17.4 µs with S ENTINEL, 27.6 µs with F ENCE and 47.9 µs with DeepEP; at 4096 tokens, F ENCE reaches 68–71% of the 900 GB/s unidirectional NVLink bandwidth.

Kimi-K3

200 100 50 20

2× 1× 1

The size of the rank-major workspaces depends only on the number of tokens, so the whole MoE layer can be captured in a CUDA graph, in which dispatch, the MoE compute kernels and combine use Programmatic Dependent Launch (PDL) (NVIDIA, 2026a) to overlap each kernel with its predecessor. Every synchronization is a point where a failed peer stalls the others, so every wait has a timeout and traps with an error message instead of hanging the engine.

DeepEP

NVLinkOneSided Fence

4

16 64 256 1K 4K 1 4 tokens per rank

16 64 256 1K 4K

Figure 5. NVLinkOneSided completes dispatch and combine faster than DeepEP at every batch. Round trip on eight B300 GPUs; bottom: speedup over DeepEP. 1 token per rank

256 tokens per rank

Dispatch

Dispatch

Combine

Combine

Fence Sentinel Fence Sentinel 0

5

10

15 0 10 critical path (µs)

launch, PDL waits election, fence, flags remote loads, reduce

routing wait for peers copy-out, reset

20

30

remote stores sanitize other

Figure 6. S ENTINEL shortens the work around the wait for peers, not the wait. Critical path inside the dispatch and combine kernels, from IKET timestamps; DeepSeek-V4.1-Flash on eight B300 GPUs. Launch, PDL waits include F ENCE’s counter-reset kernel; election, fence, flags: F ENCE’s serial chain; sanitize: F ENCE’s kernel that invalidates the expert ids past the slot counts, mostly overlapped with dispatch; copy-out, reset: S ENTINEL’s handling of received slots; other: kernel exit and S ENTINEL’s local reduction.

F ENCE or S ENTINEL. S ENTINEL is faster at decoding batches and F ENCE at large ones, with the crossover at 256 tokens per rank (Figure 5, bottom). In-Kernel Event Tracing (IKET) (NVIDIA, 2026b) shows why (Figure 6). At one token per rank, both protocols wait about equally long for their peers, and S ENTINEL saves the work around the wait: its dispatch skips F ENCE’s serial chain and extra kernels and takes half as long, and its combine pushes each partial result when it is ready instead of waiting for all peers and pulling with remote loads. As the batch grows, the transfers dominate with either protocol, while S ENTINEL’s per-slot work, copying out and resetting every received slot and running more combine blocks than fit on the GPU at once, cancels its saving. Payload precision. Low-precision payloads pay off at large batches, where F ENCE is used (Figure 7): at 4096

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink Dispatch

BF16 FP8

speedup

3× 3.5× 2×

DeepSeek-V4.1-Flash

2.0×

1.9× 1

4

16 64 256 1K 4K 1 4 tokens per rank

1.5 1.0 2.0 1.5 1.0 16

32

16 64 256 1K 4K

Figure 7. Low-precision formats accelerate dispatch and combine at large batches; dashed: the byte ratio, their theoretical limit. NVLinkOneSided with F ENCE, DeepSeek-V4.1-Flash on eight B300 GPUs; top: latency of each phase; bottom: speedup over BF16.

tokens per rank, NVFP4 makes dispatch 3.0× faster than BF16 for a byte ratio of 3.5×, and MXFP8 1.8× for 1.9×, while at one token all formats take the same time. FP8 combine halves the bytes but is only 1.4× faster, as a kernel must first read and quantize each BF16 partial result; with per-element E4M3, it shows no statistically significant accuracy loss on downstream tasks. 5.3

2.0

8

1×

MoE layer

Table 4 lists the layer and its stages. Although built from separate kernels, our layer is faster than the fused MegaMoE up to 128 tokens per rank, by up to 30%; from 256 tokens MegaMoE leads, by up to 35% at 4096, as it hides the communication behind the grouped GEMMs. Our layer is faster than DeepEP + DeepGEMM at every batch, by up to 2.4×, mostly through dispatch and combine. Skewed routing. Real routing is imbalanced, and the rank that holds the popular experts limits the layer. We model this with Zipf routing (Table 5): each token draws k distinct experts with probability proportional to i−α , where i is the expert’s position in a random order (α = 0: uniform). At one token per rank, the grouped GEMMs are bound by loading expert weights, so skew, which activates fewer experts, can even make the layers faster. At 1024 tokens per rank, the busiest rank computes up to 1.9 times as many token–expert routing pairs as a rank under perfect routing (load imbalance in Table 5), and its grouped GEMMs slow all three layers by 50–70%. Our communication grows less: a token reaches a target rank once, however many of the rank’s experts it selects, so the bytes a rank receives follow its tokens, at most P/ min(k, P ) times as many as under perfect routing, rather than its token–expert routing pairs. At α = 1.25, dispatch and combine of the busiest rank take 160 and 135 µs, against 267 and 400 µs with DeepEP

Kimi-K3

T = 128

20 10

DeepEP

NVLinkOneSided Sentinel

relative to 8 ranks

BF16 MXFP8 NVFP4

200 100 50

NVLinkOneSided Fence

Combine

T =1

time (µs)

500

64 8 16 expert-parallel ranks

32

64

Figure 8. At one token per rank, S ENTINEL’s round trip grows least from 8 to 64 GPUs. All-to-all round trip on NVL72 relative to the same implementation on 8 GPUs, at a fixed number of tokens per rank.

+ DeepGEMM, which copies each token once per selected local expert. 5.4

Large EP scaling

A larger EP group gives each GPU fewer experts, which frees memory and, at small batches, where the grouped GEMMs are bound by loading weights, reduces the weights each GPU loads. We scale EP from 8 to 64 GPUs on one NVL72 rack at a fixed number of tokens per rank. The all-to-all slows only slightly (Figure 8). At one token per rank on DeepSeek-V4.1-Flash, S ENTINEL’s round trip stays at 16.6–17.8 µs, as it has no per-peer handshake, while F ENCE’s grows from 43.0 to 47.6 µs, as it publishes and awaits one flag per peer. Kimi-K3 scales worse, as from 16 GPUs on its tokens (k = 16) reach 16 ranks instead of eight. The MoE layer gains at intermediate batches (Table 6): on 64 GPUs, a rank holds six experts of DeepSeek-V4.1-Flash instead of 48, and at 128 tokens per rank our layer takes 103 instead of 182 µs. At one token per rank, a rank activates about k experts at any EP size, and at large batches it processes T k token–expert routing pairs at any EP size, so large EP does not reduce its work in either case. Our layer stays 23–26% below MegaMoE at one token per rank, while MegaMoE gains more at intermediate batches and leads from 32 tokens per rank on 64 GPUs instead of from 128 on eight.

6

D ISCUSSION

Advantages and limitations. NVLinkOneSided performs well across the board: its all-to-all is faster than DeepEP at every batch, and its layer, built from separate kernels, is faster than the fused MegaMoE up to 128 tokens per rank on eight GPUs (§5). Keeping the all-to-all separate from MoE compute makes it adaptable: it carries any

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink Table 4. MoE layer latency (µs) on eight B300 GPUs: layer time and dispatch / MoE compute / combine stages (MegaMoE: one kernel); fastest layer in bold. NVLinkOneSided F ENCE Tokens/rank MoE layer stages DeepSeek-V4.1-Flash 1 46.0 16 79.6 128 188.5 256 208.1 1024 353.8 4096 1185.2

NVLinkOneSided S ENTINEL MoE layer stages 36.7 70.5 183.3 206.5 356.9 1198.6

12.8 / 19.6 / 14.4 13.3 / 49.9 / 17.2 16.3 / 150.0 / 23.1 22.2 / 153.4 / 33.6 55.2 / 201.3 / 99.2 187.2 / 647.9 / 351.6

MoE layer

8.3 / 23.9 / 9.8 8.5 / 54.7 / 12.9 13.5 / 150.6 / 21.5 20.5 / 154.9 / 33.8 60.8 / 206.0 / 96.2 197.4 / 665.7 / 343.3

74.7 122.0 235.9 251.2 409.6 1290.2

DE+DG stages

MegaMoE MoE layer

21.4 / 26.0 / 27.2 22.0 / 68.7 / 31.4 25.8 / 168.7 / 41.4 29.8 / 170.6 / 50.8 72.9 / 215.8 / 121.3 240.1 / 612.1 / 429.6

49.8 90.3 188.7 194.1 255.3 766.1

99.7 26.5 / 37.8 / 35.3 191.2 23.5 / 129.4 / 38.2 430.8 27.0 / 346.9 / 56.8 482.1 31.2 / 374.0 / 76.7 976.3 78.1 / 678.5 / 222.3 3091.1 269.8 / 2006.4 / 817.7

60.4 148.6 358.5 370.2 535.8 1901.9

Kimi-K3 1 16 128 256 1024 4096

42.1 8.8 / 28.1 / 13.5 122.0 9.4 / 106.5 / 12.5 346.4 13.7 / 314.8 / 21.0 393.2 20.3 / 343.4 / 32.6 732.1 61.5 / 580.7 / 98.2 2601.5 195.8 / 2084.3 / 345.3

47.7 13.1 / 23.5 / 15.1 128.4 13.4 / 100.9 / 15.0 351.1 17.2 / 313.4 / 21.5 396.1 22.8 / 342.2 / 32.0 719.5 54.9 / 561.6 / 102.9 2553.6 181.9 / 2023.0 / 360.2

Table 5. MoE layer under Zipf routing with exponent α (α = 0: uniform) on eight B300 GPUs, latency in µs. Imba. (load imbalance) is the ratio of the token–expert routing pairs that the busiest rank computes to those that each rank computes under perfect routing; 1 means balanced. Communication is the dispatch and combine of the busiest rank. Ours: NVLinkOneSided with the faster protocol + Prims-TS; DE+DG: DeepEP + DeepGEMM.

Tokens/rank

DeepSeek-V4.1-Flash Kimi-K3 Communication MoE layer Communication MoE layer α Imba. Ours DeepEP Ours DE+DG MegaMoE Imba. Ours DeepEP Ours DE+DG MegaMoE

1

0 0.5 1 1.25

1.50 1.50 1.50 1.67

16.1 15.8 15.8 15.9

1024

0 0.5 1 1.25

1.02 119.5 1.29 125.7 1.69 146.4 1.91 160.4

47.6 49.4 52.6 51.6

52.2 57.8 52.4 52.3

97.5 105.2 99.3 97.9

68.3 73.0 67.6 69.2

1.38 1.25 1.38 1.62

22.4 28.2 30.0 22.1

178.4 414.9 199.8 445.4 242.3 557.0 267.1 619.4

461.1 499.5 626.7 696.0

255.6 303.9 374.4 411.5

1.01 143.0 1.19 131.3 1.55 134.2 1.70 134.7

58.0 68.5 65.4 62.2

84.6 81.0 71.1 66.4

148.0 146.1 133.4 129.4

114.2 106.8 96.6 95.4

291.7 717.6 306.7 852.3 373.0 1075.3 400.0 1184.5

968.6 1096.7 1352.6 1477.0

538.9 638.9 814.0 918.4

quantization format and pairs with various MoE compute backends behind one interface, and it leaves room for features of the communication alone, such as fault tolerance. Its main limitation is that the EP group must fit in one NVLink domain: both protocols rely on direct loads and stores to peer memory and on 32-bit words that arrive whole, which RDMA does not offer (§2.2). Its dispatch workspace holds one region per rank of the group, so its memory grows with the EP size.

and formats follow the GEMM’s tiles, so a new grouped GEMM or precision needs a new implementation. In future work, we plan to carry the one-sided design into such a fused operator, pushing tokens without metadata into a rank-major layout and detecting their arrival from the data, so that communication also overlaps with computation at large batches.

MegaMoE. DeepGEMM MegaMoE fuses dispatch, both grouped GEMMs and combine into one persistent kernel that exchanges token indices first and then pulls the tokens into expert-major blocks sized for its GEMM tiles. Its strength is overlap: within one kernel, NVLink transfers proceed during Tensor Core work, which separate kernels cannot do, so MegaMoE leads from 256 tokens per rank (§5.3). Its weaknesses are the metadata exchange ahead of the data and its coupling: layout, transfer granularity

MoE systems. Expert parallelism was introduced with GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2022). DeepSpeed-MoE (Rajbhandari et al., 2022) co-designs the model with an inference system that combines expert, tensor and data parallelism with hierarchical all-to-all; FasterMoE (He et al., 2022) counters load imbalance by replicating popular experts and overlaps communication through fine-grained scheduling; Tutel (Hwang

7

R ELATED W ORK

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink Table 6. MoE layer latency (µs) of DeepSeek-V4.1-Flash on one GB200 NVL72 rack, by EP size; Ours: the faster protocol. Fastest layer in bold. EP size (GPUs) T

8

16

32

64

35.4 78.1 45.7

35.4 78.0 46.3

35.7 79.1 47.2

36.9 82.3 49.7

16 Ours 66.4 51.9 DE+DG 124.5 100.6 MegaMoE 83.4 63.5

47.7 88.9 58.6

49.0 90.9 56.0

1 Ours DE+DG MegaMoE

128 Ours 181.7 124.7 120.9 102.6 DE+DG 237.7 169.3 136.2 126.8 MegaMoE 179.3 121.0 91.2 86.6

et al., 2023) switches parallelism and pipelining at runtime; and MegaBlocks (Gale et al., 2023) computes experts as block-sparse operations, so that no token is dropped. MegaScale-Infer (Zhu et al., 2025) places attention and experts on separate GPUs, possibly of different types, hides the traffic between them with a ping-pong pipeline over micro-batches, and carries it with an RDMA library from the attention nodes to the expert nodes. Expert-parallel communication. DeepEP V1 (Zhao et al., 2025) has two kernel families. Its high-throughput kernels, for training and prefill, first exchange token counts and then forward the tokens between the NVLink and RDMA domains; its low-latency kernels, for decoding, send each token over RDMA, later also over NVLink, into per-expert receive buffers without a separate count exchange, and are compatible with CUDA graphs. DeepEP V2 rebuilds both on NCCL’s GPU-Initiated Networking (GIN) (Hamidouche et al., 2025) behind one buffer interface, with an expanded per-expert output layout for grouped GEMMs. Perplexity’s kernels (Licker et al., 2025b;a) split sending and receiving into separate kernels and deliver per-expert outputs, through NVSHMEM across nodes and direct stores within a node. NCCL EP (Goldman et al., 2026) offers both modes on NCCL’s device API; its low-latency mode can also return the tokens in a rank-major layout. UB-X (NVIDIA, 2026c), an experimental NVLink library in NCCL, computes dispatch offsets from a routing matrix that every rank holds and provides a combine that pushes with sentinel synchronization. Collective protocols. NCCL’s LL and LL128 protocols (NVIDIA, a; Hu et al., 2025) and MSCCL++’s LL packets (Shah et al., 2025) pair data with flag words so that receivers detect arrival without fences, at the cost of transferring the flags; NVSHMEM’s put-with-signal instead pairs each transfer with a separate signal word (NVIDIA, b). Shen

et al. (2026) measure memory barriers at over a microsecond each on GB200 and build barrier-free AllReduce from LL and sentinel synchronization on NCCL’s device API. In dispatch and combine, sentinel synchronization meets two new problems: quantized payloads may take the sentinel’s bit pattern, and the slots that receive tokens on a rank depend on the routing (§4.2). In-band completion over onesided writes also appears in RDMA systems, which validate remote objects or messages by per-cache-line versions or by the arrival of their last bytes (Dragojević et al., 2014; Kalia et al., 2014), and HPC separates completion from data transfer through MPI-3 epochs (Mes, 2015) and Notified Access (Belli & Hoefler, 2015).

Overlap and fusion. Comet (Zhang et al., 2025) overlaps expert computation with communication at tile granularity, and TileLink (Zheng et al., 2025) generates overlapped kernels from tile-level primitives. FlashDMoE (Aimuyo et al., 2025) and DeepGEMM MegaMoE (DeepSeek-AI, 2025) fuse dispatch, expert computation and combine into one persistent kernel, and FarSkip-Collective (Anonymous, 2026) changes the model so that blocking collectives overlap with computation. These reduce how much of the communication is exposed; NVLinkOneSided reduces the cost of the communication itself, which matters most in decoding, where there is little computation to hide it behind.

8

C ONCLUSION

NVLinkOneSided implements dispatch and combine as one-sided communication within an NVLink domain: each source rank pushes its tokens into rank-major workspaces using only its own routing information, MoE compute reads them through slot indices, and two protocols, S ENTINEL for small batches and F ENCE for large ones, synchronize the exchange. On Blackwell GPUs, NVLinkOneSided is faster than DeepEP at every batch, and the resulting MoE layer, built from separate kernels, achieves better performance than DeepGEMM MegaMoE at decoding batches. NVLinkOneSided is available in FlashInfer3 and is integrated into vLLM (Kwon et al., 2023), SGLang (Zheng et al., 2024) and TensorRT-LLM (NVIDIA, 2026e).

ACKNOWLEDGEMENTS We thank Itamar Rosenfeld Rauch, Chanan Maoz, Samuel Nordmann, Nikita Korobov, Yicheng Qiang, Ke Wen, Xiaofan Li, Jack Yang, Hao Lu and Ao Tang for their help and discussions. 3 https://docs.flashinfer.ai/generated/fla shinfer.moe_ep.NVLinkOneSidedAlltoAll.html

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

R EFERENCES Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024. Aimuyo, O. J., Oh, B., and Singh, R. FlashDMoE: Fast distributed MoE in a single kernel. arXiv preprint arXiv:2506.04667, 2025. Anonymous. FarSkip-Collective: Unhobbling blocking communication in mixture of experts models. In Proceedings of Machine Learning and Systems (MLSys), 2026. arXiv:2511.11505. Aubrey, K. and Stam, N. Inside NVIDIA Blackwell Ultra: The chip powering the AI factory era. NVIDIA Technical Blog, https://developer.nvidia.com /blog/inside-nvidia-blackwell-ultra-t he-chip-powering-the-ai-factory-era/, August 2025. Belli, R. and Hoefler, T. Notified access: Extending remote memory access programming models for producerconsumer synchronization. In IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2015. DeepSeek-AI. DeepSeek-V3 technical report. preprint arXiv:2412.19437, 2024.

arXiv

DeepSeek-AI. DeepGEMM. https://github.com /deepseek-ai/DeepGEMM, 2025. DeepSeek-AI. DeepInfer: High-performance, large-scale inference framework for DeepSeek models. https:// github.com/deepseek-ai/DeepInfer, 2026a. Commit 5537415, decode benchmarks in README and doc/inference_deployment_qa.md. DeepSeek-AI. DeepSeek-V4.1-Flash. https://huggin gface.co/deepseek-ai/DeepSeek-V4.1-F lash, 2026b. Dragojević, A., Narayanan, D., Hodson, O., and Castro, M. FaRM: Fast remote memory. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2014. Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. FlashInfer Contributors. FlashInfer Prims-TS fused MoE API. https://docs.flashinfer.ai/api/f used_moe.html#prims-ts-fused-moe, 2026. Accessed 2026-10-30.

Gale, T., Narayanan, D., Young, C., and Zaharia, M. MegaBlocks: Efficient sparse training with mixture-ofexperts. In Proceedings of Machine Learning and Systems (MLSys), 2023. Goldman, A., Boker, N., Sheraizin, M., Admoni, N., Polyakov, A., Bhattacharya, S., Yu, F., Sun, K., Theodorakis, G., Yin, H.-C., Gootzen, P.-J., Shafi, A., Ravid, A., Di Girolamo, S., Dinan, J., Li, X., Gorentla Venkata, M., and Bloch, G. NCCL EP: Towards a unified expert parallel communication API for NCCL. arXiv:2603.13606, 2026. Hamidouche, K., Bachan, J., Markthub, P., Gootzen, P.J., Agostini, E., Jeaugey, S., Shafi, A., Theodorakis, G., and Gorentla Venkata, M. GPU-initiated networking for NCCL. arXiv:2511.15076, 2025. He, J., Zhai, J., Antunes, T., Wang, H., Luo, F., Shi, S., and Li, Q. FasterMoE: Modeling and optimizing training of large-scale dynamic pre-trained models. In ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), 2022. Hu, Z. et al. Demystifying NCCL: An in-depth analysis of GPU communication protocols and algorithms. arXiv preprint arXiv:2507.04786, 2025. Hwang, C., Cui, W., Xiong, Y., Yang, Z., Liu, Z., Hu, H., Wang, Z., Salas, R., Jose, J., Ram, P., Chau, H., Cheng, P., Yang, F., Yang, M., and Xiong, Y. Tutel: Adaptive mixture-of-experts at scale. In Proceedings of Machine Learning and Systems (MLSys), 2023. InfiniBand Trade Association. InfiniBand Architecture Specification, Volume 1, Release 1.4, 2020. Kalia, A., Kaminsky, M., and Andersen, D. G. Using RDMA efficiently for key-value services. In ACM SIGCOMM, 2014. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In ACM Symposium on Operating Systems Principles (SOSP), 2023. Lamport, L. Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21(7): 558–565, 1978. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR), 2021.

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

Licker, N., Chen, L., and Hoang, P. Efficient and portable mixture-of-experts communication. Perplexity Research Blog, https://www.perplexity.ai/hub/bl og/efficient-and-portable-mixture-o f-experts-communication, 2025a. Licker, N., Hu, K., Zaytsev, V., and Chen, L. pplx-kernels: Perplexity MoE kernels. https://github.com/p erplexityai/pplx-kernels, 2025b. Lustig, D., Sahasrabuddhe, S., and Giroux, O. A formal analysis of the NVIDIA PTX memory consistency model. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2019. Markthub, P., Dinan, J., Potluri, S., and Howell, S. Improving network performance of HPC systems using NVIDIA Magnum IO NVSHMEM and GPUDirect Async. NVIDIA Technical Blog, https://deve loper.nvidia.com/blog/improving-netwo rk-performance-of-hpc-systems-using -nvidia-magnum-io-nvshmem-and-gpudire ct-async/, November 2022. MPI: A Message-Passing Interface Standard, Version 3.1. Message Passing Interface Forum, 2015. Moonshot AI. Kimi-K3. https://huggingface.co /moonshotai/Kimi-K3, 2026. NVIDIA. NCCL: NVIDIA collective communications library. https://github.com/NVIDIA/nccl, a.

NVIDIA. Parallel Thread Execution ISA, Version 9.4, 2026d. https://docs.nvidia.com/cuda/p arallel-thread-execution/. NVIDIA. TensorRT-LLM. https://github.com/N VIDIA/TensorRT-LLM, 2026e. PyTorch Contributors. PyTorch symmetric memory. http s://pytorch.org/docs/stable/symmetri c_memory.html. Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. DeepSpeedMoE: Advancing mixture-of-experts inference and training to power next-generation AI scale. In International Conference on Machine Learning (ICML), 2022. Recio, R., Metzler, B., Culley, P., Hilland, J., and Garcia, D. A remote direct memory access protocol specification. RFC 5040, IETF, 2007. Shah, A. et al. MSCCL++: Rethinking GPU communication abstractions for cutting-edge AI applications. arXiv preprint arXiv:2504.09014, 2025. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2017.

NVIDIA. NVSHMEM. https://developer.nvid ia.com/nvshmem, b.

Shen, S., Korzh, A., Bachan, J., Chen, T., Goel, A., Schneider, L., Kousha, P., He, Z., Jeaugey, S., Iskra, K., Chandawala, N., Hammond, J. R., and Hoefler, T. Every microsecond matters: Achieving near speed-of-light latency in GPU collectives. arXiv:2607.16100, 2026.

NVIDIA. Enabling fast inference and resilient training with NCCL 2.27. https://developer.nvidia.com /blog/enabling-fast-inference-and-res ilient-training-with-nccl-2-27/, 2025.

Zhang, S. et al. Comet: Fine-grained computationcommunication overlapping for mixture-of-experts. In Proceedings of Machine Learning and Systems (MLSys), 2025.

NVIDIA. CUDA programming guide: Programmatic dependent launch. https://docs.nvidia.com/cu da/cuda-programming-guide/04-special -topics/programmatic-dependent-launch. html, 2026a. Accessed 2026-10-30.

Zhao, C., Zhou, S., Zhang, L., Deng, C., Xu, Z., Liu, Y., Yu, K., Li, J., and Zhao, L. DeepEP: An efficient expertparallel communication library. https://github.c om/deepseek-ai/DeepEP, 2025. V1: tag v1.2.1; V2.5: branch v2.5, 2026-09-29.

NVIDIA. IKET profiling (In-Kernel event tracing). NVIDIA CUTLASS documentation 4.7.0, https:// docs.nvidia.com/cutlass/4.7.0/media/ docs/pythonDSL/guides/iket_profiling .html, 2026b. Accessed 2026-10-30.

Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

NVIDIA. UB-X: Ultra bandwidth—eXperimental NVLink collectives. https://github.com/NVIDIA/nc cl/tree/v2.32.3- 1/contrib/nccl_ubx, 2026c. NCCL contrib; first released with NCCL 2.30.7.

Zheng, S. et al. TileLink: Generating efficient computecommunication overlapping kernels using tile-centric primitives. In Proceedings of Machine Learning and Systems (MLSys), 2025.

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

Zhu, R., Jiang, Z., Jin, C., Wu, P., Stuardo, C. A., Wang, D., Zhang, X., Zhou, H., Wei, H., Cheng, Y., Xiao, J., Zhang, X., Liu, L., Lin, H., Chang, L.-W., Ye, J., Yu, X., Liu, X., Jin, X., and Liu, X. MegaScale-Infer: Efficient mixtureof-experts model serving with disaggregated expert parallelism. In ACM SIGCOMM Conference (SIGCOMM), 2025.

AI U SAGE D ISCLOSURE AI coding assistants were used to help draft and edit the text and to write the plotting and benchmark scripts. All technical content, the design, the experiments and the reported numbers were produced or verified by the authors. No experimental data was generated by AI.

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

A

L ATENCY TABLES

Tables A.1 and A.2 list dispatch, combine and round trip of the all-to-all (§5.2) for S ENTINEL, F ENCE and DeepEP from 1 to 4096 tokens per rank. Tables A.3 and A.4 list the MoE layer (§5.3) with each protocol, MegaMoE and DeepEP + DeepGEMM; Table A.5 gives the payload formats (Figure 7), and Table A.6 the layer under Zipf routing. On one GB200 NVL72 rack (§5.4), Table A.7 lists the allto-all and Table A.8 the layer with perfect routing.

B

M EASUREMENT DETAILS

The B300 node runs CUDA 13.5, and the tokens are quantized to MXFP8 before the timed span. Ten warm-up and 100 timed iterations run back to back in one CUDA graph, with L2 flushed before each; we report the span from the first to the last kernel, averaged over iterations and ranks. All-to-all outputs match a reference bitwise, and every layer matches one to a relative L2 error of at most 5.7 × 10−3 . We time phases from CUPTI kernel records in back-to-back CUDA graph replays, with no barrier or event between phases: a barrier before every phase hides the waiting a protocol does in steady state, and events in the graph keep Programmatic Dependent Launch (PDL) from overlapping the kernels around them. The round trip of the all-to-all is the sum of dispatch and combine. Each stage of the MoE layer (Table 4) spans its first kernel’s start to its last kernel’s end; PDL lets stages overlap slightly, so they add up to more than the layer. The breakdown in Figure 6 comes from IKET timestamps that every warp records when it enters and leaves each step, traced back from the block that finishes each phase last and averaged over ranks and iterations; instrumentation changes the round trip by at most 6%.

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

Table A.1. All-to-all latency (µs), DeepSeek-V4.1-Flash, 8×B300: dispatch / combine / round trip. Tokens/rank

S ENTINEL disp. comb. total

1 8.2 2 9.0 8.3 4 8 8.3 8.5 16 32 9.3 64 10.8 128 13.5 256 20.6 512 37.2 1024 60.4 2048 106.1 4096 197.0

F ENCE disp. comb.

9.1 17.4 14.2 9.5 18.5 13.5 11.6 19.9 13.0 11.7 20.0 14.2 12.1 20.6 13.6 13.5 22.9 13.9 16.3 27.0 14.6 21.7 35.2 17.4 34.5 55.1 23.2 55.7 92.9 35.1 98.0 158.4 57.4 180.8 286.9 99.9 341.7 538.7 188.2

total

DeepEP disp. comb.

13.3 27.6 23.2 13.2 26.8 21.5 13.3 26.2 21.4 13.6 27.8 21.4 16.0 29.6 21.7 16.8 30.7 22.0 18.2 32.8 23.0 22.2 39.6 25.2 32.5 55.7 30.7 54.1 89.2 42.1 96.1 153.5 68.9 178.6 278.5 120.3 342.4 530.6 227.5

24.7 25.6 26.1 26.5 26.8 28.2 30.6 36.5 48.0 69.3 113.3 205.0 404.1

total 47.9 47.0 47.5 47.8 48.5 50.2 53.6 61.7 78.7 111.4 182.2 325.3 631.6

Table A.2. All-to-all latency (µs), Kimi-K3, 8×B300: dispatch / combine / round trip. Tokens/rank

S ENTINEL disp. comb. total

1 8.6 8.9 2 4 8.7 9.0 8 9.2 16 32 9.4 64 10.7 128 13.8 256 19.9 512 38.7 1024 60.7 2048 105.7 4096 191.5

F ENCE disp. comb.

8.6 17.2 12.7 8.7 17.6 13.0 11.1 19.7 12.4 11.5 20.5 13.2 11.7 20.9 13.1 12.8 22.2 13.2 15.6 26.2 13.1 20.7 34.5 16.1 32.1 52.0 21.8 50.6 89.2 33.6 89.1 149.8 54.8 165.2 270.8 96.0 316.1 507.6 180.1

total

DeepEP disp. comb.

11.3 24.0 23.2 11.2 24.2 23.8 11.5 23.9 25.1 11.3 24.4 24.4 14.0 27.2 25.0 14.7 27.9 25.1 16.7 29.8 26.5 20.7 36.7 29.1 30.9 52.7 33.6 51.6 85.1 43.9 90.2 145.0 66.9 167.2 263.3 117.4 321.1 501.1 223.9

26.2 29.0 29.9 30.8 31.2 32.3 35.9 40.7 51.0 71.6 111.4 199.1 376.3

total 49.3 52.8 55.0 55.2 56.2 57.4 62.4 69.7 84.6 115.5 178.3 316.4 600.3

Table A.3. MoE layer latency (µs), DeepSeek-V4.1-Flash, 8×B300, MXFP4 weights and MXFP8 activations: layer time and dispatch / MoE compute / combine, each stage from its first kernel’s start to its last kernel’s end. NVLinkOneSided F ENCE Tokens/rank MoE layer stages 1 2 4 8 16 32 64 128 256 512 1024 2048 4096

46.0 12.8 / 19.6 / 14.4 47.9 12.8 / 21.2 / 14.6 51.2 12.3 / 25.5 / 14.2 59.5 13.1 / 32.4 / 14.7 79.6 13.3 / 49.9 / 17.2 113.3 13.5 / 83.2 / 17.8 177.9 13.5 / 146.6 / 18.9 188.5 16.3 / 150.0 / 23.1 208.1 22.2 / 153.4 / 33.6 259.3 33.2 / 170.5 / 57.0 353.8 55.2 / 201.3 / 99.2 637.9 99.6 / 355.5 / 184.5 1185.2 187.2 / 647.9 / 351.6

NVLinkOneSided S ENTINEL MoE layer stages 36.7 8.3 / 23.9 / 9.8 39.2 8.9 / 26.6 / 9.8 44.6 8.1 / 29.7 / 12.1 52.1 8.5 / 37.2 / 11.9 70.5 8.5 / 54.7 / 12.9 105.8 9.1 / 88.7 / 14.2 171.8 10.4 / 147.1 / 16.2 183.3 13.5 / 150.6 / 21.5 206.5 20.5 / 154.9 / 33.8 260.5 37.5 / 171.5 / 53.9 356.9 60.8 / 206.0 / 96.2 645.1 106.2 / 363.3 / 179.8 1198.6 197.4 / 665.7 / 343.3

MegaMoE DE+DG MoE layer MoE layer stages 49.8 52.4 57.8 68.1 90.3 123.0 184.7 188.7 194.1 214.9 255.3 428.3 766.1

74.7 21.4 / 26.0 / 27.2 80.5 21.8 / 28.4 / 30.0 85.8 21.6 / 34.4 / 29.8 98.4 21.9 / 46.1 / 30.4 122.0 22.0 / 68.7 / 31.4 160.2 22.7 / 104.4 / 33.2 227.4 23.6 / 168.1 / 35.7 235.9 25.8 / 168.7 / 41.4 251.2 29.8 / 170.6 / 50.8 300.3 44.7 / 181.0 / 74.4 409.6 72.9 / 215.8 / 121.3 672.3 127.8 / 320.2 / 224.3 1290.2 240.1 / 612.1 / 429.6

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

Table A.4. MoE layer latency (µs), Kimi-K3, 8×B300, MXFP4 weights and MXFP8 activations: layer time and dispatch / MoE compute / combine, each stage from its first kernel’s start to its last kernel’s end. NVLinkOneSided F ENCE Tokens/rank MoE layer stages 1 2 4 8 16 32 64 128 256 512 1024 2048 4096

47.7 13.1 / 23.5 / 15.1 52.7 13.0 / 27.8 / 16.0 61.8 13.1 / 37.7 / 15.5 84.7 14.1 / 59.4 / 15.5 128.4 13.4 / 100.9 / 15.0 211.9 13.5 / 183.5 / 15.9 334.5 14.8 / 303.2 / 17.6 351.1 17.2 / 313.4 / 21.5 396.1 22.8 / 342.2 / 32.0 515.6 33.8 / 419.3 / 58.8 719.5 54.9 / 561.6 / 102.9 1342.9 97.7 / 1060.3 / 189.7 2553.6 181.9 / 2023.0 / 360.2

NVLinkOneSided S ENTINEL MoE layer stages 42.1 8.8 / 28.1 / 13.5 47.2 8.9 / 32.6 / 14.9 57.4 8.9 / 42.6 / 15.2 80.7 9.2 / 65.2 / 16.0 122.0 9.4 / 106.5 / 12.5 206.4 9.7 / 189.4 / 13.9 328.9 10.7 / 309.4 / 16.3 346.4 13.7 / 314.8 / 21.0 393.2 20.3 / 343.4 / 32.6 515.3 38.9 / 423.8 / 54.1 732.1 61.5 / 580.7 / 98.2 1367.5 109.6 / 1093.9 / 179.3 2601.5 195.8 / 2084.3 / 345.3

MegaMoE DE+DG MoE layer MoE layer stages 60.4 67.5 78.6 106.7 148.6 231.2 350.9 358.5 370.2 418.4 535.8 1007.5 1901.9

99.7 26.5 / 37.8 / 35.3 109.2 26.0 / 45.5 / 37.4 120.6 24.2 / 58.3 / 38.2 146.2 23.4 / 85.0 / 37.8 191.2 23.5 / 129.4 / 38.2 281.8 24.4 / 216.4 / 41.0 416.0 25.5 / 344.2 / 46.3 430.8 27.0 / 346.9 / 56.8 482.1 31.2 / 374.0 / 76.7 620.6 46.2 / 451.5 / 121.5 976.3 78.1 / 678.5 / 222.3 1789.6 138.8 / 1232.6 / 422.6 3091.1 269.8 / 2006.4 / 817.7

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

Table A.5. F ENCE with other payload formats on eight B300 GPUs: bytes per token and target rank, and round trip (µs). The last two rows of each model include quantization to MXFP8, on the source rank or, after a BF16 dispatch, on the target rank. Dispatch

Combine

Bytes disp. / comb.

Tokens per rank 1 128 1024 4096

DeepSeek-V4.1-Flash BF16 BF16 MXFP8 BF16 FP8 (1×128) BF16 NVFP4 BF16 MXFP8 FP8

10288 / 10240 29.0 46.3 194.3 684.5 5328 / 10240 28.7 40.7 154.9 532.7 5328 / 10240 28.6 41.1 155.3 532.7 2928 / 10240 28.1 37.6 137.4 456.0 5328 / 5120 28.0 39.9 131.4 438.9

MXFP8, at the source BF16, at the target

5328 / 10240 30.4 43.4 158.9 547.7 10288 / 10240 30.6 48.8 214.2 768.1

Kimi-K3 BF16 BF16 MXFP8 BF16 FP8 (1×128) BF16 NVFP4 BF16 MXFP8 FP8

7296 / 7168 25.6 42.7 183.5 644.0 3824 / 7168 25.0 37.7 146.5 502.0 3824 / 7168 25.1 37.6 146.6 502.2 2144 / 7168 24.8 35.2 128.8 430.1 3824 / 3584 24.3 36.5 124.2 419.9

MXFP8, at the source BF16, at the target

3824 / 7168 26.4 40.3 149.7 514.5 7296 / 7168 25.2 42.7 196.8 701.8

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

Table A.6. MoE layer under Zipf routing with exponent α on eight B300 GPUs: layer time (µs) of Ours, DE+DG and MegaMoE. Ratio is ours relative to MegaMoE, and Imba. is the load imbalance of Table 5. Ours: NVLinkOneSided with the faster protocol + Prims-TS († : F ENCE); DE+DG: DeepEP + DeepGEMM.

Routing

T

DeepSeek-V4.1-Flash

Kimi-K3

Ours DE+DG MegaMoE Ratio Imba.

Ours DE+DG MegaMoE Ratio Imba.

Perfect

1 36.7 16 70.5 128 183.3 1024 353.8†

74.7 122.0 235.9 409.6

49.8 90.3 188.7 255.3

0.74 0.78 0.97 1.39

1.00 1.00 1.00 1.00

42.1 122.0 346.4 719.5†

99.7 191.2 430.8 976.3

60.4 148.6 358.5 535.8

0.70 0.82 0.97 1.34

1.00 1.00 1.00 1.00

Zipf, α = 0

1 52.2 16 154.1 128 188.8† 1024 414.9

97.5 208.8 233.7 461.1

68.3 167.3 188.8 255.6

0.76 0.92 1.00 1.62

1.50 1.09 1.03 1.02

84.6 302.9 347.5 717.6†

148.0 387.8 432.3 968.6

114.2 328.7 359.0 538.9

0.74 0.92 0.97 1.33

1.38 1.23 1.03 1.01

Zipf, α = 0.5

1 57.8 16 154.6 128 204.2 1024 445.4†

105.2 209.0 250.2 499.5

73.0 165.1 196.1 303.9

0.79 0.94 1.04 1.47

1.50 1.31 1.25 1.29

81.0 295.5 386.7 852.3†

146.1 376.3 456.4 1096.7

106.8 320.2 384.9 638.9

0.76 0.92 1.00 1.33

1.25 1.36 1.20 1.19

Zipf, α = 1

1 52.4 16 142.2 128 233.8 1024 557.0

99.3 179.5 279.2 626.7

67.6 140.5 221.1 374.4

0.77 1.01 1.06 1.49

1.50 71.1 1.62 257.5 1.67 445.8 1.69 1075.3†

133.4 322.8 539.1 1352.6

96.6 273.7 449.1 814.0

0.74 0.94 0.99 1.32

1.38 1.59 1.56 1.55

Zipf, α = 1.25

1 52.3 16 132.1 128 240.0 1024 619.4

97.9 163.2 289.7 696.0

69.2 129.7 225.8 411.5

0.76 1.02 1.06 1.51

1.67 66.4 1.93 218.0 1.91 447.1 1.91 1184.5†

129.4 274.0 550.0 1477.0

95.4 233.9 453.5 918.4

0.70 0.93 0.99 1.29

1.62 1.70 1.71 1.70

Table A.7. All-to-all round trip (µs, dispatch + combine) on one GB200 NVL72 rack with perfect routing, by expert-parallel ranks (on ranks / 4 compute trays) and tokens per rank. DeepSeek-V4.1-Flash

Kimi-K3

Ranks

T

F ENCE

S ENTINEL

DeepEP

F ENCE

S ENTINEL

DeepEP

8

1 16 128 1024

43.0 43.1 48.9 160.1

17.5 17.8 31.7 151.6

44.7 44.2 57.8 173.9

34.6 36.0 44.1 150.6

17.5 18.2 31.5 144.0

53.6 57.6 70.9 179.0

16

1 16 128 1024

41.8 41.8 48.8 171.5

16.6 18.4 33.0 160.2

45.9 47.1 61.7 186.2

48.4 50.6 68.1 295.0

21.5 23.2 57.9 308.3

80.0 63.5 92.6 335.9

32

1 16 128 1024

43.2 44.3 53.8 183.3

17.3 18.2 34.7 169.7

48.6 49.5 66.4 203.1

53.7 57.9 76.6 310.0

21.8 25.4 61.3 320.9

73.5 65.1 97.1 341.6

64

1 16 128 1024

47.6 46.7 58.0 191.4

17.8 19.1 36.7 179.8

54.2 55.7 81.6 213.7

52.5 56.0 78.7 320.8

23.6 26.1 64.2 340.6

79.1 90.0 105.9 378.9

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

Table A.8. MoE layer latency (µs) on one GB200 NVL72 rack with perfect routing, by expert-parallel ranks and tokens per rank: NVLinkOneSided (the faster protocol; † : F ENCE), DeepEP + DeepGEMM and MegaMoE. –: not tested, as the Prims-TS grouped GEMMs need at least k experts per rank (Kimi-K3 has 14 on 64 GPUs). DeepSeek-V4.1-Flash

Kimi-K3

Ranks

T

Ours

DE+DG

MegaMoE

Ours

DE+DG

MegaMoE

8

1 16 128 256 512 1024 4096

35.4 66.4 181.7 208.4 348.2 438.1 1378.8

78.1 124.5 237.7 256.9 317.2 481.8 1457.3

45.7 83.4 179.3 188.1 222.0 294.1 902.4

40.7 117.5 341.5 491.5 624.3 881.4 3123.6†

116.5 200.6 429.9 517.4 728.2 1152.7 3537.7

57.3 143.8 344.7 373.3 473.1 658.0 2283.5

16

1 16 128 256 512 1024 4096

35.4 51.9 124.7 200.3 252.7 440.0 1446.6†

78.0 100.6 169.3 197.2 272.2 431.4 1432.2

46.3 63.5 121.0 140.1 172.1 276.9 818.1

43.6 84.7 286.0 395.6 560.1† 1013.3† 3372.0†

109.5 151.3 300.8 419.1 652.5 1113.8 3520.2

54.4 105.4 214.3 250.3 330.3 595.3 1885.1

32

1 16 128 256 512 1024 4096

35.7 47.7 120.9 153.0 265.3 463.8 1560.7†

79.1 88.9 136.2 171.0 264.8 445.6 1481.1

47.2 58.6 91.2 114.2 173.4 265.0 824.5

43.7 66.8 234.0 322.0 570.9† 996.3† 3604.2†

107.8 142.9 258.2 396.8 637.0 1013.9 3425.8

53.1 86.7 167.0 207.7 340.1 580.5 2018.7

64

1 16 128 256 512 1024 4096

36.9 49.0 102.6 172.8 309.0† 487.0 1759.0†

82.3 90.9 126.8 165.3 253.3 422.5 1473.1

49.7 56.0 86.6 112.6 164.1 250.9 789.9

– – – – – – –

110.3 122.8 242.1 347.6 567.4 937.8 3345.2

56.7 74.6 143.2 196.4 302.3 524.1 1799.4

Related documents

• INC-MoE: Overlapping In-Network Collectives and Expert Computation for Expert-Parallel Mixture-of-Experts Inference#1473581
• INC-MoE: Overlapping In-Network Collectives and Expert Computation for Expert-Parallel Mixture-of-Experts Inference#1467663
• FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving#998172
• Launch-Bound and Substitutable: Why Three Inference Optimizations Fail to Pay Off in Mixture-of-Experts Models#780755
• Launch-Bound and Substitutable: Why Three Inference Optimizations Fail to Pay Off in Mixture-of-Experts Models#779380