ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill

ASAP: A DiSaggregated and Asynchronous Inference System for MoE Prefill Weiwei Chen, Shuang Chen*, Lele Li, Qiang Hu, Han Li, Xin Ye, Ming Yan, Zhibin Yu Huawei

arXiv:2606.22541v1 [cs.DC] 21 Jun 2026

Abstract

DeepSeek-V3 prefill instance [13] utilizes an EP size of 32, distributed across 32 GPUs over 4 compute nodes. However, accommodating this broad EP scale introduces architectural friction for the attention stage, which must adopt a hybrid of Tensor Parallelism (TP) and Data Parallelism (DP). Due to the prohibitive inter-device communication overhead incurred by large TP, the TP degree is strictly bounded (typically ≤ 8). As a result, the system configures the DP size as EP/T P to process concurrent request batches [27, 40]. Consequently, the aforementioned DeepSeek-V3 instance enforces a TP size of 8 and a DP size of 4 during its attention phase. Problem. Despite the theoretical efficiency of these hybrid parallelism strategies, our profiling reveals severe resource underutilization and inflated inference latencies during the compute-intensive prefill phase of production MoE serving. This inefficiency stems from a fundamental execution mismatch: the inevitable DP imbalance coupled with the rigid synchronization barriers inherent in state-of-the-art inference systems. As depicted in Figure 1a, heterogeneous requests are independently batched and routed to distinct attention DP groups. However, because the subsequent MoE stage operates over a globally shared expert pool, these isolated DP groups are forced to synchronize at strict barriers before and after every MoE layer. This lockstep execution paradigm creates a severe straggler effect, where the entire system stalls until the slowest attention DP group or the most congested expert completes its task. In production deployments, online inference clusters are subjected to highly stochastic request arrivals, extreme sequence length skew, unpredictable prefix cache hit rates, etc [7, 49]. Such heterogeneity results in imbalanced workload (thus inconsistent latency) across attention DP groups, a phenomenon we term DP Imbalance [31]. Related Work. Existing literature has extensively explored expert load balancing [11, 13, 34], yet the performance degradation caused by DP imbalance has been significantly underestimated in the context of online MoE serving. Because production LLM serving is characterized by highly stochastic arrivals and heavy-tailed sequence length distributions [7, 49], achieving perfect attention DP balancing is practically impossible. On another front, recent efforts in long-context acceleration [9, 14, 21, 23, 24, 26, 36, 45, 47, 54] successfully tackle

Mixture-of-Experts (MoE) models have become the de facto standard for scaling large language models. To maintain computational efficiency, modern MoE serving systems typically employ a hybrid parallelism strategy, combining Data Parallelism (DP) for attention stages with Expert Parallelism (EP) for MoE stages. However, this design necessitates frequent global synchronization barriers between attention DP groups and experts. In online serving, significant variance in request arrival rates and sequence lengths inherently leads to DP imbalance, causing severe synchronization stalls that degrade Time-to-First-Token (TTFT) and system throughput. We present ASAP, an asynchronous inference system specifically designed to accelerate the prefill phase of MoE models. ASAP disaggregates the attention and MoE stages and implements a fully asynchronous execution pipeline. This is achieved through a suite of specialized asynchronous communication primitives and four coordinated optimizations across request scheduling and model execution, which collectively dismantle global synchronization barriers. We implement and evaluate ASAP on CloudMatrix384 super-nodes, demonstrating that it improves SLO-compliant prefill throughput by 90% compared to state-of-the-art synchronous serving solutions.

1

Introduction

Background. Large-scale Mixture-of-Experts (MoE) architectures have become the de facto paradigm for serving highcapacity models efficiently. By selectively activating a sparse subset of experts per token [15, 16, 17, 35, 39, 41, 42], MoEs decouple model capability from computational cost. For example, DeepSeek-V3 [13] achieves a total scale of 671B parameters while restricting active parameters to only 37B per token. Since the vast majority of these parameters reside in expert weights, Expert Parallelism (EP) is commonly applied to the MoE stage, distributing experts across an array of accelerators. A wide EP configuration fundamentally reduces the memory footprint per device, directly alleviating the severe memory bottlenecks typical in LLM inference. Specifically, a * Corresponding author. Email: [email protected]

1

Communication Communication

C C

A AIDLE

IDLE

A A

IDLE IDLE

M M M M M M

Request Load Request stream Load 2. Maintaining Compute Efficiency under Fragmented stream and Out-of-Order Execution: Asynchrony introduces Prefill Instance-3 "fragmented" execution; instead of one massive batch, MoE Prefill Instance-3 Prefill Instance-2 devices receive independent, smaller batches, risking a drop Prefill Instance-2 Prefill Instance-1 in arithmetic intensity. Furthermore, DP imbalance forces Attentio Prefill Instance-1 MoE devices to execute layers out-of-order, where the spe-1 Attention h c DP-1 cific layer ID is only resolved at runtime. This dynamic acth1 Device b Request 1 DP-1 t execution pattern significantly increases kernel dispatch bba Request Device 1 Scheduler overhead from the host CPU, potentially introducing amore Attentio ba tch2 Attention Scheduler execution bubbles than the gains obtained from asynchrony. tch DP-2 2 2 DP-2 Our Work. We present ASAP, a high-performance infer- Device

Idle TimeIdle IDLE IDLE

Time

… …

MoE MoE

All2All All2All

Attention M A Attention M Request Batch-1 Request Batch-2 Batch-1 A Request Batch-2 A Request Batch-N Request Batch-N

All2All All2All

A

A A

(a) Synchronous Time

Request Batch-1 Request Batch-2 Batch-1 A Request Request Batch-2 A Request Batch-N Request Batch-N

A MA C

C C M A A

C

C C C C

M M M C (b) Asynchronous M C

C C

M Time C M C SAVED LATENCY SAVED LATENCY

ence system that realizes fully asynchronous MoE prefill. Device 2 ASAP disaggregates attention and MoE stages onto separate devices, interconnected via asymmetric, asynchronous communication primitives to enable non-blocking data transfers. The key is to create a dedicated shared memory buffer on each device that is visible to all devices and acts as a “superhub” between senders and receivers. This is the foundation of ASAP’s asynchronous execution pipeline. To fully leverage this asynchronous architecture and mitigate the efficiency risks of asynchrony, ASAP integrates Prefill four key runtime optimizations: (1) Length-aware batchPrefill I ing toRequest preserveQueue MoE compute intensity; (2) Request Dual-batch in§3.3: Scheduler CPU Request Queue S=4K §3.3: Request Scheduler CPU terleaving to maximize attention device duty cycles; (3) S=2K Length-Aware Dual-Batch S=4K S=2K S=2.4K Communication-computation overlapping via a tripleDual-Batch Length-Aware S=0.5K S=2.4K Batchingand (4)Interleaving stream design to S=3K maximize hardware utilization; A S=0.5K Interleaving Batching S=3K layer-oblivious MoE Super Kernel that enables bubble-free Attn D kernel dispatching, allowing out-of-order execution without Attn DP-1 NPU NPU NPUAttn NPUDP N NPUAttn NPUDP-1 host-side bottlenecks. NPU NPU NP NPU NPU NPU NPU In summary, we make the following contributions:

Figure 1: Comparison between current synchronous and ideal asynchronous execution under heterogeneous request batches. Synchronous execution (Figure 1a) requires all request batches (launched on different attention DP groups) to synchronize before the MoE computation. An ideal asynchronous execution pipeline (Figure 1b) eliminates the barrier, and allows faster request batches to progress at their own pace.

super-long requests, but remain oblivious to the execution bubbles caused by heterogeneous short-to-medium requests (e.g., prompts with < 32K tokens). Our characterization in Section 2.2.1 shows that DP imbalance does not necessarily come from super-long requests. Co-executing various shortto-medium requests in a synchronous inference system still stalls the pipeline and degrades overall throughput. Motivation. Figure 1b illustrates an ideal, barrier-free, §3.2:an Disaggregated 1. To the best of our knowledge, we are the first to build asynchronous execution paradigm. By eliminating global §3.2: Disaggregated A Asynchronous Comm asynchronous execution pipeline for online inference. synchronization, heterogeneous requests are enabled to truly Asynchronous Commu “flow” at their own pace without waiting for or delaying others. 2. We systematically analyze the prefill phase MoE of MoE serv- MoE EP-2 EP-1 This approach effectively minimizes resource idling, which siMoE EP-1 EP-2 ing. We identify an inherent DP imbalance problem in MoENPU NPU multaneously reduces inference latency and improves overall NPU NPU synchronous systems, causing severe resource idling and throughput. limited prefill performance. Challenges. Transitioning from synchronous to asynAtten MOE Devices Attention Devices chronous MoE serving presents two fundamental architectural Attent MOE Devices3. We design ASAP, the first asynchronous MoE inference Attention Devices Attn DP1 NPU 1 system hurdles: Attn DP1 Attn DP DP1 NPU1that dismantles global synchronization barriers to Attn TP-1 2 TP-2 NPU DP1 TP-1 2 Attn DP Attn DP … enable independent execution of request batches. 1 2 1. Decoupling Execution Dependencies with AsynNPU2 DP TP-2 Attn DP2

1 TP-T … TP-1

① Poll

Attn DPD

TP-2 DPTP-T 1 TP-1 … AttnCommunication DP2 … Poll NPUE Attn DPD chronous Primitives: Traditional 1 TP-2 DP1TP-1 Flags … TP-T 4. ① We … ① implement PollNPUE and evaluate ASAP on CloudMatrix384 [56] 1 TP-1 Flags TP-T TP-2 … TP-1 2 3 1 systems tightly Attention DP and MoE on D DP … TP-2 stages ① Poll Attn DPcouple 1 TP-1 1 D … 1 2 Flags TP-2 supernode. Extensive evaluation results show that … TP-T TP-1 2 3 1 DPD DP …1 TP-2 DPD TP-T … 1 11 2 22 Flags ③Clear TP-2 ① Poll the same Attn devices, enforcing a lock-step execution 2that Buffer 3 1 DPTP-T 1 2 D TP-1 … 1 2 TP-T Shared … ③Clear ASAP improves SLO-compliant prefill throughput by 90% 1 2 ①Flags Poll TP-T 2 3 1 DPD TP-2 Flags NPUD1 … 1 11 2 22 … Buffer ③Clear ②Copy Data Shared TP-TTP-1 precludes overlapping different computation stages. Flags NPUD1 Flags 1 2 compared to state-of-the-art synchronous baselines. TP-2 1 Shared Buffer ③Clear ②Copy Data 2 3 DPD TP-1 … Flags 1 21 Shared Buffer TP-T 2 3 1 Transitioning to an Attention-MoE decoupled architecture DPD TP-2 ②Copy … Data Flags 1 ③Clear TP-T ②Copy Data NPU Shared B D2 ① Write Shared Buffer ③Clear requires not NPU onlyD2physical resource disaggregation, but Flags Shared Bu ① Write Private Memory Shared Buffer ②Copy Data Data ②Copy Flags Private Memory ②Copy Data also a new class of asymmetric Data and asynchronous com②Copy D Private Memory ②Set 2 Background and Motivation ②Set munication primitives. Current collective operationsPrivate (e.g., Memory NPUDT Flags NPUDT Flagsdesigned for synchronous Private M Private Memory Peer-to-Peer, All-to-All) are Private Me Private Memory async-dispatch-recv async-dispatch-send blocking bulk transfers; the lack of efficient primitives In this section, we provide the necessary background on MoE asyn async-dispatch-recv async-dispatch-send async that support fine-grained, non-blocking token routing is a serving, and present a series of characterization studies to primary reason why fully asynchronous MoE inference motivate the need for a barrier-free, asynchronous system to remains elusive. optimize MoE prefill performance.

2

All-to-All

Expert1 All-to-All

Attention 1 DP-1 tch a Request b Device 1 Scheduler batc h2 Attention DP-2

Expert2 Expert3 Expert4

Device 2

+ Decode Instances

+