ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill

ASAP: A DiSaggregated and Asynchronous Inference System for MoE Prefill Weiwei Chen, Shuang Chen*, Lele Li, Qiang Hu, Han Li, Xin Ye, Ming Yan, Zhibin Yu Huawei

arXiv:2606.22541v1 [cs.DC] 21 Jun 2026

Abstract

DeepSeek-V3 prefill instance [13] utilizes an EP size of 32, distributed across 32 GPUs over 4 compute nodes. However, accommodating this broad EP scale introduces architectural friction for the attention stage, which must adopt a hybrid of Tensor Parallelism (TP) and Data Parallelism (DP). Due to the prohibitive inter-device communication overhead incurred by large TP, the TP degree is strictly bounded (typically ≤ 8). As a result, the system configures the DP size as EP/T P to process concurrent request batches [27, 40]. Consequently, the aforementioned DeepSeek-V3 instance enforces a TP size of 8 and a DP size of 4 during its attention phase. Problem. Despite the theoretical efficiency of these hybrid parallelism strategies, our profiling reveals severe resource underutilization and inflated inference latencies during the compute-intensive prefill phase of production MoE serving. This inefficiency stems from a fundamental execution mismatch: the inevitable DP imbalance coupled with the rigid synchronization barriers inherent in state-of-the-art inference systems. As depicted in Figure 1a, heterogeneous requests are independently batched and routed to distinct attention DP groups. However, because the subsequent MoE stage operates over a globally shared expert pool, these isolated DP groups are forced to synchronize at strict barriers before and after every MoE layer. This lockstep execution paradigm creates a severe straggler effect, where the entire system stalls until the slowest attention DP group or the most congested expert completes its task. In production deployments, online inference clusters are subjected to highly stochastic request arrivals, extreme sequence length skew, unpredictable prefix cache hit rates, etc [7, 49]. Such heterogeneity results in imbalanced workload (thus inconsistent latency) across attention DP groups, a phenomenon we term DP Imbalance [31]. Related Work. Existing literature has extensively explored expert load balancing [11, 13, 34], yet the performance degradation caused by DP imbalance has been significantly underestimated in the context of online MoE serving. Because production LLM serving is characterized by highly stochastic arrivals and heavy-tailed sequence length distributions [7, 49], achieving perfect attention DP balancing is practically impossible. On another front, recent efforts in long-context acceleration [9, 14, 21, 23, 24, 26, 36, 45, 47, 54] successfully tackle

Mixture-of-Experts (MoE) models have become the de facto standard for scaling large language models. To maintain computational efficiency, modern MoE serving systems typically employ a hybrid parallelism strategy, combining Data Parallelism (DP) for attention stages with Expert Parallelism (EP) for MoE stages. However, this design necessitates frequent global synchronization barriers between attention DP groups and experts. In online serving, significant variance in request arrival rates and sequence lengths inherently leads to DP imbalance, causing severe synchronization stalls that degrade Time-to-First-Token (TTFT) and system throughput. We present ASAP, an asynchronous inference system specifically designed to accelerate the prefill phase of MoE models. ASAP disaggregates the attention and MoE stages and implements a fully asynchronous execution pipeline. This is achieved through a suite of specialized asynchronous communication primitives and four coordinated optimizations across request scheduling and model execution, which collectively dismantle global synchronization barriers. We implement and evaluate ASAP on CloudMatrix384 super-nodes, demonstrating that it improves SLO-compliant prefill throughput by 90% compared to state-of-the-art synchronous serving solutions.

1

Introduction

Background. Large-scale Mixture-of-Experts (MoE) architectures have become the de facto paradigm for serving highcapacity models efficiently. By selectively activating a sparse subset of experts per token [15, 16, 17, 35, 39, 41, 42], MoEs decouple model capability from computational cost. For example, DeepSeek-V3 [13] achieves a total scale of 671B parameters while restricting active parameters to only 37B per token. Since the vast majority of these parameters reside in expert weights, Expert Parallelism (EP) is commonly applied to the MoE stage, distributing experts across an array of accelerators. A wide EP configuration fundamentally reduces the memory footprint per device, directly alleviating the severe memory bottlenecks typical in LLM inference. Specifically, a * Corresponding author. Email: chenshuang0804@gmail.com

1

Communication Communication

C C

A AIDLE

IDLE

A A

IDLE IDLE

M M M M M M

Request Load Request stream Load 2. Maintaining Compute Efficiency under Fragmented stream and Out-of-Order Execution: Asynchrony introduces Prefill Instance-3 "fragmented" execution; instead of one massive batch, MoE Prefill Instance-3 Prefill Instance-2 devices receive independent, smaller batches, risking a drop Prefill Instance-2 Prefill Instance-1 in arithmetic intensity. Furthermore, DP imbalance forces Attentio Prefill Instance-1 MoE devices to execute layers out-of-order, where the spe-1 Attention h c DP-1 cific layer ID is only resolved at runtime. This dynamic acth1 Device b Request 1 DP-1 t execution pattern significantly increases kernel dispatch bba Request Device 1 Scheduler overhead from the host CPU, potentially introducing amore Attentio ba tch2 Attention Scheduler execution bubbles than the gains obtained from asynchrony. tch DP-2 2 2 DP-2 Our Work. We present ASAP, a high-performance infer- Device

Idle TimeIdle IDLE IDLE

Time

… …

MoE MoE

All2All All2All

Attention M A Attention M Request Batch-1 Request Batch-2 Batch-1 A Request Batch-2 A Request Batch-N Request Batch-N

All2All All2All

A

A A

(a) Synchronous Time

Request Batch-1 Request Batch-2 Batch-1 A Request Request Batch-2 A Request Batch-N Request Batch-N

A MA C

C C M A A

C

C C C C

M M M C (b) Asynchronous M C

C C

M Time C M C SAVED LATENCY SAVED LATENCY

ence system that realizes fully asynchronous MoE prefill. Device 2 ASAP disaggregates attention and MoE stages onto separate devices, interconnected via asymmetric, asynchronous communication primitives to enable non-blocking data transfers. The key is to create a dedicated shared memory buffer on each device that is visible to all devices and acts as a “superhub” between senders and receivers. This is the foundation of ASAP’s asynchronous execution pipeline. To fully leverage this asynchronous architecture and mitigate the efficiency risks of asynchrony, ASAP integrates Prefill four key runtime optimizations: (1) Length-aware batchPrefill I ing toRequest preserveQueue MoE compute intensity; (2) Request Dual-batch in§3.3: Scheduler CPU Request Queue S=4K §3.3: Request Scheduler CPU terleaving to maximize attention device duty cycles; (3) S=2K Length-Aware Dual-Batch S=4K S=2K S=2.4K Communication-computation overlapping via a tripleDual-Batch Length-Aware S=0.5K S=2.4K Batchingand (4)Interleaving stream design to S=3K maximize hardware utilization; A S=0.5K Interleaving Batching S=3K layer-oblivious MoE Super Kernel that enables bubble-free Attn D kernel dispatching, allowing out-of-order execution without Attn DP-1 NPU NPU NPUAttn NPUDP N NPUAttn NPUDP-1 host-side bottlenecks. NPU NPU NP NPU NPU NPU NPU In summary, we make the following contributions:

Figure 1: Comparison between current synchronous and ideal asynchronous execution under heterogeneous request batches. Synchronous execution (Figure 1a) requires all request batches (launched on different attention DP groups) to synchronize before the MoE computation. An ideal asynchronous execution pipeline (Figure 1b) eliminates the barrier, and allows faster request batches to progress at their own pace.

super-long requests, but remain oblivious to the execution bubbles caused by heterogeneous short-to-medium requests (e.g., prompts with < 32K tokens). Our characterization in Section 2.2.1 shows that DP imbalance does not necessarily come from super-long requests. Co-executing various shortto-medium requests in a synchronous inference system still stalls the pipeline and degrades overall throughput. Motivation. Figure 1b illustrates an ideal, barrier-free, §3.2:an Disaggregated 1. To the best of our knowledge, we are the first to build asynchronous execution paradigm. By eliminating global §3.2: Disaggregated A Asynchronous Comm asynchronous execution pipeline for online inference. synchronization, heterogeneous requests are enabled to truly Asynchronous Commu “flow” at their own pace without waiting for or delaying others. 2. We systematically analyze the prefill phase MoE of MoE serv- MoE EP-2 EP-1 This approach effectively minimizes resource idling, which siMoE EP-1 EP-2 ing. We identify an inherent DP imbalance problem in MoENPU NPU multaneously reduces inference latency and improves overall NPU NPU synchronous systems, causing severe resource idling and throughput. limited prefill performance. Challenges. Transitioning from synchronous to asynAtten MOE Devices Attention Devices chronous MoE serving presents two fundamental architectural Attent MOE Devices3. We design ASAP, the first asynchronous MoE inference Attention Devices Attn DP1 NPU 1 system hurdles: Attn DP1 Attn DP DP1 NPU1that dismantles global synchronization barriers to Attn TP-1 2 TP-2 NPU DP1 TP-1 2 Attn DP Attn DP … enable independent execution of request batches. 1 2 1. Decoupling Execution Dependencies with AsynNPU2 DP TP-2 Attn DP2

1 TP-T … TP-1

① Poll

Attn DPD

TP-2 DPTP-T 1 TP-1 … AttnCommunication DP2 … Poll NPUE Attn DPD chronous Primitives: Traditional 1 TP-2 DP1TP-1 Flags … TP-T 4. ① We … ① implement PollNPUE and evaluate ASAP on CloudMatrix384 [56] 1 TP-1 Flags TP-T TP-2 … TP-1 2 3 1 systems tightly Attention DP and MoE on D DP … TP-2 stages ① Poll Attn DPcouple 1 TP-1 1 D … 1 2 Flags TP-2 supernode. Extensive evaluation results show that … TP-T TP-1 2 3 1 DPD DP …1 TP-2 DPD TP-T … 1 11 2 22 Flags ③Clear TP-2 ① Poll the same Attn devices, enforcing a lock-step execution 2that Buffer 3 1 DPTP-T 1 2 D TP-1 … 1 2 TP-T Shared … ③Clear ASAP improves SLO-compliant prefill throughput by 90% 1 2 ①Flags Poll TP-T 2 3 1 DPD TP-2 Flags NPUD1 … 1 11 2 22 … Buffer ③Clear ②Copy Data Shared TP-TTP-1 precludes overlapping different computation stages. Flags NPUD1 Flags 1 2 compared to state-of-the-art synchronous baselines. TP-2 1 Shared Buffer ③Clear ②Copy Data 2 3 DPD TP-1 … Flags 1 21 Shared Buffer TP-T 2 3 1 Transitioning to an Attention-MoE decoupled architecture DPD TP-2 ②Copy … Data Flags 1 ③Clear TP-T ②Copy Data NPU Shared B D2 ① Write Shared Buffer ③Clear requires not NPU onlyD2physical resource disaggregation, but Flags Shared Bu ① Write Private Memory Shared Buffer ②Copy Data Data ②Copy Flags Private Memory ②Copy Data also a new class of asymmetric Data and asynchronous com②Copy D Private Memory ②Set 2 Background and Motivation ②Set munication primitives. Current collective operationsPrivate (e.g., Memory NPUDT Flags NPUDT Flagsdesigned for synchronous Private M Private Memory Peer-to-Peer, All-to-All) are Private Me Private Memory async-dispatch-recv async-dispatch-send blocking bulk transfers; the lack of efficient primitives In this section, we provide the necessary background on MoE asyn async-dispatch-recv async-dispatch-send async that support fine-grained, non-blocking token routing is a serving, and present a series of characterization studies to primary reason why fully asynchronous MoE inference motivate the need for a barrier-free, asynchronous system to remains elusive. optimize MoE prefill performance.

2

All-to-All

Expert1 All-to-All

Attention 1 DP-1 tch a Request b Device 1 Scheduler batc h2 Attention DP-2

Expert2 Expert3 Expert4

Device 2

+ Decode Instances

+

 

Attn DP-1

NPU  NPU NPU NPU 





6SDUVH$$WWHQWL WWHQWLRRQQ 6SDUVH

Attn DP-2

NPU NPU NPU NPU

…

Attn DP-D

1RQ  1RQ NPU NPU NPU NPU QHDU OLQOLHDU

§3.2: Disaggregated Attention-MoE  and Asynchronous . . . . Communication . Primitives.... . . .

6HTXHQFH HQJWK 6HTXHQFH //HQJWK MoE EP-2

MoE EP-1

NPU NPU Attention (a)

MoE EP-E

…

QHDU 5HJL /L/L QHDU 5HJL RQRQ

..

6HTXHQFH/HQJWK /HQJWK 6HTXHQFH

NPU

. .

(b) MoE

Figure 3: Latency scaling increasing sequence length. Attentionwith Devices MOE Devices Attn DP1 Attn DP2

U2

Attn DPD

2.1

NPUD1

NPU NPUD2D14 NPU DT NPU 3 NPU 1 1 2 2 3Shared 4D24 D2 Buffer 1 2 3 NPU 4 11 22 33 4DT4 NPU DT Shared Buffer Shared Buffer 1 2 3Shared 4 Buffer 1 2 3 4 Shared Buffer 2 3 1 Shared Buffer Shared Buffer Shared Buffer Shared Buffer ②Copy Data ① Poll Flags ③Clear Flags 1

2

3

LLM Inference Preliminaries

;

;

;

     

1.00 0.75 Min: 31 Max: ~100K Avg: ~5K 0.50 PDF of Prompt Length 0.25 CDF of Prompt Length 0.00 0 25K 50K 75K 100K

CDF

; ;

×10 4

Sequence Length

(applicability to the decode phase is discussed in Section 4.4). Upon the ingress of a user request, a cluster-level load balancer routes it to a designated prefill instance, typically following a round-robin policy. Within the assigned instance, the request is aggregated into a batch, and dispatched to a specific attention DP group by the instance-level request scheduler. After the first token is generated, the request along with its KV cache is dispatched to a designated decode instance to generate subsequent tokens. To accommodate the massive parameter scale of MoE models, each serving instance typically employs wide Expert Parallelism (e.g., EP = 32 in DeepSeek V3 [13]) to distribute experts across multiple accelerators, thereby alleviating the memory pressure per device [13, 55]. For the attention stage, Tensor Parallelism (TP) is usually applied to reduce attention latency. However, due to the large volume of communication, TP is generally limited to values of no more than 8. To match the large size of the EP, the attention stage also adopts Data Parallelism (i.e., DP = EP/T P), as shown in Figure 2, .

§3.4: Model Executor CPU Comm-Comp Bubble-Free HFWL 3RL ,Q,IOQHIOFWL RQRQ 3RL QWQW Overlapping Dispatching

/DWHQF\ /DWHQF\PV PV

/DWHQF\PV PV

S=3K

§3.3: Request Scheduler CPU Length-Aware 7RWDO/DWHQF\ Dual-Batch /DWHQF\ Batching 7RWDO  Interleaving KWQL J,QGH[HU /L/LJJKWQL QQJ,QGH[HU

;

2.0 1.5 1.0 0.5 0.0

Figure 4: Impact of batch Figure 5: CDF and PDF of sequence size on attention latency un- lengths from a production dataset on der a fixed total sequence Huawei Cloud, demonstrating extreme length of 32k tokens. request variance.

Prefill Instance S=4K

     

%DWFK

Figure 2: A typical MoE serving cluster consists of a number of serving instances. A user request is routed to a designated instance through a cluster-level load balancer. It is then aggregated into a request batch with other incoming requests, and dispatched to a specific attention DP group by the instance-level request scheduler.

S=2K S=2.4K

l ar s ear gs

/DWHQF\PV

…

Prefill Instance-3 Prefill Instance-2 Prefill Instance-1

est Queue

NPUE

Load Balancer

PDF

Request stream

NPU1 NPU2 NPU3

Prefill and Decode Phases. In LLM serving, a user ① Write Data request ②Set Flags first undergoes the prefill phase to generate the initial token, followed by the decode phase to generate NPU subsequent Private Memory tokens autoregressivelyasync-combine-recv [8, 51]. Inference latency is primarily async-combine-send characterized by Time-to-First-Token (TTFT) for prefill and Time-Per-Output-Token (TPOT) for decode. Since TTFT reflects how long the user has to wait before getting responses, its service-level objective (SLO) is usually within a few seconds [10, 19]. In contrast, SLO of TPOT is usually at tens of milliseconds, to keep up with the user’s reading speed. Given their divergent profiles—prefill being computebound and decode being memory-bound—it has become the standard practice to adopt prefill-decode disaggregation in production-grade inference frameworks [22, 37, 48, 53]. MoE Architectures. MoE models replace monolithic feedforward networks (FFN) with a set of sparse, specialized subnetworks (experts). By activating only a fraction of parameters per token, MoE enables massive model scaling without a proportional increase in inference FLOPs. This architecture has rapidly become the de facto standard for state-of-theart LLMs, as evidenced by the recent releases of DeepSeekV3 [13], Kimi-K2 [42], and Qwen3-MoE [50]. MoE Serving Infrastructure. Figure 2 illustrates the typical architecture of an MoE serving cluster that is composed of a number of serving instances. Prefill-decode disaggregation is adopted, and we focus on illustrating the prefill phase E

In this paper, we focus on optimizing performance of the prefill instances in MoE serving with a combination of attention DP and expert EP.

2.2

Characterization of MoE Inference

To thoroughly analyze the runtime characteristics of the MoE prefill phase, we deploy DeepSeek-V3.2 [30], a state-of-theart open-sourced MoE model released in Dec 2025, on 32 Ascend NPUs [56]. Our setup follows the baseline configuration described in the DeepSeek technical report [13], utilizing a hybrid parallelism strategy of EP = 32, DP = 8, and T P = 4. Comprehensive specifications of the model and hardware platform are detailed in Section 5.1. 2.2.1

Attention

The attention stage is compute-intensive for the prefill phase. Figure 3a plots attention latency as a function of sequence length s for a batch size of one. It shows that the latency 3

Request Batch-N M C

Prefill Instance

exhibits a quadratic correlation with s. Although DeepSeekQueue V3.2 incorporates DeepSeek Sparse AttentionRequest (DSA) [30] S=4K where the sparse attention computation itself scalesS=2K linearly, S=2.4K S=3K O(s), the associated lightning indexer retains anS=0.5K O(s2 ) complexity. The combined effect of these operations dictates the overall quadratic scaling. In production, request schedulers dynamically bundle multiple requests into a single batch to maximize hardware utilization. However, batching to a uniform total sequence length does not yield deterministic execution time. Figure 4 demonstrates the attention latency for a fixed total budget of 32k tokens across varying batch sizes. A batch size of 32 (1k toMOE Devices Attention kens perDevices request) compared to a batch size of 1 (32k tokens NPU DP per Attn request) reveals a stark latency disparity of up NPU to 4.2×. DP Attn DP Poll This significant varianceDParises because the①Flags attention … NPU com① Poll 3 plexity requests is…2 governed by the sum of DP 1 Attnfor DP a batch of n DP Flags ③Clear ①nPoll 2 2 3 1 DP Shared … Buffer the squares of individual sequence lengths, O( s ∑ Flags NPU Flags i=1 i ) [46], ③Clear ②Copy Data Shared Buffer 2 3 1 DP Flags rather than the square of their ②Copytotal Data sum. ③Clear

§3.3: Request Scheduler CPU Length-Aware Dual-Batch Interleaving Batching Attn DP-1

NPU NPU NPU NPU

TP-1 1 TP-2 … TP-T TP-1 1 TP-2 … TP-T TP-1 TP-2 TP-1 D … 1 TP-2 TP-T TP-1… TP-T TP-2 D … TP-T TP-1 TP-2 D … TP-T

2

D

D1

NPUD2

① Write Data ②Set Flags

Shared Buffer ②Copy DataPrivate Memory Private Memory

MoE EP-1

2.2.2

…

Attn DP-D NPU NPU NPU NPU

NPU

…

MoE EP-E NPU

Figure 6: Attention ASAP Overview. Devices

MOE Devices

Attn DP1 Attn DP2

NPU NPU Huawei Cloud 1 traces. While the mean prompt length sits Attn DP NPU 1 2 3 4 NPU at 5k tokens, the distribution exhibits aNPU massive variance— NPU NPU4 4 3 NPU 1 1 2 2 3Shared Buffer 1 2 to a heavy 3 NPU 4 extending up to ranging from a mere 31 tokens tail 11 22 33 44 NPU Shared Buffer Shared Buffer NPU 1 2 3Shared 4 Buffer 100k tokens. As established in Section 2.2.1, excluding the 1 2 3 4 Shared Buffer 2 3 1 Shared Buffer ① Write Data Shared Buffer impact of prefix cache, simply matching the total token count Shared Buffer Shared Buffer ②Set Flags ②Copy Data ① Poll Flags across attention DP groups is still fundamentally inadequate. NPU ③Clear Flags Perfect load balancing would require that the sum of squared Private Memory sequence lengths (O(∑ s2i ))async-combine-recv precisely match, a constraint that async-combine-send is virtually impossible to satisfy dynamically.

2

D1

D

E

Flags

1

D2 D1

DT D2 D2 DT DT

2

3

E

The combination of O(s2 ) attention complexity, heavytailed sequence length, and other factors (e.g., prefix cache) inherently causes latency variations across attention DP groups. Such DP imbalance leads to execution bubbles and resource idling in synchronous systems.

MoE

Conversely, the latency of the MoE stage is dictated primarily by the aggregate token count within a batch. Figure 3b shows the scaling of MoE latency with s under perfect expert load balancing. It reveals that MoE latency remains initially flat, transitioning to linear scaling only after surpassing an inflection point (approximately 2k tokens). This plateau occurs because the MoE stage is strictly memory-bound at low token count; the hardware must load massive expert weights from memory while performing relatively few computation operations. As sequence length grows, arithmetic intensity improves, pushing the system into a compute-bound regime. While the exact inflection point varies across hardware platforms and model configurations, this dual-regime scaling behavior—an initial memory-bound plateau followed by a linear compute-bound phase—is an intrinsic characteristic of MoE computation.

3

ASAP Design

In this section, we present ASAP, an asynchronous inference framework designed to accelerate the prefill phase of MoE models. By replacing the rigid barriers of conventional synchronous systems with a fully decoupled and asynchronous execution pipeline, ASAP targets minimizing resource idling and maximizing prefill throughput under SLO constraints. To facilitate the subsequent discussion, Table 1 lists all the symbols, definitions, and the representative configurations (detailed in Section 5.1) used throughout this paper.

3.1

The MoE stage requires a minimum aggregate token count to saturate hardware compute units and transition from a memory-bound plateau to a high-efficiency linear regime.

2.3

MoE EP-2

NPU

Attention latency scales quadratically with individual reNPU Private Memory quest lengths. Consequently, the aggregate token count of async-dispatch-recv async-dispatch-send a batch is a poor predictor of execution time. DT

Attn DP-2

NPU NPU NPU NPU

§3.2: Disaggregated Attention-MoE and Asynchronous Communication Primitives

1

1

§3.4: Model Executor CPU Comm-Comp Bubble-Free Overlapping Dispatching

Overview

Figure 6 provides an overview of the ASAP architecture. ASAP first adopts a disaggregated design by partitioning attention and MoE stages onto separate hardware devices. Specifically, the attention stage occupies D × T NPUs, while the MoE stage is deployed across E NPUs , resulting in a total hardware footprint of D × T + E devices per prefill instance. To bridge these disaggregated components, we develop a suite of asynchronous communication primitives that support

DP Imbalance in Online Serving

In production environments, inference clusters are subjected to highly stochastic request arrivals and extreme heterogeneity in sequence lengths, prefix cache hit rate, etc [7, 49]. Figure 5 visualizes the sequence length distribution from real-world

1

A token service in production. The service and company names are kept anonymous for double blindness.

4

Table 1: Symbol Definition. Representative configurations are detailed in Section 5.1.

D T E Etotal K L H Dsize S

Representative Configurations 4 4 16 256 8 61 7168 16 bits 32k

Definition

MoE Attention

Symbol

Table 2: Shared Buffer Structure. Example size is derived from the representative configurations specified in Table 1.

Number of attention DP groups TP in each attention DP group EP Number of experts in the model TopK in the model Number of layers in the model Hidden size of the model Data size Maximum batch sequence length

ID Memory Size Example Size Description ① K × S/T 64KB Expert IDs ② H × K × S × Dsize/T 0.9GB Expert results ③ E/8 <1KB E-bit bitmap flag ① D × T × Etotal /E <1KB Token metadata ② D × H × K × S × Dsize 14GB Tokens (hidden states) ③ D × T /8 <1KB D T-bit bitmap flags

3.2.1

Asynchronous Dispatch

This occurs between an attention device and a set of MoE devices following each attention layer. Each MoE device allocates a shared buffer partitioned into D regions (one per attention DP group), with each region further subdivided into T rows (one per TP member within that DP group). As illustrated in Figure 7a, the shared buffer consists of three sub-regions: ① token metadata stores token counts per expert (Etotal /E integers; ② token payload stores the actual token hidden states; and ③ a T -bit bitmap indicating data readiness for each TP member. Execution Flow: An attention device NPUi j invokes async-dispatch-send to write its local tokens into the j-th row of the i-th region in each target MoE device. To ensure data integrity, the sender employs a backpressure mechanism: if a target flag is already set, the write blocks until the receiver clears the flag. Once the transfer completes, the sender sets its corresponding bit in the bitmap (③). On the receiver side, the MoE device executes async-dispatch-recv by polling the bitmaps across all regions. When all T flags for a specific DPi are set, the MoE device migrates the token metadata (①) and hidden states (②) to its private memory and clears the bitmap ③ to acknowledge availability for subsequent transfers.

non-blocking 1-to-N/N-to-1 data transfers. This structural decoupling forms the foundation of our barrier-free inference pipeline, allowing the attention and MoE stages to progress independently without global synchronization. Based upon this foundation, ASAP incorporates four key mechanisms within the request scheduler and model executor to maintain high hardware utilization: 1. Length-aware batching to preserve MoE computational efficiency even under fragmented workloads; 2. Dual-batch interleaving to prevent attention devices idling during active MoE processing; 3. Communication-computation overlapping via a triplestream design to maximize hardware resource utilization; 4. Bubble-free kernel dispatching to eliminate host-side dispatching overheads during out-of-order execution.

3.2 Asynchronous Communication Primitives Conventional peer-to-peer and all-to-all collectives are inherently blocking, requiring explicit handshaking between senders and receivers. To eliminate this bottleneck, we implement a distributed shared-memory abstraction, where each device allocates a globally visible buffer as a decentralized communication hub. These buffers are statically allocated during initialization and persist throughout the framework’s lifetime. Table 2 details the buffer structure and memory footprint on each device. Under this model, senders write to the target buffer and immediately resume computation without waiting for receiver acknowledgment. Similarly, receivers independently retrieve data from these buffers upon completing local tasks. We leverage this abstraction and develop four specialized primitives, two for asynchronous dispatch and another two for asynchronous combine. These are designed to replace their synchronous counterparts, both of which are traditionally implemented as blocking all-to-all collectives.

3.2.2

Asynchronous Combine

This retrieves computed expert results from multiple MoE devices back to their originating attention DP group after each MoE layer. The attention device’s shared buffer comprises: ① expert IDs for token-to-expert mapping; ② expert results partitioned into E segments (one per MoE device); and ③ an E-bit bitmap flag that indicates result arrival from each MoE device. Execution Flow: Upon completing the computation for DPi , an MoE device invokes async-combine-send to write expert results into the shared buffer of the T attention devices belonging to DPi , and sets the corresponding completion bit. The attention device executes async-combine-recv, monitoring the bitmap flags. Once all activated expert results are received, the attention device migrates the payload to its private memory and clears the flags for the next iteration. 5

MOE Devices Attention Devices

n Devices

DP1

TP-1 Attn DP1 DP1 TP-2 … TP-T TP-1 DP 2 TP-2 DP1 Attn … … TP-T TP-1 TP-2 TP-1 … 2 1 DPD DP … 1 TP-2 … Attn DPD TP-T TP-1 TP-T 2 1 DPD TP-2 … … TP-T TP-1NPUD1 TP-2 2 DPD … 1 TP-T

DP2

① Write Layer ID Data MoE-Li： H ②Set

Layer 1

x

Layer i

…

…

MoE-LL： H

Layer x (a) Asynchronous Dispatch L

NPU

NPU

Devices AttentionMOE Devices

Attn DP1 Attn DP2

NPU1

MoE EP-E

…

NPU

MOE Dev

Attn DP1 Attn DP2 NPU

NPU1 D1 NPUD1 NPU Kernel Dispatch 1 Attn DPD NPU NPU NPUD2D14 1 2 3 NPUD2D14 NPU2 NPUDT NPU DT NPU2 NPU NPU4D24 3 NPU 1 1 2 2 3Shared 4D24 3 NPU 1 1 2 2 3Shared D2 Buffer D24 Buffer 1 2 3 NPU 4 1 2 3 NPU DT DT 11 22 33 44 NPU 1 2 3 4 DT NPU DT 1 2 3Shared 4 Shared Buffer Shared Buffer NPU3 Buffer Buffer 1 2 3Shared 4 Shared NPU3 1 2 3Shared 4 Buffer Buffer 1 2 3 4 1 2 3 4 Shared Buffer MoEShared 2 3 1 Shared Buffer 2 DataBuffer 3 1 Shared Buffer ① Write ① Write Data Shared Buffer Shared Buffer Shared Buffer Shared Buffer Shared ②Set FlagsBuffer Shared Buffer ②Set Flags ① Poll Flags②Copy Data ① Poll Flags LL ②Copy Data NPU NPUE E ③Clear Flags ③Clear Flags

Attn DPD

MoE Super Kernel

1

2

3

Layer CPU 1 NPU … Pre-calculated address list Layer Li … x H i Private Memory Private Memory … CPU async-combine-send async-combine-recv async-combine-recv Layer ID Layer (b) Asynchronous Combine L NPU

IDLE

Figure 7: Asynchronous communication primitives and the shared buffer design. The buffer structure on each device is detailed in Table 2. Senders write data and set flags into the target device’s shared buffer, returning to computation without synchronization. Receivers monitor these flags to retrieve and clear data as their own tasks progress.

3.3

Lx Attention of layer x

Request Scheduler

Async-combine

Async-dispatch-send:

2

Async-combine-recv: L1 Main:

L1

2 L8 4 1

1

3 L2

L2

3

5

2 3 L1B1-L1L8 L1

4 5 L8

Async-dispatch-send: 1 Async-dispatch-recv: Main: Async-combine-send:

4

L9

1

2

3

L2 4

5

Figure 8: Dual-batch interleaving and the triple-stream design for communication-computation overlapping. For each attention DP group, two request batches are co-scheduled to interleave their layerwise attention processing together. All devices employ a tripleB2-L1 stream design to further overlap asynchronous communication primitives with the main active computation kernels.

Length-Aware Batching

As characterized in Section 2.2.2, the MoE stage requires a minimum token count (e.g., 2k tokens) to reach the computebound linear regime. Because the MoE stage processes batches from independent attention DP groups serially, maintaining a high per-batch token count is critical to saturate hardware utilization in an asynchronous environment. Our resource scheduler enforces a policy that aggregates requests into batches exceeding the inflection point in Figure 3b (e.g., 2k tokens in our setup). Notably, since ASAP allows DP groups to progress independently, the batching algorithm is significantly simplified as it no longer needs to optimize for load balancing across attention DP groups. 3.3.2

Async-combine-recv: L8 Main:

MOE

While the asynchronous framework eliminates synchronization stalls, it introduces new challenges for maintaining high Async-combine Attention hardwareAsync-dispatch utilization. Specifically, two primary issues arise. MoE Attention of the interleaved batch First, in a decoupled architecture, MoE devices process reTime quest batches sequentially rather than concurrently across all B1’-L1 B1-L2 tn DP-1: B1-L1 4 4 D1 attention DP1 groups. This leads to reduced token density B2-L1 B2’-L1 ttn DP-2: per MoE may compromise computational 2 invocation, 5 2 which efficiency. resource disaggregation creates execution B3’-L1 B3-L1Second, ttn DP-3: 3 6 3 gaps; attention devices become idle while their respective batches are in3 the MoE stage. Dispatch: 1 2 4 To address these challenges, we propose two scheduling B1’-L1 B1-L1 B2-L1 B3-L1 MOE Computation: optimizations as follows. Combine: 1 2 3 4 3.3.1

Async-dispatch

Lx Attention of layer x of the interleaved batch Lx MoE of layer x Time

Attn DP-1

patch-send

Flags

x

MoE EP-E MoE EP-2 MoE EP-1 … NPU

Attention Devices

MOE Devices NPU1

…

NPUDT

…

NPUD2

NPU

TP-1 TP-2 2 NPU2 DP1NPU … ① Poll TP-TTP-1 ① Poll TP-2 DP1NPU … … E Flags TP-1TP-T Flags NPUE ① Poll TP-1 ① Poll 3 TP-2 1 … 2 3 DPD DP … 1 TP-2 Flags Flags TP-T TP-1… ③Clear TP-T ③Clear ① Poll 3 TP-2 ① Poll 2 3 DPD … 1 Shared Buffer Shared … Buffer Flags TP-T Flags Flags Flags ③Clear ②Copy Data ③Clear ②Copy TP-1 Data Shared Buffer 3 Shared Buffer 2 3 1 DPD TP-2 Flags … Flags ②Copy Data TP-T ②Copy Data ③Clear ③Clear NPUD2 Shared Buffer ① Write Shared Buffer Flags Flags ②Copy DataPrivate Memory ②Copy DataPrivate Memory Data L1 … Private Memory Private Memory ②Set NPUDT Flags Private Memory Private Memory async-dispatch-recv async-dispatch-recv async-dispatch-send

MoE-L1： H

NPUD1

MoE EP-2

NPU

Attn DP-2

n DPD

MoE EP-1

As illustrated in Figure 8, the attention DP group interleaves the layer-wise processing of the two batches (the dark red and light red bars). When one batch is offloaded to the MoE stage, the attention devices immediately commence processing for the other batch of the same layer. The efficiency of this interleaving depends on the relative latencies of the attention and MoE stages. • Attention-limited cases: If attention latency significantly exceeds MoE latency, each batch may experience queueing delays on attention devices. As shown in Figure 3, for sequences exceeding 16k tokens, MoE latency accounts for less than 15% of attention latency. Consequently, we disable interleaving for these long sequences, allowing them to exclusively occupy an attention DP group to minimize queuing from dual-batch interleaving.

Dual-Batch Interleaving

To eliminate attention device idling during MoE execution, we porpose dual-batch interleaving. Rather than scheduling a single batch immediately upon reaching the inflection point, ASAP waits for additional requests to form a dual-batch pair, and co-schedule the two batches to an idle DP group. 6

async-combi

MoE Super Kernel

Layer H x MoE-L1： H MoE-L1： 1 x Layer 1 MoE-L1： H

x

…

…

Layer i

x …

… …

…

…

LayerH x MoE-LL： H MoE-LL： L x Layer L MoE-LL： H

…

Layer MoE Super Kernel Layer 1 1 Pre-calculated address list … Pre-calculated address list … Pre-calculated address list Layer H L1 … Li …L1 LL… Li H… LL x H L1 … Li … LL x Layer i i … … Layer Layer ID Layer ID Layer ID L Layer L

…

… …

…

Layer ID Layer ID Layer H x MoE-Li： H MoE-Li： yer ID i x Layer i MoE-Li： H

Layer L

x

Kernel Execution Kernel Execution Kernel DispatchKernel Dispatch Kernel Execution Kernel Dispatch

MoE Super Kernel MoE Super Kernel

Layer 1

(a) Per-layer MoE kernels.

x

(b) Unified MoE Super Kernel

Layer CPU 1CPU NPU … NPU Layer i … CPU Layer CPU L NPU NPU

CPU IDLE IDLE NPU

Time Time IDLE IDLE IDLE MoE-Li IDLE MoE-LiMoE-Lj

MoE-Li

MoE-Lj

Time

MoE-Lj

Time

CPU MoE Super MoE Super NPU Kernel MoE Super Kernel MoE Super Kernel Kernel

TimeSuper MoE Kernel

Figure 10: The layer-oblivious design of the MoE Figure 9: Comparison between standard per-layer MoE kernels and the proposed MoE Super Kernel. Block “H” denotes the tensor of hidden states, Block “Layer i” Super Kernel allows ahead-of-time dispatching, Lx Attention eliminating idling between kernels. Async-dispatch Async-combine Lx Attention of layer of layer x Async-dispatch Async-combinedevice x denotes the tensor of the expert weights of the ith layer. Lx Attention of layer x of the interleaved Lx MoE ofbatch Lx layer xLx MoE of layer x Attention of the interleaved batch Lx Attention of layerofx layer x Async-dispatch Async-combine Attn DP-1

Attn DP-1

Attn DP-2

Attn DP-2

MOE

MOE

Async-combine Async-dispatch Attention Async-combine Async-dispatch MoE Attention of the interleaved batch MoE of the interleaved batch Async-combine Async-dispatch Attention Attention Time Time MoE Attention of the interleaved batch B1’-L1 B1-L2 B1’-L1B1-L1 B1-L2 Time Attn DP-1: B1-L1 Attn DP-1: 1 1 4 4

Attention

Attn DP-1

Lx Attention of layer x of theAsync-combine-recv: interleaved batch Lx MoE of layer x Async-combine-recv:

Main: Async-combine-recv:

L8 Main:

2

L8L8

2

4

4 L9L8

Time

Time

2 Time

L9

4

1

1 2

2

3 5

3

1

3

2

2

3

3

3

1

6

2

2

3

2

6

2

4

4

1

1

4

25 4

2

2

1

4

4

3

3

3

1

5

3

4

6

4

2

3

4

MOE

1

1

Attn DP-2

4 • MoE-limited cases: Conversely, if attentionMain: latency ofL8the Async-dispatch-send: execution order is highly nondeterministic, driven entirely by 2 4L9 2 Async-dispatch-send: L8 interleaved batch is shorter than MoE latency, then idling the independent progress 2 4 Async-combine-recv: 1 Async-dispatch-send: 3 of each attention DP group. Async-combine-recv: 1 3 L1 L2 L1 L2 L2 L2 Main: L1 L1 Main: of attention devices will persist. However, with lengthAsync-combine-recv: 1 3 This dynamic execution order poses a significant challenge 3 1 5 5 Async-dispatch-send: B2-L1 Attn B1-L1 B1’-L1 L1 L1 1 B2-L1DP-2: B2’-L1B1-L2 L2 3 L2 Main: Async-dispatch-send: Attn DP-2: aware batching,B2’-L1attention execution typically surpasses for kernel dispatching. Typically, host CPUs launch NPU ker3 5 B3’-L1 Async-dispatch-send: AttnB2’-L1 DP-3: B2-L1 B3’-L1 B3-L1 B3-L1 Attn DP-3: several milliseconds, comfortably overlapping with the sub-1 nels asynchronously in advance, effectively hiding the host1 3 4 5 Async-dispatch-recv: 1 2 3 4 25 Async-dispatch-recv: B3’-L1 B3-L1 millisecond to few-millisecond MoE latency. Main: L8L2 L1 L8 L2 argument preparation, memory Dispatch: Main: side L1B1-L1L8 L1 L1B1-L1 L8 dispatch overhead (e.g., Dispatch: Async-dispatch-recv: 1 2 3 4 5 5 1 4 25 3 4 Async-combine-send: MOE Computation: 1 2 3 B1’-L1 B2-L1 B3-L1 B1’-L1 Async-combine-send: B1-L1 B2-L1 B3-L1 B1-L1 MOE preparation, runtime API calls) behind acL1B1-L1allocation, L8 L1 L8 tiling L2 Main: ch: Computation: Combine: Combine: 5 execution. However, because the exact layer 1 tive 2 accelerator 3 4 B1’-L1 Async-combine-send: B1-L1 B2-L1 B3-L1 utation: 3.4 Model Executor ne: ID to be executed next by the MoE stage is resolved dyThe model executor serves as the core engine for inference namically at runtime, kernels cannot be pre-launched. This B2-L1 execution. It orchestrates parallelism strategies, interfaces B2-L1 forces the CPU dispatch overhead—ranging from tens to hunwith the underlying hardware, and dispatches NPU kernels. B2-L1 of microseconds—onto the critical path (Figure 10). dreds Two primary runtime optimizations are proposed to minimize Given that individual MoE kernel execution spans merely resource idling and maximize system throughput. sub-millisecond to a few milliseconds, this dispatch overhead constitutes a non-negligible performance bubble. 3.4.1 Communication-Computation Overlapping To eliminate these execution bubbles, we redesign the standard per-layer MoE kernel (Figure 9a) into a unified layerCommunication typically accounts for 10% to 20% of tooblivious MoE Super Kernel (Figure 9b). MoE kernel is tal execution latency [18]. To effectively mask communicatypically a Grouped Matrix Multiplication (GMM), with hidtion overhead, ASAP employs a triple-stream concurrency den states being the left tensor, and expert weights of a specific model on each device (Figure 8). A dedicated compute stream layer being the right matrix. MoE Super Kernel introduces executes the model’s forward pass, while two independent three key modifications: communication streams handle asynchronous data transfers, 4

ensuring that the hardware compute units remain saturated. Specifically, the main compute stream of each attention device interleaves the layer-wise forward passes of the dual batches, while that of each MoE device processes MoE layers out-of-order. Each communication stream handles one of the four asynchronous primitives. By mapping these independent tasks to isolated hardware streams and allocate dedicated hardware resources to each stream, ASAP approaches noninterfering execution (detailed in Section 4). 3.4.2

• Global Weight Access: The kernel is granted pointer access to the expert weights across all L layers. Because these weights are already statically resident in the MoE device’s HBM, this modification incurs zero additional memory footprint. • Pre-calculated Address Indexing: The memory offset for each layer’s weight tensor is pre-computed and stored in an on-device address array for constant-time lookup. • Dynamic Resolution: The layer ID is treated as a dynamic device-side argument rather than a host-side compilation constant. The kernel directly indexes the address array using the input layer ID to locate the correct target tensor before executing the GMM operation.

Bubble-Free Kernel Dispatching

As shown in Figure 6, the MoE stage is a shared by multiple attention DP groups. Due to asynchronous processing, the MoE module executes layers in a nondeterministic, out-oforder manner. For instance, it may interleave the execution of Layer 8 from one DP group with Layer 1 from a newly scheduled batch in another DP group (Figure 8). This layer

In this way, the MoE Super Kernel is oblivious of layer ID, allowing ahead-of-time kernel dispatching and eliminating device idling between MoE kernels (Figure 10). 7

Time

4.2

Scale Up UB Plane L2 Level Switch

ASAP is integrated into a production-grade inference framework [22] based on PyTorch 2.1 and CANN 8.3 [2]. For our primary evaluation of DeepSeek-V3.2, we adopt a disaggregated Attention-MoE configuration with D = 4, T = 4, E = 16, totaling 32 NPUs. This configuration maintains the same device count as the standard DP8-T P4-EP32 synchronous setup [13]. Note that ASAP supports other combinations of parallelism degrees, especially when D × T ̸= E thanks to the disaggregated architecture. The optimal values of D, T , and E can be derived from design space exploration [22, 55], which is orthogonal to our work. Due to the HBM capacity constraints (which must host the KV cache on attention devices), we set the maximum sequence length per device to 8k. With T P = 4, this allows individual request batches up to 32k tokens (i.e., S = 32k); requests longer than 32k are routed to dedicated prefill instances utilizing Sequence Parallelism (SP). While our implementation and evaluation focus on these limits, ASAP’s architecture is orthogonal to memory capacity and scales with available hardware resources.

Node-1

Node-2 Node48

CPU

UB-1 Switch UB-1 Switch UB-1 Switch

CPU DPU CPU DPU NPU 1 DPU NPU 1 NPU 1

UB-1 UB-7 Switch Switch UB-1 Scale Up UB Plane UB-7 Switch L1 level switch Switch UB-1 UB-7 Switch Switch

NPU 2 NPU 2 NPU 2

NPU 3 NPU 3 NPU 3

NPU 16 NPU 16 NPU 8

Figure 11: Hardware architecture of the CloudMatrix384 supernode. The supernode comprises 48 nodes, each integrating 8 NPUs. All 384 NPUs are interconnected via a two-level hierarchical UB plane, providing 400 GB/s bandwidth between any NPU pair.

3.5

Putting It All Together

Upon arrival, user requests are aggregated via length-aware batching to ensure optimal MoE token density. Once a pair of batches is formed, they are co-scheduled to an idle attention DP group using dual-batch interleaving. The attention devices alternate their layer-wise processing, achieving seamless communication-computation overlapping by immediately computing the next request batch while invoking async-dispatch-send of the previous batch in separate communication streams. Concurrently, MoE devices process these decoupled workloads sequentially and out-of-order via the MoE Super Kernel, completely eliminating host-side stalls through bubble-free kernel dispatching. Finally, expert results are returned via async-combine-send, allowing the originating attention devices to eagerly advance to the next layer. Ultimately, this orchestrated pipeline fully dismantles global DP synchronization barriers, unlocking highthroughput, asynchronous MoE inference.

4

ASAP Implementation

4.1

Hardware Platform

Serving Framework and Deployment

4.3

ASAP Components

Asynchronous Communication Primitives: The four asynchronous communication primitives ( async-dispatch-send, async-dispatch-recv, async-combine-send, and async-combine-recv) are implemented in AscendC [1] with approximately 4,000 lines of code (LoC) in total. The shared memory buffer on each device is statically allocated using aclrtMalloc [4]. Reads and writes from one NPU to the remote memory buffer of another are called through DataCopy or DataCopyPad interfaces [3] whose latency is at the granularity of microseconds. MoE Super Kernel: We extended the baseline MoE GMM kernel (about 3,000 LoC) to support layer-oblivious execution. By introducing a layer-ID input tensor and a pre-calculated weight address list (roughly 200 LoC), the MoE Super Kernel can resolve memory offsets dynamically at runtime. Triple-Stream Design: To mitigate resource contention between concurrent communication and computation, we partition hardware cores for our triple-stream design. Specifically, we allocate all 24 cube cores and 24 vector cores to the main compute stream. Each of the two communication streams is assigned 12 dedicated vector cores, which is sufficient to drive the non-compute-intensive data transfer logic. While this static partitioning isolates execution units, residual contention persists at the shared memory hierarchy, particularly within the L2 cache and HBM bandwidth [2]. Our empirical profiling reveals that on attention devices, even with core partitioning, concurrent communication-related memory traffic interferes with compute-intensive kernels, leading to a measurable elongation of attention latency. Consequently,

ASAP is implemented and deployed on the Huawei CloudMatrix384 supernode [22, 56] (Figure 11), which interconnects 384 Ascend 910 NPUs. Each NPU die has 24 cube cores, 48 vector cores and 64 GB HBM. The NPUs are linked via a Unified-Bus (UB) [28] mesh, providing a shared memory abstraction with a uniform bi-directional 400 GB/s bandwidth between any device pairs in the supernode. This architecture supports remote direct memory access with microsecond-level latency. 8

we selectively deploy the triple-stream design only on MoE devices. Host-Side Logic: We modified approximately 1,000 LoC in the request scheduler and model executor to implement length-aware batching, dual-batch interleaving, and bubblefree kernel dispatching.

4.4

trace dataset from Huawei Cloud’s, shown in Figure 5 and discussed in Section 2.3. Requests longer that 32k are excluded, as discussed in Section 4.2. We set the SLO for TTFT at 5 seconds. Each run lasts for five minutes. Metrics. All the systems decouple prefill and decode phases onto different devices, and we evaluate only the performance of the prefill instance. We evaluate: (1) Mean TTFT across varying request rates (RPS), and (2) SLO-compliant Throughput, defined as the maximum RPS sustained without violating the 5s SLO. Baselines. We compare ASAP against two synchronous state-of- the-art inference systems: • Default: A standard scheduling and execution logic in the in-house inference framework [22], similar to vLLM [25]. The resource scheduler aggregates incoming requests into batches of similar total token counts, to balance loads between DP groups.

Discussion

Portability Beyond Supernodes. ASAP does not strictly require supernode architectures. While UB-based load/store primitives provide ultra-low intra-node latency, the asynchronous paradigm is platform-agnostic. The logic remains robust over RDMA-based interconnects; even with RDMA latencies of hundreds of microseconds, the performance gains from dismantling global synchronization barriers (which can span tens of milliseconds) significantly outweigh the communication overhead. Applicability to Decode Phase. While ASAP can be applied to the decode phase, the benefits are likely to be less pronounced for two reasons: (1) Complexity Scaling: Decode attention is O(s) rather than O(s2 ), resulting in lower variance in attention latency and more manageable DP imbalance. (2) Arithmetic Intensity: The decode phase generates only one token per request. Accumulating enough tokens to saturate MoE efficiency (e.g., 2k tokens) would introduce queuing delays that may negatively affect decode SLO. Therefore, ASAP primarily focuses on the prefill phase.

5 5.1

• ChunkedPrefill [5]: Long sequences are split into 8k chunks, which get processed sequentially. With T P = 4, this means that the maximum sequence length on each attention device is reduced from 8k to 2k. This mitigates the sequence-length variance and DP imbalance problem.

5.2

End-to-End Performance

Figure 12 shows the mean TTFT of each inference system with increasing RPS, and Figure 13 reports the normalized throughput under the 5s SLO. We observe that: • For each inference system, mean TTFT is flat at the beginning, and increases almost exponentially after an inflection point. ASAP significantly delays this inflection point. It consistently outperforms all the synchronous baselines across the entire range of request rates. For Default, load balancing using aggregate sequence length is proven to be ineffective, as discussed in Section 2.2.1. For ChunkedPrefill, the reduced variance in sequence length indeed mitigates DP imbalance, thus outperforming Default. However, the system is still synchronous, and does not completely eliminate the DP imbalance issue.

Evaluation Methodology

MoE Model. We evaluate ASAP using Deepseek-V3.2 [30], a state-of-the-art open-sourced MoE model [30], released in December 2025. DeepSeek-V3.2 features a massive parameter scale of 671B, with 256 experts and 1 shared expert. It activates only 37B parameters (8 experts and 1 shared expert) per token. Deepseek-V3.2 incorporates DSA, a novel sparse attention mechanism. Model Deployment. Following the baseline deployment in Deepseek’s technical report [13], one prefill instance of the synchronous baseline utilizes 32 NPU dies with DP = 8, T P = 4 and EP = 32. To ensure a fair comparison, ASAP employs a disaggregated DP = 4, T P = 4 and EP = 16 configuration, maintaining the same device count as synchronous systems. All experiments are conducted on CloudMatrix384 supernodes (detailed in Section 4.1). Workloads. Similar to prior work [25, 47], we evaluate various request rate and lengths. We simulate realistic serving scenarios with request inter-arrival times following a Poisson distribution [43]. Request rate is quantified by request-persecond (RPS). Request lengths are sampled from a production

• At a low load (RPS = 1) when requests rarely get queued in any of the inference systems, ASAP achieves mean TTFT of 350ms, representing a 34.3% and 9.8% reduction compared to Default (533ms) and ChunkedPrefill (388ms), respectively. As load increases, the benefits of asynchronous execution become more pronounced. At RPS = 4, ASAP reduces TTFT by 54.9% and 41.8% over Default and ChunkedPrefill, respectively. At RPS = 8, Default is unable to sustain the request rate, and ASAP reduces mean TTFT by 68.3% over ChunkedPrefill. • ASAP sustains an RPS of 20, whereas Default and ChunkedPrefill reach their limits at 6.8 and 10.5 RPS, respectively. This translates to 194% and 90% throughput improvement, demonstrating ASAP’s superior ability to ab9

    





   

[[ [[

 

 



 

536 536

 

 

 

   'HIDXOW &KXQNHG3UHILOO $HROXV 'HIDXOW &KXQNHG3UHILOO $HROXV

%DVHOLQH %DVHOLQH

Figure 13: Normalized throughput under SLO.

  



. . . . . .  .  .  .  .  .  .  .  .  .  .  .  .



6HTXHQFH/HQJWK

Figure 15: Latency decomposition at QPS=4. The left and right bars at each sequence length show mean TTFT under Default and ASAP, respectively. For Default, TTFT is decomposed into kernel time, request queuing delay, and synchronization waiting delay. For ASAP, TTFT is decomposed into kernel and non-kernel delay.

5.4



6\QF3HHUWR3HHU $V\QF'LVSDWFK

    

       

1XPEHURIWRNHQV

Figure 14: Communication latency with increasing token count.

Communication Efficiency

The foundation of ASAP is its non-blocking, asymmetric and asynchronous communication. If not designing a new set of primitives, one has to use existing synchronous peer-to-peer (P2P) communication to realize asymmetric communication between attention and MoE devices. For instance, after attention computation, if the attention device needs to dispatch its tokens to n MoE devices. Then, the P2P communication will be invoked n times, Each time, the attention device (i.e., sender) has to handshake with each MoE device (i.e., receiver), and the data transfer will not be done until the receiver has received the data transfer. If the receiver is doing its own computation task, the sender will be blocked until the receiver finished the ongoing task and acknowledged the data transfer. Therefore, in p2p communication, the communication latency include (1) the handshake time which is a constant, (2) the data transfer time which is linear to data amount, (3) extra receiving delay which fluctuates depending on the status of each receiver, and (4) the total communication latency is n times of the communication latency between the sender and each receiver. On the contrary, ASAP’s async-dispatch-send is non-blocking, so it doesn’t need to handshake with the receiver. It transfers data and immediately returns back future tasks. There is no extra receiving delay either because each receiver performs

sorb heterogeneous workloads that traditionally trigger severe synchronization stalls.

5.3



kernel overheads dominate for short sequences. For instance, for requests under 512 tokens, synchronization delay and request queuing delay account for 55% and 30% of the overall TTFT. This confirms the "straggler effect" where short requests are penalized the most by global barriers. In contrast, ASAP reduces non-kernel delay up to 80% for the same sequence range. By allowing request batches to "flow" asynchronously, ASAP effectively converts idle synchronization time into productive computation intervals. As sequence length increases, the relative impact of synchronization diminishes, yet ASAP maintains a consistent edge by eliminating residual execution bubbles.

'HIDXOW77)7 6\QFKURQL]DWLRQGHOD\ 5HTXHVWTXHXLQJGHOD\ $HROXV77)7 $HROXV1RQNHUQHOGHOD\ .HUQHOWLPH



/DWHQF\PV

 

 

Figure 12: Mean TTFT with increasing request-persecond (RPS). 

 

&RPPXQLFDWLRQ/DWHQF\PV

 

[[

 

1RUP7KURXJKSXW

 

 

1RUP7KURXJKSXW

'HIDXOW 'HIDXOW &KXQNHG3UHILOO &KXQNHG3UHILOO $HROXV $HROXV

0HDQ77)7PV

0HDQ77)7PV

 

Latency Decomposition

To understand the source of the performance gains, Figure 15 decomposes the TTFT of Default and ASAP at RPS=4. for Default, TTFT is decomposed into kernel time (the actual computation latency), request queuing delay (waiting time after request arrival and before request processing), and synchronization waiting delay (due to DP imbalance). For ASAP, due to the small value, we do not further decompose nonkernel latency which mainly comes from queuing on attention devices due to dual-batch interleaving. Each bar represents a range of requests. For example, the first and second group of bars in Figure 15 represent mean latency of requests under 512 tokens, and requests between 512 and 1024 tokens, respectively. Latency increases with request sequence length due to increasing kernel time. Kernel time is the same under the two systems. However, in synchronous systems (Default), non10

 







 

:LWKRXW2YHUODSSLQJ :LWK2YHUODSSLQJ

  

          

         

Figure 16: Comparison of mean TTFT with/without dual-batch interleaving.

Figure 17: Comparison of mean TTFT with/without communication-computation overlapping.

asyn-dispatch-recv whenever it’s ready, without blocking the sender. Also, the sender can perform data transfer to n devices at the same time in parallel. Figure 14 compares the communication latency of the synchronous P2P and the asynchronous async-dispatch, with an increasing number of tokens. Both curves show a linear trend with token count. Sending t tokens implies sending data size of t ∗ H ∗ Etotal /E. For DeepSeek V3.2, this is 63MB per 1k tokens. We can see that latency of synchronous p2p is 4× of async-dispatch under 1k tokens, and 5.8× under 8k tokens. The difference increases with more data. Note that the latency of our asynchronous communication primitive by leveraging supernode is significantly lower than that without support of the high-bandwidth peer-to-peer network in supernodes. Prior work [33] develops a set of asynchronous p2p primitives on GPUs leveraging the IBGDA Connection. The reported communication latency to send 512 tokens is over 0.5ms. However, the latency of our primitive is less than 0.1ms under the same token count.



:20R(6XSHU.HUQHO :0R(6XSHU.HUQHO 



 



          

536

Figure 18: Comparison of mean TTFT with/without the bubble-free kernel dispatching brought by MoE Super Kernel.

these execution gaps, enabling the system to sustain high request rate under latency constraints. Consequently, it improves the SLO-compliant throughput by 14.3% (from 17.5 to 20 RPS). This result underscores that dual-batch interleaving is essential for maximizing attention device duty cycles and maximizing the overall system throughput. 5.5.2

Effectiveness of Communication-Computation Overlapping

The efficacy of our triple-stream design in masking communication overhead is analyzed in Figure 17. In low-load scenarios, overlapping yields negligible TTFT reduction as the system operates with ample resource headroom. As the request rate intensifies, the overlapping becomes critical bottleneck, which increases the SLO-compliant throughput from 17.8 to 20, an 12.4% improvement. 5.5.3

Effectiveness of Bubble-Free Kernel Dispatching

Figure 18 quantifies the gains from the MoE Super Kernel that enables bubble-free kernel dispatching. At low RPS, the bubble-free kernel dispatching reduces TTFT by roughly 13ms. This is because the traditional kernel dispatching latency from the host CPU is 220 microseconds each layer measured on the CloudMatrix384 supernode. Since there are 61 layers in DeepSeek V3.2, this translates to 220 ∗ 61 ≈ 13.4ms saving in TTFT. The latency saving also brings higher throughput. Compared to the systems without the MoE Super Kernel, our bubble-free kernel dispatching increases the SLO-compilant throughput by 6%.

Ablation Studies

We perform a series of ablation studies to show the effectiveness of each mechanism in ASAP. 5.5.1





536

536

5.5

0HDQ/DWHQF\PV



0HDQ/DWHQF\PV

0HDQ/DWHQF\PV

6LQJOH%DWFK 'XDO%DWFK



Effectiveness of Dual-Batch Interleaving

Figure 16 evaluates the impact of dual-batch interleaving on mean TTFT across varying request rates. At low RPS, where request arrivals are sparse, the dual-batch mechanism introduces a marginal TTFT increase of approximately 6.5%. This overhead stems from the difference in attention and MoE latency, as illustrated in Figure 8: if the attention latency of one batch (B1 ) exceeds the MoE latency of the co-scheduled batch (B2 ), B2 must wait for the attention devices to become available, introducing minor stalls. However, as the RPS increases, the system without interleaving suffers from frequent attention-stage idling during MoE offloading, leading to rapid queue accumulation. By contrast, ASAP’s dual-batch interleaving effectively reclaims

6

Related Work

Long-context LLM Inference. Extensive research has focused on accelerating attention computation for long sequences. FlashAttention [12] and FlashDecoding [20] optimize memory I/O through tiling and recomputation, though they do not alter the quadratic complexity O(s2 ) to request sequence length s in the prefill phase. Other approaches, such 11

as sparse attention [9] and linear attention [24], attempt to reduce this complexity. Our characterization (Section 2) and evaluation (Section 5) have already applied sparse attention, and we demonstrate the necessity and benefits of ASAP’s asynchronous execution. Sequence Parallelism (SP) [6, 26, 32] partitions long sequences across devices to enable concurrent processing. LoongServe [47] and Infinite-LLM [29] further refine this by dynamically adjusting SP degrees or dedicating specialized clusters for long-context requests. Additionally, ChunkedPrefill [5] and BucketServe [52] mitigate request variance by decomposing long prompts or grouping similarlength requests. While these techniques are orthogonal and complementary to ASAP, they remain within the synchronous execution paradigm; they merely alleviate rather than eliminate the idle bubbles caused by intrinsic workload variance.

ASAP, the first asynchronous inference system to speed up the prefill phase of MoE models. Experiments show that ASAP improves throughput under SLO by 90% over the state-of-the-art inference systems.

References [1] Ascend C Custom Operator Development and Usage. https://www.mindspore.cn/tutorials/experts /en/r2.3.1/operation/op_custom_ascendc.htm l, 2025. [2] Compute Architecture for Neural Networks (CANN). https://www.hiascend.com/cann, 2025. [3] Data Movement in Ascend C. https://www.hiasce nd.com/document/detail/zh/canncommercial/8 3RC1/API/ascendcopapi/atlasascendc_api_07_ 0102.html, 2025.

Disaggregated LLM Serving Systems. The computational disparity between different inference phases has led to the rise of disaggregated architectures. Prefill-decode disaggregation, exemplified by DistServe [53], Splitwise [37], and Aegaeon [48], separates the compute-bound prefill and memory-bound decode stages to minimize interference. Mooncake [38] extends this by pooling CPU and DRAM resources for a distributed KV cache. However, these systems primarily focus on inter-phase isolation and do not address the intra-phase latency fluctuations driven by dynamic workloads. Recently, Attention-MoE disaggregation has been proposed to enable independent scaling of attention and MoE modules, including frameworks such as MegaScale-Infer [55] and Step3 [44]. However, they still rely on synchronous barriers across attention DP groups and do not tackle the DP imbalance dilemma. Furthermore, these frameworks are predominantly tailored for the decoding stage. In contrast, ASAP specifically targets the prefill stage, introducing an asynchronous execution pipeline to tackle resource idling due to DP imbalance. Asynchronous Inference Systems. Expert-as-a-Service (EaaS) [33] designs a CPU-free asynchronous peer-to-peer communication library for MoE serving. However, it mainly targets scaling at the granularity of each expert, and does not tackle DP imbalance. The communication library includes only peer-to-peer primitives, and does not provide asymmetric 1-to-N/N-to-1 communication primitives. On the contrary, ASAP adopts a set of efficient, asynchronous, and asymmetric communication primitives. Also, the communication latency of our primitives is much smaller than that reported in EaaS by taking advantage of supernodes.

7

[4] Memory Management of Ascend NPU. https://www. hiascend.com/document/detail/zh/canncommer cial/83RC1/API/appdevgapi/aclpythondevg_01 _0110.html, 2025. [5] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369, 2023. [6] William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. Striped attention: Faster ring attention for causal transformers. arXiv preprint arXiv:2311.09431, 2023. [7] Jingwei Cai, Dehao Kong, Hantao Huang, Zishan Jiang, Zixuan Ma, Qingyu Guo, Zhenxing Zhang, Guiming Shi, Mingyu Gao, and Kaisheng Ma. Characterizing cloud-native llm inference at bytedance and exposing optimization challenges and opportunities for future ai accelerators. In 2026 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 1–19. IEEE, 2026. [8] Shiheng Cao, Junmin Wu, Junshi Chen, Hong An, and Zhibin Yu. Amali: An analytical model for accurately modeling llm inference on modern gpus. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, ISCA ’25, page 1495–1508, New York, NY, USA, 2025. Association for Computing Machinery.

Conclusion

[9] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.

Existing synchronous inference systems introduce costly barriers under DP imbalance in online MoE serving. We present 12

[10] Yujeong Choi, Yunseong Kim, and Minsoo Rhu. Lazy batching: An sla-aware batching system for cloud machine learning inference. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 493–506, 2021.

[20] Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++: Faster large language model inference on gpus. arXiv preprint arXiv:2311.01282, 2024. [21] Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, and Ninghui Sun. Memserve: Context caching for disaggregated llm serving with elastic memory pool. arXiv preprint arXiv:2406.17565, 2024.

[11] Peizhuang Cong, Aomufei Yuan, Shimao Chen, Yuxuan Tian, Bowen Ye, and Tong Yang. Prediction is all moe needs: Expert load distribution goes from fluctuating to stabilizing, 2024. [12] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memoryefficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135, 2022.

[22] Junhao Hu, Jiang Xu, Zhixia Liu, Yulong He, Yuetao Chen, Hao Xu, Jiang Liu, Jie Meng, Baoquan Zhang, Shining Wan, Gengyuan Dan, Zhiyu Dong, Zhihao Ren, Changhong Liu, Tao Xie, Dayun Lin, Qin Zhang, Yue Yu, Hao Feng, Xusheng Chen, and Yizhou Shan. Deepserve: serverless large language model serving at scale. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’25, USA, 2025. USENIX Association.

[13] DeepSeek-AI, Aixin Liu, Bei Feng, and etc Bing Xue. Deepseek-v3 technical report, 2025. [14] Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens, 2023.

[23] Yunpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xiaoxing Ma, Lijuan Yang, Hao Chen, Shupeng Li, and Penghao Zhao. Advancing transformer architecture in long-context large language models: A comprehensive survey, 2024.

[15] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts. arXiv preprint arXiv:2112.06905, 2022.

[24] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR, 2020. [25] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611–626, New York, NY, USA, 2023. Association for Computing Machinery.

[16] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), January 2022. [17] Ze-Feng Gao, Peiyu Liu, Wayne Xin Zhao, Zhong-Yi Lu, and Ji-Rong Wen. Parameter-efficient mixture-ofexperts architecture for pre-trained language models. arXiv preprint arXiv:2203.01104, 2022.

[26] Dacheng Li, Rulin Shao, Anze Xie, Eric Xing, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. Lightseq: : Sequence level parallelism for distributed training of long context transformers. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023), 2023.

[18] Raja Gond, Nipun Kwatra, and Ramachandran Ramjee. Tokenweave: Efficient compute-communication overlap for distributed llm inference. arXiv preprint arXiv:2505.11329, 2025. [19] Ruihao Gong, Shihao Bai, Siyu Wu, Yunqian Fan, Zaijun Wang, Xiuhong Li, Hailong Yang, and Xianglong Liu. Past-future scheduler for llm serving under sla guarantees. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’25, page 798–813, New York, NY, USA, 2025. Association for Computing Machinery.

[27] Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120, 2022. [28] Heng Liao, Bingyang Liu, Xianping Chen, Zhigang Guo, Chuanning Cheng, Jianbing Wang, Xiangyu Chen, Peng 13

Dong, Rui Meng, Wenjie Liu, Zhe Zhou, Ziyang Zhang, Yuhang Gai, Cunle Qian, Yi Xiong, Zhongwu Cheng, Jing Xia, Yuli Ma, Xi Chen, Wenhua Du, Shizhong Xiao, Chungang Li, Yong Qin, Liudong Xiong, Zhou Yu, Lv Chen, Lei Chen, Buyun Wang, Pei Wu, Junen Gao, Xiaochu Li, Jian He, Shizhuan Yan, and Bill McColl. Ub-mesh: A hierarchically localized nd-fullmesh data center network architecture. IEEE Micro, 45(5):20–29, 2025.

[38] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pages 155–170, Santa Clara, CA, February 2025. USENIX Association. [39] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation AI scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 17–23 Jul 2022.

[29] Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, and Wei Lin. Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache. arXiv preprint arXiv:2401.02669, 2024. [30] Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, and Chen Dong. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025.

[40] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2020.

[31] Guowei Liu, Hongming Li, Yaning Guo, Yongxi Lyu, Mo Zhou, Yi Liu, Zhaogeng Li, and Yanpeng Wang. Revealing the challenges of attention-ffn disaggregation for modern moe models and hardware systems. arXiv preprint arXiv:2602.09721, 2026.

[41] Yehui Tang, Xiaosong Li, Fangcheng Liu, Wei Guo, Hang Zhou, Yaoyuan Wang, Kai Han, Xianzhi Yu, Jinpeng Li, Hui Zang, Fei Mi, Xiaojun Meng, Zhicheng Liu, Hanting Chen, Binfan Zheng, Can Chen, Youliang Yan, Ruiming Tang, Peifeng Qin, Xinghao Chen, Dacheng Tao, and Yunhe Wang. Pangu pro moe: Mixture of grouped experts for efficient sparsity. arXiv preprint arXiv:505.21411, 2025.

[32] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:310.01889, 2023. [33] Ziming Liu, Boyu Tian, Guoteng Wang, Zhen Jiang, and etc Peng Sun. Expert-as-a-service: Towards efficient, scalable, and robust large-scale moe serving. arXiv preprint arXiv:2509.17863, 2025.

[42] Kimi Team, Yifan Bai, Yiping Bao, and etc Guanduo Chen. Kimi k2: Open agentic intelligence, 2025.

[34] Haiyue Ma, Zhixu Du, and Yiran Chen. Moe-gps: Guidlines for prediction strategy for dynamic expert duplication in moe load balancing, 2025.

[43] W.J. Thompson. Poisson distributions. Computing in Science and Engineering, 3(3):78–82, 2001.

[35] Siyuan Mu and Sen Lin. A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07137, 2025.

[44] Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, and Siyu Chen. Step-3 is large yet affordable: Model-system co-design for cost-effective decoding. arXiv preprint arXiv:2507.19427, 2025.

[36] Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, Tri Dao, Yida Wang, and Ravi Netravali. Marconi: Prefix caching for the era of hybrid llms. In M. Zaharia, G. Joshi, and Y. Lin, editors, Proceedings of Machine Learning and Systems, volume 7. MLSys, 2025.

[45] Yujie Wang, Shiju Wang, Shenhan Zhu, Fangcheng Fu, Xinyi Liu, Xuefeng Xiao, Huixia Li, Jiashi Li, Faming Wu, and Bin Cui. Flexsp: Accelerating large language model training via flexible sequence parallelism. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’25, page 421–436, New York, NY, USA, 2025. Association for Computing Machinery.

[37] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture, ISCA ’24, page 118–132. IEEE Press, 2025. 14

[46] Zheng Wang, Anna Cai, Xinfeng Xie, Zaifeng Pan, Yue Guan, Weiwei Chu, Jie Wang, Shikai Li, Jianyu Huang, and Chris Cai. {WLB-LLM}:{Workload-Balanced} 4d parallelism for large language model training. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), pages 785–801, 2025.

[55] Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, and Yang Cheng. Megascale-infer: Efficient mixture-of-experts model serving with disaggregated expert parallelism. In Proceedings of the ACM SIGCOMM 2025 Conference, pages 592–608, 2025.

[47] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, SOSP ’24, page 640–654, New York, NY, USA, 2024. Association for Computing Machinery.

[56] Pengfei Zuo, Huimin Lin, Junbo Deng, Nan Zou, Xingkun Yang, Yingyu Diao, Weifeng Gao, Ke Xu, Zhangyu Chen, and Shirui Lu. Serving large language models on huawei cloudmatrix384. arXiv preprint arXiv:2506.12708, 2025.

[48] Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. Aegaeon: Effective gpu pooling for concurrent llm serving on the market. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, pages 1030–1045, 2025. [49] Yuxing Xiang, Xue Li, Kun Qian, Wenyuan Yu, Ennan Zhai, and Xin Jin. Servegen: Workload characterization and generation of large language model serving in production. arXiv preprint arXiv:2505.09999, 2025. [50] An Yang, Anfeng Li, and etc Baosong Yang. Qwen3 technical report, 2025. [51] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: efficient execution of structured language model programs. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA, 2024. Curran Associates Inc. [52] Wanyi Zheng, Minxian Xu, Shengye Song, and Kejiang Ye. Bucketserve: Bucket-based dynamic batching for smart and efficient llm inference serving. arXiv preprint arXiv:2507.17120, 2025. [53] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210, 2024. [54] Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Xiao Chuanfu, Dahua Lin, and Chao Yang. Sampleattention: Nearlossless acceleration of long context llm inference with adaptive structured sparse attention. In M. Zaharia, G. Joshi, and Y. Lin, editors, Proceedings of Machine Learning and Systems, volume 7. MLSys, 2025. 15

Related documents

• Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism#158558
• US04: Requirements for Asynchronous Collective Subroutines#634673
• The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation#1069931
• US04: Specifications for Asynchronous Collective Subroutines#634672
• A Multi-Engine Dataflow for MoE Decoding on Scratchpad-Based Tensor Accelerators#1050941