arXiv:2609.07536v1 [cs.DC] 7 Sep 2026
Analytical Resource Management for Fine-grained MoE Computation-Communication Overlap Hongyu Liu
Minyu Cui
Miquel Pericàs
Chalmers University of Technology and University of Gothenburg Gothenburg, Sweden [email protected]
Linnaeus University Växjö, Sweden [email protected]
Chalmers University of Technology and University of Gothenburg Gothenburg, Sweden [email protected]
Abstract—Fine-grained computation–communication overlap in distributed Mixture-of-Experts (MoE) inference allows communication to begin as partial compute results become ready. However, cooperative thread arrays (CTAs) performing computation and communication contend for finite residency capacity on the streaming multiprocessors (SMs). Because a resident CTA generally retains its allocated SM resources until completion, CTAs that cannot be co-resident must wait for resources to become available, resulting in wave-like execution. A fixed resource partition cannot adapt to changes in input size, routed expert load, and kernel configuration, potentially causing a communication backlog or reducing expert compute parallelism. We present a wave-quantized analytical model and launchtime resource manager for dependency-coupled overlap pipelines. Using the current routed-tile counts, kernel occupancy, GPU residency constraints, and split-level readiness dependencies, it selects the communication-CTA count and resource partition before each kernel launch without candidate execution, per-workload profiling, or kernel recompilation. We integrate the method into the public COMET A100 implementation in the FLUX codebase. We evaluate three MoE models on four NVIDIA A100 GPUs under several parallelism strategies at the GEMM2+GatherRS operator, complete post-router MoE layer, and complete-model prefill levels. Across 15 real-p90 workloads, the analytical selector achieves 3.22% mean regret relative to the measured oracle with a mean solver overhead of 0.157 µs. Over COMET, our method achieves geometric-mean speedups of 2.528× at the GEMM2+GatherRS operator, with a maximum of 4.218×; 1.771× at the complete post-router MoE layer, with a maximum of 2.584×; and 1.185× for complete-model prefill, with a maximum of 1.439×. For the representative TP=2/EP=2 configuration, complete-model prefill achieves geometric-mean speedups over COMET of 1.336×, 1.211×, and 1.232× for Granite, Qwen, and DeepSeek-V2-Lite, respectively, with a best speedup of 1.439×. At every feasible TP=2/EP=2 sequence length of at least 4,096, our implementation outperforms COMET and the other state-of-the-art MoE systems, Megatron core-TE and FastMoE TP+NCCL. Index Terms—Mixture-of-Experts, computation– communication overlap, GPU resource management, analytical performance modeling
I. I NTRODUCTION Recent advances in large language models (LLMs) have been driven in part by scaling model parameters and training data. Scaling dense models, however, increases both memory consumption and the computation performed for every token, causing training and inference costs to grow rapidly. Mixture-
of-Experts (MoE) models offer a more scalable alternative: they increase parameter capacity by incorporating many experts while sparsely activating only a small subset for each token, thereby avoiding a proportional increase in computational overhead. As models and expert sets continue to grow, a single GPU can no longer store or efficiently execute all experts. Distributed MoE inference must place experts across multiple GPUs, send routed tokens to the devices that host their selected experts, and return the expert outputs to the devices that own the original tokens. Prior studies report that this crossdevice communication can account for as much as 47% of end-to-end model execution time in representative models and frameworks [1]. Communication thus limits not only an individual MoE layer but also the scaling benefit expected from adding GPUs. Overlapping this communication with expert GEMMs without degrading their efficiency is therefore critical to scaling distributed MoE inference. A representative solution [1] is dependency-aware, threadblock-specialized fine-grained overlap. This technique exposes result readiness at fine granularity, allowing a communication CTA to process a local result as soon as it becomes available rather than waiting for the entire GEMM to finish. Implementations may assign compute and communication work to separate CTAs, either in concurrently executing kernels or within a fused kernel. These role-specialized CTAs keep longlatency communication instructions out of compute CTAs, thereby avoiding direct instruction-level interference with computation. However, both CTA classes compete for limited SM residency and shared GPU resources. This introduces a resource-partitioning problem that ultimately determines the effectiveness of overlap. With insufficient communication resources, ready results cannot be transferred and reduced promptly, leaving a communication tail on the critical path. With excessive communication resources, expert-GEMM parallelism decreases and computation becomes the bottleneck. Therefore, effective overlap requires resource management to balance the two CTA classes under dependency constraints to minimize the overall pipeline latency, rather than maximizing either throughput in isolation. An imbalanced allocation can create backpressure or resource contention and may perform worse than a non-overlapped schedule.
No single communication-CTA allocation is optimal for every workload. Token count, expert matrix dimensions, the number of routed expert tiles, and the pipeline split granularity change computation and communication work at different rates. Moreover, GPUs execute integer numbers of CTAs and tile waves, so performance changes discontinuously at wave boundaries instead of scaling smoothly with a resource ratio. A fixed partition cannot adapt to this shape-dependent behavior. Profiling candidate configurations for every shape can identify an efficient partition, but it incurs deployment and tuning costs and does not provide a principled solution for an unseen workload. This leads to our central question: given the workload shape, dependency structure, and kernel resource requirements before execution, can we directly derive the resource partition between computation and communication CTAs without pershape profiling? The key difficulty is dependency–residency coupling: communication CTAs can advance only after their inputs become ready, yet their residency reduces the SM capacity available to compute CTAs. Because both sides progress in discrete CTA waves, the optimal partition minimizes overall pipeline latency rather than either side’s isolated throughput.
Fig. 1. Post-router MoE path and its evaluated COMET/FLUX implementation. The highlighted GEMM2+GatherRS operator is the optimization target.
Based on this observation, we formulate resource management for thread-block-specialized overlap as dependencyconstrained resource partitioning and develop a wavequantized analytical makespan model. Rather than adding new precompiled kernel variants, we insert an analytical launch-time resource manager between conventional kernel dispatch and the actual launch. This design decouples kernel implementation from workload-specific resource allocation: the same binaries adapt at runtime without recompilation, candidate-kernel execution, warm-up timing, or consulting a per-workload winner table. The core analytical decision takes approximately 0.16 µs and therefore incurs negligible overhead. The contributions of this paper are threefold:
II. BACKGROUND AND P ROBLEM F ORMULATION
Problem characterization and empirical finding. We characterize CTA allocation in dependency-coupled, thread-block-specialized overlap as a non-preemptive GPU-residency partitioning problem, and identify the dependency-residency coupling through which communication progress and compute capacity jointly determine pipeline performance. • Analytical model and launch-time resource management. We develop a dependency-constrained, wavequantized makespan model and implement a lightweight runtime resource manager between precompiled kernel dispatch and launch. Using the current workload, actual routed tile counts, kernel residency, and dependencies, it directly derives communication CTA count C, determines the associated communication reservation R(C), and computes the remaining compute capacity P (C). • System implementation and cross-model, multi-level evaluation. We integrate this analytical launch-time resource manager into COMET’s public A100 implementation in the FLUX codebase [1], [2] and evaluate its accuracy and performance benefits at the three mea-
•
Router
Logical operations Dispatch exchange
GEMM1
Activation
GEMM2
Return / combine
implemented as AllGather + GEMM1 compute
Activation
GEMM2 + GatherRS
communication
optimized
Evaluated COMET/FLUX realization complete post-router MoE layer
surement levels: the GEMM2+GatherRS operator, the complete post-router MoE layer, and complete-model prefill. The reported speedups arise solely from improved CTA resource allocation: our method does not reduce FLOPs, routed tokens, or communication volume, and it preserves numerical correctness. At the first two levels, we evaluate Granite-3.1-1B-A400M, Qwen1.5-MoEA2.7B, and DeepSeek-V2-Lite under uniform, real-p50, and real-p90 routing workloads. At the complete-model prefill level, we compare our system with upstream COMET, Megatron core-TE, and FastMoE TP+NCCL using model-matched inputs and identical execution and measurement boundaries.
This section provides the execution context needed to understand the resource management problem studied in this work. We first describe the dataflow of a distributed MoE layer, then explain how fine-grained computation–communication overlap forms a dependency pipeline executed by compute and communication CTAs, and finally formulate resource partitioning within this pipeline as the problem addressed in this paper. A. Distributed MoE Execution Given a set of input hidden states, the router selects the top-k experts for each token and produces token–expert assignments. The post-router MoE layer studied in this work follows the dataflow of the existing fused implementation and consists of three consecutive operations: 1) AllGather+GEMM1. AllGather collects the participating input segments. As the required data arrives, grouped GEMM1 performs the gate and up projections. 2) Activation. A gated activation combines the two projections to form the intermediate representation for each expert. 3) GEMM2+GatherRS. Grouped GEMM2 projects the expert intermediate representations back to the hidden dimension. GatherRS aggregates the top-k expert contributions, performs ReduceScatter, and writes the token outputs for each rank.
Grouped GEMM2 tile-task space experts
split s₀
tile task (compute CTA)
split s₁
…
⋮ e E−1
split s L−1
⋯
e₀ ⋮
⋮
⋱ ⋯
⋮
Grouped GEMM2 kernel · resident compute-CTA pool global tile sequence
claim next tile task
GEMM2 compute
write output tile
publish completed seg.
GatherRS kernel · resident communication-CTA pool ready-segment queue
gather / reorder
claim tile tasks
top-k reduction data
ReduceScatter + writeback
token outputs by rank
ready signal
Fig. 2. Split-level execution of the GEMM2+GatherRS operator. Compute CTAs publish a segment after all tiles in the corresponding split complete; communication CTAs consume ready segments.
Fig. 1 relates the logical post-router operator sequence to the evaluated COMET/FLUX realization. Our resource decision acts on the final GEMM2+GatherRS operator. To avoid ambiguity, we use GEMM2+GatherRS operator for this joint computation–communication operator, complete post-router MoE layer for the path from AllGather+GEMM1 through the GatherRS output writeback, and complete-model prefill for a complete model prefill forward pass, including attention, routing, MoE blocks, GPU-resident KV-cache updates, and the last-token LM head. These three boundaries isolate the direct effect of the resource decision, its propagation through the MoE layer, and its model-level impact, respectively. B. Fine-Grained Overlap and GPU Residency A CUDA thread block, also called a cooperative thread array (CTA), comprises multiple warps and is scheduled on one SM. While resident, a CTA occupies threads, registers, and shared memory; once admitted in ordinary kernel execution, it normally retains these resources until completion. These resource requirements jointly determine how many CTAs can reside on an SM. A CTA is therefore neither a single warp nor an SM: one SM may host multiple CTAs when resources permit, whereas one CTA is not split across multiple SMs. In the fine-grained overlap path studied here, compute CTAs repeatedly claim and execute grouped-GEMM2 tile tasks, as shown in Fig. 2. A segment becomes ready only after every tile task in its corresponding split finishes; the compute side then publishes its data and readiness signal. Communication CTAs consume these ready segments to aggregate expert outputs, reduce them, and write back the corresponding result. Communication can therefore begin before the complete GEMM2 finishes, forming a dependency pipeline in which computation publishes completed segments and communication consumes them. A communication CTA can perform useful work only after its input becomes ready, and the same communication CTAs service multiple splits in sequence. Concurrent residency alone therefore does not guarantee balanced progress. Both CTA classes share finite GPU residency. Increasing the number of communication CTAs can process more ready
communication tiles in parallel and reduce communication waves, but it also occupies SM resources that would otherwise be available to compute CTAs. Compute parallelism consequently decreases and later split-release times may be delayed. Conversely, too few communication CTAs cause ready splits to queue and leave a communication tail on the completion path of the GEMM2+GatherRS operator. Both workloads execute in integer CTA waves: completion time changes materially only when a resource adjustment adds or removes a wave. Performance as a function of the resource partition therefore exhibits plateaus and steps rather than smooth linear scaling. Existing fine-grained overlap mechanisms already specify tile decomposition, split-level readiness, and the concurrent execution of the two CTA classes. Our work instead asks how, once this mechanism is fixed, a particular invocation should partition residency between the two classes so that exposed concurrency becomes effective overlap. C. Optimization Scope and Problem Formulation We study the public A100 implementation of COMET in the FLUX codebase [1], [2]. This path fixes the compute and communication kernels, their tile- and split-level readiness protocol, and their dataflow. The public A100 path uses a fixed communication-CTA count C = 3, which can be overridden manually but is not selected analytically for each invocation. Let W describe an invocation’s workload, including its matrix dimensions, split count, and routed expert-tile counts; let K describe the tile shape and resource footprint of the kernels selected by dispatch; let H describe the GPU SM count and residency constraints; and let Gdep denote the split-level dependency graph between compute results and communication tasks. The only decision variable is the communicationCTA count C for the current launch. Given C, the communication parallelism, its required residency reservation, and the remaining compute-CTA capacity are jointly determined. To isolate the effect of resource partitioning, comparisons across C hold W, K, and Gdep fixed. The same inputs and router assignments produce the same GEMM tiles, perform the same arithmetic work, and communicate the same data. The resource manager changes only the communication-CTA grid and the resulting compute-CTA reservation. The problem therefore does not reroute tokens or change a kernel algorithm; it changes how the same work and dependencies occupy GPU execution resources. Let T (C | W, K, H, Gdep ) denote the completion time from the launch of the first compute CTA to the writeback of the final communication result under a given partition. The legal set C contains integer CTA counts that reserve sufficient communication residency while retaining positive compute capacity. Before the actual launch, we seek C ∗ = arg min T (C | W, K, H, Gdep ) .
(1)
C∈C
This objective imposes three requirements: communication CTAs must obtain enough residency to consume ready splits,
compute CTAs must retain enough capacity to produce subsequent results, and the selector cannot learn the answer by executing and timing candidate kernels. The core problem is therefore to use the workload, kernel residency, and readiness dependencies available before launch to predict the two discrete wave sequences analytically and select the communicationCTA count that minimizes GEMM2+GatherRS completion time. The following section derives the progress-preserving reservation, the compute and communication wave counts, the split-readiness recurrence, and the corresponding launch-time selection algorithm. III. D EPENDENCY-C ONSTRAINED , WAVE -Q UANTIZED R ESOURCE M ANAGEMENT This section presents our dependency-constrained, wavequantized model and its runtime resource manager. A. Optimization Scope
a given communication-CTA parallelism, not an independent tuning parameter. Register usage limits the current GatherRS kernel to one resident communication CTA per A100 SM. Consequently, ocomm = 1,
R(C) = C.
(3)
Each concurrent communication CTA therefore requires a distinct SM residency position. Based on its thread, register, and dynamic shared-memory requirements, the selected 128 × 128 GEMM2 kernel admits at most two resident compute CTAs per SM, i.e., ocomp = 2. After reserving R(C) SM positions, the resident compute capacity is P (C) = ocomp G − R(C) . (4) For the current A100 kernel family, this reduces to P (C) = 2(G − C).
(5)
Adding one communication CTA therefore has two opposing effects: it adds one communication worker but removes one compute SM, or two resident compute-CTA slots. This dependency–residency coupling forms the resource constraint of the pipeline model.
Optimization scope. Across the TP=2/EP=2 evaluation matrix, the ratio of baseline standalone GEMM2+GatherRS latency to complete post-router MoE layer latency has a geometric mean of 78.2%. The corresponding geometric means are 85.3%, 75.6%, and 74.2% for Granite, Qwen, and DeepSeekV2-Lite, respectively. Thus, this operator is a major component of the complete post-router MoE layer. It also exhibits a clear resource mismatch. A fixed GatherRS communication-CTA count cannot adapt to changes in token count, matrix dimensions, split count, or the number of GEMM2 tiles, and can produce a suboptimal tradeoff between insufficient communication parallelism and reduced GEMM2 capacity. In contrast, AllGather+GEMM1 already launches computation incrementally from AllGather readiness and does not exhibit the same fixed-CTA partitioning bottleneck. We therefore optimize the CTA resource partition of GEMM2+GatherRS while leaving AllGather+GEMM1 and the activation unchanged. Fig. 3 summarizes the control path from per-invocation metadata to the selected launch configuration and its logical CTA-to-SM residency. The following subsections define the modeled quantities and selection procedure.
The routed counts Me determine only the GEMM tiles implied by the router; our method neither changes expert assignments nor reschedules work by expert popularity. Given C, the compute waves per split are Wcomp Vcomp (C) = . (7) P (C)
B. Dependency-Constrained Resource Partitioning
Communication-CTA work. Each rank ultimately owns approximately ⌈M/D⌉ original tokens. The current GatherRS
Resource notation. Table I summarizes the static relationship between the GPU and the two kernel roles. Minimum progress-preserving reservation. For all C communication CTAs to acquire residency and advance the dependency pipeline, the minimum reservation is C . (2) R(C) = ocomm With a smaller reservation, simultaneous residency of all communication CTAs is not guaranteed, and additional queueing can delay the consumption of ready splits. With a larger reservation, unchanged communication occupancy provides no additional parallelism while further reducing compute capacity. Hence, R(C) is the minimum spatial partition that realizes
C. Wave-Quantized Pipeline Model The partition must account for the actual work on both sides. Table II defines the workload and tile quantities used below. comp Compute-CTA work. Expert e requires ⌈Me /TM ⌉ tiles along the token dimension. The output width of one split is ⌈N/L⌉, requiring ⌈⌈N/L⌉/TNcomp ⌉ hidden-dimension tiles. The compute-CTA work in one split is therefore ! E X ⌈N/L⌉ Me . (6) Wcomp = g comp TM TNcomp e=1
TABLE I R ESOURCE NOTATION FOR THE GEMM2+G ATHER RS OPERATOR . Symbol Definition G C ocomm ocomp R(C) P (C)
SMs available to the GEMM2+GatherRS operator; 108 on the evaluated A100. GatherRS communication CTAs launched; the primary decision variable. Maximum resident communication CTAs per SM. Maximum resident GEMM2 compute CTAs per SM. Minimum SM residency positions reserved for C communication CTAs. Resident compute-CTA slots remaining after the reservation.
(a)
Software control plane at runtime Workload M, N, K, L, D routed expert loads {M e }
(b) Kernel tile shapes · Ocomp , Ocomm threads · registers · SMEM
A100 G = 108 SMs 65,536 regs · 164 KB SMEM / SM evaluate C = 1, …, C max
Analytical launch-time resource manager Tile-work estimator W comp , W comm
Residency model R(C), P(C)
Dependency + wave makespan A s (C), q(C), F s (C), T(C)
Select C* = arg min T(C)
Selected A100 residency
R(C*)
communication-reserved SMs
SM SM SM comm CTA comm CTA … comm CTA G − R(C*)
remaining compute SMs
SM comp CTA comp CTA
Launch configuration C* · R(C*) · P(C*)
…
SM comp CTA comp CTA
Fig. 3. Launch-time resource selection and A100 residency. (a) The analytical manager selects C ∗ , R(C ∗ ), and P (C ∗ ) from workload, kernel, and GPU inputs. (b) The resulting compute and communication residency.
comm communication tile is TM × TNcomm = 128 × 1024. Communication-CTA work per split is ⌈M/D⌉ ⌈N/L⌉ Wcomm = × . (8) comm TM TNcomm
With C communication CTAs, the required waves are Wcomm Vcomm (C) = . C
(9)
These two wave expressions expose the central tradeoff. Increasing C typically reduces Vcomm (C), but also reduces P (C) and can increase Vcomp (C). Both expressions contain ceilings, so performance changes through steps and plateaus rather than continuously. Adding a CTA within the same wave plateau may provide no benefit; crossing a communicationwave boundary can suddenly improve performance, while simultaneously crossing a compute-wave boundary can abruptly degrade it. Split-level dependency. Wave counts describe how many execution rounds each role requires, but not when those rounds may execute. GEMM2 and GatherRS are not independent: communication can process only a split that compute CTAs have completed and published, while the same communication CTAs sequentially serve multiple splits. The latency of the
TABLE II W ORKLOAD AND TILE NOTATION . Symbol
Definition
M
Global tokens before top-k expansion; M = BS in prefill. Batch size. Number of participating ranks (world size). Number of local experts on the current rank. Token–expert assignments received by local expert e. GEMM2 output dimension (Transformer hidden size). Tensor-parallel local GEMM2 reduction dimension. Number of splits in the GEMM2+GatherRS pipeline. Number of input groups in grouped GEMM2. Compute-tile dimensions along M and N . Communication-tile dimensions along tokens and hidden size. Compute tiles in one split. Communication tiles per split and rank.
B D E Me N K L g comp comp TM , TN comm , T comm TM N Wcomp Wcomm
GEMM2+GatherRS operator therefore depends on both work quantities, compute publication times, and the communication queue. The persistent GEMM visitor places the expert problems of each split contiguously in a global tile sequence. With P (C) resident compute slots, compute CTA b takes global tiles b, b+ P (C), b+2P (C), . . .. This assignment allows CTAs to execute some tiles of later splits before every tile of an earlier split completes. The model therefore derives each split’s publication boundary from the global tile sequence rather than treating splits as isolated compute phases. We use one compute-CTA wave as the time unit. For split s ∈ {0, . . . , L − 1}, completing the first s + 1 splits requires (s + 1)Wcomp compute tiles. Its readiness time is (s + 1)Wcomp . (10) As (C) = P (C) Compute and communication waves have different service times. Let ρ be the service-time ratio of one communicationCTA wave to one compute-CTA wave. The time to process one split with C communication CTAs is q(C) = ρVcomm (C).
(11)
Communication for split s can begin only after its result is published at As (C) and the communication CTAs finish the preceding split at Fs−1 (C). Defining F−1 (C) = 0, the completion time of split s is Fs (C) = max As (C), Fs−1 (C) + q(C). (12) When As (C) > Fs−1 (C), communication waits for computation; otherwise, a ready split waits for communication service. The predicted latency of the GEMM2+GatherRS operator is the completion time of the final split: T (C) = FL−1 (C).
(13)
This recurrence places cross-split compute look-ahead, splitreadiness dependencies, and sequential communication service on one timeline, capturing pipeline fill, steady state, and drain. Launch and fixed metadata costs common to every C do not affect the partition choice, so the model compares candidates using their relative T (C).
Algorithm 1 CPU-side communication-grid selection Require: M, N, K, L, D, {Me }, kernel tile shape and occupancy, G 1: Compute Wcomp and Wcomm 2: for C = 1 to Cmax do 3: R(C) ← ⌈C/ocomm ⌉ 4: P (C) ← ocomp (G − R(C)) 5: Compute all split-readiness times As (C) 6: Compute Vcomm (C) and q(C) 7: F−1 (C) ← 0 8: for s = 0 to L − 1 do 9: Fs (C) ← max(As (C), Fs−1 (C)) + q(C) 10: end for 11: end for 12: C ∗ ← arg minC∈C T (C) 13: return communication grid C ∗ , communication reservation R(C ∗ ), compute capacity P (C ∗ )
Relative wave service rate. The parameter ρ maps compute and communication waves to a common time scale. To separate communication service from split-readiness waiting, we measure both service rates at a fixed C = 16, where communication CTAs can continuously obtain ready work. Compute completes 25 cumulative waves in 497.36 µs, while communication continuously serves two splits in 400.19 µs, giving 400.19/2 ≈ 10.05. (14) ρref = 497.36/25 The reference rate is adjusted for GEMM2 compute intensity and communication scale. A larger reduction dimension K, or a larger GEMM2 tile, increases the work in a compute wave and thereby decreases the time of a communication wave relative to it; more ranks increase relative ring-communication cost. At runtime, we use comp comp Kref TM,ref TN,ref D ρ = ρref , comp comp K TM TN Dref
(15)
where Kref = 352, comp comp TM,ref = TN,ref = 128,
Dref = 4.
(16)
The conversion expresses As (C), q(C), and Fs (C) in compute-wave units. The model can then compare the communication waves eliminated by adding CTAs against the additional compute waves caused by their SM reservation. D. Launch-Time Analytical Selection Before kernel launch, the resource manager receives the current workload M, N, K, L, D, host-side routed counts {Me }, the selected GEMM2 kernel’s tile shape and occupancy, and the GPU SM count G. For the current A100 kernel family, the legal communication-CTA set is C = {1, 2, . . . , Cmax },
Cmax = 16.
(17)
The upper bound comes from a one-time kernel-family saturation characterization; it is not an A100 hardware limit. Section IV validates this boundary. The resource manager executes the procedure shown in Algorithm 1 entirely on the CPU.
When multiple C values have the same wave-quantized T (C), the selector uses the number of empty slots in the final compute wave, ∆comp (C) = Vcomp (C)P (C) − Wcomp ,
(18)
as a tie breaker without additional fitted coefficients, preferring the candidate with smaller ∆comp (C). This rule reduces empty slots in the final incomplete compute wave without using measurement noise to distinguish candidates on the same predicted plateau. The loop evaluates at most 16 analytical candidates. It neither launches nor times candidate GPU kernels and does not query a table of measured per-shape winners. Its complexity is O(E + L|C|): the E term reduces routed expert tiles, while L and |C| are both small in the current implementation. E. System Integration and Correctness The resource manager sits between conventional kernel dispatch and the actual launch. Existing dispatch still selects the GEMM2 compute kernel, after which the manager reads its host-side tile shape and maximum resident blocks per SM. The GPU SM count is cached during operator initialization. Tile shape and occupancy are queried again only when dispatch selects a different GEMM2 kernel, avoiding repeated hardware-query calls on every MoE layer invocation. At runtime, the manager preferentially uses host-side expert split counts already available on the execution path to compute X Me (19) comp . TM e If routed counts exist only on the device, the system uses a normalized-load estimate derived from the total assignments and expert count rather than introducing a device-to-host copy or synchronization solely for scheduling. The system launches C ∗ GatherRS communication CTAs, reserves R(C ∗ ) SM residency positions for communication, and limits the remaining compute capacity to P (C ∗ ). The modeled partition therefore matches the actual kernel launch. The baseline and our method reuse the same GEMM2 compute kernel, GatherRS communication kernel, and readiness protocol. Input tensors, weights, router assignments, GEMM tiles, communication volume, activations, and outputs remain unchanged; only C and its resulting R(C)/P (C) partition differ. An explicit user configuration retains highest priority, and disabling analytical selection restores the upstream fixed configuration and hot path. The measured improvements therefore come from rebalancing the same compute and communication work, not from reducing model work or relaxing correctness requirements. The current implementation and service-rate parameters instantiate the model for a single-node A100/NVLink kernel family. The same modeling method applies to another finegrained overlap pipeline when compute and communication CTAs compete for GPU residency, a describable readiness DAG connects them, and both workloads can be expressed as tile waves. Such an instantiation must, however, rederive
the kernel’s ocomp , ocomm , tile work, dependency recurrence, legal C range, and service rate. IV. E XPERIMENTAL R ESULTS A. Experimental Setup 1) Hardware and Software: All experiments run on the same compute node with four NVIDIA A100-SXM4-40GB GPUs connected by NVLink. They use BF16 precision and evaluate three parallelism configurations: TP = 4/EP = 1, TP = 2/EP = 2, and TP = 1/EP = 4, where TP and EP denote the tensor-parallel and expert-parallel degrees and TP × EP = 4 in every configuration. We use PyTorch 2.7.1, CUDA 12.6, NCCL 2.26.2, CUTLASS commit df8a550, and COMET A100 V2 commit 19831ca; Granite uses one pipeline split, and the other models use two. 2) Models and Workloads: Table III summarizes the three evaluated models. All experiments use batch size B = 4 and sequence length S ∈ {1024, 2048, 4096, 8192, 16384}. GEMM2+GatherRS operator and MoE layer experiments use uniform routing with equal expert probabilities and real traces produced by the native model routers on LMSYS-Chat-1M [3]. For each model and sequence length, we select the actual assignment tensors nearest the empirical p50 and p90 of Rexpert = maxe Le /(M k/E). COMET and our method replay the same selected tensor. Complete-model prefill instead uses the natural per-layer routing of the forward pass. The three models, three routing workloads, and five input lengths form 45 evaluated model–routing–sequence configurations for each TP/EP configuration. 3) Measurement Methodology and Correctness: After 20 warm-up iterations, operator and layer measurements collect 60 ABBA-interleaved samples per policy, while completemodel and multi-backend measurements collect 20 per backend. We report median maximum-rank latency and 95% confidence intervals from 20,000 block-stratified bootstrap replicates. COMET and ours pass four-rank allclose and produce identical complete-model outputs. Against Hugging Face references, all models preserve last-token top-1 predictions, and Qwen and DeepSeek-V2-Lite additionally pass elementwise allclose. 4) Baselines: The primary baseline is the public COMET A100 path described in Section II. Ours changes only C and the resulting residency partition while retaining the same kernels, assignments, arithmetic work, and communication volume. For complete-model prefill, we additionally compare with Megatron core-TE and FastMoE TP+NCCL [7]–[9]. Megatron core-TE uses Transformer Engine GroupedLinear TABLE III M O E MODEL CONFIGURATIONS . H AND F DENOTE THE HIDDEN AND EXPERT-FFN SIZES , RESPECTIVELY.
with NCCL AllGather and ReduceScatter, whereas FastMoE TP+NCCL uses its tensor-parallel expert path with the same NCCL collectives. Due to page limitation, we present detailed results for the representative balanced configuration when T P = 2, EP = 2 and summarize the other two configurations in Fig. 8. B. Predictor Accuracy and Runtime Overhead For each model’s real-p90 TP=4/EP=1 workload at all five sequence lengths, we obtain the measured oracle by sweeping C = 1, . . . , 16 and selecting the lowest median latency. We report regret = (Tpredicted − Toracle )/Toracle . Across the 15 workloads, the mean and maximum regrets are 3.22% and 10.21%, respectively (Table IV). The legal domain C ≤ 16 is frozen by a one-time A100 kernel-family characterization; an extended sweep to C = 32 on a communication-intensive boundary workload found no out-of-domain improvement. The mean standalone solver overhead is 0.157 µs. All subsequent measurements include routedcount extraction, metadata handling, and operator dispatch. C. GEMM2+GatherRS Operator Evaluation Fig. 4 reports the absolute latency of the GEMM2+GatherRS operator and its speedup over upstream COMET. The five-length geometric-mean speedups for uniform, real-p50, and real-p90 are 2.951×, 3.034×, and 3.086× for Granite; 2.856×, 2.813×, and 2.905× for Qwen; and 2.870×, 2.948×, and 2.945× for DeepSeek-V2-Lite. The 95% confidence interval is above 1× at every point. The gains come from resource rebalancing rather than less communication or GEMM work. For Granite real-p50 at S = 2048, for example, latency of the GEMM2+GatherRS operator decreases from 2.8824 to 0.8564 ms, a 3.366× speedup. The best TP=2/EP=2 result is Qwen real-p90 at S = 16384, where latency decreases from 22.0088 to 5.7067 ms, or 3.857×. As S and the model shape change, the predictor adjusts C from the compute and communication wave counts, avoiding the persistent mismatch between communication progress and GEMM capacity created by COMET’s static resource partition. Similar aggregate gains on uniform and real traces show that the predictor remains effective across routing distributions, despite changes in the compute-tile count caused by expertboundary rounding.
TABLE IV P REDICTOR REGRET RELATIVE TO THE MEASURED ORACLE . Evaluation matrix
Model
H
F
Granite-3.1-1B-A400M [4] 1024 512 Qwen1.5-MoE-A2.7B [5] 2048 1408 DeepSeek-V2-Lite [6] 2048 1408
Routed experts Top-k 32 60 64
8 4 6
Mean Maximum
Granite real-p90, five shapes 0.91% Qwen real-p90, five shapes 7.14% DeepSeek-V2-Lite real-p90, five shapes 1.60%
4.13% 10.21% 6.36%
All 15 workloads
10.21%
3.22%
COMET
uniform
lines:
p50
p90 4.0
20
3.5
10
3.0
5
2.5
2
2.0
Speedup (×)
Latency (ms, log scale)
Ours
1.5
1
1.0
S
1k
2k
4k
8k
16k
1k
Granite-3.1-1B-A400M
2k
4k
8k
16k
1k
Qwen1.5-MoE-A2.7B
2k
4k
8k
16k
DeepSeek-V2-Lite
TP2/EP2. Bars within each S: uniform → p50 → p90. Line speedup = COMET latency / Ours latency.
Fig. 4. Latency and speedup of the GEMM2+GatherRS operator versus sequence length S under uniform and real-router p50/p90 workloads with TP=2 and EP=2. COMET
uniform
lines:
p50
p90 2.5
20 2.0
10 5
1.5
2 1.0
1
S
Speedup (×)
Latency (ms, log scale)
Ours
1k
2k
4k
8k
16k
1k
Granite-3.1-1B-A400M
2k
4k
8k
16k
1k
Qwen1.5-MoE-A2.7B
2k
4k
8k
16k
DeepSeek-V2-Lite
TP2/EP2. Bars within each S: uniform → p50 → p90. Line speedup = COMET latency / Ours latency.
latency:
Ours
speedup:
Ours
COMET
Megatron core-TE
FastMoE TP+NCCL
Megatron core-TE
FastMoE TP+NCCL 1.6
2k 1k
1.4
500 1.2
200 100
S
OOM
1k
2k
4k
8k
Granite-3.1-1B-A400M
16k
1k
2k
4k
8k
Qwen1.5-MoE-A2.7B
16k
1k
2k
4k
8k
Speedup (×)
Latency (ms, log scale)
Fig. 5. Complete post-router MoE layer latency and speedup versus sequence length S under uniform and real-router p50/p90 workloads with TP=2 and EP=2.
1.0
16k
DeepSeek-V2-Lite
TP2/EP2. Line speedup = COMET latency / corresponding method latency.
Fig. 6. Complete-model prefill latency and speedup versus sequence length S across four distributed MoE implementations with TP=2 and EP=2.
D. Complete Post-Router MoE Layer Evaluation Fig. 5 reports complete post-router MoE layer latency and speedup for TP=2 and EP=2. Because our method leaves AllGather+GEMM1 and the activation unchanged, the layerlevel speedups are smaller than the operator-level results. The five-length geometric-mean speedups for uniform, real-p50, and real-p90 are 2.241×, 2.271×, and 2.290× for Granite; 1.918×, 1.875×, and 1.897× for Qwen; and 1.929×, 1.939×, and 1.950× for DeepSeek-V2-Lite. For example, Granite realp90 at S = 4096 decreases from 6.0426 to 2.7754 ms, a 2.177× speedup; DeepSeek-V2-Lite real-p90 at S = 2048
decreases from 5.6394 to 2.8993 ms, a 1.945× speedup. At every evaluated point with B = 4 and S ≥ 1024, the 95% confidence interval for the complete post-router MoE layer remains above 1×. E. Complete Model Prefill Evaluation Fig. 6 reports complete-model prefill latency, and Table V gives all paired results for COMET and our method for TP=2 and EP=2. Granite and Qwen cover five sequence lengths and obtain geometric-mean speedups of 1.336× and 1.211×, respectively. DeepSeek-V2-Lite covers four feasible lengths
and has a geometric-mean speedup of 1.232×. The best point is Granite at S = 8192, where latency decreases from 539.44 to 374.76 ms, a 1.439× speedup. As the input grows, the time saved in the GEMM2+GatherRS operator accumulates across MoE blocks, offsetting launch-time metadata and dispatch costs and surviving the unchanged attention, normalization, router, KV-cache update, and LM-head work. Fig. 7 consolidates the pointwise speedups across all three measurement levels. Fig. 6 also compares four implementations under identical model inputs and measurement boundaries. Table VI reports the geometric-mean speedup of our method over each implementation across lengths; DeepSeek-V2-Lite uses its four feasible lengths, while the other models use five. These means do not imply that our method is always fastest at the shortest input. At S = 1024, Megatron core-TE has the lowest latency on all three models; at Qwen S = 2048, its 206.78 ms is also faster than our 227.05 ms. The crossover appears as tile-wave counts grow: our method leads Granite and Qwen from S = 4096 onward and DeepSeek-V2-Lite from S = 2048 onward. It is the fastest of all four implementations at every feasible TP=2/EP=2 point with S ≥ 4096. This trend matches the design target: fixed-partition costs become more exposed when enough token tiles form multiple compute and communication waves, whereas short workloads are more readily dominated by launches, collectives, and unchanged dense operations. DeepSeek-V2-Lite complete-model prefill at B = 4, S = 16384 exceeds the memory capacity of a 40-GB A100. We retain the Out-of-Memory (OOM) result rather than reducing the batch size or sequence length or changing the KV-cache boundary. The GEMM2+GatherRS operator and complete post-router MoE layer at the same shape do not retain the full model state, so they remain feasible and are included in the first two result sets. Accordingly, these results apply to the complete-model prefill boundary defined above, not to complete online-serving or decode throughput.
TABLE V C OMPLETE - MODEL PREFILL RESULTS AGAINST COMET WITH TP=2 AND EP=2. Model
S COMET (ms) Ours (ms) Speedup
Granite Granite Granite Granite Granite
1024 2048 4096 8192 16384
128.78 185.19 292.25 539.44 1130.41
111.59 141.11 209.49 374.76 807.37
1.154× 1.312× 1.395× 1.439× 1.400×
Qwen Qwen Qwen Qwen Qwen
1024 2048 4096 8192 16384
178.32 276.08 471.08 911.45 1982.85
158.68 227.05 380.32 724.61 1618.15
1.124× 1.216× 1.239× 1.258× 1.225×
DeepSeek-V2-Lite DeepSeek-V2-Lite DeepSeek-V2-Lite DeepSeek-V2-Lite
1024 2048 4096 8192
211.28 350.85 648.49 1404.36
182.21 276.77 511.66 1134.45
1.160× 1.268× 1.267× 1.238×
TABLE VI G EOMETRIC - MEAN COMPLETE - MODEL PREFILL SPEEDUP OF OURS OVER EACH BACKEND WITH TP=2 AND EP=2. Model
COMET Megatron core-TE FastMoE TP+NCCL
Granite Qwen DeepSeek-V2-Lite
1.336× 1.211× 1.232×
1.064× 1.002× 1.026× <1 slower
Granite-3.1-1B-A400M
1.151× 1.125× 1.176×
>1 faster
Qwen1.5-MoE-A2.7B
DeepSeek-V2-Lite
1k
2k
4k
8k
16k
1k
2k
4k
8k
16k
1k
2k
4k
8k
16k
uniform
2.17
3.49
2.96
2.92
3.42
2.05
3.29
2.66
3.19
3.32
2.32
3.06
2.58
3.27
3.25
p50
2.27
3.37
3.15
3.15
3.40
2.22
3.08
2.67
2.89
3.34
2.31
3.13
2.59
3.46
3.44
p90
1.95
3.17
3.61
3.47
3.62
2.10
2.61
2.98
3.27
3.86
2.05
2.87
2.97
3.54
3.59
uniform
1.78
2.53
2.13
2.28
2.58
1.54
2.07
1.83
2.03
2.19
1.61
2.03
1.83
2.12
2.10
p50
1.96
2.29
2.09
2.49
2.57
1.56
1.76
1.78
2.17
2.19
1.66
1.79
2.06
2.11
2.11
p90
1.83
2.43
2.18
2.57
2.53
1.59
1.83
1.81
2.17
2.16
1.59
1.95
2.05
2.13
2.09
COMET
1.15
1.31
1.40
1.44
1.40
1.12
1.22
1.24
1.26
1.23
1.16
1.27
1.27
1.24
OOM
Megatron
0.73
0.89
1.15
1.35
1.36
0.80
0.91
1.05
1.15
1.15
0.82
1.01
1.14
1.17
OOM
FastMoE
0.81
1.04
1.26
1.37
1.39
0.95
1.11
1.19
1.21
1.20
1.04
1.20
1.24
1.23
OOM
(a)
(b)
(c)
TP2/EP2. (a) GEMM2+GatherRS stage; (b) complete post-router MoE layer; (c) complete-model prefill.
Speedup = baseline latency / Ours latency.
Fig. 7. Pointwise speedup with TP=2/EP=2 at the GEMM2+GatherRS operator, complete post-router MoE layer, and complete-model prefill levels. Values are baseline latency divided by our latency; green and red indicate speedup and slowdown. Underlines mark row maxima, and OOM indicates an infeasible run. Megatron and FastMoE denote Megatron core-TE and FastMoE TP+NCCL. TP4/EP1
TP2/EP2
(a) Stage
TP1/EP4 (c) Prefill
(b) MoE layer
5.0
3.0
3.0
2.0
1.6 1.4 1.2
1.0
1.0
1.0
Granite QwenDeepSeek Granite QwenDeepSeek Granite QwenDeepSeek Metric: geometric-mean speedup over COMET (×).
Fig. 8. Speedup over COMET across the three four-GPU TP/EP configurations. Bars show geometric means and whiskers show pointwise ranges: (a) GEMM2+GatherRS operator; (b) complete post-router MoE layer; (c) complete-model prefill.
F. Cross-Configuration Summary Fig. 8 summarizes the speedups over COMET across all three TP/EP configurations. Every configuration maintains a pointwise minimum above 1× at all three measurement levels. The lowest observed speedup is 1.058×, obtained for Qwen complete-model prefill with TP=4/EP=1 at S = 1024. TP=2/EP=2 achieves the largest geometric-mean speedup at every measurement level, followed by TP=1/EP=4 and TP=4/EP=1. Across all configurations, the geometric-mean speedups over COMET are 2.528× at the GEMM2+GatherRS operator, 1.771× at the complete post-router MoE layer, and 1.185× for complete-model prefill. The corresponding maximum speedups are 4.218×, 2.584×, and 1.439×, respectively. V. R ELATED W ORK Prior work spans distributed MoE execution, fine-grained computation-communication overlap, and GPU resource management. These areas address different levels of the execution stack: distributed MoE systems organize operators and
devices, overlap systems construct fine-grained dependency pipelines, and resource-management systems allocate execution capacity among concurrent GPU tasks. Distributed MoE systems coordinate expert placement, token dispatch and combination, grouped expert computation, and interdevice communication. Tutel adaptively selects parallelization strategies, All-to-All algorithms, and pipeline degrees [10]. PipeMoE models computation and communication to select a pipeline depth [11]. ScheMoE schedules computation and communication over a task graph [12]. FasterMoE uses expert shadowing to mitigate device-level load imbalance caused by dynamic routing [13]. These approaches primarily determine where a task runs, how it is partitioned, and when it starts, These approaches primarily determine where a task runs, how it is partitioned, and when it starts, thereby exposing more opportunities for overlap. But they do not explicitly derive the resident compute/communication CTA ratio within a finegrained dependency-coupled operator. Fine-grained overlap efforts reduce synchronization at operator boundaries by exposing compute-to-communication dependencies at tile or split granularity. Recent works use readiness signaling, tile scheduling, work reordering, concurrent kernels, and thread-block specialization to allow communication to consume partial results before all computation completes [14], [15]. COMET, the direct predecessor of this work, constructs such dependency pipelines through shared-tensor dependency analysis, tile rescheduling, horizontal fusion, and thread-block specialization [1]. FLUX [2] decomposes computation and communication to tile granularity and combines kernel fusion with dependency synchronization to reduce exposed communication time. TileLink instead provides tile-centric primitives and a compiler backend that generates overlap kernels with lower programming effort [16]. These systems establish execution mechanisms and programming abstractions for fine-grained overlap. FlashDMoE further places dispatch, expert computation, and combine in a single persistent GPU kernel, using device-initiated communication to remove host orchestration from the critical path [17]. Our work begins after such dependency-coupled execution has been established and models and controls the resource allocation between its compute and communication CTAs. State-of-the-art approaches have also studied how GPU execution resources should be allocated and scheduled across concurrent operators, communication tasks, and tiled workloads. NanoFlow jointly selects operation order and GPU resource allocation for LLM serving through profiling- and interference-aware search [18]. Lagom uses a unified cost model and measurement-guided iterative search to configure multiple computation and communication operators in distributed LLM training [19]. DeepEP V2 analytically determines SM and queue-pair requirements for expert-parallel communication from traffic and link characteristics [20], while Stream-K analyzes the wave-quantization loss caused by unevenly distributed GEMM tiles [21]. These studies establish the importance of resource contention, communication provisioning, and discrete work waves. Our work
focuses on a narrower dependency-coupled MoE operator, combining non-preemptive CTA residency, wave quantization, and fine-grained readiness dependencies to select the compute–communication allocation analytically at launch time without per-shape profiling. VI. C ONCLUSION This paper presents a launch-time resource manager for fine-grained computation–communication overlap in distributed MoE inference. Given the workload, CTA residency constraints, and split-level dependencies, it selects the communication-CTA count and the compute/communication resource partition before kernel launch. We integrate the method into the COMET A100 implementation in FLUX. Across three MoE models and three TP/EP configurations on four A100 GPUs, the method achieves geometric-mean speedups over COMET of 2.528× for the GEMM2+GatherRS operator, 1.771× for the complete post-router MoE layer, and 1.185× for complete-model prefill. The corresponding maximum speedups are 4.218×, 2.584×, and 1.439×. All 95% confidence intervals for the operator and layer measurements remain above 1×. Future work will evaluate the analytical model on newer GPU architectures and multi-node interconnects, and extend resource selection to decode and other computation– communication operators with fine-grained dependencies. ACKNOWLEDGMENT Computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), funded by the Swedish Research Council. The computations were performed on the Alvis system hosted by Chalmers e-Commons at Chalmers University of Technology. The authors used OpenAI Codex to assist with language editing. The authors reviewed and revised the resulting text and take responsibility for the manuscript. R EFERENCES [1] S. Zhang et al., “COMET: Fine-grained computation-communication overlapping for mixture-of-experts,” in Proc. Mach. Learn. Syst. (MLSys), 2025. [2] L.-W. Chang et al., “FLUX: Fast software-based communication overlap on GPUs through kernel fusion,” arXiv preprint arXiv:2406.06858, 2024. [3] L. Zheng, W.-L. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang, “LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset,” in Proc. Int. Conf. Learn. Representations (ICLR), 2024. [4] IBM Granite Team, “Granite-3.1-1B-A400M-Base,” [Online]. Available: https://huggingface.co/ibm-granite/granite-3.1-1b-a400m-base, 2024, accessed: Jul. 27, 2026. [5] Qwen Team, “Qwen1.5-MoE: Matching 7b model performance with 1/3 activated parameters,” [Online]. Available: https://qwenlm.github.io/ blog/qwen-moe/, 2024, accessed: Jul. 27, 2026. [6] DeepSeek-AI et al., “DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model,” arXiv preprint arXiv:2405.04434, 2024. [7] NVIDIA Corporation, “Megatron-Core,” [Online]. Available: https:// github.com/NVIDIA/Megatron-LM, 2026, accessed: Jul. 24, 2026. [8] ——, “Transformer Engine,” [Online]. Available: https://github.com/ NVIDIA/TransformerEngine, 2026, accessed: Jul. 24, 2026.
[9] J. He, J. Qiu, A. Zeng, Z. Yang, J. Zhai, and J. Tang, “FastMoE: A fast mixture-of-expert training system,” arXiv preprint arXiv:2103.13262, 2021. [10] C. Hwang et al., “Tutel: Adaptive mixture-of-experts at scale,” in Proc. Mach. Learn. Syst. (MLSys), vol. 5, 2023. [11] S. Shi, X. Pan, X. Chu, and B. Li, “PipeMoE: Accelerating mixtureof-experts through adaptive pipelining,” in Proc. IEEE Conf. Comput. Commun. (INFOCOM), 2023, pp. 1–10. [12] S. Shi, X. Pan, Q. Wang, C. Liu, X. Ren, Z. Hu, Y. Yang, B. Li, and X. Chu, “ScheMoE: An extensible mixture-of-experts distributed training system with tasks scheduling,” in Proc. 19th Eur. Conf. Comput. Syst. (EuroSys), 2024, pp. 236–249. [13] J. He, J. Zhai, T. Antunes, H. Wang, F. Luo, S. Shi, and Q. Li, “FasterMoE: Modeling and optimizing training of large-scale dynamic pre-trained models,” in Proc. 27th ACM SIGPLAN Symp. Principles and Practice of Parallel Programming (PPoPP), 2022, pp. 120–134. [14] M. Cui, A. Wingkvist, and M. Ericsson, “Fine-grained computationcommunication overlap via tile-level signaling and scheduling for mixture-of-experts,” arXiv preprint arXiv:2607.19539, 2026. [15] H. Ke, X. Li, M. Liu, Q. Mao, T. Wu, Z. Huang, L. Chen, Z. Wang, Y. Zhang, Z. Zhu, G. Dai, and Y. Wang, “FlashOverlap: A lightweight design for efficiently overlapping communication and computation,” arXiv preprint arXiv:2504.19519, 2025. [16] S. Zheng et al., “TileLink: Generating efficient compute-communication overlapping kernels using tile-centric primitives,” in Proc. Mach. Learn. Syst. (MLSys), 2025. [17] O. J. Aimuyo, B. Oh, and R. Singh, “FlashDMoE: Fast distributed MoE in a single kernel,” arXiv preprint arXiv:2506.04667, 2025. [18] K. Zhu et al., “NanoFlow: Towards optimal large language model serving throughput,” in Proc. 19th USENIX Symp. Operating Systems Design and Implementation (OSDI), 2025, pp. 749–765. [19] G. Xu et al., “Lagom: Unleashing the power of communication and computation overlapping for distributed LLM training,” arXiv preprint arXiv:2602.20656, 2026. [20] DeepSeek-AI, “DeepEP: An efficient expert-parallel communication library,” [Online]. Available: https://github.com/deepseek-ai/DeepEP, 2026, accessed: Jul. 24, 2026. [21] M. Osama, D. Merrill, C. Cecka, M. Garland, and J. D. Owens, “Stream-K: Work-centric parallel decomposition for dense matrix-matrix multiplication on the GPU,” arXiv preprint arXiv:2301.03598, 2023.