arXiv:2605.05049v1 [cs.DC] 6 May 2026
Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism Sajal Dash
Feiyi Wang
Oak Ridge National Laboratory Oak Ridge, USA Email: [email protected]
Oak Ridge National Laboratory Oak Ridge, USA Email: [email protected]
MoE [13], exacerbate these challenges further: they produce many tall-and-skinny GEMMs with poor hardware utilization, inflate activation memory, and require all-to-all collectives involving a large number of participating processes. These difficulties compound when training on shared HPC platforms which are primarily designed for modeling and simulation workloads and feature non-uniform communication fabrics across GPU nodes. Prior work such as X-MoE [14] has demonstrated that communication overhead becomes a dominant performance bottleneck in exactly this setting, yet no holistic framework exists for systematically characterizing, quantifying, and mitigating these inefficiencies across both model and platform architecture. In this work, we develop mathematical models to quantify memory, compute, and communication requirements for diverse MoE architectures under different distributed training strategies and their combinations, parameterized by empirically measured platform characteristics — including memory capacity, GPU throughput, and network bandwidth. We validate these models I. I NTRODUCTION through comprehensive experimental training runs using frameTransformer based Large Language Models (LLMs) have works including DeepSpeed-TED [15], DeepSpeed-MoE [12], demonstrated consistent performance gains with increasing and X-MoE [14] on the Frontier supercomputer [16]. model size [1], [2]. However, training these models at scale Using our resource modeling framework alongside experis resource-intensive: a mixed-precision training run requires imental profiling, we identify prominent sources of missed approximately 20 bytes of memory per parameter, roughly performance: (i) all-to-all latency at scale caused by expert 6 floating-point operations per parameter per token, and parallelism over non-uniform interconnects; (ii) load imbalsubstantial inter-device communication [3], [4]. The Mixture- ance from skewed expert assignment, resulting in low GPU of-Experts (MoE) mechanism, in which only a fraction of the utilization during early stages, and, in case of no external model parameters get sparsely activated for each token, has mechanisms, throughout the majority of training; and (iii) a emerged as a promising approach to achieving dense-models’ lack of system-aware hybrid parallelization strategies. We then performance at a reduced computational cost with savings develop targeted solutions for each. proportional to the sparsity factor [5], [6]. State-of-the-art models such as Mixtral [7], DeepSeek [8], Qwen [9], and A. Contributions Kimi [10] have adopted MoE to deliver superior performance 1) Analytical and Empirical Resource Modeling: We at substantially reduced training cost. Despite these advantages, develop a mathematical model for estimating memory, MoE training presents its own unique challenges. Relative to compute, and communication utilization during MoE a parameter-matched dense model, MoE training introduces training across a range of architectures and parallelizacomputational load imbalance across devices, an elevated tion configurations, and empirically validate it through memory footprint from storing both model parameters and micro-benchmarking, code instrumentation, and hardware transient activation tensors, and high inter-device communiprofiling. cation volume under expert parallelism — the most widely 2) Piper: Pipeline Parallelism for Localizing Commuused MoE distribution method [11], [12]. Emerging MoE nication: We introduce Piper, a framework that applies architectures with fine-grained experts such as DeepSeekpipeline parallelism to intra-layer parallelization strategies
Abstract—Frontier models increasingly adopt Mixture-ofExperts (MoE) architectures to achieve large-model performance at reduced cost. However, training MoE models on HPC platforms is hindered by large memory footprints, frequent largescale communication across heterogeneous networks, and severe workload imbalance. To characterize these challenges, we develop a mathematical model that quantifies memory, compute, and communication requirements for MoE configurations under various parallelization schemes, verified through micro-benchmarking, code instrumentation, and hardware profiling. Our analysis identifies performance bottlenecks: all-to-all latency at scale from expert parallelism, insufficient compute-communication overlap, low GPU utilization from imbalanced skinny GEMMs, and the absence of platform-aware hybrid parallelization strategies. To address these, we introduce Piper, a framework that leverages resource modeling to identify efficient training strategies for MoE models on target HPC platforms, applying pipeline parallelism with optimized schedules. Piper achieves 2–3.5× higher MFU than state-of-the-art frameworks such as X-MoE, and a novel all-to-all algorithm delivers 1.2X–9X bandwidth over vendor implementation.
in order to localize and overlap expensive collective GPU without tensor splitting, fine-grained designs generate communications, and uses the resource model to automat- tall-and-skinny GEMMs with poor hardware utilization, inflate ically identify efficient training configurations for target activation memory by factor mm m, and require all-to-all HPC platforms. Piper achieves 2–3.5X higher Model collectives spanning many GPUs. FLOP Utilization (MFU) compared to state-of-the-art MoE training framework, X-MoE [14]. B. Distributed Training Frameworks and Hybrid Parallelism 3) Topology-Aware All-to-All Algorithm: We design a Training large MoE models requires combining data parDragonfly-topology-aware hierarchical all-to-all algorithm allelism (DP) [4], tensor parallelism (TP) [19], pipeline that exploits the dependency structure of asynchronous parallelism (PP) [20], [21], and expert parallelism (EP) [11], point-to-point communications, groups traffic over slower where different experts are placed on different devices and inter-node and inter-cabinet links, and saturates NICs tokens are routed via all-to-all collectives. uniformly, achieving 1.5X–4X the bandwidth of vendorDeepSpeed-MoE [12] combines expert parallelism with provided implementations. tensor parallelism and ZeRO memory sharding [4], primarily 4) Expert Migration for Load Balancing: We develop an targeting coarse-grained architectures. DeepSpeed-TED [15] expert migration technique in which GPUs hosting the extends this by jointly optimizing across Tensor, Expert, and same layer periodically exchange experts to re-balance Data parallelism axes. load, incurring an amortized overhead of less than 5% of X-MoE [14] targets fine-grained expert architectures, identotal training time. tifying activation memory and all-to-all scope as primary 5) Trillion-Scale MoE Training: Using Piper and our bottlenecks. It introduces zero-padding for load balancing, resource modeling tool, we devise and validate trainredundancy-based communication bypassing, and sequenceing strategies for several state-of-the-art MoE models sharded parallelism, successfully outperforming DeepSpeedat 20–50% MFU, and demonstrate training of trillionMoE and Tutel at hundreds-of-billions scale. However, for parameter MoE models at 20% MFU on the Frontier 500B+ models, X-MoE achieves only 5% MFU. supercomputer. For reference, X-MoE reported training Tutel [22] provides efficient MoE dispatch and combine of a 545B parameter model at 5.23% MFU. kernels with dynamic top-K routing and adaptive parallelism II. BACKGROUND AND R ELATED W ORK switching, but focuses on the dispatch kernel rather than end-toWe survey prior work across five dimensions relevant end training strategy selection and does not cover attention-layer to this paper: MoE model architectures, distributed training parallelization. A general limitation across these frameworks is the absence frameworks and hybrid parallelism, collective communication of platform-aware hybrid parallelism planning that jointly algorithms, load balancing techniques, and pipeline parallelism. accounts for memory, compute, and communication constraints, We also review HPC platform topology characteristics that a gap Piper directly addresses (Section III). motivate our system-level design choices. Table I summarizes the architectural parameters of representative state-of-the-art C. Pipeline Parallelism MoE models. A. MoE Architectures The MoE mechanism replaces the feed-forward network (FFN) sublayer in a Transformer block [2] with a collection of expert FFNs, each receiving only the tokens routed to it by a learned gating function [5]. There are two primary architectural streams. Coarse-grained MoE. Early large-scale MoE models such as GShard [11], Switch Transformer [6], and the Mixtral family [7] employ a small number (typically 8–64) of large experts whose FFN dimension matches a comparably sized dense model, with top-1 or top-2 routing per token. Because individual experts frequently exceed single-GPU memory capacity, they require tensor parallelism or sharded data parallelism, substantially complicating the communication pattern. Fine-grained MoE. Pioneered by DeepSeek-MoE [13] and followed by Qwen3 [17], Kimi K2 [18], and others, this approach decomposes each expert into many smaller experts; reducing FFN dimension by a factor m while increasing expert count proportionally and selects a larger top-K (K ∈ [6, 8, 16]). 16]) per token. While individual experts now fit within a single
Pipeline parallelism partitions model layers across devices using micro-batching [20]. The 1F1B schedule [21] reduces pipeline bubble fraction and peak activation memory over GPipe [23]; interleaved variants [3] and ZB-H1/H2 schedules [24] reduce bubble overhead further. These techniques target dense models where communication occurs between layers rather than within them. Piper extends pipeline parallelism to the intra-layer axis introduced by expert parallelism, enabling computation-communication overlap within MoE layers. D. Load Balancing Uneven token distribution across experts reduces effective GPU throughput. Common mitigations include auxiliary loadbalancing losses [6], [11], token dropping [6], expert-choice routing [25], and the auxiliary-loss-free bias-adjustment strategy of DeepSeekV3 [8]. All operate at the routing level and cannot correct device-level imbalance from oblivious expert placement. Our expert migration approach (Section VI) complements these methods by physically redistributing experts based on observed load.
TABLE I: State-of-the-Art Mixture-of-Experts (MoE) Model Configurations Model
Total Params
Active Params
Total Experts
Active / Token
Layers
Hidden Size
FFN Dim (per expert)
Context
Train Tokens
DeepSeek-V2 DeepSeek-V3 DeepSeek-V3.2
236B 671B 671B
21B 37B 37B
162a 257b 257b
6R + 2S 8R + 1S 8R + 1S
60 61 61
5,120 7,168 7,168
1,536 2,048 2,048
128K 128K 128K
8.1T 14.8T —
Mixtral 8×7B Mixtral 8×22B
∼47Bc 141B
∼13B 39B
8 8
Top-2 Top-2
32 56
4,096 6,144
14,336 16,384
32K 64K
— —
Qwen3-30B-A3B Qwen3-235B-A22B
30B 235B
3B 22B
128d 128d
Top-8 Top-8
48 94
2,048 7,168
768 2,048
128K 128K
∼36T ∼36T
Llama 4 Scout Llama 4 Maverick
109B 400B
17B 17B
17e 129e
1R + 1S 1R + 1S
∼48 ∼48
∼5,120 ∼5,120
∼8,192 ∼8,192
10M 1M
40T 40T
Arctic
480B
17B
128f
Top-2
—
—
∼3,660
128K
3.5T
Kimi K2
∼1T
32B
384
Top-8
61
7,168
2,048
128K
15.5T
a 160 routed (R) + 2 shared (S) experts.
b 256 routed + 1 shared expert.
c Mixtral replicates only the FFN layers; attention weights are shared, giving
∼47B total rather than 8 × 7B = 56B. d No shared experts; uses global-batch load-balancing loss. e Maverick uses alternating dense and MoE layers; values marked ∼ are approximate (not fully disclosed by Meta). f Arctic is a Dense-MoE hybrid: 10B dense transformer backbone + residual 128 × 3.66B MoE MLP. R = routed, S = shared.
E. Collective Communication Algorithms Under expert parallelism, each MoE layer incurs four all-toall operations (two per forward pass and two per backward), making all-to-all a dominant latency contributor at scale [14], [12]. Flat all-to-all. NCCL [26] and RCCL [27] perform direct point-to-point transfers between all process pairs, which is bandwidth-optimal under a uniform topology but performs poorly on hierarchical networks where inter-node bandwidth is significantly lower than intra-node bandwidth. Hierarchical all-to-all. Tutel [22], FasterMoE [28], and HetuMoE [29] use a two-phase approach, intra-node aggregation followed by reduced inter-node transfers, substantially reducing cross-node messages. However, these algorithms treat the inter-node network as homogeneous. Dragonfly topology. HPC systems such as Frontier [16] employ Dragonfly networks [30] with high-bandwidth intragroup links and sparser inter-group links. Topology-oblivious algorithms cause unnecessary contention on slower inter-group links. Our topology-aware all-to-all (Section V) explicitly models this three-level hierarchy (intra-node, intra-group, intergroup) and coordinates asynchronous point-to-point communication to eliminate idle cycles. F. Analytical Performance Modeling Roofline analysis [31] bounds achievable performance by arithmetic intensity and memory bandwidth, informing kernel optimization for attention and FFN layers [32]. Korthikanti et al. [33] develop analytical models for pipeline bubble and activation memory in dense Transformer training; PaLM [34] uses empirical roofline fitting to project hardware efficiency across configurations. For MoE models, interactions among expert parallelism, routing, and load imbalance introduce variables that dense-model frameworks do not capture. No prior work provides a unified model spanning memory, compute,
and communication jointly across different parallelization dimensions validated on a real HPC platform, a gap we address in Section III. III. P IPER : A F RAMEWORK FOR P IPELINING M O E T RAINING VIA R ESOURCE M ODELING Piper is a framework for efficient MoE model training on HPC platforms. Its design rests on two observations. First, existing frameworks such as DeepSpeed-MoE, DeepSpeedTED, and X-MoE, distribute all model components across large, groups of GPUs, forcing expensive collective communications (4 all-reduce for tensor parallelism, 2 all-gather for sharded data parallelism, 4 all-to-all for expert parallelism) to span many ranks simultaneously. Second, pipeline parallelism, which is the standard tool for bounding communication group size in dense model training [3], has not been applied to MoE training due to the added complexity of intra-layer expert parallelism. Piper closes this gap by composing pipeline parallelism with expert parallelism. It organizes P GPUs into a P P × EP device mesh: P P pipeline stages, each staffed by EP GPUs that handle one partition of experts for L/P P layers via expertdata parallelism. Confining expert-parallel communication to a small, topologically local group of GPUs, ideally within a single node or a single-hop Rosetta switch group on Frontier allows Piper to exploit fast intra-node interconnects and avoid the high latency of large-scale all-to-all collectives. We expand on Tutel [22] to support the underlying expert-parallelism. Piper consists of four components, described in turn: (i) an analytical resource model that estimates memory, compute, and communication for any (P P, EP ) configuration and a model architecture (Section III-A1); (ii) a micro-benchmarking suite that measures platform-specific bandwidth and throughput to parameterize the model (Section IV-A); (iii) a performance estimator that scores valid configurations by predicted MFU (Section IV-C); and (iv) a pipelined training executor that
implements the selected strategy with an efficient 1F1B schedule (Section VII). At the core of our framework is Pipeline Parallelism on top of expert parallelism (Figure 2). We use Tutel for facilitating expert-parallelism. Piper framework has a resource modeling component, a micro-benchmarking suit, a performance estimator, and a pipelined training tool that partitions MoE models across layers.
grad, 4 Bytes for fp32 master copy, and (4 + 4 = 8) Bytes fp32 momentum and variance. We first establish a lower bound on the memory by assuming everything fits in a hypothetical GPU with infinite memory so that no model parallelism is required (Table III). There are 4d2model attention parameters (WK , WQ , WV ∈ Rdmodel ×dmodel ) for Multi-Head Attention (MHA). Each of the FFN experts has 3dmodel df f n weights since there are three weight matrices (Wup , Wgate ∈ Rdmodel ×df f n and A. Resource Modeling for MoE Training Wdown ∈ Rdf f n ×dmodel ). Accounting for the activation memory, intermediate output Training large MoE models faces two coupled resource is calculated in half precision (2 Bytes), for b sequences with s constraints: (i) memory pressure from model parameters, optimizer states, and expert activations, and (ii) communication sequence length, each expert receives se ; E[se ] ≈ bsk tokens. E latency from all-to-all collectives under expert parallelism. For a single expert with SwiGLU activations, each token activatWe develop analytical models for both, parameterized by the ing an expert creates 3d f f n (with fused kernel) or 4df f n values notation in Table II, and use them to prune the (P P, EP ) (up, gate, down). Since E[s ] = bsk/E, per expert activation e search space to configurations that are memory-feasible and to memory is 2Bytes × bsk/E × (3d f f n + dmodel ). Activation rank feasible configurations by predicted throughput. memory from the attention module is 2Bytes × (6bsdmodel a) Notations: We will use the notations listed in Table II (Q, K, V projections, Attention output, output projections) + for resource modeling. 2Bytes × 2bHs2 (Attention score, softmax output) (Table III). With, flash attention, 4bHs2 → 2bHs. TABLE II: Symbol definitions. Symbol d L LMoE H dh E Es k nmat dMoE ffn dd ffn PP EP P = P P × EP g s b bµ = b/M M = α · PP α i βp Mfw
Description Model hidden dimension (dmodel ) Total transformer layers (assume all are MoE layers) Number of MoE layers; L − LMoE are dense Number of attention heads Per-head dimension (H · dh = d) Routed experts per MoE layer Shared (always-active) experts per MoE layer Top-k routing (experts activated per token) Weight matrices/expert: 3 (SwiGLU) Expert FFN intermediate dimension Dense FFN intermediate dimension Pipeline parallel degree Expert parallel degree Total GPU count GPUs per node Sequence length (tokens) Global batch size (sequences) Microbatch size Total microbatches per gradient step Microbatch multiplier Pipeline stage index, 0 ≤ i ≤ P P − 1 Bytes per parameter (on GPU) Framework overhead (RCCL buffers, etc.)
1) Modeling Memory: a) Modeling memory under expert-data parallelism: The total memory required for training an MoE model constitutes a) static memory (parameter + optimizer states + gradients) and b) activation memory. Mixed-precision training stores parameters in multiple formats simultaneously. The total bytes consumed on GPU per parameter is 16, 2 Bytes for fp16 param, 2 Bytes for fp16
Model
#Parameters
Model Memory
Activation Memory
Attention
4d2model
64d2model
12bs dmodel + 4bHs2
Experts
3Edmodel df f n
48Edmodel df f n
2bsk(3df f n + dmodel )
TABLE III: Estimating training memory
Mu = L 64d2model + 48Edmodel df f n + 12bsdmodel (1) + 4Hbs2 + 2bsk 3df f n + dmodel 2) Memory with Expert Data Parallelism: In expert data parallelism, the world size is P = EP . The non-expert modules (attention) get replicated across P GPUs and each GPU gets E/EP experts. The total memory consumption increases due to the replicated attention module. However, the per GPU memory requirement comes down to 48E Medp = L 64d2model + dmodel df f n + 12bsdmodel EP (2) 2bsk + 4Hbs2 + 3df f n + dmodel EP a) Memory Under Pipelined Expert Parallelism: In the hybrid Pipeline-Expert parallelism, we divide the P GPUs into P P × EP grid and there are EP pipeline parallel groups of size P P and there are P P expert-parallel groups of size EP . In each of the P P pipeline stages, there are EP GPUs hosting l = L/P P layers. Each of these EP GPUs is replicating the non-expert parameters (attention, router) and hosting E/EP experts in an expert-data parallel way. So, each GPU’s memory is l × (attention-memory + E/EP expert-memory). The peak activation memory in GPipe schedule is for all of the M
Fig. 1: Piper framework for efficient MoE training
∆M = M1F1B (0) − M1F1B (P P − 1) 4b 2bsk L(P P − 1) 12b sdmodel + Hs2 + (3dffn + dmodel ) = PP M M M · EP (5) That means, the first stage needs to hold (P P − 1)-times more activation memory than the last stage. This creates heavily skewed memory pressure across pipeline stages. B. Modeling Communication 1) Communication Under Expert-Data Parallelism: Under expert data parallelism, major communication happens due to the activation values. There are two phases of all-to-all communications. microbatches since at the steady state all microbatches needs a) Dispatch and Combine: Each data-parallel router to stay alive in the memory. routes it’s bsk tokens (bs tokens, but each token gets to k experts) to E experts. Under proper load-balancing, every 48E L GP ipe 2 GPU sends bsk/EP tokens to every other GPU. The com64dmodel + dmodel df f n Medp×pp = PP EP munication volume between a pair of GPUs is 2bsk/EP and 2bsk 2 total communication volume during the dispatch all-to-all is + 12bsdmodel + 4Hbs + 3df f n + dmodel 2bsk EP EP × = (EP − 1)bsk. These are tokens, so they (3) 2 EP have dmodel dimensions, in fp16/bf16, the total message size b) 1F1B Pipeline Schedule: In 1F1B scheduling, stage i is 2(EP − 1)bsk × dmodel bytes. Individual message size is (0-indexed, i ∈ {0, 1, . . . , P P − 1}) holds (P P − i) in-flight 4bskdmodel bytes. microbatch activations simultaneously at peak. Each microbatch ≈ EP Combine is the same communication in the reverse direction. has size b/M , where M is the total number of microbatches. So, the message volumes are same. The per-GPU send volume Hence, the per-GPU memory for stage i is: model during dispatch is 2bskd bytes (fp16), giving a per-NIC EP 4bskdmodel injection load of EP ·BNIC seconds at NIC bandwidth BNIC . L 48E 1F1B Medp×pp (i) = × 64d2model + · dmodel dffn + (P P − i) Since combine is the reverse operation, the total all-to-all PP EP latency per MoE layer in the forward pass is bounded by: ! 12b 4b 2bsk 4bskdmodel sdmodel + Hs2 + (3dffn + dmodel ) Ta2a ≥ (6) M M M · EP EP · BNIC (4) This bound is tight when NICs are uniformly saturated; The memory difference between the first and last stage is: Section V discusses why the flat RCCL all-to-all fails to Fig. 2: Pipeline Parallelism on Expert Parallelism
D. Pipelined Training Execution Among valid configurations, Piper ranks them by estimated MFU. Then it interfaces PyTorch Distributed Pipeline Parallelism with expert-parallel or any other hybrid intranode distribution library such as Tutel. We instrumented 1F1B schedule and installed synchronization mechanisms among expert-parallel group members. We expanded Tutel and PyTorch’s pipeline parallelism mechanism to work with each other under a two-dimensional parallelization (three, counting external data parallelism). IV. P ERFORMANCE M ODELING THROUGH M ICRO - BENCHMARKING Resource modeling gives us a realistic expectation regarding viable distributed training strategies in terms of number of nodes, degrees of expert and pipeline data parallelism, whether to adopt memory saving techniques such as checkpoint activation, offloading, etc. This can be done largely statically by the mathematical formulas we developed in Section III-A and considering system properties such as HMB memory. Once we find candidate strategies to train our model without running into out of resource (e.g., OOM) error. To estimate the model flops utilization or MFU for each of these strategies, we need to micro-benchmark the HPC
TFLOPS (QKV proj + causal SDPA + out proj)
achieve it on Dragonfly topologies and how our topology-aware platform for various phases of the MoE training workflow. With pipelined expert-data parallelism, the typical flow is algorithm addresses this. 2) Communication under Pipeline and Expert-Data Paral- attn → routing → dispatch a2a → expert → combine a2a → P 2P ×l lelism: World size P = EP × P P . For each stage hosting l-layers, there are 2l all-to-all communication calls in the So, we need to run micro-benchmarking on computational forward pass. performance and communication latency. Between tow stages a batch of P2P communication happens. Each of the EP GPUs hosting the last layer of the ith stage A. Micro-benchmarking Computation sends 2bsdmodel bytes to it’s counterpart to the first layer of the Every transformer layer has two major components, non(i + 1)st stage. Total message sent between two stages in this expert attention module, and multiple Feed Forward Network stage is 2EP × bsdmodel bytes. Now, under different pipeline (FFN) experts. In an expert-data parallel setting, we need to scheduling, number of stages participating in this concurrent distribute the experts equally across the g gpus in a node and communication varies. replicate the attention module in a data-parallel way across all g gpus. Each GPU needs to host and process one attention C. Finding Valid Parallelization Strategies w/o OOM module and E/g experts. We start with separate optimization of these two parts and identify the suitable kernels optimized for each part. P P × EP = n × g (total GPU count) (7) 1) Attention Performance: For attention part, different EP | E (EP divides expert count) (8) models have different model dimension, and the flash-attention PP ≤ L ( ≥ 1 layer per stage) (9) kernel is optimized for only a fixed set of head-dimension. EP ≤ g · Nh (EP within fast-interconnect domain) We benchmark a single GPU to identify the best performing (10) head-dimension. We need to choose a set of best throughput generation < batch size, head dimension > tuples. Figure 3 Mpeak (0) ≤ CGPU (worst-case stage fits in HBM) (11) shows our measure performances for various MoE model where Nh is the number of nodes sharing a single-hop architectures. interconnect (e.g., Nh = 4 for a Rosetta switch group Attention Module Throughput vs Sequence Length on Frontier). The fourth constraint ensures that all-to-all AMD Instinct MI250X · fp16 · B=4 · 8 MoE Models 120 communication during expert dispatch stays within the fast DeepSeek-V3 intra-group fabric. The fifth uses the stage-0 peak from the 110 Grok-1DeepSeek-V2 DBRX 1F1B model, which is the binding memory constraint across Mixtral-8×22B OLMoE-1B-7B all stages. 100 TFLOPS 100 Mixtral-8×7B
90 Qwen2-57B-A14B
80 70
GQA (6 models blue/green/purple family) MHA / MLA DeepSeek-V2 MHA / MLA DeepSeek-V3
Projdominated
60
512 1K
2K
4K
Sequence Length
8K
Fig. 3: Achievable throughput for various models for different sequence lengths. 2) Expert Performance: Similarly, for the FFN part, which mostly requires GEMM operations, we need to identify the set of (num tokens, batch size, expert dimension) tuples that gives us the best throughput. For the fine-grained experts where GEMM between many tall and skinny matrices are involved, the search becomes more involved (Figure 4). Then, from these two sets, we need to identify the tuples with agreeing batch size. Choosing this batch size becomes a critical choice, especially in the context of pipeline parallelism since choosing the right number of micro batch size, and right value for micro batch size will determine pipeline bubble and computation efficiency.
MoE Expert GEMM Throughput vs Batch Size · Average (Up + Down) / 2 AMD Instinct MI250X · fp16 · 8 MoE Models (c)
Avg TFLOPS (Up + Down) / 2
100
Mixtral-8×22B Mixtral-8×7B
DBRX Grok-1
120
Qwen2-57B-A14B
100 TFLOPS
DeepSeek-V3
DeepSeek-V2 OLMoE-1B-7B
80 60 40 20 0
200
OLMoE-1B-7B Mixtral-8×7B Qwen2-57B-A14B DeepSeek-V2
Memory-latency bottleneck region
400
8001K
2K
4K
8K10K
20K
Total Tokens in Batch (log scale)
Mixtral-8×22B DBRX Grok-1 DeepSeek-V3 40K
80K100K
Fig. 4: MoE GEMM Performance B. Micro-benchmarking Communication
and identifies the three phases of communication along with their dependency structure in order to maximize concurrency across phases. At the intra-node level, it enforces maximum locality by exploiting GPU-to-NIC affinity. At the inter-node level, it leverages the Dragonfly grouping structure by treating the four nodes connected to a common Rosetta switch as a single communication locality domain. Finally, when possible, it constrains node allocation to within the same rack entirely, avoiding the slowest inter-rack communication links. A. Communication Group Construction We assign GPUs sharing a NIC to the same inter-node communicator, ensuring that inter-node traffic from a given GPU is always injected through its affinitized NIC. This prevents NIC contention and allows all four NICs on a node to operate at full bandwidth simultaneously.
In our framework, we incur two types of communication, all-to-all within an expert-parallel group and send-recv between two pipeline stages. Since, between stage send-recv communication happens following a synchronization across the expert parallel group, there are EP concurrent P2P communications between two stages. For benchmarking all-to-all bandwidth, we vary the number of GPUs from 2 to 64 spanning 1-8 nodes, for various message sizes (Figure 5).
Fig. 6: Communication groups for Topological all-to-all
B. Three-Phase Algorithm and Dependency Structure The three phases exhibit the following dependency structure: Phase I ∥ Phase II → Phase III
Fig. 5: Benchmarking All-to-all bandwidth for various message sizes. Bandwidth drops significantly as soon as all-to-all involves inter-node communicaiton C. MFU Estimation Let tcompute and tcomm be the attention + expert and A2A + P2P totals summed over the full step. Then: Fmodel tcompute × , π̂eff · G · tcompute tstep | {z } | {z }
(13)
Phase I (intra-node all-to-all) is fully independent of the internode phases, since all source-destination pairs are known locally and require no prior accumulation. Phase III depends on the completion of Phase II, as it redistributes the received remote data within the node. We exploit the Phase I independence by launching it concurrently with Phase II or Phase III, hiding a significant portion of intra-node communication latency behind inter-node transfers. Figure 7 demonstrates our three phases and their overlapping strategies.
tcompute tcomm = 1−b−C. Rack-Aware Node Allocation tstep tstep When the job scheduler permits, HALO constrains node hardware efficiency compute fraction (12) allocation to within a single rack to eliminate inter-rack traffic, which traverses the slowest links in the Dragonfly topology. V. HALO: H IERARCHICAL A FFINITY- AWARE Within a rack, nodes are further grouped by Rosetta switch L OCALITY-O PTIMIZED A LL - TO -A LL affinity: the four nodes sharing a switch form a single locality Our algorithm pursues five design goals. It saturates all domain, and inter-group communication is batched across this four NICs on a single node during inter-node communication, boundary to minimize contention on inter-switch links. MFU =
Algorithm 1: HALO A LLT OA LL Input: S ∈ RN ×D send buffer, N = num nodes×R communication groups and rank metadata from S ETUP C OMMS Output: O ∈ RN ×D fully transposed output buffer
Phase 1: Intra-node Extraction // Contiguous copy of this node’s R rows 7 L ← S[local start : local start+R] // shape (R, D)
6
8
// Intra-node exchange: rank ℓ receives row ℓ from every local peer P1 ← A LLT OA LL S INGLE(L, internal group)
Phase 2: Inter-node Exchange // Pack all remote-destined rows with a single I NDEX S ELECT kernel 10 idx ← nR, . . . , nR+R−1 n ∈ N− // pre-built LongTensor 9
on GPU 11
Fs2 ← I NDEX S ELECT(S, dim = 0, idx ) // one GPU kernel;
Fig. 7: Three phases of our HALO all-to-all algorithm. Phase I can happen concurrently with either of Phase II and Phase III, but Phase III needs to wait for Phase II to complete. D. Comparison against torch.dist.all to all We compare HALO against torch.dist.all_to_all backed by RCCL across varying node counts and message sizes (Figure 8). HALO achieves 1.1×–9× lower latency for configurations of 16 nodes or more. The crossover at 16 nodes reflects the threshold at which inter-rack communication becomes dominant under the flat RCCL implementation — precisely the regime where HALO’s rack-aware grouping and NIC saturation strategy yield the largest gains. At smaller scales, where all GPUs fit within a single switch group, both algorithms perform comparably since single-hop communication already saturates available bandwidth.
shape (M R, D)
16 nodes 128 GPUs
2.0×
14 15 16
WAITA LL()
1.4× 1.2× 1.0×
4.2
8.4 16.8 33.6 67.1 34.2 68.4 36.9 1 2 5
.5
47
21
Data Size / Rank (MB)
95
.8
.6
16
33
4.2
.1
67
8.4
13
26
6.9
53
8.4
42
5.56×
.6
.1
33
67
4.2
13
8.4
6.9
26
3.44×
3.0× 2.0×
.7
73
53
10
.5
47
21
512 nodes 4096 GPUs
3.54× 3.50×
3.5×
4.36×
4.0×
.8
16
Data Size / Rank (MB)
5.0×
2.0× 1.81×
2.0×
4.0×
6.0×
3.0×
3.0×
95
.7
73
10
256 nodes 2048 GPUs
4.40×
Speedup vs. Reference
8.4
Data Size / Rank (MB)
3.54×
4.0×
1.0× 4.2
42
128 nodes 1024 GPUs
5.0×
.8 16
// Scatter repacked rows to final owners on this node Fr3 ← A LLT OA LL S INGLE(Fs3 , internal group) // rank
3.0×
1.0×
3.03×
3.0× 2.5× 2.0× 1.5× 1.0×
1.0×
19
4.0×
2.0×
Phase 3: Intra-node Redistribution // Transpose Ph2 recv buffer; one GPU kernel — no intermediate allocation 18 Fs3 ← R ESHAPE P ERMUTE V IEW(Fr2 , M, R, D), [1, 0, 2] , [R, M D] // row ℓ = all remote data for local rank ℓ
5.0×
0.6×
4.0×
17
6.0×
0.8×
4.94× 4.67× 4.44×
5.0×
7.0×
1.6×
2.1
// flush all RDMA ops
64 nodes 512 GPUs
8.06× 8.03× 7.92×
8.0×
Speedup vs. Reference
13
32 nodes 256 GPUs
9.0×
1.80× 1.78×
1.8× 1.75×
Speedup vs. Reference
12
// Batched P2P over world group (RDMA); one send+recv slice per remote node for i, n ∈ E NUMERATE(N− ) do peer ← n · R + local rank IR ECV(Fr2 [iR : (i+1)R], src = peer ) IS END(Fs2 [iR : (i+1)R], dst = peer )
Speedup vs. Reference
// Persistent buffers — allocated once on first call, reused thereafter R×D 2 P1 ∈ R // Ph1 recv s r M R×D 3 F2 , F2 ∈ R // Ph2 send/recv flat buffers s r R×M D 4 F3 , F3 ∈ R // Ph3 send/recv flat buffers N ×D 5 O∈R // output
Speedup vs. Reference
local start ← node id · R; N− ← {n ∈ [0, num nodes) : n ̸= node id }; M ← |N− |
Speedup vs. Reference
1
1.0× .6 33
.1 67
1
.2 34
2
.4 68
5
.9 36
Data Size / Rank (MB)
10
.7 73
21
.5 47
9 42
5
0.5× .6 33
.1 67
V0 Sequential V1 Ph1 Ph2 Ph3
1
.2 34
.4 68
2
5
.9 36
.7 73
10
Data Size / Rank (MB) V2 Ph2 Ph1 Ph3
.5 47
21
dist.all_to_all
9 42
5
.1
67
4.2
13
8.4
26
6.9
53
.7
73
10
Data Size / Rank (MB)
.5
47
21
95
42
Baseline
Fig. 8: Latency comparison of Neighborhood-all to all algorithm against the RCCL based torch.dist.all to all algorithm.
ℓ receives its (M D)-wide slice
// Write Phase 1 (local) and Phase 3 (remote) results into output buffer 20 O[local start : local start+R] ← P1 r 21 T ← P ERMUTE V IEW (F3 , R, M, D), [1, 0, 2] // shape (M, R, D)
for i, n ∈ E NUMERATE(N− ) do 23 O[nR : (n+1)R] ← T[i]
22
24
return O
VI. L OAD BALANCING THROUGH E XPERT M IGRATION The load across GPUs emerging from router’s expert selection can vary over training period. Initially, all experts are preferred by the router equally; however small random difference can make some experts more favorable. These favored experts get more tokens, and consequently more gradient updates. This creates a positive feedback loop where a subset of experts become more capable than others, and end up getting more tokens consistently. This causes expert collapse. In the middle part of the training, experts become specialized in different types of input. The load distribution stabilizes, and
different experts develop distinct activation patterns across the TABLE IV: Per-Layer Worst-Case Expert Migration Message feature space. Towards the end, the specialization becomes Size and Latency Per GPU (48 × E × dmodel × dffn /G bytes, G = 8, bandwidth = 50 GB/s) more concrete. Some experts remain underutilized. Unless there is an auxiliary load-balancing loss or other Send Size / Latency Model E/layer dmodel dffn load balancing mechanism in place, expert workload remains GPU (GB) (ms) skewed. So, a static expert parallelism at the beginning of the Switch-Base 128 768 2,048 1.21 24.2 training cannot ensure that all the GPUs within a layer are Mixtral 8×7B 8 4,096 14,336 2.63 52.6 receiving balanced workload. While load balancing loss helps Mixtral 8×22B 8 6,144 16,384 4.50 90.0 with load balancing in the middle stages of the training, it Grok-1 8 6,144 32,768 9.00 180.0 still suffers from prolonged period of load imbalance. Figure 9 GLaM (1.2T) 64 8,192 32,768 102.88 2057.6 shows expert-routing distribution in different layer becomes DeepSeek-V2 160 5,120 1,536 7.04 140.8 balanced after almost one billion token consumption. DeepSeek-V3 256 7,168 2,048 21.00 420.0 Existing MoE training methods are averse to expert reassignment for load balancing due to the perceived overhead of expert migration. Under existing distribution techniques that With average intra-node all-to-all bandwidth 50GB/s, the distribute experts among many nodes through slow Ethernet, worst case expert migration will only take tens of milliseconds this caution is justified. However, when we successfully utilize for most models. However, we will perform this migration pipeline parallelism to localize the experts from the same incrementally and intermittently so that only a subset of the layer to a small number of GPUs connected via fast link experts need to migrate to balance GPU workload. To rebalance (NVLink or Infinity Fabric), we should reconsider the dynamic with minimal swaps, we develop a hill-climbing swapping based load balancing. In the absence of load balancing loss, the algorithm (Algorithm 19). expert demands are still skewed, but load across the GPUs can be balanced. Even with the expert collapse, the final model Algorithm 2: Hill-Climbing Swap-Based Minimal Rebalancing performance does not suffer. Input: Groups G = {G1 , . . . , GK }, max iterations T = 100 Output: Rebalanced groups G, swap count c 1 c ← 0; 2 for t = 1, . . . , T do P 3 sk ← n∈Gk n ∀k; + 4 k ← arg maxk sk ; 5 k− ← arg mink sk ; 6 δ ← sk+ − sk− ; 7 best swap ← None, ∆∗ ← 0; 8 foreach (i, n1 ) in Gk+ do Fig. 9: Expert load distribution during the training process. 9 foreach (j, n2 ) in Gk− do The model has a 350M base model, with 16 experts and expert 10 δ ′ ← |(sk+ − n1 + n2 ) − (sk− − n2 + n1 )|; 11 if δ ′ < δ and (δ − δ ′ ) > ∆∗ then frequency 0.5. 12 ∆∗ ← δ − δ ′ , best swap ← (i, j);
A. Dynamic Expert Migration
13 14
if best swap ̸= None then (i∗ , j ∗ ) ← best swap; Gk+ [i∗ ] ↔ Gk− [j ∗ ]; c ← c + 1; else break;
We extend the router class to maintain token distribution 15 16 throughout the training. An external scheduler (could be 17 intermittent interrupt in the training loop) inspects the growing 18 load imbalance, and whenever it crosses a pre-determined threshold, it can run our migration algorithm to identify 19 return G, c the minimal number of intra-group (expert-parallel group) migrations of experts among GPUs. The cost of migration will be the cost of an all-to-all in the expert parallel group VII. S CALABLE T RAINING OF SOTA M O E M ODELS where individual expert’s migration cost is the latency of moving 48dmodel df f n (3dmodel df f n expert parameters, and each parameter needs 16 Bytes for master copy, optimizer states, We have developed resource modeling and a throughput and gradients). In the worst case, when we do a complete re- estimator through micro-benchmarking of Frontier superassignment from scratch, that would mean we will be moving computer to identify the viable and performant distributed all E experts, and the individual send message size will training strategies. For example to train a model (super model be 48(E/EP )dmodel df f n (3dmodel df f n Bytes. Assuming a with 545B parameters), we computed the required memory single layer fits in a node (8 GPUs) with EP=8, the individual for various node count with different distribution strategies send-message sizes for various SOTA model is in Table IV. (Figure 10).
Total GPU Memory vs. EP Size (External Checkpoint: Attn + MoE) 200 175 150 125 100 75 64 GB HBM limit
50 25
16 bytes/param: 2B fp16 weight + 2B fp16 grad + 4B fp32 master + 4B fp32 m1 + 4B fp32 m2 Activation: ⌈L/pp⌉ ⋅ pp ⋅ 2 b s dm ⋅ 2 B (2 checkpoint boundaries/layer: attn module input + MoE module input)
0
4
8
16
32
64
Expert Parallel Size (EP)
128
256
Fig. 10: Identifying viable training strategies for a model by estimating its per-GPU memory requirement. Here, we investigate training strategy of a 615 B parameter model. And, just like this analysis suggests, we can train this model with at least 64 nodes.
A. Trroughput Ceiling : Training a Single Layer in a Node Every transformer layer has two major components, nonexpert attention module, and multiple Feed Forward Network (FFN) experts. In an expert-data parallel setting, we need to distribute the experts equally across the G gpus in a node and replicate the attention module in a data-parallel way across all G gpus. Each GPU needs to host and process one attention E module and experts. G After getting the suitable configuration, we train a single layer of the SOTA models on a single Frontier node using expert-parallelism and observe their training performance (Figure 11). We were able to fit a single layer to a single node for each of the models we tried. With memory constraint under pipelined expert-parallelism and pipeline bubble under various schedules, this sets up the upper limit of training performance of any model using Piper.
120 100
TFLOPS MFU (%) Act. checkpointing
102.8 53.8%
60 50
80 60
74.2
55.4
38.8%
40
29.0% 46.8 24.5%
40
37.8 19.8%
30 20
20
10
0
0
2 B B ut ick 22 co s ×7 kV er s ee ams 8× ams l 8 ams av am 4 S ram a l S r a r r r r M a p r pa a a e 4 m pa ixt pa ixt De 5B p M 7B p Lla 2B M 54B ma 9B 4 1 10 23 Lla 52
MFU (%)
Memory per GPU (GB)
225
70
32 nodes (256 GPUs) 64 nodes (512 GPUs) 128 nodes (1024 GPUs)
Throughput (TFLOPS)
Total (model + activation) Model only (weights + optimizer)
Fig. 12: Training throughput of SOTA MoE models using Piper (sequence length 4096). Hatching denotes activation checkpointing. the impact on the MFU (Figure 12). From Table 12, we observe that training models with traditional large experts achieves best performance while the fine-grained experts achieve lower performance. C. Comparison Against Other MoE Training Frameworks We compare Piper’s training throughput for different model sizes against state of the art training frameworks such as Tutel, DeepSpeed MoE, DeepSpeed TED, and most prominent XMoE. Since X-MoE is the leading open-source framework for fine-grained experts training, we compare Piper against these tools using fine-grained MoEs. Our framework Piper can train these models with only a fraction of their requirement at 2-3.6X throughput (Figure 13).
MoE Layer Training Throughput Frontier Supercomputer (Single-Node EDP) 140
Throughput (TFLOPS)
Meta Mistral DeepSeek Moonshot Snowflake mean = 102.2
129.4 117.5
120
112.4
109.6
100
104.2
80
84.3
81.7
78.3
DeepSeek V3
Kimi K2
DeepSeek V2
60 40 20 0
Mixtral 8×22B
Mixtral 8×7B
Llama 4 Maverick
Llama 4 Scout
Arctic
Fig. 11: Training efficiency of a single layer of the SOTA models. B. Training the Full Model
Fig. 13: Throughput comparison against SOTA MoE training frameworks. Piper can train the small, medium, large, and super model using 8, 32, 80, and 512 MI250X GPUS instead of 256 and 1024 GPUs used by X-MoE.
With our resource modeling, we identified valid parallelization strategies for various SOTA models and train them using their original size. We selectively use activation checkpointing to fit the model into a small number of GPUs while minimizing
D. Training Trillion(s) Parameter Models We take a dense model [dmodel = 5120, df f n = 20480, L = 32, k = 2] (10 Billion parameters, let’s call it M10B) and
scale out its parameter count by scaling the number of experts (E). We start with 16 experts on 8 nodes and then scale E proportionally. We use 128 experts on 64 nodes (512 GPUs), and 256 experts on 128 nodes (1024 GPUs). With this scaling, we can train a 862 Billion parameter model using 512 GPUs at 39.38 TFLOPs, and a 1.7 Trillion parameter model using 1024 GPUs at 33 TFLOPS. This is a form of weak scaling, and the scaling efficiency is 73% from 64 GPUs to 1024 GPUs (Figure 14). Throughput and Model Size vs. Number of GPUs
60
Throughput (TFLOPs) Ideal throughput (45.15 TFLOPs) 2000 Model Size (B params) 1721 B
1750
E=256, PP=4
40.88
40
1500
39.38
37.52
33.04
1250
30
1000
20
750
862 B
E=128, PP=4
10 0
Model Size (Billion Parameters)
Throughput (TFLOPs)
50 45.15
110 B
64
E=16, PP=4
500
432 B
250
E=64, PP=8
128 110 B
256 Number of GPUs
E=32, PP=4
512
1024
0
Fig. 14: Scaling M10B models with experts. VIII. C ONCLUSION AND D ISCUSSION We presented Piper, a holistic MoE training framework that co-designs distributed training strategy through mathematical resource modeling, empirical micro-benchmarking, and platform-aware performance estimation. Its central contribution is the application of pipeline parallelism on top of traditional expert-data parallelism, which proves critical in mitigating largescale all-to-all latency. Piper achieves 2–3.6× the MFU of XMoE on SOTA models, demonstrating that deep understanding of the target HPC platform translates directly into training throughput gains. Implemented in Python and PyTorch, Piper is portable across platforms, and its micro-benchmarking suite can characterize arbitrary HPC systems. As new MoE architectures emerge, Piper’s modular design allows incremental integration of kernel optimizations, improved schedules, and dynamic load-balancing strategies to deliver continued efficiency gains.
R EFERENCES [1] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020. [Online]. Available: https://doi.org/10.48550/arXiv.2001.08361 [2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems, vol. 30, 2017. [Online]. Available: https://doi.org/10.48550/arXiv.1706.03762 [3] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro et al., “Efficient large-scale language model training on GPU clusters using Megatron-LM,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2021. [Online]. Available: https://doi.org/10.1145/3458817.3476209 [4] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “ZeRO: Memory optimizations toward training trillion parameter models,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2020. [Online]. Available: https://doi.org/10.1109/SC41405.2020.00024 [5] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparselygated mixture-of-experts layer,” in International Conference on Learning Representations (ICLR), 2017. [Online]. Available: https: //doi.org/10.48550/arXiv.1701.06538 [6] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2101.03961 [7] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand et al., “Mixtral of experts,” arXiv preprint arXiv:2401.04088, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.04088 [8] DeepSeek-AI, “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2501.12948 [9] Qwen Team, “Qwen1.5-MoE: Matching 7B model performance with 1/3 activated parameters,” 2024. [Online]. Available: https: //qwenlm.github.io/blog/qwen-moe/ [10] Kimi Team, “Kimi k1.5: Scaling reinforcement learning with LLMs,” arXiv preprint arXiv:2501.12599, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2501.12599 [11] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “GShard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668, 2020. [Online]. Available: https://doi.org/10.48550/arXiv.2006.16668 [12] S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He, “DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation AI scale,” in Proceedings of the 39th International Conference on Machine Learning (ICML), 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2201.05596 [13] D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu et al., “DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models,” arXiv preprint arXiv:2401.06066, 2024. [Online]. Available: https: //doi.org/10.48550/arXiv.2401.06066 [14] Z. Chi, L. Dong, S. Ma, R. Pang, S. Huang, X.-L. Mao, and F. Wei, “X-MoE: Scalable and finetunable sparse mixture-of-experts transformer for on-device inference,” arXiv preprint arXiv:???, 2024, wARNING: original arXiv ID 2405.13089 is incorrect and belongs to an unrelated paper. Correct ID needs to be verified. [15] Microsoft DeepSpeed Team, “DeepSpeed tensor, expert, and data parallelism,” Microsoft, Tech. Rep., 2022. [Online]. Available: https://www.deepspeed.ai/tutorials/mixture-of-experts-inference/ [16] S. Atchley, C. Zimmer, J. Lange, B. Grodowitz, S. Oral et al., “Frontier: Exploring exascale,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2023. [Online]. Available: https://doi.org/10.1145/3581784.3607089 [17] Qwen Team, “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2505.09388 [18] Kimi Team, “Kimi k2: Open agentic intelligence,” arXiv preprint arXiv:2507.20534, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2507.20534
[19] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti et al., “Efficient large-scale language model training on GPU clusters using Megatron-LM,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2021. [Online]. Available: https://doi.org/10.1145/ 3458817.3476209 [20] D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “PipeDream: Generalized pipeline parallelism for DNN training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP), 2019. [Online]. Available: https://doi.org/10.1145/3341301.3359646 [21] D. Narayanan, A. Phanishayee, K. Shi, X. Chen, and M. Zaharia, “Memory-efficient pipeline-parallel DNN training,” in Proceedings of the 38th International Conference on Machine Learning (ICML), 2021. [Online]. Available: https://doi.org/10.48550/arXiv.2006.09503 [22] C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram et al., “Tutel: Adaptive mixture-of-experts at scale,” in Proceedings of Machine Learning and Systems (MLSys), vol. 5, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2206.03382 [23] Y. Huang, Y. Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen, “GPipe: Efficient training of giant neural networks using pipeline parallelism,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019. [Online]. Available: https://doi.org/10.48550/arXiv.1811.06965 [24] P. Qi, X. Wan, G. Huang, and M. Lin, “Zero bubble pipeline parallelism,” arXiv preprint arXiv:2401.10241, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2401.10241 [25] Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon, “Mixture-of-experts with expert choice routing,” in Advances in Neural Information Processing Systems (NeurIPS), 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2202.09368 [26] NVIDIA Corporation, “NVIDIA Collective Communications Library (NCCL),” 2023. [Online]. Available: https://developer.nvidia.com/nccl [27] AMD Inc., “ROCm Collective Communications Library (RCCL),” 2023. [Online]. Available: https://github.com/ROCm/rccl [28] J. He, J. Zhai, T. Antunes, H. Wang, F. Luo, S. Shi, and Q. Li, “FasterMoE: Modeling and optimizing training of large-scale dynamic pretrained models,” in Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), 2022. [Online]. Available: https://doi.org/10.1145/3503221.3508418 [29] X. Nie, P. Zhao, X. Miao, T. Zhao, and B. Cui, “HetuMoE: An efficient trillion-scale mixture-of-expert distributed training system,” arXiv preprint arXiv:2203.14685, 2022. [Online]. Available: https: //doi.org/10.48550/arXiv.2203.14685 [30] J. Kim, W. J. Dally, S. Scott, and D. Abts, “Technology-driven, highly-scalable dragonfly topology,” in Proceedings of the 35th Annual International Symposium on Computer Architecture (ISCA), 2008. [Online]. Available: https://doi.org/10.1109/ISCA.2008.19 [31] S. Williams, A. Waterman, and D. Patterson, “Roofline: An insightful visual performance model for multicore architectures,” Communications of the ACM, vol. 52, no. 4, pp. 65–76, 2009. [Online]. Available: https://doi.org/10.1145/1498765.1498785 [32] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,” in Advances in Neural Information Processing Systems (NeurIPS), 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2205.14135 [33] V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,” in Proceedings of Machine Learning and Systems (MLSys), 2023. [Online]. Available: https://doi.org/10.48550/ arXiv.2205.05198 [34] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts et al., “PaLM: Scaling language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2204.02311