FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training Shuyao Qi
Haoyuan Liu
Shizhen Zhao
Shanghai Jiao Tong University Shanghai, China [email protected]
Shanghai Jiao Tong University Shanghai, China [email protected]
Shanghai Jiao Tong University Shanghai, China [email protected]
arXiv:2604.19654v1 [cs.DC] 21 Apr 2026
Abstract
Modern distributed training employs parallel dimensions (Data Parallelism (DP), Tensor Parallelism (TP), EP, Pipeline Parallelism (PP)), all configured offline assuming deterministic tensor shapes and communication volumes [1, 8, 14, 18]. MoE routing breaks this assumption. As Figure 1(a) shows, per-device token counts fluctuate widely across micro-batches even during stable training, and the imbalance pattern is datadependent: different training corpora and stages produce different routing distributions. Coarse-grained mitigations such as auxiliary balancing losses [2, 5] constrain the router’s expressiveness, degrading model quality. The alternative, accepting the imbalance, leads to significant wasted compute: as Figure 1(b) shows, in synchronous training every device waits for the slowest, and load imbalance wastes on average 18.6% of GPU time per MoE layer. Prior work on dynamic MoE scheduling attempts to address this, but fails to fully hide the overhead of rebalancing. Tutel [4] and SmartMoE [16] switch between predefined parallel configurations, but their strategies partition expert weights across GPUs, always introducing additional communication. FasterMoE [3] replicates hot experts within the EP domain (shadow expert) and pipelines dispatch with computation; however, pipelining and shadow expert splits Grouped GEMM (general matrix multiply) into smaller perexpert kernels, degrading GPU utilization, and its predictionbased scheduling degrades under changing routing patterns. Triton Distributed [19] fuses computation with communication for TP-parallel MoE, but the overlap consumes SM resources, reducing compute efficiency. A deeper compatibility issue is common to all these approaches. In practice, specialized MoE communication libraries such as DeepEP [17] and FUSCO [20] deliver much better performance than All-to-All collectives through custom dispatch protocols. Importantly, these libraries perform bulk token transfers without support for staged delivery. This breaks the staged pipelining assumed by prior overlapbased approaches: splitting communication into stages on top of these backends adds extra volume rather than hiding latency (Section 3.3). We observe that on the NVIDIA Hopper architecture, the NVLink Copy Engine combined with symmetric memory provides a path for intra-node communication that is entirely
Fine-grained, per-micro-batch load balancing is essential for efficient Mixture-of-Experts (MoE) training, yet every prior dynamic scheduling scheme pays for it with extra communication that is hard to hide—especially on modern bulktransfer backends such as DeepEP. We make a simple but consequential observation: on the NVIDIA Hopper architecture the NVLink Copy Engine can move data between intranode GPUs without consuming any SM cycles, effectively providing a nearly free communication channel that runs in parallel with compute kernels. FEPLB turns this idle hardware into a new parallel dimension for MoE load rebalancing. Its Two-Phase Dispatch first routes tokens across nodes via the standard EP backend, then redistributes dynamic-expert tokens and weights within the NVLink domain through the Copy Engine at nearly zero cost, while a lightweight CPU scheduler runs concurrently with static expert computation. Because FEPLB uses only Copy Engine and CPU that are orthogonal to those consumed by EP and PP, it coexists with existing parallel strategies without reconfiguration. On GLM-5’s MoE layers (128 experts, no auxiliary loss, up to 16 H100 GPUs), FEPLB reduces the token straggler by 51– 70% and the GEMM straggler by 50–68% with no measurable EP communication overhead. Its advantage grows with the EP degree: at EP = 8, it achieves 2× lower token straggler than FasterMoE.
Keywords Mixture of Experts; Dynamic Parallelism; Load Balancing; Distributed Training
1
Introduction
Mixture-of-Experts (MoE) has become the dominant architecture for scaling large language models. Models such as DeepSeek-V3 [7], Kimi-K2 [13], and GLM-5 [15] reach trillions of parameters while activating only a small fraction per token. Under Expert Parallelism (EP), each device holds a disjoint subset of experts; the per-device computational load depends on how many tokens the learned router assigns, a quantity that varies randomly per micro-batch. 1
Shuyao Qi, Haoyuan Liu, Shizhen Zhao
Figure 1: (a) Per-GPU token distribution (stacked) across 7,000 training iterations for GLM-5’s MoE layer (128 experts, EP = 8, without auxiliary loss). The varying band widths reveal persistent, random load imbalance across all eight GPUs. (b) Wasted GPU time analysis: the upper panel shows total MoE layer time vs. wasted time (the gap between the slowest and average device); the lower panel shows the efficiency loss ratio. On average, load imbalance wastes 18.6% of GPU time per MoE layer. invisible to GPU computation. Unlike SM-based communication (NCCL, fused kernels) that competes for compute resources, the Copy Engine operates on a dedicated hardware data path and proceeds through NVLink without consuming any SM cycles or interfering with concurrent Grouped GEMM execution. This hardware capability has been mostly unused for MoE load balancing. Building on this observation, we design FEPLB, a system that introduces a new parallel dimension dedicated to dynamic, per-micro-batch load scheduling. FEPLB uses TwoPhase Dispatch: Phase 1 routes tokens to the NVLink domain via EP, with static expert tokens dispatched normally and dynamic expert tokens collected for intra-node processing; Phase 2 redistributes tokens and copies expert weights within the NVLink domain via the Copy Engine at nearly zero cost. A CPU-side load balancer decides per micro-batch which experts to rebalance, running concurrently with static expert computation. Because the Copy Engine, CPU scheduling, and the time during static expert computation are all resources unused by EP and PP, FEPLB achieves orthogonality by construction: it coexists with existing parallel dimensions without reconfiguring parallelism or replacing the EP communication backend. We evaluate FEPLB on the MoE layers of GLM-5 (128 routed experts, without auxiliary loss) across three PP/EP configurations on up to 16 NVIDIA H100 GPUs. Our contributions are:
(1) We propose dynamic load balancing as a new parallel dimension for MoE, orthogonal to EP and PP, and identify the design principle that enables its introduction: resource-level separation from existing parallel domains. (2) We design Two-Phase Dispatch, which separates internode EP routing (Phase 1) from SM-free, NVLink-CEbased intra-node load rebalancing that redistributes both tokens and expert weights (Phase 2), achieving orthogonality by construction. (3) We present a per-micro-batch expert weight redistribution algorithm that runs on the CPU concurrently with static expert computation, with a configurable dynamic expert count (dyn) and minimum token threshold. (4) We demonstrate that FEPLB reduces token straggler by 51–70% and GEMM straggler by 50–68% with no measurable EP communication overhead. Its advantage grows with EP degree, achieving 2× lower token straggler than FasterMoE at EP = 8.
2 Design of FEPLB 2.1 Design Principle: Orthogonal Dynamic Parallelism The central design principle is that FEPLB’s parallel dimension must be orthogonal to EP and PP. Concretely: (1) EP communication must be unaffected, with no additional internode traffic or NIC interference; (2) PP scheduling must be 2
FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training
Table 1: Parallel dimensions in MoE training and their hardware resource usage. FEPLB introduces a new dynamic dimension that uses resources (CPU, NVLink Copy Engine) orthogonal to those used by EP and PP. Dimension
Scope
Communication
Compute
EP PP FEPLB
Inter-node Inter-node Intra-node
RDMA / NVLink RDMA / NVLink NVLink Copy Engine
GPU SMs GPU SMs CPU
Figure 3: FEPLB system architecture. The training loop (top) periodically optimizes expert placement via a Router Predictor. Within each micro-batch (bottom), static experts follow the standard EP path while dynamic experts use Two-Phase Dispatch with SM-free NVLink weight redistribution. Figure 2: Two-Phase Dispatch. Phase 1: EP dispatch routes static expert tokens to assigned devices and collects dynamic expert tokens into the NVLink domain (inter-node, ∼50 GB/s). Phase 2: NVLink Copy Engine redistributes dynamic expert tokens and copies expert weights within the node (SM-free, ∼900 GB/s). No crossnode balancing is performed, preserving EP’s internode communication pattern.
In Phase 2, the NVLink Copy Engine redistributes dynamic expert tokens and copies expert weights from overloaded to underloaded devices within the NVLink domain, consuming zero SM cycles. The Copy Engine operates at 900 GB/s bidirectional on H100 NVLink 4.0, on a hardware path separate from both the inter-node NICs used by EP and the SMs used for Grouped GEMM. FEPLB rebalances only within the NVLink domain, leaving EP’s inter-node communication untouched. Because each token is still processed by the same expert with identical weights, weight redistribution preserves exact MoE semantics. This scope is limited by the current NVLink topology, not by design. On NVIDIA SuperPod architectures (e.g., GB200 NVL72) with all-to-all NVLink across 72 GPUs, Phase 2 can rebalance the entire EP group without cross-node communication.
unchanged, as FEPLB operates within a single MoE layer; (3) GPU SMs must not be consumed, so all rebalancing communication must be SM-free. Table 1 shows how FEPLB achieves this via resource-level separation. EP and PP both use RDMA/NVLink and GPU SMs; FEPLB uses NVLink Copy Engine and CPU. These resource sets do not overlap, making FEPLB a true parallel dimension rather than an ad-hoc scheduling trick.
2.3 2.2
Two-Phase Dispatch
System Architecture and Overlap Strategy
FEPLB operates at two timescales (Figure 3). At the macro level, a Router Predictor periodically optimizes expert-to-device assignment based on historical routing statistics, executed at checkpoint time to spread out migration cost. At the micro level, each micro-batch uses Two-Phase Dispatch for fine-grained rebalancing. Experts on each device are partitioned into static and dynamic categories, controlled by parameter dyn. With 128 experts and EP = 8, each device hosts 16; setting dyn = 4 makes 4 dynamic (eligible for copying) and 12 static (always local).
Two-Phase Dispatch decomposes MoE token processing into two stages (Figure 2). In Phase 1, static expert tokens are dispatched to their assigned devices via the standard EP backend (e.g., DeepEP), following the unmodified EP path. Dynamic expert tokens are collected from the entire EP domain into the corresponding NVLink domain (i.e., the local node) and permuted there, preparing them for intra-node redistribution. After Phase 1, each device within the NVLink domain knows the exact per-expert token counts for dynamic experts, which may be highly unbalanced across devices. 3
Shuyao Qi, Haoyuan Liu, Shizhen Zhao
Table 2: Per-layer MoE execution time (forward / backThe per-micro-batch timeline exploits this partition. First, ward, ms) across PP/EP configurations under five meththe router produces routing decisions on all devices. Phase 1 ods. then dispatches static expert tokens to their assigned devices via the standard EP backend while collecting dynamic expert tokens from the EP domain into the NVLink domain and perPP/EP Before LB FasterMoE Triton Dist. Tutel FEPLB muting them. The GPU then begins computing static experts 4/2 8.2/14.9 7.9/14.0 13.1/22.8 8.0/17.1 7.9/14.4 on SMs; concurrently, the CPU load balancer analyzes the 4/4 7.3/13.2 6.9/12.2 15.3/24.0 7.2/15.2 6.8/12.1 actual token distribution and decides which dynamic experts’ 2/8 6.9/12.5 6.3/11.1 22.8/30.0 6.8/14.5 6.0/10.6 weights to copy, and the NVLink Copy Engine transfers those weights along with the corresponding permuted tokens to Software framework. Our baseline is the Megatron-LM target devices, all SM-free. Once static expert computation MoE framework with DeepEP for dispatch, NVIDIA Transand weight copy both complete, the GPU computes dynamic former Engine for mixed-precision training, and multi-stream experts with rebalanced load. Finally, the Combine phase cuBLAS-based Grouped GEMM. FEPLB is implemented on returns results to source devices. top; all configurations share the same communication and Static experts serve a dual purpose: they contribute to the computation kernels. model’s output, and their computation provides a time window during which the load balancer and weight copy finish Model configuration. We use a reduced-layer variant of without blocking the critical path. The CPU load balancer GLM-5 [15] (18 layers instead of the original 78), preservruns on a dedicated thread, triggered by router completion, ing the original MoE layer architecture (128 routed experts, and issues weight copy commands to the NVLink Copy Entop-𝑘 routing, no auxiliary loss). Since FEPLB operates indegine on dedicated CUDA streams without SM involvement. pendently within each MoE layer, reducing the layer count The load balancer runs a greedy algorithm: it repeatedly does not affect per-layer evaluation. selects the busiest dynamic expert on the most overloaded deWe evaluate three PP/EP configurations covering different vice and copies that expert’s weights (along with its tokens) parallelism trade-offs: PP = 4, EP = 2 (8 GPUs); PP = 4, EP = 4 to the most underloaded device. A minimum token thresh(16 GPUs); PP = 2, EP = 8 (16 GPUs). Each configuration asold 𝜏 prevents copying experts with too few tokens. Each signs 128/EP experts per device (64, 32, or 16 respectively). copy migrates an entire expert rather than splitting its token batch: Grouped GEMM performance is highly sensitive to Baselines. We compare five configurations: (1) Before LB: per-expert batch size under the roofline model, and splitting standard EP with no load balancing; (2) FasterMoE [3]: shadowtokens would produce smaller matrix multiplications in the expert replication with pipe = 1 (no overlap) and pipe = 2 memory-bound regime. Migrating whole experts also makes (pipelined dispatch). For fair comparison, we re-implement the algorithm deterministic: given the same routing deciFasterMoE with SM-free NVLink CE transfers and DeepEP sions, every device independently derives the same weight dispatch. Unless noted, results refer to pipe = 1; (3) Triton copy plan without coordination. The algorithm completes in Distributed [19]: TP-parallel MoE with fused computationapproximately 50 𝜇s on a single CPU core, well within the communication kernels; (4) Tutel [4]: adaptive switching static expert computation window. between EP and DP modes; (5) FEPLB: our system with TwoThe memory overhead is modest: FEPLB allocates max_num_dyn× Phase Dispatch. 𝑊expert per device for copied weights. For GLM-5, each exMetrics. We report two straggler metrics that directly quanpert is 72 MiB; with max_num_dyn = 8, the buffer is 576 MiB tify load imbalance. The token straggler is max𝑑 𝑇𝑑 −𝑇¯ , where (<0.7% of 80 GB HBM3), reused across all MoE layers. 𝑇𝑑 is the per-GPU token count and 𝑇¯ is the mean. It measures the excess token count on the most loaded device. The 3 Evaluation ¯ where 𝐺𝑑 is the per-GPU GEMM straggler is max𝑑 𝐺𝑑 − 𝐺, We evaluate FEPLB on per-layer execution time, EP commuGrouped GEMM execution time. It measures the wall-clock nication overhead, load balance quality across methods and computation time wasted waiting for the slowest device. EP degrees, and sensitivity to the dynamic expert count. Both metrics are averaged over the full training run.
3.1
Experimental Setup
3.2
Hardware. Experiments run on NVIDIA H100 SXM5 GPUs (80 GB HBM3) connected via NVLink 4.0 (900 GB/s bidirectional per GPU) within each node and 400 Gbps InfiniBand between nodes.
Per-Layer Execution Time
Table 2 reports per-MoE-layer execution time. Triton Distributed is 1.6–3.3× slower than the baseline on the forward pass: its fused computation-communication kernels consume SM resources for communication, and the overhead grows 4
FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training
Figure 4: EP communication time (Dispatch and Combine) for Before LB, FasterMoE (pipe = 1 and pipe = 2), and FEPLB. FasterMoE with pipe = 2 adds up to 46.8% dispatch overhead due to additional communication volume on top of DeepEP. FEPLB introduces no measurable overhead (<1%). as more GPUs participate in the collective. Tutel matches or slightly improves forward time but adds 15–16% backward overhead due to additional communication from weight partitioning across GPUs. FEPLB consistently matches or outperforms all baselines. At PP = 2, EP = 8, forward time drops from 6.9 ms to 6.0 ms (−13%) and backward from 12.5 ms to 10.6 ms (−15%). At PP = 4, EP = 2, FEPLB’s forward matches FasterMoE (7.9 ms) but backward is slightly higher (14.4 vs. 14.0 ms): with EP = 2, each device hosts 64 experts, and since FEPLB migrates entire experts to preserve Grouped GEMM efficiency, rebalancing granularity is limited at low EP.
3.3
Figure 5: Token straggler (top) and GEMM straggler (bottom) across three PP/EP configurations for Before LB, FasterMoE, and FEPLB. FEPLB achieves 51–70% token straggler reduction and 50–68% GEMM straggler reduction, increasingly outperforming FasterMoE as EP grows.
Orthogonality Verification: EP Communication Overhead
Table 3: Token straggler (max − mean) across configurations. Percentages show reduction relative to Before LB.
Figure 4 tests whether each approach alters EP communication performance (measured at EP = 8). FasterMoE with pipe = 1 shows negligible overhead, matching FEPLB. However, pipe = 2 incurs 46.8% additional dispatch time and 40.2% additional combine time, validating the observation from Section 1: pipelining requires splitting communication into stages, which adds volume on top of bulk-transfer backends like DeepEP. FEPLB avoids this: Phase 2 operates after Phase 1 on a separate hardware path, with overhead below 1%.
3.4
Load Balance Quality
PP/EP
Before LB
FasterMoE
FEPLB
4/2 4/4 2/8
2,278 4,649 6,666
1,014 (−55%) 2,471 (−47%) 4,036 (−39%)
1,107 (−51%) 1,697 (−63%) 2,021 (−70%)
low EP, though FEPLB already edges ahead on GEMM straggler (50% vs. 46%). As EP increases, FasterMoE’s prediction accuracy degrades under sparser, less predictable routing distributions: its token straggler reduction drops from 55% to 39%. FEPLB’s reactive approach improves in the opposite direction, from 51% to 70%. At EP = 8, FEPLB achieves 2×
Figure 5 compares load balance quality across all PP/EP configurations. FEPLB and FasterMoE exhibit opposite scaling trends. At EP = 2, FasterMoE achieves slightly better token straggler reduction (55% vs. 51%) thanks to relatively stable routing at 5
Shuyao Qi, Haoyuan Liu, Shizhen Zhao
Table 4: GEMM straggler in ms (max − mean) across configurations. Percentages show reduction relative to Before LB. PP/EP
Before LB
FasterMoE
FEPLB
4/2 4/4 2/8
0.316 0.652 1.110
0.170 (−46%) 0.380 (−42%) 0.625 (−44%)
0.157 (−50%) 0.247 (−62%) 0.352 (−68%)
4
Related Work
MoE systems and parallel strategies. GShard [5], Switch Transformer [2], DeepSeek-V3 [7], and GLM-5 [15] scaled MoE to trillions of parameters, while Megatron-LM [12] and DeepSpeed-MoE [9] built distributed training infrastructure. Auto-parallelization frameworks [1, 6, 8, 10, 11, 14, 18] solve for parallel strategies offline under deterministic assumptions violated by MoE routing. FEPLB complements these systems by dynamically reconfiguring within each MoE layer. Dynamic scheduling and communication. Tutel [4] switches between EP and DP modes but partitions weights, adding communication. SmartMoE [16] selects from a pre-computed strategy menu. FasterMoE [3] replicates hot experts; even re-implemented with SM-free NVLink CE and DeepEP, its predictive approach degrades as EP increases (Section 3.4). Triton Distributed [19] fuses TP-parallel MoE with communication but reduces available SMs. Specialized libraries such as DeepEP [17] and FUSCO [20] perform bulk transfers without staged delivery, breaking the pipelining assumed by prior overlap approaches. FEPLB is compatible with these backends and differs by reconfiguring every micro-batch through resource-level separation.
5
We present FEPLB, a system that introduces dynamic load balancing as a new parallel dimension orthogonal to EP and PP via resource-level separation: NVLink Copy Engine for intra-node redistribution, CPU for scheduling, separate from the RDMA NICs and GPU SMs used by existing dimensions. Two-Phase Dispatch separates inter-node EP routing from SM-free intra-node rebalancing, preserving exact MoE semantics and supporting fully auxiliary-loss-free training. On GLM-5’s MoE layer (128 experts, no auxiliary loss, up to 16 H100 GPUs), FEPLB reduces token straggler by 51–70% and GEMM straggler by 50–68% with no measurable EP overhead, achieving 2× lower token straggler than FasterMoE at EP = 8. Current limitations include whole-expert migration without token-level splitting, which limits granularity at low EP. This work shows that the dynamic overheads of conditional computation can be eliminated by using hardware resources that current frameworks leave idle.
Figure 6: Token straggler as a function of the dynamic expert count dyn = 2, 4, 8 across three PP/EP configurations. Higher dyn provides more rebalancing flexibility, with diminishing returns beyond dyn = 4.
lower token straggler (2,021 vs. 4,036) and 1.8× lower GEMM straggler (0.352 vs. 0.625 ms) than FasterMoE.
3.5
Conclusion
Sensitivity to Dynamic Expert Count References
The parameter dyn controls how many experts per device are eligible for dynamic copying. Figure 6 evaluates dyn = 2, 4, 8. Even dyn = 2 achieves substantial token straggler reduction, because the imbalance is usually caused by a few busy experts. Increasing dyn yields diminishing returns: dyn = 2 to 4 adds 1–3 percentage points, and 4 to 8 another 1–3 points. dyn = 4 is a practical default.
[1] Zhenkun Cai, Xiao Yan, Kaihao Ma, Yidi Wu, Yuzhen Huang, James Cheng, Teng Su, and Fan Yu. 2022. TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism. IEEE Transactions on Parallel and Distributed Systems 33, 8 (2022), 1967–1981. doi:10. 1109/TPDS.2021.3132413 [2] William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39. 6
FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training
[3] Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. 2022. Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 120–134. [4] Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. 2023. Tutel: Adaptive mixture-of-experts at scale. Proceedings of Machine Learning and Systems 5 (2023), 269–287. [5] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 (2020). [6] Zongbiao Li, Xiezhao Li, Yinghao Cui, Yijun Chen, Zhixuan Gu, Yuxuan Liu, Wenbo Zhu, Fei Jia, Ke Liu, Qifeng Li, et al. 2024. Automatically Planning Optimal Parallel Strategy for Large Language Models. arXiv preprint arXiv:2501.00254 (2024). [7] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [8] Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. 2022. Galvatron: Efficient transformer training over multiple gpus using automatic parallelism. arXiv preprint arXiv:2211.13878 (2022). [9] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International conference on machine learning. PMLR, 18332–18346. [10] Ruifeng She, Bowen Pang, Kai Li, Zehua Liu, and Tao Zhong. 2025. Automatic Operator-level Parallelism Planning for Distributed Deep Learning–A Mixed-Integer Programming Approach. arXiv preprint arXiv:2503.09357 (2025). [11] Ziji Shi, Le Jiang, Ang Wang, Jie Zhang, Xianyan Jia, Yong Li, Chencan Wu, Jialin Li, and Wei Lin. 2023. TAP: Accelerating large-scale DNN training through tensor automatic parallelisation. arXiv preprint arXiv:2302.00247 (2023). [12] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019). [13] Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. 2025. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 (2025). [14] Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, et al. 2022. Unity: Accelerating {DNN} training through joint optimization of algebraic transformations and parallelization. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 267–284. [15] Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. 2025. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471 (2025). [16] Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, and Jidong Zhai. 2023. {SmartMoE}: Efficiently training {SparselyActivated} models through combining offline and online parallelization. In 2023 USENIX Annual Technical Conference (USENIX ATC 23). 961–975.
[17] Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Jiashi Li, and Liang Zhao. 2025. DeepEP: an efficient expert-parallel communication library. https://github.com/ deepseek-ai/DeepEP. [18] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. 2022. Alpa: Automating inter-and {Intra-Operator} parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 559–578. [19] Size Zheng, Wenlei Bao, Qi Hou, Xuegui Zheng, Jin Fang, Chenhui Huang, Tianqi Li, Haojie Duanmu, Renze Chen, Ruifan Xu, et al. 2025. Triton-distributed: Programming overlapping kernels on distributed ai systems with the triton compiler. arXiv preprint arXiv:2504.19442 (2025). [20] Zhuoran Zhu, Chunyang Zhu, Hao Lin, Xu Fu, Yiming Zhou, Quanlu Zhang, Zhenhua Li, Feng Qian, Chao Yu, Boxun Li, et al. 2025. FUSCO: High-Performance Distributed Data Shuffling via TransformationCommunication Fusion. arXiv preprint arXiv:2512.22036 (2025).
7