Conceptio › Archive › arXiv CS

arXiv CSopen access

MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cloud distributed-computing parallel-computing

distributed computing, parallel computing, cloud

MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production

arXiv:2605.08962v1 [cs.DC] 9 May 2026

Chunyu Xue1,2,∗ , Yangrui Chen1 , Jianyu Jiang1 , Ningxin Zheng1 , Junda Feng1 , Jingji Chen1 , Shixiong Zhao1 , Shen Yan1 , Yi Lin1 , Lei Shi1 , Zanbo Wang1 , Lishu Luo1 , Faming Wu1 , Haibin Lin1 , Xin Liu1 , Yanghua Peng1,† , Quan Chen2,† 1 ∗

ByteDance Seed, 2 Shanghai Jiao Tong University

Work done at ByteDance Seed, † Corresponding authors

Abstract As the foundational component of versatile AI applications, training an multimodal large language model (MLLM) relies on multimodal datasets with dynamic modality mixture proportions and sample length distributions. However, existing MLLM systems remain inefficient under dynamic workloads, due to statically coupled decisions of resource allocation and model parallelization between encoders and the LLM backbone. This paper presents MegaScale-Omni, an industrialgrade MLLM training system tailored for dynamic workload adaption and hyper-scale deployment. MegaScale-Omni is built upon the training scheme of encoder-LLM multiplexing with three key innovations: (1) Decoupled parallelism strategies with long-short sequence parallelism for encoders to process variable-length samples, and full-fledged 5D parallelism for the LLM backbone, both organized under a communication-efficient parallelization layout. (2) Unified encoder-LLM representations for flexible, extensible colocation, and a new paradigm of encoder-LLM joint pipeline with workload resilience. (3) Workload balancing techniques via decentralized grouped reordering in data loaders and adaptive resharding from encoder to LLM ranks. MegaScale-Omni is deployed as the foundation of our in-house large-scale MLLM training tasks with thousands of GPUs. Our experimental results demonstrate 1.27×–7.57× throughput improvement under production-grade dynamic workloads, as compared to four state-of-the-art systems. Date: May 12, 2026 Correspondence:

Quan Chen [email protected]

[email protected],

Introduction

Yanghua

Peng

plications [4, 16, 17, 34, 45], including content generation, embodied intelligence, and visual understanding. Architecturally, an MLLM consists of an LLM backbone together with a set of modality-specific encoders, decoders, and adapters, as depicted in Figure 1(a). The training data of each modality is first processed by its corresponding encoder to generate unified embeddings, then concatenated as the input to the LLM backbone, as shown in Figure 1(b).

Multimodal large language models (MLLMs) have become a foundational component for general-purpose artificial intelligence. By encoding heterogeneous modality data as the input to a large language model (LLM), MLLMs extend the capabilities of text-based models to additional modalities such as image [12], audio [54], and video [31]. These models have achieved notable progress across a range of ap1

Encoder

LLM Backbone

Text samples

Image samples

What is an MLLM? Analyze content in

Decoder Decoder

(a) General MLLM architecture

Image E.

Text Encoder T

Pretraining

Mixed training

LLM

ViT

text (100%)

image (99%)

image (92%)

text-32K (52%) image (41%)

video (1%)

video (8%) video (7%)

text-128K (45%) text-512K (35%) image (55%) image (48%) video (7%)

video (10%)

Process

(b) Mul;modal data process example

(a) Dynamic inter-modality proporAons in mulA-phase training

Figure 1 Multimodal LLM architecture and data processing. CDF

1.0

Unfortunately, the training efficiency of existing MLLM systems remains limited in production environments. From our experience with large-scale, real-world MLLM training, we identify that the main obstacle lies in the dynamic variation of relative

DS-A (image) DS-A (text) DS-B (image) DS-B (text)

0.5

0.0

Token/sec (×1e5)

Encoder

In/Out adapters

Modality raAo

Encoder

Sample length (#token)

(b) Dynamic sample length distribuAon in dataset (DS)

0.4

DS-A + text-512K

DS-B + text-512K 0.6

2.84× gap 0.2

2.13× gap

0.4 0.2

0.0

9:1 7:3 5:5 3:7 1:9

Image-text raAo

0.0

OOM

9:1 7:3 5:5 3:7 1:9

Image-text raAo

Figure 2 Dynamic workloads in hyper-scale MLLM training and resulting throughput degradation in existing systems.

workloads between encoders and the LLM backbone throughout training. This dynamism stems

primarily from two factors: the dynamic modality data ratios and the variability in multimodal input sample lengths.

Current MLLM training systems fall short in handling dynamic workloads, as they statically couple resource allocation and model parallelization between encoders and the LLM backbone. Some frameworks (e.g., Megatron-LM [40]) treat encoders as embedding layers of the LLM, allocating resources and parallelizing the entire MLLM in a unimodal manner using strategies such as pipeline parallelism (§2.1). Consequently, when encoder workloads rise, the first LLM pipeline stage becomes a compute and memory bottleneck, leading to a 2.84× throughput gap and out-ofmemory (OOM) issues as shown in Figure 2(c). Other systems (e.g., DistTrain [55]) statically disaggregate encoders and the LLM backbone as two separate models, each with fixed, distinct resources and parallelism strategies. Such disaggregation suffers from encoding bottlenecks when encoders are assigned excessive workloads, while inducing substantial device bubbles when encoders have insufficient samples. Another recent work, Optimus [13], decouples resource allocation and parallelization assuming static workloads, generating decisions solely from model information. Thus, it optimizes for each specific workload but fails to handle dynamic ones.

Specifically, to attain balanced model capability across modalities, MLLM training is typically divided into multiple phases, each with distinct ratios of modality-specific data [2, 5, 16]. Figure 2(a) presents a multi-phase training recipe for a Vision-Language Model (VLM). The process begins with a pretraining phase, where the LLM and Vision Transformer (ViT) [12] are separately trained on text and image data, respectively. Then, in phase P0, adapters are trained on image and video data while the LLM and ViT remain frozen. In subsequent phases, researchers carefully adjust the ratios of both modalities and task domains to balance the model capability when training the entire MLLM. Some approaches further adapt modality data ratios smoothly throughout these phases, such as every one or a few steps, while the research community continues to explore more dynamic multimodal data mixing strategies. Accordingly, the relative workloads of encoders and the LLM backbone shift continuously as training proceeds. In addition, mixing datasets across modalities and task domains exacerbates the skewness of sample length distribution in training batches. Figure 2(b) reports statistics for two datasets with image-text pairs. As observed, significant disparities exist not only across modalities (e.g., 1K/3.8K for average text/image lengths) but also across datasets of the same modality (e.g., 2.71× for image length). When mixing audio and text datasets, the average difference reaches up to 17.6× (Figure 5). Such disparities lead to workload imbalance for encoders either across modalities or within the same modality, ultimately degrading the training efficiency.

Given the above limitations, an efficient system should decouple resource allocation and model parallelization, while adapting to dynamic workloads. Specifically, by employing decoupled parallelism strategies, we can parallelize encoders with higher concurrency and more resources, i.e., with larger data parallelism (DP) degrees across all GPUs, to avoid bottlenecking LLM execution. For the LLM, evidently, the ideal choice should be inheriting all full-fledged parallelization techniques from unimodal training [9, 33, 36]. Meanwhile, by colocating encoders and the LLM across GPUs, we can temporally 2

orchestrate their execution and reactively adapt their elapsed times in response to workload shifts. These design considerations constitute the key insight of this paper, which we refer to as encoder-LLM multiplexing.

sive experiments demonstrate that MegaScale-Omni improves training throughput by 1.27×–7.57× under dynamic multimodal workloads across various model and hardware scales, compared to four stateof-the-art baselines. We also share our practical operational experience of deploying MegaScale-Omni at hyper-scale, including performance analysis, MFU optimizations, and solutions to engineering failures.

On top of this training scheme, we build MegaScaleOmni, a MLLM training system for dynamic workload adaption and hyper-scale deployment. MegaScaleOmni consists of three key innovations to train a production-grade MLLM with high efficiency, scalability, and engineering flexibility. ▷ First, MegaScale-Omni decouples parallelization between encoders and the LLM backbone with tailored strategies for variable-length, long-context workloads and hyper-scale deployment. For encoders, MegaScale-Omni proposes long-short sequence parallelism, which unifies DP, Ulysses SP, and ZeRO2/3 to process variable-length samples simultaneously. For the LLM, MegaScale-Omni incorporates full-fledged 5D parallelism with a communication-efficient layout to mitigate execution interference.

Background and Motivation

2.1

Industrial MultiModal Training

Multimodal large language model (MLLM) has been widely trained and deployed as the foundation of AI applications, such as GPT-4o [34], Qwen2.5-VL [5], and Seedream-3.0 [14]. Scenarios. Our production environments involve three typical MLLM training scenarios. (1) VisionLanguage [7] integrates visual encoders, such as ViT [12] for images and VideoSwin [31] for videos, with the LLM backbone to generate textual answers. (2) Visual Understanding and Generation [29] comprehends visual information and synthesizes images from textual prompts via Variational Auto Encoder (VAE) [24]. (3) Triple-Modality [54] unifies visual, auditory, and textual modalities using Universal Speech Model (USM, for understanding) [54] and Discrete Speech Encoder (for generation) [44] to encode auditory inputs.

▷ Second, MegaScale-Omni proposes unified encoderLLM representations to enable flexible, extensible colocation across GPU ranks. This is motivated by the distinct parallelism strategies and intricate data dependencies between encoders and the LLM, which prevent simply wrapping encoders as LLM layers or nestedly inserting encoder operations into LLM codes in codebases. The ideas behind is to abstract encoders as anchors to the LLM pipeline, while encapsulating complex parallelization and communication details. Building on representations, MegaScale-Omni proposes a new paradigm of encoder-LLM joint pipeline, featuring uniform, on-demand encoder insertion to maintain structural stability under dynamic workloads.

Parallelism Strategies. Large-scale MLLM training involves multiple parallelism strategies. (1) Data parallelism (DP) partitions the global training batch across multiple GPUs, with each replicating model parameters and optimizer states. Zero Redundancy Optimizer (ZeRO) [37, 57] extends DP by distributing model states across GPUs to reduce per-device memory footprint. (2) Tensor parallelism (TP) [40] splits layer parameters along hidden dimension across GPUs for memory and computation reduction. (3) Pipeline parallelism (PP) [32, 33] groups layers into stages and splits the global batch into multiple microbatches, pipelining stage execution to improve end-toend throughput. (4) Sequence parallelism (SP) splits inputs and activations across GPUs along sequence dimension to reduce per-device memory footprint and computation. It consists of three major variants, including Megatron SP [25], Ulysses SP [20], and context parallelism (CP) [3]. (5) Expert parallelism (EP) is tailored for Mixture-of-Expert (MoE) models [38], partitioning experts across GPUs for memory reduction and parallel computation.

▷ Lastly, MegaScale-Omni proposes multiple techniques to optimize workload balance under multimodal data and hyper-scale parallelization. For skewed dataset distributions, MegaScale-Omni employs grouped reordering in decentralized data loaders to balance workloads across encoders. When resharding multimodal embeddings from encoder to LLM ranks, MegaScale-Omni balances workloads across LLM ranks and mitigates communication stragglers via adaptive sample sharding and symmetric dispatching. MegaScale-Omni has been deployed as the foundation for most large-scale MLLM training tasks in our company, scaling to thousands of GPUs. Exten3

(a) Training data

More imbalance or padding

(b) Intui0ve packing

text512K image (35%) (55%)

video (10%)

Token frac+on Token frac+on

Training process

text32K image video (52%) (41%) (7%)

30% 23%20% 11%9%

text-32K 47% 25% 13% 4% 3%

text-512K

1.0 OpenI. (image) OpenI. (text) RefCO. (image) RefCO. (text)

0.5

Uniform max length

0.0

code zh-en 32% 21% subject 13%11% reason. 6% inﬂuen.

image + video book-L synth.L 42% code-S 18% 12% 9% 7% zh-en-S subj.-S

Number of tokens

Figure 3 MLLM mixed training and hybrid packing. Modality ra+o

(b) Audio datasets

1.0

CDF

Image samples

(a) Image datasets

Mixed-modality sequence ✕

Global batch

Text samples

Global batch

Single-modality sequence

0.5

0.5 LibriSpeech GigaSpeech

0.0

400

800 1200 1600

Number of tokens

BytedLong BytedShort 0.0

10K

20K 30K 40K

Number of tokens

Figure 5 Data distributions of image (OpenImages [26], RefCOCOg [53]), audio (LibriSpeech [35], GigaSpeech [52]), and text (BytedLong, BytedOCR, both productional datasets) modalities, measured with encoded sample length.

cap4on ground. OCR video interl.

cap4on OCR video ground. educate.

steps, which prevents static workload optimization within a single phase. As an example, one of our triple-modality training tasks initializes the imagetext mixture ratio to 1 : 1 for the first 10B tokens, then gradually increasing the audio modality ratio to reach an image-audio-text mixture of 13 : 74 : 13.

image + video

Figure 4 Dynamic proportions across modalities and top-5 tasks in training recipe. -L/-S denote long/shortcontext.

Dynamic Data Distributions. Mixing datasets across modalities and task domains exacerbates the skewness of sample length distribution, as illustrated in Figure 5. For instance, the average sample length of OpenImages [26] is 3.8K, while those of LibriSpeech [35] and BytedLong are 0.34K and 6K, respectively. Even within the same modality, RefCOCOg [53] has an average length of 1.4K, which is 2.71× shorter than OpenImages. Such disparities in sample length lead to workload imbalance across encoders, both between different modalities and within the same modality. Specifically, data loaders retrieve samples based on a fixed dataset ratio, such as a 1 : 1 ratio in the number of image and text samples. By ignoring the skewed data distribution, some encoders may be assigned excessively long samples, while others process only short ones.

Mixed Training. Modern large-scale MLLM training constructs data batches by mixing multi-source datasets across diverse modalities and task domains (Figure 3(a)). A common intuition from unimodal training is to pack samples of the same modality into sequences and batch those of different modalities, as shown in Figure 3(b). This introduces heavy imbalance or substantial padding between sequences due to dynamic relative workloads across modalities (§2.2). For instance, text samples with longer lengths offer less packing opportunities than images. Therefore, industrial practice employs hybrid packing, allowing packing samples across modalities for better alignment. As depicted in Figure 3(c), samples of different modalities are packed into sequences of roughly uniform length, then batched into a global batch.

2.2 Characterizing Dynamic Workloads in Production

Dynamism of Modality Computation. Computational heterogeneity across different modalities further amplifies the impact from dynamic multimodal datasets. We illustrate this using triple-modality training as an example, with a sequence length of 16K, batch size of 32, and image-audio-text ratio of 4 : 4 : 2. ViT-1B [12], USM-2B [54], and LLaMA12B [41] consume 14.7K, 4.54K, and 29.0K TFLOPs, respectively, as they comprise distinct computation operations (e.g., convolutional layers in USM). This heterogeneity further amplifies the variance in computation time of different modalities, given that audio samples typically have shorter average lengths than image samples as illustrated above. Moreover, some modality-specific operations also introduce workload dynamism. For example, ViT processes images with dynamic resolutions [11], generating variable-length

Dynamic Inter-Modality Ratios. Figure 4 shows one of our VLM training tasks with text, image, and video datasets. Researchers design a multiphase training recipe to schedule dataset processing order [5, 8, 16], adjusting the ratio of modalities in each phase for distinct objectives, such as longcontext enhancement with 35% text-512K, 55% image, and 10% video samples. Different task domains of the same modality also exhibit varying proportions (e.g., 47% book v.s. 13% code texts). This further shifts the relative workloads across modalities due to variable-length samples (e.g., book-L yields more training tokens than code-S ). Moreover, some approaches smoothly adjust modality ratios throughout training phases, such as at every one or a few 4

Users

tokens after patchifying via a fixed down-sampling strategy (e.g., 3 × 14 × 14 pixels per token).

2.3 Encoder-LLM Multiplexing under Dynamic Workloads 2.3.1

Long-Short Encoders Seq Parallel

Parallel Conﬁg

Full-Fledged LLM 5D Parallel

MM Datasets

Workload Balancer (§5) Data Loading (§5.1) Grouped Reordering

Data Reshard (§5.2)

Why Existing Systems are Inefficient?

Current MLLM training systems fall into two major categories. The first is the unimodal-like category (e.g., Megatron-LM [40], Transformers [49]), which prepends encoders to the first LLM pipeline stage to replace text embedding layers. Here, the entire MLLM is parallelized like a unimodal text-based LLM, i.e., sharing the same resources and parallelism strategies between encoders and the first stage. While intuitive, this unimodal-like system suffers from heavy compute and memory bottlenecks at the first stage when encoder workloads increase (e.g., occupying up to 30% MLLM TFLOPs from model profiles). This degrades training efficiency with up to 2.84× throughput drop and OOM issues (Figure 2(c)).

Coloca4on Representa4on (§4.2) EncodeAnchor

Adap4ve Sharding

Joint Orchestra4on (§4.3)

Symmetric Dispatch

WL-Resilient Pipeline

Produc4onal GPU Cluster

Figure 6 System architecture of MegaScale-Omni.

across all GPU ranks, to avoid bottlenecking the LLM pipeline. For the LLM, the ideal way is to inherit full-fledged parallelization techniques from unimodal training, such as virtual pipeline [33] and efficient operators [9, 10], to boost LLM-part efficiency. ❷ Colocate encoders and the LLM backbone on shared resources for workload-resilient joint orchestration. Specifically, by deploying both the encoder and LLM model states across all available GPU ranks, it is possible to temporally schedule the fine-grained execution of encoder and LLM computations. In response to relative workload shifts, the proportion of their elapsed times could be reactively and dynamically adapted, thereby avoiding device idling caused by insufficient encoder samples or LLM pipeline bubbles.

The second category is disaggregation (e.g., DistMM [18], DistTrain [55]), which statically disaggregates encoders and the LLM backbone as two separate models, each with distinct resources and parallelism strategies. However, as the relative workloads between encoders and the LLM shift, such static disaggregation undergoes either encoding bottlenecks that induce LLM pipeline bubbles (when encoders handle workloads exceeding their allocated resources), or prolonged device idling (when encoders lack sufficient samples).

System Overview

MegaScale-Omni is a production-grade MLLM training system designed for dynamic workload adaption while scaling up to thousands of GPUs. Building on the encoder-LLM multiplexing scheme as presented in §2.3.2, it addresses three major challenges to achieve efficient system design.

Another recent MLLM system, Optimus [13], prevents coupling resource allocation and parallelization decisions for encoders and the LLM, yet parallelizing models based on static, synthetic workloads instead of productional dynamic ones. It synthetizes image-only batches with fixed sizes and statically inserts encoder computations into LLM bubbles. As workloads shift, the optimality of Optimus is broken. 2.3.2

MLLM Model

Model Mul4plexer (§4) Decoupled Paralleliza4on (§4.1)

▷ Efficient decoupled encoder-LLM parallelization under long context and hyper-scale. MegaScaleOmni ensures no interference from encoder parallelization to full-fledged parallelization of the LLM backbone. Specifically, it avoids bottlenecking the LLM pipeline or complicating pipeline schedules with complex data dependencies. Moreover, it addresses the inefficiency of static encoder parallelization when handling variable-length samples. For instance, using sequence parallelism (SP) is beneficial for processing long-context samples within memory limits; for short-context samples, however, sharding with SP is prone to GPU underutilization and throughput drops.

Opportunities.

Given limitations of existing systems, we propose a new MLLM training scheme — encoder-LLM multiplexing, for industrial MLLM training with workload resilience. The key ideas of this scheme include: ❶ Decouple MLLM parallelization and employ distinct parallelism strategies for encoders and the LLM backbone. Specifically, we find that the encoders should be parallelized with high concurrency and enough resources, such as with data parallelism (DP)

▷ Flexible encoder-LLM colocation and joint 5

DP-2

DP-3

Heavy SP-1

Samples

Node DP state G1 G2

State shiC

DP-1

GPU execu,on

Timeline

Sta,c DP-SP node par,,on

(a) Spa,al dividing DP/SP nodes

…

(b) Temporal shiCing between DP/SP

LD (Long)

CPU LD (Short)

GPU

AllG.

QKV

A<n

SP state all-to-all

Distributed data loader Global reorder

addresses the impracticality of colocating encoders and the LLM in production environments. With distinct parallelism strategies and intricate data dependencies, intuitively treating encoders as LLM layers and wrapping with unified parallelization modules (e.g., PyTorch DDP [27]) is infeasible. Nestedly inserting encoder operations into LLM pipeline codes is also not allowed, which are shared by multiple projects in industrial codebases. Moreover, MegaScale-Omni proposes principled orchestration that retains structural stability under dynamic workloads, given that naive strategies like aggresively inserting encoders into LLM bubbles lead to increased bubbles as workloads shift.

Local reorder

workload-resilient orchestration. MegaScale-Omni

…

LD (Short) …

AllG.

QKV

DP state

All2All

A<n

All2All

…

SP state

Figure 7 Long-short sequence parallelism for encoders. Each node maintains an instance of parallelized encoders.

▷ Efficient encoder-LLM workload balancing during data loading and resharding. MegaScaleOmni enhances workload balance across encoders when loading multimodal data. Deploying encoders across all ranks exacerbates workload imbalance under skewed data distribution, hindering intuitive solutions like global sample reordering due to substantial communication. Additionally, MegaScaleOmni balances computation across LLM ranks and communication when resharding data from encoder to LLM ranks, as naive strategies such as uniform sequence sharding cause compute and communication stragglers.

Model Parallelization

In this section, we describe the detailed system design of MegaScale-Omni built on encoder-LLM multiplexing scheme.

4.1

Decoupled Parallelization

4.1.1 Long-Short Sequence Parallelism for Encoder We first introduce the parallelization of modalityspecific encoders and detail the underlying rationale. Long-Short Sequence Parallelism. To efficiently process variable-length samples, MegaScaleOmni proposes long-short sequence parallelism (LSSP), which unifies DP, Ulysses SP, and ZeRO2/3 strategies for encoder parallelization. As shown in Figure 7(a), an intuitive strategy for unification is to spatially dividing DP and SP nodes. This method statically partitions nodes into two disjoint groups, where each group exclusively parallelizes encoders using either DP (for short samples) or Ulysses SP (for long samples). When encoder workloads shift, such as an increase in long samples due to an elevated proportion of long-context datasets, significant workload imbalance arises between DP and SP nodes.

Architecture. Figure 6 depicts the MegaScaleOmni architecture with two main components: Model Multiplexer and Workload Balancer. As the core component, the multiplexer decouples encoder-LLM parallelization by employing long-short sequence parallelism for encoders and full-fledged 5D parallelism for the LLM backbone (§4.1). To flexibly colocate encoders and the LLM across GPU ranks, it provides unified model representations by abstracting encoders as anchors to the LLM pipeline codes (§4.2). Leveraging the representations, the multiplexer jointly orchestrates encoder-LLM execution through a workload-resilient pipeline with uniform, on-demand encoder insertion (§4.3). To alleviate workload imbalance caused by dynamic multimodal workloads, the balancer enhances decentralized data loaders with grouped reordering and zero redundancy filtering techniques (§5.1). During data transfer between encoder and LLM ranks, the balancer reshards embeddings and gradients through adaptive sample sharding and symmetric dispatching (§5.2).

Instead, MegaScale-Omni designs LSSP to temporally shift node states between DP and Ulysses SP strategies. As depicted in Figure 7(b), for each microbatch, each node locally reorders samples by length and divides them by a length threshold η. Samples not exceeding η are encoded in the DP state, while those longer than η are encoded in the SP state with additional all-to-all communication. Since DP and Ulysses SP both replicate encoder parameters across ranks (or sharding with ZeRO2/3), state shifting introduces no additional model resharding overhead. Figure 7(c) shows the execution breakdown of LSSP. After loading short and long samples 6

DP=2

PP=4

Model ParallelizaHon GPU Cluster

28 24 25 26 27

Parameters or Sequences Encoders

DP=8

…

20 16 17 18 19

…

0 1 2 3

12 8 9 10 11

…

TP=2 CP=2

LLM 4

…

asynchronously, LSSP all-gathers layer parameters (e.g., QKV weights), computes outputs (e.g., QKV projection), and shifts to the next state. Elapsed times of the DP and SP states are reactively adjusted in response to dynamic workloads. This method efficiently balances workloads across GPU ranks without cross-node sample reordering.

ZeRO=4 SP=4 Colocated LLM stage Encoders

Modality1

DP=8

Modality2

i Rank i DP group

Parallelism Selection Rationale. The principles of encoder parallelization include high concurrency and no interference to the LLM execution. Therefore, data parallelism (DP) emerges as the first-class citizen because it boosts encoding concurrency without intensive communication. For long-context samples, we adopt Ulysses sequence parallelism (SP) [20] alongside DP to shard patchified tokens1 across ranks within each DP group. This stems from two major reasons. First, for long-context samples (e.g., 512K) and medium-sized encoders (e.g., 1B parameters), activations dominate the overall memory consumption. Compared to TP that replicates activations across GPU ranks, Ulysses SP shards them along sequence dimension with SP × memory reduction (SP is the Ulysses SP degree). Second, Ulysses SP shows better compatibility and workload balance than other SP variants. Compared to Megatron SP, which is tightly integrated with TP, Ulysses SP enables flexible integration with other parallelism strategies (e.g., ZeRO3). Another SP variant, CP, is prone to workload imbalance due to causal attention masking [48]. Conversely, Ulysses SP preserves full attention per rank with optimal balance by sharding attention along head dimension. When scaling to larger encoders, we employ ZeRO2/3 to distribute encoder states and parameters, as it is well compatible to both DP and Ulysses SP [20]. To prevent P2P deadlock risks in hyper-scale deployment and support complex LLM pipelines [33, 36], we do not adopt PP for encoder parallelization, as it complicates encoder-LLM data dependencies and joint pipeline schedules [13]. 4.1.2

Replicate

Heavy intra-node comm

Light inter-node comm

4-GPU node Node rack

Figure 8 The layout of encoder-LLM parallelization.

reduces local batch size while introducing constant communication overhead of model parameters. As a result, when scaling to thousands of GPUs, FSDP is prone to communication bottleneck and computation underutilization. For sequence parallelism, we mainly employ Ulysses for workload balance and ease of use; for overlong contexts (e.g., >512K), CP is selected because scaling Ulysses SP is limited by the number of attention heads (each rank requires at least one head [20]). Communication-Efficient Layout. To eliminate communication congestion between encoders and the LLM backbone, for each modality, MegaScaleOmni exclusively colocates an encoder with each LLM pipeline stage, as shown in Figure 8. In this layout, communication-intensive groups such as encoder SP and LLM TP are confined intra nodes to exploit highbandwidth links, while being temporally scheduled to avoid congestion. Inter-node communication, such as P2P operations between encoders and the LLM backbone, are jointly scheduled based on their data dependencies and pipeline schedules. Notably, LLM stages share a homogeneous encoder assignment, e.g., all having two encoders of Modality1 and Modality2, rather than assigning Modality1 to P P 0 and P P 1 while Modality2 to P P 2 and P P 3. This is for workload balancing of encoders across LLM stages, given the computational heterogeneity of different modalities.

Encoder-LLM Parallelization Layout

We then introduce the parallelization layout of encoders and the LLM backbone.

4.2

Full-Fledged 5D Parallelism for LLM. To exploit highly optimized training techniques [9, 33], MegaScale-Omni inherents full-fledged 5D parallelism for the LLM backbone, including DP (ZeRO1), TP, PP (with diverse schedules), SP, and EP (for MoE). Fully Sharded Data Parallel (FSDP) [57] is not selected as an alternative to TP or PP, as it further

Colocation Representation

In MegaScale-Omni, modality encoders are colocated with pipeline stages of the LLM backbone to harness resource multiplexing. The basic idea is to flexibly insert encoders into LLM pipeline schedules, which reduces engineering complexity compared to from-scratch re-implementation, generalizes to diverse pipelines (e.g., 1F1B [32]), and naturally reuses existing LLM techniques. Despite the benefits, encoder insertion is non-trivial in industrial practice.

1 Samples are segmented into discrete patches, each as a token [12].

def llm_fw_bw_pipeline_1f1b(step_func): pp = get_pipeline_rank() # SPMD-style # 1. Warmup phase for i in range(num_warmups): in = recv(src=pp-1) if pp > 0 else data_iter.next() out = forward_step(step_func, in, i) Inserted ✕ send(out, dst=pp+1) if pp < P-1 else pass # 2. Steady phase in = recv(src=pp-1) if pp > 0 else data_iter.next() Hooked for i in range(num_remains): out = forward_step(step_func, in, i+num_warmups) out_grad = send_recv(out, pp+1) if pp < P-1 else pass in_grad = backward_step(prev_in, prev_out, out_grad) in = send_recv(in_grad, pp-1) if pp > 0 else d.next() # 3. Drain phase for i in range(num_warmups): out_grad = recv(src=pp+1) if pp < P-1 else pass in_grad = backward_step(prev_in, prev_out, out_grad) send(in_grad, dst=pp-1) if pp > 0 else pass

✓

Warmup

Steady

Drain

PP0 1 2 3 4 5 Encoder 1 5 2 6 3 7 4 8 5

PP1

123

1 4 2 5 3 6 4 7 5 8 6

PP2

1 3 2 4 3 5 4 6 5 7 6 8 7

PP3

7 8

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 LLM fwd

LLM bwd

Microbatch index

(a) LLM pipeline implementa,on and schedule

if (condition_1(pp, i)): in = load_data(pp, i) out = self.encoders[0].fwd(in) send_act_to_pp0(out) elif (condition_2(pp, i)): out_grad = recv_grad_from_pp0() self.encoders[1].bwd(out_grad) else: ... # Other code segments

and maximum sequence length. Below is an example with two encoders and ZeRO3 enabled: EncoderAnchor ([ ViT , USM ] , cfg ( zero3 = True ) )

(b) Nested encoder inser,on class EncoderAnchor: def __init__(self, enc_cls, *cfg): # Parallelized w. LSSP self.encoders = [enc_cls(*cfg)] def hook(self, llm_stage, on_fwd): def __hook(): pp, mb_idx = get_position() pass,i = pp_schedule[pp,mb_idx] if pass == ‘fwd’: in = load_balanced_data(…) out = self.encoders[i].fwd(in) async_send(out,dst=0) elif pass == ‘bwd’: o_grad = async_recv(src=0)

Instead of intrusively inserting into LLM codes, engineers only need to hook the anchor to the LLM stage in their customized step_func(). For example, to insert 5-8th and 9-12th encoder microbatches before the 5th and 6th LLM forward microbatches across PP0-3 (green dashed blocks in Figure 9(a)), the corresponding anchor is registered as follows (green dashed line in the figure):

self.encoders[i].bwd(o_grad) llm_stage.register(__hook,on_fwd)

Figure 9 Illustration of native LLM pipeline codes, nested encoder insertion, and unified encoder representation.

anchor . hook ( llm_stage , True ) # step_func ()

Then, the data flow of encoder and LLM microbatches is defined in a JSON-like format:

Strawman: Nested Code Insertion. A natural way for colocation is to insert encoder operations into specified positions of LLM codes in a nested manner. However, LLM pipeline follows a “single program, multiple data” (SPMD) paradigm with complex schedules [39, 40], as shown in Figure 9(a). Each PP rank concurrently executes the same function (e.g., llm_fw_bw_pipeline_1f1b()) yet with different data. To precisely insert encoders, additional information including “(PP rank, microbatch index)” is needed for positioning, such as condition_1(pp, i) in Figure 9(b). This nested insertion makes encoderLLM codes hard to develop, maintain, and optimize. Worse still, industrial codebases share the same training infrastructure across multiple projects, hindering intrusive code modifications to LLM codes.

{5: (0 , [ -1 ,5]) , 6: (1 , [ -2 ,5]) , ...}

The key denotes the encoder microbatch index, while the value (pp, [left,right]) denotes the insertion before right LLM microbatch (after left microbatch) on pipeline rank pp. Negative values denote backward microbatches. At runtime, the pp_schedule is exposed to anchors to instruct the next encoder microbatch. The data flow must satisfy the data dependencies between encoders and the LLM, such as positioning 5th encoder microbatch prior to that of the LLM.

4.3

Unified Encoder Representation. To address the above issues, we present an unified encoder representation that expresses all encoder insertion operations while remaining non-intrusive to LLM codes. The key idea is to decouple encoder positioning within LLM codes from their data flow. For example, in Figure 9(a), the encoder computation (red block) is “positioned” below the LLM forward_step() in warmup phase (red dashed line). From the data perspective, this computation processes 5th microbatch between LLM 4th forward and 1th backward microbatches (red arrow).

Joint Orchestration

Uniform Pipeline with Workload Resilience. Despite the flexibility enabled by the aforementioned representation, aggressively inserting encoders to the LLM pipeline would assign more microbatches to the later LLM stages, as shown in Figure 10(a). When encoder workloads increase, this non-uniform insertion disrupts the structure of the original LLM pipeline. Specifically, when the encoding time of the first stage increases by ∆t, the last stage undergoes a m delay of (N−1 /N0m )∆t, where Nim denotes the number of microbatches for stage i. As a result, LLM microbatches, which were closely fitted in the original pipeline, become heavily misaligned (red arrows in Figure 10(a)), inducing 1.63× bubbles and P2P deadlock risk [22, 50].

As outlined in Figure 9(c), MegaScale-Omni represents encoders as an anchor to the LLM pipeline schedule. The EncoderAnchor is instantiated by specifying: (1) modality encoder classes to be colocated on this PP rank, (2) parallelism configurations such as whether to enable ZeRO2/3 or activation offloading, and (3) data configurations such as patch size

To achieve workload resilience, we propose uniformly inserting encoders across LLM stages as illustrated in Figure 10(b). Specifically, we find that LLM stage latencies remain relatively stable under workload shifts 8

1 PP0 1 1 2 3 1 2 3 2 … PP1 2 3 1 2 3 1 2 1 3 2 … PP2 1 2 3 1 2 3 1 1 2 2 3 3 … Encoder expand 1.63× bubbles PP0 1 1 2 3 1 2 3 1 2 … PP1 2 3 12312 1 3 2 … PP2 1 2 1 2 3 1 2 3 1 2 3 3 … Non-uniform scaling

Async send (embeds, DP→CP) PP0 1

1 1 234123 4

PP1 2

PP2 3

PP3 4

On-demand inser7on 5

5 5 2 6 3 7 4 8 1 5 2 6 3 7 4

Async recv (grads, CP→DP) 1

1 8 5

9 9

Encoder1 Encoder2

1 2 3 4 1 2 3 Stable 123412

1 4 2

1 3 2 4 3

6 5 3 6 4 7 1 8 2 5 3 6 4 7 5 7 5 4 6 1 7 2 8 3 5 4 6 5 7 6

123 41 1 2 2 3 3 4 4

Uniform scaling

(a) Aggresive inser7on in dynamic workloads

8 5 1 6 2 7 3 8 4 5 5 6 6 7 7 Loop body

2 8 6 10 10 9 3

3 8 7 11 11 9 3

4 8 8 12 12 9

LLM stages Balance group Stable to loads Scaled to loads

(b) Workload-resilient, interleaved-1F1B pipeline

Figure 10 Illustration of workload-resilient encoder-LLM pipeline schedules under dynamic workloads.

5.1

(blue dashed block in the figure), owning to the uniform sequence length used in hybrid packing. Conversely, given workload balance across encoders, their latencies scale uniformly up or down under shifting workloads (red dashed block), retaining structural stability to avoid interfering with LLM execution.

Decentralized Grouped Reordering. Prior works such as DistTrain [55] and DynaPipe [22] rely on global reordering to balance training samples across microbatches, since they adopt centralized data loaders on the first pipeline stage. In contrast, MegaScale-Omni distributes encoders across all pipeline stages (i.e., all GPU ranks), making centralized data loader prone to limited storage read concurrency and high overhead from large-scale dispatching to all GPU ranks. To address this, we employ decentralized data loaders to enhance data loading concurrency, in which each worker reads different parts of data simultaneously. This design also alleviates the host memory contention with activation offloading. However, global reordering becomes infeasible due to the costly cross-node and even cross-rack communication, as well as the risk of all-to-all hangs.

On-Demand Encoder Insertion. To uniformly insert encoders, an intuitive way is to compute all encoder forward microbatches before the begin of LLM pipeline, while all backward ones after its completion. While having no interference to the LLM pipeline execution, this approach poses significant memory pressure on GPU or host storage (if offloading is enabled), causing contention with other memory-bound operations such as multi-source data loaders (§7.4). In addition, temporally “isolating” encoders and the LLM abandons opportunities for fine-grained overlap, such as overlapping encoder data loading and parameter all-gather with LLM stage computations.

To reconcile workload balancing and overhead, we propose decentralized grouped reordering, which divides GPU ranks into reordering groups based on the network locality, as illustrated in Figure 11. In each group, we first initiate a metadata all-gather operation to exchange data size information. Then, we reorder modality samples using the Karp algorithm [6] and redistribute data via an all-to-all operation. This technique incurs no negative convergence impact for three reasons. First, encoder data loading follows the principle of independent and identically distributed (i.i.d.) sampling, while sample reordering across DP replicas remains equivalent to no reordering after gradient synchronization. Second, encoder outputs are restored to the original distribution of data loaders before organized as LLM inputs. Lastly, after the LLM backward pass, gradients undergo a post-reorder step to restore the original distribution.

Therefore, we propose on-demand inserting encoders before needed, as shown in Figure 10(b). For example, PP0 computes 5th microbatch for both Encoder1 and Encoder2 exactly before the output is needed by the LLM computation. Other stages (PP13) concurrently compute subsequent microbatches and asynchronously send to PP0. This reduces peak memory footprint of encoder activations by computing backward early and reclaiming memory after it, while overlapping encoder communication with the LLM. Notably, on-demand insertion has impact on sample reordering in data loaders (§5.1). The more microbatches computed in the balance group consecutively, the higher balancing efficiency achieved across encoders because the balancer have more available samples and thus larger reordering space.

Balancing Encoder Data Loading

Workload Balancing

Zero Redundancy Filtering. To alleviate I/O bandwidth congestion of remote storage, industrial practice only loads data to the first TP rank of PP0 and then locally broadcasts to other TP ranks [56]. However, since encoders partially compute microbatches of the same batch, in MegaScale-Omni, each stage should first load the entire batch and then shard

The efficiency of encoder-LLM multiplexing is closely impacted by workload balancing performance of MegaScale-Omni. We therefore introduce dedicated techniques to address workload imbalance arising from dynamic workloads.

D1 G1

D2 G2

D3 G3

{5,4,3,2} {6,7,5,6} {3,4,3,2} {5,4,5,7}

D6 G6

D7 G7

D8 G8

{3,4,3,2} {6,7,8,6} {5,4,5,4} {3,4,3,7}

Metadata All-Gather Intra-Group Reorder

Intra-Group Reorder {8,4,4,3} {7,5,3,3} {7,5,4,3} {6,6,4,2}

Data All-to-All G2

Texts

Audios

Sharding

Reduce buﬀer

GPU 1

GPU 2

GPU 1

GPU 2

G1 GPU rank

Encoder rank 1

Encoder rank 2

Encoder rank 1

Encoder rank 2

Modality sample

Sharding & All-to-All

Reordering group

{7,5,3,3} {7,5,4,2} {6,5,4,3} {6,5,4,2}

Images Decentral

D1 dataloader

LLM SP rank 1

{x,x,x} Metadata

LLM SP rank 2

All-Reduce Sharding LLM CP rank 1 LLM CP rank 2

AllGather

(a) Balance for LLM Ulysses SP

AlltoAll

(b) Balance for LLM CP

Figure 12 Balancing encoder-LLM resharding for triplemodality. Encoder and LLM ranks are colocated on 2 GPUs.

Figure 11 Decentralized grouped reordering workflow. 8 GPUs with the group size of 4, the microbatch size is 4.

across LLM CP ranks, while intra-sample sharding with fixed CP degree causes redundant communication for short samples [15, 48]. We thus only shard long samples (e.g., Image and Text in the figure) and integrate hybrid data parallelism [15] to process sharded samples with CP and unsharded ones with DP.

it based on PP ranks. This results in redundant data transfers and memory footprint in both remote I/O and TP broadcast. To mitigate data redundancy, we integrate a filter to remote data loaders, leveraging metadata (e.g., PP world size, batch size) to prefetch data to be consumed by each stage. While effective, the zero redundancy loaders cannot be checkpointed and resumed as usual. This is because vanilla checkpointing functions only store loader states of TP0+PP0 [43], which are incomplete after filtered. We address this by maintaining a data buffer on DP0 to track pre-filtered data. In each checkpointing, __getstate__() fetches complete pre-filtered data and stores in state_dict; upon resumption, __setstate__() function reloads the data and re-filters to ensure consistency with the original execution flow.

Symmetric Dispatching. After sharding, we dispatch the sharded embeddings to LLM ranks under the principle of symmetry. As shown in Figure 12(a), we use a symmetric all-to-all operation for Ulysses SP to reduce memory footprint (§4.1.1), given that the data on each encoder rank is uniformly sharded. For CP, however, all-to-all operations become asymmetric due to intra-sample sharding, which further leads to complex rank mapping and communication bottleneck from imbalanced data dispatching. We therefore use an all-reduce operation with recycled memory buffer to aggregate samples on each LLM CP rank, then sharding and merging tokens as usual (Figure 12(b)).

5.2 Balancing Encoder-LLM Data Resharding Since encoders and the LLM backbone are parallelized with different strategies, it is necessary to reshard embeddings for communication. An intuition is to first shard embeddings on each encoder rank, then route each shard to its destination rank on LLM PP0. This approach requires complex global all-toall rank mapping between encoder and LLM ranks. Instead, we propose “send-then-reshard” strategy for better code maintainability. The encoder outputs of each PP stage are asynchronously sended to PP0, then resharded by adaptive sharding and symmetric dispatching.

Implementation and Optimizations

MegaScale-Omni is built on Megatron-LM [40] and has been integrated to our in-house codebase as the foundation of most large-scale MLLM training tasks in our company. Parallelism Tuning. When scaling to thousands of GPUs, directly determining the optimal parallelism configurations for encoders and the LLM backbone is non-trivial. Manual tuning remains cost-ineffective, not only in terms of economic overhead but also in engineering time. To efficiently identify the optimal configurations, we perform parallelism tuning on a “scaled-down” proxy cluster that mimics the original hyper-scale cluster. This proxy cluster proportionately reduces the DP degrees of encoders and LLM, the number of GPUs, and the global batch size. It preserves the same optimal configuration as the

Adaptive Sample Sharding. To balance data resharding with variable-length samples, we employ adaptive sharding strategies based on the SP variant used for LLM parallelization. (1) For Ulysses SP (Figure 12(a)), we uniformly shard data along sequence dimension, as Ulysses restores the full sequence dimension before attention. (2) For context parallelism (CP, Figure 12(b)), sharding data along sequence dimension leads to imbalanced attention 10

Table 1 Models and workloads mainly used in evaluation.

original cluster, since all DP workers share identical execution schedules and operators. Selective Activation Offloading. MegaScaleOmni selectively unifies activation offloading and recomputation with distinct strategies for the LLM and encoder parts. For the LLM, we implement chunk-level overlap in virtual pipeline for computation and communication, using separate CUDA streams for D2H/H2D copies [15]. Open-source Megatron-LM [40], however, only supports vanilla CPU offloading without conjunction with pipeline or recomputation.

7.1

Experimental Setup

LLM

Batch Size

Seq-Len

Workload-A Workload-B Workload-C Workload-D

ViT1B ViT2.4B ViT10B ViT10B

LLaMA12B LLaMA70B LLaMA70B GPT175B

32 64 128 256

16384 16384 8192 8192

(2) Cluster-B : A hyper-scale productional cluster with thousands of GPUs. Detailed specifications are hidden due to business and confidential requirements of our company. Models and Workloads. We evaluate MegaScaleOmni mainly using vision-language models (VLMs), with variously sized ViTs as image encoder and LLaMA/GPT as the LLM backbone, as detailed in Table 1. For both MegaScale-Omni and baselines, Parallelism tuning is employed under each workload to locate near-optimal configurations. We further evaluate system performance under triple-modality workloads by additionally employing USM as the audio encoder. We set the maximum sequence length to 16K for hybrid packing, while reducing it to 8K for Workload-C/D to avoid overly large OOM proportions for the baselines. For VLM experiments, we use two open-source visual datasets (OpenImages [26], RefCOCOg [53]) and one productional textual dataset (BytedLong with document length of up to 512K). We use industrial-grade multi-source data loaders to load multimodal data in a streaming manner.

Efficient Operators. MegaScale-Omni incorporates advanced operator-level optimizations to further enhance computation and communication efficiency. For attention layers, we adopt FlashAttention2 [10] to improve workload partitioning for long-context inputs. For dense layers, we reduce communication overhead using Flux [9], which fuses GEMM operations with all-gather and reduce-scatter collectives in tensor parallelism into large kernels for fine-grained overlap. In addition, MegaScale-Omni enables communication overlap between encoder and LLM microbatches. Specifically, LLM gradient synchronization is overlapped with encoder computations, while encoder data loading and parameter all-gather are overlapped with LLM computations.

Evaluation

Encoder

We employ this cluster to evaluate MegaScaleOmni against other baselines and conduct ablation studies.

For the encoder, we overlap offloading communication with layer computation in a layer-wise manner because of its non-pipeline parallelization. The choice between offloading and recomputation is made at the operator granularity, since encoder operators are typically smaller than those of the LLM, which makes solely offloading hard to perfectly overlap with computation. Accordingly, we selectively recompute memory-intensive operators (e.g., core attention) and only offload compute-intensive operators (e.g., MLP projection). To achieve zero activation residency on GPUs, we further offload the persistent input activations required for recomputation in each transformer layer.

Name

Baselines. We compare MegaScale-Omni with four state-of-the-art systems: (1) Megatron-LM [40] is a widely-used, large-scale training system with various parallelism strategies, supporting MLLM training by treating encoders as the embedding layers of the first LLM pipeline stage. (2) Megatron-Dist is a Megatron-based system enhanced with encoder-LLM disaggregation [55], as the original system is closed-source. It treats encoders as the first pipeline stage and the LLM backbone as other stages with distinct resources and parallelism strategies. (3) AutoParallel is a 3D parallelization system for MLLMs built on Alpa [58]. It treats encoders as embedding layers of the LLM, while featuring automated pipeline construction to address pipeline imbalance. Due to lacking vision-language bench-

Testbeds. All experiments are conducted on two clusters: (1) Cluster-A: A cluster with 512 GPUs. Each node has 8 GPUs, 1.8TB memory, 88 vCPUs, and NVLink connection between GPUs. All nodes are connected via 8*400 Infiniband (RDMA) network. 11

Megatron-LM

1.49× 1.32×

1.54×

0.8

0.0

1.65×

3.30×

1.44× 1.48×

AutoParallel 1.40×

0.6

1.0 0.5

Megatron-Dist

×1e5

1.37× 1.27×

×1e5 0.5 0.4 7.57× 0.2 0.1 0.0

0.2

9:1

7:3

5:5

3:7

0.0

1:9

9:1

7:3

Image-text ra7o

5:5

3:7

1:9

Image-text ra7o

(a) Workload-A (64-GPU)

Op7mus

MegaScale-Omni 1.49×

1.48× 1.39×

1.20×

0.2 9:1

7:3

5:5

3:7

Image-text ra7o

(b) Workload-B (128-GPU)

×1e5 0.6

1.98× 1.81× 1.37×

1.47×

0.4

0.3

0.4

OOM

Thr. (token/sec)

×1e5 1.5

1:9

0.0

OOM

9:1

7:3

5:5

3:7

1:9

Image-text ra7o

(d) Workload-D (512-GPU)

Thr. (token/sec)

Figure 13 Training throughput (number of processed tokens per second) across Workload-A/B/C/D and image-text mixture ratios, with fixed sequence length of 16K/8K. Missing bars indicate OOM issues occur in the specified configurations. Megatron-LM ×1e5 0.8 0.4 0.0

Megatron-Dist 6.88×

AutoParallel

Op7mus

1.63×

1.37×

5:5

3:7

MegaScale-Omni

1.98× across 64, 128, 256, and 512 GPUs, respectively. The improvement on 512 GPUs is relatively moderate because all baselines suffer from OOM at the 9 : 1 mixing ratio. Across all workloads and hardware scales, the benefits of MegaScale-Omni grow increasingly significant as the ratio of image samples increases, with improvements reaching up to 5.08×. These results align with our expectations, as the encoder-LLM multiplexing scheme, which is the core design of MegaScale-Omni, targets both workload resilience and strong scalability. In contrast, Megatron-LM and Megatron-Dist confront degraded performance with up to 6.05× throughput drops and OOM issues, due to either the PP0 bottleneck or statically allocated encoder resources. By comparison, Alpa and Optimus remain more stable when the image-text ratio grows, because of their ability to adjust pipeline stage partitions (thereby alleviating the PP0 bottleneck) and colocate encoders across LLM ranks (thereby improving encoding concurrency).

1.50×

OOM 9:1

7:3

1:9

Thr. (token/sec)

(a) Sequence length = 16384 ×1e5 4.72×

1.69×

1.34×

0.8 0.4 0.0

OOM

9:1

7:3

5:5 3:7 Image-text ra7o (b) Sequence length = 32768

1:9

Figure 14 Training throughput of Workload-C and 256 GPUs with sequence length scaling to 16K and 32K.

marks, we implement VLMs based on Transformers Flax [49]. We pre-generate parallelization plans to bypass its labor-intensive searching. (4)Optimus [13] is a state-of-the-art MLLM training system that colocates encoders and the LLM with distinct parallelization for static workloads. It features fixed encoder-LLM 3D parallelization and multi-tiered bubble scheduling (for DP, TP, and PP) to reduce bubbles.

7.2

Scaling with Sequence Lengths. Figure 14 further illustrates the throughput of MegaScale-Omni on Workload-C and 256 GPUs when scaling to longer sequence lengths. It is observed that MegaScale-Omni still remains stable performance and outperforms baselines by up to 6.88× and 4..72× under sequence length of 16K and 32K, respectively. This arises from two main reasons. First, MegaScale-Omni employs long-short sequence parallelism to parallelize encoders across all GPU ranks, efficiently processing longcontext image samples. Second, MegaScale-Omni balances compute workloads across ranks for both encoders and the LLM backbone via grouped reordering and adaptive resharding, thereby avoiding stragglers when computing more image and text samples within longer sequences. Other baselines, instead, underperform due to either memory pressure on limited GPU ranks or lacking sequence parallelism support for encoders or even the LLM backbone. In our training practice with thousands of GPUs, MegaScale-Omni

End-to-End Performance

We evaluate the end-to-end training efficiency of MegaScale-Omni across various performance metrics, using Cluster-A with 512 GPUs and WorkloadA/B/C/D in Table 1. Throughput over Training Scales and Mixture Ratios. Figure 13 compares the training throughput of MegaScale-Omni and four baselines across different workloads and image-text mixture ratios under scales of 64 to 512 GPUs. As observed, MegaScale-Omni consistently outperforms all baselines across different training scales and image-text sample mixing ratios. Compared to the strongest baseline, MegaScale-Omni improves training throughput by up to 1.54×, 3.30×, 7.57×, and 12

OOM

60 40 20

9:1 7:3 5:5

3:7 1:9

Image-text ra+o (a) First pipeline stage (bo#leneck)

×1e5

Thr. (token/sec)

Megatron-Dist+Oﬄoad AutoParallel+Recomp. Op+mus+Recomp. MegaScale-Omni

Memory usage (GB)

Megatron+Oﬄoad

OOM

60 32 30 28 26

9:1 7:3 5:5

3:7 1:9

Image-text ra+o (b) Last pipeline stage

20 10 0

9:1 7:3 5:5 3:7 9:1

MegaScale-Omni

30 20 10 0

MegaScale-Omni ×1e5

0.4 1.28× 1.38× 1.58× 1.72× 0.4 1.28× 1.46× 1.72× 2.01× 0.4 1.28× 0.3

0.3

0.2

0.1

1:1:8 2:2:6 3:3:4 4:4:2

1:1:8 3:1:6 5:1:4 7:1:2

0.1

1.45×

1.40×

1.37×

1:1:8 1:3:6 1:5:4 1:7:2

Image-audio-text raGo

(a) Scaling image and audio

(b) Scaling Image

and maximum sequence lengths. As illustrated in Figure 16(a), MegaScale-Omni improves up to 23.0% MFU against baselines across mixtures, while remaining stable as encoder workloads shift. When fixing the mixture ratio as 7 : 3 and shifting the sequence length in Figure 16(b), MegaScale-Omni exhibits the greatest scaling potential by increasing MFU from 17.0% to 38.2%. Notably, all results are measured in production environments with dynamic multimodal workloads, rather than on static synthetized inputs. This setting poses substantial challenges for MFU improvement, as evidenced by a 17% MFU gap between real and synthetized workloads in MegaScale-Omni.

OOM

MFU (%)

Megatron-Dist

MFU (%)

Megatron

% % % % 0. 23.0 18.8 14.7 11.6 1

Megatron-Dist

Figure 17 Triple modality experiments of Workload-B and 128 GPUs with three scaling strategies.

Figure 15 Memory of Workload-B and 128 GPUs across mixtures, with sequence length of 16K and PP degree of 4. 40

Megatron-LM ×1e5

OOM 4K 8K 12K 16K 20K 24K 28K 32K

(a) Scaling with image-text raEos (b) Scaling with sequence lengths

Figure 16 MFU of Workload-B and 128 GPUs (a) across mixture ratios with sequence length of 16K, and (b) across sequence lengths with image-text mixture ratio of 7 : 3.

Triple-Modality Experiments. We further evaluate the training performance of MegaScale-Omni under dynamic triple-modality workloads. As shown in Figure 17, MegaScale-Omni consistently outperforms Megatron-LM and Megatron-Dist with up to 2.01× higher throughput over different image-audiotext mixture ratios. It is observed that when scaling up the proportion of image and audio samples, MegaScale-Omni remains resilient to increased encoder workloads, whereas the baselines suffer from throughput degradation of up to 41.5%. This is attributed to not only the workload-resilient encoderLLM pipeline but also the efficient optimizations to balance image and audio workloads across GPU ranks.

efficiently scales to sequence lengths of up to 512K, enhancing the model capability under long-context multimodal data. Memory Footprint over Mixture Ratios. We study the memory consumption of MegaScale-Omni and other baselines over different proportions of image and text samples. Figure 15(a) and (b) show the memory of the first and last pipeline stages, respectively. The first stage is typically the bottleneck because it stores the largest number of activations [28]. For MegaScale-Omni and baselines, all memory optimizations are enabled. As observed, for the first stage, Megatron-LM consumes the most memory (up to 68.1GB) as all encoders are prepended to PP0; Optimus, Alpa, and DistTrain exhibit moderate memory footprint yet increased as the image-text ratio grows (up to 1.75×). MegaScale-Omni consistently consumes less memory by up to 2.21× through distributing encoder activations across all GPU ranks and selectively offloading/recomputing encoder and LLM activations. For the last stage, MegaScaleOmni consumes more memory than Megatron-LM and Megatron-Dist. This is reasonable because it colocates encoders on this stage, whereas the two baselines only execute LLM computation on it.

7.3

Ablation Studies

Performance Breakdown. To understand how each optimization contributes to the overall performance, we present the speedup breakdown of MegaScale-Omni in Figure 18. As observed, disabling encoder-LLM multiplexing by prepending encoders as the LLM layers has the most significant impact on training efficiency, causing throughput degradations of 60.6% and 52.9% for Workload-A and Workload-B, respectively. This demonstrates the pivotal status of the multplexing scheme, including decoupled parallelization and joint pipeline. Another dominant optimization is the workload balancing in both data

MFU over Dynamic Workloads. Figure 16 presents the MFU scaling of MegaScale-Omni and baselines across different image-text mixture ratios 13

0.0

Thr. (token/sec)

% .0% .3% .9% -18.7 -18 -16 -9 % .6%-50.7 0.5 -60

Thr. (token/sec)

×1e5 1.0

×1e5 0.8

.6% 9% -9.6 -18.1%-11 -18.8% % % .9 5 0.4 -52.9 -4

Thr. (token/sec)

0.4 0.3 0.2 0.1 0.0

1.43× 1.19× 1.00×

1.39× 1.19× 0.85×

1.33×

1.30×

1.27× 1.28×

Throughput Overhead

1.25× 1.13×

1.04×

0.4 0.3

1.00×

800 600 400 200 0

128

Reordering group size

1 (no bala.)

(b) Workload-B (128-GPU)

Figure 20 Throughput and overhead tradeoff with Workload-B and 128 GPUs across reordering group sizes.

Figure 18 Performance breakdown on Workload-A/B with image-text ratio of 7 : 3 and sequence length of 16K. Megatron-LM

0.5

0.0

(a) Workload-A (64-GPU)

×1e5

0.6

Overhead (ms)

w/o mul?plex w/o WL-balance w/o LSSP w/o tuning w/o oﬄoading w/o OP-overlap MegaScale-Omni

Megatron-Dist

MegaScale-Omni

ure 20 shows the throughput and communication overhead of MegaScale-Omni across different reordering group sizes. As observed, a larger reordering group enhances training throughput by up to 1.33× yet causes higher overhead for data all-to-all operations (738ms for size 128), with the effect of diminishing returns existed when continuously enlarging the group.

1.44× 1.28× 1.22× 1.14× 0.96× 0.83× 0.72× 0.70× OOM

PP=DP=4, PP=DP=4, PP=DP=4, PP=DP=4, PP=TP=8, TP=8,VPP=1 TP=8,VPP=0 DP=2,VPP=1 TP=8,VPP=1, TP=8,VPP=1, w/o. oﬄoad ViT FSDP

Figure 19 Performance of encoder-LLM multiplexing for Workload-B and 128 GPUs across parallelism strategies, with image-text ratio of 7 : 3 and sequence length of 16K.

7.4

Hyper-Scale Training Experience

In this subsection, we present the operational and engineering experience of training our in-house MLLMs using MegaScale-Omni in Cluster-B with thousands of GPUs.

loading and resharding, as w/o WL-balance undermines throughput by 50.7% and 45.9%. This stems from severe workload imbalance in both encoder and LLM computations, which is caused by skewed data distributions in multimodal datasets. Moreover, disabling long-short sequence parallelism for encoders leads to throughput degradation of up to 18.7%, since encoders suffer from imbalanced computation across ranks. For other optimizations, disabling parallelism tuning, overlap in selective activation offloading, and overlap in efficient operators leads to throughput degradations of 18.1%, 16.3%, and 18.8%, respectively.

Performance Analysis. Figure 21 presents the throughput and loss results of our MLLM training task. As observed, MegaScale-Omni remains generally stable under multimodal workloads with the average training throughput of about 8M tokens per second. Over the process, MegaScale-Omni underwent training restarts of 59 times, most of which were automatically recovered from hardware and software faults. Experts also employed periodic tensor checks to ensure stable convergence, initially for all communication tensors yet significantly affects throughput (second black edged block). In later steps, we only checked output tensors of encoders, which mitigated throughput degradation. As shown in Figure 21(b), we observed transient loss spikes for ViT due to the large learning rate used in early steps. Simply reducing learning rate mitigated these spikes yet degraded final model quality. We thus manually restarted the training to bypass loss anomalies in early steps; whereas in later steps, loss spikes recovered automatically in most cases.

Robustness of Multiplexing. To study the robustness of encoder-LLM multiplexing over parallelism configurations, we exclusively evaluate the multiplexer by disabling other optimizations of MegaScale-Omni. As shown in Figure 19, MegaScaleOmni consistently outperforms Megatron-LM and Megatron-Dist across different parallelism strategies with up to 1.64× throughput improvements. The comparisons span multiple dimensions, including disabling virtual pipeline (#VPP layers per stage), enlarging PP degrees, disabling activation offloading, and employing FSDP for ViT. These results demonstrate that benefits of MegaScale-Omni originate from its advanced architectural design, rather than experttuned parallelism strategies.

MFU Optimizations. We have made significant efforts to enhance the MFU performance of MLLM training. One of our encountered problems is the network interference between non-training communication operations (e.g., all-gather operations in multi-source token counter for modality ratio control) and model DP all-gather operations used in the distributed optimizer [23, 40]. This results in

Workload Balance and Overhead Analysis. We further delve into the performance of workload balancing optimizations by studying the “efficiencyoverhead” tradeoff in grouped data reordering. Fig14

1.2

1.1

Check

Training restarts

4 2 0

1.0 0.9

Training init 0

20K

(§5.1), enabled cross-NUMA node memory access under high pressure, and selectively adjusted offloading plans based on fine-grained memory monitoring.

Loss anomaly

Loss

Thr. (M token/sec)

40K

60K

80K

Steps

(a) End-to-end training throughput

100K

0.8

The second problem is the existance of loader state saving stragglers. Under 512K-length multimodal workloads, the overhead of saving loader states randomly fluctuates from 6 to 20 minutes, where the slowest straggler slowdowns the entire training process. We therefore employed asynchronous loader state snapshot with ahead-of-time state preparation, reducing the overhead to about 800ms.

20K 40K 60K 80K 100K

Steps

(b) Training loss curve

Figure 21 Performance of hyper-scale MLLM training in our production environments with thousands of GPUs.

a training delay of about 150ms and throughput drop of 2.2%. Intuitive approaches such as reducing counter frequency or using a separate process group for non-training communication either cause inaccurate metric collection or increase risks of communication hangs. Therefore, our solution is to relocate related code segments to the end of pipeline functions, which overlaps the counter all-gather (between DP groups) with encoder backward computation (within each DP group).

Communication and Checkpointing Anomalies. We have observed that all-to-all and P2P primitives became unstable and prone to communication hangs when the group size exceeded 512. This arises from over-sharded send and receive tensors for medium-size operations during data transmission. Due to the vulnerability of naive P2P communication used to transfer sharded, flattened 1D tensors across ranks, we encountered random checkpoint saving hangs in hyper-scale training. To address this issue, we adopted non-P2P operations to save checkpoints by restructuring N-D sharded tensors through offset and length-based indexing. Moreover, the firsttime checkpointing overhead after restart reached about 15 minutes [43], which became non-negligible under frequent training restarts. To eliminate this, we employed a persistent saving plan cache for all checkpoints in HDFS when model parallelization remains unchanged.

Another optimization is employing the “forward-thenbackward” pipeline [19] to trade memory for less bubbles in 512K-length MLLM training. Due to the sample length variability in multimodal workloads, some microbatches may be assigned overly long samples with much longer processing time [48], while others consist mostly of short ones. Such heterogeneity introduces substantial internal bubbles in warmup and drain phases of 1F1B-like pipeline. This is because, after issuing a number of microbatches equal to the pipeline degree, the first stage must wait until the first microbatch completes on the last stage. We therefore took another route by using “forwardthen-backward” pipeline to issue all microbatches in warmup and drain phases to fill bubbles, while reducing memory with more selective offloading and recomputation (§6). As a result, we significantly improved MFU and saved about 2/3 GPU hours for our training task.

Related Work

Large Model Training Framworks. A lot of efforts have been dedicated to improving the training efficiency of large models. Megatron-LM [40], DeepSpeed [39], and PyTorch FSDP [57] are the most popular open-source training frameworks with various parallelism strategies [3, 20, 25, 32, 33, 38, 40]. Some works [1, 9, 46] explore overlapping communication and computation operators to reduce device idle time and improve resource utilization. Alpa [58], nnScaler [30], and Unity [42] automatically identify the optimal hybrid parallelism strategy on specified resources. These frameworks focus on generic largescale models rather than multimodal LLM training, and thus lack tailored designs to efficiently address MLLM architectures, encoder-LLM orchestration, and dynamic multimodal workloads.

Data Loader Problems. We have in-depth analyzed the fault records of multi-source data loaders used in training. The first problem is the host memory contention between memory offloading and multiprocess data loading. While the activation offloading in MegaScale-Omni reduces GPU memory footprint and overlaps communication, we observe nonnegligible host memory fragmentation that degrades memory access efficiency. Moreover, distributed data loaders typically employ multiple workers to alleviate data loading bottleneck, binding each to a NUMA node for fast memory access. To alleviate contention, we employed parallel data loading across PP ranks

MultiModal Training Systems. With the advent of multimodal tasks, recent works have studied building dedicated training systems for multimodal 15

LLMs (MLLMs). Megatron-LM [40] and Transformers [49] support training MLLMs but still in the way of unimodal training, treating encoders as embedding layers of the LLM backbone. DistMM [18] and DistTrain [55] propose disaggregating encoders and the LLM backbone as two separate models, while exploring heterogeneity-aware model partitioning, resource allocation, and data balancing for MLLMs. Optimus [13], GraphPipe [21], Spindle [47], and PipeWeaver [51] propose either fine-grained multi-tiered bubble exploitation or dynamic pipeline scheduling algorithms to reduce execution bubbles. These researches represent valuable academic efforts in exploring the generic paradigm of multimodal training, providing insights that inspire the design choices of MegaScale-Omni. However, from our experience, they either introduce substantial engineering complexity or rely on idealized algorithms that can lead to unstable performance in industrial deployment. MegaScale-Omni is designed as a foundational system that remains extensible to advanced scheduling algorithms to further improve MLLM training efficiency.

Conclusion

This paper presents MegaScale-Omni, an industrialgrade MLLM training system for dynamic workload adaption and hyper-scale deployment with thousands of GPUs. The core idea of MegaScale-Omni is the encoder-LLM multiplexing scheme, which decouples parallelization, colocates resources, and jointly orchestrates the execution for encoders and the LLM backbone. It also employs data reordering and resharding optimizations to balance encoder and LLM workloads. Experiments demonstrate that MegaScale-Omni improves training throughput by 1.27×–7.57× under dynamic multimodal workloads. We hope our insights will inspire future research and advance generic paradigms for MLLM training.

References

Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024. URL https://arxiv.org/abs/2403.17297.

[1] Nvidia transformer engine. https://github.com/ NVIDIA/TransformerEngine, 2022. [2] Internlm technical report. https://github.com/ InternLM/InternLM-techreport, 2023. [3] Nvidia context parallel package. https://docs. nvidia.com/megatron-core/developer-guide/ latest/api-guide/context_parallel.html, 2025. [4] Gemini 2.5 flash. https://deepmind.google/ models/gemini/flash/, 2025. [5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https://arxiv.org/abs/2502.13923.

[9] Liwen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Ziheng Jiang, Haibin Lin, et al. Flux: Fast software-based communication overlap on gpus through kernel fusion. arXiv preprint arXiv:2406.06858, 2024. [10] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691.

Analysis of [6] S. Boettcher and S. Mertens. the karmarkar-karp differencing algorithm. The European Physical Journal B, 65(1):131–140, August 2008. ISSN 1434-6036. doi: 10.1140/epjb/ e2008-00320-9. URL http://dx.doi.org/10.1140/ epjb/e2008-00320-9.

[11] Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, Avital Oliver, Piotr Padlewski, Alexey Gritsenko, Mario Lučić, and Neil Houlsby. Patch n’ pack: Navit, a vision transformer for any aspect ratio and resolution, 2023. URL https://arxiv.org/abs/2307.06304.

[7] Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, Megan Richards, Samuel Lavoie, Pietro Astolfi, Reyhane Askari Hemmat, Jun Chen, Kushal Tirumala, Rim Assouel, Mazda Moayeri, Arjang Talattof, Kamalika Chaudhuri, Zechun Liu, Xilun Chen, Quentin Garrido, Karen Ullrich, Aishwarya Agrawal, Kate Saenko, Asli Celikyilmaz, and Vikas Chandra. An introduction to visionlanguage modeling, 2024. URL https://arxiv.org/ abs/2405.17247.

[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929. [13] Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng, Haibin Lin, and Minlan Yu. Optimus: Accelerating large-scale multi-modal llm training by bubble exploitation. arXiv preprint arXiv:2408.03505, 2024.

[8] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song,

[14] Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, Wei Liu, Yichun Shi, Shiqi Sun, Yu Tian, Zhi Tian, Peng Wang, Rui Wang, Xuanda Wang, Xun Wang, Ye Wang, Guofeng Wu, Jie Wu, Xin Xia, Xuefeng Xiao, Zhonghua Zhai, Xinyu Zhang, Qi Zhang, Yuwei Zhang, Shijia Zhao, Jianchao Yang, and Weilin Huang. Seedream 3.0 technical report, 2025. URL https://arxiv.org/ abs/2504.11346. [15] Hao Ge, Junda Feng, Qi Huang, Fangcheng Fu, Xiaonan Nie, Lei Zuo, Haibin Lin, Bin Cui, and Xin

Liu. Bytescale: Efficient scaling of llm training with a 2048k context length on more than 12,000 gpus, 2025. URL https://arxiv.org/abs/2502.21231.

bei Li, Mingrui Chen, Peng Liu, Ruihang Miao, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Gong, Zixin Zhang, Hongyu Zhou, Jianjian Sun, Brian Li, Chengting Feng, Changyi Wan, Hanpeng Hu, Jianchang Wu, Jiangjie Zhen, Ranchen Ming, Song Yuan, Xuelin Zhang, Yu Zhou, Bingxin Li, Buyun Ma, Hongyuan Wang, Kang An, Wei Ji, Wen Li, Xuan Wen, Xiangwen Kong, Yuankai Ma, Yuanwei Liang, Yun Mou, Bahtiyar Ahmidi, Bin Wang, Bo Li, Changxin Miao, Chen Xu, Chenrun Wang, Dapeng Shi, Deshan Sun, Dingyuan Hu, Dula Sai, Enle Liu, Guanzhe Huang, Gulin Yan, Heng Wang, Haonan Jia, Haoyang Zhang, Jiahao Gong, Junjing Guo, Jiashuai Liu, Jiahong Liu, Jie Feng, Jie Wu, Jiaoren Wu, Jie Yang, Jinguo Wang, Jingyang Zhang, Junzhe Lin, Kaixiang Li, Lei Xia, Li Zhou, Liang Zhao, Longlong Gu, Mei Chen, Menglin Wu, Ming Li, Mingxiao Li, Mingliang Li, Mingyao Liang, Na Wang, Nie Hao, Qiling Wu, Qinyuan Tan, Ran Sun, Shuai Shuai, Shaoliang Pang, Shiliang Yang, Shuli Gao, Shanshan Yuan, Siqi Liu, Shihong Deng, Shilei Jiang, Sitong Liu, Tiancheng Cao, Tianyu Wang, Wenjin Deng, Wuxun Xie, Weipeng Ming, Wenqing He, Wen Sun, Xin Han, Xin Huang, Xiaomin Deng, Xiaojia Liu, Xin Wu, Xu Zhao, Yanan Wei, Yanbo Yu, Yang Cao, Yangguang Li, Yangzhen Ma, Yanming Xu, Yaoyu Wang, Yaqiang Shi, Yilei Wang, Yizhuang Zhou, Yinmin Zhong, Yang Zhang, Yaoben Wei, Yu Luo, Yuanwei Lu, Yuhe Yin, Yuchu Luo, Yuanhao Ding, Yuting Yan, Yaqi Dai, Yuxiang Yang, Zhe Xie, Zheng Ge, Zheng Sun, Zhewei Huang, Zhichao Chang, Zhisheng Guan, Zidong Yang, Zili Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, and Yibo Zhu. Step-audio: Unified understanding and generation in intelligent speech interaction, 2025. URL https://arxiv.org/abs/2502.11946.

[16] Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, Jingji Chen, Jingjia Huang, Kang Lei, Liping Yuan, Lishu Luo, Pengfei Liu, Qinghao Ye, Rui Qian, Shen Yan, Shixiong Zhao, Shuai Peng, Shuangye Li, Sihang Yuan, Sijin Wu, Tianheng Cheng, Weiwei Liu, Wenqian Wang, Xianhan Zeng, Xiao Liu, Xiaobo Qin, Xiaohan Ding, Xiaojun Xiao, Xiaoying Zhang, Xuanwei Zhang, Xuehan Xiong, Yanghua Peng, Yangrui Chen, Yanwei Li, Yanxu Hu, Yi Lin, Yiyuan Hu, Yiyuan Zhang, Youbin Wu, Yu Li, Yudong Liu, Yue Ling, Yujia Qin, Zanbo Wang, Zhiwu He, Aoxue Zhang, Bairen Yi, Bencheng Liao, Can Huang, Can Zhang, Chaorui Deng, Chaoyi Deng, Cheng Lin, Cheng Yuan, Chenggang Li, Chenhui Gou, Chenwei Lou, Chengzhi Wei, Chundian Liu, Chunyuan Li, Deyao Zhu, Donghong Zhong, Feng Li, Feng Zhang, Gang Wu, Guodong Li, Guohong Xiao, Haibin Lin, Haihua Yang, Haoming Wang, Heng Ji, Hongxiang Hao, Hui Shen, Huixia Li, Jiahao Li, Jialong Wu, Jianhua Zhu, Jianpeng Jiao, Jiashi Feng, Jiaze Chen, Jianhui Duan, Jihao Liu, Jin Zeng, Jingqun Tang, Jingyu Sun, Joya Chen, Jun Long, Junda Feng, Junfeng Zhan, Junjie Fang, Junting Lu, Kai Hua, Kai Liu, Kai Shen, Kaiyuan Zhang, Ke Shen, Ke Wang, Keyu Pan, Kun Zhang, Kunchang Li, Lanxin Li, Lei Li, Lei Shi, Li Han, Liang Xiang, Liangqiang Chen, Lin Chen, Lin Li, Lin Yan, Liying Chi, Longxiang Liu, Mengfei Du, Mingxuan Wang, Ningxin Pan, Peibin Chen, Pengfei Chen, Pengfei Wu, Qingqing Yuan, Qingyao Shuai, Qiuyan Tao, Renjie Zheng, Renrui Zhang, Ru Zhang, Rui Wang, Rui Yang, Rui Zhao, Shaoqiang Xu, Shihao Liang, Shipeng Yan, Shu Zhong, Shuaishuai Cao, Shuangzhi Wu, Shufan Liu, Shuhan Chang, Songhua Cai, Tenglong Ao, Tianhao Yang, Tingting Zhang, Wanjun Zhong, Wei Jia, Wei Weng, Weihao Yu, Wenhao Huang, Wenjia Zhu, Wenli Yang, Wenzhi Wang, Xiang Long, XiangRui Yin, Xiao Li, Xiaolei Zhu, Xiaoying Jia, Xijin Zhang, Xin Liu, Xinchen Zhang, Xinyu Yang, Xiongcai Luo, Xiuli Chen, Xuantong Zhong, Xuefeng Xiao, Xujing Li, Yan Wu, Yawei Wen, Yifan Du, Yihao Zhang, Yining Ye, Yonghui Wu, Yu Liu, Yu Yue, Yufeng Zhou, Yufeng Yuan, Yuhang Xu, Yuhong Yang, Yun Zhang, Yunhao Fang, Yuntao Li, Yurui Ren, Yuwen Xiong, Zehua Hong, Zehua Wang, Zewei Sun, Zeyu Wang, Zhao Cai, Zhaoyue Zha, Zhecheng An, Zhehui Zhao, Zhengzhuo Xu, Zhipeng Chen, Zhiyong Wu, Zhuofan Zheng, Zihao Wang, Zilong Huang, Ziyu Zhu, and Zuquan Song. Seed1.5-vl technical report, 2025. URL https://arxiv.org/abs/2505.07062.

[18] Jun Huang, Zhen Zhang, Shuai Zheng, Feng Qin, and Yida Wang. DISTMM: Accelerating distributed multimodal model training. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1157–1171, Santa Clara, CA, April 2024. USENIX Association. ISBN 978-1-939133-39-7. URL https://www.usenix. org/conference/nsdi24/presentation/huang. [19] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019. [20] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of ex-

[17] Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jing-

treme long sequence transformer models, 2023. URL https://arxiv.org/abs/2309.14509.

[29] Chao Liao, Liyang Liu, Xun Wang, Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation, 2025. URL https://arxiv.org/abs/2505.05472.

[21] Byungsoo Jeon, Mengdi Wu, Shiyi Cao, Sunghyun Kim, Sunghyun Park, Neeraj Aggarwal, Colin Unger, Daiyaan Arfeen, Peiyuan Liao, Xupeng Miao, Mohammad Alizadeh, Gregory R. Ganger, Tianqi Chen, and Zhihao Jia. Graphpipe: Improving performance and scalability of dnn training with graph pipeline parallelism, 2024. URL https://arxiv.org/abs/ 2406.17145.

[30] Zhiqi Lin, Youshan Miao, Quanlu Zhang, Fan Yang, Yi Zhu, Cheng Li, Saeed Maleki, Xu Cao, Ning Shang, Yilei Yang, et al. {nnScaler}:{ConstraintGuided} parallelization plan generation for deep learning training. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 347–363, 2024.

[22] Chenyu Jiang, Zhen Jia, Shuai Zheng, Yida Wang, and Chuan Wu. Dynapipe: Optimizing multitask training through dynamic pipelines. In Proceedings of the Nineteenth European Conference on Computer Systems, pages 542–559, 2024.

[31] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer, 2021. URL https://arxiv.org/abs/2106.13230. [32] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1–15, 2019.

[23] Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin, and Xin Liu. Megascale: Scaling large language model training to more than 10,000 gpus, 2024. URL https://arxiv.org/abs/2402.15627.

[33] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.

[24] Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https://arxiv.org/ abs/1312.6114. [25] Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models, 2022. URL https://arxiv.org/abs/2205.05198.

[34] OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex BakerWhitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern,

[26] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7):1956–1981, March 2020. ISSN 15731405. doi: 10.1007/s11263-020-01316-z. URL http: //dx.doi.org/10.1007/s11263-020-01316-z. [27] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020. [28] Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. Terapipe: Token-level pipeline parallelism for training large-scale language models. In International Conference on Machine Learning, pages 6543–6552. PMLR, 2021.

Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O’Connell, Ian O’Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu,

Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276. [35] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, 2015. doi: 10.1109/ICASSP.2015.7178964. [36] Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. Zero bubble pipeline parallelism. arXiv preprint arXiv:2401.10241, 2023. [37] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20:

International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020.

Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024. URL https://arxiv.org/abs/2409.12191.

[38] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeedmoe: Advancing mixture-of-experts inference and training to power next-generation ai scale, 2022. URL https://arxiv.org/abs/2201.05596.

[46] Shibo Wang, Jinliang Wei, Amit Sabne, Andy Davis, Berkin Ilbeyi, Blake Hechtman, Dehao Chen, Karthik Srinivasa Murthy, Marcello Maggioni, Qiao Zhang, et al. Overlap communication with dependent computation via decomposition in large deep learning models. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pages 93–106, 2022.

[39] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 3505–3506, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984. doi: 10.1145/3394486.3406703. URL https://doi. org/10.1145/3394486.3406703.

[47] Yujie Wang, Shenhan Zhu, Fangcheng Fu, Xupeng Miao, Jie Zhang, Juan Zhu, Fan Hong, Yong Li, and Bin Cui. Spindle: Efficient distributed training of multi-task large models via wavefront scheduling, 2025. URL https://arxiv.org/abs/2409.03365. [48] Zheng Wang, Anna Cai, Xinfeng Xie, Zaifeng Pan, Yue Guan, Weiwei Chu, Jie Wang, Shikai Li, Jianyu Huang, Chris Cai, Yuchen Hao, and Yufei Ding. Wlb-llm: Workload-balanced 4d parallelism for large language model training, 2025. URL https://arxiv. org/abs/2503.17924.

[40] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.

[49] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Art Natural Language Processing. pages 38–45. Association for Computational Linguistics, October 2020. URL https://www. aclweb.org/anthology/2020.emnlp-demos.6.

[41] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. [42] Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, et al. Unity: Accelerating {DNN} training through joint optimization of algebraic transformations and parallelization. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 267– 284, 2022.

[50] Tianyuan Wu, Lunxi Cao, Hanfeng Lu, Xiaoxiao Jiang, Yinghao Yu, Siran Yang, Guodong Yang, Jiamang Wang, Lin Qu, Liping Zhang, and Wei Wang. Adaptra: Straggler-resilient hybrid-parallel training with pipeline adaptation, 2025. URL https: //arxiv.org/abs/2504.19232. [51] Zhenliang Xue, Hanpeng Hu, Xing Chen, Yimin Jiang, Yixin Song, Zeyu Mi, Yibo Zhu, Daxin Jiang, Yubin Xia, and Haibo Chen. Pipeweaver: Addressing data dynamicity in large multimodal model training with dynamic interleaved pipeline, 2025. URL https: //arxiv.org/abs/2504.14145.

[43] Borui Wan, Mingji Han, Yiyao Sheng, Zhichao Lai, Mofan Zhang, Junda Zhang, Yanghua Peng, Haibin Lin, Xin Liu, and Chuan Wu. Bytecheckpoint: A unified checkpointing system for llm development. arXiv preprint arXiv:2407.20143, 2024. [44] Dingdong Wang, Mingyu Cui, Dongchao Yang, Xueyuan Chen, and Helen Meng. A comparative study of discrete speech tokens for semanticrelated tasks with large language models, 2024. URL https://arxiv.org/abs/2411.08742.

[52] Yifan Yang, Zheshu Song, Jianheng Zhuo, Mingyu Cui, Jinpeng Li, Bo Yang, Yexing Du, Ziyang Ma, Xunying Liu, Ziyuan Wang, Ke Li, Shuai Fan, Kai Yu, Wei-Qiang Zhang, Guoguo Chen, and Xie Chen. Gigaspeech 2: An evolving, large-scale and multidomain asr corpus for low-resource languages with automated crawling, transcription and refinement, 2025. URL https://arxiv.org/abs/2406.11546.

[45] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin.

[53] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling con-

text in referring expressions, 2016. URL https: //arxiv.org/abs/1608.00272. [54] Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara Sainath, Pedro Moreno, ChungCheng Chiu, Johan Schalkwyk, Françoise Beaufays, and Yonghui Wu. Google usm: Scaling automatic speech recognition beyond 100 languages, 2023. URL https://arxiv.org/abs/2303.01037. [55] Zili Zhang, Yinmin Zhong, Ranchen Ming, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, and Xin Jin. Disttrain: Addressing model and data heterogeneity with disaggregated training for multimodal large language models. arXiv preprint arXiv:2408.04275, 2024. [56] Juntao Zhao, Qi Lu, Wei Jia, Borui Wan, Lei Zuo, Junda Feng, Jianyu Jiang, Yangrui Chen, Shuaishuai Cao, Jialing He, Kaihua Jiang, Yuanzhe Hu, Shibiao Nong, Yanghua Peng, Haibin Lin, Xin Liu, and Chuan Wu. Overlord: Ultimate scaling of dataloader for multi-source large foundation model training, 2025. URL https://arxiv.org/abs/2504.09844. [57] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023. [58] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. Alpa: Automating inter-and intra-operator parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 559–578, 2022.

Record · ID 175220 · SHA-256 ad5368309ca96dbb
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.