2026-05-12
Accelerating Compound LLM Training Workloads with Maestro Xiulong Yuan∗ , Hongqing Chen∗ , Jiaxuan Peng∗ , Fan Zhou∗ , Zhixiang Ruan∗ , Zekun Wang∗ , Bo Zheng, Rui Men, Haiquan Wang, Zhipeng Zhang, Langshi Chen, Man Yuan, Jiaqi Gao, Zhengping Qian, Junyang Lin, Yong Li† , Wei Lin, Junhua Wang, Jingren Zhou Alibaba Group ∗
Equal contribution.
†
Corresponding authors.
arXiv:2605.10501v1 [cs.DC] 11 May 2026
Abstract Compound LLM training workloads—such as knowledge distillation and multimodal LLM (MLLM) training—are gaining prominence. These workloads typically comprise heterogeneous components that differ significantly in parameter scale, execution mode (forward-only or full forward-backward pass), and sequence length. Besides, component activation can be data-dependent: in MLLM training, for instance, modality-specific components are activated only when the input contains corresponding modality data, resulting in dynamic computational paths and irregular runtime workloads. Conventional training frameworks, designed for monolithic models, cannot handle the dual heterogeneity of static configurations and runtime behavior. By enforcing one-size-fits-all training configurations across components and ignoring input-induced workload variations—they suffer from suboptimal throughput and poor GPU utilization on emerging compound workloads. In this paper, we introduce Maestro, a section-centric training framework systematically addresses the aforementioned dual heterogeneity challenges. Maestro first reconstructes the compound workload computation into a coarse-grained section graph. Each section is capable of configuring its own tailored parallelism strategy, micro-batch size, and even data-parallel degree. This fine-grained, component-aware mechanism for training configuration and resource allocation effectively addresses the static heterogeneity inherent across different model components. To tackle data-dependent runtime workload irregularity, Maestro further introduces a wavefront scheduling algorithm that dynamically reorders input samples to orchestrate computation across concurrently active sections while strictly preserving cross-section data dependencies. By doing so, it maximizes inter-section parallelism and minimizes section stalls, thereby improving overall hardware utilization. Maestro has been deployed in production for millions of GPU hours and has demonstrated significant real-world impact: it reduces GPU resource consumption by approximately 40% on critical workloads—including knowledge distillation and multimodal model training—thereby validating its effectiveness in large-scale, real-world training scenarios. In this paper, we share Maestro’s design and implementation details to empower the broader research and engineering community in advancing next-generation compound LLM training.
1
Introduction
The training paradigm for modern large language models (LLMs) is shifting from homogeneous, monolithic Transformer stacks toward compound workloads composed of heterogeneous, functionally specialized components that interact in complex ways (Zhang et al., 2025a; Xu et al., 2025a). This shift is particularly evident in the rise of multimodal large language model (MLLM) architectures, which integrate multiple specialized sub-modules—such as modality-specific encoders for feature extraction, an auto-regressive LLM backbone for cross-modal reasoning, and modality-specific decoders for generating outputs across text, image, audio, and other modalities—into a unified training pipeline (Xu et al., 2025b; Team, 2025; AI et al., 2025; Bai et al., 2026). Another prominent example of such compound workloads is knowledge distillation, in which a student model is trained to emulate the behavior of one or more teacher models that are often more capable through carefully designed loss functions that facilitate knowledge transfer for efficient model compression. The teacher and student models frequently differ in architecture, parameter size, and execution mode: teachers typically perform inference-only forward passes, while the student undergoes full forward-backward training (AI, 2025). These diverse functional components exhibit markedly different computational, memory, and communication characteristics. Yet, conventional training frameworks typically enforce a uniform global configuration—such as a single parallelism strategy and fixed micro-batch size—across all components. 1
This failure to account for component-level heterogeneity often results in suboptimal training efficiency and underutilized hardware resources (Zhang et al., 2025b). Moreover, the training recipes in these compound workloads differ significantly from those used for conventional uniformly stacked models. Their unique training strategies—combined with data-dependent computation in sub-modules—introduce dynamic runtime irregularities. For instance, vision-language model (VLM) training typically leverages a heterogeneous dataset comprising both text-only and text-image samples, aiming to strengthen visual grounding capabilities while maintaining robust performance across general tasks. During training, text-image samples activate the vision encoder for feature extraction, whereas text-only samples bypass it entirely and are routed directly to the LLM backbone. This input-dependent activation pattern yields dynamic computation graphs that vary across batches and thus introduces workload irregularities during runtime (Wang et al., 2025; Bai et al., 2025). When extended to omni-modal models that accept images, audio, and text as inputs and generate both text and audio outputs, the training data distribution becomes markedly more heterogeneous—spanning text-only, text-image, text-audio, and text-image-audio combinations. This expanded modality spectrum intensifies dynamic runtime irregularities, as the activation patterns of modality-specific encoders and decoders vary substantially across batches, further exacerbating the scheduling challenges posed by input-dependent computation graphs (Ye et al., 2025; Xu et al., 2025b). Optimizing workloads that involve computation of such input-dependent, sparsely activated components lies beyond the scope of existing training frameworks like Megatron-LM, which were primarily designed for monolithic Transformer models. Even with an optimally tuned static configuration, dynamic load imbalance between data-parallel and pipeline-parallel ranks at runtime can readily cause performance degradation and suboptimal resource utilization. To address the aforementioned challenges, we introduce Maestro, a training framework specifically designed for modern compound LLM training workload that treats heterogeneous components as first-class citizens. Maestro first partitions the original computation graph into sections, where each section groups sub-modules with similar computational, memory, and communication characteristics. Each section can then be assigned tailored training configurations—such as parallelization strategy, micro-batch size, and even data-parallel group size—enabling fine-grained resource allocation and training strategy optimization across heterogeneous model components. To effectively handle runtime irregularity arising from data-dependent section activation, Maestro incorporates a wavefront scheduler that orchestrates section execution to maximize inter-section concurrency while minimizing data-dependent stalls—prioritizing optimal performance of sections residing on the critical path. Furthermore, to enable efficient inter-section data transfer with negligible interference to GPU kernel execution, we design and implement an asynchronous, asymmetric message queue based on one-sided remote direct memory access. This mechanism automatically shards and transfers tensors across parallelism domain boundaries with minimal CPU involvement and near-zero GPU overhead. Experiments on compound workloads—including knowledge distillation and multimodal model training—demonstrate that Maestro achieves up to 1.7× higher training throughput compared to Megatron-LM. Deployed in production for millions of GPU hours, Maestro has accelerated the training of our latest Qwen3.5 model series while reducing GPU resource consumption by approximately 40%.
2
Compound LLM Training and Challenges
The rise of compound LLM training workloads has fundamentally undermined the assumption of workload uniformity that underpins conventional training systems. In this section, we dissect two representative sources of heterogeneity—multimodal model training and knowledge distillation—and analyze how each introduces distinct resource demands and runtime workload irregularities that collectively present critical challenges to training efficiency optimization in existing frameworks. 2.1
Multimodal model training
Multimodal large language models (MLLMs) integrate modality-specific encoders and decoders with a shared autoregressive LLM backbone, forming a multi-stage computational pipeline. This pipeline proceeds sequentially through three phases: (1) modality-specific feature extraction by encoders, (2) cross-modal fusion and reasoning within the LLM backbone, and (3) modality-conditioned output generation via dedicated decoders. The resulting workload exhibits pronounced heterogeneity across components—not only in parameter scale but also in computational intensity and memory footprint—posing fundamental challenges for uniform resource allocation strategies. For example, in the Qwen3-VL-235B-A22B model (Bai et al., 2025), the vision encoder contains only 0.4B parameters, whereas the LLM backbone reaches 235B model scale (Qwen3-VL-Team, 2026). Despite its small parameter count, the vision encoder processes extremely long sequences of visual tokens before applying a downsampling operation (e.g., 4:1) along the sequence dimension to produce compact 2
Figure 1: Architecture of Qwen3-VL model. embeddings for the LLM. This attention computation over long sequences results in disproportionately high computational overhead for end-to-end training relative to its model size. To address this bottleneck, context parallelism is the main necessary parallelism strategy for this component to accelerate attention computation. In contrast, the LLM backbone confronts a multifaceted optimization challenge. Even when trained on data with moderate sequence lengths, its massive parameter scale imposes severe memory pressure that cannot be alleviated by any single parallelism method. Instead, a hybrid parallelism configuration—orchestrating tensor, pipeline, expert, and context parallelism in concert—is essential to jointly optimize memory footprint, computational density, and communication efficiency. This stands in sharp opposition to the vision encoder’s singular reliance on context parallelism, epitomizing the static heterogeneity across components that undermines uniform configuration assumptions in conventional frameworks. In practice, MLLMs are typically trained on heterogeneous datasets comprising both text-only and multimodal samples (Wang et al., 2025). Text-only samples bypass all modality-specific encoders entirely, whereas multimodal samples dynamically activate one or more encoders (e.g., vision or audio encoders) based on input composition. This data-dependent activation pattern introduces runtime irregularity: during multimodal sample processing, the LLM backbone stalls while awaiting modality embeddings; conversely, during text-only processing, modality encoders remain largely idle. When pipeline parallelism is employed, this sample-wise imbalance manifests as dynamic pipeline bubbles that degrade end-to-end throughput. Critically, the degradation scales adversely with pipeline depth: as models grow larger and require deeper pipelines, the relative performance penalty intensifies—leading to disproportionately greater resource wastage despite increased hardware investment (Bai et al., 2026; Team, 2025; Chen et al., 2025). 2.2
Knowledge Distillation
Figure 2: The training workload in knowledge distillation. Beyond architectural heterogeneity within a single model, modern training workloads increasingly exhibit cross-model heterogeneity: multiple interacting models—each with distinct execution semantics—are co-scheduled within a single logical training job. Knowledge distillation epitomizes this paradigm, tightly coupling one or more frozen teacher models with a trainable student model in an integrated loop. Critically, these components manifest fundamental execution asymmetry. Teacher models participate exclusively in the forward pass to generate soft targets or logits (Hinton et al., 2015; Gou et al., 2021), bypassing backpropagation and parameter updates entirely. In stark contrast, the student model undergoes the full training lifecycle—forward propagation, loss computation, backpropagation, and optimizer-driven 3
parameter updates—bearing the entirety of gradient-related computation and memory pressure. This execution asymmetry translates into stark disparities in resource requirements and utilization patterns. Although both teacher and student models may demand comparable GPU memory for parameter storage, only the student incurs the full training memory footprint—optimizer states, activations, and gradients—while the teacher, being frozen, requires merely inference-time resources. Consequently, in self-distillation scenarios (Zhao et al., 2026; Hubotter et al., 2026), where teacher and student share identical architectures, enforcing a uniform parallelism configuration becomes fundamentally suboptimal. The teacher’s lightweight forward-only workload, when subjected to the same aggressive sharding strategy as the student, suffers from excessive device fragmentation—yielding underutilized compute units and unnecessary inter-device communication overhead, despite its minimal computational demands. Furthermore, increasing the micro-batch size substantially improves the teacher model’s throughput with negligible additional memory overhead, while imposing significant activation memory pressure on the student model. Conventional training systems enforce a uniform micro-batch size across all components within a job—dictated solely by the student’s memory constraints—forcing the teacher to operate at suboptimal batch scales. This one-size-fits-all policy inevitably leads to underutilized GPU compute resources and degraded end-to-end training efficiency.
3
Maestro System Design and Implementation
Figure 3: Overview of Maestro. We illustrate Maestro’s overall workflow in Figure 3. Given a compound LLM workload—such as MLLM training, Maestro first decomposes models into distinct logical sections based on component boundaries and execution semantics.It then logically partitions the GPU cluster into dedicated resource groups, each sized and shaped according to its section’s parallelism strategy (tensor, pipeline, expert, context) and fanout configuration. Each resource group independently initializes its collective communication groups and manages checkpoint loading and saving for its corresponding section. Crucially, Maestro establishes crosssection communication channels to enable tensor transfer and dynamic resharding across heterogeneous parallelism degrees—particularly between sections employing different tensor parallelism (TP) or context parallelism (CP) degrees. During each training iteration, given a global batch of samples, Maestro leverages an wavefront scheduler that dynamically reorders samples based on their activated sections and exploitable inter-section parallelism. This runtime-aware orchestration simultaneously maximizes cross-section concurrency and suppresses inter-section pipeline bubbles induced by data-dependent activation patterns. Importantly, Maestro guaranties training equivalence—i.e., it produces identical model updates as the original unmodified training process—and introduces no degradation in model performance.
4
3.1
Section Construction
We first discuss section construction in this part. The section serves as a first-class abstraction for training orchestration in Maestro. In principle, each section corresponds to a logically independent functional component. This not only enables fine-grained, component-aware resource allocation, but also allows sections to execute concurrently to the greatest possible extent. However, certain scenarios demand more sophisticated construction strategies. For instance, in Kullback-Leibler (KL) divergence loss based distillation training (Gou et al., 2021), the teacher model needs to provide logits tensor to the student model for loss computation. The size of a logits tensor is determined by the vocabulary size and sequence length, whereas the hidden state tensor—used to compute these logits in the final output layer—is governed by the hidden dimension and sequence length. Since the vocabulary size is often an order of magnitude (or more) larger than the hidden dimension, a preferred section construction strategy is to colocate the student model with the final output layers of all teacher models within the same section as shown in Figure 4. This way, during execution, the student only needs to fetch the compact hidden states from the teacher models, rather than the much larger logits, thereby significantly alleviating cross-section communication pressure. Another scenario arises in omni-modal model training. Since samples containing both image and audio modalities typically constitute a relatively small proportion of the dataset, the corresponding encoders are often activated in a mutually exclusive manner during computation. Given that the audio and image encoders have comparable model sizes, they can be co-located within the same section to improve device utilization and minimize resource fragmentation.
Figure 4: Section construction in knowledge distillation training. 3.2
Section Hyper-Parameters Optimization
After the first decomposition step, each section s ∈ S is then assigned a tailored training configuration C s = {DP s , T P s , P P s , CP s , mbss }, specifying its degrees of data, tensor, pipeline, and context parallelism, along with its micro-batch size. In existing frameworks (Shoeybi et al., 2019), all sections always share the same data-parallel degree. However, this always leads to a fundamental throughput imbalance, existing among these sections. To illustrate this issue, consider a knowledge distillation workload: the teacher model performs only the forward pass to generate soft labels and thus achieves higher computational efficiency compared to the student model, which executes both forward and backward passes, when both models operate under their respective optimal configurations. Consequently, the faster sections (e.g., the teacher) often suffer from underutilized resources. To bridge this gap and maximize resource utilization, Maestro introduces a fan-out mechanism that decouples their data-parallel execution while preserving correctness. As illustrated in Figure 5, each data-parallel rank in the teacher section processes k = 2 input samples and distributes the resulting outputs to k distinct DP ranks in the student section. This allows the student section to consume one sample per DP rank at its natural pace, while the faster teacher section achieves an effective throughput with a lower GPUbudget. The fan-out enforces a precise relationship between their parallelism degrees: DP tec =
DP stu . fanout
(1)
Therefore, the training configuration turns to be C s = {DP s , T P s , P P s , CP s , mbss , fanouts } based on this mechanism. To maximize overall throughput, the system must allocate resources according to the optimal combination of these per-section configurations. This motivates formulating the problem as an optimization: find {C s } that minimizes the end-to-end iteration time f ({C s }), subject to hardware resource, memory and hyper-parameters constraints: 5
Figure 5: Illustration of fan-out mechanism.
• Resource constraint:
X
N s ≤ N GPUs ,
s∈S
• Memory constraint: maxs Mgs ≤ M GPU , ∀s ∈ S, g∈G
• fanout constraint: DP f r × fanout = DP sr , where (f r, sr ) ∈ E in G(S, E ), where N s denotes the number of GPUs allocated to section-s; G s is the set of GPUs assigned to section-s (with |G s | = N s ), and Mgs represents the total memory consumption on GPU g when executing section-s, including model parameters, optimizer states, activations, and gradients; f r, sr ∈ S denote two sections connected by a directed data-flow edge in the section dependency graph G = (S, E ) , with f r being the faster section (e.g., teacher) and sr the slower section (e.g., student). Formally: min
f ({C s })
s.t.
X
{C s }
N s ≤ N GPUs , (2)
s∈S
maxs Mgs ≤ M GPU , g∈G
∀s ∈ S,
DP f r × fanout = DP sr ,
∀(f r, sr ) ∈ E.
However, directly solving Equation 2 is infeasible in practice. Although the valid parallelism degrees for each section are restricted to integer divisors of model-specific structural parameters (e.g., hidden dimension, number of attention heads, and sequence length), the joint configuration space across all sections remains combinatorial due to the multiplicative interaction of per-section choices. Furthermore, every candidate configuration must simultaneously satisfy global resource limits and per-device memory constraints. Given the exponential growth of feasible combinations and the high cost of evaluating each configuration, exhaustive or naive search quickly becomes prohibitively expensive, necessitating a principled and scalable optimization approach. To overcome these challenges, we start from a key observation: in compound LLM workloads—such as multimodal training or distillation training—system throughput is overwhelmingly bottlenecked by a critical section, which incurs the highest computational cost per sample or consumes the largest fraction of end-to-end execution time (e.g., the LLM in vision–language training or the student model in distillation). This critical section effectively defines the critical path of the workload: in an ideal scenario where it operates at peak efficiency without stalling, its throughput establishes a fundamental upper bound for the entire pipeline. Guided by this insight, we adopt a hierarchical optimization strategy that prioritizes the critical path: first, we configure the critical section to approach its theoretical throughput limit; then, we adapt other auxiliary sections to ensure they neither stall nor exert backpressure on the critical path. Specifically, our solution proceeds in two stages: 1. Stage 1 (Critical-first): Find a near-optimal configuration C crit for the critical section that maximizes its computational efficiency (e.g., MFU), subject to per-GPU memory and hardware constraints. 2. Stage 2 (Auxiliary-adaptive): Given C crit and a chosen fanout, configure each auxiliary section saux with the minimal GPU count such that its computation fully overlaps with the critical section and introduces no stalls. 6
This two-stage decomposition offers two key benefits. First, it dramatically reduces the complexity of solving the original joint optimization problem. By decoupling the search into a sequence of smaller subproblems, the combinatorial explosion across sections is avoided. Once the near-optimal configuration for the critical section is identified, each auxiliary section can be configured independently and efficiently, without revisiting the global search space. Second, the approach achieves near-optimal end-to-end throughput. Because pipeline performance is fundamentally bounded by the critical path, maximizing the efficiency of the critical section brings the system close to its theoretical throughput limit. Resources allocated to auxiliary sections are kept to the minimum necessary to avoid stalls, ensuring that any overhead remains negligible. As a result, our method delivers high system efficiency while remaining computationally tractable. 3.3
Section Communication
Figure 6: Illustration of data resharding and communication across sections.
Following the steps above, managing tensor transfers across sections becomes essential. Since each section may adopt heterogeneous parallelism configurations—particularly in tensor parallelism (TP) and context parallelism (CP)—along with distinct fan-out mechanisms, correctly and efficiently handling data resharding is critical. Moreover, as section executions are highly asynchronous with respect to one another, the communication process must not interfere with GPU computation and should achieve effective overlap with kernel execution to maximize hardware utilization. To satisfy this requirement, we design and implement a high-performance, asynchronous, asymmetric message queue that supports the M-to-N communication pattern—a novel paradigm distinct from conventional point-to-point or collective communication in LLM training scenarios—to accommodate heterogeneous TP/CP configurations across section boundaries. During communication establishment, the message queue decouples M-to-N communication into multiple point-to-point channels. Each channel comprises a CPU subchannel for metadata (e.g., tensor shape, section name, position within the TP/CP group, etc.) and a GPU subchannel for tensor data; both employ one-sided operations to enable asynchronous communication with negligible CPU/GPU interference. At runtime, each sender invokes the push API to transmit a tensor to remote receivers. The MessageQueue first leverages the CPU subchannel to exchange metadata and reserve a slot in the destination device’s memory; the sender GPU then pushes tensor data directly without awaiting the receiver’s participation. On the receiving side, the pull API dequeues the earliest message. When multiple senders contribute to a single tensor, the API automatically gathers the sharded fragments. Finally, the receiver updates local memory statistics to facilitate subsequent requests. With this MessageQueue, we can efficiently accommodate diverse cross-section communication patterns—such as fan-out communication among data-parallel ranks and M-to-N transfers across heterogeneous TP/CP parallelism domains. 3.4
Section Computation Scheduling
While section-tailored training strategies and fine-grained resource allocation form the foundation of Maestro’s efficiency, the decisive factor in unlocking its full performance potential lies in wavefront scheduling—a mechanism that maximizes inter-section parallelism while minimizing idle time induced 7
by cross-section data dependencies. During each training iteration, given a global batch of samples and the resulting constructed sections, Maestro first analyzes the sections that each sample will activate. In principle, one could globally search for an optimal sample ordering that minimizes the aggregated idle time across all sections. However, this combinatorial optimization is computationally intractable for practical workloads. As we noted earlier, in most scenarios we have encountered, the critical section—the bottleneck stage that dominates the execution timeline—has a decisive impact on overall training efficiency. Thus, we design our scheduling policy to prioritize the critical section: it is never allowed to stall waiting for forward or backward passes from other sections. This ensures the critical section remains continuously saturated, thereby maximizing overall training throughput. Guided by the critical-section-first principle, a 6-tuple Sample(t_f _bc, t_f _c, t_f _ac, t_b_bc, t_b_c, t_b_ac) is adopted by Maestro to model each sample, where t_f _bc, t_f _c, and t_f _ac denote the execution times before, within, and after the critical section during the forward pass, respectively; t_b_bc, t_b_c, and t_b_ac denote the corresponding times during the backward pass. We implement a heuristic scheduling algorithm that produces a near-optimal schedule in O (N 2 ) time complexity where N denotes the number of samples per DP rank. In practical training scenarios with data parallelism, each DP rank typically processes tens to hundreds of samples. Moreover, since the scheduling process could fully overlaps with GPU execution, the O (N 2 ) complexity introduces negligible overhead to end-to-end training latency. In detail, the scheduling process operates in two phases as shown in Algorithm 1: First, it sorts all samples in ascending order of t_f _bc—prioritizing those that reach the critical section earliest—and initializes the result schedule with the top-ranked sample. Subsequently, for each remaining sample, the algorithm evaluates all feasible insertion positions within the partially constructed result schedule. For each candidate position, it simulates the end-to-end execution timeline across all sections to compute the resulting makespan (total execution time), with inter-section overlap explicitly modeled; the algorithm then commits the insertion that minimizes this metric. This greedy insertion heuristic effectively approximates critical-path scheduling: by prioritizing samples with minimal t_f _bc and dynamically selecting insertion points that suppress pipeline bubbles, it maintains continuous saturation of the critical section while opportunistically packing non-critical work into idle slots. The approach thus achieves near-optimal throughput without incurring the combinatorial cost of exhaustive search. When the data parallelism degree exceeds 1, Maestro first partitions the global batch across DP ranks with the goal of balancing the distribution of activated sections. Each DP rank then independently executes the scheduling algorithm described above. For sections with fanout > 1, each DP rank merges the per-rank schedules from the fanout DP ranks of its downstream section via round-robin interleaving. This ensures fair progression across parallel branches and prevents starvation of any downstream consumer, which is critical for maintaining balanced pipeline occupancy under heterogeneous data dependencies. Algorithm 1 SectionSchedulingAlgorithm 1: Given Samples = List[(sample_idx, t_f _bc, t_f _c, t_f _ac, t_b_bc, t_b_c, t_b_ac)]. 2: InitialSampleOrder = SortByF orwardT imeBef oreCriticalSection(Samples). 3: ResultOrder = [InitialSampleOrder [0]] 4: for Sample ∈ InitialSampleOrder [1 :] do 5: best_position, best_makespan = 0, IN F 6: for position ∈ Range(0, length(ResultOrder ) + 1) do 7: current_makespan = calculate_makespan(ResultOrder, position, Sample) 8: if current_makespan < best_makespan then 9: best_makespan, best_position = current_makespan, position 10: end if 11: insert_sample(ResultOrder, best_position, Sample) 12: end for 13: end for
We illustrate our approach using a vision-language model (VLM) training example, as shown in Figure 7. In this workload, two sections are constructed: the ViT section and the LLM section. The ViT section has a fanout of 4, meaning each data-parallel replica in the ViT section supplies data to four replicas in the LLM section. With a global batch size of 12, each LLM replica processes 3 samples per iteration. Clearly, the LLM section is the critical section in this scenario. The samples in their original order are represented in Sample(t_f _bc, t_f _c, t_f _ac, t_b_bc, t_b_c, t_b_ac) as: (0.1, 1, 0, 0, 2, 0.2), (0, 1, 0, 0, 2, 0), (0, 1, 0, 0, 2, 0), and (0.2, 1, 0, 0, 2, 0.4).... Here, t_f _bc and t_b_ac actually describe the forward and backward execution time in the ViT encoder respectively; a value of t_f _bc = 0 indicates a text-only sample that bypasses the ViT section and is fed directly into the LLM section. Since no additional modules follow the LLM backbone, all samples have t_f _ac = 0. Applying Algorithm 1 to samples on each DP rank and merging the resulting schedules at the ViT section yields the execution order shown in Figure 7. We can 8
see that under this ordering, the LLM section remains continuously saturated—never stalling to await forward or backward passes from the ViT section. Consequently, ViT computation is fully overlapped with LLM execution, contributing zero critical-path overhead to end-to-end training throughput.
Figure 7: Data scheduling for VLM training with fanout = 4 over a global batch of 12 samples.
4
Evaluations and Case Study
We benchmark Maestro’s efficacy on two representative production-scale workloads that embody distinct forms of heterogeneity: (1) Multimodal training of the recently released Qwen3.5-400B-A17B and Qwen3Next-80B-A3B models on a dataset with a 32K sequence length—exemplifying static heterogeneity (vision encoder vs. LLM backbone) and dynamic heterogeneity (text-only vs. text-image sample activation); and (2) KL divergence-based knowledge distillation, where a frozen Qwen3.5-400B-A17B teacher guides the training of a Qwen3-Next-80B-A3B student (Qwen3-Team, 2026)—epitomizing cross-model execution asymmetry (forward-only teacher vs. full backpropagation student). All experiments are conducted on Alibaba Cloud’s large-scale GPU clusters. We adopt Megatron-LM (Shoeybi et al., 2019)—the foundation upon which Maestro is built—as our primary baseline to ensure a fair comparison under identical lowlevel optimizations. This baseline represents the state-of-the-art in homogeneous parallel training and serves as the de facto standard for large-scale LLM training systems. In all experiments, we ensure that the critical section’s resource allocation remains identical to the baseline configuration to enable a fair comparison. Specifically, for VLM training where the baseline employs 512 GPUs, we allocate exactly 512 GPUs to Maestro’s LLM section and provision additional resources exclusively for the ViT section. Similarly, in distillation experiments, the student section is allocated the same GPU count as the baseline, with extra resources dedicated to the teacher section. We evaluate training performance using two primary metrics: (1) end-to-end token throughput, and (2) per-gpu token throughput—the latter isolates the efficiency gain attributable to Maestro’s sectioning mechanism by factoring out scale effects. In the VLM training experiments, we further measure relative efficiency against text-only training to quantify whether the performance overhead introduced by the ViT encoder on the LLM backbone has been fully eliminated. Notably, this methodology—comparing multimodal throughput to text-only baselines—has also been adopted by recent community efforts including Kimi-K2.5 (Bai et al., 2026) and LongCat-Flash-Omni (Team, 2025).
9
4.1
Vision-Language Model Training
We first evaluate Maestro on training two recently released models, the Qwen3.5-400B-A17B and the Qwen3-Next-80B-A3B on a 32K-sequence lengthed multimodal dataset. Qwen3.5-400B-A17B is a native multimodal model, whereas for Qwen3-Next-80B-A3B we integrate the ViT encoder from Qwen3.5400B-A17B to evalute multimodal training at this model scale. The models contain two structurally and computationally asymmetric components: a ViT encoder for image feature extraction and an LLM backbone for cross-modal reasoning. In Maestro, these map naturally to two distinct logical sections—the ViT section and the LLM section. The ViT processes raw image patches into visual tokens, which undergo a 4:1 downsampling before concatenation with text tokens to form the complete input for the LLM backbone, as illustrated in Figure 1. Consequently, the ViT operates on a significantly longer token sequence than the effective visual context ultimately consumed by the LLM. To accommodate this disparity, we employ section-specific parallelization strategies: the ViT section primarily leverages context parallelism to efficiently handle its long input sequences, while the LLM section adopts its own optimal parallelism configuration—tailored to its model scale and sequence characteristics. Besides, in practice, training VLM typically requires careful curation of the vision-to-text data mix. For instance, Kimi-K2.5 (Bai et al., 2026) adopts a 1:9 vision-to-text sample ratio, while LongCat-Flash-Omni (Team, 2025) uses a 1:2 ratio. This inherent modality skew creates a natural opportunity for cross-section computation overlap. With our wavefront scheduling mechanism, we can fully hide the computational cost of the ViT section—effectively eliminating its contribution to the end-to-end training time.
Figure 8: Maestro performance on multimodal training. In both experiments, the LLM backbone uses the best available parallelism strategy and allocates an additional 12.5% of resources to the ViT section. Specifically, Qwen3.5-400B-A17B is trained with a pipeline parallel size of 4, whereas Qwen3-Next-80B-A3B is trained without pipeline parallelism. As shown in Figure 8, Maestro achieves approximately 1.4× higher end-to-end training throughput than Megatron-LM on Qwen3.5-400B-A17B and 1.20× higher throughput on Qwen3-Next-80B-A3B. Moreover, even when accounting for the additional resources allocated to the ViT section, Maestro still delivers 1.24× and 1.067× higher per-gpu throughput, respectively—demonstrating improved resource utilization. Finally, both experiments achieve 100% relative training efficiency with regard to end-to-end throughput compared to text-only training, confirming that the performance overhead of the ViT section has been completely eliminated. 4.2
Distillation
We further evaluate a knowledge distillation scenario where Qwen3.5-400B-A17B serves as the teacher model and distills knowledge to the Qwen3-Next-80B-A3B student model via KL divergence loss. This training paradigm requires the frozen teacher to transmit logits to the student during the forward pass. Given the stark execution asymmetry—forward-only teachers versus full backpropagation for the student—one could naturally partition the workload into distinct sections: one section per teacher model and a separate section for the student. However, as noted earlier, the logits tensor can be orders of magnitude larger than the hidden state from which it is computed in the teacher model’s output layer. For instance, in Qwen3.5-series models (), the vocabulary size is 250K while the hidden dimension is merely 4K—rendering the logits tensor 62.5× larger in size. To circumvent the prohibitive inter-section communication overhead of transferring such voluminous tensors, we colocate the teacher’s final output layer with the student model within the same logical section, enabling compact hidden-state transfer instead. 10
Furthermore, because the teacher section performs only forward passes (with frozen parameters), increasing its micro-batch size (MBS) substantially boosts throughput with negligible memory overhead. For instance, as shown in Figure 9, scaling the teacher MBS from 1 to 4 achieves a 2.6× throughput improvement while the peak memory consumption remains nearly flat. This insight enables fan-out execution: a single DP rank in the teacher section can concurrently serve multiple downstream DP ranks in the student section. Consequently, Maestro reduces the additional GPU allocation required for the teacher without compromising end-to-end throughput—effectively improving resource utilization.
Figure 9: Normalized throughput and peak memory of the teacher model across micro-batch sizes. The experiments allocate an additional 25% GPU resources to the teacher section relative to the baseline. As shown in Figure 10, Maestro achieves 1.75× higher end-to-end token throughput and 1.4× higher per-gpu token throughput compared to Megatron-LM—demonstrating simultaneous improvements in both absolute training performance and resource efficiency.
Figure 10: Maestro performance on distillation training.
5
Related Work
5.1
LLM Training
Data Parallelism. In data parallelism (Li et al., 2020), the entire model, including all parameters, is replicated identically across all participating devices. During training, the global mini-batch is partitioned into equal-sized sub-batches, with each device processing one sub-batch independently through the full forward and backward passes. After the backward pass completes on each device, the computed gradients for all parameters are synchronized across the device group using collective operations, ensuring that every replica accumulates the sum of gradients over the entire mini-batch. The optimizer then applies identical parameter updates on all devices, maintaining model consistency. Based on this, ZeRO (Rajbhandari et al., 2020) further partitions optimizer states, gradients, and model parameters within the data-parallel group to reduce per-rank memory footprint. Tensor Parallelism. Tensor parallelism (TP) (Shoeybi et al., 2019; Chang et al., 2024), also known as model parallelism, partitions individual model layers (e.g., attention heads or feed-forward networks) 11
across multiple devices. For instance, the weight matrices of linear layers are split column-wise or rowwise, and intermediate activations are communicated via All-Gather or Reduce-Scatter operations. TP reduces per-device memory usage and enables training of extremely wide layers, but introduces finegrained communication between devices at every layer, which can become a bottleneck on low-bandwidth interconnects. Pipeline Parallelism. Pipeline parallelism (PP) (Huang et al., 2019; Narayanan et al., 2019) partitions a model vertically into multiple sequential stages, each assigned to a dedicated device or group of devices. To overlap computation across stages, the global mini-batch is divided into smaller micro-batches that flow through the pipeline in a staggered manner—enabling concurrent execution and improving hardware utilization. A line of work (Huang et al., 2019; Narayanan et al., 2019; 2021; Qi et al., 2024) has focused on designing advanced scheduling strategies that optimize throughput, memory footprint, and pipeline occupancy. Expert Parallelism. Expert parallelism (EP) (Singh et al., 2023; Cai et al., 2024) is specialized for sparse mixture-of-experts architectures, where each input token is routed to a small subset of expert subnetworks out of a large pool. In EP, experts are distributed across devices, and each device hosts a disjoint subset of experts. During the forward pass, tokens are dispatched to the devices hosting their selected experts; after expert computation, results are gathered back to the originating device. EP minimizes redundant computation and memory footprint by ensuring that only activated experts are executed and stored, but incurs dynamic communication costs due to data-dependent routing. It is often combined with DP and TP to scale MoE models efficiently. 5.2
MLLM Training
Multimodal training extends LLM training by incorporating additional, typically smaller modality encoders. This introduces model heterogeneity: the encoder and LLM decoder exhibit vastly different compute and memory demands, necessitating distinct parallelization strategies for each component. Moreover, multimodal inputs bring dynamic data heterogeneity—such as variable sequence lengths in visual tokens—which causes fluctuating per-sample computation times. In pipeline parallelism, this variability leads to dynamic, data-dependent pipeline bubbles, significantly degrading throughput and complicating scheduling. Optimus (Feng et al., 2025) addresses model heterogeneity by prioritizing the LLM backbone: it allocates GPU resources to the LLM first and then fits the encoders into the remaining memory budget. This design inherently bottlenecks encoder throughput, as they must operate under a severely constrained memory budget dictated by the LLM’s dominant footprint. Recently, DistTrain (Zhang et al., 2025b) proposes inter- and intra-microbatch reordering algorithms to balance computation across data-parallel groups and reduce pipeline bubbles caused by variable visual data lengths, respectively. However, its focus is limited to alleviating imbalance within pipeline parallelism by minimizing PP bubbles. Similarly, PipeWeaver (Chen et al., 2025) addresses data-length dynamicity by searching for pipeline schedules tailored to the current training batch.
6
Conclusion
As model architectures evolve from monolithic layer stacks to composites of heterogeneous functional components, the foundational assumption of workload uniformity—upon which conventional training frameworks are built—is violated. This breakdown leads to suboptimal training efficiency and underutilized resources. The problem is further exacerbated when these components are data-dependent activated, as runtime irregularities amplify performance degradation. To address this emerging class of compound workloads, we propose Maestro. Maestro decomposes the original computation graph into distinct sections, enabling each section to adopt a tailored training strategy for fine-grained resource allocation. To mitigate runtime irregularities caused by sparse section activation, we further introduce a wavefront scheduling algorithm that maximizes cross-section parallelism and minimizes stall time due to inter-section data dependencies. Experiments on compound workloads—such as native multimodal training and knowledge distillation—demonstrate that Maestro significantly improves both cluster-wide training throughput and per-GPU utilization. In the VLM training scenario, Maestro achieves 100% relative performance compared to text-only training, fully eliminating the performance overhead of the additional ViT module through section computation scheduling. We share the design and implementation of Maestro in the hope of empowering the community to run next-generation compound LLM workloads with optimal performance.
References Inclusion AI, Biao Gong, Cheng Zou, Chuanyang Zheng, Chunluan Zhou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Dandan Zheng, Fudong Wang, et al. Ming-omni: A unified multimodal model for 12
perception and generation. arXiv preprint arXiv:2506.09344, 2025. Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai. meta. com/blog/llama-4-multimodal-intelligence/, 2025. Kimi Team Yifan Bai, Yifan Bai, Yiping Bao, S. H. Cai, Yuan Cao, Y. Charles, H. S. Che, Cheng Chen, Guanduo Chen, Huarong Chen, Jia Chen, Jiahao Chen, Jianlong Chen, Jun Chen, Kefan Chen, Liang Chen, Ruijue Chen, Xinhao Chen, Yanru Chen, Yanxu Chen, Yicun Chen, Yimin Chen, Yingjiang Chen, Yuankun Chen, Yujie Chen, Yutian Chen, Zhirong Chen, Ziwei Chen, Dazhi Cheng, Min-Rong Chu, Jialei Cui, Jiaqi Deng, Muxi Diao, Haochen Ding, Mengfan Dong, Meng-Wei Dong, Yuxin Dong, Yuhao Dong, Angang Du, Chenzhuang Du, Dikang Du, Lingxiao Du, Yulun Du, Yu Fan, Shengjun Fang, Qiulin Feng, Yichen Feng, Garimugai Fu, Kelin Fu, Hongcheng Gao, Tong Gao, Yu-Zheng Ge, Shangyi Geng, Chengyang Gong, Xiaochen Gong, Zhuoma Gongque, Qizheng Gu, Xinran Gu, Yi-Chao Gu, Longyu Guan, Yuanying Guo, Xiaoru Hao, Weiran He, Wen-Lin He, Yunjia He, Chao Hong, Hao-Xing Hu, Jiaxi Hu, Yan-Ni Hu, Zhenxing Hu, Ke Huang, Ruiyuan Huang, Weixiao Huang, Zhiqi Huang, Tao Jiang, Zhejun Jiang, Xinyi Jin, Yu Jing, Guokun Lai, Aidi Li, C. Li, Cheng Li, Fang Li, Guanghe Li, Guanyu Li, Haitao Li, Haoyang Li, Jia Li, Jingwei Li, Junxiong Li, Lincan Li, Mo Li, Weihong Li, Wentao Li, Xinhang Li, Xinhao Li, Yang Li, Yanhao Li, Yiwei Li, Yuxiao Li, Zhaowei Li, Zheming Li, Weilong Liao, Jiawei Lin, Xiaohan Lin, Zhishan Lin, Zi-Cong Lin, Cheng Liu, Chenyu Liu, Hongzhang Liu, Liang Liu, Shaowei Liu, Shudong Liu, Shuran Liu, Tian-Bo Liu, Tianyu Liu, Weizhou Liu, Xiangyan Liu, Yangyang Liu, Yanming Liu, Yibo Liu, Yuanxin Liu, Yue Liu, Zhengying Liu, Zhongnuo Liu, Enzhe Lu, Haoyu Lu, Zhiyuan Lu, Junyu Luo, Tongxu Luo, Yashuo Luo, Long Ma, Yingwei Ma, Shaoguang Mao, Yuan Yuan Mei, Xinhong Men, Fanqing Meng, Zhiyong Meng, Yibo Miao, Mi Ni, Kun Ouyang, Siyuan Pan, Bo Pang, Yuchao Qian, Ruoyu Qin, Zeyu Qin, Jiezhong Qiu, Bowen Qu, Zeyu Shang, Youbo Shao, Tianxiao Shen, Zhennan Shen, Juanfeng Shi, Lidong Shi, Sheng-Peng Shi, Feifan Song, Pengwei Song, Tianhui Song, Xiaoxi Song, Hongjin Su, Jianlin Su, Zhaochen Su, Linpeng Sui, Jinsong Sun, Junyao Sun, T. Sun, Flood Sung, Yunpeng Tai, Chuning Tang, Heyi Tang, Xiao Yu Tang, Zhe-Chen Tang, Jiawen Tao, Shiyuan Teng, Chaoran Tian, Pengfei Tian, Ao Wang, Bowen Wang, Chensi Wang, Chuang Wang, Congcong Wang, Dingkun Wang, Dinglu Wang, Dongliang Wang, Feng Wang, Hailong Wang, Haiming Wang, Heng Wang, Huaqing Wang, Hui Wang, Jiahao Wang, Jinhong Wang, Jiuzheng Wang, Kaixin Wang, Linian Wang, Qibin Wang, Shengjie Wang, Shuyi Wang, Si Wang, Wei Wang, Xiaochen Wang, Xinyuan Wang, Yao Wang, Yejie Wang, Yipu Wang, Yiqin Wang, Yucheng Wang, Yuzhi Wang, Zhaoji Wang, Zhaowei Wang, Zhengtao Wang, Zhexu Wang, Zihan Wang, Zizhe Wang, Chu Wei, Ming Wei, Chuan Wen, Zichen Wen, Chengjie Wu, Haoning Wu, Junyan Wu, Rucong Wu, Wenhao Wu, Yuefeng Wu, Yuhao Wu, Yuxin Wu, Zijian Wu, Chenjun Xiao, Jin Xie, Xiao-Ming Xie, Yuchong Xie, Yifei Xin, Bo-Kai Xing, Boyu Xu, Jianfan Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinbo Xu, Xinran Xu, Yangchuan Xu, Yichang Xu, Yuemeng Xu, Zelai Xu, Zi-Yang Xu, Junjie Yan, Yuzi Yan, Guang-Fu Yang, Hao Yang, Junwei Yang, Kai Yang, Ning Yang, Ruihan Yang, Xiaofei Yang, Xinlong Yang, Ying Yang, Yi Yang, Zhen Yang, Zhilin Yang, Zonghan Yang, Haotian Yao, Dan Ye, Wen guang Ye, Zhuorui Ye, Bohong Yin, Chengzhen Yu, Long Yu, Tao Yu, Tianxiang Yu, Enming Yuan, Mengjie Yuan, Xiao feng Yuan, Yang Yue, Weihao Zeng, Dunyuan Zha, Haobing Zhan, Dehao Zhang, Hao Zhang, Jin Zhang, Puqi Zhang, Qiao Zhang, Rui Zhang, Xiaobin Zhang, Y. Zhang, Yadong Zhang, Yangkun Zhang, Yichi Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yushun Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Chenguang Zhao, Feifan Zhao, Jinxian Zhao, Shuai Zhao, Xiangyu Zhao, Yikai Zhao, Zijia Zhao, Huabin Zheng, Ruihan Zheng, Shao-Jiang Zheng, Tengyang Zheng, Jun-Xing Zhong, Longguang Zhong, Weiming Zhong, M. Zhou, Runjie Zhou, Xinyu Zhou, Zaida Zhou, Jinguo Zhu, Liya Zhu, Xinhao Zhu, Yuxuan Zhu, Zhengxin Zhu, Jingze Zhuang, Weiyu Zhuang, Ying Zou, and Xinxing Zu. Kimi k2.5: Visual agentic intelligence. 2026. URL https://api.semanticscholar.org/CorpusID:285269548. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report, 2025. URL https://arxiv.org/abs/2511.21631. Weilin Cai, Juyong Jiang, Le Qin, Junwei Cui, Sunghun Kim, and Jiayi Huang. Shortcut-connected expert parallelism for accelerating mixture-of-experts. arXiv preprint arXiv:2404.05019, 2024. Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Chengji Yao, Ziheng Jiang, et al. Flux: Fast software-based communication overlap on gpus through kernel fusion. arXiv preprint arXiv:2406.06858, 2024. 13
Xing Chen, Zhenliang Xue, Zeyu Mi, Hanpeng Hu, Yibo Zhu, Daxin Jiang, Yimin Jiang, Yixin Song, Yubin Xia, and Haibo Chen. Pipeweaver: Addressing data dynamicity in large multimodal model training with dynamic interleaved pipeline. arXiv preprint arXiv:2504.14145, 2025. v1 [cs.DC] 19 Apr 2025. Referenced in Maestro.pdf as [3]. Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng, Haibin Lin, and Minlan Yu. Optimus: Accelerating {Large-Scale}{Multi-Modal}{LLM} training by bubble exploitation. In 2025 USENIX Annual Technical Conference (USENIX ATC 25), pp. 161–177, 2025. Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International journal of computer vision, 129(6):1789–1819, 2021. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019. Jonas Hubotter, Frederike Lubeck, Lejs Deen Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation. 2026. URL https://api.semanticscholar.org/CorpusID:285102353. Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020. Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM symposium on operating systems principles, pp. 1–15, 2019. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the international conference for high performance computing, networking, storage and analysis, pp. 1–15, 2021. Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. Zero bubble (almost) pipeline parallelism. In The Twelfth International Conference on Learning Representations, 2024. Qwen3-Team. Qwen3-next-80b-a3b. https://qwen.ai/blog?from=research.latest-advancements-l ist&id=4074cca80393150c248e508aa62983f9cb7d27cd&, 2026. Accessed: 2026-02-10. Qwen3-VL-Team. Qwen3-vl. https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct, 2026. Accessed: 2026-02-05. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–16. IEEE, 2020. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, and Abhinav Bhatele. A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training. In Proceedings of the 37th International Conference on Supercomputing, pp. 203–214, 2023. Meituan LongCat Team. Longcat-flash-omni technical report. ArXiv, abs/2511.00279, 2025. URL https: //api.semanticscholar.org/CorpusID:282740018. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Yuchen Duan, Xuehui Wang, Songze Li, Xiangyu Zhao, Haodong Duan, Nianchen Deng, Bin Fu, Yinan He, Yi Wang, Conghui He, Botian Shi, Junjun He, Ying Xiong, Han Lv, Lijun Wu, Wenqi Shao, Kai Zhang, Hui Deng, Biqing Qi, Jiaye Ge, Qipeng Guo, Wenwei Zhang, Yuzhe Gu, Wanli Ouyang, Limin Wang, Min Dou, Xizhou Zhu, Tong Lu, Dahua Lin, Jifeng Dai, Bowen Zhou, Weijie Su, Kaiming Chen, Yu Qiao, Wenhai Wang, and Gen Luo. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. ArXiv, abs/2508.18265, 2025. URL https://api.semanticscholar.org/CorpusID:280710824. 14
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. ArXiv, abs/2503.20215, 2025a. URL https://api.semanticscholar.org/CorpusID:277322543. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xianzhong Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Ke Chen, Xue Lian Liu, Peng Wang, Ming Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jing-Jun Zhou, and Junyang Lin. Qwen3-omni technical report. ArXiv, abs/2509.17765, 2025b. URL https://api.semanticscholar.org/CorpusID:281420796. Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang, Ligeng Zhu, Yuanhang Su, Sean Lin, AnChieh Cheng, Zhen Wan, Jinchuan Tian, Yuming Lou, Dong Yang, Zhijian Liu, Yukang Chen, Ambrish Dantrey, Ehsan Jahangiri, Sreyan Ghosh, Da fu Xu, Ehsan Hosseini-Asl, Danial Mohseni Taheri, Vidya Murali, Sifei Liu, Jason Lu, Oluwatobi Olabiyi, Frank Wang, Rafael Valle, Bryan Catanzaro, Andrew Tao, Song Han, Jan Kautz, Hongxu Yin, and Pavlo Molchanov. Omnivinci: Enhancing architecture and data for omni-modal understanding llm. ArXiv, abs/2510.15870, 2025. URL https://api.semanticsc holar.org/CorpusID:282203145. Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, et al. Jenga: Effective memory management for serving llm with heterogeneity. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, pp. 446–461, 2025a. Zili Zhang, Yinmin Zhong, Yimin Jiang, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, Daxin Jiang, and Xin Jin. Disttrain: Addressing model and data heterogeneity with disaggregated training for multimodal large language models. In Proceedings of the ACM SIGCOMM 2025 Conference, pp. 24–38, 2025b. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Selfdistilled reasoner: On-policy self-distillation for large language models. 2026. URL https://api.sema nticscholar.org/CorpusID:285050713.
15