HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters
Chong Li1 , Zhengdao Yu1 , Nelson Lossing1 , Thibaut Tachon1 , Pierre Leca1 , Etienne Filhol1 , Yujie Yuan1 , Chong Bao2 , Teng Su2 1
arXiv:2607.03229v1 [cs.DC] 3 Jul 2026
2
Huawei Fourier Research Center, Paris, France Huawei Technologies Co., Ltd, Hangzhou, China Abstract
Modern AI applications have expanded beyond text-only interaction into a wide range of multimodal scenarios, making multimodal large language models (MLLMs) crucial for both research and industry. However, compared with traditional decoder-only LLM training, large-scale MLLM training often shows much lower MFU. We analyze the key pain points in MLLM training and introduce Mpipe, which uses a schedule algebra to derive concrete runtime behavior from a compact schedule specification. From this algebra, Mpipe derives transpose, a multimodal-aware heterogeneous parallel schedule that remaps modality-encoder computation into otherwise idle pipeline regions. On Ascend 910C NPU clusters, Mpipe achieves 2.70× speedup in a small-scale setting and 1.21× speedup in a 512-card large-scale setting.
1. Introduction
chased at runtime, by per-iteration schedule search (Xue et al., 2026b) or data-dependent parallelism (Xue et al., 2026a). It can instead be relocated off the critical path into the pipeline’s intrinsic warmup bubbles by a single static placement. We propose Mpipe, which formalizes parallel schedules as a small composable algebra and derives transpose, a schedule that places encoder computation in those bubbles, as one point in that algebra. The resulting schedule stays static and invariant to the modality mix, with no runtime scheduling overhead. Our contributions are:
Modern AI applications are no longer limited to text-only interaction. In many widely used scenarios, such as visual question answering, document understanding, video analysis, embodied agents, and content generation, models are expected to jointly understand language together with images, videos, and other modal inputs. This trend has made multimodal large language models (MLLMs) an increasingly important direction in both research and industry. As model parameters rapidly scale, training frontier models increasingly depends on large accelerator clusters. For example, Llama 3 reports production training on clusters with up to 16K GPUs (Grattafiori et al., 2024). MLLMs further increase this demand beyond decoder-only LLMs. In addition to the language backbone, they must process high-resolution images, videos, and other modality inputs through extra model components, which increases computation, memory footprint, and data movement.
• A schedule algebra for parallel training. We formalize parallel pipeline schedules as a small composable algebra in which a schedule is a list of perregion skeletons, and a single derivation maps a cut, schedule, and model to concrete runtime behavior: device placement, collective communication, and execution order, from which we read a cost model that predicts the step makespan. We prove a backwardfootprint lemma and a schedule-invariance corollary: one schedule is correct whether the modality encoder is frozen or trained.
However, large-scale MLLM training often achieves much lower MFU (model FLOPs utilization) than decoder-only LLM training (Zhang et al., 2025). Figure 1 compares the MFU of representative LLM and MLLM training workloads.
• Transpose. From this algebra Mpipe derives transpose, a static schedule that replicates the encoder across pipeline ranks, runs it inside the warmup bubbles, and gathers its output into the first LLM stage. The placement is decided once, adds no runtime scheduling overhead, and is invariant to the modality mix.
The dominant source of this gap is the modality encoder, which sits at the input side of the pipeline: its highly variable work depends only on data available before the LLM pipeline begins. Such source-side variance need not be 1
HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters Table 1. Approximate component heterogeneity under one MLLM training configuration. ViT computation is measured with visual length fixed to 4× the LLM backbone sequence length, where the LLM backbone sequence length is 8K.
MFU (Model FLOPs Utilization) 60% 41%
44.57% Component
40%
LLM backbone ViT encoder
20%
20%
13.5%
0% LLM LLaMA3
LLM DeepSeek
MLLM QwenVL2
Parameter memory (per layer)
Activation memory (recomputed)
Computation (FLOPs)
∼4.95 GB ∼0.01 GB
∼1.79 GB ∼1.63 GB
∼ 4.76 × 1018 ∼ 2.88 × 1017
ity content is often much heavier than text: a long text sequence may only occupy a few kilobytes, while a highresolution image can occupy tens of megabytes and highdefinition video can reach gigabytes. It is also more variable: images may have different resolutions, videos may have different lengths, and some samples may be pure text with no additional modality content at all. After preprocessing, these differences become different modality token sequence lengths. Since attention-based encoder computation depends strongly on sequence length, high-resolution images or long videos can require much more computation and activation memory than small images or text-only samples. In MLLM training, both the ratio and the workload of modality content can vary across samples. For example, vision-language corpora often mix images with very different resolutions, such as high-resolution photographs and low-resolution web images, making the encoder workload vary dynamically from one microbatch to the next.
MLLM ViT+LLaMA3 +SD
Figure 1. MFU comparison between representative LLM and MLLM training workloads. The MLLM measurements are sourced from OrchMLLM (Zheng et al., 2025) and DistTrain (Zhang et al., 2025).
• End-to-end results. On Ascend 910C NPU clusters, Mpipe delivers 2.70× speedup in a small-scale setting and 1.21× at 512 cards, with no change to the training loss.
2. From LLM to MLLM Training Beyond traditional NLP tasks, MLLMs must handle diverse inputs such as images, videos, or audio before the language model can reason over them. A typical MLLM therefore contains modality encoders for parsing these inputs, an LLM backbone for reasoning, and sometimes modality generators for output synthesis. Between these components, projector layers are commonly used to align modality representations with the LLM backbone feature space (e.g., NExT-GPT (Wu et al., 2024)). MLLM training also often proceeds in multiple stages: to align feature spaces across modalities, some modules may be frozen while projector or alignment layers are updated.
At the component level, modality encoders are often much narrower than the LLM backbone, but they can still introduce comparable activation memory and non-negligible computation. Table 1 gives one representative profile: the LLM backbone dominates parameter memory, while the ViT encoder has much smaller parameter memory but comparable activation memory and non-negligible computation. In our 512-NPU training traces, under pipeline parallel (PP) execution, microbatches with few or no images spend only about 200–300 ms in encoder-stage computation, whereas image-heavy microbatches can take more than 1 s. Such instability inserts dynamic bubbles into the pipeline.
The key structural change is that MLLMs no longer execute a single homogeneous Transformer stack. They separate modality parsing from language reasoning, and this separation naturally introduces components with different functional roles. A ViT-style encoder (Dosovitskiy et al., 2021) extracts and aligns modality features, the aligned tokens are then reasoned over in the LLM backbone, and the reasoning result may be passed to modality generators such as U-Net (Ronneberger et al., 2015) or DiT (Peebles & Xie, 2023) to synthesize modality-specific outputs. Therefore, the encoder, projector, LLM backbone, and generator have naturally different responsibilities and process different workloads; they are not interchangeable copies of the same repeated block, but form a heterogeneous model pipeline.
In summary, efficient MLLM training faces two challenges. First, component heterogeneity: modality encoders and LLM backbones have different memory and computation profiles. Second, dynamic encoder workloads: multimodal inputs change the amount of encoder computation and activation memory at microbatch granularity. These challenges motivate the system-level comparison in the next section.
3. Limitations of Existing MLLM Training Systems Training MLLMs in practice usually relies on AI training frameworks, such as PyTorch (Li et al., 2020), Megatron-
The input side adds another source of difficulty. Modal2
HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters
LM (Shoeybi et al., 2019), and Hyper-Parallel1 . These frameworks manage low-level training details and expose reusable parallelism templates for deploying large models. This abstraction works well for repeated Transformer stacks, but becomes challenging when applied to heterogeneous and dynamic MLLM workloads.
tions the encoder and backbone into dedicated, modalityaware pipeline segments across the same ranks to alleviate the load imbalance between encoder stages and LLM backbone stages. It also prefetches metadata for upcoming microbatches and splits large modality workloads to improve balance. Nevertheless, DIP still keeps the encoder coupled inside the LLM backbone pipeline. It mitigates the mismatch through this placement and runtime load balancing, but the architectural heterogeneity of MLLMs remains part of the pipeline itself. Moreover, runtime metadata prefetching, workload splitting, and schedule search are overlapped off the critical path at the scales DIP evaluates, so their cost at much larger cluster scale remains undemonstrated.
Tensor parallelism (Shoeybi et al., 2019) is effective for splitting large hidden matrices in a repeated Transformer stack, but the same partition may provide little memory relief for a much narrower encoder. Pipeline parallelism (Huang et al., 2019) distributes layers across stages, but encoder stages and LLM backbone stages can carry very different workloads, injecting substantial bubbles into the pipeline. Dynamic encoder workloads further complicate standard parallel execution and amplify the pipeline effect above. Data parallelism (Dean et al., 2012) assigns different input subsets to different replicas, so an image-heavy replica can become a straggler while a text-only replica waits. Sequence and context parallelism (Korthikanti et al., 2023; Liu et al., 2024) can reduce long-sequence attention memory, but applying the same sequence split to small or empty modality inputs may introduce communication without much useful memory reduction.
Across these three directions, the encoder remains coupled inside the LLM backbone pipeline, and each system pays for heterogeneity and dynamic workloads through profiling, search, or runtime balancing. As a result, a static plan must either be recomputed every iteration or risks being invalidated by the next microbatch. Mpipe takes a different direction: it relocates encoder computation out of the critical path and into the pipeline’s intrinsic warmup bubbles through a single static placement, so it requires no periteration profiling or search and absorbs the dynamic workload rather than planning around it. We present this design in the next section.
The preceding section identifies two challenges for efficient MLLM training: component heterogeneity and dynamic encoder workloads. Beyond these general-purpose templates, existing MLLM training systems approach these challenges from three directions.
4. Mpipe Mpipe trains MLLMs, which pair a modality encoder with an LLM backbone. The two blocks are heterogeneous, and that is the difficulty. The encoder is a source block. Its cost depends on the input, such as image resolution, sequence length, or the number of tiles. The LLM layers are uniform. If the encoder runs as a normal pipeline stage, its variance is exposed on the critical path: it stretches the stage it occupies, and the resulting bubble spreads down the pipeline.
Spatial resource partitioning. DistTrain (Zhang et al., 2025) assigns hardware resources separately to encoders, LLM backbones, and generators according to their modulelevel load, and customizes parallel strategies for different modules. This cluster-level partitioning can match a specific encoder workload, but such load-specific partitioning is hard to keep effective under dynamic encoder workloads. Temporal bubble exploitation. Optimus (Feng et al., 2025) adjusts the temporal schedule to exploit LLM backbone pipeline bubbles for MLLM training. It first analyzes computation dependencies, then profiles encoder computation, decomposes it into kernel-level units, and temporally schedules these units into LLM backbone pipeline bubbles. This fine-grained scheduling can reduce exposed bubbles by overlapping encoder computation with otherwise idle pipeline slots. However, this schedule depends on the profiled encoder workload. When encoder workloads change dynamically, the values obtained from a static profile become less reliable, making the pre-planned overlap less effective and leading to suboptimal results.
Mpipe removes this exposure with transpose. Transpose replicates the encoder on every pipeline rank and runs each rank’s encodes inside the warmup bubble. Encoder work then overlaps the pipeline fill instead of taking its own stage (Figure 2). We derive transpose from a small schedule algebra (§4.1). A schedule is a list of per-region skeletons, one skeleton per depth region. A derivation maps it to runtime behavior and fixes three things: where each region runs (placement), which collectives cross region boundaries, and the order of forward and backward events, including the warmup and cooldown slots. In this algebra Mpipe is ⟨transpose, 1f1b⟩: a transpose skeleton on the encoder, 1F1B on the backbone. The derivation gates backward work on the model’s trainable bits, not on the schedule. So one transpose schedule serves a frozen or a trained encoder, unchanged. Section 4.2 develops transpose as a
Runtime load balancing. DIP (Xue et al., 2026b) parti1
https://atomgit.com/mindspore/ hyper-parallel
3
HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters 1
1
2
2
3
1
3
4
2
4
5
3
1
5
2
5
3
1
2 1
6
4
1
5
3 2
1
6
6
4
4 3
2
5 4
3
6
5 4
2
3
4
1
2
3
6
1
2
6
1
5
Replicated region (transpose) puts the whole region on every rank and runs data parallelism over the microbatches. Its forwards run in the warmup bubble, and its backwards, when trainable, in the cooldown. Different schedules are points in one space. VPP-3 is ⟨1f1b, 1f1b, 1f1b⟩: three Sharded regions woven into one wavefront. Mpipe is ⟨transpose, 1f1b⟩.
6
DataLoad
1
1 2 3
1
3
4
4 5
6
1
2
3
4
5
2
3
4
5
6
1
2
3
4
1
2
3
4
5
6
2
3
2
3
4
5
6
1
2
1
2
3
4
5
6
1
1
2
3
4
5
6
1 2 3 4 5 6 2
1 5
6
1
Encoder LLM
Notation. P is the number of pipeline ranks (rank ∈ {0, . . . , P −1}) and M the number of microbatches (mb ∈ {1, . . . , M }). A region is in {1, . . . , k}, and a slab is a contiguous layer interval. [X] is a list of X. X ? is an optional X, present only when trainability allows it (Lemma 1). The derivation reads the model in two places: each region’s layer count, which with the cut gives the per-rank slabs, and its trainable bit, which gates all backward work. It then emits three outputs.
Figure 2. Transpose overview. Instead of exposing modality encoder computation as a regular pipeline stage, Mpipe transposes encoder execution into otherwise idle warmup bubbles of the LLM pipeline.
method. 4.1. Schedule Algebra A schedule applies one skeleton to each depth region of the model. A model is a sequence of layers U = ⟨u1 , . . . , uL ⟩. The algebra does not depend on a region’s role. This paper uses the encoder / LLM-backbone split and leaves generator (sink) regions to future work. Two inputs define a parallelization. A cut sets k−1 interior boundaries c1 < · · · < ck−1 in (0, L), with c0 = 0 and ck = L. It splits U into k contiguous regions Π1 , . . . , Πk , one per interval. A schedule σ = ⟨σ1 , . . . , σk ⟩ gives one skeleton to each region. Composition is region concatenation. Each seam in Table 2 is a layout change. The pair Replicated → Replicated is the only one with no change: two replicated regions share one ByMicrobatch owner, so the seam is the identity and the two regions collapse into one. We therefore drop this seam and forbid adjacent replicated regions. Every valid schedule then has a normal form in which transposes are separated by a sharded region. A schedule is valid when its cut tiles U and it is in normal form. Both checks use (c, σ) only. So ⟨transpose, 1f1b⟩ and ⟨transpose, 1f1b, transpose⟩ are valid, while ⟨transpose, transpose⟩ reduces to ⟨transpose⟩. The function derive maps a valid (c, σ) and the model to runtime behavior.
layout ::= Sharded | Replicated owner ::= ByLayer(rank → slab) | ByMicrobatch(mb → rank) comm ::= Gather | Scatter | Send | RecvGrad group ::= ReplicaGroup | DataGroup seam : region × region event ::= Fwd(region, rank, mb) | Bwd(region, rank, mb) | Coll(comm, seam, mb) | DpReduce(region, group) | Loss(mb) | StepBarrier edge ::= Activation | Gradient | Turnaround | Accumulate | Sequence placement : region → layout × owner collectives : { seams: [seam×comm×comm? ], reduces: [region×group] } order : { nodes: rank → [event], edges: [edge] } The unit of placed work is a stage: the pair (region, rank) that one rank runs. The runtime numbers it stage index = rank + (region−1)P , so VPP-3’s three regions are the three virtual stages each rank interleaves. A Sharded region emits one Fwd(region, r, m) per rank r, on its ByLayer slab. A Replicated region emits one at r = ByMicrobatch(m). Placement gives each region its layout and an owner that splits its work; the ByMicrobatch owner assigns a replicated region’s microbatches to ranks. Collectives is a derived view of adjacent layouts. Each seam’s forward comm comes from the layout pair (Table 2). Its backward comm? fires under Lemma 1. The reduces field records each trainable region’s weight reduction over its
skeleton ::= 1f1b | gpipe | transpose schedule ::= ⟨skeleton1 , . . . , skeletonk ⟩ derive : cut × schedule × model → placement × collectives × order A skeleton fixes a region’s forward/backward order. Its one derived property is the layout: layout(transpose) = Replicated, and layout(1f1b) = layout(gpipe) = Sharded. We write layoutj := layout(σj ) for region j. A Sharded region pipelines its layers across the ranks. A 4
HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters
Π2 (Sharded). A (·)? is present only when the encoder is trainable.
group: ReplicaGroup for a transposed region, DataGroup for a sharded one. A DataGroup reduce is a no-op when that axis has one member, i.e. pure pipeline parallelism. Order is one dependency graph over the stage-indexed events. nodes(r) is its projection onto rank r, and edges are its five typed arcs. The graph is acyclic by construction, so there is no cyclic-wait deadlock, and it fixes the forward/backward turnaround and every activation’s live interval.
placement : enc 7→ (Replicated, ByMicrobatch), llm 7→ (Sharded, ByLayer) collectives : seams : [((enc, llm), Gather, Scatter? )]
Table 2. Seam collective for each pair of adjacent layouts. The two cross-layout rows are mirrors. Replicated → Replicated has no layout change, so its seam is the identity. We omit it and forbid adjacent replicated regions. This is the only missing entry, and it is what makes validity, and hence composition, non-trivial. layoutj → layoutj+1
fwd
Replicated → Sharded Sharded → Sharded Sharded → Replicated Replicated → Replicated
Gather Scatter Send RecvGrad Scatter Gather identity (omitted)
reduces : [(enc, ReplicaGroup)? , (llm, DataGroup)] order edges (microbatch m, rank dropped) : Fwd(enc, m) → Coll(Gather, (enc, llm), m) → Fwd(llm, m) → Loss(m) → Bwd(llm, m),
bwd
Bwd(llm, m) forks: ⇝ DpReduce(llm, DataGroup) → StepBarrier ⇝ Coll(Scatter, (enc, llm), m)? → Bwd(enc, m)? → DpReduce(enc, ReplicaGroup)? → StepBarrier
Lemma 1 (Backward footprint). A region’s backward work is fixed by the model, through three trainability gates. Order the regions input-to-loss as Π1 , . . . , Πk . For Πj : (i) Πj sends its input-activation gradient upstream (the reverse collective on its input seam (Πj−1 , Πj )) iff ∃ i < j : trainable(Πi ); (ii) Πj receives a gradient from downstream (the reverse collective on its output seam (Πj , Πj+1 )) iff ∃ i ≤ j : trainable(Πi ); (iii) Πj reduces weight gradients iff trainable(Πj ). The send and receive gates differ by one region. Πj receives but does not send exactly when it is trainable and no region before it is: the trainable source.
The chain gives the edges for one microbatch, with the rank index dropped; ⇝ marks a fork into independent branches. Fwd(enc, m) stands for Fwd(enc, ByMicrobatch(m), m), and Fwd(llm, m) for the family over the llm slabs on every rank. The per-rank schedule nodes(r) is this graph projected onto rank r; Figure 2 shows it schematically. Forward arcs are Activation edges. The Loss → Bwd pivot is a Turnaround. Each Bwd emits an Accumulate edge to its DpReduce. When its region has a trainable strict predecessor (Lemma 1(i)), it also emits a Gradient edge into its reverse seam. The two DpReduce → StepBarrier arcs are independent Sequence edges, with no order between the two reduces. Bwd(llm, m) is never gated, since the backbone is always trainable. But its Gradient edge, the Scatter? that ships the LLM’s input-activation gradient back to the encoder, fires only when the encoder is trainable (Lemma 1(ii) at enc equals Lemma 1(i) at llm). The encoder is the trainable source of Lemma 1. It receives the Scatter? and runs its weight backward Bwd(enc, m)? . This is an Accumulate edge only: the encoder has no predecessor, so no Gradient edge, and it sends nothing upstream. The Activation edge from each warmup Fwd(enc, m) to its cooldown consumer is the long-lived encoder state.
Proof. A region’s input-activation gradient is used only upstream, by Πj−1 and earlier. A frozen region still passes it through. So Πj sends iff some region before it is trainable (∃ i < j). Πj receives iff it runs any backward at all, that is iff Πj or some region before it is trainable (∃ i ≤ j). The two index sets differ only at i = j. Finally, Πj ’s receive is Πj+1 ’s send (the collective carries exactly Πj+1 ’s inputactivation gradient), so the seam gate and the region gate are one predicate. □ Corollary 1 (Schedule-invariance). Gates (i) through (iii) use only trainable(·) and the cut, never σ. So a region’s backward footprint does not depend on the schedule: the encoder has the same footprint whether it is transposed (Replicated) or pipelined (Sharded). The footprint says which backward events fire. The collective type for each one still depends on σ through Table 2. One transpose schedule therefore needs no per-trainability variant: derive reads trainable, not σ.
A second derivation. derive is not tied to one schedule. VPP-3, ⟨1f1b, 1f1b, 1f1b⟩, places every region as (Sharded, ByLayer) and gives two interior seams, ((Π1 , Π2 ), Send, RecvGrad? ) and ((Π2 , Π3 ), Send, RecvGrad? ). The seam argument of Coll is what tells them apart, which is why it is not optional. Each rank owns one slab of all three regions, so the stage-indexed events are the interleaved 1F1B wavefront.
Instantiation. Take Mpipe, ⟨transpose, 1f1b⟩. The encoder is enc = Π1 (Replicated) and the backbone is llm = 5
HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters
Π1 again sends no input-activation gradient. The same six events and five edges describe both schedules; only the cut and the skeletons differ.
instead of taking an extra stage on the critical path. The owner map is a free choice, which Mpipe uses to match heavier encodes to longer warmup slots. This PP-internal replication is not the same as a separate data-parallel group for the encoder. The encoder forward runs in the same pipeline group, and its output feeds the first LLM stage at once through the seam gather. So Mpipe adds no global aggregation and no cross-group synchronization on the encoder path. The encoder output is used where it is made, inside the pipeline that needs it. We keep the locality and independence of data-parallel work without a second collective domain.
The same schedule, two footprints. Scheduleinvariance (Corollary 1) is one axis. Freezing is the other. Freezing the encoder leaves σ, the placement, and the forward order unchanged, but deletes every (·)? . The order collapses to Fwd(enc, m) → Coll(Gather, . . . ) → Fwd(llm, m) → Loss(m) → Bwd(llm, m) → DpReduce(llm, DataGroup) → StepBarrier, with no reverse Scatter, no Bwd(enc), and no replica reduce. The schedule comes from the user; the footprint comes from the model.
4.3. Cost Model The same two outputs that give memory, placement and order, also give a cost model. We weight each event with a duration cost : event → R≥0 and read the step time off the order graph, so a schedule’s cost is computed, not measured. A compute event runs on one rank and costs the roofline of its slab; a collective moves an n-byte payload over its group at the α-β rate of the SGL bridging model (Li & Hains, 2012): cost(Fwd) = cost(Bwd) = max W/F, D/B ,
Memory as a derived view. Memory is read off (placement, order) with no extra output. An activation is live from when it is produced to its last use, and a rank’s peak is the running sum of live sizes. The transposed encoder is one long Activation edge, live across the whole LLM span; transpose hides it in the warmup bubble instead of paying for an extra sharded stage with the same lifetime. Applicability. The two composition axes, a depth cut with one skeleton per region and an owner map that weaves Sharded regions into one wavefront, are enough for every schedule we use: 1F1B, GPipe, interleaved VPP, and ⟨transpose, 1f1b⟩. The algebra also extends in three graded steps. A fold (a reflected ByLayer owner plus a wave skeleton) reaches Hanayo (Liu et al., 2023). An event split (a split backward plus a fused forward-backward node) reaches DualPipeV (DeepSeek-AI, 2025). Both are conservative: the present algebra is the degenerate case. A bidirectional feed, as in DualPipe (DeepSeek-AI, 2024), needs a new input and stays outside. Our use-case sits inside the closed core. We treat 1f1b, gpipe, and transpose as composable parallel skeletons in the sense of Cole (Cole, 1989), applied to the execution schedule of a fixed model rather than to an algorithm.2
cost(Coll) = cost(DpReduce) = α + β n, where W is the slab’s FLOPs at peak rate F , D its HBM traffic at bandwidth B, and n the payload bytes. These weights turn order into a weighted graph in which each rank’s events follow their nodes(r) order. The step makespan is its longest path, and a rank’s bubble is its idle time: X T = max cost(e), p ∈ paths(order)
bub(r) =
T−
X
e∈p
cost(e).
e∈nodes(r)
SGL sums its supersteps in sequence; a pipeline overlaps them, so T is a longest path, not a sum of stage times. Because T comes from the schedule alone, it can be checked against the measured step time.
4.2. Transpose Transpose is the Replicated skeleton of §4.1, read as a systems mechanism. Replicating the encoder on every rank is a scatter-gather step in the sense of SGL (Li & Hains, 2012). The replication scatters the microbatches across the ranks. Each rank runs the encoder forward for the microbatches its ByMicrobatch owner gives it, on its own, inside its warmup bubble. The Replicated → Sharded seam then gathers the per-rank encoder outputs into the first LLM stage (Figure 2). Encoder work overlaps the pipeline fill
This makes transpose quantitative. Write wr for the warmup bubble on rank r, its idle time before the first LLM forward, and Er for the encoder work that rank runs, P Er = m: ByMicrobatch(m)=r cost(Fwd(enc, r, m)). Transpose places Er in that window, so it hides min(Er , wr ) and exposes spillr = max 0, Er − wr
2
The derivation is defined for every valid cut. The present instantiation uses the two-region encoder/LLM cut; intermediate cuts are future work.
on the critical path. When every Er ≤ wr the encoder is free, and the step pays only the worst rank’s overflow, 6
HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters Table 3. Experiment A: average end-to-end step time on a production-scale MLLM workload.
maxr spillr . The ByMicrobatch owner is the assignment that minimizes this exposed spill, a min-max balancing of encoder work against per-rank warmup slack; matching heavier encodes to longer warmup slots is its greedy form.
Model
Baseline Avg. Step Time
Mpipe Avg. Step Time
Speedup
16.26 s
13.42 s
1.21×
ViT+DeepSeek
The model predicts where transpose helps. When the encoder is small against the warmup slack, as when the backbone dominates at scale, Er ≤ wr and the encoder leaves the step untouched off the bubble; when it is large against the slack, the residual maxr spillr is exposed and the gain shrinks. This is the trend we measure in §5.
Table 4. Experiment B: average end-to-end step time on an 8device workload. Model
Baseline Avg. Step Time
Mpipe Avg. Step Time
Speedup
11.01 s
4.07 s
2.70×
Qwen3.5 MLLM
ble. In the algebra they are just another Sharded region, for example ⟨transpose, 1f1b, 1f1b⟩, and are pipelined rather than moved. Runtime systems cover these cases with periteration search (Xue et al., 2026b) or data-dependent parallelism (Xue et al., 2026a). Mpipe keeps the schedule static and loss-preserving. It trades that coverage for near-zero scheduling overhead and reproducible runs, and we evaluate it in the encoder-bound case where this trade pays off.
Compositional cost. The longest path above is a global property of the assembled graph, but it can also be read compositionally. Give each schedule a timing denotation JσK: a function that maps the arrival times of its inputs, one per rank and microbatch, to the completion times of its outputs. Each such function is monotone and linear in the (max, +) semiring, since a completion time is the maximum of an input-ready time and a resource-free time, plus a duration. Concatenating two schedules σ and σ ′ composes their denotations, with the seam Coll as the transfer function between them: Jσ ′ K ◦ JCollK ◦ JσK. Cost is therefore a homomorphism from schedules to (max, +) transfer functions, on the same composition the algebra already uses. The step makespan is this composed function applied to the microbatch injection times, and its steady-state period is the function’s (max, +) eigenvalue, the bottleneck stage. Transpose needs no special case: the warmup slack w is just when the LLM region becomes ready on each rank, part of the arrival times the composition already carries, so the spill max(0, E −w) is what its transfer function returns rather than a separate rule. For our acyclic, singleorientation order, the makespan read this way equals the operational longest path above, the standard agreement between longest paths in a timed event graph and (max, +) linear behavior (Baccelli et al., 1992). We state this as a remark and leave the contention-aware proof, which threads the per-rank state through the composition, to future work.
5. Experiments We conduct experiments to evaluate the overall performance of Mpipe on MLLM training workloads. Environment. We implement and evaluate Mpipe in Hyper-Parallel. The evaluation runs on a 512-device CloudMatrix384 (Zuo et al., 2025). Mpipe runs inside the framework’s existing pipeline runtime: derive lowers a schedule to the placement, collectives, and order of §4.1, and the Replicated → Sharded gather uses CANN/HCCL, with no change to the LLM backbone’s pipeline kernels. Experiment A: production-scale MLLM workload. We design this experiment to simulate the effect of Mpipe under practical industry MLLM training workloads. The model is deployed on the full 512-device environment and uses the ViT encoder from Qwen2-VL (Wang et al., 2024) together with a DeepSeek-V3 (DeepSeek-AI, 2024) LLM backbone. The dataset is an internal multimodal dataset. Since DistTrain has not released its implementation, we construct a DistTrain-like baseline in our environment: one pipeline stage is allocated to the encoder with a customized parallel strategy, while the LLM backbone applies conventional 5D parallelism. Table 3 shows that Mpipe achieves 1.21× speedup over this baseline.
Scope. Mpipe handles the main, source-side form of MLLM data variance: the encoder workload. The schedule algebra (§4.1) moves it into pipeline bubbles as a static schedule that does not depend on the modality mix. Three cases fall outside this, and the same algebra states each one. (i) When a microbatch’s encoder work is larger than the warmup slack, part of it spills onto the critical path; the owner map reduces this spill but cannot remove it. (ii) Variance in the LLM sequence length makes the backbone stages themselves data-dependent. This is interior to the pipeline and is handled by sequence bucketing or packing, not by placement. (iii) Output-side generators are pipeline sinks. Their input comes from the backbone, not from the start of the step, so they cannot be transposed into a bub-
Experiment B: generality study. To quickly validate the generality of Mpipe, we also run an 8-device experiment with a recently released Qwen3.5-based MLLM (Qwen Team, 2026) on the CapsFusion image-text dataset (Yu et al., 2024). The baseline follows Megatron-LM (Shoeybi et al., 2019). As shown in Table 4, Mpipe reduces the average step time from 11.01 s to 4.07 s, achieving 2.70× speedup. 7
HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters
Analysis. The relative speedup of transpose depends on the encoder’s share of the step and on how much warmup bubble is available to absorb it (§4.3). This is why the two settings differ in magnitude: in Experiment A the large backbone dominates the step, so relocating the smaller encoder yields 1.21×, whereas in the controlled setting of Experiment B the encoder is a larger share of the step and the gain reaches 2.70×. Because the two settings also differ in model and baseline, they are not a controlled scaling study; a sweep that isolates scale and the resulting encoderto-bubble ratio is left to future work. In both cases the gain comes from a single static schedule with no per-iteration scheduling cost, and transpose leaves the training loss unchanged, since it only relocates computation.
express an algorithm, whereas we compose them to express the execution schedule of a fixed model.
6. Related Work
Limitations and future work. When per-microbatch encoder work exceeds the warmup slack, part of it spills onto the critical path; the static owner map reduces this spill but cannot remove it, and a loss-preserving metadataguided reordering that closes the residual is left to future work. We also instantiate transpose only on the two-region encoder/LLM cut with 1F1B; the algebra supports richer schedules (a fold toward Hanayo, an event split toward DualPipeV, a bidirectional feed toward DualPipe), whose evaluation we leave to future work.
7. Conclusion MLLM training exposes component heterogeneity and dynamic encoder workloads that make conventional LLM parallelism less effective at scale. Mpipe casts parallel schedules as a small schedule algebra and derives their concrete pipeline behavior, including transpose, a heterogeneous parallel schedule for MLLM training. Experiments on Ascend 910C NPU clusters show that Mpipe reduces end-to-end training step time on both small-scale and 512card settings.
Pipeline schedules. Pipeline parallelism partitions a model into stages and streams microbatches through them. GPipe (Huang et al., 2019) flushes each batch; 1F1B and interleaved (VPP) schedules reduce the bubble at fixed memory; recent work pushes further with wave-like (Hanayo (Liu et al., 2023)), zero-bubble (DualPipeV (DeepSeek-AI, 2025)), and bidirectional (DualPipe (DeepSeek-AI, 2024)) schedules. Mpipe does not propose one more schedule in isolation. It treats these as points in a single algebra: 1F1B, GPipe, and interleaved VPP are instances, and the recent schedules are short, named extensions of it (§4.1). Within that algebra, ⟨transpose, 1f1b⟩ is the schedule that handles the encoder/LLM heterogeneity of MLLM training.
References Baccelli, F., Cohen, G., Olsder, G. J., and Quadrat, J.-P. Synchronization and Linearity: An Algebra for Discrete Event Systems. Wiley, 1992. Cole, M. I. Algorithmic Skeletons: Structured Management of Parallel Computation. Research Monographs in Parallel and Distributed Computing. Pitman/MIT Press, London, 1989.
MLLM training systems. DistTrain (Zhang et al., 2025), Optimus (Feng et al., 2025), and DIP (Xue et al., 2026b) target encoder/backbone heterogeneity through spatial partitioning, temporal bubble profiling, and runtime load balancing respectively; OrchMLLM (Zheng et al., 2025) and MegaScale-Omni (Xue et al., 2026a) adapt resources and parallelism to dynamic multimodal workloads in production. As discussed in §3, all of these keep the encoder coupled inside the backbone pipeline and pay for heterogeneity and dynamism at runtime. Mpipe instead relocates the encoder out of the pipeline with a single static placement, which removes the per-iteration cost and is invariant to the modality mix.
Dean, J., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., and Ng, A. Y. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024. URL https:// arxiv.org/abs/2412.19437. DeepSeek-AI. DualPipe: A bidirectional pipeline parallelism algorithm for computation-communication overlap in DeepSeek-V3/R1 training. https://github. com/deepseek-ai/DualPipe, 2025. DualPipeV schedule; accessed 2026-06-24.
Schedules as composable skeletons. Mpipe’s view of schedules as composable parallel patterns follows the line of parallel algorithmic skeletons (Cole, 1989), which hide orchestration behind structured collectives rather than raw point-to-point messaging (Gorlatch, 2004), and the BSPstructured, cost-predictable skeleton libraries (Loulergue et al., 2005; Javed & Loulergue, 2009; Li & Hains, 2012). The difference is one of level: those compose skeletons to
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, 8
HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters
N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https:// openreview.net/forum?id=YicbFdNTTy.
Liu, Z., Cheng, S., Zhou, H., and You, Y. Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’23). Association for Computing Machinery, 2023. doi: 10.1145/3581784. 3607073.
Feng, W., Chen, Y., Wang, S., Peng, Y., Lin, H., and Yu, M. Optimus: Accelerating large-scale multi-modal llm training by bubble exploitation. In Proceedings of the 2025 USENIX Conference on USENIX Annual Technical Conference, USENIX ATC ’25, USA, 2025. USENIX Association. ISBN 978-1-939133-48-9.
Loulergue, F., Gava, F., and Billiet, D. Bulk synchronous parallel ML: Modular implementation and performance prediction. In International Conference on Computational Science (ICCS), volume 3515 of LNCS, pp. 1046– 1054. Springer, 2005.
Gorlatch, S. Send-receive considered harmful: Myths and realities of message passing. ACM Transactions on Programming Languages and Systems (TOPLAS), 26(1):47– 56, 2004. doi: 10.1145/963778.963780.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In International Conference on Computer Vision, pp. 4195–4205, 2023.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https:// arxiv.org/abs/2407.21783.
Qwen Team. Qwen3.5-Omni technical report. arXiv preprint arXiv:2604.15804, 2026. URL https:// arxiv.org/abs/2604.15804.
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., and Chen, Z. GPipe: Efficient training of giant neural networks using pipeline parallelism. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc.
Ronneberger, O., Fischer, P., and Brox, T. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Springer, 2015. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training multibillion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. URL https://arxiv.org/abs/1909.08053.
Javed, N. and Loulergue, F. OSL: Optimized bulk synchronous parallel skeletons on distributed arrays. In Advanced Parallel Processing Technologies (APPT), volume 5737 of LNCS, pp. 436–451. Springer, 2009.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-VL: Enhancing visionlanguage model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. URL https://arxiv.org/abs/2409.12191.
Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models. In Proceedings of Machine Learning and Systems, volume 5, 2023. Li, C. and Hains, G. SGL: towards a bridging model for heterogeneous hierarchical platforms. International Journal of High Performance Computing and Networking (IJHPCN), 7(2):139–151, 2012.
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. NExTGPT: Any-to-any multimodal llm. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.
Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., and Chintala, S. PyTorch Distributed: Experiences on accelerating data parallel training. Proceedings of the VLDB Endowment, 13(12):3005–3018, 2020. doi: 10. 14778/3415478.3415530.
Xue, C., Chen, Y., Jiang, J., Zheng, N., Feng, J., Chen, J., Zhao, S., Yan, S., Lin, Y., Shi, L., Wang, Z., Luo, L., Wu, F., Lin, H., Liu, X., Peng, Y., and Chen, Q. Megascale-omni: A hyper-scale, workload-resilient system for multimodal llm training in production. In Proceedings of the 21st European Conference on Computer Systems (EuroSys ’26). Association for Computing Machinery, 2026a. doi: 10.1145/3767295.3803587.
Liu, H., Zaharia, M., and Abbeel, P. RingAttention with blockwise transformers for near-infinite context. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=WsRHpHH4s0.
Xue, Z., Hu, H., Chen, X., Jiang, Y., Song, Y., Mi, Z., Zhu, Y., Jiang, D., Xia, Y., and Chen, H. DIP: Efficient large 9
HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters
multimodal model training with dynamic interleaved pipeline. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’26, pp. 618–632, New York, NY, USA, 2026b. Association for Computing Machinery. ISBN 9798400723599. doi: 10.1145/3779212.3790154. URL https://doi. org/10.1145/3779212.3790154. Yu, Q., Sun, Q., Zhang, X., Cui, Y., Zhang, F., Cao, Y., Wang, X., and Liu, J. CapsFusion: Rethinking imagetext data at scale. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14022–14032, Seattle, WA, USA, 2024. doi: 10.1109/ CVPR52733.2024.01330. Zhang, Z., Zhong, Y., Jiang, Y., Hu, H., Sun, J., Ge, Z., Zhu, Y., Jiang, D., and Jin, X. DistTrain: Addressing model and data heterogeneity with disaggregated training for multimodal large language models. In Proceedings of the ACM SIGCOMM 2025 Conference, pp. 24–38. ACM, 2025. doi: 10.1145/ 3718958.3750472. URL https://doi.org/10. 1145/3718958.3750472. Zheng, Y., Xiao, B., Shi, L., Li, X., Wu, F., Li, T., Xiao, X., Zhang, Y., Wang, Y., and Liu, S. OrchMLLM: Orchestrate multimodal data with batch post-balancing to accelerate multimodal large language model training. arXiv preprint arXiv:2503.23830, 2025. URL https: //arxiv.org/abs/2503.23830. Zuo, P., Lin, H., Deng, J., Zou, N., Yang, X., Diao, Y., Gao, W., Xu, K., Chen, Z., Lu, S., et al. Serving large language models on Huawei CloudMatrix384. arXiv preprint arXiv:2506.12708, 2025. URL https: //arxiv.org/abs/2506.12708.
10