Arachne: Orchestrating Cascades for Efficient Text-to-Video Model Training Peng Yu2,1 , Yuankai Fan1,∗ , Yang Qiu1 , Tian Li1 , Bihuan Chen2 , Yin Chen1 , Qizhen Weng1,∗ 1
Input Noise Predicted noise
(a) Training workflow of a typical T2V model, consisting of a VAE, a text encoder, and multiple DiT blocks. Bs =3
58.6s 38.9s
5
Bs
=1
Frames=37
13.4s
2
DP=4
Frames=105
=1 3
=6
Frames=113
Bs
Abstract—The rising demand for AI-generated videos is fueled by advances in large-scale Text-to-Video (T2V) models, trained on extensive datasets of video clips spanning diverse resolutions and durations. To address this data heterogeneity, current training methods often use a bucketing strategy that groups samples into discrete buckets for efficiency. However, this approach struggles to scale with compute and data volumes under static parallelism schemes, such as data and sequence parallelism, leading to significant workload imbalances and hardware under-utilization. In this paper, we present Arachne, a novel training framework for efficient T2V model training at scale. Arachne decomposes the training process into fine-grained computational units, called cascades, orchestrating their distributed execution and synchronization across the cluster through coordinated spatial and temporal optimization. Our comprehensive evaluation demonstrates that Arachne reduces iteration time by up to 65% over leading frameworks, exhibiting a positive scaling trend where its performance advantages amplify as training scale grows.
Bs
arXiv:2607.01701v1 [cs.DC] 2 Jul 2026
Institute of Artificial Intelligence (TeleAI), China Telecom, Shanghai, China; {fanyk1, lit117, cheny304, wengqzh}@chinatelecom.cn, [email protected] 2 College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China; [email protected], [email protected]
52.6s
1
I. I NTRODUCTION Video synthesis with deep generative models is transforming diverse domains, including media content creation [1], [3], [40], [43], interactive video games [6], [44], [49], and world simulation [11], [14], [36]. Among these, Text-to-Video (T2V) models emerge as the leading and rapidly advancing frontier, driven by advances in generative algorithms [15], [28], [29], [38] , scalable model architectures [13], [32], [35], and unprecedented availability of compute and data resources [18], [33], [46]. Current state-of-the-art T2V models [5], [20], [43], [45], [48], have converged on the latent diffusion paradigm [38] as their architectural blueprint. As illustrated in Fig. 1a, this paradigm involves a Variational Autoencoder (VAE) [19] first compressing video into a latent space, where a Diffusion Transformer (DiT) [35], guided by a text encoder, is then trained to predict artificially added noise. Here, the success of this paradigm in generating high-quality video relies heavily on training with massive datasets [20], [43], [53]. Training on these massive datasets is typically accelerated using data parallelism (DP) [25], where the workload is evenly distributed across multiple devices. However, due to the data’s inherent heterogeneity, a local batch randomly sampled for each DP rank will contain a mix of videos with varying ∗ Corresponding authors: Yuankai Fan, Qizhen Weng.
Frames=13
Data Buckets
Text Encoder Forward
VAE Forward
DiT Forward
DiT Backward
Idle
(b) Illustration of a T2V model training step under the default bucketing (4 frame-length buckets) and parallelism configurations (DP=4, SP=4), showing significant GPU idle time (hatched regions) induced by the gradient synchronization barrier following the DiT backward pass.
durations and resolutions, making it difficult to efficiently organize the data into uniform batches for training. The widely adopted solution is the bucketing strategy [20], [41], [53], which pre-groups videos by similar resolutions and durations so that each DP rank can sample a local batch from a single bucket. In addition, for buckets containing long sequences that exceed single-device memory, sequence parallelism (SP) [17], [21], [24], [26] is utilized to partition a single data sequence across a dedicated group of devices. Challenges. However, while bucketing ensures uniformity within each local batch, it still leaves a considerable workload imbalance across DP ranks. The strategy attempts to mitigate this issue by incorporating adaptive batch sizing1 , but such adjustments are fundamentally constrained by GPU memory 1 Buckets with shorter sequences are assigned larger batch sizes to equalize the total tokens (i.e., batch size × sequence length) processed per rank, subject to GPU memory constraints
limits and the rigidity of static parallelism. This is clearly illustrated by the example in Fig. 1b. In a representative training step in the T2V model HunyuanVideo [20], executed on four DP=4 ranks, each with an SP=4 degree, four distinct batches from separate buckets are fed into the SP groups, with sequence lengths varying from 13 to 113 frames. Due to the static parallelism configuration applied to all groups, the rank processing the longest sequence (113 frames) takes 58.6 seconds, whereas the one with the shortest (13 frames) completes in only 6.0 seconds, resulting in 52.6 seconds of idle time for the fastest rank. This example shows the substantial resource under-utilization inherent in current training schemes. The challenge of mitigating inefficiencies from static parallelism under data heterogeneity has spurred the development of dynamic parallelism approaches in the Large Language Model (LLM) field, primarily targeting the communication overhead of short text sequences [12], [47]. However, such a focus is mismatched with the realities of T2V generation, where the combination of significantly longer video sequences and compute-intensive DiT architectures makes computation, not communication, the defining performance bottleneck (more details can be seen in §III). Proposed Method. This paper proposes Arachne, a novel training framework tailored for T2V model training. At its core, Arachne introduces a unified strategy that integrates fine-grained workload decomposition with dynamic spatiotemporal orchestration. It first breaks down the training process into minimal schedulable units known as cascades, each a self-contained task defined by a bespoke configuration that specifies its target model components, input data, parallelization strategy, and scheduling parameters. This decomposition empowers Arachne to execute these cascades with flexibility and efficiency via spatio-temporal orchestration. Specifically, Arachne introduces three components that work collaboratively to optimize T2V training. 1 Cascadelevel parallelism planner performs temporal optimization by decomposing the training iteration into a Directed Acyclic Graph (DAG) of dependent cascades, co-optimizing the launch time and parallel configuration for each to minimize the theoretical makespan on abstract, homogeneous hardware. 2 Topology-aware resource mapper then performs spatial optimization by taking the planner’s logical, topology-agnostic plan and, considering the physical network topology, mapping each cascade onto a specific set of GPUs to create a communication-efficient physical execution plan. Finally, 3 Runtime executor enacts the concrete spatio-temporal plan by coordinating data handoff across dependent cascades and regulating gradient accumulation within parallelism groups, thereby ensuring both correctness and efficiency. We evaluate Arachne’s performance on three large-scale T2V models, including Wan2.1 [45], CogVideoX [48], and HunyuanVideo [20], and compare it with two leading training frameworks—Megatron-LM [39] and DeepSpeed [37]—as well as FlexSP [47], a representative training framework tailored for variable-length corpora. Our experimental results
show that, under realistic T2V training scenarios, Arachne achieves significant end-to-end speedups, reducing average iteration time by up to 65%, 59%, and 35% compared to these frameworks, respectively. To further evaluate scalability, we examine Arachne across diverse model architectures, workload heterogeneities, and cluster sizes, which reveals a clear scaling trend: as training scale increases, the speedup achieved by Arachne grows correspondingly, reflecting the “scaling-law” behavior observed in model performance under realistic large-scale training scenarios. Contributions. We make the following contributions: 1) We present Arachne, a novel training framework for efficient large-scale T2V model training. Its design is informed by a systematic analysis of T2V’s unique workload characteristics, revealing performance bottlenecks distinct from those encountered in the LLM domain. 2) We propose a cascade-level spatio-temporal orchestration paradigm that mitigates workload imbalance arising from data heterogeneity. This paradigm temporally optimizes the launch time and parallel configuration for each fine-grained cascade, while spatially mapping it to communicationefficient GPU placements. 3) Through comprehensive experiments, we show that Arachne achieves significant and consistent speedups over leading baselines, demonstrating both effectiveness and robust scalability across diverse T2V models, workload heterogeneities, and cluster configurations. 4) We will open-source the full implementation of Arachne and the T2V training workloads used in our evaluation, fostering reproducibility and enabling further research. II. BACKGROUND This section provides an overview of the core architectures, training paradigms, and parallelization strategies used in modern T2V models. A. T2V Diffusion Models Modern T2V models typically follow a multi-module latent diffusion paradigm [38], as illustrated in Fig. 1a, comprising a VAE for compression, a text encoder for conditioning, and a DiT backbone for denoising. In this paradigm, the VAE (often a 3D causal VAE [50]) encodes raw video clips into compact spatio-temporal latent representations. Noise is added to these latents, and under text conditioning, the DiT is trained to predict the added noise in DDPM [15] or the velocity toward clean latents in Flow Matching [29], [31]. B. T2V Training Strategies Training T2V models on large-scale, heterogeneous datasets typically relies on two core strategies: multi-stage curriculum learning to ensure stable convergence, and data bucketing to enable efficient batching. Curriculum Strategy. To make training T2V models computationally tractable and stable, models follow a multistage, coarse-to-fine curriculum [7], [20], [53]. Training starts
on massive, low-resolution video-text data to establish textvideo alignment, and then progressively fine-tunes on smaller, higher-quality datasets at increasing resolutions to improve visual fidelity and temporal coherence.
Unlike LLMs that rely on Tensor and Pipeline Parallelism (TP/PP) to distribute massive parameters [16], [39], T2V training is primarily constrained by the activation memory of long video sequences. Consequently, practitioners favor data-centric parallelism: Data Parallelism (DP) for throughput scaling, and Sequence/Context Parallelism (SP/CP) to shard long contexts [10], [45], [53]. Data Parallelism (DP). DP replicates the model on each device while sharding the training data. During each step, all devices process their local data in parallel before a global gradient synchronization ensures a consistent model update. This approach is often enhanced with optimizers like ZeRO [37] that also partition optimizer states, gradients, and parameters to further reduce per-device memory. Sequence/Context Parallelism (SP/CP). SP and CP shard long video contexts across devices to reduce activation memory. Ulysses-style SP uses head partitioning by splitting attention heads across devices and relying on All-to-All communication to enable full-sequence self-attention for each head shard [17]. In contrast, CP uses sequence partitioning by sharding Q/K/V along the sequence dimension and computing attention block-wise while circulating K/V slices in a ring, which enables overlapping communication with computation [4], [23], [30], [34].
Probability Density
0.8 LLM Mean (1,391) 0.4
K
1K
26 2
13
66 K
K
33 K
16
8K
4K
2K
1K
51 2
25 6
8
0.0
Sequence Length
Fig. 2: Sequence-length distributions for two T2V datasets (Koala [46] and an internal 1080p dataset called Lynx) and two LLM datasets (CommonCrawl and GitHub). The x-axis is shown in log scale for readability. The vertical dashed lines mark the average sequence lengths for each domain.
after VAE encoding, even short video clips generate thousands of tokens, making the shortest T2V sequences more computationally demanding than the vast majority of LLM sequences (e.g., the 99.5% shorter than 8K tokens). • Concentrated. Unlike the long-tail distribution of LLM workloads that spans several orders of magnitude (e.g., from 128 to 128K tokens), T2V workloads exhibit a high relative concentration. Despite a large absolute token variance, the dynamic range of T2V sequences (e.g., 33K to 262K) is limited to a small ratio (∼ 8×). This clustering within a high-cost computational region leads to a sustained and predictable heavy load. • Discrete. Since sequence length increases in discrete increments determined by video frame counts, marginal variations in video duration can lead to substantial computational disparities. Such discontinuity is a major source of inefficiency that amplifies the load imbalance across pre-defined training buckets. 58.6s 45.2s
SP=2,4,8,2
Limited Benefits
Bs =3
C. Parallelisms in T2V Training
T2V Mean (104,355)
T2V Datasets Koala-36M Lynx (1080P)
1.2
12
Bucketing Strategy. To batch heterogeneous videos, naive zero-padding incurs massive computational waste, while adapting sequence packing [22] to 3D video tensors introduces extreme implementation complexity and inefficient attention masking. As a practical alternative, the bucketing strategy [20], [53] pre-categorizes videos into discrete buckets based on fixed resolutions and durations, thereby guaranteeing strict intrabatch homogeneity during training.
LLM Datasets CommonCrawl GitHub
1.6
x2
3.1s
5
Frames=37 Bs
x4
2
x8
17.2s
3
x2
Bs
=6
Frames=113
30.1s
1
Frames=13
Data Buckets
Observation 1: Right-shifted, concentrated, and discrete sequence-length distributions in video workloads. The primary distinction between T2V and LLM workloads lies in their sequence-length distributions, as depicted in Fig. 2. Comparing T2V datasets with commonly used LLM corpora highlights three salient characteristics of video workloads that challenge conventional scheduling assumptions: they are rightshifted, concentrated, and discrete. • Right-shifted. The entire distribution is shifted dramatically to the right as a direct consequence of the data modality;
Frames=105
=1
DP=4
Bs
Existing distributed training systems, heavily optimized for LLMs, fall short for T2V generation, whose distinct sequence lengths and computation patterns create new bottlenecks. We highlight two key observations that expose this mismatch and motivate a T2V-aware system design.
=1
III. O BSERVATIONS
Text Encoder Forward
VAE Forward
DiT Forward
DiT Backward
Idle
Fig. 3: Training step corresponding to Fig. 1b, with parallelism reconfigured across buckets (SP=8 for the 113-frame bucket and SP=2 for the 13- and 37-frame buckets). Observation 2: Computation as the bottleneck and the limits of LLM-style scheduling. A second defining characteristic of the T2V workloads is their extreme computational intensity, stemming from the DiT architecture’s full selfattention over a unified sequence of 3D patches of length
Cascade-grained Parallelism Planner Cascade T1
Data A
Data B
Buckets
Modules
One Training Step
T2
A
B
A
A
T3
B
T4
B
Data Bucket - Batch Size: 1 - Resolution: 720p - Frames: 37 Parallelism Config - Strategy: SP - SP Degree: 4 Module - Model Name: DiT Schedule Info - Launch Time: T4
Topology-aware Resource Mapper T1
Computational cascades
T2
T3
B
B
A
Pool of cascades
A B
DAG
Timeline
Runtime Executor
Cascade
Node A
Node B
Network topology
Text encoder, VAE and DiT Modules in a T2V model
A
Data Bucket - Batch Size: 1 - Resolution: 720p - Frames: 37 Parallelism Config - Strategy: SP - SP Degree: 4 Module - Model Name: DiT Schedule Info - Launch Time: T3 - Placement: GPU 0,1(A), 0,1(B)
A A
B
B
A
B
Timeline
Timeline
Communication-efficient DAG
Execution Plan
Forward pass (and backward pass) of modules
Pipeline Execution System idle time
Fig. 4: Overview of Arachne. L, which incurs quadratic (O(L2 )) complexity [42], [51], [52]. When applied to the inherently long sequences described above, this operation becomes exceedingly compute-intensive, emerging as the dominant performance bottleneck. Clearly, this characteristic limits the applicability of many approaches derived from the LLM domain [12], [47]; existing frameworks are typically designed for text corpora with high variability in sequence length, where communication overhead across short sequences is the primary performance concern. However, these communication-oriented approaches are insufficient for compute-bound workloads observed in T2V. Example 1. Continuing the example in Fig. 1b, we apply the communication-oriented dynamic parallelism method introduced in [47], which increases parallelism for the 113frame sequence to SP=8 while reducing it for the 13- and 37-frame sequences to SP=2, respectively. As shown in Fig. 3, such optimization yields only limited benefits (gray region): while the overall latency is reduced from 58.6s to 45.2s, the performance bottleneck merely shifts to the 105-frame bucket, leaving substantial GPU idle times (i.e., 3.1s, 17.2s, and 30.1s). This example shows that applying communicationoriented optimizations in LLMs is challenging for T2V training due to the computation-intensive nature of the workload. Motivation for Our Approach. Our analysis reveals that the interplay of heterogeneous video sequences and computeintensive DiT architecture poses a unique training challenge for T2V models: one that existing parallelism approaches, including recent dynamic parallelism from the LLM field, are ill-equipped to solve. This inadequacy stems from two constraints that are inherent to all such approaches: (1) a static spatial assignment, which binds each data sample to a fixed set of GPUs for its entire lifecycle, and (2) a strict temporal synchronization, which mandates that all samples within an iteration launch simultaneously. Confronted with the profound heterogeneity of T2V data, these twin rigidities are the direct cause of the severe workload imbalance and resource underutilization that plague current systems. To overcome this rigidity, we introduce fine-grained spatiotemporal orchestration, a paradigm enabled by decomposing each sample’s computation into minimal, schedulable units (i.e., cascades). Such fine-grained decomposition unlocks independent control across both dimensions: on the temporal
axis, we can precisely schedule each cascade’s launch time and assign it a bespoke parallel configuration to control its duration. On the spatial axis, we can map it onto a specific set of GPUs to create a communication-efficient physical layout. This flexible orchestration mitigates system idle time inherent in rigid schemes, thereby improving load balance and minimizing the end-to-end training makespan. IV. S YSTEM D ESIGN A. Overview Arachne is a distributed training framework for T2V workloads by addressing the challenges outlined in §III. A high-level view of Arachne is shown in Fig. 4, which illustrates its three main components: Cascade-level parallelism planner, Topology-aware resource mapper, and Runtime executor. We first provide an overview of each component and detail its design in the following subsections. Cascade-level parallelism planner. The planner ( Fig. 4- 1 ) performs temporal orchestration, directly dismantling the strict synchronization of traditional paradigms. It first decomposes each sample’s execution into a DAG of fine-grained, schedulable cascades. It then formulates the scheduling of these cascades as an optimization problem to minimize the global theoretical makespan, generating a logical execution plan that co-optimizes the launch time and parallel configuration for each cascade. Topology-aware resource mapper. While the planner generates a temporally optimal logical plan, it does so on a homogeneous hardware abstraction. The mapper (illustrated in Fig. 4- 2 ) addresses this by performing spatial orchestration to create a communication-efficient plan that breaks the spatial rigidity of fixed resource allocation. To achieve this, it refines the logical DAG via a two-level spatial arrangement: (1) At the intra-cascade level, it selects optimal physical GPUs to minimize a cascade’s internal communication. (2) At the intercascade level, it co-locates dependent cascades to maximize data locality and reduce transfer costs. Runtime executor. Runtime executor (illustrated in Fig. 43 ) is the engine that enacts the complete spatio-temporal optimized execution plan crafted by the planner and mapper. It brings the optimized schedule to life by launching each cascade on its designated GPUs according to the precise
timeline. Critically, the Executor manages the complex intercascade dependencies, handling state transitions (e.g., data handoffs) and synchronizations across different parallelism groups. To ensure correctness amidst this highly dynamic execution, it also implements a novel synchronization primitive for heterogeneous gradient accumulation, guaranteeing efficient and accurate gradient aggregation.
T ≥ ec ,
∀c ∈ C
(3)
sc ≥ ep , ∀p ∈ Pc , ∀c ∈ C X xc,k = 1, ∀c ∈ C
(4)
X
(Memc,k · xc,k ) ≤ W,
Example 3. Consider the case depicted in Fig. 4- 1 . The cascade is characterized by: (1) input data: one 37-frame 720p video clip; (2) model specification: DiT; (3) parallelization strategy: SP degree of 4; (4) GPU allocation: GPU 0 and 1 on both node A and node B; and (5) launch time: T3 . TABLE I: Nomenclature for the MILP Formulation. Decision and Auxiliary Variables Launch time of cascade c End time of cascade c Binary variable: 1 if cascade c uses parallelism degree k, 0 otherwise The makespan of the schedule (objective to be minimized)
T
Sets and Indices I M C
Set of bucket samples i in the current batch Set of model modules m (e.g., VAE) Set of all cascades c, where each c corresponds to a pair (i, m) Set of all available parallelism degrees k Set of active cascades at time t, where At = {c ∈ C | sc ≤ t < e c } Set of direct predecessors of cascade c
K At Pc Parameters Lc,k Memc,k N W
Latency of cascade c with parallelism degree k Peak memory requirement of cascade c with parallelism degree k Total number of available GPUs Maximum memory capacity of a single GPU
Minimize
T
k · xc,k ≤ N,
∀t ≥ 0
(7)
Problem Formulation. We formalize cascade scheduling as a makespan minimization problem, aiming to determine when each cascade executes and how much parallelism it uses to minimize overall execution time. To focus on the intrinsic scheduling complexity, we treat GPUs as a homogeneous resource pool and intentionally abstract away concrete GPU placement; the resulting formulation therefore captures a logical scheduling optimum that jointly optimizes execution order and parallelism under dependency, memory, and aggregate resource constraints. The primary input is a dependency graph constructed from each training batch, where nodes represent cascades and edges encode data dependencies. Based on this abstraction, we model the scheduling problem as a Mixed-Integer Linear Programming (MILP) formulation, with notation summarized in Table I. The objective is to minimize the total makespan (T ) by jointly optimizing each cascade’s launch time sc and parallel configuration xc,k . The formulation enforces fundamental constraints on unique parallelism selection, precedence ordering, and per-GPU memory limits (Eqs. (4)–(6)), as well as an event-based resource constraint that bounds the number of concurrently active GPUs (Eq. (7)). The efficacy of our MILP formulation (§IV-B) requires accurate apriori performance estimates. To this end, we developed two cost models2 to predict the execution latency Lc,k and peak GPU memory Memc,k for each cascade c with any potential parallelism degree k. Cost Model for VAE. The VAE’s 3D convolutional layers generate prohibitively large activation tensors when processing high-resolution or long-duration videos [9], [27]. To prevent out-of-memory (OOM) errors, the VAE distributes independent volumetric sub-tiles across GPUs. While this tile-level parallelism ensures memory efficiency, it severely complicates performance prediction. We resolve this via a hierarchical model synergizing lightweight simulation with machine learning. To estimate end-to-end latency for a parallelism degree k, a top-level simulator distributes standardized “base tiles” across workers round-robin, bounding the total latency by the makespan of the bottleneck GPU. Crucially, individual basetile latencies are predicted via a two-stage hybrid approach: an analytical baseline (linear regression on tile volume) refined by an XGBoost [8] model, which fits residuals using one-hot
(1)
subject to: ec =
(6)
c∈At k∈K
A fundamental question for Arachne is how to determine the optimal execution strategy for a given training step. To this end, Cascade-level parallelism planner decomposes the training process into a set of basic schedulable units, referred to as cascades, and formulates their execution as a global makespan minimization problem, thereby deriving an optimized execution plan. In this section, we first provide a formal definition of the cascade and then present the corresponding optimization formulation. D EFINITION 1. (Cascade) A cascade is the minimal, selfcontained computational unit that can be directly executed within existing distributed training frameworks. Each cascade defines the configuration required for distributed execution, including input data, module specifications, parallelization strategy, GPU allocation, and scheduling parameters.
sc ec xc,k
∀c ∈ C
k∈K
X X B. Cascade-level Parallelism Planner
(5)
k∈K
X k∈K
(Lc,k · xc,k ) + sc ,
∀c ∈ C
(2)
2 We exclude the cost model for the text encoder in the T2V model, as its contribution to the total makespan is negligible compared to the dominant bottleneck of processing long video sequences. This is due to the typically short text sequences in T2V training (e.g. under 256 tokens).
encoded spatial dimensions (e.g., 256 × 256) to capture shapedependent hardware execution efficiencies. Cost Model for DiT. For the Transformer-based DiT module, the cost model provides peak memory and latency estimates essential for the planner to derive memory-feasible, makespanminimizing schedules. We formulate these estimates using batch size B and sequence length S. To prevent OOM errors, the per-GPU peak memory (MemDiT ) accounts for a fixed model-state component (Memstates )—sharded according to the device count N and ZeRO stage—and a variable activation component. This activation memory is distributed across k GPUs based on the total token count (B ·S) and the empirically profiled per-token memory (Memtoken ): B · S · Memtoken (8) MemDiT (B, S, k) = Memstates + k The latency model (LDiT ) decomposes execution time into parallelizable computation and communication. To reflect on the DiT architecture, it utilizes regression coefficients α1 and α2 to model linear operations (e.g., MLPs) via α1 (B·S), while capturing the O(S 2 ) complexity of the self-attention mechanism via α2 (B · S 2 ). The unified formulation is expressed as: 2
LDiT (B, S, k) =
Vcomm α1 (B · S) + α2 (B · S ) + (9) k BWeff (k)
where Vcomm is the communication volume, and BWeff (k) is the effective bandwidth determined by the parallelism degree k and its corresponding intra- or inter-node characteristics. Algorithm 1 Genetic Algorithm for Cascade Scheduling 1: Input: cascades C, cost model M, and number of GPUs N 2: Hyper-parameters: population size p, generations G, mutation
rate µ, and elitism size ϵ
3: Output: best makespan T ∗ and best schedule S ∗ 4: T ∗ ← ∞, S ∗ ← None ▷ Initialize the best-found solution 5: P ← I NITIALIZE P OPULATION(C, M, p) ▷ Initialize a
population P of p chromosomes 6: for g = 1 → G do 7: {Si }, {Ti } ← E VALUATE F ITNESS(P, C, M, N )
▷ Find a short makespan 8: Tgen , Sgen ← F IND B EST I N G ENERATION({Si }, {Ti }) 9: if Tgen < T ∗ then ▷ Update the global best solution 10: T ∗ ← Tgen 11: S ∗ ← Sgen 12: end if 13: P ← E VOLVE N EXT G ENERATION(P, {Ti }, µ, ϵ) ▷ Via elitism, crossover, and mutation 14: end for 15: return T ∗ , S ∗ ▷ Return the best schedule
Genetic Algorithm for Heuristic Cascade Scheduling. While the MILP formulation in §IV-B defines the theoretical optimum, the multi-module architecture of T2V workloads (e.g., Text Encoder, VAE, DiT) inherently decomposes each sample into multiple distinct cascades, across which the joint optimization of parallelism configurations and timeline scheduling induces a combinatorial explosion. We therefore
adopt a genetic algorithm (GA) as an anytime heuristic to decouple these decision dimensions, rendering the massive search space computationally tractable and yielding effective schedules within practical runtime constraints. Specifically, each chromosome encodes a complete assignment of parallelism degrees {xc,k } for all cascades (Alg. 1, line 5). Its fitness is evaluated in a topology-agnostic manner by a fast list scheduler that heuristically determines launch times {sc } and the resulting makespan by prioritizing the critical path (lines 6–7). The population iteratively evolves through elitism, crossover, and mutation (line 13), enabling efficient exploration of the parallelism space and progressive refinement toward lower makespans. C. Topology-aware Resource Mapper Cascade-level parallelism planner treats GPUs as a homogeneous resource pool, which can result in initial allocations with substantial communication overhead. To mitigate this, Topology-aware resource mapper incorporates physical hardware topology during cascade placement, completing the planner’s logical schedule into a topology-aware, end-to-end execution plan for efficient T2V training. Intra-cascade Placement. Topology-aware resource mapper prioritizes the communication patterns in a single cascade for resource assignments. This is achieved by using a four-level hierarchical prioritization scheme that minimizes communication latency and prevents network bottlenecks. • Priority 1 (Within-Node Placement). The highest priority is to place an entire cascade within a single compute node. That is, if a node has enough available GPUs to satisfy its parallelism degree, Topology-aware resource mapper restricts placement to that node. Such localization ensures that collective communication operations (e.g., All-Reduce, All-to-All) are confined to high-bandwidth, low-latency intra-node interconnects such as NVLink. • Priority 2 (Balanced Cross-Node Placement). When withinnode placement is infeasible, the priority is to distribute the cascade crossing nodes in a balanced manner. By balanced, we refer to an even distribution of inter-node communication that helps prevent localized congestion and reduces the likelihood of network hotspots. Example 4. For a cascade with a parallelism degree of 4 (SP=4), the asymmetric 1+3 placement in Fig. 5(a) funnels cross-node traffic through a single shared path, leading to severe contention on the PCIe bus, NIC, and the corresponding switch uplink/downlink resources (indicated by the fire symbol). In contrast, the symmetric 2+2 placement in Fig. 5(b) balances communication across NICs, thereby mitigating congestion and improving performance. • Priority 3 (NIC-Coordinated Placement). Given a balanced inter-node placement (e.g., a 2+2 distribution), the third priority is to select specific GPUs that optimize data paths across the network fabric. This strategy leverages a common multi-rail cluster architecture where same-numbered NICs across different nodes are connected to the same leaf switch.
GPU↔NIC: PCIe Gen5 x16 (~64 GB/s one-way)
Spin 1
Spin 2
Spin 1
NIC↔Leaf / Leaf↔Spine: NDR 400 Gb/s/link
Spin 2
Spin 1
Leaf 0
Leaf 1
Leaf 2
Leaf 3
Leaf 0
Leaf 1
Leaf 2
Leaf 3
Leaf 0
NIC 0
NIC 1
NIC 2
NIC 3
NIC 0
NIC 1
NIC 2
NIC 3
NIC 0
Spin 2
Leaf 1
Leaf 2
Leaf 3
NIC 1
NIC 0
NIC 1
PCIE
Server1
Server2
(a) Unbalanced Placement (1+3)
Server1
Server2
(b) Balanced Placement (2+2)
Server1
Server2
(c) NIC-coordinated Balanced Placement
Fig. 5: Comparison of three different resource placement strategies. Example 5. Fig. 5(c) illustrates the placement in which NIC indices are aligned across nodes (0,1 on both), confining all inter-node communication to a single leaf switch (e.g., data from GPU 0 (node A) to GPU 1 (node B) travels via the local PCIe bus to NIC 1 (node A), through leaf switch 1, reaches NIC 1 (node B), and finally arrives at GPU 1 (node B), completely avoiding the high-latency spine switch). The balanced placement in Fig. 5(b), however, assigns node A’s GPUs to NICs 0,1 and node B’s to 2,3, forcing all inter-node traffic through the spine switch. • Priority 4 (Internal Communication-Aware Placement). On modern GPU servers, latency between a GPU and a given NIC may vary due to traversal across PCIe switches or NUMA domains. In cases where multiple GPU combinations achieve equivalent NIC coordination, this strategy considers the topology of intra-node communication and selects the combination that minimizes the cumulative communication distance to the assigned NICs, thereby reducing intra-node latency and ensuring the most efficient path to the inter-node network. Inter-cascade Placement. For interdependent cascades (e.g., a VAE passing its output to a DiT), we co-design placement with the data handoff mechanism introduced in Runtime executor (§IV-D) to enable efficient transfers. Topology-aware resource mapper follows a prioritized approach to achieve this: Its primary objective is to maximize the physical GPU overlap between the producer and consumer. A high degree of overlap allows Runtime executor to perform efficient in-place handoff, where shared GPUs act as a distributed cache and eliminate the need for explicit data movement. When such overlap is not feasible, the mapper optimizes a secondary objective—improving the efficiency of Runtime executor’s handoff engine. It does so by co-locating the main rank of either the producer or the consumer with the central broker rank, thereby confining Gloo-based data transfers to highspeed intra-node interconnects and minimizing overhead for unavoidable data movement. D. Runtime Executor Once it receives the optimized DAG from Topology-aware resource mapper, the primary role of Runtime executor is to translate this static plan into a live, dynamic execution
on the cluster. It achieves this by treating each GPU as a dynamic resource that is rapidly re-tasked according to the DAG’s spatio-temporal schedule. For instance, a single GPU might process a VAE cascade for one data sample and then be immediately re-assigned to a DiT cascade for another. To manage this fluidity, the Executor is responsible for two critical functions detailed below: orchestrating efficient intercascade data handoffs and resolving the complex challenge of heterogeneous gradient accumulation. Optimized Inter-cascade Data Handoff. Runtime executor’s key function in managing inter-cascade data dependencies (e.g., passing a latent tensor from a VAE to a DiT) is an adaptive data handoff mechanism. This mechanism avoids the prohibitive memory footprint of naively buffering tensors globally by leveraging the physical locality of the producer and consumer GPU groups, as determined by Topology-aware resource mapper. The mechanism operates in two modes based on this locality. First, for overlapping GPU groups, it performs a highly efficient in-place handoff, where common GPUs act as a distributed cache. If the consumer group is a superset, a localized broadcast from these caching GPUs efficiently updates the new peers. Second, for disjoint groups, a decoupled asynchronous engine allows the producer to “fire-and-forget” its output tensor via a stream-pipelined pack-and-offload path, where tensors are packed on a secondary CUDA stream and copied to host memory asynchronously, then routed through locality-driven multi-broker nodes over a dedicated CPU-side Gloo channel that runs entirely in parallel with performancecritical NCCL collectives like All-to-All. Consumers further hide residual latency via lookahead prefetching of upcoming transfers. This adaptive strategy ensures inter-cascade data dependencies are resolved with near-zero overhead on the critical training path. Heterogeneous Gradient Accumulation. A key challenge arising from Runtime executor’s dynamic cascade dispatch is heterogeneous gradient accumulation. The DAG schedule breaks the traditional one-to-one GPU-to-sample-gradient mapping, leading to a complex gradient landscape. For instance, while processing four data chunks (A, B, C, D), one GPU subgroup might accumulate a composite gradient of
Stage 1
Stage 2
Average Iteration Time (s)
(Dataset=WebVid, Resolution=360p) 30 25
Stage 3
(Dataset=Koala, Resolution=720p)
(Dataset=Lynx, Resolution=1080p)
50 1.13x
1.09x
1.18x
22.07s
40
50 1.43x
1.42x
1.27x
20
40
1.65x
32.96s
30
30
20
20
10
10
1.59x 1.35x
26.80s
15 10 5
Megatron-LM
DeepSpeed
FlexSP
Arachne
Megatron-LM
DeepSpeed
FlexSP
Megatron-LM
Arachne
DeepSpeed
FlexSP
Arachne
Fig. 6: Average iteration time (in seconds) across three training stages for the HunyuanVideo-13B model. The annotations above the Arachne bars indicate relative speedups compared to the baseline systems. TABLE II: Architectural details of the T2V models evaluated. T2V Models
#Parameters #Layers #Attn Heads Hidden Dims
Wan2.1 CogVideoX HunyuanVideo
1.3B 5B 13B
Megatron-LM Average GPU Idle Ratio (%)
GA +GB , a second subgroup could hold a different composite GC + GD , and a third might only contain a pure gradient GA . This heterogeneity, where GPUs hold fundamentally different pre-summed gradient combinations, renders a conventional global All-Reduce mathematically incorrect, as it cannot disentangle the components. To address this, Runtime executor introduces a two-phase representative-based synchronization strategy. An exact minimum set cover selects the fewest representative GPUs whose gradients collectively span all components. These representatives first All-Reduce among themselves, then each simultaneously broadcasts Gfinal to its non-representative peers, with both phases preferentially reusing existing execution groups. This ensures mathematically consistent gradient updates across all GPUs with minimal synchronization overhead.
30 30 60
12 30 24
DeepSpeed
FlexSP
50 40
1536 1920 3072
Arachne
47.0 45.8
34.6
38.8 37.6 33.6
32.8
30
26.8
23.5
20
12.1
9.0
10
Stage 1
Stage 2
8.1
Stage 3
V. E XPERIMENTS A. Experimental Setup Platform and Protocols. All experiments run on a 64GPU NVIDIA H100-80GB cluster (8 GPUs/node via NVLink, 8×400 Gbps InfiniBand NICs per node) using PyTorch 2.7.1 and CUDA 12.6. End-to-end results (§V-B) are reported on 16 GPUs, while scalability evaluations (§V-C) use up to 64 GPUs. To ensure fair comparison, all systems use the same bucket configuration, adaptive batching, and activation checkpointing. We cap the maximum sequence length at 129 frames as a practical trade-off to maximize data coverage while preventing baseline OOM errors. Performance is averaged over 50 iterations following a 10-iteration warm-up. Baselines. We compare Arachne with three state-of-the-art distributed training systems, namely Megatron-LM [39] and DeepSpeed [37], as well as FlexSP [47], a representative LLM training system optimized for variable-length corpora. Here, Megatron-LM (DP with ZeRO-1 and CP) and DeepSpeed (ZeRO and Ulysses-style SP) serve as strong static baselines, utilizing fixed parallelism configurations (e.g., DP=4, SP=4) manually tuned for optimal performance. FlexSP adapts its SP degree per iteration based on bucket-level workload variations. In contrast to the baselines’ static configurations, Arachne dynamically generates and executes an optimized fine-grained spatio-temporal plan for the cascades in each training iteration. Workloads and Models. To evaluate training performance under realistic conditions, we adopt the progressive curriculum
Fig. 7: Average GPU idle ratio across three training stages.
from Open-Sora [53] (§II-B) across three stages: Stage 1 on WebVid [2] (∼7M clips, 360p), Stage 2 on Koala [46] (∼20M clips, 720p), and Stage 3 on an internal Lynx dataset (∼10M clips, 1080p). We use three representative T2V models spanning 1.3B to 13B parameters to cover diverse architectures: Wan2.1-1.3B [45] (standard self/cross-attention), CogVideoX-5B [48] (3D full attention), and HunyuanVideo13B [20] (hybrid dual/single-stream). The architectural details are summarized in Table II. B. End-to-End Performance Overall Results. The main results for the training of HunyuanVideo are presented in Fig. 6. Overall, Arachne consistently outperforms all baselines across all three stages, with its performance advantage becoming increasingly pronounced from Stage 1 (360p) to Stage 3 (1080p). As data heterogeneity and computational density intensify at higher resolutions, the resulting workload imbalance exacerbates, thereby amplifying the benefits of Arachne’s fine-grained execution. One remarkable outcome is that Arachne achieves peak speedups of 65% over Megatron-LM, 59% over DeepSpeed, and 35% over FlexSP in Stage 3. It is worth noting that the absolute average iteration time in Stage 3 (26.80s) is “counterintuitively” lower than that in
Throughput (Frames/GPU/Second)
Megatron-LM
DeepSpeed
Model Size Scalability
Workload Heterogeneity
Cluster Size Scalability 0.8
1.13x 0.88
1.11x
1.18x
1.19x
1.33x
2.0 1.35x
0.8 1.21x
1.78
0.82
1.24x
0.79 1.21x 1.34x 1.35x
1.25x
1.21x 1.39x 1.40x
1.5
Arachne
1.0
2.51
2.5
FlexSP
0.6
1.43x
1.40x 1.26x 0.71
CogVideoX (5B)
1.22x 1.40x 1.37x
1.40x
0.4 0.2
0.2
0.5
Model Sizes
1.43x
0.68 0.62 1.23x 1.40x
0.4
1.0
Wan2.1 (1.3B)
0.6
0.71 1.26x 1.40x
HunyuanVideo (13B)
53 frames
81 frames
Max Frames
105 frames
16
32
64
#GPUs
Fig. 8: Throughput scalability evaluation of Arachne under increasing training complexity, evaluated on HunyuanVideo-13B, across model size, workload heterogeneity (via larger maximum frame windows), and cluster size. Stage 2 (32.96s). This observation actually stems from hardware memory constraints at 1080p, which cap the maximum sequence length at 57 frames and thus reduce overall computation by skewing the workload distribution toward shorter sequences. Nevertheless, the computational disparity across buckets with different frame counts is substantially larger in Stage 3 (1080p), leading to more severe workload imbalance under static or coarse-grained parallelization. Notably, by effectively mitigating this amplified imbalance through finegrained spatio-temporal orchestration, Arachne is able to capitalize on the increased heterogeneity in Stage 3, yielding its largest relative performance gains in this setting. We next analyze the average GPU idle ratio, defined as the fraction of total training time during which GPUs remain idle, in Fig. 7. Across all three baselines, the idle ratio increases from Stage 1 to Stage 3, indicating that higherresolution training exacerbates workload imbalance. FlexSP partially mitigates this trend by forming heterogeneous SP groups based on per-iteration sequence lengths, but its idle ratio remains substantial (rising to 33.6% in Stage 3). In contrast, Arachne exhibits an inverse trend by dropping its idle ratio to 8.1% across stages, confirming that fine-grained orchestration effectively converts growing data heterogeneity into richer scheduling opportunities rather than wasted cycles. C. Scalability Evaluation To evaluate the scalability of Arachne, we conduct experiments across three dimensions of large-scale training: computational intensity, workload heterogeneity, and cluster size. Our exploration centers around the following three questions: (i) How does Arachne perform as the training task becomes increasingly compute-intensive? (ii) To what extent can Arachne sustain efficiency under diverse and imbalanced workloads? (iii) How well does Arachne scale to larger clusters? Model-size Scalability. We evaluate Arachne’s scalability with computational intensity by benchmarking across three T2V models of increasing size (Wan2.1-1.3B, CogVideoX-5B, and HunyuanVideo-13B) under a fixed Stage 2 configuration, as shown in Fig. 8 (left). The results reveal that Arachne’s
performance advantage consistently amplifies with model scale: its throughput speedup over Megatron-LM grows from 35% on the 1.3B model to 43% on the 13B model. As larger models magnify the penalty of workload imbalance, both rigid static baselines and coarse-grained adaptations (e.g., FlexSP) suffer disproportionately. Arachne, conversely, effectively converts the resulting idle time into productive computation, demonstrating its ability to scale with computational intensity and underscoring its readiness for future, even larger models. Workload-Heterogeneity Scalability. Next, Fig. 8 (middle) evaluates scalability against workload heterogeneity by expanding the maximum frame window (53, 81, and 105 frames) for HunyuanVideo (Stage 2), sampling progressively wider, more imbalanced dataset portions (10.8%, 46.9%, and 70.4% in Fig. 2). As data diversity increases, Arachne’s throughput speedup over Megatron-LM surges from 21% (53 frames) to 35% (105 frames). Fundamentally, Arachne actively exploits this amplified workload imbalance as a richer scheduling space for its fine-grained orchestration, proving its robustness for diverse real-world datasets. Cluster-size Scalability. Finally, we evaluate cluster scalability by expanding the hardware from 16 to 32 and 64 GPUs for HunyuanVideo under a fixed Stage 2 configuration in Fig. 8 (right). Across all tested scales, Arachne maintains a stable and significant throughput advantage of 30% to 40% over both baselines. While scaling up heightens scheduling complexity with more concurrent cascades, Arachne effectively harnesses the simultaneously expanded spatial resource pool to unlock greater placement flexibility, thereby absorbing large-scale workload imbalances and highlighting its robust practicality for large-scale production training environments. In summary, experiments across these three dimensions demonstrate that Arachne exhibits a clear scaling trend, consistently achieving efficiency gains with large-scale training scenarios. This positions Arachne as a robust and futureready system for large-scale training under ever-expanding computational and data demands.
TABLE III: Ablation study on HunyuanVideo-13B, showing the average iteration time (in seconds), with the corresponding slowdown shown in parentheses. “w/o Mapper” denotes removing Topology-aware resource mapper entirely, while “w/o Inter-cascade” and “w/o Intra-cascade” partially disable the inter- and intra-cascade placement optimization, respectively. Method
Stage 1
Stage 2
Stage 3
Arachne
22.07
32.96
26.80
for this iteration is reduced to 31.4s, representing a 30.5% reduction from the 45.2s FlexSP baseline (as shown in Fig. 3).
w/o Inter-cascade 22.98 (+4.13%) 34.29 (+4.04%) 27.78 (+3.64%) w/o Intra-cascade 23.76 (+7.68%) 35.76 (+8.48%) 30.41 (+13.67%) 23.89 (+8.26%) 37.09 (+12.53%) 31.89 (+19.26%) w/o Mapper
Fig. 9: Execution timeline visualization in case study.
D. Ablation Study
E. Case Study To analyze how Arachne achieves its performance gains, we conduct a detailed case study of HunyuanVideo using Stage 2 workloads, combining execution-timeline visualization and workload-balancing analysis. Execution Timeline Visualization We revisit the example introduced in Fig. 3 to provide a before-and-after comparison against the execution schedule generated by Arachne. As shown in Fig. 9, Arachne produces an optimized spatiotemporal execution by exercising fine-grained control over both temporal scheduling and spatial allocation of cascades, breaking the rigid constraints imposed by prior schedulers. Temporally, independent cascades are staggered to fill idle compute periods: shorter cascades from data A and B are dispatched to overlap with longer-running ones. Spatially, Arachne decouples cascades from fixed GPU sets to enable dynamic resource allocation. This behavior is exemplified by data D, whose text encoder, VAE, and DiT cascades are dynamically assigned 4 GPUs (12--15), 6 GPUs (8--13), and 12 GPUs (0--11), respectively. As a result, the makespan
Bottlenect: Rank 8 TFLOPS: 6729.0
Bottlenect: Rank 0 TFLOPS: 5274.9 Bottlenect: Rank 12 TFLOPS: 4630.9
Fig. 10: Per-rank TFLOPS distribution in a training iteration. Hatched regions show under-utilization relative to the bottleneck rank. Coefficient of Variation (CV) measures imbalance. Megatron-LM serves as the representative static baseline.
Coefficient of Variation (CV)
To validate the effectiveness of the resource placement strategies introduced in Arachne for minimizing communication overhead, we conduct an ablation study of Topologyaware resource mapper. All configurations are benchmarked on HunyuanVideo with three-stage training workloads from our end-to-end evaluation (§V-B). Table III presents the results. The results show that disabling Topology-aware resource mapper significantly increases iteration times across all stages, with the most pronounced impact in Stage 3, which slows down by a substantial 19.26%. A finer-grained analysis further reveals a clear hierarchy within the mapper’s components. The intra-cascade placement strategy is the dominant contributor; disabling it in Stage 3 causes a 13.67% slowdown, far greater than the 3.64% slowdown from disabling the inter-cascade strategy. These findings validate our mapper’s design, particularly its emphasis on optimizing intra-cascade communication. More fundamentally, they demonstrate that an optimized temporal plan is only truly effective when coupled with topologyaware spatial orchestration, confirming the necessity of our holistic spatio-temporal orchestration paradigm.
0.8
(Lower is better ↓ )
Megatron-LM FlexSP Arachne
0.6
0.4
0.2
0.0 5
10 15 20 25 30 35 40 Consecutive Training Iterations
45
50
Fig. 11: Temporal stability measured by CV of per-rank TFLOPS over 50 consecutive training iterations. Workload Balancing Analysis. To attribute the observed performance gains to improved workload balancing, we analyze both the spatial distribution and temporal stability of perrank computation. Fig. 10 presents the per-rank TFLOPS distribution within a single training iteration. Compared to all baselines, Arachne achieves a more balanced workload across GPU ranks, as evidenced by both a lower CV and smaller imbalance gaps. Fig. 11, on the other hand, measures the CV of per-rank TFLOPS over 50 consecutive training iterations. Arachne consistently maintains a low CV, remaining around 0.10 across iterations, whereas the baselines exhibit substantially higher variance over time. Taken together, these results demonstrate that Arachne’s fine-grained spatiotemporal orchestration yields a workload allocation that is both spatially balanced and temporally stable, underpinning its sustained training throughput improvements.
VI. C ONCLUSION We presented Arachne, a novel distributed training framework for efficient T2V model training, based on the finegrained spatio-temporal orchestration paradigm. Decomposing the training workload into fine-grained cascades, Arachne optimized their physical placement and execution timeline, addressing the severe workload imbalance caused by data heterogeneity. The comprehensive evaluation on leading T2V models showed that Arachne achieved up to 65% reduction in iteration time compared to state-of-the-art training frameworks. More importantly, it revealed that the performance advantage of this approach scales with both model complexity and workload heterogeneity, confirming its robustness and practicality as a solution for large-scale generative model training in the future. R EFERENCES [1] Ivan Bacher, Hossein Javidnia, Soumyabrata Dev, Rahul Agrahari, Murhaf Hossari, Matthew Nicholson, Clare Conran, Jian Tang, Peng Song, David Corrigan, and François Pitié. An advert creation system for 3d product placements. In Machine Learning and Knowledge Discovery in Databases: Applied Data Science Track - European Conference, ECML PKDD, volume 12460 of Lecture Notes in Computer Science, pages 224–239. Springer, 2020. DOI: https://doi.org/10.1007/ 978-3-030-67667-4 14. [2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. DOI: https://doi.org/10.1109/ICCV48922.2021. 00175. [3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets. 2023. DOI: https://doi.org/10.48550/arXiv.2311.15127. [4] William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. Striped attention: Faster ring attention for causal transformers. 2023. DOI: https://doi.org/ 10.48550/arXiv.2311.09431. [5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. Technical report, OpenAI, February 2024. https: //openai.com/index/video-generation-models-as-world-simulators/. [6] Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation. In International Conference on Learning Representations (ICLR), 2025. [7] Shoufa Chen, Chongjian Ge, Yuqi Zhang, Yida Zhang, Fengda Zhu, Hao Yang, Hongxiang Hao, Hui Wu, Zhichao Lai, Yifei Hu, TingChe Lin, Shilong Zhang, Fu Li, Chuan Li, Xing Wang, Yanghua Peng, Peize Sun, Ping Luo, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Goku: Flow based video generative foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. DOI: https://doi.org/10. 1109/CVPR52734.2025.02190. [8] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. ACM, 2016. DOI: https://doi.org/10.1145/2939672.2939785. [9] Yu Cheng and Fajie Yuan. Leanvae: An ultra-efficient reconstruction vae for video diffusion models. 2025. DOI: https://doi.org/10.48550/ arXiv.2503.14325. [10] Weichen Fan, Chenyang Si, Junhao Song, Zhenyu Yang, Yinan He, Long Zhuo, Ziqi Huang, Ziyue Dong, Jingwen He, Dongwei Pan, Yi Wang, Yuming Jiang, Yaohui Wang, Peng Gao, Xinyuan Chen, Hengjie Li, Dahua Lin, Yu Qiao, and Ziwei Liu. Vchitect-2.0: Parallel transformer for scaling up video diffusion models. 2025. DOI: https://doi.org/10. 48550/arXiv.2501.08453.
[11] Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, and Hongyang Zhang. The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control. arXiv preprint arXiv:2412.03568, 2024. DOI: https://doi.org/ 10.48550/arXiv.2412.03568. [12] Hao Ge, Fangcheng Fu, Haoyang Li, Xuanyu Wang, Sheng Lin, Yujie Wang, Xiaonan Nie, Hailin Zhang, Xupeng Miao, and Bin Cui. Enabling Parallelism Hot Switching for Efficient Training of Large Language Models. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (SOSP ’24), pages 178–194. ACM, November 2024. DOI: https://doi.org/10.1145/3694715.3695969. [13] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, FeiFei Li, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. In Proceedings of the European Conference on Computer Vision (ECCV), volume 15137 of Lecture Notes in Computer Science, pages 393–411. Springer, 2024. DOI: https://doi.org/10.1007/978-3-031-72986-7 23. [14] David Ha and Jürgen Schmidhuber. World models. 2018. DOI: https: //doi.org/10.48550/arXiv.1803.10122. [15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020. DOI: https://dl.acm.org/doi/10.5555/3495724. 3496298. [16] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems (NeurIPS), pages 103–112, 2019. DOI: https://dl.acm.org/doi/10.5555/3454287.3454297. [17] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. 2023. DOI: https://doi.org/10.48550/ arXiv.2309.14509. [18] Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A largescale video dataset with long durations and structured captions. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. DOI: https://dl.acm.org/doi/10.5555/ 3737916.3739467. [19] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. 2013. DOI: https://doi.org/10.48550/arXiv.1312.6114. [20] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. 2024. DOI: https://doi.org/10.48550/arXiv.2412.03603. [21] Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. In D. Song, M. Carbin, and T. Chen, editors, Proceedings of Machine Learning and Systems, volume 5, pages 341–353. Curan, 2023. https://proceedings.mlsys.org/paper files/paper/2023/file/ 80083951326cf5b35e5100260d64ed81-Paper-mlsys2023.pdf. [22] Mario Michael Krell, Matej Kosec, Sergio P. Perez, and Andrew W. Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027, 2021. DOI: https://doi.org/10.48550/ arXiv.2107.02027. [23] Dacheng Li, Rulin Shao, Anze Xie, Eric P Xing, Joseph E Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. Lightseq:: Sequence level parallelism for distributed training of long context transformers. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023). [24] Dacheng Li, Rulin Shao, Anze Xie, Eric P Xing, Xuezhe Ma, Ion Stoica, Joseph E Gonzalez, and Hao Zhang. Distflashattn: Distributed memoryefficient attention for long-context llms training. In First Conference on Language Modeling (COLM), 2024. [25] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experiences on accelerating data parallel training. Proceedings of the VLDB Endowment, 13(12):3005– 3018, 2020. DOI: https://doi.org/10.14778/3415478.3415530. [26] Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence Parallelism: Long Sequence Training from System
Perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2391–2404. Association for Computational Linguistics, 2023. DOI: https://doi.org/10.18653/v1/2023.acl-long.134. [27] Zongjian Li and ... Wf-vae: Enhancing video vae by wavelet-driven energy flow for latent video diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. DOI: https://doi.org/10.1109/CVPR52734.2025.01656. [28] Sungbin Lim, EUN BI YOON, Taehyun Byun, Taewon Kang, Seungwoo Kim, Kyungjae Lee, and Sungjoon Choi. Score-based generative modeling through stochastic evolution equations in hilbert spaces. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 37799–37812. Curran Associates, Inc., 2023. DOI: https://dl.acm.org/doi/10.5555/3666122.3667767. [29] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. 2022. DOI: https://doi.org/10.48550/arXiv.2210.02747. [30] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. [31] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusionbased generative models with scalable interpolant transformers. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. DOI: https://doi.org/10.1007/978-3-031-72980-5 2. [32] Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, YuanFang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. Transactions on Machine Learning Research (TMLR), 2025. [33] Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to-video generation. In The Thirteenth International Conference on Learning Representations (ICML), 2025. [34] NVIDIA. Context parallelism. https://docs.nvidia.com/megatron-core/ developer-guide/latest/user-guide/features/context parallel.html, 2024. [35] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4172–4182, 2023. DOI: https://doi.org/10.1109/ICCV51070.2023.00387. [36] Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, Lei Bai, Wanli Ouyang, and Ruimao Zhang. Worldsimbench: Towards video generation models as world simulators. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. [37] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’20), pages 1–16. IEEE Press, 2020. DOI: https://doi.org/10.1109/SC41405. 2020.00024. [38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. DOI: https: //doi.org/10.1109/CVPR52688.2022.01042. [39] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. 2019. DOI: https: //doi.org/10.48550/arXiv.1909.08053. [40] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In International Conference on Learning Representations (ICLR), 2023. [41] Step-Video Team. Step-video-t2v technical report: The practice, challenges, and future of video foundation model. 2025. DOI: https: //doi.org/10.48550/arXiv.2502.10248. [42] Xin Tan, Yuetao Chen, Yimin Jiang, Xing Chen, Kun Yan, Nan Duan, Yibo Zhu, Daxin Jiang, and Hong Xu. Dynamic sparsity in largescale video dit training. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS ’26, page 101–116, New York, NY, USA, 2025. DOI: https://doi.org/10.1145/3760250.3762216.
[43] The Movie Gen Team at Meta. Movie gen: A cast of media foundation models. 2025. DOI: https://doi.org/10.48550/arXiv.2410.13720. [44] Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. In International Conference on Learning Representations (ICLR), 2025. [45] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, et al. Wan: Open and advanced large-scale video generative models. 2025. DOI: https://doi.org/10.48550/arXiv.2503.20314. [46] Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, Fei Yang, Pengfei Wan, and Di Zhang. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. DOI: https://doi.org/10. 1109/CVPR52734.2025.00789. [47] Yujie Wang, Shiju Wang, Shenhan Zhu, Fangcheng Fu, Xinyi Liu, Xuefeng Xiao, Huixia Li, Jiashi Li, Faming Wu, and Bin Cui. FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’25), pages 421–436. ACM, 2025. DOI: https://doi.org/10.1145/3676641.3715998. [48] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer. In International Conference on Learning Representations (ICLR), 2025. [49] Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. [50] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–tokenizer is key to visual generation. 2023. DOI: https://doi.org/10.48550/arXiv. 2310.05737. [51] Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, and Hao Zhang. Fast video generation with sliding tile attention. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. [52] Peiyuan Zhang, Haofeng Huang, Yongqi Chen, Will Lin, Zhengzhong Liu, Ion Stoica, Eric P. Xing, and Hao Zhang. Faster video diffusion with trainable sparse attention. 2025. DOI: https://doi.org/10.48550/ arXiv.2505.13389. [53] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. 2024. DOI: https: //doi.org/10.48550/arXiv.2412.20404.