Conceptio › Archive › arXiv CS

arXiv CSopen access

Mosaic: Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cloud distributed-computing parallel-computing

distributed computing, parallel computing, cloud

arXiv:2605.18710v1 [cs.DC] 18 May 2026

Mosaic: Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing Yanbo Wang

Yuxuan Wang

Chen Chen∗

[email protected] Shanghai Jiao Tong University China Institute of Artificial Intelligence (TeleAI), China Telecom China

[email protected] Shanghai Jiao Tong University China

Chunyu Xue

Yu Feng

Anbang Wu

[email protected] Shanghai Jiao Tong University China

Quan Chen

Yin Chen

Qizhen Weng

[email protected] Shanghai Jiao Tong University China

[email protected] Institute of Artificial Intelligence (TeleAI), China Telecom China

Abstract

Alignment

With the wide adoption of Multimodal Models (MMs) in real-world scenarios, it is significant to efficiently train the emerging MMs exhibiting increasingly complex module architectures. For MM deployment, existing works allocate a GPU to only one MM module following a temporalmultiplexing manner; this compromises the training efficiency because a single module often fails to attain high GPU utilization. To improve GPU utilization and attain efficient MM training, we propose to deploy MMs in a temporalspatial multiplexing manner, allowing multiple MM modules to colocate on a GPU with well-controlled resource quotas. In this paper, we propose Mosaic, an efficient MM training system applying temporal-spatial multiplexing. We first develop a flexible and lightweight execution engine that supports MM training with arbitrary resource quotas, and then build a comprehensive and accurate performance model to estimate module execution time under any possible allocation plan. With the performance model, we further adopt powerful heuristics to work out high-quality MM deployment plans in an efficient manner. Testbed experiments confirm that Mosaic can effectively improve the training efficiency of popular MMs, with a training speedup of up to 1.31×.

Qwen3 LLM

Encoders Vision

Encoders Text

(a) CLIP [50]

Vision

Text

(b) Qwen3-VL [3]

Figure 1. MMs comprise diverse dependent modules.

26, 83]. As shown in Fig. 1, an MM is typically composed of multiple modules: input encoders, a shared backbone, and output decoders, forming a directed acyclic graph. In particular, given the deepening adoption of MMs in massive edge scenarios, an emerging trend for MM development is to maintain a modest model size (typical edge-grade MMs are smaller than 10B [6, 26, 28, 73]) yet increase the number of involved modalities (i.e., from two or three modals to omnimodals [2, 34, 59]). Moreover, for competitiveness, such MMs often need to be routinely re-trained on the daily-collected data [4, 13, 64, 80], rendering it of paramount significance to train them efficiently with the provisioned GPUs [21, 67]. Given an MM with highly heterogeneous modules, it is challenging to deploy the MM modules over the GPU set for high efficiency. For example, the classical Megatron-LM scheme [54] parallelizes each module over all the GPUs, which is inefficient due to the scaling overhead. DistMM [21] and Spindle [67] schemes instead partition the modules into execution waves that temporally multiplex the cluster and, for each wave, minimize the cross-module execution time gap with module-adaptive GPU allocation. However, in those schemes each GPU is exclusively allocated to only one module

Introduction

With the rapid development of AI techniques, Multimodal Models (MMs) [7, 32, 50, 63] are now widely adopted in many real-world scenarios, especially in edge devices for autonomous driving [20, 62, 70] and embodied intelligence [10, ∗ Corresponding author.

Wang et al.

at a time; in modern MMs with heterogeneous modules, some modules with low compute intensity often fail to achieve high utilization with their allocated GPUs, which further leads to compromised training efficiency (Fig. 4). Therefore, to attain efficient MM training, we need to allow multiple modules to spatially multiplex a GPU for improved utilization. That is, overall we need to enforce temporal-spatial multiplexing for MM deployment: first map the modules of an MM to sequentially-executed stages for temporal GPU multiplexing, and then, within each stage, map each module to a set of compute SMs potentially on multiple GPUs. Ideally, with a proper MM-stage and stage-GPU mapping plan, we can minimize the cross-module bubbles while attaining high GPU utilization, thereby optimizing the MM training efficiency. In realizing this insight, there are three key requirements. First, we need to maintain a flexible execution engine supporting module training with arbitrary SM allocations. Second, we need to build a comprehensive performance model to help evaluate any possible stage-GPU mapping solution. Third, we need to find a high-quality mapping solution in an efficient manner. In this paper, we propose Mosaic, an MM training system that exploits temporal-spatial GPU multiplexing for high efficiency. Mosaic is composed of three parts: (1) Mosaic Multiplexing Engine—which allows multiple MM modules to arbitrarily multiplex a GPU in a flexible and lightweight manner, (2) Mosaic Performance Model—which can accurately estimate the execution efficiency of a module under any possible spatial multiplexing plan, and (3) Mosaic Mapping Solver—which can promptly find a high-quality temporal (MM-stage) and spatial (stage-GPU) multiplexing solution. Specifically, we make non-trivial innovations in each solution aspect. In the Mosaic Multiplexing Engine, considering the SM partitioning granularity and isolation overheads, we choose to adopt the GreenContext (GC) multiplexing technique [42], which allocates fine-grained SM resources to a module with a separate GC-stream; we also propose streampool pre-creation to mitigate GC preparation overheads in the critical path. Meanwhile, in the Mosaic Performance Model, to support fractional SM allocation on multiple GPUs, we build a comprehensive scaling surface for each module; besides, to handle the performance interferences of colocated modules, we add both the additive and multiplicative rectification items to rectify the performance model, which proves to be more accurate than existing modeling methods. Finally, in the Mosaic Mapping Solver, we jointly solve the MM-stage and stage-GPU mapping problem with high quality by combining the Greedy Agglomerative Hierarchical Clustering (GAHC) heuristic with the Google CP-SAT solver; we also incorporate the early-pruning and result-caching techniques for even better solving efficiency. We have implemented Mosaic with 8K LoCs, and evaluated its performance in a testbed with 32 H100 GPUs. Our end-to-end experiments with popular MMs show that, by

making higher GPU utilization with spatial multiplexing, Mosaic improves the training efficiency by up to 1.31× over the state-of-the-art methods. Meanwhile, our deep-dive experiments further confirm that each Mosaic innovation—in the multiplexing engine, performance model, as well as mapping solver—does make non-negligible contribution to the overall performance superiority. In summary, this paper makes three key contributions. • We identify temporal-spatial GPU multiplexing as an effective deployment paradigm for training heterogeneous MMs, where dependent modules temporally share GPUs and parallel modules spatially share SM resources. • We design Mosaic, a practical MM training system that realizes this paradigm through a GreenContext-based multiplexing engine, an interference-aware performance model, and a joint MM-stage/stage-GPU mapping solver. • We implement Mosaic and demonstrate on a 32-H100 testbed that it improves training efficiency by up to 1.31× over state-of-the-art MM training schemes.

Background and Motivation

2.1

Multimodal Models: The Emerging Trend

Multimodal Models (MMs), which jointly process inputs of diverse forms (e.g., text, vision, and audio) [1, 15, 32], are increasingly significant for diverse scenarios like embodied intelligence [31] and autonomous driving [76]. As illustrated in Fig. 1 and Fig. 2, an MM is composed of multiple (encoder, decoder, and backbone) modules with mutual dependencies, which form a Directed Acyclic Graph (DAG). For example, CLIP [50] is a classical MM that aligns the features of different modalities via contrastive learning, and Qwen3-VL [3] uses an LLM backbone to merge the inputs respectively from the vision and text encoders. In particular, given the widespread and also in-depth adoption of MMs in many real-world application scenarios [37, 57, 61]—where the MMs are increasingly deployed on edge hardware like cars, robots, and phones [26, 53, 56, 74, 81], the recent days have witnessed a trend to develop modest-size yet high-modal-complexity MMs: maintaining a modest size allows the MM to be hosted by low-end edge devices, and supporting more modalities can make the MM more powerful for realistic tasks. Fig. 2 shows such MM examples. ImageBind [16] aligns 6 modalities (image, text, audio, depth, thermal and IMU ) into one embedding space, Unified-IO 2 [35] combines 5 encoders and 3 decoders with an LLM backbone, and OFASys [2] integrates 9 encoders and 6 decoders with a universal model. As revealed by Table 1, those modules have strong architecture heterogeneity. Moreover, as shown in Fig. 2d, in some cases multiple MMs may work together on a shared device [31, 69], which also form a logically-complex MM requiring joint training. Meanwhile, for competitiveness, MMs need to be timely refreshed with routinely collected data. Recent studies already 2

Mosaic : Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing

CNN

BEP

Audio

Video

Struct

GAN

G2P

Box

Imu

Depth

Models (LLMs) typically of a large size, the dependent modules (aka model layers) are often deployed in GPU clusters following the Pipeline Parallel (PP) paradigm [22, 39]. However, PP deployment incurs warm-up and cool-down bubbles which waste the GPU resources; for our targeting edge-grade MMs which are not large (and can thus be kept in GPU memory during training), it would be more efficient to have them temporally-multiplex the GPUs [21, 54, 67], such that no GPUs have to stand by idly waiting for the output of upstream modules. Then we turn to deploying the parallel modules, which are particularly challenging for MMs due to the substantial module heterogeneity.

Heat

Vicuna

Alignment

Uni-2 LLM

OFASys LLM

OPT

(a) ImageBind

(b) Unified-IO 2

(d) CTVLM

Figure 2. Edge-grade MMs with high modal complexity. Model

Module

Layers

Dim.

TFLOPs

Qwen3-VL (8.1B)

Qwen3LLM Vision Text

36 27 1

4096 4096 4096

22.27 2.58 0.15

145.2 82.4 2.1

Unified-IO 2 (3.8B)

UIO-2 LLM Vision Audio Text

48 11 11 1

3072 768 768 3072

16.70 1.48 1.06 0.10

110.5 24.6 21.8 4.5

ImageBind (1.2B)

Vision Audio Text

24 12 12

1024 768 768

4.17 2.09 1.04

35.2 22.8 20.5

OFASys (6.3B)

OFASys LLM Vision Text Audio

36 8 4 6

1280 1280 1280 1280

4.80 1.35 0.72 0.95

41.6 18.2 12.5 14.8

Deploying parallel MM modules. Regarding the deployment of parallel (yet heterogeneous) MM modules, a series of works have been proposed in the literature. As illustrated in Fig. 3a, the classical Megatron-LM framework [54] adopts a symmetric allocation strategy, copying each module to all the provisioned GPUs following the Data Parallel (DP) paradigm [30] (different modules are still executed sequentially via temporal multiplexing). Although straightforward, this paradigm is often inefficient: the usually overaggressive parallelization would amplify the communicationto-computation ratio and incur remarkable per-module execution slowdown. To address such inefficiency, later MM deployment works enable asymmetric, module-adaptive GPU allocation. As shown in Fig. 3b, DistMM [21] assigns disjoint sets of GPUs to different modules (each module still enforces DP over its allocated GPUs). Its objective is to balance the execution times of parallel modules; however, because resource allocation is restricted to integer GPU counts, the training system frequently suffers from sub-optimality due to rounding error. These errors may leave substantial duration misalignment between modules, resulting in resource idle time prior to the cross-module data-merging barriers. To further alleviate such rounding-caused module duration inconsistency, as shown in Fig. 3c, Spindle [67] proposes to decompose the modules into finer-grained slices. Note that, although this improved granularity allows for better temporal alignment, it comes at the cost of significantly higher programming complexity and increased coordination overhead due to the frequent synchronization required between slices.

Table 1. Architecture information of representative MMs (for TFLOPs information, the input configuration is the same as in Table 2). Compute intensity (CI) is measured in FLOPs/Byte.

report treating MM updating as daily workloads: FoMo-inFlux [64] and D-MoLEs [14] have both focused on continual MM training in real-world scenarios, and Modyn [4] has further explored how to orchestrate recurring model updates in data-centric pipelines. Hence, it is increasingly significant to train MMs efficiently with the provisioned GPU set. 2.2

Efficiency-oriented MM Deployment

Optimizing model deployment for high training efficiency is a classical research problem in the literature, yet it remains challenging for MMs comprising heterogeneous modules organized in DAGs. Essentially, deploying a DAG-structured MM involves two decision aspects: (1) how to deploy the dependent modules, and (2) how to deploy the parallel modules. Here, we walk through existing deployment methods concerning the two aspects.

Common limitations of existing deployment paradigms. However, despite the differences, the aforementioned paradigms commonly exhibit a fundamental limitation: they assign each GPU to only one MM module, assuming that each module must exclusively occupy the GPUs allocated to it; such an allocation restriction inherently compromises the resultant MM training efficiency. In fact, it often occurs that a single MM module fails to attain high utilization on the GPUs allocated to it. Specifically, in Table 1, we list the module-level architecture information

Deploying dependent MM modules. Regarding the deployment of dependent MM modules, the data dependencies between upstream and downstream modules prevent their concurrent execution. For conventional Large Language 3

Wang et al. 346.14ms

T V

Idle

Time

(a) Megatron-LM

(b) DistMM

T V

Time

T V

A Time

(a) A Colocated Method

Figure 3. Behaviors of different MM deployment schemes when training the CLIP model on four GPUs. V, T, and A denote vision-encoder, text-encoder and alignment modules.

,WHUDWLRQ7LPH *388WLOL]DWLRQ

*388WLOL]DWLRQ

376.8ms

,WHUDWLRQ7LPHPV

416.5ms

432.9ms V

WURQ LVW00 LQGOH RVDLF 0 ' 6S

D 0HJ

(b) Efficiency/utilization Results

Figure 4. Benefits of spatial GPU multiplexing. Allowing vision (V) and text (T) encoders to colocate reduces total iteration time and improves overall GPU utilization.

of typical MMs, including their Compute Intensity (CI). Table 1 suggests that the per-module compute intensity can vary by over an order of magnitude across different modules, exhibiting strong cross-module heterogeneity. For example, for the Qwen3-VL-8B model, the text-encoder module has a CI of 2.1 FLOPs/Byte, the vision-encoder module has a CI of 82.4 FLOPs/Byte, yet the LLM backbone as a module has a CI of 145.2 FLOPs/Byte. In the meantime, modern GPUs feature hundreds of streaming multiprocessors (SMs) [41], and a module with low CI usually fails to fully monopolize those SMs, thereby yielding low GPU utilization. By contrast, by relaxing the exclusive GPU allocation constraint, we can potentially attain higher GPU utilization and further yield faster MM training. As shown in Fig. 4a, with modern spatial multiplexing techniques [40, 42, 43], we can colocate multiple MM modules on a GPU. By properly allocating a portion of the GPU SMs to each module based on its computing characteristics, it is possible that each colocated module can complete at around the same time. In this way, the provisioned GPUs can (1) avoid temporal resource wastage while (2) attaining higher utilization during their busy time. To confirm, we train the CLIP model on a 4×H100 GPU server under different deployment paradigms. As revealed by Fig. 4b, compared to those existing paradigms in Fig. 3, a spatial-multiplexing-enabled deployment solution illustrated in Fig. 4a can indeed attain a higher GPU utilization (with an improvement of 29.9% over the status quo Spindle paradigm) and further a shorter iteration time. This result confirms the necessity to enable spatial GPU multiplexing in MM deployment.

Mosaic Solution Design

In this section, we present the design details of Mosaic, an MM training system exploiting temporal-spatial GPU multiplexing for high compute efficiency. We will first mathematically formulate the problem and make the system overview in Sec. 3.1, and then we elaborate each design component in Sec. 3.2, Sec. 3.3, and Sec. 3.4. Finally, we discuss some peripheral design issues in Sec. 3.5. 3.1

Problem Formulation and System Overview

Problem formulation. For clarity, we first formulate the problem of efficiency-oriented MM deployment. We represent each MM training job as a DAG, ⟨M, E⟩, where M denotes the MM modules and E denotes the data dependencies between modules. Under the temporal-spatial multiplexing mode, MM training needs to be executed in sequential stages, each comprising a set of parallel modules: modules in different stages temporally multiplex the GPU resources, whereas modules within a stage spatially multiplex the GPU resources. Following this mode, the MM deployment problem can be expressed as conducting two-level mappings: (1) MM-stage mapping—map the MM modules to temporally ordered stages with the module dependency preserved, and (2) stage-GPU mapping—map the parallel modules of each stage to the GPU resources with fine-grained SM allocation enabled. Next, we respectively formulate the two mapping problems. 1) Formulating the outer MM-stage mapping problem. Suppose the execution plan consists of 𝑛 (𝑛 is also a decision variable) stages S = ⟨𝑆 1, 𝑆 2, . . . , 𝑆𝑛 ⟩, which are executed sequentially on a pool of GPU resources G. We let 𝑇iteration represent the training iteration time which is the optimization target, then the MM-stage optimization problem can be formulated as:

Insight. In summary, when training MMs comprising heterogeneous modules, we need to exploit temporal-spatial GPU multiplexing for best training efficiency. For dependent MM modules, we need to adopt temporal GPU multiplexing to avoid resource bubbles; more importantly, for parallel MM modules, we need to adopt spatial GPU multiplexing to enhance the GPU utilization. Next, we will explore how to enforce this insight into practice. 4

Mosaic : Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing

∗ 𝑇iteration (M, G) =

𝑛 ∑︁

min 𝑛,S=⟨𝑆 1 ,𝑆 2 ,...,𝑆𝑛 ⟩

∗ 𝑇stage (𝑆𝑖 , G),

techniques [40, 42, 43], they, however, lack the allocation amount flexibility or bring large overheads when preparing the desired SM execution slots for MM modules in each stage. • Second, to solve the above optimization problems, a prerequisite is to obtain a comprehensive performance model, 𝑇module (𝑚, A), that can estimate the execution time of a module when colocated with other modules at arbitrary resource ratios. However, with partial GPU allocation allowed, the performance modeling space is much larger than existing ones; worse, due to complicated interferences among colocated modules, it is hard to accurately predict the module performance for arbitrary colocating plans. • Third, it also presents a non-trivial challenge to solve the formulated problems in a practical manner. Given the increasing MM complexity and the nested nature of the two optimization problems, the solution space is highly complex, and it is hard to find a high-quality deployment solution within a modest time budget.

(1)

𝑖=1

s.t.

𝑆𝑖 = M,

(2)

𝑖 ∈ [𝑛] −1

(𝑚) < S −1 (𝑚 ′ ),

∀(𝑚, 𝑚 ′ ) ∈ E. (3)

The objective formula (1) means to minimize total iteration time, which, given the temporal multiplexing nature, is modeled as the sum of all stages’ execution times. Here ∗ 𝑇stage (𝑆𝑖 , G) means the shortest possible execution time of stage 𝑆𝑖 given the GPU set G, which is obtained by solving the later stage-GPU mapping problem. The constraint formula (2) states that all modules are covered, and formula set (3) ensures the data dependency is complied with by the mapping plan, where S −1 (𝑚) denotes the index of the stage that contains module 𝑚. 2) Formulating the inner stage-GPU mapping problem. In solving the above optimization problem, for each candidate MM-stage mapping solution (i.e., fixing 𝑆𝑖 and G), we need ∗ to acquire 𝑇stage (𝑆𝑖 , G)—the shortest stage execution time attained under the optimal module deployment (stage-GPU 𝑔 mapping) plan. Specifically, we let 𝑎𝑚 ∈ [0, 1] be the SM 𝑔 𝑔∈ G quota of GPU 𝑔 allocated to module 𝑚, A = 𝑎𝑚 𝑚∈𝑆 be the overall stage-GPU mapping plan, and 𝑇module (𝑚, A) be the execution time of module 𝑚. Then the stage-GPU mapping problem can be formulated as: ∗ 𝑇stage (𝑆, G) =

max 𝑇module (𝑚, A)

min

To harvest the benefit of temporal-spatial multiplexing in real-world MM deployments, we need to address all three challenges in our proposed system. System overview. As shown in Fig. 5, we develop Mosaic, an MM training system that exploits temporal-spatial GPU multiplexing for high training efficiency. Mosaic is composed of three components: the Mosaic Multiplexing Engine, the Mosaic Performance Model, and the Mosaic Mapping Solver.

(4)

• First, the Mosaic Multiplexing Engine supports module training with an arbitrary number of SMs on a GPU. To support such flexibility while being light-weight for runtime use, we adopt the Green Context (GC) technique and further propose GC-stream pool precreation to amortize the maintenance overheads. • Second, the Mosaic Performance Model can accurately estimate the execution time of an MM module when it is allocated any number of GPU SMs and colocated with any MM modules. We build a scaling surface for comprehensive modeling and also incorporate the additive and multiplicative interference items for accurate modeling. • Third, the Mosaic Mapping Solver can efficiently find a high-quality temporal (MM-stage) and spatial (stageGPU) multiplexing plan for an MM. It combines the Greedy Agglomerative Hierarchical Clustering (GAHC) heuristic with the CP-SAT solver to get high-quality solutions, and also adopts early-pruning and resultcaching techniques to further enhance the solving efficiency.

𝑔∈G

A={𝑎𝑚 }𝑚∈𝑆 𝑚∈𝑆 𝑔

s.t.

∑︁

𝑔

𝑎𝑚 ≤ 1,

∀𝑔 ∈ G.

(5)

𝑚∈𝑆

The objective formula (4) aims to minimize the end-to-end stage latency, estimated as the slowest module’s latency. Constraint formulas (5) prevent over-allocation of computation resources. The memory resource constraint can be formulated similarly; it is omitted here for simplicity, but always complied with in practice. Note that existing methods like 𝑔 DistMM and Spindle essentially restrict the allocation 𝑎𝑚 to binary values {0, 1}, and from a mathematical point of view, such a solution space restriction naturally compromises the solution quality. Challenges. While the above formulas clearly depict what an ideal solution should be, it is still difficult to find and apply that optimal solution in practice. Specifically, there are three challenges. • First, to support spatial multiplexing with controlled GPU quota, we need to prepare a flexible and lightweight training engine that can execute MM modules with arbitrary SM allocation amounts. While modern GPUs support a series of spatial multiplexing

Next, we elaborate on the solution design in each Mosaic component in greater detail. 5

Wang et al.

MM Deployment Planning Mosaic Performance Model (§3.3) Per-Module Scaling Surfaces

Deployment

Cross-Module Interference Coefficient

Mosaic Mapping Solver (§3.4) Module-Stage Mapping

GC Streams: Module 0

Module-GPU Mapping

&/,3

4ZHQ9/

(a) VRAM overhead comparison between MPS and Green Context. Green Context features 95% fewer memory overhead.

Mosaic Mulplexing Engine (§3.2)

GC Streams: Module 1

036

... GC Streams: Module 2

2YHUKHDGPV

Module 2

95$0&RQVXPSWLRQ0%

Module 1

Module 3

Module 0

User

0362YHUKHDG *&2YHUKHDG ,WHUDWLRQ7LPHRQ0HJDWURQ/0

&/,3

4ZHQ9/

(b) Latency overhead comparison. Green Context achieves the least overhead, with a latency of less than 30 ms.

Figure 6. Green Context is lightweight in both memory and time overhead.

Figure 5. Overview of our Mosaic system.

3.2

for each module with the desired SM quota. Nonetheless, creating or destroying such GC-stream would incur nonnegligible overheads in the critical path (e.g., for reclaiming the stream objects as well as the associated GC states). Our later testbed evaluations (Fig. 11b) show that, when training the Imagebind model on 8 × H100, it takes up to 6.63% of the per-iteration time to manage the GC-streams in a on-demand (create-and-then-destroy) manner. Such overheads would be substantial when conducting stage transition for multiple times within each MM training iteration. Therefore, to eliminate stream preparation overheads in the critical path, we choose to create a Green Context stream pool in advance. Moreover, we note that MM training is an iterative process where the same spatial-colocation plan would be repeated in each iteration. That is, given the MM modules and the available GPUs, all the desirable GC-stream configurations to appear is essentially a white-box information— acquirable with offline profiling. Therefore, in our Mosaic system, we pre-create—at training commencement—a pool of GC-streams-to-use with the desired SM quotas; once a training stage completes in an iteration, the modules of the next training stage can be immediately launched, without incurring any stream-creation/destruction delay in the critical path. This way, we can make flexible and light-weight resource provisioning at per-module granularity, well supporting the enforcement of our solution paradigm.

Flexible and Light-weight Spatial Multiplexing

To exploit spatial multiplexing for efficient MM training, we need to first provide a flexible training runtime. Specifically, that runtime should (1) support arbitrary partitioning of a GPU’s computing resources to different MM modules— per the solution requirement of Eq. 4. Moreover, as indicated in Fig. 4a, that GPU partitioning scheme should (2) be dynamically-tunable during the training process—given the periodical occurrence of stage transitions. Choosing among popular multiplexing techniques for best flexibility. Based on the two requirements above, we surveyed the popular spatial multiplexing techniques supported by modern NVIDIA GPUs. The Multi-Instance GPU (MIG) [40] technique can partition a GPU into up to seven instances, each fully isolated with its own memory, cache, and compute cores. However, the MIG technique cannot partition GPU SMs at arbitrary portions (must follow fixed configurations). Meanwhile, Multi-Process Service (MPS) [43] is a classical spatial multiplexing technique that allows allocating an arbitrary number of GPUs to a process (i.e., a MM training job in our problem). However, MPS itself does not support fine-grained, intra-process resource isolation at permodule level. Recently, Green Context (GC) [42] has been developed as an advanced spatial multiplexing technique that supports intra-process SM isolation at CUDA stream level; besides, as shown in Fig. 6, its memory and time overheads are much less than MPS. Therefore, we choose to adopt the GC techniques in Mosaic: by mapping a MM module to a CUDA-stream with designated SM quota, we can realize module-level adaptive GPU resource allocation.

3.3

Thorough and Accurate Performance Modeling

With spatial sharing enabled, a MM module can be allo𝑔 cated an arbitrary SM portion on each GPU: 𝑎𝑚 ∈ [0, 1]. This, however, complicates the estimation of 𝑇module (𝑚, A) (the latency of module 𝑚 under allocation A) for two reasons: (1) The allocation space expands from a discrete set of GPU counts to a continuous multi-dimensional surface, making exhaustive profiling infeasible; (2) Spatial sharing introduces complex interference patterns when modules are

Light-weight allocation adjustment with a pre-created stream pool. While Green Context supports fine-grained GPU multiplexing, once a GC stream is launched, its SM quota can however not be adjusted at runtime. Therefore, we need to prepare a distinct Green Context stream (GC-stream) 6

Mosaic : Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing

Heat Encoder Imu Encoder

2.4 1.6 0.8 0.0 0.1

8 0.4

0.7

Rat io

1.0

e gre

Iteration Time

Audio Encoder Depth Encoder

performance model. The scaling surface 𝑇module (𝑚, 𝑑𝑚 , 𝑎𝑚 ) constructed via grid sampling primarily captures computebound (SM) contention. However, MM modules are also significantly I/O-intensive. Under the NVIDIA GC mechanism, co-located modules share a common GPU memory plane, making memory bandwidth contention a major source of execution inefficiency. As illustrated in Fig. 8a, even with a fixed SM quota 𝑎𝑚 , a module’s computing efficiency fluctuates remarkably as the available memory bandwidth varies. As in Fig. 8a, when the text-encoder’s compute quota is fixed, increasing the co-located audio-encoder’s compute quota aggravates bandwidth contention and further prolongs the text-encoder’s iteration time. To model this without exhaustive profiling of module colocation combinations, we use memory bandwidth utilization 𝐵(𝑚, 𝑎𝑚 ) ∈ [0, 1] as a generic interference proxy. This metric is collected concurrently during grid sampling at no extra cost. We then adjust the baseline latency by an I/O interfer𝑔 ence term Δ𝑚 ; for a module spanning multiple GPUs G𝑚 , the execution time is governed by the worker encountering the maximum delay:

Text Encoder Vision Encoder

2.4 1.6 0.8 0.0 0.1

0.4

0.7

Rat io

(a) ImageBind

1.0

ree

g De

(b) Unified-IO 2

Figure 7. MM modules’ scaling curves are smooth with respect to DP Degree and SM Ratio.

collocated onto the same GPU. These challenges render existing solutions inadequate and here we respectively address them. Towards comprehensive modeling with symmetry-based solution pruning and smoothness-based grid sampling. To overcome the first challenge, Mosaic builds a scaling surface through symmetry-based pruning and smoothnessbased sampling. While Mosaic theoretically allows a module to receive heterogeneous SM quotas across different GPUs, such asymmetric configurations are inherently suboptimal for data-parallel replicas: synchronous iterations are limited by the slowest worker, any asymmetry creates stragglers and results in wasted resource cycles on faster GPUs. Consequently, we prune the search space by enforcing symmetric 𝑔 allocations. Let G𝑚 = {𝑔 ∈ G | 𝑎𝑚 > 0} be the set of GPUs assigned to module 𝑚, and 𝑑𝑚 = |G𝑚 | be its data-parallel degree, we then require that: ∀𝑚, ∃𝑎𝑚 , ∀𝑔 ∈ G𝑚 ,

rectified 𝑇module (𝑚, A) = 𝑇module (𝑚, A) + max Δ𝑚 . 𝑔

The remaining question is how to estimate the delay Δ𝑚 when multiple modules are co-located on the same GPU. We observe that linear approximations of aggregate utilization—used in works like LLMStation [19]—fail to capture high-intensity contention. Specifically, as shown in Fig. 8b, the additive-only model attains only 𝑅 2 = 0.86 whereas ours reaches 0.97 (𝑅 2 is the coefficient of determination [9], and a value closer to 1 indicates a better fitting performance), because it underestimates the rapidly increasing runtime when aggregate bandwidth pressure approaches saturation. Consistent with recent studies [55, 79], we find that marginal interference is often multiplicatively augmented by the resource demands of peers. We therefore employ a composite formulation to fit the universal interference coefficients 𝑒 1, 𝑒 2, 𝑒 3 : ∑︁ Ö 𝑔 Δ𝑚 = 𝑒 1 + 𝑒 2 𝐵(𝑚, 𝑎𝑚 ) + 𝑒 3 𝐵(𝑚, 𝑎𝑚 ), (8) 𝑔

𝑔

𝑎𝑚 = 𝑎𝑚 .

(7)

𝑔∈ G𝑚

(6)

By enforcing a uniform quota 𝑎𝑚 across all 𝑑𝑚 replicas, we effectively collapse the high-dimensional allocation mesh into a tractable two-dimensional space – each module’s deployment is thus simplified to a tuple (𝑑𝑚 , 𝑎𝑚 ). Since empirical observations in Fig. 7 show that latency varies smoothly along both dimensions, Mosaic approximates this surface with sparse grid sampling rather than exhaustive measurement. Specifically, we profile 𝑑𝑚 at powers-of-two (e.g., 1, 2, 4, 8) and 𝑎𝑚 at decile increments (i.e., 0.1, 0.2, . . . , 1.0). This sparse mesh provides a high-fidelity approximation of scaling surface the scaling surface 𝑇module (𝑚, A) = 𝑓𝑚 (𝑑𝑚 , 𝑎𝑚 ) while reducing profiling overhead by orders of magnitude, striking a practical balance between model accuracy and characterization efficiency.

𝑚∈ M𝑔

where M𝑔 = {𝑚 ∈ 𝑆𝑖 |𝑔 ∈ G𝑚 } is set of modules co-located on GPU 𝑔. As shown in Fig. 8b, this additive-multiplicative approach accurately tracks non-linear degradation, providing the high-fidelity estimation required for optimal allocation. 3.4

Efficient and High-quality Decision Making

With the ability to accurately estimate 𝑇module (𝑚, A), we can now turn to the scheduling problem as defined in Sec. 3.1. Mosaic needs to partition modules into an ordered list of stages S = ⟨𝑆 1, 𝑆 2, . . . , 𝑆𝑛 ⟩, where each stage 𝑆𝑖 ⊆ M contains modules that execute in parallel, and the stage order respects every dependency in E. For each stage 𝑆𝑖 , Mosaic must also choose an optimal allocation plan A𝑆𝑖 over the

Performance model rectification with awareness to I/O interferences. To address the second challenge—the non-linear interference arising from co-location—Mosaic incorporates an I/O-aware correction mechanism into its 7

Text Enc. Iter. Time (ms) Bandwidth Utilization

100 80

1050

900

40 20

750 0 10 20 30 40 50 Audio Enc. SM Quota (%)

Colocated Runtime (ms)

1200

Bandwidth Utilization (%)

Text Enc. Iter. Time (ms)

Wang et al. 250 200

Algorithm 1 Mosaic’s Temporal-Spatial Mapping Solver

Additive-only Additive-multiplicative Ground Truth

Require: MM DAG ⟨M, E⟩, GPU set G Ensure: Stage list S and allocations ⟨A𝑆1 , . . . , A𝑆𝑛 ⟩ 1: S ← ⟨{𝑚} | 𝑚 ∈ M⟩ in a topological order 2: for all 𝑆 ∈ S do 3: (𝑇𝑆 , A𝑆 ) ← StageEval(𝑆, G) 4: while |S| > 1 do 5: Δbest ← 0; Best ← ⊥ 6: for all (𝑆𝑥 , 𝑆 𝑦 ) ∈ Pairs(S) do 7: if merging 𝑆𝑥 and 𝑆 𝑦 violates E then 8: continue 9: if EarlyPrune (𝑆𝑥 , 𝑆 𝑦 , Δbest ) then 10: continue 11: (𝑇𝑆𝑥 ∪𝑆 𝑦 , A𝑆𝑥 ∪𝑆 𝑦 ) ← StageEval(𝑆𝑥 ∪ 𝑆 𝑦 , G) cached if seen 12: Δ ← 𝑇𝑆𝑥 + 𝑇𝑆 𝑦 − 𝑇𝑆𝑥 ∪𝑆 𝑦 13: if Δ > Δbest then 14: Δbest ← Δ; Best ← (𝑆𝑥 , 𝑆 𝑦 )

150 100 50 0

50 60 70 80 90 100 Aggregate Bandwidth Pressure (%)

(a) Increasing SM quota of the (b) Estimated text-encoder peraudio-encoder affects efficiency formance with the additive-only of the colocated text-encoder and and additive-multiplicative methoverall bandwidth utilization. ods, plus the ground truth.

Figure 8. Memory bandwidth contention does degrade performance, and simple linear modeling is insufficient. We train two modules (text-encoder and audio-encoder) from the OFASys model on an H100 GPU.

15: 16:

GPU set G to minimize stage Í latency. The resulting iteration time is 𝑇iteration (S, G) = 𝑆𝑖 ∈ S 𝑇stage (𝑆𝑖 , G). This decision has two coupled levels. The upper level decides which independent modules should be grouped into the same stage, while the lower level decides whether the modules inside a candidate stage can be placed onto the available GPUs with acceptable latency. The coupling makes exhaustive search intractable: the number of possible stage partitions grows as the Bell number of |M |, and each partition still requires solving a complex per-stage resource mapping problem. Mosaic address this with Alg. 1 which we will detail next.

17: 18:

(Line 12)

if Best = ⊥ then break (𝑆𝑥 , 𝑆 𝑦 ) ← Best Apply merge of 𝑆𝑥 and 𝑆 𝑦 in S.

19: return S, ⟨A𝑆 | 𝑆 ∈ S⟩

for every module, the deployment options whose predicted latency meets the target 𝜏, then ask whether one option per module can be selected and placed within the resource constraints. This joint selection-and-placement check is devised as a multi-constraint feasibility problem and solved with established CP-SAT [47] engine. The smallest feasible 𝜏 is exactly 𝑇stage (𝑆, G). Additionally, we set a tunable hyperparameter SM-quota search granularity to discretize each module’s fractional SM quota into candidate allocation levels. Empirically, we set it to 10% and justify this choice in the Sec. 5.4.

A high-quality algorithm to solve Problem 4: combining GAHC heuristic with CP-SAT solver. Mosaic starts from a conservative stage list in which every module forms its own stage (Line 1). In each round, it considers all pairs of current stages that can be legally merged without violating the dependency order. For a candidate merge 𝑆𝑥 ∪ 𝑆 𝑦 , Mosaic invokes StageEval to compute the optimal latency 𝑇𝑆𝑥 ∪𝑆 𝑦 and the corresponding optimal allocation A𝑆𝑥 ∪𝑆 𝑦 . The merge benefit is Δ(𝑆𝑥 , 𝑆 𝑦 ) = 𝑇𝑆𝑥 + 𝑇𝑆 𝑦 − 𝑇𝑆𝑥 ∪𝑆 𝑦 .

⊲

Enhancing algorithm efficiency with early-pruning and result-caching. The preceding procedure yields highquality mappings, but it is far from fast in practice, largely because evaluating candidate merges repeatedly invokes StageEval. We therefore apply early-pruning and resultcaching to cut this cost. First, early-pruning (Line 9 in Alg. 1) drops candidate merges that cannot possibly beat the best merge gain Δbest already found in the current greedy round. Before running StageEval, we test whether each module in the would-be merged stage still has a non-empty set of deployment options under which a merge gain strictly larger than Δbest is attainable; if any module fails, the pair cannot outperform Δbest and we skip the relatively costly full CP-SAT solver call. Second, result-caching (Line 11 in Alg. 1) memorizes the outcome of StageEval. We store the solved latency and

(9)

Mosaic applies the merge with the largest positive Δ; if no merge can further reduce the iteration time, the search terminates (Line 15∼18). This greedy merging is an instance of Greedy Agglomerative Hierarchical Clustering (GAHC), a classical heuristic for progressive grouping. For each candidate stage proposed by the upper-level GAHC search, StageEval solves the optimal intra-stage allocation (i.e., Eq. (4)). Its min–max objective is hard to hand to a standard optimization solver as-is, so we apply binary search on a scalar target latency 𝜏 and convert it into repeated feasibility tests. For each trial 𝜏, we collect, 8

Mosaic : Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing

allocation (𝑇𝑆 , A𝑆 ) the first time a stage composition 𝑆 appears and return the cached entry on later requests. This is well motivated: the outer greedy loop runs for many rounds and evaluates a large number of candidate merges per round, while committing to only one merge at the end of each round, so the same intermediate stage compositions are revisited often—yielding plentiful cache hits and avoiding repeated CP-SAT work for identical module sets. With the above two techniques, we can effectively reduce search overhead while preserving the solution quality, which will be confirmed by our evaluations later in Sec. 5.3.3. 3.5

inference scenarios, the batch size is determined by the runtime request intensity; such batch size uncertainty yields a larger performance modeling space. Meanwhile, the solution paradigm would also be different (e.g., batch size and resource provisioning plan may also be part of the solution).

Implementation

Mosaic is implemented with ∼8K lines of code, including Mosaic Multiplexing Engine with ∼3K LoC, Mosaic Performance Model with ∼3K LoC, and Mosaic Mapping Solver with ∼2K LoC. Mosaic Multiplexing Engine is a PyTorch-based [46] distributed runtime. It manages module-specific communication groups and reuses them across stages to avoid unnecessary communicator initialization. It also implements the CUDA stream pooling optimization as described in Sec. 3.2. Mosaic Performance Model conducts profiling for each module, sweeping chosen DP degrees and SM quotas, recording iteration time, GPU memory usage, SM utilization, and memory bandwidth utilization into structured JSON files, and further manages scaling curve fitting. Mosaic Mapping Solver implements Alg. 1 with Google OR-Tools CP-SAT [47], encoding group selection, GPU placement, and SM-quota assignment as integer variables with memory under SMcapacity constraints. It stores colocation groups as module bitmasks to realize result-caching for StageEval, and also applies early-pruning for merging candidates before calling the solver.

Discussions

Impact of data heterogeneity. Data heterogeneity is a hot research topic for MM training, which manifests in two aspects: intra-modal heterogeneity and inter-modal heterogeneity. We note that they do not affect the effectiveness of Mosaic. Regarding intra-modal (sample-length) heterogeneity, a series of data preparation techniques, like padding [27], truncation [48] and pruning [52], are commonly used in mainstream training practices [7, 68, 78]. We follow such practices and ensure that the ultimate inputs of a module have the same length. Regarding inter-modal (sample-format) heterogeneity, we note that its impact has already been priced in when building our module-specific performance models: the module execution time is jointly affected by the module size, sample format and batch size, all of which are fixed during the training process and thereby well captured by Mosaic.

Compatibility with pipeline parallelism. While in this paper we focus on modest-size edge-grade MMs where pipeline parallelism (PP) [22, 39] is inappropriate (Sec. 2.2), for cloudhosted cutting-edge MMs with hundreds of billions of parameters [17, 38, 45, 49, 77], PP deployment would be unavoidable, which means to host the encoders/decoders and LLM backbones in separate GPU sets—each forming a pipeline stage. We note that our temporal-spatial multiplexing principle can still be applied together with PP. With colocated module execution, we can enhance the GPU utilization of the encoder/decoder modules. In that sense, some of their allocated GPUs can be yielded to other pipeline stages, which, with proper load balancing methods [65, 66], can help to further improve the end-to-end pipeline execution time.

Evaluation

This section evaluates Mosaic performance on representative MM training workloads over state-of-the-art baseline methods (Sec. 5.1). The end-to-end performance results in Sec. 5.2 suggest that Mosaic can remarkably improve the training efficiency and GPU utilization. Further microscopic studies in Sec. 5.3 demonstrate the effectiveness of each of Mosaic’s core design innovations. In Sec. 5.4 we conduct sensitivity studies to evaluate Mosaic performance across diverse resource scales and SM-quota search granularity. 5.1

Experimental Setup

Hardware. All experiments are conducted on a GPU cluster consisting of 4 HGX nodes, each equipped with 8 NVIDIA H100 (80GB) GPUs and 64 Intel(R) Xeon(R) Platinum 8468 CPU cores. The CUDA version is 13.0. Within each node, GPUs are connected by NVLink with 900 GBps bandwidth. Across nodes, RDMA communication is provided by 8 InfiniBand NICs per node, each with 400 Gbps bandwidth.

Extensibility to inference workloads. When serving MM inference requests, the problem also exists that a single MM module fails to utilize the allocated GPU. Moreover, in some scenarios, a single input sample may trigger multiple MM inference tasks: for example, in V-Bench [23] evaluation, a video sample is processed by multiple MM raters (e.g., DINO [5], ViT [8] and RAFT [60]). In such cases, it is appealing to further enable cross-MM module colocation to improve the overall benchmarking efficiency. That said, we note that the research challenges for inference workloads are different. Different from training scenarios, in typical

Models. In our experiments, we respectively train six representative MM models: CLIP [50], Qwen3-VL [3], UnifiedIO 2 [35], ImageBind [16], OFASys [2], and CTVLM [36]. All of them are elaborated in Sec. 2.1. In particular, Unified-IO 2, ImageBind, and OFASys each can be viewed as a model family, in which, depending on the multi-modal tasks, the modules 9

Wang et al.

Image Video Audio Depth

Sequence length Resolution Channels

Mosaic

Value 2048 tokens

1200 Makespan (ms)

Text

Parameter

512×512 RGB

Frames Resolution

32 512×512

Sampling rate Duration

16 kHz 8s

Resolution

224×224

Thermal

Resolution

256×256

IMU

Axes Sampling rate Duration

6 100 Hz 8s

Action

Sequence length

256 tokens

Box

Coordinates

(𝑥 1, 𝑦1, 𝑥 2, 𝑦2 )

DistMM

Megatron-LM

Qwen3-VL

Uni2 4000

1500

3000

1000

2000

300

500

1000

900 600

ImageBind

0 OFASys

3000

1500

3000

2000

1000

CTVLM 4000

2000

500

1000

1000 0

Figure 9. Per-iteration time when training different MMs. Mosaic

Table 2. Training data configurations. GPU Utilization

provisioned can be flexibly combined to form diverse MMs. Regarding the training input data, as shown in Table 2, we apply the data preparation methods discussed in Sec. 3.5 to enforce a uniform (yet modal-specific) sample length (size) for each module; all are typical values suggested by their official documentations. The training batch size is by default set to 32 and degrades to 16 for single-node experiments.

CLIP

Spindle 60

Megatron-LM Uni2 40

40 20

20 0 ImageBind

GPU Utilization

DistMM Qwen3-VL

Baselines and metrics. We compare Mosaic with the three baselines previously elaborated in Sec. 2.2: MegatronLM [54], DistMM [21], and Spindle [67]. Our primary metric to evaluate the training efficiency is the per-iteration training makespan. Additionally, we report hardware efficiency via Compute-Warps-in-Flight [44]1 . 5.2

Spindle

CLIP 2000

2000 Makespan (ms)

Modality

40 40 20

0 OFASys

CTVLM 60 40

20 0

Figure 10. GPU utilization when training different MMs.

Mosaic does achieve the highest GPU utilization for all the models: on average, the GPU utilization under Mosaic is 47.0%, yet under Spindle, DistMM, and Megatron-LM, they are respectively 38.3%, 31.0%, and 26.8%. In particular, for the highly-heterogeneous OFASys model, the most underutilized GPU under Spindle, which hosts the IMU module, has a utilization of 24.7%, yet under Mosaic that utilization is increased to 56.3%. This demonstrates that, by enabling spatial multiplexing, Mosaic effectively enhances the overall GPU utilization, thereby improving the MM training efficiency.

End-to-End Performance

Fig. 9 depicts the average per-iteration time of each MM under different deployment methods. It confirms that Mosaic consistently achieves the best training efficiency, with 1.07×– 1.31× speedup over Spindle (the second best), 1.10×–1.42× over DistMM, and 1.17×–1.48× over Megatron-LM. Such superiority aligns with our previous analysis in Sec. 2.2. Moreover, we also notice that the performance benefit of Mosaic is relatively larger for more complex MMs: the improvement of Mosaic over Spindle is 1.07× for CLIP, yet it is 1.31× for OFASys (which has 9 encoder modules). This is because for MMs with substantially heterogeneous modules, enforcing full-GPU allocation, as done in existing works, would render some GPUs highly underutilized. To confirm, we further resort to the average per-GPU utilization information shown in Fig. 10. It suggests that

5.3

Microscopic Performance Deep Dive

In this part, we respectively study the effectiveness of a series of techniques adopted in Mosaic. 5.3.1

Superiority of Mosaic Multiplexing Engine.

Recall that in Sec. 3.2, to mitigate the runtime provisioning overhead of diverse GC-streams, we propose to pre-create a pool of GC-streams with the desired quotas. To check its necessity, we measure the average time overhead to create GC-stream of different SM quotas (from 0.1 to 0.9; each point is repeated 30 times, and the error-bars show the upper/lower

1We avoid the misleading coarse-grained GPU Utilization as it only reflects whether any kernel is active and may remain high even when SM resources are under-saturated. In contrast, Compute-Warps-in-Flight directly reflects the amount of compute-active warps on SMs, making it a better indicator.

Mosaic : Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing 80

40 30 0.02

0.01

0.2

0.4 0.6 SM Quota

0.8

900

On-demand 818

800

500

798

765 727

709

700 600

Mosaic

662 594

562

Inference-unaware Additive-item only Full-model

60 40 20 0

d -VL ys P Bin CLI Qwen3 Image OFAS

Normalized E2E Time

On-demand

Prediction Error (%)

Launch Time

Mosaic

Iteration Time (ms)

4 6 Number of Modules

Inference-unaware Full-model Additive-item only

1.0

0.95

0.9

0.93

0.89

0.91 0.87

0.86 0.83

0.8 0.7

0.76

4 6 Number of Modules

(a) GC-stream preparation la- (b) End-to-end iteration time untency under different schemes. der different schemes.

(a) Prediction error across differ- (b) Normalized E2E time under ent modeling strategies. different modeling strategies.

Figure 11. GC-stream pre-creation remarkably reduces provision latency and benefits end-to-end iteration time.

Figure 12. Mosaic’s interference-aware performance model outperforms the other baseline methods in both prediction error and end-to-end performance.

5.3.2

Search Time (s)

10 10 10 10 10 10

Timeout (> Module 6)

2 1 0

−1 −2

Brute-force GAHC GAHC+caching Mosaic mapping-solver

2 4 6 8 10 12 14 16 18 20 Number of Modules

(a) Search time ablation study.

100 Optimal Ratio (%)

bounds). As shown in Fig. 11a, on-demand GC-stream creation takes a time overhead of around 37 ms; in contrast, by creating the GC-streams a priori, the amortized GC-stream launching overhead shrinks to around 0.013 ms, achieving a remarkable overhead reduction. We further evaluate how this GC-stream launching speedup translates to the end-to-end performance benefit. Fig. 11b compares the average per-iteration time of four typical MMs (CLIP, Qwen3-VL, ImageBind, and OFASys) with and without GC-stream pre-creation. Across these workloads, Mosaic reduces end-to-end iteration time by 2.4%–6.6%, with an average reduction of about 4.6%. This confirms that such a technique is indispensable to Mosaic performance superiority.

100% Optimality

96 94 92 90

2 4 6 8 10 12 14 16 18 20 Number of Modules

(b) Quality of Mosaic’s solution compared to genuine optimal from exhaustive enumeration.

Figure 13. Mosaic’s search algorithm finds near-optimal solution with high time efficiency.

Superiority of Mosaic Performance Model.

Recall that in Sec. 3.3, we have built an interference-aware performance model which includes both the additive and multiplicative items. To confirm the superiority of that performance model, we resort to the ablation study depicted in Fig. 12a, in which we evaluate the modeling accuracy for the OFASys model by setting up varying module numbers. Specifically, we evaluate three modeling schemes: (1) inference-unaware, which directly uses the per-module scaling surfaces, (2) additive-item only, which only includes the additive item when fitting the performance model, and (3) full model, which uses the full model depicted by Eq. 8. The ground-truth latency is obtained by actually executing each tested colocation plan. As shown in Fig. 12a, the full-model method consistently attains the highest modeling accuracy. For example, with 8 modules in the OFASys model, it attains an average prediction error of merely 3.65%, much smaller than the other methods (11.39% and 17.85%, respectively). This confirms the superiority of our performance model in Eq. 8. In Fig. 12b, we further show the resultant per-iteration time of the OFASys model under different performance modeling methods; all results are normalized by the value under

the interference-unaware performance model. Fig. 12b reveals that the performance model in Eq. 8 achieves a training speedup of 11%–24%. Moreover, that speedup level increases when the OFASys model comprises more modules, because with more modules, the cross-module performance interference is more salient. In summary, our interference-aware performance model is crucial to the superiority of Mosaic. 5.3.3

Superiority of Mosaic Mapping Solver.

Recall that in Sec. 3.4, to solve the module-stage mapping and module-GPU mapping problem, we apply the GAHC heuristic and also incorporate early-pruning and result-caching for better efficiency. To confirm the effectiveness of those methods, we compare the efficiency and quality performance of Mosaic mapping-solver (Alg. 1) against multiple baselines: (1) brute-force, which finds the best mapping scheme by exhaustive grid search, (2) GAHC, which uses the standard GAHC algorithm without pruning or caching, (3) GAHC+caching, which further incorporates result caching into GAHC, and (4) Mosaic mapping-solver, which incorporates both earlypruning and result-caching with GAHC. 11

Wang et al.

5.4 5.4.1

2.0 1.5 Mosaic Spindle

1.0

DistMM Megatron

16 Number of GPUs

(a) Throughput under different GPU cluster scales.

100

75 90

25 0

30% 20% 10% 5% 1% Profiling Grid Density

Optimal Rate (%)

2.5

(b) Search-time/quality trade-off under different SM-quota search granularities.

Figure 14. Sensitivity of Mosaic to cluster scale and SMquota search granularity on the OFASys model.

Sensitivity Analysis

the median optimality ratio is only 89.10% and 91.18%. Reducing the granularity to 10% (the default value for main experiments) improves the median optimality ratio to 94.27% with a moderate 15.87 s search time, forming a practical knee point in the trade-off. Further refining the search granularity brings limited additional benefit. Compared with 10%, the 5% and 1% settings make the search 2.03× and 6.85× slower, respectively, but improve median optimality by only 0.73 and 1.18 percentage points. These results suggest that a 10% SM-quota search granularity captures most of the benefit of fine-grained spatial allocation while avoiding the rapidly increasing overhead of near-continuous search.

Sensitivity to GPU Cluster Scale.

We first examine whether Mosaic remains effective as the available resource pool scales. This experiment fixes a four-module OFASys workload, varies the GPU pool size from 8 to 32 GPUs and compares their throughput (defined 1 as iteration time ). As shown in Fig. 14a, Mosaic consistently achieves the highest throughput across all evaluated GPU pool sizes. Compared with Megatron-LM under the same GPU pool size, Mosaic improves throughput by 1.68×, 1.48×, and 1.31× on 8, 16, and 32 GPUs, respectively. Against the strongest baseline, Spindle, the corresponding speedups are 1.30×, 1.20×, and 1.12×; against DistMM, they are 1.52×, 1.36×, and 1.22×. This trend indicates that spatial multiplexing is particularly valuable when the GPU pool is tight. With fewer GPUs, integer and exclusive-GPU allocation policies have less room to absorb module-level imbalance, so lightweight modules are more likely to leave sub-GPU compute capacity unused. Mosaic instead allows complementary modules to share GPUs with fractional SM quotas, improving aggregate throughput under the same resource budget. As the GPU pool grows, the relative pressure on each allocation decision decreases, and the speedup gradually narrows, but Mosaic still preserves the best throughput at every scale. 5.4.2

125 Search Time (s)

Normalized Throughput

Fig. 13a shows the search time of the optimal deployment plan for the OFASys model (under varying module number). According to Fig. 13a, as the module number increases, the brute-force method quickly becomes intractable (surpassing the time budget of 1800s). By contrast, all the other methods can find the optimal solution within the time budget for up to 20 modules. In particular, the Mosaic mapping-solver, which combines GAHC with both early-pruning and result-caching, attains the best efficiency; this demonstrates the necessity of the two techniques. Meanwhile, regarding the solution quality, as suggested in Fig. 13b (the optimal plan is obtained by exhaustive search), the deployment plan found by our Mosaic mapping-solver achieves 100% optimality (in per-iteration time) for up to 4 modules, and remains near-optimal as the search space grows. The median optimality ratio is around 94.27% with 10 modules. Therefore, Mosaic mapping-solver performs well in both solution efficiency and quality.

Related Works

In this section, we present other related works additional to those we have mentioned in Sec. 2.2, in aspects respectively in distributed model deployment, spatial GPU multiplexing and efficient training of large-scale MMs. Distributed model deployment. With the increasing use of neural network models, deploying large neural network models on distributed GPUs has been a hot research topic in the literature. Apart from the traditional DP [30], TP [54] and PP [22, 39] parallelization schemes which are elaborated in Sec. 2.1, recently some new parallelization schemes also emerge, including Sequence Parallel (SP) [29], Context Parallel (CP) [33] and Expert Parallel (EP) [51]. Meanwhile, given the increasing model deployment complexity, a series of autoparallelization works, like Alpa [82] and Metis [65] have also been proposed. However, these works commonly focus on Large Language Models, which are highly symmetric and executed in a purely sequential manner; their parallelization methods are over-simplified for MMs that have multiple heterogeneous module branches.

Sensitivity to SM-quota Search Granularity.

We next study the effect of the SM-quota search granularity introduced in Sec. 3.4, to understand how Mosaic balances search time and solution quality. Fig. 14b reports the resulting search time and median optimality ratio compared to the optimal plan found by exhaustive enumeration. Coarse granularities substantially reduce search overhead but compromise the solution quality: at 30% and 20%, the search finishes in 5.32 s and 8.46 s, while 12

Mosaic : Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing

Spatial GPU Multiplexing. Spatial GPU multiplexing has been widely studied to improve GPU utilization, overwhelmingly for inference acceleration. For example, Salus [75] provides fine-grained GPU sharing primitives for DL applications, REEF [18] enables microsecond-scale preemption for concurrent DNN inferences, Orion [58] further improves GPU sharing performance for ML applications through interference-aware scheduling, and MuxServe [11] exploits spatialtemporal multiplexing for LLM serving. To the best of our knowledge, we are the first that exploit spatial GPU multiplexing to speed up the training efficiency of emerging MMs.

[4] Böther, M., Robroek, T., Gsteiger, V., Holzinger, R., Ma, X., Tözün, P., and Klimovic, A. Modyn: Data-centric machine learning pipeline orchestration. Proc. ACM Manag. Data 3, 1 (Feb. 2025). [5] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 9650–9660. [6] Chen, W., Li, Z., and Xin, S. Omnivlm: A token-compressed, subbillion-parameter vision-language model for efficient on-device inference, 2024. [7] Chen, Z., Wang, W., Cao, Y., et al. Expanding performance boundaries of open-source multimodal models with model, data, and testtime scaling. arXiv, 2024. arXiv:2412.05271. [8] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (2021). [9] Draper, N. R., and Smith, H. Applied Regression Analysis, 3 ed. John Wiley & Sons, 1998. [10] Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, P. PaLM-e: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning (23–29 Jul 2023), A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202 of Proceedings of Machine Learning Research, PMLR, pp. 8469–8488. [11] Duan, J., Lu, R., Duanmu, H., Li, X., Zhang, X., Lin, D., Stoica, I., and Zhang, H. MuxServe: Flexible spatial-temporal multiplexing for multiple LLM serving. In Proceedings of the 41st International Conference on Machine Learning (2024), vol. 235 of Proceedings of Machine Learning Research, PMLR, pp. 11905–11917. [12] Feng, W., Chen, Y., Wang, S., Peng, Y., Lin, H., and Yu, M. Optimus: Accelerating {Large-Scale} {Multi-Modal} {LLM} training by bubble exploitation. In 2025 USENIX Annual Technical Conference (USENIX ATC 25) (2025), pp. 161–177. [13] Garg, S., Farajtabar, M., Pouransari, H., Vemulapalli, R., Mehta, S., Tuzel, O., Shankar, V., and Faghri, F. Tic-clip: Continual training of clip models. In International Conference on Learning Representations (2024), B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, Eds., vol. 2024, pp. 16649–16684. [14] Ge, C., Wang, X., Zhang, Z., Chen, H., Fan, J., Huang, L., Xue, H., and Zhu, W. Dynamic mixture of curriculum LoRA experts for continual multimodal instruction tuning. In Proceedings of the 42nd International Conference on Machine Learning (13–19 Jul 2025), A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, Eds., vol. 267 of Proceedings of Machine Learning Research, PMLR, pp. 19011–19033. [15] Gemini Team, Google. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Technical report, 2025. [16] Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023). [17] Google DeepMind. Gemini 3.1 Pro model card. Model card, 2026. [18] Han, M., Zhang, H., Chen, R., and Chen, H. Microsecond-scale preemption for concurrent gpu-accelerated dnn inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (2022), USENIX Association, pp. 539–558. [19] He, Y., and et al. LLMStation: Resource multiplexing in tuning and serving large language models. In Proceedings of the 2025 USENIX

Efficient training of large-scale, cloud-grade MMs. Deploying large-scale, cloud-grade MMs in hundreds or thousands of GPUs has been the research focus of a series of recent works. For example, Cornstarch [24], Optimus [12], GraphPipe [25], PipeWeaver [72], and MegaScale-Omni [71] all seek to advance the parallelization schemes for such largescale MMs, for which PP optimization (against module and data heterogeneity) is the key focus. By contrast, in this paper, we focus on edge-grade MMs, which would be even more significant than cloud-grade MMs in terms of the entity number and scenario diversity [37]. Meanwhile, as elaborated in Sec. 3.5, our temporal-spatial multiplexing method can be jointly applied with more advanced parallelization schemes (exemplified by PP) when training large MMs.

Conclusion

This paper introduced Mosaic, a system for efficient training of MMs comprising heterogeneous modules. Motivated by the growing module heterogeneity of modern multimodal models, Mosaic maps dependent modules into sequential stages while allowing parallel modules within each stage to share GPUs with fine-grained SM allocations. Mosaic realizes this approach with three coordinated components: a flexible GreenContext-based multiplexing engine, a performance model that captures scaling behavior and colocation interference, and a mapping solver that efficiently finds highquality temporal and spatial deployment plans. Our evaluation on representative MMs and a 32-H100 cluster shows that Mosaic consistently improves GPU utilization, achieving up to 1.31× training-efficiency improvement over existing methods. Overall, Mosaic shows that carefully planned temporal-spatial multiplexing can effectively reduce crossmodule bubbles and unlock underutilized GPU resources in MM training.

References [1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). [2] Bai, J., et al. Ofasys: A multi-modal multi-task learning system for building generalist models. arXiv, 2022. arXiv:2212.04408. [3] Bai, S., Cai, Y., Chen, R., et al. Qwen3-vl technical report. arXiv, 2025. arXiv:2511.21631. 13

Wang et al.

Annual Technical Conference (2025), USENIX Association. [20] Hegde, D., Yasarla, R., Cai, H., Han, S., Bhattacharyya, A., Mahajan, S., Liu, L., Garrepalli, R., Patel, V. M., and Porikli, F. Distilling multi-modal large language models for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2025), pp. 27575–27585. [21] Huang, J., Zhang, Z., Zheng, S., Qin, F., and Wang, Y. {DISTMM}: Accelerating distributed multimodal model training. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) (2024), pp. 1157–1171. [22] Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., and Chen, Z. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems (NeurIPS) (2019). arXiv:1811.06965. [23] Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., Wang, Y., Chen, X., Wang, L., Lin, D., Qiao, Y., and Liu, Z. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024). [24] Jang, I., Lu, R., Bansal, N., Chen, A., and Chowdhury, M. Efficient distributed MLLM training with Cornstarch, 2025. [25] Jeon, B., Wu, M., Cao, S., Kim, S., Park, S., Aggarwal, N., Unger, C., Arfeen, D., Liao, P., Miao, X., Alizadeh, M., Ganger, G. R., Chen, T., and Jia, Z. Graphpipe: Improving performance and scalability of dnn training with graph pipeline parallelism. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (2025). [26] Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E. P., Sanketi, P. R., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C. OpenVLA: An open-source vision-language-action model. In Proceedings of The 8th Conference on Robot Learning (2025), vol. 270 of Proceedings of Machine Learning Research, PMLR, pp. 2679–2713. [27] Krell, M. M., Kosec, M., Perez, S. P., and Fitzgibbon, A. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027 (2022). [28] Li, B., Li, Y., Li, Z., Liu, C., Liu, W., Niu, G., Tan, Z., Xu, H., Yao, Z., Yuan, T., Zhou, D., Zhuang, Y., Yan, S., Dai, G., and Wang, Y. Megrez-omni technical report, 2025. [29] Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120 (2021). [30] Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704 (2020). [31] Liu, C., Qian, Y., Tang, D., Zhu, H., Pang, J., and Cai, Q. From insight to action: Embodied multi-agent system integrating vision language model for digital twin-assisted human-robot collaborative assembly. Journal of Manufacturing Systems 85 (2026), 531–556. [32] Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems 36 (2023), 34892–34916. [33] Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. In The Twelfth International Conference on Learning Representations (2024). [34] Liu, Z., Dong, Y., Wang, J., Liu, Z., Hu, W., Lu, J., and Rao, Y. Ola: Pushing the frontiers of omni-modal language model. arXiv preprint arXiv:2502.04328 (2025). [35] Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

(2024). [36] Lu, S., Guo, L., Wang, W., Zhao, Z., Yue, T., Liu, J., and Liu, S. Collaborative training of tiny-large vision language models. In Proceedings of the 32nd ACM International Conference on Multimedia (2024), pp. 4928– 4937. [37] MarketsandMarkets. Edge ai software market worth 8.89 billion by 2031. Web Page, 2026. [38] Moonshot AI. Kimi K2.5: Visual agentic intelligence. Technical blog, 2026. [39] Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., and Zaharia, M. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP) (2019). [40] NVIDIA. Multi-instance gpu user guide. NVIDIA Documentation, 2025. Accessed 2026-01-20. [41] NVIDIA. Cuda c++ programming guide. https://docs.nvidia.com/cu da/cuda-c-programming-guide/, 2026. [42] NVIDIA Corporation. Cuda c++ programming guide: Green contexts. https://docs.nvidia.com/cuda/cuda-programming-guide/04-specialtopics/green-contexts.html, 2025. Accessed: 2026-02-04. [43] NVIDIA Corporation. Cuda multi-process service (mps) overview. https://docs.nvidia.com/deploy/pdf/CUDA Multi Process Service Overview.pdf, 2025. Accessed: 2026-02-04. [44] NVIDIA Corporation. NVIDIA Nsight Systems User Guide: GPU Metrics. NVIDIA Corporation, 2026. Version 2026.2. [45] OpenAI. Introducing GPT-5.4. OpenAI product release, 2026. [46] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019). [47] Perron, L., and Furnon, V. Or-tools. [48] Portes, J., Trott, A., Havens, S., King, D., Venigalla, A., Nadeem, M., Sardana, N., Khudia, D., and Frankle, J. Mosaicbert: A bidirectional encoder optimized for fast pretraining. In Advances in Neural Information Processing Systems (2023), vol. 36. [49] Qwen Team. Qwen3.5-397B-A17B model card. Hugging Face model card, 2026. [50] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (2021), M. Meila and T. Zhang, Eds., vol. 139 of Proceedings of Machine Learning Research, PMLR, pp. 8748–8763. [51] Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. DeepSpeed-MoE: Advancing mixture-ofexperts inference and training to power next-generation AI scale. In Proceedings of the 39th International Conference on Machine Learning (2022), vol. 162 of Proceedings of Machine Learning Research, PMLR, pp. 18332–18346. [52] Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsieh, C.-J. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems (2021), vol. 34. [53] Shao, Z., Yu, Z., Yu, J., Ouyang, X., Zheng, L., Gai, Z., Wang, M., Kuang, Z., and Ding, J. Imp: Highly capable large multimodal models for mobile devices. IEEE Transactions on Multimedia 27 (2025), 2961– 2974. [54] Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019). [55] Shubha, S. S., Shen, H., and Iyer, A. USHER: Holistic interference avoidance for resource optimized ML inference. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), USENIX Association, pp. 947–964. 14

Mosaic : Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing

[56] Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beisswenger, J., Luo, P., Geiger, A., and Li, H. DriveLM: Driving with graph visual question answering. In Computer Vision – ECCV 2024 (2024), Springer, pp. 256–274. [57] Siru, C., Yuanchao, S., Cong, W., and Jiming, C. A survey on edge multimodal large models: compression, inference acceleration, and applications. National Science Open. [58] Strati, F., Ma, X., and Klimovic, A. Orion: Interference-aware, fine-grained gpu sharing for ml applications. In Proceedings of the Nineteenth European Conference on Computer Systems (2024), EuroSys ’24, Association for Computing Machinery, pp. 1075–1092. [59] Team, Q. Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804 (2026). [60] Teed, Z., and Deng, J. RAFT: Recurrent all-pairs field transforms for optical flow. In Proceedings of the European Conference on Computer Vision (2020), pp. 402–419. [61] The Business Research Company. On-device multimodal ai market report 2026. Market report, 2026. [62] Tian, X., Gu, J., Li, B., Liu, Y., Wang, Y., Zhao, Z., Zhan, K., Jia, P., Lang, X., and Zhao, H. Drivevlm: The convergence of autonomous driving and large vision-language models. In Proceedings of The 8th Conference on Robot Learning (06–09 Nov 2025), P. Agrawal, O. Kroemer, and W. Burgard, Eds., vol. 270 of Proceedings of Machine Learning Research, PMLR, pp. 4698–4726. [63] Tschannen, M., et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding. arXiv, 2025. arXiv:2502.14786. [64] Udandarao, V., Roth, K., Dziadzio, S., Prabhu, A., Cherti, M., Vinyals, O., Hénaff, O., Albanie, S., Akata, Z., and Bethge, M. A practitioner's guide to real-world continual multimodal pretraining. In Advances in Neural Information Processing Systems (2024), A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37, Curran Associates, Inc., pp. 133801–133845. [65] Um, T., Oh, B., Kang, M., Lee, W.-Y., Kim, G., Kim, D., Kim, Y., Muzzammil, M., and Jeon, M. Metis: Fast automatic distributed training on heterogeneous GPUs. In 2024 USENIX Annual Technical Conference (USENIX ATC 24) (Santa Clara, CA, July 2024), USENIX Association, pp. 563–578. [66] Wang, Y., Wang, Y., Chen, C., Xue, C., Weng, Q., Chen, Y., Li, Z., Zhu, X., Yang, Y., Chen, Q., et al. Suika: Efficient and high-quality re-scheduling of 3d-parallelized llm training jobs in shared clusters. In Proceedings of the 21st European Conference on Computer Systems (2026), pp. 2002–2021. [67] Wang, Y., Zhu, S., Fu, F., Miao, X., Zhang, J., Zhu, J., Hong, F., Li, Y., and Cui, B. Spindle: Efficient distributed training of multi-task large models via wavefront scheduling. arXiv, 2024. arXiv:2409.03365. [68] Wen, Z., Gao, Y., Li, W., He, C., and Zhang, L. Token pruning in multimodal large language models: Are we solving the right problem?, 2025. Findings of ACL 2025. [69] Wu, Y., Li, D., Chen, Y., Jiang, R., Zou, H. P., Huang, W.-C., Li, Y., Fang, L., Wang, Z., and Yu, P. S. Multi-agent autonomous driving systems with large language models: A survey of recent advances, resources, and future directions. Findings of the Association for Computational Linguistics: EMNLP 2025 (2025). [70] Xu, Z., Zhang, Y., Xie, E., Zhao, Z., Guo, Y., Wong, K.-Y. K., Li, Z., and Zhao, H. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters 9, 10 (2024), 8186–8193. [71] Xue, C., Chen, Y., Jiang, J., Zheng, N., Feng, J., Chen, J., Zhao, S., Yan, S., Lin, Y., Shi, L., et al. Megascale-omni: A hyper-scale, workloadresilient system for multimodal llm training in production. In Proceedings of the 21st European Conference on Computer Systems (2026), pp. 675–692. [72] Xue, Z., Hu, H., Chen, X., Jiang, Y., Song, Y., Mi, Z., Zhu, Y., Jiang, D., Xia, Y., and Chen, H. Pipeweaver: Addressing data dynamicity in

large multimodal model training with dynamic interleaved pipeline. arXiv, 2025. arXiv:2504.14145. [73] Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Chen, C., Li, H., Zhao, W., He, Z., Chen, Q., Zhou, R., Zou, Z., Zhang, H., Hu, S., Zheng, Z., Zhou, J., Cai, J., Han, X., Zeng, G., Li, D., Liu, Z., and Sun, M. Efficient GPT-4V level multimodal large language model for deployment on edge devices. Nature Communications 16, 1 (jul 2025), 5509. [74] Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Chen, C., Li, H., Zhao, W., He, Z., Chen, Q., Zhou, R., Zou, Z., Zhang, H., Hu, S., Zheng, Z., Zhou, J., Cai, J., Han, X., Zeng, G., Li, D., Liu, Z., and Sun, M. Efficient GPT-4V level multimodal large language model for deployment on edge devices. Nature Communications 16, 5509 (2025). [75] Yu, P., and Chowdhury, M. Salus: Fine-grained gpu sharing primitives for deep learning applications. In Proceedings of Machine Learning and Systems (2020). [76] Yurtsever, E., Lambert, J., Carballo, A., and Takeda, K. A survey of autonomous driving: Common practices and emerging technologies. IEEE access 8 (2020), 58443–58469. [77] Z.ai. GLM-4.6. Technical blog, 2025. [78] Zhang, D., Qi, S., Wu, Y., Xiao, X., Wang, X., and Chen, L. Fast-slow efficient training for multimodal large language models via visual token pruning, 2026. [79] Zhang, S., Xu, A., Chen, Q., Zhao, H., Cui, W., Wang, Z., Li, Y., Xiao, L., and Guo, M. Efficient Performance-Aware GPU sharing with compatibility and isolation through kernel space interception. In 2025 USENIX Annual Technical Conference (USENIX ATC 25) (2025), USENIX Association, pp. 1003–1019. [80] Zhao, H., Zhu, F., Guo, H., Wang, M., Wang, R., Meng, G., and Zhang, Z. Mllm-cl: Continual learning for multimodal large language models, 2025. [81] Zhen, H., Qiu, X., Chen, P., Yang, J., Yan, X., Du, Y., Hong, Y., and Gan, C. 3D-VLA: A 3D vision-language-action generative world model. In Proceedings of the 41st International Conference on Machine Learning (2024), vol. 235 of Proceedings of Machine Learning Research, PMLR, pp. 61229–61245. [82] Zheng, L., Li, Z., Zhang, H., Zhuang, Y., Chen, Z., Huang, Y., Wang, Y., Xu, Y., Zhuo, D., Xing, E. P., Gonzalez, J. E., and Stoica, I. Alpa: Automating inter- and Intra-Operator parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (Carlsbad, CA, July 2022), USENIX Association, pp. 559–578. [83] Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., Vuong, Q., Vanhoucke, V., Tran, H., Soricut, R., Singh, A., Singh, J., Sermanet, P., Sanketi, P. R., Salazar, G., Ryoo, M. S., Reymann, K., Rao, K., Pertsch, K., Mordatch, I., Michalewski, H., Lu, Y., Levine, S., Lee, L., Lee, T.-W. E., Leal, I., Kuang, Y., Kalashnikov, D., Julian, R., Joshi, N. J., Irpan, A., Ichter, B., Hsu, J., Herzog, A., Hausman, K., Gopalakrishnan, K., Fu, C., Florence, P., Finn, C., Dubey, K. A., Driess, D., Ding, T., Choromanski, K. M., Chen, X., Chebotar, Y., Carbajal, J., Brown, N., Brohan, A., Arenas, M. G., and Han, K. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Proceedings of The 7th Conference on Robot Learning (06–09 Nov 2023), J. Tan, M. Toussaint, and K. Darvish, Eds., vol. 229 of Proceedings of Machine Learning Research, PMLR, pp. 2165–2183.

Record · ID 200441 · SHA-256 86af6077f983a588
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.