HeteroVDiff: Pipelining Video Diffusion across Mobile NPU and GPU with Shared Weights

HeteroVDiff: Pipelining Video Diffusion across Mobile NPU and GPU with Shared Weights Zheyu Shen, Ziyao Wang, Yexiao He, Guoheng Sun, Yuning Zhang, Yiting Wang, Wanghao Ye, Ang Li

arXiv:2610.16244v1 [cs.DC] 12 Oct 2026

University of Maryland, College Park College Park, MD, USA {zyshen,ziyaow,yexiaohe,ghsun,yuning,ywang144,wy891,angliece}@umd.edu

Abstract Video generation is moving from the cloud to phones for privacy and offline use, and the applications often ask for several clips at a time. So video generation on phones is a throughput problem before it is a latency problem. Contemporary mobile SoCs carry both a GPU and a neural processing unit (NPU) on shared DRAM. Yet on-device video diffusion systems run the denoiser on one of them. The other engine either sits idle or only decodes the finished latent afterwards. The heterogeneous designs developed for mobile LLMs do not carry over, because they synchronise the two engines at every operator and partition weight tensors between the engines rather than sharing one stored copy. In this paper we first measure the platform limits that decide what can run on this class of SoC. They include how large one buffer may be, whether both runtimes can map the same buffer, and how much activation memory the NPU toolchain lets one graph hold. Drawing on these measurements, we store the base weights once in INT8 in the layout the NPU reads natively. The GPU uses the same bytes directly as integer dot-product operands, so neither runtime rearranges the weights on load. We then flatten the denoising trajectory into its full sequence of block executions and cut it once at a block boundary, so the GPU runs the prefix and the NPU the suffix. We place the cut from measured rather than modelled rates. Those rates include the decoder that only the NPU can run. They are taken while both engines run together, because each engine’s speed changes when the other is active. By employing these techniques we present HeteroVDiff, which serves a Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym ’XX, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX

multi-expert video diffusion model across the NPU and the GPU using 48% less weight memory than giving each engine its own copy. Generating videos back to back on a Galaxy S25 with decoding inside the pipeline, HeteroVDiff sustains 1.35× the stream throughput of NPU-only execution. On VBench-2.0, HeteroVDiff matches FP32 in the total score over all 18 dimensions.

CCS Concepts • Computer systems organization → Heterogeneous (hybrid) systems; • Human-centered computing → Ubiquitous and mobile computing design and evaluation methods; • Computing methodologies → Neural networks.

Keywords On-device Inference, Video Diffusion, Heterogeneous SoC, NPU, Mobile GPU, Model Serving ACM Reference Format: Zheyu Shen, Ziyao Wang, Yexiao He, Guoheng Sun, Yuning Zhang, Yiting Wang, Wanghao Ye, Ang Li. 2018. HeteroVDiff: Pipelining Video Diffusion across Mobile NPU and GPU with Shared Weights. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX). ACM, New York, NY, USA, 19 pages. https://doi.org/XXXXXXX.XXXXXXX

1

Introduction

Generative video diffusion has progressed from closed research demonstrations to open model releases and product previews [5, 22, 51, 55, 62]. With these models, an application no longer selects a precomputed asset but synthesises a short clip for a specific user when it is needed. Representative applications include personalised video advertising, audio-driven avatars, video virtual try-on, and interactive scene generation [17, 19, 50, 66]. Most of these applications need many clips at once. An advertising feed renders one clip per product, an avatar one per audio segment, a try-on session one per shortlisted garment, and scene generation one per user action. Such an application therefore waits for a whole set of clips, and it cares about how long the set takes

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Applications

Video Diffusion Model

Ads

Semantic Expert

Try-on

NPU

Detail Expert

Smartphone SoC GPU

Idle

Runs the model Shared DRAM Scenes

Figure 1: Applications that generate video from userspecific inputs run on a phone whose SoC carries both a programmable NPU and a GPU on shared DRAM. The workload is a multi-expert diffusion transformer that switches weight sets part-way along its denoising trajectory. Published on-device systems run that transformer on one engine. The other either sits idle or only decodes the finished latent afterwards.

rather than how long any single clip takes. On a phone one clip already takes minutes, so the time for a set comes down to the rate at which the device completes clips. That rate decides whether such an application is usable, and we call it stream throughput. The phone already has the hardware to raise this rate, but no published system uses all of it. A modern smartphone integrates a programmable neural processing unit (NPU) and a graphics processing unit (GPU) that share the same physical DRAM [6, 38, 41, 42, 53]. Yet on-device diffusion systems run the denoiser on one of them, as Figure 1 shows. Some works [4, 57, 71] shrink the model until it fits mobile constraints and run it on a single engine, and recent video systems [11, 18] place the whole diffusion transformer on the NPU. One system [69] uses the GPU only for the decoder, so the two engines run in sequence rather than together. In every case the GPU stays outside the denoising computation. The idle GPU matters because such sets of clips have strong reasons to be generated on the device rather than remotely. When conditioning data contain personal information, remote processing raises privacy and compliance considerations [9, 30, 49]. Remote execution is exposed to network latency, and under load to queueing before denoising begins [1, 52]. At scale, centralised generative inference also incurs recurring cloud-GPU serving cost per clip [33]. Image diffusion and language models have already moved towards on-device execution [6, 31, 71]. The video systems above follow the same path, but repeated transformer execution creates a different deployment problem for video. Two properties of modern video diffusion make it hard to run on a phone. First, one video executes 𝐿 transformer

Shen et al.

blocks over 𝑁 denoising steps, yielding 𝑁 ·𝐿 block executions against a shared latent. Few-step distillation [44, 48] reduces 𝑁 from tens to a few [23, 65], but it does not remove this repeated structure. Second, recent multi-expert diffusion models specialise denoisers for different noise regimes [3, 10, 51, 60]. For example, the Dual-expert Consistency Model (DCM) [32] freezes its semantic expert and adds timestepdependent embeddings and a low-rank adaptation (LoRA) of the attention projections for detail refinement. TimeStep Master also attaches timestep-specific LoRA experts to a shared base model [73]. Together, these properties create long accelerator occupancy and a large persistent weight footprint. In a stream these costs are paid again for every clip. The time a clip occupies the accelerator sets how fast clips complete, and the weights must stay in memory for the whole stream. A cross-engine system could use the two engines and the shared DRAM to address both costs. With a stream of clips, the GPU can run part of the denoising computation while the NPU runs the rest, so both engines are busy at the same time. Because they share physical DRAM, they can also read one stored copy of the weights instead of holding one each. This does not make a single video faster, because its trajectory is serial and the GPU is slower per block, but it shortens the period between finished videos in the stream. That period is the stream throughput the applications above wait on, so we target it rather than single-video latency. Three deployment obstacles stand between this opportunity and a practical cross-engine system. First, cross-engine execution requires a compatible stored weight representation without maintaining engine-specific converted copies. Each accelerator runtime normally expects weights in its own layout, so adding an engine requires a separately converted representation. Every converted copy adds to the memory the weights occupy, and multi-expert models add expert parameters on top. Second, a partition must balance two asymmetric engines. But the balance is not a simple ratio of block rates, because the decoder runs only on the NPU and adds a fixed per-video cost to its side. Fine-grained GPU–NPU execution has been used for mobile LLMs, where HeteroInfer partitions each weight tensor so that both engines compute the same layer concurrently [6]. Applied to video diffusion, a boundary at every block execution would require 𝑁 ·𝐿 synchronisation points per video, each of which has to make the whole activation available to both runtimes before either engine proceeds. Cutting only at step boundaries avoids the coordination but leaves too few places to balance two engines of very different speed. Third, the platform has limits that no data sheet states. How large one buffer may be, whether both runtimes can map the same buffer, and how much activation memory the NPU toolchain lets one graph hold are all capped, and the caps are found

Pipelining Video Diffusion with Shared Weights

only by trying. When both engines run at once they also compete for DRAM bandwidth and power, so a block takes a different time than running alone. No existing system clears all three obstacles, so a multi-expert video model still has no exact cross-engine serving path on a phone. We present HeteroVDiff, a system that serves a multiexpert video diffusion model across the NPU and the GPU of one phone, evaluated on a Snapdragon 8 Elite with a twoexpert model. It answers the three obstacles with three novel designs. First, the base weights are stored once in INT8 using the layout the NPU reads natively. The GPU uses the same bytes directly as operands of its integer dot-product instruction, so neither runtime rearranges the weights on load. Both runtimes map the same physical pages, and the NPU dequantises the block it is executing to FP16 in a scratch buffer that the next block overwrites, so each engine keeps its own fastest arithmetic while the memory stores only one copy. Second, the denoising trajectory is flattened into its 𝑁 ·𝐿 block executions and cut once at a block boundary, which may lie inside a step. The GPU runs the block executions before the cut, and the NPU runs the rest and then the decoder, so in a stream the GPU starts video 𝑖+1 while the NPU finishes video 𝑖. Every block still runs in order on the full activation, so the partition is algorithmically exact. We place the cut from rates measured while both engines run together, with the decoder’s per-video cost. Third, the platform limits are measured first and designed around. We measure how large a buffer can be, which buffers both runtimes can map, how much activation memory one graph may hold, and how much each engine speeds up or slows down when the other runs. The size limits set how the weights are divided into buffers, the co-run rates set the cut, and the same rates predict the throughput. We evaluate HeteroVDiff on a Galaxy S25 running DCMWan2.1, measuring performance and scoring output quality. On a stream of 49-frame videos, HeteroVDiff completes one every 677 s and NPU-only every 917 s, a stream-throughput gain of 1.35×. The gain rises to 1.50× at 17 frames. The gain holds under a service-level objective (SLO). HeteroVDiff sustains 5.3 videos per hour against 3.9 for NPU-only while every video meets the deadline. The two-engine pipeline holds 1.47 GiB of weights, half the 2.83 GiB that data parallelism needs for its two working sets. Quality is unchanged within 0.2 pp of FP32 in VBench Quality score and within 1.3 pp in VBench-2.0 score. This paper makes three contributions:

• One weight copy for two engines. We store the base weights once in INT8 with the NPU’s native layout, and the GPU consumes the same bytes directly. Both runtimes map the same physical pages.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Timestep embedding and expert schedule step 0

step 1

step 2

semantic expert (base)

step 3

step 4

step 5

detail expert (base + LoRA)

Denoising loop, N = 6 steps

DiT block (×L = 30) layer norm self-attention

zt

zt−1 layer norm patchify

cross-attention

unpatchify

Euler update

layer norm MLP next step, t decreases

after step 5, z0 Condition y

VAE decoder

precomputed umT5-xxl text embedding

3-D conv

semantic expert (base)

DiT

detail expert (base + LoRA)

Decoder

49 frames 832×480

Figure 2: Inference in DCM-Wan2.1, the two-expert video diffusion model we deploy. Top: steps 0–2 use the semantic expert, which is the shared base, and steps 3–5 (the switch step 𝜅=3) the detail expert as the same base with a rank-128 attention LoRA and its own timestep embedding. Middle: one step is one forward pass. The latent 𝑧𝑡 is patchified, passes 𝐿=30 DiT blocks whose cross-attention reads the precomputed umT5 text embedding 𝑦, and unpatchify and one Euler update give 𝑧𝑡 −1 . Bottom: after step 5 the variational-autoencoder (VAE) decoder, a stack of 3-D convolutions, turns 𝑧 0 into 49 frames at 832×480. • One cut with no approximation. We flatten the trajectory into 𝑁 ·𝐿 block executions and cut it once at a block boundary. Every block runs in order on the full activation. • Measured platform limits as design inputs. We measure how large a buffer both runtimes can register, how much activation memory the NPU toolchain lets one graph hold, and how co-execution changes each engine’s rate. The design built from these numbers sustains 1.35–1.50× the stream throughput of NPUonly execution on a Galaxy S25.

2 Background 2.1 Video Diffusion and Multi-Expert Models A video diffusion model denoises a Gaussian latent 𝑧𝑡 over 𝑁 timesteps [13, 47]. Each step patchifies 𝑧𝑡 ∈ R𝐶𝑧 ×𝐹𝑧 ×𝐻𝑧 ×𝑊𝑧 with patch size 𝑝 into 𝑧¯𝑡 ∈ R𝑀 ×𝑑𝑧 with 𝑀 = 𝐹𝑧 (𝐻𝑧 /𝑝) (𝑊𝑧 /𝑝)

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

tokens. A 480p 49-frame clip gives 𝑀≈2×104 . The tokens pass 𝐿=30 Diffusion Transformer (DiT) blocks [37], and block ℓ outputs 𝑋𝑡,ℓ ∈ R𝑀 ×𝑑 . An Euler update then yields 𝑧𝑡 −1 . After the last step a 3-D convolutional VAE decoder produces the frames. Multi-expert models specialise denoiser parameters by timestep. Wan2.2-A14B [51] switches by timestep between two 14 B experts. They need 28 GB in INT8 and do not fit. TimeStep Master [73] attaches LoRA experts to one backbone. DCM [32] shares parameters by construction (Figure 2). The semantic expert is a 1,419 M shared base. The detail expert adds a rank-128 LoRA on the attention projections (94.4 M) and its own timestep-embedding block (25.6 M), and it takes over from step index 𝜅=3 of six fixed timesteps. We therefore serve DCM-Wan2.1. Two properties matter. A step is one forward pass with no second branch to overlap. Six steps give too few partition points to balance the GPU and NPU, so §4 cuts at block granularity.

Shen et al.

Table 1: Running the whole workload on one engine, alone and while the other engine is busy. Co-execution moves the two engines in opposite directions. The NPU solo figure is the mean of ten videos generated back to back with decoding inside the pipeline.

NPU, solo NPU, co-run GPU, solo GPU, co-run

per DiT block

VAE

per video

3,352 ms 3,101 ms 12,633 ms 12,800 ms

308 s 275 s — —

916.7 s 825 s — —

3.1 Whole videos are the wrong scheduling unit 2.2

Snapdragon 8 Elite Silicon

The Galaxy S25 uses the Snapdragon 8 Elite [42], whose two accelerators share DRAM. Its Hexagon V79 NPU couples the HVX vector unit with the HMX matrix unit. HMX multiplies 32×32 INT8 or FP16 tiles held in VTCM in a fixed tile order [42, 43]. The NPU is reached over FastRPC [39] and runs offline-compiled QNN context binaries [40] with fixed shapes and an activation-footprint limit. The Adreno 830 GPU [38] runs OpenCL with INT8 dp4a and no matrix unit. Usable memory is 10.85 GiB. The runtimes share memory only through DMA-BUF descriptors [29] from AHardwareBuffer [2] or the Hexagon heap, which the CPU, GPU and DSP map at once. QNN accepts memory only in this form. Appendix A expands this section.

3

Motivation

This section reports what we measured when we deployed a dual-expert video diffusion model on a Galaxy S25. The workload is DCM-Wan2.1 [32] at its released configuration. It generates 49 frames at 832×480 in 𝑁 =6 denoising steps. The diffusion latent has 𝐶𝑧 =16 channels and 𝐹𝑧 =13 temporal positions. Its spatial dimensions are 𝐻𝑧 =60 and 𝑊𝑧 =104. The DiT uses 𝑝=2 and 𝑑=1536, giving 𝑑𝑧 =64 values per latent patch and 𝑀=20,280 tokens. Each step runs 𝐿=30 DiT blocks. The measurements expose three linked questions: why whole-video assignment fails, how both runtimes can access the same base-weight copy, and where the cut should fall.

Video generation on a phone today keeps only one engine busy [11, 18, 69]. As shown in Table 1, our NPU-only pipeline produces one video every 916.7 s with the decoder inside the pipeline, and the GPU is idle for all of it. Almost half of the SoC’s programmable accelerator does no work. The obvious way to use both engines is to hand each of them whole videos, and that fails for two independent reasons. First, at the solo rates in Table 1, the 𝑁 ·𝐿=180 DiT block executions take 2,350 s on the GPU and 603 s on the NPU, a 3.90× difference. Second, the GPU cannot finish a video at all, because the Qualcomm AI Runtime (QAIRT) GPU backend defines no Conv3d and the decoder never runs there. Using both engines therefore requires splitting each trajectory once between them. The next feasibility question is whether both runtimes can access the model without duplicating its weights. As shown in Table 2, the two engines do not agree on a precision either. The GPU runs a block fastest in INT8 and pays 3.37× for FP16, while the NPU runs the same block fastest in FP16 and pays 2.96× for INT8. The GPU is fast in INT8 because its integer dot-product instruction does four INT8 multiplies at once, whereas floating point does one; FP16 does not help because the GPU converts every FP16 value to FP32 before computing on it anyway. The NPU is fast in FP16 because its matrix unit computes on FP16 natively; INT8 is slower because the hardware cannot read signed INT8 results directly, so each GEMM needs an extra quantise pass before it and a dequantise pass after it, and those passes cost more than the GEMM itself. A shared representation therefore cannot simply be whichever format each engine prefers.

Pipelining Video Diffusion with Shared Weights

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 2: One DiT block on each engine at three precisions, same block and same input at 𝑀=20,280. Each multiplier is against that engine’s own fastest precision, shown in bold. The two engines prefer opposite formats. engine GPU NPU

3.2

INT8

FP16

FP32

12.6 s 42.3 s (3.37×) 37.8 s (3.01×) 9.95 s (2.96×) 3.36 s 3.95 s (1.17×)

Shared weights require per-block regions

Running two engines naively duplicates the weights. Each runtime stages tensors in its own layout, so one enginenative FP16 copy occupies 2.82 GB and two copies occupy 5.64 GB. That is about half of the 11.65 GB the kernel reports usable, and everything else on the phone has to fit in what is left. The obvious repair is to keep one base-weight copy that both runtimes read. The platform will not allow us to allocate it naively. Adreno advertises a maximum OpenCL buffer of 1024 MiB, but Android’s gralloc refuses a BLOB allocation at exactly 229 B, so OpenCL never gets the chance to refuse. A 511 MiB region registers into a QNN Hexagon context and an OpenCL context at the same time, while 1024 MiB takes the device down. The design that survives divides the shared base weights into one region per DiT block, with every region below the verified limit. This resolves the storage constraint; the remaining question is how to divide the trajectory between the two engines.

3.3

The cut must be block-level and measurement-driven

A single handoff divides each trajectory into a GPU prefix and an NPU suffix. The NPU suffix also runs the VAE decoder, so the cut must balance the complete pipeline rather than the DiT alone. With 𝑁 =6, a step-level cut moves work only in units of 𝐿=30 block executions. At the co-run rates in Table 1, the neighbouring step-level cuts occur after blocks 30 and 60. The first leaves the GPU idle for 356 s, while the second leaves the NPU idle for 121 s. A block-level cut can be placed between them and still requires only one handoff per video. Selecting that block boundary requires the co-run GPU block time 𝑇𝐺 , NPU block time 𝑇𝑁 , and NPU decoder time 𝑇𝑉 . These rates depend on the workload, concurrent execution, and thermal state. The cut must therefore be derived from measurements under the target deployment conditions. Together, these measurements motivate a single-cut pipeline. Its design must satisfy three requirements. First,

the shared base weights must be allocated at per-block granularity because a monolithic cross-runtime allocation is unavailable. Second, the trajectory must use one block-level cut because step-level cuts cannot balance the two pipeline stages. Third, the cut must be derived from co-run 𝑇𝐺 , 𝑇𝑁 , and 𝑇𝑉 measured under a stated cooling condition. Section 4 develops the resulting design, and Section 6 reports the measurements under these platform conditions.

4 System Design 4.1 Overview HeteroVDiff shares one physical copy of the base weights in shared memory between the GPU and NPU, then executes each denoising trajectory across both accelerators. Each video requires one cross-engine handoff. Figure 3 shows the system: a CPU coordinator, the two executors that consume one INT8 artifact, and the shared DRAM that holds it; Figure 4 shows how the trajectory is cut between the executors. HeteroVDiff treats the 𝐵 = 𝑁 ·𝐿 DiT block executions of one video as a single flattened sequence. The GPU executes the prefix [0, 𝑘𝑏 ) DiT blocks, while the NPU executes the suffix [𝑘𝑏 , 𝐵) and decodes the final latent with the VAE. Both engines read the same stored base-weight artifact. It contains INT8-quantised weights in the NPU’s native HMX tile layout and one FP32 scale per output column. The GPU consumes these weights directly as dp4a operands. The NPU uses FP16 operands for its GEMMs and caches the dequantised feed-forward weights for the current block (§4.2). The GPU writes the hidden state 𝑋𝑡,ℓ ∈ R𝑀 ×𝑑 and the patchified running latent 𝑧¯𝑡 ∈ R𝑀 ×𝑑𝑧 at the cut. The NPU then resumes inside the same denoising step (§4.3). The selected cut also falls before the first expert switch, so the GPU runs only the semantic expert and the NPU handles the switch locally (§4.3). The GPU GEMMs use INT8 operands, while the NPU GEMMs use FP16 operands (§4.2). The GPU keeps the state between blocks in FP16 and uses a separate scale for each GEMM input. These choices limit error accumulation across the quantised block sequence (§4.4). At each denoising step, the host CPU projects the current latent into DiT tokens, maps the final hidden state back to a latent-space prediction, and applies the Euler update to obtain the latent for the next step. The mechanisms assume only a stack of DiT blocks whose base weights quantise to INT8 and a denoising trajectory that repeats those blocks on one latent, so 𝑁 and 𝐿 are free in the flattened sequence. Neither assumption involves DCM or its LoRA experts. Three rules govern the design, and each is forced by a measurement rather than chosen for elegance. (1) Share the weight representation, not the compute format. Both engines read the same INT8 base-weight

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY runs on the CPU

CPU coordinator Map regions

runs on the GPU

runs on the NPU

Shen et al. INT8 base weight, one copy in shared DRAM

handoff state

Shared DRAM: one INT8 artifact, one region per block b0

b1

b2

b3

One region, NPU tile order One handoff

detail expert / LoRA

attention base

FFN

b4

b5

…

b6

Handoff

Outside the shared region scales

LoRA A, B

b29

zt̄

Xt, ℓ

timestep emb.

Two executors, the same INT8 bytes Dispatch blocks

GPU executor (OpenCL)

NPU executor (FastRPC)

Block prefix [0, kb)

Block suffix [kb, B) and decoder

Figure 3: HeteroVDiff in one view. The CPU coordinator (left) maps each per-block weight region into both runtimes at start-up, dispatches the block prefix [0, 𝑘𝑏 ) to the GPU and the suffix [𝑘𝑏 , 𝐵) to the NPU, where 𝐵=𝑁 ·𝐿 counts the block executions of one video, and performs the single hand-off, whose hidden state 𝑋𝑡,ℓ and latent 𝑧¯𝑡 live in the same shared DRAM (lavender). Both executors (bottom right) read the same INT8 bytes: the GPU uses them as integer dot-product operands and rescales in the epilogue, while the NPU dequantises the feed-forward weights once per block into an FP16 scratch and applies the adapter factors inside its block kernel. The base is stored once in shared DRAM (top right), one region per block in the NPU’s tile order; the detail expert’s factors and timestep embedding live outside it. artifact. The GPU uses the weights directly for INT8 dot products, whereas the NPU dequantises them for FP16 computation (§4.2). (2) Share weights one DiT block at a time. The platform cannot create one cross-runtime buffer large enough for the full base model. HeteroVDiff therefore allocates the shared base weights at per-DiT-block granularity (§4.2). (3) Cut once, at a block boundary. A step-level cut offers only 𝑁 +1 positions. Handoffs after every DiT block add repeated communication, so HeteroVDiff instead places one block-level cut in the flattened sequence (§4.3).

4.2

Cross-Engine Single-Copy Base-Weight Storage

Section 3 established two constraints on weight storage; Figure 3 shows the artifact that satisfies them, how each executor consumes it, and what stays outside it. Each shared region must fit within the platform limit, and the same stored representation must support INT8 GPU GEMMs and FP16 NPU GEMMs. HeteroVDiff therefore uses one canonical INT8 base-weight artifact and divides it into one region per DiT block.

For a weight matrix 𝑊 ∈ R𝑑in ×𝑑out , HeteroVDiff assigns one INT8 scale to each output column: 𝑠𝑗 =

max𝑖 |𝑊𝑖 𝑗 | , 127 

𝑄𝑖 𝑗 = clip [−127,127] round



𝑊𝑖 𝑗 𝑠𝑗

(1)

 .

It then stores 𝑄 in the 32×32 tiled layout required by the Hexagon matrix unit. Tiles are ordered first by outputcolumn tile and then by input-row tile. The byte address of 𝑄𝑖 𝑗 is     𝑖 𝑗 𝑑 in off (𝑖, 𝑗) = + 1024 32 32 32   𝑖 mod 32 + 128 + ( 𝑗 mod 32)4 + (𝑖 mod 4). 4 (2) Each DiT block is stored in a separate region that satisfies the allocation limit measured in Section 3.2. The same tiled representation is consumed differently by the two engines. The last term in Equation 2 places four consecutive input values for a fixed output column in four contiguous bytes, which is exactly the char4 operand Adreno’s integer dot-product instruction takes. The GPU reads these bytes directly and applies the activation and weight scales in the GEMM epilogue as 𝑦 𝑗 = acc 𝑗 (𝑠𝑥 𝑠 𝑗 ) +𝑏 𝑗 , where acc 𝑗 is the integer accumulator, 𝑠𝑥 the calibrated activation scale of §4.4, and 𝑏 𝑗 the output bias. For the feed-forward layer, the NPU dequantises both weight matrices to FP16 once at the start

Pipelining Video Diffusion with Shared Weights

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

of each block. All feed-forward chunks in that block reuse the block-local FP16 scratch buffer. The remaining weights follow their kernel-specific staging paths. The NPU’s HMX tile layout is the canonical storage format because the NPU requires this layout in hardware. The GPU can support it by changing its OpenCL index calculation. A GPU-native row-major layout would instead make the NPU reorder the same weights repeatedly while staging a block. At runtime, the GPU and NPU map each per-block region into their own address spaces while referring to the same physical pages in shared memory. The two runtimes therefore share one physical copy of the base weights. Their runtime handles and virtual addresses remain separate, but no second base-weight allocation is created. This removes cross-engine duplication of the base weights. Expert-specific parameters and the NPU’s temporary FP16 scratch remain separate. The factorised expert representation stores the shared attention base𝑊 , the rank-128 LoRA factors 𝐴 and 𝐵, and the expert-specific timestep embeddings separately. The detail expert evaluates

[0, 𝑘𝑏 ). The NPU executes the suffix [𝑘𝑏 , 𝐵) and decodes the final latent, so no additional handoff is needed.

𝑦 = 𝑥𝑊 + (𝑥𝐴⊤ ) (2𝐵) ⊤,

Figure 4: Partitioning the flattened trajectory. Top: one video is 𝐵=180 block executions followed by the VAE decode, cut once at 𝑘𝑏 =51 inside step 1 (red). The GPU runs the prefix, the NPU the suffix and the decoder; only the hidden state and the running latent cross, once per video, and because 𝑘𝑏 ≤𝜅𝐿=90 the expert switch stays on the NPU. Bottom: over a stream, the GPU prefix of the next video overlaps the NPU suffix and decode of the current one, so the period is the longer of the two stages (Equation 4); the decoder’s fixed cost on the NPU moves the balance point later than the block rates alone would place it.

(3)

while both experts reuse the same feed-forward weights. Keeping the factors separate avoids storing a second fullrank attention copy and avoids requantising a small adapter after it is merged into the base. Both device runners consume this representation at run time. The NPU runs the two factor GEMMs inside its block kernel and adds the result to the base projection in FP16. The GPU computes the same low-rank delta and fuses it into the GEMM epilogue before requantisation. Merging the adapter into an INT8 base is not merely unnecessary: the adapter’s magnitude is about one third of the base’s requantisation noise, so a merged INT8 copy cannot represent the two experts distinguishably at weight level, whereas the factorised path applies the delta at full precision. The factorised expert representation is the only part of HeteroVDiff that is specific to DCM. A singleexpert model omits it, and a model whose experts each carry a full base stores one shared artifact per expert under the same design. The NPU reuses its feed-forward scratch block by block. The temporary FP16 copy is therefore limited to the block currently executing rather than the full model. Section 6 reports the memory, latency, and fidelity effects of these choices.

4.3

Partitioning the Flattened Trajectory

To obtain finer placement than a step-level cut without introducing repeated handoffs, HeteroVDiff flattens the trajectory into 𝐵 = 𝑁 ·𝐿 block executions and places one blocklevel cut at index 𝑘𝑏 (Figure 4). The GPU executes the prefix

runs on the GPU

runs on the NPU

semantic expert

detail expert

One video: 180 block executions block 0

block 29

s0 s1 s2 s3 s4 s5 VAE

VAE decoder

Stream: two-stage pipeline GPU

prefix 1

time → prefix 2

NPU

suffix 1

prefix 3 VAE

suffix 2

VAE

period = max(kbTG, (B−kb) TN + TV)

For a stream of videos, the GPU prefix and NPU suffix form a two-stage pipeline. Let 𝑇𝐺 and 𝑇𝑁 denote the GPU and NPU time per DiT block, and let 𝑇𝑉 denote the NPU time for VAE decoding. The steady-state period is
period(𝑘𝑏 ) = max 𝑘𝑏 𝑇𝐺 , (𝐵−𝑘𝑏 ) 𝑇𝑁 + 𝑇𝑉 . (4) The two terms cross at 𝐵 𝑇𝑁 + 𝑇𝑉 . (5) 𝑇𝑁 + 𝑇𝐺 The integer optimum 𝑘𝑏∗ is the neighbouring integer that gives the smaller period. The decoder cost appears only in the NPU stage, so omitting 𝑇𝑉 would balance the DiT blocks rather than the complete video pipeline. The three latency values are not fixed properties of the hardware. HeteroVDiff measures 𝑇𝐺 , 𝑇𝑁 , and 𝑇𝑉 while both engines run under the target workload and thermal condition. It then evaluates the neighbouring integer cuts and selects 𝑘=

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

the one with the lower predicted period. The current runner fixes 𝑘𝑏 at startup and does not adjust it online. Only the state needed to resume the current denoising step crosses at the cut. The GPU hands off the hidden state 𝑋𝑡,ℓ and the patchified running latent 𝑧¯𝑡 used by the Euler update. Because 𝑘𝑏 need not be a multiple of 𝐿, the NPU may resume in the middle of a denoising step. Its block kernel accepts a starting block and a block count, so it skips the patch embedding and loads the transferred hidden state. The offline artifact stores the metadata needed to resume execution on the NPU, including the AdaLN parameters, expert choice, and Euler step size for each denoising step. The GPU therefore hands off only the two state tensors. The cut index 𝑘𝑏 is a runtime parameter selected at startup; the artifact stores only the per-step metadata needed to resume at any block boundary. The cut also determines whether expert routing crosses the engine boundary. The expert switch remains on the NPU when 𝑘𝑏 ≤ 𝜅𝐿, where 𝜅𝐿 is the first block that uses the detail expert. The GPU therefore runs only the semantic expert, while the NPU selects the detail expert locally without moving expert parameters. During artifact generation, HeteroVDiff records the expert used by each timestep so that routing remains correct when the timestep schedule changes.

4.4

Activation Numerics and Calibration

The GPU keeps the hidden state between DiT blocks in FP16, the last row of the GPU executor in Figure 3. Before each INT8 GEMM, it quantises the GEMM input using a scale calibrated for that block and input. This allows the GPU to use INT8 dot products without keeping intermediate states in INT8. The NPU keeps the hidden state between blocks in FP32 and narrows it to FP16 only where it enters a GEMM, because the state accumulates 30 gated additions per step and grows to about 25 in magnitude by the last block, and it never serves as a GEMM operand, so the wider format costs nothing. Before deployment, we calibrate each GEMM input using the actual timestep schedule and AdaLN modulation. For each block and GEMM input, we divide the largest observed absolute value by 127 to obtain a fixed INT8 scale. The same scale is then used throughout inference. We follow prior timestep-aware calibration methods [25, 45, 46, 70]. These choices reduce quantisation error but do not make INT8 execution identical to full-precision execution. Section 6 evaluates its effect on generated-video quality.

Shen et al.

5

Implementation

This section follows the execution path from runtime setup through the DiT kernels and VAE decoder, then describes artifact generation and validation. HeteroVDiff is about 25 k lines of new code. The NPU kernels are C with HVX and HMX intrinsics, added to the open-source htp-ops-lib operator package [12]. The GPU executor and the coordinator are C++ and OpenCL C. Artifact export, quantisation and validation are Python, and deviceside orchestration is shell.

5.1

Runtime and Work Placement

HeteroVDiff consists of a device-side coordinator, an OpenCL GPU DiT executor, and a FastRPC NPU DiT executor. The coordinator assigns the block executions before the cut to the GPU executor and the rest to the NPU executor, launches them for each video, and sequences the one handoff between them. At the handoff the GPU passes two tensors, the hidden state and the running latent, and the NPU resumes at block 𝑘𝑏 even when the cut falls inside a denoising step. Each executor runs the per-step patch embedding, output head and Euler update on the CPU in its own process, in place on the shared buffers the accelerator has just written, without copying them. Both DiT executors bypass the QAIRT graph runtime. The NPU VAE decoder uses the QNN Hexagon Tensor Processor (HTP) backend separately after the last block of each video, so in a stream the GPU prefix of video 𝑖+1 overlaps the NPU suffix and the decode of video 𝑖. The NPU DiT executor calls a custom Hexagon operator package directly. The QNN HTP backend prevents external operators from using HMX. It also cannot serialise the full attention graph at the deployment shape without slicing it. The GPU DiT executor uses hand-written OpenCL because the QNN GPU backend does not support the required fully connected operator. Both executors expose the same blockrange interface and read the artifact described in §4.2. At startup, each block’s base weights are placed in a shared DMA-BUF. The NPU maps the buffer through FastRPC. The GPU imports the same buffer through Qualcomm’s OpenCL host-pointer extension. When the two executors run as separate processes, the NPU-side owner passes the per-region file descriptors to the GPU process over a Unix-domain socket. The runtimes use different handles and virtual addresses for the same physical pages in shared memory. The rank-128 adapters, detail-expert timestep embeddings, and the NPU’s block-local FP16 scratch remain separate from this shared base. Appendix B describes the NPU and GPU DiT kernels and the VAE decoder.

Pipelining Video Diffusion with Shared Weights

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Throughput gain over NPU-only (×)

49 frames M = 20,280 1.5 1.0 0.5

1.00×

0.99×

25 frames M = 10,920 1.40×

1.39×

1.35×

1.32× 1.00×

33 frames M = 14,040

1.00×

1.50×

1.42× 1.00×

0.97×

0.98× 0.48×

0.44×

0.40×

0.35×

1.50×

1.41×

0.99×

17 frames M = 7,800

Videos / hour within the SLO

0.0 6 4

D = 1375 s 3.9

5.3 3.9

3.9

2 0

9 6

D = 924 s

8.2

5.8

5.8

5.8

3 0

ly ly DP nsor Diff -on PU-on U -Te oV P G N ero Heter t e H

0

12 8

D = 708 s

11.5

7.6

7.6

7.4

0

0

D = 494 s

16.3

12 10.9

10.9

10.7

6

4

ly ly DP nsor Diff -on PU-on U -Te oV P G N ero Heter t e H

18

0

ly ly DP nsor Diff -on PU-on U -Te oV P G N ero Heter t e H

0

0

ly ly DP nsor Diff -on PU-on U -Te oV P G N ero Heter t e H

Figure 5: Every configuration at four clip lengths. Top: stream throughput as the gain over NPU-only (the vertical line at 1× is the NPU-only baseline). Bottom: videos per hour with every video inside the SLO, a per-video deadline 𝐷 of 1.5× the NPU-only latency. DP drops to the NPU-only bar because its GPU-arm videos exceed the deadline 𝐷 and only its NPU arm counts. Same device, active cooling, shared-base plus factorised-expert weight path, ten-video streams.

6 Evaluation 6.1 Experimental Setup Platform and workloads. All device measurements use one Samsung Galaxy S25 (Snapdragon 8 Elite, SM8750) with the Hexagon V79 NPU and Adreno 830 GPU, and the usable memory is 10.85 GiB as measured. The model is DCM-Wan2.11.3B with 𝑁 =6 denoising steps, 𝐿=30 blocks per step, and 832×480 output. The semantic expert on steps 0–2 and the detail expert on steps 3–5 (𝜅=3). We vary the clip length over 49, 33, 25 and 17 frames, which sets the token count to 𝑀=20,280, 14,040, 10,920 and 7,800 while the number of block executions per video stays 𝐵=𝑁 ·𝐿=180. Prompt embeddings are computed once offline, so the text encoder is not resident during generation. Thermal and power protocol. Every timed run is cold. The phone sits on an external cooler with the screen off and stays plugged in. Both controls hold the device in one operating state, so that the differences we report come from the schedule rather than from heat or a draining battery. Metrics and measurement. The primary metric is stream throughput, reported as seconds per video over a stream of 10 content-diverse prompts generated back to back with decoding inside the pipeline. For the two-engine pipeline it is the

steady-state period after the initial fill. We also report singlevideo latency from launch to finished decode, SLO attainment and resident memory. The SLO is a per-video deadline of 1.5× the NPU-only single-video latency at the same workload. Output quality is scored on the seven VBench quality dimensions, and on 18 VBench-2.0 dimensions. Configurations. All self-built configurations consume the same INT8 base-weight artifact and apply the factorised expert representation of §4.2 at run time, so they differ only in how execution is divided between the two engines. NPU-only runs the whole trajectory and the decoder on the NPU. It is the strongest single engine and serves as the (1×) reference throughout. GPU-only runs the DiT on the GPU and decodes on the NPU, because the vendor GPU backend has no 3D convolution and therefore cannot run the VAE decoder. DP is naive data parallelism in which both engines generate whole videos from one prompt queue with separate working sets. The NPU worker decodes the GPU worker’s latents as they arrive. Hetero-Tensor is the per-GEMM column split of HeteroInfer [6] in which both engines compute a slice of every GEMM and exchange results after each one. HeteroInfer has no public code, so we implemented the split ourselves following the paper. HeteroVDiff places one block-level cut at the 𝑘𝑏 that Equation 4 selects from co-run rates for each workload, namely 51, 54, 59 and 61.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Shen et al.

FP16, one copy per engine Merged-expert INT8, one per engine

2 2

5,379 3,746

– 52.5

5.25 3.71

NPU-only GPU-only DP Hetero-Tensor (column slices) HeteroVDiff (shared pages)

1 1 2 1 1

1,423 1,512 2,846 1,423 1,423

52.5 – 52.5 52.5 78.8

1.44 1.48 2.83 1.44 1.47

6.2

End-to-End Throughput and SLO

Figure 5 plots the gain over NPU-only for every configuration at all four workloads. HeteroVDiff raises stream throughput over NPU-only at every clip length. At 49 frames it completes a video every 677 s against 917 s, which is 5.3 against 3.9 videos per hour, a gain of 1.35×. The gain rises to 1.50× at 17 frames. The deployed cuts are the ones the rate model selects for each workload, 𝑘𝑏 =51, 54, 59 and 61, all before the first expert switch at block 𝜅𝐿=90. Naive data parallelism comes closest on throughput and loses on the SLO axis. It reaches 1.32–1.42×, 0.6–6.4% behind HeteroVDiff at the four workloads. Under the SLO every video DP adds over NPU-only misses the deadline. Each DP video is generated entirely by one engine, so the added videos all come from the GPU arm. A GPU-arm video takes 2,620 s at 49 frames against the deadline of 1,375 s. The added videos are therefore unusable, and DP’s SLO-compliant rate is its NPU arm alone. That is the NPU-only rate, 3.9 videos per hour against HeteroVDiff’s 5.3. HeteroVDiff meets the deadline because every video crosses both engines and none takes longer than 1,325 s. Hetero-Tensor finishes a video in 860.5 s at 49 frames, 0.94× NPU-only, because the perGEMM split saves about 50 ms of GEMM time per block and pays 155 ms of waiting and cache maintenance, so the 4,680 hand-offs per video cost more time than the GPU’s slices save. GPU-only completes at 0.35–0.48×. The gain grows as clips shorten, and the cut has to be placed from co-run rates (§6.5). As 𝑀 falls, the GPU-to-NPU block-time ratio drops from 3.9 to 2.7, so the balanced cut moves later and the GPU carries more of each video, and the NPU’s co-run speed-up grows from 10% to 16–19%.

Numeric path

FP32 95.10 96.34 99.49 97.35 73.61 58.30 65.92 83.19 NPU path 95.02 96.29 99.48 97.31 75.28 58.09 66.00 83.25 GPU path 94.52 96.07 99.34 97.22 74.17 56.62 66.28 82.74 HeteroVDiff, 𝑘𝑏 =51 94.67 96.05 99.34 97.27 75.83 57.38 66.20 83.02

1.47 GiB of weights. Data parallelism holds two artifacts, 2.83 GiB. One copy per engine needs 3.71 GiB in merged INT8 and 5.25 GiB in FP16. Table 3 counts tensor sizes because resident-set counters move by tens of megabytes with page cache and compressed swap. Both engines map the same artifact pages. The GPU reads the attention base in place and stages only the current block’s feed-forward pair. The deployed cut keeps every detail-expert block on the NPU. Adding the GPU to NPU-only therefore costs only 26.3 MiB. The kernel charges the shared attention base half to each process. A GPU that loaded the same weights from the file would hold a private copy of about 540 MB. The factorised expert representation stores each DiT block once in 47.3 MiB, so the dual-expert artifact is 1.39 GiB. Of that 1,423 MiB the attention base is 541.4 MiB, the feedforward pair 787.5 MiB, the scales 2.6 MiB and the rank-128 factors 91.5 MiB. The NPU’s FP16 copy of one block’s feedforward pair adds 52.5 MiB. Merging the detail expert into a second attention copy would take 1.83 GiB and one FP16 copy per engine 2.82 GB (§3.2). The expert switch is also free. Applying the factors at run time moves no base weights, whereas merged weights would reload the whole attention base in every video. Working buffers add about 0.4 GiB. Two hidden-state carriers each hold 59.4 MiB in FP16 for a 49-frame clip. The GPU’s de-fused attention keeps one 256-row block of scores and probabilities, 178 MiB at 𝑀=20,280, growing with 𝑀 rather than 𝑀 2 . The hand-off writes the hidden state and latent once per video, 130 MB. HeteroVDiff needs about 1.9 GiB at steady state, leaving the rest of the 10.85 GiB the kernel reports usable to the decoder and the application.

6.4 6.3

Memory Analysis

Sharing one stored copy of the base weights between the two engines is what saves the memory. HeteroVDiff holds

ct c on Ba sis ckg t en rou cy nd con Tem si s ten po cy ral flic ker Mo ing tio ns mo o thn Dy ess nam ic d egr Ae ee sth et i cq ual Im ity agi ng qu alit Qu y ali ty sco re

Copies Base + adapter Transient Total (MiB) (MiB) (GiB)

bje

Configuration

Table 4: VBench [14] quality dimensions and Quality score (%) of the four numeric paths defined in the text, full prompt suite with five samples per prompt.

Su

Table 3: Weight memory of each configuration at steady state. Base is the shared attention base, the feedforward pair and the scales, and adapter is the rank-128 factors. Transient is the NPU’s FP16 copy of the current block’s feed-forward pair and the GPU’s staged block. Total is their sum in GiB.

Output Quality

We score four numeric paths of the same model, prompts and six-step schedule, which differ only in the arithmetic of the DiT blocks. FP32 is the unquantised reference. The NPU

Pipelining Video Diffusion with Shared Weights

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Figure 6: Qualitative samples. Six uniformly spaced frames from each of four 832×480, 49-frame videos generated under the deployed HeteroVDiff numeric path (𝑘𝑏 =51) for VBench prompts, printed on each row. path consumes the shared INT8 base weights dequantised to FP16 and computes in FP16, the GPU path consumes the same INT8 weights with INT8 activations, and HeteroVDiff applies the GPU path to block executions [0, 51) and the NPU path to the remaining 129, matching the deployed cut. Figure 6 shows frames from the deployed path, and Table 4 scores all four paths on the full VBench prompt suite, while Table 6 in Appendix C does the same on VBench-2.0. The shared INT8 artifact costs no measurable quality. On VBench the NPU path is within 0.21 pp of FP32 on six of the seven dimensions and +0.06 pp in Quality score, the GPU path stays within 1.7 pp on every dimension and loses 0.45 pp, and HeteroVDiff sits between them at −0.17 pp. VBench-2.0, which judges faithfulness with vision– language-model (VLM) and detector-based scorers rather than perceptual quality, does not separate the four paths. The Total scores lie within 1.3 pp of each other and none of the quantised arms falls below FP32. Per-dimension differences run in both directions on every arm, and the largest, +11.5 pp on multi-view consistency, is on a dimension the benchmark scores from only 8 to 10 videos per arm, so one video moves it by 10 pp or more. We therefore read VBench-2.0 as evidence of no separation at this prompt count, not of an improvement from quantisation.

6.5

Ablations

6.5.1 Where to cut. As shown in Table 5, a cut at a block boundary beats a cut at a step boundary because no step boundary reaches the balance point. With 𝑁 =6 the only step boundaries are multiples of 𝐿=30. The two that bracket the deployed cut each leave one engine idle. At 𝑘𝑏 =30 the GPU

Table 5: Where to cut the trajectory, at 49 frames. Each row is the same pipeline with a different cut index 𝑘𝑏 , over six videos generated back to back with decoding inside the pipeline. Stage times are per period. Idle is the gap between the two stages. Cut selected by

𝑘𝑏

GPU (s)

NPU (s)

Idle (s)

Period (s) ↓

Step boundary: the only positions a step-level partition offers step before 𝑘 ∗ 30 431 765 334 768 (+12.5%) step after 𝑘 ∗ 60 782 650 132 782 (+14.5%) Block boundary: the rate model chooses the index block rates alone 34 443 723 280 726 (+6.3%) solo rates 57 747 655 92 752 (+10.1%) co-run rates, with 𝑇𝑉 51 676 675 1 683

finishes its prefix 334 s before the NPU finishes the suffix and the decode. At 𝑘𝑏 =60 the NPU waits 132 s for the GPU. The periods are 12.5% and 14.5% above the deployed cut. A block boundary narrows that gap to 1 s with the same single hand-off. Selecting 𝑘𝑏 among block boundaries requires co-run rates and the decoder term 𝑇𝑉 of Equation 4. Balancing block rates alone without 𝑇𝑉 gives 𝑘𝑏 =34 and a period 6.3% above the deployed cut because the NPU still owes a decode that no longer fits in the GPU’s shadow. Solo rates give 𝑘𝑏 =57 and a period 10.1% above it because they over-estimate the GPU and hand it 6 blocks too many. Co-run rates with 𝑇𝑉 give the deployed 𝑘𝑏 =51 and balance the two stages to within 1 s. Every period is reported at the deployed run’s decoder time 𝑇𝑉 =267 s because that time drifted from 186 to 270 s between measurement days.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Shen et al.

deployed SLO

Videos / hour within the SLO

HeteroVDiff

15 DP

NPU-only

10

Hetero-Tensor GPU-only

5

0 1.25×

1.5×

1.75×

2×

3×

SLO deadline D (× NPU-only single-video latency)

Figure 7: Videos per hour within the SLO against the deadline factor 𝑓 at 17 frames, where the deadline is 𝐷 = 𝑓 × the NPU-only single-video latency. HeteroVDiff clears every deadline in the sweep since its worst video takes 1.24× that latency. DP and GPU-only both need 2.09×, and below that DP delivers exactly the NPUonly rate. The vertical rule marks the deployed 1.5× SLO.

SLO sensitivity. Figure 7 sweeps the deadline factor 𝑓 from 1.25 to 3 at 17 frames, sets 𝐷 = 𝑓 × 329.3 s and plots videos per hour within the SLO for the five configurations. Two thresholds decide the figure. The worst HeteroVDiff video takes 1.24× the NPU-only latency because it crosses both engines, while a DP video on the GPU arm takes 2.09×. HeteroVDiff therefore clears every deadline in the sweep and holds 16.4 videos per hour, the highest line throughout. Up to 𝑓 =2 every video DP generates on its GPU arm takes longer than 𝐷 and misses the deadline. Only the videos from its NPU arm count, and that arm alone completes 10.9 videos per hour, the same as NPU-only. At 𝑓 =3 the deadline is long enough for the GPU-arm videos to count, so DP rises to 15.5. Even with this loose deadline DP stays below HeteroVDiff’s 16.4. GPU-only starts meeting the deadline at the same point and reaches 5.2. Hetero-Tensor stays at 10.7 at every deadline. HeteroVDiff therefore keeps its gain at every deadline an application would realistically set. Its only losing case is a deadline below 1.24×, where every cross-engine video misses. The cut is chosen when the runner starts, so for such a deadline the runner simply runs NPU-only or a small 𝑘𝑏 .

7

Related Work

Heterogeneous inference on mobile SoCs. Concurrent execution across a phone’s processors is established for language and perception models [6, 8, 15, 36, 53, 59, 61], and differs from HeteroVDiff in two ways. They synchronise at every

layer or tensor slice, about 400 µs per crossing by HeteroInfer’s measurement [6]. Video diffusion makes that far more expensive because every block runs once per denoising step. Our six-step, 30-block workload executes 𝑁 ·𝐿=180 blocks per video, so a per-layer scheme would cross at least 180 times where HeteroVDiff crosses once. They also keep weights in each runtime’s own layout. HeteroInfer gives the engines disjoint slices of one matrix. And BlastNet, CoDL, and 𝜇Layer duplicate or reformat blocks per processor [16, 21, 28]. HeRo aliases one buffer across the QNN and OpenCL runtimes through CL_MEM_USE_HOST_PTR [24] only for inter-stage activations, while weights stay one copy per backend. None lets two runtimes consume the same persistent weight bytes, the problem §4.2 solves.

Cross-engine interference. Co-execution on mobile SoCs has been characterised as a cost, or at best as aggregate bandwidth recovered from idle memory channels [6]. HaX-CoNN models GPU–DSP contention as slowdown [8], Agent.xpu reports NPU–GPU interference on an AI PC as degradation or neutrality [54], Sereno finds on this SoC that NPU inference degrades foreground rendering [58], and the one documented cross-component dynamic voltage and frequency scaling (DVFS) coupling in mobile inference is antagonistic [68]. Their co-runners are separate models or interactive rendering [34, 63]; none places a cut from rates that co-execution itself changes, so HeteroVDiff measures its rates with both engines running, where they move in opposite directions.

On-device diffusion. Prior on-device diffusion makes the model smaller and runs it on one engine. Image systems pair efficient architectures with GPU-aware kernels on a single engine [26, 71]; Speed Is All You Need already keeps a materialised score matrix with a partially fused softmax on Adreno, reserving fused FlashAttention for head dimension 40 [7], and we re-adopt that structure at head dimension 128, in INT8, and at video sequence length rather than claim it. Video systems retrain compact models [56, 57], prune tokens or distil steps [4, 18, 27], or page blocks in on demand [20], all orthogonal to HeteroVDiff, which keeps the model and changes where it runs. The closest vendor systems run entirely on the Hexagon NPU and leave the GPU idle [11, 18]. MobileWan also splits its block sequence, but within the NPU to fit memory, unloading and reloading at every boundary; S2 DiT gives the GPU only the decoder, so its engines run in sequence [69]; CODA disaggregates video diffusion across two units, but needs near-memory hardware phones lack and overlaps only classifier-free guidance branches our model lacks [67].

Pipelining Video Diffusion with Shared Weights

8

Conclusion

HeteroVDiff serves a multi-expert video diffusion model across the NPU and the GPU of one phone from a single stored copy of its weights. The base weights live in one INT8 artifact laid out as the NPU reads it and consumed by the GPU as integer dot-product operands. The second expert is a rank128 factor on top of that base. And the denoising trajectory is cut once at a block boundary so that the GPU runs the prefix of the next video while the NPU finishes the current one. On a Galaxy S25 this completes a 49-frame video every 677 s against 917 s for NPU-only execution, a stream-throughput gain of 1.35× that rises to 1.50× at 17 frames, while staying within 0.2 pp of FP32 in VBench Quality score and 1.3 pp in VBench-2.0 Total score. The general lesson is that two accelerators should share weight bytes rather than arithmetic on a shared-memory SoC, and the cut between them must be placed from co-run rates that include the decoder. The results come from one SoC generation with a cut selected offline and applied statically. The cut is a start-up parameter, so a runner could recompute it between videos from the rates it has just observed. The mechanisms themselves depend only on two accelerators sharing one DRAM, and should carry over to other SoCs of that kind.

References [1] Sohaib Ahmad, Qizheng Yang, Haoliang Wang, Ramesh K Sitaraman, and Hui Guan. 2025. Diffserve: Efficiently serving text-to-image diffusion models with query-aware model scaling. Proceedings of Machine Learning and Systems 7 (2025). [2] Android Developers. 2024. Native Hardware Buffer (AHardwareBuffer). https://developer.android.com/ndk/reference/group/ahardware-buffer. Android NDK API Reference. Accessed: 2026-09-02. [3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. 2022. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022). [4] Haitam Ben Yahia, Denis Korzhenkov, Ioannis Lelekas, Amir Ghodrati, and Amirhossein Habibian. 2025. Mobile Video Diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 19450–19460. doi:10.1109/ICCV51701.2025.01808 arXiv:2412.07583. [5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. 2024. Video generation models as world simulators. OpenAI Blog (2024). [6] Le Chen, Dahu Feng, Erhu Feng, Yingrui Wang, Rong Zhao, Yubin Xia, Pinjie Xu, and Haibo Chen. 2025. Characterizing mobile soc for accelerating heterogeneous llm inference. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 359–374. [7] Yu-Hui Chen, Raman Sarokin, Juhyun Lee, Jiuqiang Tang, Chuo-Ling Chang, Andrei Kulik, and Matthias Grundmann. 2023. Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPUAware Optimizations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 4651-4655.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY [8] Ismet Dagli and Mehmet E. Belviranli. 2024. Shared Memorycontention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chips. In PPoPP ’24: Proceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, 2024. doi:10.1145/3627535.3638502 [9] European Parliament and Council of the European Union. 2016. Regulation (EU) 2016/679 (General Data Protection Regulation). Official Journal of the European Union, L 119, pp. 1–88. https://eurlex.europa.eu/eli/reg/2016/679/oj 27 April 2016. [10] Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. 2023. Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 10135–10145. [11] Mohsen Ghafoorian, Denis Korzhenkov, Adil Karjauv, Ioannis Lelekas, Noor Fathima, Spyridon Stasis, Hanno Ackermann, Boris van Breugel, Markus Nagel, Fatih Porikli, et al. 2026. MobileWan: Closing the Quality Gap for Mobile Video Diffusion. arXiv preprint arXiv:2607.06173 (2026). [12] Zixu Hao, Jianyu Wei, Tuowei Wang, Minxing Huang, Huiqiang Jiang, Shiqi Jiang, Ting Cao, and Ju Ren. 2026. Scaling llm test-time compute with mobile npu on smartphones. In Proceedings of the 21st European Conference on Computer Systems. 2157–2172. [13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33. 6840–6851. [14] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. 2024. Vbench: Comprehensive benchmark suite for video generative models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 21807–21818. [15] Joo Seong Jeong, Jingyu Lee, Donghyun Kim, Changmin Jeon, Changjin Jeong, Youngki Lee, and Byung-Gon Chun. 2022. Band: coordinated multi-dnn inference on heterogeneous mobile processors. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services. 235–247. [16] Fucheng Jia, Deyu Zhang, Ting Cao, Shiqi Jiang, Yunxin Liu, Ju Ren, and Yaoxue Zhang. 2022. CoDL: Efficient CPU-GPU Co-execution for Deep Learning Inference on Mobile Devices. In MobiSys ’22: Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services. doi:10.1145/3498361.3538932 [17] Anuj Kapoor and Madhav Kumar. 2025. Frontiers: Generative AI and personalized video advertisements. Marketing Science 44, 4 (2025), 733–747. [18] Animesh Karnewar, Denis Korzhenkov, Ioannis Lelekas, Noor Fathima, Adil Karjauv, Mohsen Ghafoorian, and Amirhossein Habibian. 2026. Neodragon: Mobile video generation using diffusion transformer. In International Conference on Learning Representations, Vol. 2026. 39506– 39537. [19] Johanna Karras, Yingwei Li, Nan Liu, Luyang Zhu, Innfarn Yoo, Andreas Lugmayr, Chris Lee, and Ira Kemelmacher-Shlizerman. 2024. Fashion-vdm: Video diffusion model for virtual try-on. In SIGGRAPH Asia 2024 Conference Papers. 1–11. [20] Bosung Kim, Kyuhwan Lee, Isu Jeong, Jungmin Cheon, Yeojin Lee, and Seulki Lee. 2026. Mobile-Oriented Video Diffusion: Enabling Textto-Video Generation on Mobile Devices Without Retraining, Compression, or Pruning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026, pp. 3961-3971. doi:10.1109/WACV61042.2026.00386

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY [21] Youngsok Kim, Joonsung Kim, Dongju Chae, Daehyun Kim, and Jangwoo Kim. 2019. 𝜇layer: Low latency on-device inference using cooperative single-layer acceleration and processor-friendly quantization. In Proceedings of the Fourteenth EuroSys Conference 2019. 1–15. [22] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. 2024. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 (2024). [23] Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Y Wang. 2024. T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. Advances in neural information processing systems 37 (2024), 75692– 75726. [24] Maoliang Li, Jiayu Chen, Zihao Zheng, Ziqian Li, Xinhao Sun, Guojie Luo, Chenchen Liu, and Xiang Chen. 2026. HeRo: Adaptive Orchestration of Agentic RAG on Heterogeneous Mobile SoC. arXiv preprint arXiv:2603.01661 (2026). [25] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. 2023. Q-Diffusion: Quantizing Diffusion Models. In IEEE/CVF International Conference on Computer Vision (ICCV). [26] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. 2023. SnapFusion: Textto-Image Diffusion Model on Mobile Devices within Two Seconds. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Main Conference Track. [27] Zhuojin Li, Hsin-Pai Cheng, Hong Cai, Shizhong Han, and Fatih Porikli. 2026. CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2026. [28] Neiwen Ling, Xuan Huang, Zhihe Zhao, Nan Guan, Zhenyu Yan, and Guoliang Xing. 2022. Blastnet: Exploiting duo-blocks for crossprocessor real-time dnn inference. In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems. 91–105. [29] Linux Kernel Developers. 2024. Buffer Sharing and Synchronization (dma-buf). https://www.kernel.org/doc/html/latest/driver-api/dmabuf.html. The Linux Kernel documentation. Accessed: 2026-09-02. [30] Bo Liu, Ming Ding, Sina Shaham, Wenny Rahayu, Farhad Farokhi, and Zihuai Lin. 2021. When machine learning meets privacy: A survey and outlook. ACM Computing Surveys (CSUR) 54, 2 (2021), 1–36. [31] MLC LLM. 2025. Mlc llm: Universal llm deployment engine with ml compilation. [32] Zhengyao Lv, Chenyang Si, Tianlin Pan, Zhaoxi Chen, Kwan-Yee K. Wong, Yu Qiao, and Ziwei Liu. 2025. Dual-Expert Consistency Model for Efficient and High-Quality Video Generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). arXiv:2506.03123. [33] Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. 2024. Spotserve: Serving generative large language models on preemptible instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1112–1127. [34] Aryama V Murthy, Yashas N Kotre, Prathmesh Sharma, Pragya Mishra, Sanjith Ganapathi, and Priyesh Shukla. 2026. Phase Matters: Characterizing Heterogeneous Vision-Language Inference on a Mobile SoC. In Proceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services Workshops. 322–327. [35] Mergen Nachin, Digant Desai, Stephen Jia, Chen Lai, Mengwei Liu, Jacob Szwejbka, Raziel Alvarez, RJ Ascani, Dave Bort, Manuel Candales, et al. 2026. ExecuTorch-A Unified PyTorch Solution to Run ML Models On-Device. Proceedings of Machine Learning and Systems 8 (2026),

Shen et al. 569–609. [36] Keondo Park, You Rim Choi, Inhoe Lee, and Hyung-Sin Kim. 2023. PointSplit: Towards On-device 3D Object Detection with Heterogeneous Low-power Accelerators. In IPSN ’23: Proceedings of the 22nd International Conference on Information Processing in Sensor Networks. doi:10.1145/3583120.3587045 [37] William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Transformers. In IEEE/CVF International Conference on Computer Vision (ICCV). 4195–4205. [38] Qualcomm Technologies, Inc. 2024. Adreno 830 GPU Architecture and OpenCL Programming Reference. Qualcomm Technologies, Inc. Qualcomm technical reference manual. [39] Qualcomm Technologies, Inc. 2024. FastRPC: Remote Procedure Call Framework for Hexagon DSP. Qualcomm Technologies, Inc. Qualcomm Hexagon SDK documentation. [40] Qualcomm Technologies, Inc. 2024. Qualcomm AI Engine Direct SDK. https://www.qualcomm.com/developer/software/qualcomm-aiengine-direct-sdk. Accessed: 2026-09-02. [41] Qualcomm Technologies, Inc. 2024. Qualcomm Hexagon V79 Programmer’s Reference Manual. Qualcomm Technologies, Inc. https: //docs.qualcomm.com/doc/80-N2040-60 Accessed: 2026-09-02. [42] Qualcomm Technologies, Inc. 2024. Snapdragon 8 Elite Mobile Platform Product Brief. Product Brief. https://www.qualcomm.com/ smartphones/products/8-series/snapdragon-8-elite-mobile-platform Accessed: 2026-09-02. [43] Qualcomm Technologies, Inc. 2026. Qualcomm Hexagon V81 HMX Programmer’s Reference Manual. Qualcomm Technologies, Inc. https: //docs.qualcomm.com/doc/80-N2040-62/ Rev. AA, March 19, 2026. [44] Tim Salimans and Jonathan Ho. 2022. Progressive Distillation for Fast Sampling of Diffusion Models. In International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=TIdIXIpzhoI [45] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. 2023. Post-training Quantization on Diffusion Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [46] Junhyuk So, Jungwon Lee, Daehyun Ahn, Hyungjun Kim, and Eunhyeok Park. 2023. Temporal Dynamic Quantization for Diffusion Models. In Advances in Neural Information Processing Systems (NeurIPS). [47] Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020). [48] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2023. Consistency models. arXiv preprint arXiv:2303.01469 (2023). [49] State of California. 2018. California Consumer Privacy Act of 2018 (CCPA). California Civil Code §§ 1798.100 et seq. https://oag.ca.gov/ privacy/ccpa As amended. [50] Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. 2024. EMO: Emote Portrait Alive—Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions. In European Conference on Computer Vision (ECCV). [51] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. 2025. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 (2025). [52] Xiaofei Wang, Yiwen Han, Victor C. M. Leung, Dusit Niyato, Xueqiang Yan, and Xu Chen. 2020. Convergence of Edge Computing and Deep Learning: A Comprehensive Survey. IEEE Communications Surveys & Tutorials 22, 2 (2020), 869–904. [53] Jianyu Wei, Ting Cao, Shijie Cao, Shiqi Jiang, Shaowei Fu, Mao Yang, Yanyong Zhang, and Yunxin Liu. 2023. Nn-stretch: Automatic neural network branching for parallel inference on heterogeneous multiprocessors. In Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services. 70–83.

Pipelining Video Diffusion with Shared Weights [54] Xinming Wei, Jiahao Zhang, Haoran Li, Jiayu Chen, Haoning Guan, Rui Qu, Maoliang Li, Xiang Chen, and Guojie Luo. 2025. Agent. xpu: Efficient scheduling of agentic llm workloads on heterogeneous soc. arXiv preprint arXiv:2506.24045 (2025). [55] Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. 2025. Video models are zero-shot learners and reasoners. arXiv preprint arXiv:2509.20328 (2025). [56] Yushu Wu, Yanyu Li, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ke Ma, Arpit Sahni, Ju Hu, Aliaksandr Siarohin, Dhritiman Sagar, et al. 2025. Taming diffusion transformer for efficient mobile video generation in seconds. arXiv preprint arXiv:2507.13343 (2025). [57] Yushu Wu, Zhixing Zhang, Yanyu Li, Yanwu Xu, Anil Kag, Yang Sui, Huseyin Coskun, Ke Ma, Aleksei Lebedev, Ju Hu, et al. 2025. Snapgenv: Generating a five-second video within five seconds on a mobile device. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2479–2490. [58] Tong Xin, Xinrui Shi, Mingkai Dong, and Zeyu Mi. 2026. Inference in the Shadows: Taming Memory Bandwidth Contention in Mobile {LLM} Inference with Sereno. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). 2299–2318. [59] Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. 2025. Fast On-device LLM Inference with NPUs. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (ASPLOS ’25). doi:10.1145/3669940.3707239 [60] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. 2023. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems 36 (2023), 41693–41706. [61] Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. 2024. PowerInfer-2: Fast Large Language Model Inference on a Smartphone. arXiv preprint arXiv:2406.06282 (2024). [62] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. 2025. Cogvideox: Text-tovideo diffusion models with an expert transformer. In International Conference on Learning Representations (ICLR). arXiv:2408.06072. [63] Zilin Ye and Qianyi Huang. 2026. Understanding Concurrent DNN Interference on Mobile SoCs. In Proceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services Workshops (MobiSys Workshops ’26), 2026. doi:10.1145/3812836.3814775 [64] Juheon Yi and Youngki Lee. 2020. Heimdall: mobile GPU coordination platform for augmented reality applications. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1–14. [65] Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. 2025. From Slow Bidirectional to Fast Autoregressive Video Diffusion Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [66] Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T Freeman, and Jiajun Wu. 2025. Wonderworld: Interactive 3d scene generation from a single image. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 5916–5926. [67] Yuanpeng Zhang, YuXuan Wu, Yitong Xiao, Chenhao Xue, Yi Ren, Cong Li, Yihan Yin, Dimin Niu, and Guangyu Sun. 2026. CODA: Algorithm-Hardware Co-design for Edge Video Diffusion via NMPEnabled Compute-Cache Operator Disaggregation. arXiv preprint arXiv:2607.14908 (2026). [68] Zongpu Zhang, Pranab Dash, Qiang Xu, Y Charlie Hu, Jian Li, and Haibing Guan. 2026. Rethinking DVFS for Mobile LLMs: Unified

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Energy-Aware Scheduling with CORE. Proceedings of Machine Learning and Systems 8 (2026), 186–198. [69] Lin Zhao, Yushu Wu, Aleksei Lebedev, Dishani Lahiri, Meng Dong, Arpit Sahni, Michael Vasilkovsky, Hao Chen, Ju Hu, Aliaksandr Siarohin, et al. 2026. S2dit: Sandwich diffusion transformer for mobile streaming video generation. arXiv preprint arXiv:2601.12719 (2026). [70] Tianchen Zhao, Tongcheng Fang, Haofeng Huang, Enshu Liu, Rui Wan, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, et al. 2024. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation. arXiv preprint arXiv:2406.02540 (2024). [71] Yang Zhao, Yanwu Xu, Zhisheng Xiao, Haolin Jia, and Tingbo Hou. 2024. Mobilediffusion: Instant text-to-image generation on mobile devices. In European Conference on Computer Vision. Springer, 225– 242. [72] Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Lulu Gu, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, et al. 2025. Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.21755 (2025). [73] Shaobin Zhuang, Yiwei Guo, Yanbo Ding, Kunchang Li, Xinyuan Chen, Yaohui Wang, Fangyikang Wang, Ying Zhang, Chen Li, and Yali Wang. 2025. Timestep master: Asymmetrical mixture of timestep lora experts for versatile and efficient diffusion models in vision. arXiv preprint arXiv:2503.07416 (2025).

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

A

Shen et al.

Background Details

This appendix keeps the material that §2 summarises. Reviewers do not need it to follow the paper.

Snapdragon 8 Elite SoC

L2 Cache

A.1

Multi-expert models and the DCM sampler

Multi-expert diffusion models specialise different timestep or noise regimes with different denoiser parameters. For example, Wan2.2-A14B [51] stores separate 14 B high-noise and low-noise transformers and selects between them using a fixed timestep boundary. TimeStep Master [73] attaches multiple timestep-specialised LoRA experts to a shared backbone, including eight fine-grained core experts and four coarser context experts. These experts are lightweight adapters rather than independently stored denoisers. DCM [32] freezes its distilled semantic expert and uses it as a shared backbone. It forms the detail expert by adding new timestep-embedding layers and rank-128 LoRA adapters to the attention projections. Figure 2 shows its inference pipeline. Together, the two 14 B INT8 experts of Wan2.2A14B require 28 GB. Even if the pair were factorised post hoc into a shared base plus low-rank adapters, the 14 B base alone would still exceed our target platform’s memory budget unless it were distilled to a smaller model. DCM shares parameters by construction. Its semantic expert is the shared base of 1,419 M parameters, and the detail expert adds only a rank-128 attention adapter of 94.4 M parameters and a new timestep-embedding block of 25.6 M parameters. We therefore use the two expert model DCM-Wan2.1 as our implementation and evaluation workload. This choice is dictated by storage feasibility rather than convenience. The expert boundary is a timestep threshold, and where it lands in step index depends on the sampler. DCM uses the detail expert when 𝑡≤978. Its sampler uses six fixed timesteps: [1000, 992, 982, 949, 905, 810]. Write 𝜅 for the index of the first step that uses the detail expert. Here 𝜅=3, so the first three steps use the semantic expert and the final three use the detail expert. Two properties of this model matter for the system. First, a step is one forward pass and offers no independent second branch to overlap. Second, its six steps provide too few partition points to balance the GPU and NPU effectively. We therefore consider finer-grained partitioning in §4.

A.2

Snapdragon 8 Elite silicon

The Snapdragon 8 Elite SoC [42] used in Samsung Galaxy S25 platform exposes two programmable accelerators that share a unified memory pool, shown in Figure 8. The Hexagon V79 NPU contains a VLIW DSP, a 1024-bit SIMD vector unit (HVX), and a matrix unit (HMX). HMX multiplies a

NPU

CPU

System Cache

VLIW DSP (Scalar)

HVX (Vector)

GPU HMX (Tensor)

VTCM (NPU Shared Memory)

Adreno GPU GPU Cache

Shared DRAM Unified Memory for All Subsystems

Figure 8: The Snapdragon 8 Elite SoC. The Hexagon NPU pairs a VLIW scalar DSP with the HVX vector unit and the HMX matrix unit over VTCM, its softwaremanaged on-chip memory. The CPU and the Adreno GPU keep their own caches, and all three subsystems reach the same shared DRAM.

32×32 tile of activations by a 32×32 tile of weights. Both operands must already sit in VTCM, in the tile order the hardware expects [43]. HMX supports INT2, INT4, INT8, INT16, FP8, and FP16 with mixed-precision support [42]. Reachable throughput is gated by the FastRPC [39] channel between the application processor and the compute DSP (cDSP), the subsystem that hosts the NPU. Programs are typically delivered as offline-compiled Qualcomm AI Engine Direct (QNN) context binaries [40]. The toolchain hardcodes input shapes at build time and refuses to serialise graphs whose activation footprint exceeds an internal limit, a constraint we revisit in §6. The Adreno 830 GPU [38] exposes INT8 compute through an OpenCL integer dot-product instruction (dp4a). It has no matrix unit. HMX keeps a tile resident and reuses it in hardware. On the GPU every load from DRAM must be scheduled and reused by the kernel itself. A general matrix multiplication (GEMM) reuses each load heavily and attention barely does, which is why the GPU slows down at attention. The system kernel reports 10.85 GiB of usable memory on this device. Adreno advertises a maximum OpenCL buffer size of 1024 MiB, and §3 shows that the actual limit is lower. The two runtimes share memory only through a DMA-BUF [29] file descriptor, allocated either through Android’s AHardwareBuffer API [2] or through the Hexagon runtime’s own heap. One descriptor can be mapped by the CPU, the GPU, and the DSP at the same time. QNN’s Hexagon backend accepts memory only as such a descriptor and has no way to take an OpenCL buffer directly. A tensor produced by a GPU kernel therefore reaches an NPU graph only when both runtimes register the same region.

Pipelining Video Diffusion with Shared Weights

A.3 Heterogeneous inference for generative models Two strategies dominate prior work on splitting an inference workload across multiple accelerators on the same SoC. Layer partition (e.g., HeteroInfer [6], NN-Stretch [53], and Heimdall [64]) places a contiguous prefix of model layers on one device and the suffix on another, handing off the activation tensor at each boundary. This strategy incurs at least one cross-device handoff per layer split. An LLM crosses that boundary once per token. A diffusion model would cross it once per block of every step, which is 𝑁 ·𝐿 times for a single video. Expert partition is used in mobile MoE deployment [31, 35], which places different experts on different devices and routes tokens dynamically. The denoising trajectory gives diffusion models a third axis that LLMs do not have. A video synthesiser runs the same DiT graph 𝑁 times against a common latent, with no token-level dependency between steps. Few-step distillation has pushed 𝑁 from 25−50 down to 4−8, which shrinks the trajectory but does not remove the axis. Step partition runs steps 0..𝑘−1 on one accelerator and steps 𝑘..𝑁 −1 on another, with a single latent handoff in the middle. The handoff cost is then 𝑂 (1) per video rather than 𝑂 (𝑁 ·𝐿) for 𝐿 blocks. This matters because every synchronisation between the two accelerators has a fixed cost [6].

B

Implementation Details

This appendix expands §5 with the kernel-level detail of the NPU and GPU DiT paths and the VAE decoder.

B.1

NPU DiT Kernels

The NPU executor of Figure 3 is a custom FastRPC package. Within it, HeteroVDiff ports the open-source htp-ops-lib HMX FlashAttention kernel [12] to the model’s video shapes. The executor creates a six-context HVX worker pool, enables HMX for those workers, and dispatches every attention kernel through that pool. The deployed workload does not use an attention mask, so the executor takes an unmasked path and never allocates the 𝑀×𝑀 zero-valued mask tensor that a generic kernel would read. The port fixes two library routines that were wrong for this model. The softmax exponential was a lookup table that is accurate only for small attention scores, and at this model’s score magnitudes its output correlates with an FP64 reference at only 0.65. We replace it with a polynomial exp2 in FP16 on HVX, which is correct at every magnitude we tested and costs about 1.2× in attention time. The FP32-toFP16 cast, which the executor applies wherever the FP32 residual stream enters a GEMM, used a packing instruction that interleaves its two input vectors, so the outputs came

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

out in the wrong order. One de-interleave step after it makes the cast bit-exact. The remaining changes implement the model’s attention semantics. Both self-attention and cross-attention use threeaxis rotary positions with adjacent-pair rotation. The kernels also apply the query-key normalisation gains and all projection biases supplied by the model. The executor and kernels share the same rotary tables and metadata. HMX GEMMs consume FP16 operands, so the NPU dequantises the two feed-forward matrices of the shared INT8 base once per block into a reusable FP16 scratch buffer. HVX reorders and widens the tiled INT8 values, then applies the per-column scales. All output chunks in that block reuse the converted weights before the scratch buffer is recycled for the next block. For detail-expert blocks, the kernel applies the rank-128 adapter at run time instead of merging it into the weights. A first HMX GEMM projects the block input onto the 128dimensional factor basis, a second expands it back, and an HVX pass adds the result into the FP16 projection output. The base weights never change, so the expert switch moves no weights.

B.2

GPU DiT Kernels

The GPU DiT executor of Figure 3 implements each block as a sequence of OpenCL kernels that read the shared baseweight artifact directly. Attention is split into a query-key GEMM, a row-wise softmax, and an attention-value GEMM. This structure gives the integer dot-product kernels the operand reuse of the GEMM mainloop, and it keeps each dispatch to tens of milliseconds. Our fused INT8 kernel, which launched the whole sequence as one dispatch, was killed before it completed at the deployment length. The GPU reads the canonical HMX-tiled INT8 base-weight artifact without creating a reordered copy. Four fixed-stride vector loads assemble each char4 operand, and the percolumn weight scale is applied in the GEMM epilogue. The FP16 inter-block hidden state and the scale for each GEMM input follow §4.4.

B.3

NPU VAE Decoder

The NPU runs the VAE decoder through the QNN HTP backend after the final denoising step. The QNN GPU backend does not provide the required three-dimensional convolution. On the QNN HTP backend, the floating-point version fails during graph preparation, while the INT8 version compiles. The decoder is therefore implemented as an INT8 QNN HTP graph. The first frame and later frames use separate QNN HTP context binaries. The first frame skips a temporal upsampling operation, while later frames read and update the recurrent

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

feature cache. A zero-filled cache is not equivalent to the firstframe computation, so the runtime selects the appropriate context for each frame. The recurrent feature cache also rules out decoding a whole frame at once, because the graph’s estimated allocation exceeds the toolchain’s 2 GiB per-graph ceiling. HeteroVDiff therefore divides each latent frame into spatial VAE tiles and uses the largest tile we found to compile under that ceiling, 60×40 latent pixels, three per frame. Tiles are

Shen et al.

decoded one after another, and each keeps its own recurrent cache across all frames of the clip, so only one tile’s cache is resident at a time.

C

VBench-2.0 Scores

Table 6 lists the per-dimension VBench-2.0 scores discussed in §6.4. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

Pipelining Video Diffusion with Shared Weights

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 6: VBench-2.0 [72] scores (%) of the same four numeric paths on all 18 dimensions, grouped by category. Total is the mean of the five category means. Commonsense

FP32 (no quantisation) 87.93 78.07 83.03 18.32 22.55 13.85 33.33 6.41 13.11 10.19 81.33 42.06 58.73 51.45 29.11 14.49 85.38 NPU: INT8 weights, FP16 compute 88.33 76.16 83.64 22.71 22.06 16.16 31.33 7.98 12.89 11.42 81.26 41.55 63.93 53.28 30.86 9.54 84.80 GPU: INT8 weights, INT8 activations 86.96 79.89 82.41 19.05 21.57 11.11 25.00 7.89 12.00 12.96 86.15 40.94 65.85 51.47 27.85 21.10 85.38 HeteroVDiff, 𝑘𝑏 =51 88.19 75.60 83.41 19.05 23.04 12.12 30.67 8.44 12.67 12.65 82.56 46.52 61.72 57.78 30.12 25.95 81.29

22.41 22.41 20.69 22.99

l

Physics

To ta

Creativity

Hu ma

Numeric path

Controllability

na nat om Hu y ma ni den tity Hu ma nc lo Dy nam thes ic a ttr Dy ibu nam te ic s pat ial Mo rel . tio no rde Hu r ma ni nte rac Co ti o mp n l e xp Co lot mp lex lan dsc Ca ape me ra mo tio n Div ers i ty Co mp osi ti o n Me cha nic s Th erm o ti cs Ma ter Mu ial ltivie wc on Ins sis tan t. ce pre ser Mo v. tio nr ati on alit y

Human fidelity

50.77 50.98 51.38 51.98

Related documents

• HeteroVDiff: Pipelining Video Diffusion across Mobile NPU and GPU with Shared Weights#1327834
• LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing#327586
• UnStep: Training-Free Acceleration of Causal Video Diffusion with Fewer Steps Than Distillation#1116849
• Paris 2.0: A Decentralized Diffusion Model for Video Generation#224504
• From Static to Dynamic: On-Policy Distillation from Image to Video Diffusion Models#1128644