VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion
arXiv:2605.30351v1 [cs.CV] 28 May 2026
Hidir Yesiltepe1 Jiazhen Hu1 Tuna Han Salih Meral1 Adil Kaan Akan2 Kaan Oktay2 Hoda Eldardiry1 Pinar Yanardag1 1 2 Virginia Tech fal Project Page: https://videomla.github.io
Abstract Long-rollout causal video diffusion has converged on a fixed-size sliding-window KV cache, with recent progress innovating within this layout by changing which tokens occupy the window or how their positions are encoded. The per-head KV layout itself, a dominant contributor to streaming memory and latency, has been mostly left unchanged. In this paper, we present the first study of Multi-Head Latent Attention (MLA) in video diffusion. VideoMLA replaces per-head keys and values with a shared low-rank content latent and a shared decoupled 3D-RoPE positional key, reducing per-token KV memory by 92.7% at every cached layer. We further investigate why MLA succeeds in video diffusion even though the spectral assumption often used to motivate it in language models does not hold: pretrained video attention is not low-rank, with 99%-energy effective rank far above any practical latent dimension. VideoMLA retains quality at compression ratios where direct spectral approximation would predict large reconstruction error. We show that the MLA bottleneck, rather than the pretrained spectrum, determines the effective rank: both spectral and random initialization occupy nearly the full rank budget from initialization, and training preserves this budget while adapting within it. On VBench, VideoMLA matches short-horizon streaming video diffusion baselines, achieves the best overall score at long horizons among evaluated methods, and improves throughput by 1.23× on a single B200.
1
Introduction
Causal video diffusion models [9, 4, 16, 31, 25, 24, 17, 27, 22, 3, 23, 12, 30, 13] have gained traction as the dominant approach to streaming, long-horizon video generation. Distilled from bidirectional teachers, they generate frames [31, 5, 11, 8] or chunks autoregressively while attending to a rolling key-value (KV) cache of past frames, producing minute-long videos at interactive rates on a single GPU. As models scale toward longer rollouts, the per-head KV cache increasingly defines the operating point. At Wan-1.3B scale [21], each cached token stores 2 × 12 × 128 = 3,072 dense KV scalars per layer, accounting for keys and values across 12 heads with 128 channels each. With a 21-latent-frame cache, 1,560 tokens per latent frame, and 30 transformer layers, the dense KV cache contains 3.02B scalars, or about 6.0GB in bf16/fp16. This footprint explains why recent streaming systems use fixed-size sliding-window caches: retaining all past KV states would grow linearly with rollout length. However, fixing the window only bounds the number of cached tokens; it does not reduce the per-token, per-layer cost of the per-head KV layout. Reducing this layout is therefore a direct lever for longer horizons, larger batches, and faster inference. The dominant line of recent work treats the cache as a fixed-size sliding window and innovates inside it. CausVid [27] initiated this thread by converting bidirectional diffusion into causal autoregressive generation via distribution matching distillation, with a sliding KV cache from inception. Preprint.
(a) Normalized singular values of [WK ; WV ]
(b) Cumulative spectral energy
dc = 192
10−2
1200
0.8
Effective rank rτ
σk /σ1
10−1
0.6
0.4
0.2
dc = 192 Emed = 0.458
Median across layers 10
−3
100
101
1000 800 τ = 0.90 τ = 0.95 τ = 0.99
600 400
dc = 192
200
Median across layers 102
Rank k
(c) Effective rank by layer 1400
τ = 0.99 τ = 0.95 τ = 0.90
1.0
Cumulative energy E(k)
100
103
0.0
0 0
100
200
300
400
Rank k
500
600
700
0
5
10
15
20
25
Layer index
Figure 1: Pretrained video diffusion attention is not low-rank, unlike in language models. Singular value analysis of [WK ; WV ] ∈ R3072×1536 across the 30 transformer blocks of Wan2.1T2V-1.3B. At dc = 192, the median layer captures only Emed = 0.458 of the spectral energy, and the 99%-energy effective rank exceeds 1300 in every layer. Self-Forcing [9] closed the train–test gap by conditioning training on self-generated frames within the same rolling cache. Subsequent work refined this recipe through attention-sink, token-selection, and compressed-memory mechanisms for long-range consistency [16, 25, 17, 28, 12, 29], training strategies for multi-minute rollouts and prompt switching [4, 22, 7, 24], improved distillation objectives [31, 17], and positional reparameterization such as Infinity-RoPE [24]. However, these methods all preserve the per-head KV layout that fills the window in the first place: they redistribute, reweight, compress over time, or reposition cached tokens without reducing the per-token KV state. A second, complementary line changes the attention computation itself. SANA-Video [3] replaces softmax attention with block-causal linear attention, removing the conventional KV cache and using a constant-memory cumulative state for long-video generation. SCD [2] reduces cached state by routing temporal reasoning through a 25-layer causal encoder and using a 10-layer frame-wise decoder, so only the encoder layers cache. Under the same Wan cache geometry, this reduces dense KV storage by 16.7%. VideoMLA is orthogonal: it keeps all 30 self-attention layers cached but reduces each token’s cached state from 3072 to 224 scalars, yielding an 11.4× smaller cache than SCD for the same 21-latent-frame window. Thus, rather than changing which tokens are cached, how they are positioned, or how many layers cache, VideoMLA targets the remaining factor directly: the per-token KV layout at every cached self-attention layer. In this paper, we intervene on the per-head layout itself. Building on Multi-Head Latent Attention (MLA) [14], we present VideoMLA, the first MLA-style latent KV cache for autoregressive video diffusion. VideoMLA replaces per-head keys and values with a shared low-rank content latent and a head-shared decoupled 3D-RoPE positional key, reducing per-token KV memory by 92.7% at every cached layer. This raises a puzzle: MLA is usually motivated by low-rank pretrained WK , WV [14, 10], yet Wan-1.3B (Fig. 1) has 99%-energy rank far above practical latent dimensions. VideoMLA nonetheless retains quality where direct spectral approximation would incur large reconstruction error (Fig. 2). We show that the MLA bottleneck, not the pretrained spectrum, determines the effective rank: SVD and random initialization both nearly saturate the rank budget, which training preserves with little spectral change. The design question therefore shifts from what is the intrinsic rank? to what latent budget preserves video quality? Our contributions are summarized as follows: • Latent KV caching for video diffusion. We introduce VideoMLA, an MLA-style autoregressive video diffusion model that replaces per-head keys and values with a shared content latent and a head-shared decoupled 3D-RoPE key, reducing per-token KV memory by 92.7% at every cached layer. • A spectral puzzle and rank-budgeted resolution. We show that Wan-1.3B video attention is not low-rank: the 99%-energy effective rank of [WK ; WV ] far exceeds practical latent dimensions. VideoMLA nevertheless retains quality, while both SVD and random initialization saturate the imposed rank budget from initialization and preserve it during training. • Efficient long-horizon generation. We identify the NoPE/RoPE allocation that preserves visual fidelity and motion consistency at minute-scale horizons. On VBench, VideoMLA matches short-horizon baselines, achieves the best long-horizon overall score among evaluated methods, and improves throughput by 1.23× on a single B200. 2
(a) All four budgets share a common normalized spectrum
10−2
10
dc = 64 dc = 128 dc = 256 dc = 512
100
101
102
103
(c) Effective rank tracks the budget at every dc 500
0.8
Effective rank r0.99
Cumulative energy E(k)
σk /σ1
10−1
−3
(b) Cumulative spectral energy τ = 0.99
1.0
100
0.6
0.4 dc = 64 dc = 128 dc = 256 dc = 512
0.2
dc = 64 dc = 128
400
dc = 256 dc = 512
300
200
100
0.0
0 0
100
Rank k
200
300
400
500
Rank k
600
0
5
10
15
20
25
Layer index
Figure 2: The composed operator occupies its full rank-dc budget at every dc and every layer. Singular value analysis of the composed operator Mlearned = [W↑K W↓KV ; W↑V W↓KV ] for SVDinitialized VideoMLA students at dc ∈ {64, 128, 256, 512}. (a) Median normalized spectra share a common envelope, truncated at dc . (b) Cumulative spectral energy. (c) Layer-wise 99%-energy effective rank: r0.99 ≈ 0.98 dc at every budget, uniformly across depth. The composed operator’s rank is determined by the architectural bottleneck, not by the spectral structure of the dense source.
2
Related Work
Causal Video Generation. Causal video diffusion converts a bidirectional teacher into a streaming student that generates frames or chunks autoregressively with a rolling KV cache. CausVid [27] initiated this line with Distribution Matching Distillation (DMD) [26] based causal distillation, and Self-Forcing [9] reduced train–test mismatch by training on self-generated rollouts. Subsequent work improves long-horizon stability through joint denoising and attention sinks [16], teacher-guided correction [4], causal ODE initialization [31], reward-weighted distillation and EMA sinks [17], deep sink and cache pruning [25], KV recaching for prompt switches [22], and block-relative temporal RoPE [24]. These methods improve what is stored in the window or how it is positioned, but retain the dense per-head KV layout. Efficient Causal Video Generation. A complementary line restructures attention to reduce memory or compute. SANA-Video [3] replaces softmax with block-causal linear attention and uses a constantsize cumulative state. SCD [2] separates temporal reasoning from frame-wise rendering, caching only the causal encoder. VideoSSM [28] augments sliding-window KV with an SSM-compressed global memory. These approaches reduce temporal or layer-wise memory, but do not compress the per-token, per-head KV state at every cached layer. Multi-Head Latent Attention. DeepSeek-V2 [14] introduced Multi-Head Latent Attention (MLA), replacing per-head KV with a shared low-rank latent and a decoupled positional key; DeepSeekV3 [15] scaled this design. MTLA [6] further compresses along time, while MHA2MLA [10] and TransMLA [18] convert pretrained MHA [20] or GQA [1] LLMs into MLA. These works target language deployment. We study MLA in video diffusion, where the memory profile and pretrained attention spectrum differ substantially.
3
Method
We write xt ∈ Rd for the attention input at current chunk t, where a chunk denotes a group of latent frames. Let d be the model dimension, nh the number of heads, and dh the per-head dimension, so that d = nh dh . VideoMLA introduces a shared KV latent dimension dc for cached content and splits each head into a NoPE content-scoring subspace and a RoPE positional subspace, dh = dnope + drope . h h The NoPE part is reconstructed from the shared latent and is not rotary-position encoded; the RoPE part uses a head-shared decoupled 3D-RoPE key. 3.1
Compressed KV Cache Construction
Each video latent token xt ∈ Rd produced by the backbone is first compressed into a low-rank latent that summarizes its key and value content for the rolling cache: cKV = W↓KV xt ∈ Rdc , t 3
(1)
+ channel concat Multi-Head Attention (MHA) 𝑟𝑜𝑝𝑒
+
𝑟𝑜𝑝𝑒
𝑘𝑡
token concat
𝑞𝑡
+
+
Decoupled 3D-RoPE
𝑛𝑜𝑝𝑒
𝑣𝑡
𝑊↑𝑉
+
𝑘𝑡
Value Up
𝑊↑𝐾
Key Up
𝑊𝑅𝑄
Query Rotation
𝑛𝑜𝑝𝑒
𝑞𝑡
Query Up
𝑊↑𝑄
𝑘𝑡𝑅
𝑐𝑡𝐾𝑉
𝑊↓𝐾𝑉
𝑞𝑡𝑅
𝑄
𝑐𝑡
cached
𝑊𝑅𝐾
KV Down
Compressed KV Cache 𝑐 𝐾𝑉
Key Rotation
𝑊↓𝑄
Query Down
Video Latent 𝑥𝑡
Figure 3: Overview of VideoMLA. VideoMLA replaces dense per-head KV cache in Causal Wan 2.11.3B with a low-rank latent obtained by jointly compressing keys and values through shared down/up projections, with positional information carried by a single decoupled rotated key. Orange blocks denote down projections, green blocks denote rotations, and white blocks denote up projections; latent frames are colored blue for the key/value stream and white for the query stream. Each block is annotated with the corresponding weight matrix from Section 3, named latents are shown in red. where W↓KV ∈ Rdc ×d is the joint KV down-projection (Fig. 3, KV Down). The vector cKV is t the content object written into the compressed KV cache. It has dimension dc ≪ nh dh , so it replaces the dense per-head content keys and values that would otherwise be stored for every head. Positional information is not folded into this latent; it is stored separately through the decoupled key ktR introduced in Section 3.2. Thus, each cached token stores the pair (cKV , ktR ) rather than dense t per-head KV states. The per-head keys and values needed by attention are obtained from cKV through two up-projections, t nope K KV kt,h = W↑,h ct ,
V vt,h = W↑,h cKV , t
(2)
where h ∈ {1, . . . , nh } indexes attention heads and W↑K , W↑V are the key and value up-projections (Fig. 3, Key Up and Value Up). Two properties of this construction are important. First, the same cached latent cKV is shared across all heads: a single cache read produces nh per-head keys and nh t per-head values through Eq. 2. Second, the reconstructed key carries no rotary positional information. nope It is the content-only component of the per-head key, denoted kt,h ; the positional component lives in the separate RoPE subspace. Together with the decoupled positional key, the per-token cached state is reduced from the 2nh dh scalars of a dense per-head KV cache to dc + drope scalars. In our default setting, this is 224 scalars h per token per layer, a 92.7% reduction. The query path is per-token and uses an analogous down/up structure. From xt , a query downprojection produces a query latent, and a content up-projection recovers the per-head NoPE query: Q dq cQ t = W ↓ xt ∈ R ,
(3)
nope Q Q qt,h = W↑,h ct ,
(4)
where dq is the query latent dimension and W↑Q is the Query Up projection in Fig. 3. Since queries are recomputed from the current block at every generation step, cQ t is internal to the layer and is 4
Metric
Causal Full
Causal Local
Causal Linear
MLA Local
Memory
2N D
2W D
D dh
W (dc + drope ) h
Comp. (N -th token)
ND
WD
D dh
nW (dc + drope ) h
Comp. (N tokens)
1 N 2D 2
NW D
N D dh
nN W (dc + drope ) h
Table 1: Memory and compute costs across four attention variants. For a sequence of length N with hidden dimension D, n heads, per-head dimension dh = D/n, local window W , latent KV dimension dc (dc ≪ D), and shared decoupled-RoPE dimension drope . h
never written to the KV cache. The head-sharing occurs only in the decoupled positional branch: VideoMLA uses a single RoPE key shared across heads, while the NoPE queries, NoPE keys, and values remain head-specific after up-projection. The dimension dc is the layer’s main content-cache capacity knob: it controls how aggressively the cached content is compressed and how much shared subspace the model can use for joint key-value content. The choice of dc is studied empirically in Figure 7 and Appendix. 3.2
Decoupled 3D-RoPE
The latent cache cKV is kept position-free, so that the low-rank content path can be shared across t heads and reused under sliding-window re-indexing. Positional information is instead carried by a nope separate RoPE subspace. We split each head as dh = dnope + drope , where kt,h is the reconstructed h h content key and the remaining channels form a decoupled 3D-RoPE key. As in Wan, drope is h partitioned across temporal, height, and width axes, using the corresponding high-frequency rotary bands. For each token, VideoMLA computes a single head-shared positional key rope
ktrope = RoPE3D (ktR ),
ktR = WRK xt ∈ Rdh ,
(5)
rather than nh per-head RoPE keys. The cache stores the unrotated state (cKV , ktR ); rotation is t applied only when the active attention window is assembled. This keeps cached states independent of absolute rollout time and yields a per-token cache size of dc + drope . h The query branch follows the same decomposition. From the query latent cQ t , the positional query for head h is Q Q rope R R qt,h = WR,h ct , qt,h = RoPE3D (qt,h ). (6) Attention is then computed over the concatenated NoPE and RoPE components: each head uses nope rope nope rope (qt,h , qt,h ) against (kt,h , kt ), while values remain reconstructed only from the content latent. 3.3
Training-Time Forward Pass
During training, every video latent token writes its compressed cache state (cKV , ktR ), defined in t Eqs. 1 and 5, into the KV cache as the block under denoising progresses. Attention is then computed in standard multi-head form, with the per-head content keys and values reconstructed on demand from the cached content latent through Eq. 2, and the shared positional key obtained by rotating the cached positional state ktR at use time. For a query token at position i and a cached token at position j, attention head h combines the content and positional contributions into a single score (h) scorei,j
nope nope rope qi,h · kj,h + qi,h · kjrope p = , dnope + drope h h
(7)
nope rope where qi,h and qi,h are the content and rotated positional query components from Eqs. 4 and 6, nope kj,h is the per-head content key from Eq. 2, and kjrope is the rotated shared positional key obtained from Eq. 5. The two inner products live in subspaces of dimension dnope and drope respectively, h h so the joint score is normalized by their combined dimension. A softmax over the active attention
5
Figure 4: Qualitative results. Samples generated by VideoMLA. Frames are shown at uniformly spaced timestamps from each 30s rollout, illustrating that the compressed latent KV cache preserves scene structure, subject identity, and visual fidelity over time. window followed by a weighted sum of vj,h produces the per-head output, and the head outputs are mixed through the output projection W O . (h)
The shape of scorei,j matches what a dense attention layer of the same per-head dimension would produce. As a consequence, VideoMLA substitutes for the dense self-attention module without any change to the surrounding training pipeline: chunkwise causal block masks, sink tokens, and FlexAttention kernels operate on the reconstructed per-head keys and values exactly as they would on dense ones. The only structural change relative to the dense baseline is internal to the attention layer: the cache holds (cKV , ktR ) rather than per-head K and V , and the per-head views consumed t by attention are reconstructed at use time.
4
Experiments
4.1
Setup and Dataset
Implementation Details. We implement VideoMLA on top of the Wan-2.1 T2V-1.3B backbone [21], replacing only the self-attention layers while leaving the remaining architecture unchanged. The model has 30 transformer blocks, hidden dimension 1536, 12 heads, and per-head dimension 128. Unless otherwise stated, we use dc = 192 and dq = 768, with the head dimension split into dnope = 96 and drope = 32. The decoupled 3D-RoPE channels are allocated across temporal, h h height, and width axes as (6, 5, 5) complex pairs, using the highest-frequency bands. This gives a per-token cache size of dc + drope = 224 scalars, corresponding to a 13.7× reduction from the h dense 2nh dh = 3072-scalar KV cache. Training follows the three-stage Causal Forcing pipeline [31], including Teacher Forcing, Consistency Distillation initialization to four steps, and DMD, with total batch size 128. We use learning rates 5×10−6 for Teacher Forcing and 2×10−6 for Consistency Distillation and DMD. All training experiments are run in bf16 mixed precision on a 8 × B200 GPU. Dataset. For the Consistency Distillation stage preceding DMD, we use 47,680 videos: 29,471 from OpenVid-1M [19] and 18,209 synthesized clips. Baselines. We compare VideoMLA with recent causal video diffusion methods covering standard streaming pipelines, attention-architecture redesigns, and positional reparameterizations. The streaming baselines include CausVid [27], Self-Forcing [9], Rolling-Forcing [16], Causal Forcing [31], 6
CausVid
Self Forcing
Rolling Forcing
Causal Forcing
Infinity-RoPE Deep Forcing
Reward Forcing LongLive
VideoMLA
LongSANA
Figure 5: Qualitative comparison. Long-rollout samples from VideoMLA and baseline causal video diffusion baselines under the same prompt. Each row shows uniformly spaced frames from one method. Results on 30s ↑
Model
Results on 60s ↑
User Study ↑
AQ
BC
DD
IQ
MS
SC
Overall
AQ
BC
DD
IQ
MS
SC
Overall
PA
TC
DC
Overall
Self-Forcing [9] CausVid [27] Causal Forcing [31] Rolling-Forcing [16] Deep Forcing [25] Reward Forcing [17] LongLive [22] Infinity-RoPE [24]
0.541 0.597 0.526 0.620 0.621 0.644 0.654 0.640
0.948 0.921 0.945 0.953 0.953 0.956 0.959 0.958
0.624 0.473 0.738 0.742 0.713 0.954 0.649 0.847
0.577 0.663 0.628 0.688 0.660 0.683 0.678 0.669
0.952 0.935 0.968 0.982 0.979 0.981 0.983 0.982
0.932 0.913 0.947 0.960 0.961 0.957 0.967 0.966
0.762 0.750 0.792 0.824 0.815 0.863 0.816 0.844
0.565 0.497 0.503 0.580 0.597 0.585 0.606 0.607
0.958 0.929 0.936 0.958 0.957 0.952 0.961 0.959
0.393 0.723 0.847 0.380 0.402 0.676 0.433 0.647
0.650 0.574 0.608 0.670 0.690 0.673 0.664 0.638
0.987 0.948 0.935 0.988 0.987 0.985 0.991 0.988
0.974 0.933 0.920 0.977 0.979 0.974 0.982 0.979
0.755 0.767 0.792 0.759 0.769 0.808 0.773 0.803
2.79 – 2.59 2.55 2.60 2.91 2.56 2.46
2.79 – 2.63 2.68 2.76 2.99 2.70 2.44
2.70 – 2.81 2.60 2.68 2.83 2.58 2.41
2.76 – 2.68 2.61 2.68 2.91 2.61 2.43
LongSANA [3] VideoMLA (Ours)
0.573 0.976 0.149 0.683 0.974 0.988 0.601 0.942 0.981 0.697 0.986 0.952
0.723 0.859
0.529 0.976 0.103 0.702 0.991 0.986 0.569 0.963 0.958 0.715 0.993 0.954
0.714 0.859
2.48 2.63 2.56 3.04 3.24 3.22
2.56 3.17
Table 2: Long-horizon performance and user preference comparison. Results across 30s and 60s video generation, plus user study scores. AQ: Aesthetic Quality, BC: Background Consistency, DD: Dynamic Degree, IQ: Imaging Quality, MS: Motion Smoothness, SC: Subject Consistency. User study metrics are PA: Prompt Adherence, TC: Temporal Consistency, and DC: Dynamic Consistency. Bold: best; underline: second best. Reward Forcing [17], Deep Forcing [25], and LongLive [22] and Infinity-RoPE[24]. We also compare with architectural efficiency method LongSANA [3]. 4.2
Main Results
Qualitative Results. Figure 4 shows that VideoMLA preserves subject identity, scene structure, and visual fidelity over 30-second rollouts despite replacing the dense per-head KV cache with a compact latent cache. Finally, Figure 5 shows that VideoMLA generates results comparable to representative streaming causal video baselines while requiring faster inference and substantially lower memory. These qualitative results indicate that VideoMLA improves the efficiency–memory trade-off without the pronounced fidelity, dynamism, or long-horizon stability losses observed in more aggressive compression-based alternatives. Quantitative Results. Table 2 reports long-horizon VBench results at 30s and 60s. VideoMLA achieves the best dynamic degree at both horizons, with 0.981 at 30s and 0.958 at 60s, indicating that latent KV compression does not suppress motion or lead to static generation. It also obtains the best imaging quality and motion smoothness, and reaches the highest 60s overall score of 0.859, substantially outperforming prior streaming baselines such as Reward Forcing, Infinity-RoPE, LongLive, and LongSANA. At 30s, VideoMLA is also competitive with the strongest baseline, achieving the second-best overall score while using a much smaller KV cache memory. 7
VideoMLA (Ours)
LongSANA
1 Minute Horizon
Figure 6: Long-horizon generation quality. Frames sampled across a one-minute rollout of the same prompt. (Bottom) VideoMLA sustains visual fidelity with diverse, evolving motion, while (Top) LongSANA produces near-static content that degrades over time. VideoMLA yields higher visual fidelity and more diverse motion while achieving higher generation throughput and lower latency than LongSANA, and reduces KV cache size by 92.7% relative to the Self-Forcing baseline. Throughput ↑
Latency ↓
CLIP-T ↑
CLIP-F ↑
HPSv3 ↑
Frame-wise autoregressive models NOVA [5] 0.6B 768×480 640×384 Pyramid Flow [11] 2B
2.26 1.39
14.63 87.32
0.2764 0.2888
0.9673 0.9795
2.95 8.02
Chunk-wise autoregressive models Self-Forcing [9] 1.3B 832×480 LongSANA [3] 2B 832×480 VideoMLA (Ours) 1.3B 832×480
18.06 19.35 23.96
4.19 4.48 3.38
0.3036 0.2978 0.3278
0.9689 0.9887 0.9686
9.86 7.54 9.74
Model
#Params
Resolution
Table 3: Text-to-Video quantitative comparison on VBench. Models have similar parameter sizes and resolutions. Throughput ↑ (FPS) and latency ↓ (s) measured with batch size 1 on B200. Higher is better for CLIP-T, CLIP-F, and HPSv3 scores ↑. Bold: best; underline: second best.
Efficiency Results. Table 3 shows that VideoMLA achieves the highest throughput and lowest latency among chunk-wise autoregressive models, while also obtaining the best CLIP-T score. Although LongSANA has a slightly higher CLIP-F score, this is partly due to its more static generations, which preserve frame-level similarity but reduce motion dynamics. Consistently, VideoMLA obtains a higher HPSv3 score and, as shown in Figure 6, produces sharper, more dynamic, and more temporally stable long-rollout videos than LongSANA. 4.3
Ablations
Batch Scaling Under Fixed Memory. Fig. 7 shows that VideoMLA translates cache compression into practical serving headroom on a single B200. Dense MHA reaches the memory limit at B = 28, whereas MLA shifts the OOM cliff far to the right; with dc = 64, it remains within budget even at B = 320. The per-request memory slope drops from 6.26 GB/batch for MHA to 0.57–1.43 GB/batch for MLA, a 77–91% reduction across dc ∈ {64, 128, 192, 256, 512}. Consequently, MLA supports 4.6× to at least 11.4× larger non-OOM batches under the same memory cap, with our default dc = 192 giving 8.0× batch headroom.
5
Why MLA Works in Video Diffusion: Rank Budget vs. Spectral Structure
MLA is often motivated by the assumption that the pretrained key/value maps are approximately low-rank. We test whether this explanation holds for video diffusion by analyzing the joint dense 8
(a) Peak memory vs. batch size
(c) Practical serving headroom
MHA / SF MLA dc = 64 MLA dc = 128 MLA dc = 192 MLA dc = 256 MLA dc = 512
150 125 100 75 50 25 0
white labels: aggregate fps
7
350
6.26
6 5 4 3 2 0.57 −91%
1
0.69 −89%
0.82 −87%
0.94 −85%
1.43 −77%
Maximum non-OOM batch
175
Per-batch memory slope (GB)
Peak GPU memory (GB)
(b) Per-request memory growth
B200 cap
200
300 9.1×
250
2
4
8
16
32
64
128
MHA
Batch size
64
128
192
256
6.9×
150
4.6×
100 50 0
256
8.0×
200
0 1
≥ 11.4×
512
B=28 27.8
19.0
19.5
19.3
19.1
19.1
MHA
64
128
192
256
512
MLA dc (MHA baseline at left)
MLA dc (MHA baseline at left)
Figure 7: VideoMLA increases serving headroom under a fixed B200 memory budget. Compared with dense MHA, MLA greatly reduces per-batch memory growth and shifts the OOM limit to much larger batch sizes; the default dc = 192 gives 8.0× non-OOM batch headroom. (a) Rank budget is saturated from initialization 100.0
191 SVD init Random init
190 189 188 187
0.40
99.5 99.0 98.5
98.4
98.0 97.5
97.4
SVD init Random init
0.30
0.25
0.20
96.5 2
0.35
97.0
step 0 = init
0
(c) Spectral shape changes little during Stage-1
full budget
Median σdc /σ1
Median r0.99 /dc (%)
Mean effective rank r0.99
(b) Both initializations use nearly all dc = 192
dc = 192
192
4
6
Training checkpoint step (k)
8
9.5
0
2
4
6
8
9.5
Training checkpoint step (k)
0
2
4
6
8
9.5
Training checkpoint step (k)
Figure 8: Rank-budget saturation during training. At dc = 192, both SVD and random initialization occupy nearly the full latent rank budget from initialization, with stable effective rank and spectral tail throughout training. operator [WK ; WV ] in Wan2.1-T2V-1.3B. Fig. 1 shows that this operator is not low-rank: at the default budget dc = 192, the median layer preserves only 45.8% of the spectral energy, and the 99%-energy effective rank exceeds 1300 in every layer. Thus, a direct rank-dc spectral approximation would discard most of the dense key/value energy, even though VideoMLA retains generation quality at this cache size. This mismatch suggests that MLA should not be interpreted as recovering a hidden low-rank structure in the pretrained attention weights. Instead, MLA changes the optimization problem: the composed key/value operator M = [W↑K W↓KV ; W↑V W↓KV ] is constrained by construction to have rank at most dc . Fig. 2 confirms that the learned composed operator uses this architectural budget almost fully across latent sizes. For dc ∈ {64, 128, 256, 512}, the normalized spectra share a common shape truncated at dc , and the layer-wise 99%-energy rank remains close to 0.98dc throughout the network. The effective rank is therefore set by the MLA bottleneck rather than by the spectrum of the original dense operator. We further investigate whether this is an artifact of SVD initialization. Fig. 8 compares SVD and random initialization at dc = 192 during training. Both nearly saturate the rank budget from initialization, and training preserves the effective rank and spectral tail. Thus, training does not discover a lower-rank solution or collapse the spectrum; it adapts within the imposed budget.
6
Limitations and Broader Impact
VideoMLA reduces per-token KV cache, but the latent budget cannot shrink arbitrarily. Small budgets such as dc = 64 improve memory headroom but lose fine-grained details and degrade quality, making dc a quality–efficiency trade-off. Our experiments focus on Wan2.1-T2V-1.3B and minute-scale generation; larger backbones, higher resolutions, longer horizons, and prompt switching remain future work. More efficient long-horizon generation can reduce deployment cost and broaden access to creative tools, simulation, education, and assistive media production. 9
7
Conclusion
We presented VideoMLA, the first MLA-style latent KV cache for autoregressive video diffusion. By replacing dense per-head keys and values with a shared low-rank content latent and a head-shared decoupled 3D-RoPE positional key, VideoMLA reduces per-token KV cache memory by 92.7% while preserving compatibility with standard chunk-causal generation. Our analysis shows that this success does not arise from an intrinsically low-rank pretrained key-value operator; instead, the MLA bottleneck defines a rank budget that the model uses nearly fully and adapts within during training. Empirically, VideoMLA preserves visual quality and motion at long horizons, achieves the best one-minute overall score among evaluated methods, and improves throughput with substantially lower cache memory. These results identify the per-token KV layout as an effective and complementary axis for scaling efficient long-horizon video diffusion.
Acknowledgements Pinar Yanardag is supported by the National Science Foundation under Grant No. 2543524.
References [1] Ainslie, J., Lee-Thorp, J., De Jong, M., Zemlyanskiy, Y., Lebrón, F., Sanghai, S.: Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 4895–4901 (2023) [2] Bai, X., He, G., Li, Z., Shechtman, E., Huang, X., Wu, Z.: Causality in video diffusers is separable from denoising. arXiv preprint arXiv:2602.10095 (2026) [3] Chen, J., Zhao, Y., Yu, J., Chu, R., Chen, J., Yang, S., Wang, X., Pan, Y., Zhou, D., Ling, H., et al.: Sana-video: Efficient video generation with block linear diffusion transformer. arXiv preprint arXiv:2509.24695 (2025) [4] Cui, J., Wu, J., Li, M., Yang, T., Li, X., Wang, R., Bai, A., Ban, Y., Hsieh, C.J.: Self-forcing++: Towards minute-scale high-quality video generation. arXiv preprint arXiv:2510.02283 (2025) [5] Deng, H., Pan, T., Diao, H., Luo, Z., Cui, Y., Lu, H., Shan, S., Qi, Y., Wang, X.: Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412.14169 (2024) [6] Deng, K., Woodland, P.C.: arXiv:2505.13544 (2025)
Multi-head temporal latent attention. arXiv preprint
[7] Gao, J., Chen, Z., Liu, X., Feng, J., Si, C., Fu, Y., Qiao, Y., Liu, Z.: Longvie: Multimodal-guided controllable ultra-long video generation. arXiv preprint arXiv:2508.03694 (2025) [8] Henschel, R., Khachatryan, L., Poghosyan, H., Hayrapetyan, D., Tadevosyan, V., Wang, Z., Navasardyan, S., Shi, H.: Streamingt2v: Consistent, dynamic, and extendable long video generation from text. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 2568–2577 (2025) [9] Huang, X., Li, Z., He, G., Zhou, M., Shechtman, E.: Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08009 (2025) [10] Ji, T., Guo, B., Wu, Y., Guo, Q., Shen, L., Chen, Z., Qiu, X., Zhang, Q., Gui, T.: Towards economical inference: Enabling deepseek’s multi-head latent attention in any transformerbased llms. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 33313–33328 (2025) [11] Jin, Y., Sun, Z., Li, N., Xu, K., Jiang, H., Zhuang, N., Huang, Q., Song, Y., Mu, Y., Lin, Z.: Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954 (2024) [12] Kim, Y., Hu, Q., Kuo, C.C.J., Beerel, P.A.: Memrope: Training-free infinite video generation via evolving memory tokens. arXiv preprint arXiv:2603.12513 (2026) 10
[13] Li, H., Liu, S., Lin, Z., Chandraker, M.: Rolling sink: Bridging limited-horizon training and open-ended testing in autoregressive video diffusion. arXiv preprint arXiv:2602.07775 (2026) [14] Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al.: Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 (2024) [15] Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al.: Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024) [16] Liu, K., Hu, W., Xu, J., Shan, Y., Lu, S.: Rolling forcing: Autoregressive long video diffusion in real time. arXiv preprint arXiv:2509.25161 (2025) [17] Lu, Y., Zeng, Y., Li, H., Ouyang, H., Wang, Q., Cheng, K.L., Zhu, J., Cao, H., Zhang, Z., Zhu, X., et al.: Reward forcing: Efficient streaming video generation with rewarded distribution matching distillation. arXiv preprint arXiv:2512.04678 (2025) [18] Meng, F., Tang, P., Tang, X., Yao, Z., Sun, X., Zhang, M.: Transmla: Multi-head latent attention is all you need. arXiv preprint arXiv:2502.07864 (2025) [19] Nan, K., Xie, R., Zhou, P., Fan, T., Yang, Z., Chen, Z., Li, X., Yang, J., Tai, Y.: Openvid-1m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02371 (2024) [20] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017) [21] Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.W., Chen, D., Yu, F., Zhao, H., Yang, J., et al.: Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 (2025) [22] Yang, S., Huang, W., Chu, R., Xiao, Y., Zhao, Y., Wang, X., Li, M., Xie, E., Chen, Y., Lu, Y., et al.: Longlive: Real-time interactive long video generation. arXiv preprint arXiv:2509.22622 (2025) [23] Yang, Y., Zhang, T., Huang, W., Chen, J., Wu, B., He, X., Cai, D., Li, B., Jiang, P.T.: Anchor forcing: Anchor memory and tri-region rope for interactive streaming video diffusion. arXiv preprint arXiv:2603.13405 (2026) [24] Yesiltepe, H., Meral, T.H.S., Akan, A.K., Oktay, K., Yanardag, P.: Infinity-rope: Actioncontrollable infinite video generation emerges from autoregressive self-rollout. arXiv preprint arXiv:2511.20649 (2025) [25] Yi, J., Jang, W., Cho, P.H., Nam, J., Yoon, H., Kim, S.: Deep forcing: Training-free long video generation with deep sink and participative compression. arXiv preprint arXiv:2512.05081 (2025) [26] Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Durand, F., Freeman, B.: Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems 37, 47455–47487 (2024) [27] Yin, T., Zhang, Q., Zhang, R., Freeman, W.T., Durand, F., Shechtman, E., Huang, X.: From slow bidirectional to fast autoregressive video diffusion models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 22963–22974 (2025) [28] Yu, Y., Wu, X., Hu, X., Hu, T., Sun, Y., Lyu, X., Wang, B., Ma, L., Ma, Y., Wang, Z., et al.: Videossm: Autoregressive long video generation with hybrid state-space memory. arXiv preprint arXiv:2512.04519 (2025) [29] Zhang, L., Agrawala, M.: Packing input frame context in next-frame prediction models for video generation. arXiv e-prints pp. arXiv–2504 (2025) [30] Zhao, Z., Lu, Y., Liu, Z., Song, J., Deng, J., Patras, I.: Relax forcing: Relaxed kv-memory for consistent long video generation. arXiv preprint arXiv:2603.21366 (2026) 11
[31] Zhu, H., Zhao, M., He, G., Su, H., Li, C., Zhu, J.: Causal forcing: Autoregressive diffusion distillation done right for high-quality real-time interactive video generation. arXiv preprint arXiv:2602.02214 (2026)
12
Table of Contents A Videos and Website
1
B Details on User Study
1
C Background
1
C.1 Wan2.1-T2V-1.3B Backbone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D Implementation Details
1 3
D.1 Backbone and Tokenization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
D.2 VideoMLA Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
D.3 NoPE/RoPE Split and 3D RoPE . . . . . . . . . . . . . . . . . . . . . . . . . . .
4
D.4 Chunk-Causal Sliding-Window Attention . . . . . . . . . . . . . . . . . . . . . .
4
D.5 Long-Horizon RoPE Re-indexing . . . . . . . . . . . . . . . . . . . . . . . . . .
5
D.6 Training Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
E Inference-Time Reparameterization
5
F Additional Ablations
7
A
Videos and Website
To facilitate comprehensive evaluation and improve result accessibility, we provide video results covering qualitative examples, ablation studies, comparisons, and limitations in the https:// videomla.github.io.
B
Details on User Study
We conduct a user study to evaluate perceptual quality of one-minute generations. We compare nine models and ask 50 participants to rate each video using the interface shown in Fig. 9. For each generated video, participants answer three questions: Prompt Adherence, measuring how well the video follows the prompt; Temporal Consistency, measuring whether the video remains coherent from start to end; and Dynamic Consistency, measuring whether the video contains plausible and sustained motion. Each question is rated on a five-point Likert scale, from 1) Very Bad to 5) Very Good.
C
Background
C.1
Wan2.1-T2V-1.3B Backbone
Our experiments use Wan2.1-T2V-1.3B as the base video diffusion backbone. Wan2.1-T2V-1.3B is a latent video diffusion transformer operating over spatiotemporal latent tokens rather than RGB pixels. The video is encoded by a 3D VAE that compresses the temporal dimension by 4× and each spatial dimension by 8×, so an input video V ∈ RF ×H×W ×3 is mapped to a latent tensor with temporal length 1 + ⌈(F − 1)/4⌉ and spatial resolution H/8 × W/8. The denoising model follows the rectified-flow formulation, where a clean latent x0 and Gaussian noise ϵ are linearly interpolated as xt = (1 − t)x0 + tϵ, t ∈ [0, 1], and the reverse process is parameterized by a neural velocity field and solved with Euler integration at inference time. 1
Figure 9: User Study Interface. User Study Interface for Long Video Generation
Wan2.1-T2V-1.3B uses a diffusion transformer with multi-head self-attention over video latent tokens. In our implementation, the backbone contains 30 transformer blocks, hidden dimension d = 1536, nh = 12 attention heads, and per-head dimension dh = 128. In the dense baseline, each cached token stores both keys and values for all heads, giving a per-token, per-layer KV cache size of 2nh dh = 2 × 12 × 128 = 3072 scalars. With a 21-latent-frame cache, 1,560 tokens per latent frame, and 30 cached transformer layers, this corresponds to 3.02B cached scalars, or approximately 6.0GB in bf16/fp16. This dense per-head KV layout is the main memory target of VideoMLA. The backbone uses 3D rotary position embeddings (3D-RoPE) to encode temporal and spatial token coordinates before self-attention. For latent features x ∈ RB×S×C with S = F HW , the channel dimension is partitioned across temporal, height, and width axes, and RoPE is applied separately to the corresponding coordinate subspaces before concatenation. In Wan, each RoPE dimension has a fixed maximum sequence length of 1024; although RoPE remains mathematically defined beyond this range, generation outside the positional regime observed during training can degrade attention quality. For autoregressive long-video generation, Wan2.1-T2V-1.3B is commonly used after causal distillation or self-rollout training. The model generates latent frame chunks sequentially and conditions each chunk on a rolling KV cache of previous chunks. This cache enables efficient streaming generation because previous key and value states are reused rather than recomputed. However, in the dense Wan attention layout, the cache still stores full per-head keys and values for every retained token and every cached layer. VideoMLA keeps the Wan backbone and causal rollout setting intact, but replaces this dense per-token KV state with a shared latent content cache and a decoupled head-shared 3D-RoPE key. 2
Table 4: Training and model hyperparameters. Unless otherwise stated, all experiments use the default VideoMLA setting. Hyperparameter
Value
Backbone Transformer blocks Hidden dimension d Number of heads nh Per-head dimension dh KV latent dimension dc Query latent dimension dq NoPE channels dnope h RoPE channels drope h 3D-RoPE complex pairs (t, h, w) Per-token cache size KV cache reduction Training precision Training GPUs Total batch size Teacher Forcing LR Consistency Distillation LR DMD LR Training pipeline Inference steps
D
Implementation Details
D.1
Backbone and Tokenization
Wan2.1-T2V-1.3B 30 1536 12 128 192 768 96 32 (6, 5, 5) 224 scalars 92.7% bf16 8× B200 128 5 × 10−6 2 × 10−6 2 × 10−6 TF → CD → DMD 4
VideoMLA is implemented on top of Wan2.1-T2V-1.3B. The backbone contains L = 30 transformer blocks, hidden dimension d = 1536, nh = 12 attention heads, and per-head dimension dh = 128, so that d = nh dh . The feed-forward hidden dimension is 8960. We keep the Wan text-conditioning branch and all non-attention modules unchanged, and replace only the temporal self-attention layers with the VideoMLA block described in Section 3. We train on 5-second clips at 480 × 832 resolution and 16 fps. The Wan VAE encodes each clip into a latent tensor with 16 channels, 21 latent frames, and spatial size 60 × 104. A 3D patch embedding with patch size (1, 2, 2) maps each latent frame into 30 × 52 = 1560 visual tokens. Thus, each 5-second clip contains 21 × 1560 = 32760 self-attention tokens. Autoregressive generation is performed in chunks of 3 latent frames. D.2
VideoMLA Block
Each temporal self-attention layer follows the notation of Section 3. Given an attention input xt ∈ Rd , VideoMLA first forms a shared content cache latent cKV = W↓KV xt ∈ Rdc , t and a query latent In the default model, we use dc = 192,
Q dq cQ t = W↓ xt ∈ R .
dnope = 96, h
dq = 768,
drope = 32, h
with dnope + drope = dh = 128. The down-projection shapes are h h W↓Q ∈ R768×1536 .
W↓KV ∈ R192×1536 ,
Both cKV and cQ t are normalized by RMSNorm before the corresponding up-projections. t The content key and value for head h are reconstructed from the shared cache latent: nope K KV kt,h = W↑,h ct ,
V KV vt,h = W↑,h ct .
3
Aggregating all heads, the projection shapes are nope
W↑K ∈ R(nh dh
)×dc
= R1152×192 ,
W↑V ∈ R(nh dh )×dc = R1536×192 . The NoPE query is reconstructed analogously: nope Q Q qt,h = W↑,h ct ,
nope
W↑Q ∈ R(nh dh
)×dq
= R1152×768 .
The RoPE branch is decoupled from the content cache. For each token, VideoMLA computes a single head-shared positional key rope
ktR = WRK xt ∈ Rdh , and per-head positional queries Q Q R qt,h = WR,h ct ,
WRK ∈ R32×1536 , rope
WRQ ∈ R(nh dh
)×dq
= R384×768 .
The output projection is the original attention output projection WO ∈ Rd×(nh dh ) = R1536×1536 . Therefore, each cached token stores only rope
(cKV , ktR ) ∈ Rdc +dh , t rather than dense per-head keys and values. With the default setting, this is 192+32 = 224 scalars per token per layer, compared with 2nh dh = 2 · 12 · 128 = 3072 scalars for dense MHA, corresponding to a 13.7× cache reduction. D.3
NoPE/RoPE Split and 3D RoPE
Each head is split as
dh = dnope + drope , h h where the NoPE subspace is used for content matching and the RoPE subspace is used for positionaware matching. The per-head query and key are nope rope nope rope qt,h = [qt,h ; qt,h ], kt,h = [kt,h ; kt ], where
rope R qt,h = RoPE3D (qt,h ), ktrope = RoPE3D (ktR ). The positional key is shared across heads, while the NoPE keys, NoPE queries, and values remain head-specific after up-projection.
For the default split drope = 32, the RoPE subspace contains drope /2 = 16 complex frequency pairs. h h Following the Wan 3D-RoPE factorization, these pairs are allocated across temporal, height, and width axes as (6, 5, 5), using the highest-frequency bands from the corresponding axis groups. D.4
Chunk-Causal Sliding-Window Attention
VideoMLA preserves the chunk-causal attention pattern used by the autoregressive Wan backbone. Tokens within the same 3-latent-frame chunk can attend to one another, while tokens in a later chunk cannot be attended to by earlier chunks. For long-horizon generation, attention is restricted to a fixed cache consisting of one sink latent frame and the most recent six latent frames. Since each latent frame contains 1560 tokens, the sink occupies 1560 cached token slots and the local window occupies 6 × 1560 token slots. During training, the same chunk-causal and sliding-window structure is enforced with block-sparse attention masks. During inference, the cache stores rope
cKV ∈ RB×Tcache ×dc , k R ∈ RB×Tcache ×dh , where Tcache is the number of cached tokens in the sink-plus-window context. When the cache is full, tokens outside the sink are evicted in FIFO order. The attention computation reads the active cache, reconstructs the content keys and values through W↑K and W↑V , applies 3D-RoPE to the active positional keys, and evaluates the standard per-head attention scores from Eq. (7). 4
D.5
Long-Horizon RoPE Re-indexing
For rollouts beyond the 21 latent frames seen during 5-second training, cached positional keys are stored before RoPE is applied. When an attention window is assembled, the active cached keys are assigned local temporal coordinates inside the current sink-plus-window context and are then rotated by RoPE3D following [24]. The current query chunk is rotated in the same local coordinate system. Thus, both queries and cached keys use a bounded, window-relative positional frame even after cache eviction. This re-indexing keeps the RoPE phase within the short-horizon regime observed by the backbone during training, while allowing the generated video to extend beyond the original 21-latent-frame clip length. All reported long-video rollouts use one sink latent frame, a six-latent-frame local window, and the 4-step student sampler. D.6
Training Pipeline
We train the same VideoMLA architecture in three stages on 8 NVIDIA B200 GPUs, using FSDP full sharding, bf16 mixed precision, AdamW with β1 = 0 and β2 = 0.999, and the rectified-flow denoising objective. Stage 1: Teacher Forcing. We initialize the MLA projections from an SVD-style decomposition of the pretrained Wan dense attention matrices at the target configuration (dc , dq , dnope , drope ) = (192, 768, 96, 32). h h The model is then trained as a chunk-causal flow-matching student with clean previous-block context from the teacher-encoded latents. We use learning rate 5 × 10−6 , per-GPU batch size 1, total batch size 2, gradient checkpointing, 1000 training timesteps, and timestep shift 5.0. Stage 2: Consistency Distillation. Starting from the Stage-1 checkpoint, we distill the model to a 4-step sampling schedule [1000, 750, 500, 250]. We use timestep shift 5.0 and classifier-free guidance scale 3.0. The generator learning rate is 2 × 10−6 , the critic learning rate is 4 × 10−7 , and the total batch size is 2 with gradient checkpointing enabled. Stage 3: Distribution Matching Distillation. Finally, we initialize from the Stage-2 checkpoint at iteration 2500 and fine-tune with distribution matching distillation on the same 4-step schedule. The real score is provided by the frozen teacher and the fake score is learned online. We use five critic updates per generator update, EMA weight 0.99 starting from step 1, generator learning rate 2 × 10−6 , critic learning rate 4 × 10−7 , guidance scale 3.0, and timestep shift 5.0. The total batch size is 16, obtained with per-GPU batch size 8 on 8 GPUs and gradient accumulation of 2. The Stage-3 checkpoint is used for all reported VideoMLA results, including the long-horizon evaluations.
E
Inference-Time Reparameterization
The training-time formulation in Section 3 is written to make the connection to standard multi-head attention explicit: each cached token stores (cKV , kjR ), and the per-head NoPE keys and values are j reconstructed through Eq. 2 before applying the usual attention computation. This is convenient during training because it allows VideoMLA to reuse the same block-causal masks, sink-token logic, nope and attention kernels as the dense baseline. At inference, however, explicitly reconstructing kj,h and vj,h would partially undo the benefit of latent caching by materializing dense per-head tensors after every cache read. We therefore use an equivalent reparameterization that keeps the cache and the attention computation in latent form. For the score computation, the only contribution of the reconstructed NoPE key is through its inner product with the reconstructed NoPE query. Substituting Eqs. 3, 4, and 2 into the content term of 5
Eq. 7 gives ⊤ nope nope Q Q K KV qi,h · kj,h = W↑,h ci W↑,h cj ⊤ ⊤ Q K KV = cQ W W↑,h cj i ↑,h ⊤ = cQ Ah cKV , j i where
⊤ Q K Ah = W↑,h W↑,h ∈ Rdq ×dc .
(8)
(9)
The matrix Ah depends only on learned parameters and is independent of the current sequence, cache contents, diffusion timestep, and rollout position. It can therefore be precomputed once when the model is loaded. During inference, the NoPE content score for head h is computed directly from the nope nope KV query latent cQ , without forming either qi,h or kj,h as explicit i and the cached content latent cj per-head vectors. The value path admits an analogous absorption. Let WhO denote the slice of the output projection applied to the output of head h. Using Eq. 2, V KV WhO vj,h = WhO W↑,h cj
= Bh cKV , j
(10)
where V Bh = WhO W↑,h .
(11)
Thus, the value up-projection can also be folded into the output mixer. In practice, after the attention weights for head h are computed, the weighted sum can be accumulated over the cached latents cKV j and then projected by Bh , rather than first reconstructing all dense values vj,h and then applying the output projection. The RoPE branch is kept separate from this absorption. The cache stores the unrotated, head-shared positional key kjR from Eq. 5. When the active attention window is assembled, kjR is rotated by RoPE3D (·) using the current window indexing, and the positional score term in Eq. 7 is computed as rope qi,h · kjrope .
(12)
This separation is important because RoPE is position-dependent and cannot be folded into a fixed parameter matrix in the same way as the NoPE content projections. Storing kjR unrotated also preserves the ability to re-index cached tokens within a sliding window, as described in Section 3.2. After reparameterization, the inference-time cache is never expanded into dense per-head keys and values. Each cached token contributes only a content latent cKV ∈ Rdc and a head-shared positional j rope key kjR ∈ Rdh . Therefore, the per-token cached state remains dc + drope , h
(13)
2nh dh .
(14)
instead of the dense baseline cost For an attention window of size W , cache memory traffic is reduced from
to
O(W 2nh dh )
(15)
O(W (dc + drope )) . h
(16)
With the default configuration dc = 192 and drope = 32, this corresponds to 224 cached scalars h per token per layer, compared with 2nh dh = 3072 scalars for dense MHA. The reparameterization therefore preserves the mathematical attention computation of the training-time formulation while ensuring that the inference-time implementation realizes the intended latent-cache memory and bandwidth savings. 6
(a) Latent dimension dc dc 64 128 256 512
(b) NoPE/RoPE split
Semantic↑ Quality↑ Total↑ Mem.↓ FPS↑ 77.42 82.16 82.74 82.41
79.18 84.31 84.58 84.39
78.30 83.24 83.66 83.40
32.00× 19.20× 10.67× 5.65×
26.93 27.00 26.93 26.79
dnope h
drope h
112 64 32 96
16 64 96 32
Semantic↑ Quality↑ Total↑ 74.62 79.54 75.88 83.02
78.31 82.12 80.74 84.76
76.47 80.83 78.31 83.89
Table 5: Ablation studies. Left: sweep over latent KV dimension dc . Right: decoupled RoPE dimension ablation with dnope + drope = 128. Mem.: KV cache compression ratio relative to dense h h per-token KV cache. FPS: throughput at batch size 1 on 1×H100 80 GB.
F
Additional Ablations
Table 5 studies two architectural choices: the latent KV dimension dc and the NoPE/RoPE channel split. The latent dimension controls the main quality–efficiency trade-off. At dc = 64, VideoMLA gives the largest cache compression and memory headroom, but the budget is too restrictive: both semantic and quality scores drop, consistent with the loss of fine-grained visual details under overly aggressive compression. Increasing to dc = 128 largely recovers quality while retaining a large compression ratio. Further increasing to dc = 256 or 512 gives only marginal gains, but substantially reduces the memory advantage. This suggests that the useful operating regime is not the largest possible latent dimension, but the smallest budget that preserves task-relevant video features. The NoPE/RoPE split also has a clear effect. With only 16 RoPE channels, positional capacity is too limited, leading to weak temporal and spatial anchoring. Conversely, the RoPE-heavy 32/96 split leaves too little capacity for cached content and hurts semantic fidelity. The balanced 64/64 setting improves over these extremes but remains below the content-heavy default. The best result comes from the 96/32 split, indicating that streaming video benefits from allocating most channels to the cached content path while retaining a smaller dedicated RoPE subspace for positional structure.
7