Xiang Fan1 Yuheng Wang1 Bohan Fang1 Zhongzheng Ren12 Ranjay Krishna1 1 University of Washington 2 University of North Carolina at Chapel Hill # [email protected] https://refdecoder.github.io/ Traditional Video Gen VAE Encoder + Video DiT
VAE Decoder
RᴇꜰDᴇᴄᴏᴅᴇʀ VAE Encoder + Video DiT
Reference Encoder (high-fidelity)
RefDecoder
arXiv:2605.15196v1 [cs.CV] 14 May 2026
R EF D ECODER: Enhancing Visual Generation with Conditional Video Decoding
Figure 1: (a) R EF D ECODER improves video VAE decoders by conditioning on a high-fidelity reference signal that bypasses the lossy VAE latent round trip. Given the encoded latents, the decoder injects fine-grained details that are not preserved in the VAE latent space, improving reconstruction fidelity and downstream generation quality. (b) R EF D ECODER integrates with existing video generation pipelines (Wan 2.1 shown) without requiring costly re-training of the diffusion model.
Abstract Video generation powers a vast array of downstream applications. However, while the de facto standard, i.e., latent diffusion models, typically employ heavily conditioned denoising networks, their decoders often remain unconditional. We observe that this architectural asymmetry leads to significant loss of detail and inconsistency relative to the input image. To address this, we argue that the decoder requires equal conditioning to preserve structural integrity. We introduce R EF D ECODER, Preprint.
a reference-conditioned video VAE decoder by injecting high-fidelity reference image signal directly into the decoding process via reference attention. Specifically, a lightweight image encoder maps the reference frame into the detail-rich high-dimensional tokens, which are co-processed with the denoised video latent tokens at each decoder up-sampling stage. We demonstrate consistent improvements across several distinct decoder backbones (e.g., Wan 2.1 and VideoVAE+), achieving up to +2.1 dB PSNR over the unconditional baselines on the Inter4K, WebVid, and Large Motion reconstruction benchmarks. Notably, R EF D ECODER can be directly swapped into existing video generation systems without additional fine-tuning, and we report across-the-board improvements in subject consistency, background consistency, and overall quality scores on the VBench I2V benchmark. Beyond I2V, R EF D ECODER generalizes well to a wide range of visual generation tasks such as style transfer and video editing refinement.
1
Introduction
In the dominant paradigm of video generation, a diffusion backbone denoises latent representations guided by an input conditioning, while a VAE decoder subsequently maps these latents back to pixel space [2, 39, 51]. While substantial effort has been devoted to improving the conditioning and architecture of the diffusion backbone [28, 13, 31], the VAE decoder has received comparatively little attention and remain unconditional in mainstream video diffusion models. It is typically treated as a standalone reconstruction module that operates without access to the primary conditioning signal at inference time [2, 39, 52, 17]. We observe that this architectural asymmetry creates a systematic bottleneck: even when the diffusion model faithfully preserves the reference information within the latent, the decoder must reconstruct fine-grained spatial details from a heavily compressed representation without a detailed anchor. This leads to two characteristic failure modes: (1) Progressive degradation of spatial details, where textures, edges, and high-frequency content deteriorate in frames that deviate from the conditioning image (e.g. Figure 3 baseline details); and (2) Temporal inconsistency, where appearance drifts across the sequence (e.g. Figure 4(a) human face). These artifacts emerge during latent-topixel decoding rather than the diffusion process itself, exposing the decoder as an under-exploited component that fails to leverage the visual cues already provided by the reference. However, injecting the reference image into the decoder is non-trivial for several reasons. (1) Because the decoder operates through hierarchical up-sampling stages, it remains an open question at which specific stage the reference signal should be introduced. (2) The conditioning mechanism must maintain compatibility with pretrained decoder weights, ensuring the system remains functional in its original setting, i.e., no reference image. (3) The approach should be sufficiently generic to generalize across diverse visual generation tasks and various decoder backbones. To address aforementioned problems, our key insight is that the decoder’s hidden space is far richer than the VAE latent space; encoding the reference image into this high-fidelity space and injecting it as additional self-attention tokens lets the decoder exploit details unavailable from the latent code alone. Building on this, we propose R EF D ECODER, a reference-conditioned video VAE decoder with two components: a reference image encoder, a lightweight convolutional network mapping the reference frame into the decoder’s hidden space as spatially aligned tokens (Sec. 2(a)); and a conditional token decoder that, at each up-sampling stage, concatenates reference and video latent tokens along the temporal axis and jointly processes them via a shared transformer block with rotary positional embeddings (RoPE) [33]; the tokens are then separated and up-sampled independently, preserving compatibility with the pretrained decoder (Sec. 2(b)). To prevent the decoder from ignoring the reference, we further introduce a latent token dropout strategy that randomly zeroes video latent tokens at a variable rate, forcing the model to recover spatial details from reference tokens via attention (Sec. 2(c)). Our newly introduced modules are architecture-agnostic and generalize to existing video VAE decoders, with the transformer block shared across decoder stages. Because only the decoder is modified and the encoder remains frozen, R EF D ECODER is a drop-in replacement for any existing video VAE decoder and can immediately benefit downstream diffusion models without retraining. 2
Reference Image
RefDecoder
Dropout
reyaL tuptuO
k c ol B p U
noitnettA
redocnE
ecnerefeR
×N
Output Video
Latent
Attention
k c ol B p U
noitcejorP
drawroF-deeF
noitnettA-fleS
gniddebmE hctaP
×L
Figure 2: Architecture overview of the R EF D ECODER. The reference image is encoded into tokens and injected into the decoder through shared attention blocks, enabling interactions between reference and latent tokens during decoding. We test R EF D ECODER on different video VAE backbones including Wan 2.1 [39] and VideoVAE+ [48], demonstrating consistent improvements across all of them. On the Inter4K reconstruction benchmark, R EF D ECODER achieves over 1 dB PSNR improvement compared to the unconditional baseline. In generation tasks, on the VBench I2V benchmark [14, 15], our method improves subject consistency, background consistency, motion smoothness, aesthetic quality, and overall scores. We show that the same reference injection mechanism naturally extends to decode-time style transfer. By supplying a style image as the reference, R EF D ECODER produces stylized video without any task-specific modification, validating the generality of conditional token decoding. Furthermore, in video editing tasks, R EF D ECODER enhances the fidelity of edited videos to the input reference while preserving the intended edits, demonstrating that reference conditioning as a powerful tool to mitigate the trade-off between editability and fidelity [10]. These results suggest a broader message: in image conditioned generation pipelines, the decoder is not merely a passive reconstruction module but an active participant that can—and should—leverage conditioning signals. Don’t forget the decoder.
2
Method
We propose R EF D ECODER, a reference-conditioned video VAE decoder that injects reference image tokens directly into the decoding process via joint attention over concatenated reference and video tokens, enhancing spatial fidelity and temporal coherence. The overall architecture is illustrated in Fig. 2. We build upon a standard video VAE backbone and augment its decoder with conditional token interactions, which we detail below. Reference image encoding. We start with a reference image Iref ∈ R3×H×W . Typical methods encode the reference into a low-dimensional feature space comparable to the VAE latent bottleneck (e.g. 16 channels), and obtain hierarchical features derived therefrom. To preserve high-fidelity information, we instead project the original image patches directly into a high-dimensional feature space (e.g. 512 dimensions), which is fed to the decoder at the very first stage, upsampled alongside the video features throughout decoding stages, and interacted with through injected attention layers at every stage (Section 2(b)). This setup allows multiresolution high-dimensional feature extraction within the decoding process. We keep the reference encoder minimal—applying a single convolution followed by normalization— to avoid any issue with a deeper encoder smoothing out high-frequency details. 3
H
W
After encoding, we obtain reference tokens zref ∈ RC× p × p , where C is the first decoder stage’s channel dimension and p is the VAE’s spatial compression ratio. This aligns reference tokens with the decoder’s initial feature map, enabling direct position-aware information transfer through attention. Conditional token decoding. Standard video VAE decoders rely on causal 3D convolutional upsampling layers with limited local receptive fields, which cannot recover fine-grained details lost to the narrow latent bottleneck. To let the decoder selectively retrieve details from the reference, we insert Transformer blocks into the upsampling stages of the pretrained decoder, enabling each video token to query reference tokens and extract relevant information. Our conditional token decoding involves three design goals: (1) combining reference and video tokens for joint processing; (2) preserving compatibility with the existing VAE upsampling path; and (3) minimizing parameter overhead. Token concatenation. At decoder stage s, we concatenate video tokens z(s) ∈ RCs ×Ts ×Hs ×Ws and (s) reference tokens zref ∈ RCs ×1×Hs ×Ws along the temporal dimension, yielding a tensor of shape Cs ×(1+Ts )×Hs ×Ws . This is then processed through self attention in a Transformer block. R (s)
Separation and upsampling. After attention, the output is split back into reference tokens ẑref and video tokens ẑ(s) . Each branch is then passed through the pre-trained upsampling modules independently. The reference tokens are upsampled only spatially, while the video tokens undergo both spatial and temporal upsampling. Weight sharing across stages. To reduce the parameter overhead of Transformer blocks, Transformer weights are shared across all stages. Stage-specific patch embedding layers project varying channel dimensions Cs to a unified transformer hidden dimension, enabling weight sharing without architectural constraints. We ablate the effectiveness of weight-shared blocks in Section 3.7. Latent token dropout. To encourage the decoder to actively rely on the reference rather than solely on the video latents, we apply token dropout to the latents before decoding. At each training step, every spatiotemporal position in z is independently zeroed with probability r, where r is sampled uniformly from [0, rmax ), forcing the model to retrieve missing details from the reference tokens via attention. We use rmax = 0.7 by default. Training. We initialize the VAE from a pretrained video VAE and freeze the encoder entirely. The decoder is fine-tuned at a reduced learning rate, while the newly added modules (reference image encoder and Transformer blocks) are trained from scratch. The training objective is: L = ∥x − x̂∥1 + LLPIPS (x, x̂) where x and x̂ denote the ground-truth and reconstructed video frames respectively, and LLPIPS is the LPIPS perceptual loss [59]. Random frame reference selection. While the reference image typically serves as the first frame at inference, the model should ideally be robust to a wider range of reference content. We therefore randomly select a frame from the input video as the reference during training, preventing overfitting to a fixed temporal relationship. Two-stage curriculum training. We start with training on short clips of the minimum supported number of frames and then extend to the maximum frame count to generalize to long temporal contexts.
3
Experiments
We evaluate R EF D ECODER on video reconstruction and image-to-video generation. Throughout, R EF D ECODER is used as a plug-and-play drop-in replacement for the baseline VAE decoder: the encoder, the diffusion backbone, and the inference pipeline remain unchanged. To assess generality, we apply R EF D ECODER to two architecturally distinct backbones, Wan 2.1 and VideoVAE+. We further ablate key design choices including the number of Transformer blocks, latent token dropout, and the training curriculum. 4
Wan 2.1 Wan 2.1 Wan 2.1 VideoVAE+ VideoVAE+ VideoVAE+
Baseline
Ours
Baseline
Ours
Figure 3: Reconstruction comparisons. Cropped regions from two backbones (Wan 2.1 and VideoVAE+) are shown. R EF D ECODER produces sharper reconstructions and better preserves fine details than the baseline, particularly for high-frequency content like text, human faces, and structural patterns.
3.1
Experimental Setup
Training data. Our training set contains approximately 100K videos including videos from MiraData9K [19, 62], DL3DV [19, 62], and a subset of OpenVidHD-0.4M [22]. Benchmarks. For reconstruction, we use the three test sets provided by VideoVAE+ [48]—an Inter4K [32] test split of 500 high-quality videos, a subset of WebVid [1], and a Large Motion subset of 100 videos (80 from WebVid, 20 from Inter4K) curated for complex camera and object motion—using identical splits to ensure direct comparability with [48] (see Section B.5). We report PSNR, SSIM, and LPIPS for reconstruction, and the standard 12 VBench dimensions [14, 15, 61] for generation. Implementation details. We apply R EF D ECODER to two architecturally distinct backbones, Wan 2.1 [39] and VideoVAE+ [48], which differ substantially in their channel widths and decoder depth. We freeze the encoder and train only the decoder, and drop up to 70% latent video tokens during training. We adopt a two-stage curriculum: for Wan 2.1, 5-frame clips at 480 × 832 followed by 17-frame clips; for VideoVAE+, 4-frame clips at 216 × 216 (as does [48]) followed by 16-frame clips at the same resolution. Baselines. We compare our method with the vanilla VAE autoencoders from frontier open source video models, including Wan 2.1 [39] and Hunyuan 1.5 [17], as well as state-of-the-art academic VAE models including VideoVAE+ [48] and Reducio [35]. 5
Table 1: Video reconstruction results on WebVid, Inter4K (test), and Large Motion datasets. Best results are shown in bold. Our method consistently outperforms the Wan 2.1 baseline and VideoVAE+ across most metrics. WebVid
Inter4K (test)
Large Motion
Model (Backbone)
PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓
Cosmos-Tokenizer [23] CogVideoX-VAE [52] EasyAnimate-VAE [49] CV-VAE [60] Open-Sora-Plan (WF-VAE-L) [20] IV-VAE [45] VidTok [34] HunyuanVAE [17] Reducio-VAE [35] (T/4)
31.2 32.5 31.5 31.6 32.5 33.2 33.4 33.2 32.4
89.6 92.3 90.5 90.6 91.5 92.7 93.1 92.7 88.8
0.107 0.053 0.057 0.064 0.051 0.041 0.043 0.037 0.057
31.3 32.9 32.1 32.3 33.1 33.2 34.4 33.4 29.7
88.6 92.1 90.9 90.8 92.1 92.3 93.6 92.7 86.3
0.103 0.050 0.041 0.055 0.040 0.038 0.034 0.032 0.096
30.2 31.1 30.5 30.7 31.2 31.9 31.9 32.1 26.8
86.8 89.8 88.5 88.7 89.6 90.9 90.8 91.2 78.6
0.119 0.069 0.060 0.073 0.060 0.049 0.053 0.045 0.126
Wan 2.1 VAE [39] R EF D ECODER (Wan 2.1)
32.8 33.5
91.4 92.6
0.032 0.037
33.7 34.9
93.0 94.9
0.034 0.031
30.8 31.4
89.1 90.4
0.048 0.053
(+0.7)
(+1.2)
(+.005)
(+1.2)
(+1.9)
(-.003)
(+0.6)
(+1.3)
(+.005)
VideoVAE+ [48] R EF D ECODER (VideoVAE+)
32.9 35.1
93.0 93.5
0.041 0.031
33.9 34.7
93.3 94.7
0.034 0.033
31.9 32.9
90.7 92.3
0.050 0.043
(+2.1)
(+0.5)
(-.010)
(+0.8)
(+1.6)
(-.001)
(+1.1)
(+1.6)
(-.007)
3.2
Reconstruction Results
Table 1 reports reconstruction metrics on Inter4K, WebVid, and the Large Motion subset from [48]. As we can see, R EF D ECODER consistently outperforms existing baselines, with gains of +1.2 dB PSNR on Inter4K (Wan 2.1) and +2.1 dB PSNR on WebVid (VideoVAE+), and reduced LPIPS across the majority of the settings, demonstrating fine detail recovery and architectural generalization. A per-category breakdown shows that gains are positive across every category and are largest for content-rich scenes (Appendix B.6), a setting that many existing video generation pipelines struggle with [39, 17, 52]. We additionally compare against three concurrent reference-conditioned VAEs: H3AE [46], ReducioVAE [35], and RefTok [11]. Quantitative numbers for Reducio-VAE on our benchmarks are included in Table 1. Since H3AE and RefTok have not released code or model weights, we follow each of their evaluation protocols on their respective datasets—DAVIS [24, 25] for H3AE and BAIR [7] for RefTok. R EF D ECODER trained on VideoVAE+ backbone outperforms both H3AE and RefTok (Appendix D). 3.3
Generation results
We plug R EF D ECODER into the unmodified Wan 2.1 pipeline as a drop-in decoder replacement, without any retraining of the diffusion model. To isolate the decoder’s contribution, we adopt a fixed-seed protocol: for each prompt we draw 5 random seeds once, generate diffusion latents with the unmodified Wan 2.1 pipeline, and decode the same latents with both Wan 2.1 baseline decoder and R EF D ECODER, so that diffusion sampling variance do not take effect in the pairwise comparison and the improvement is attributable solely to the decoder. Evaluation details are described in Appendix B.7. As shown in Table 2, R EF D ECODER outperforms the baseline on 11 out of 12 VBench dimensions, with the largest gains in subject and background consistency, and raises the total VBench score from 87.9 to 88.2 (+0.3). For comparison, the current top open source models on the VBench leaderboard differ by a score of 0.1. Temporal stability. Beyond per-frame quality, we assess temporal stability under the same fixed-seed protocol with flow warping error (Ewarp ) [18], temporal LPIPS (EtLPIPS ) [59], flicker (Eflicker ) [4], and CLIP consistency (CLIPcons ) [8]. R EF D ECODER reduces all three degradation metrics by 8–15% and slightly improves CLIP consistency, producing less flickering and more temporally coherent videos than baseline (Appendix C). 6
Wan Wan
Wan Ours
GT
Ours
GT Figure 4: Video generation comparisons. For the same input reference, three frames generated at different timesteps are shown. The baseline distorts the underlying scene structure, whereas R EF D ECODER preserves the structure while producing sharper and more consistent details across frames. Table 2: VBench generation eval results (12 dimensions) on the Wan 2.1 backbone. Subj. / BG denote subject/background consistency; I2V Subj./BG are their I2V-specific variants. Quality Dimensions Model
Wan 2.1 [39] 96.6 97.9 R EF D ECODER (Wan 2.1) 96.8 98.2
3.4
I2V Dimensions
Aggregate Scores
Subj. BG Motion Dyna. Aesth. Imag. I2V Subj. I2V BG Camera Quality I2V Total 98.1 98.3
47.5 47.6
65.6 66.0
70.1 70.1
98.0 98.3
99.1 99.3
19.3 19.1
81.3 81.6
94.5 87.9 94.8 88.2
Human evaluation
We conduct a blind preference study comparing R EF D ECODER with four VAE baselines: Wan 2.1, HunyuanVAE, VideoVAE+, and Reducio-VAE. Participants use a slider-overlay interface in which the two videos are shown on randomized sides and model identities are hidden. Overall, R EF D ECODER is preferred over every baseline. In reconstruction tasks, R EF D ECODER is preferred over Wan 2.1, HunyuanVAE, VideoVAE+, and Reducio-VAE, with preference rates of 92.3%, 69.2%, 63.4%, and 89.6%, respectively. In generation tasks, R EF D ECODER is preferred over Wan 2.1 with a preference rate of 82.4%. Appendix F provides further detail. 3.5
Qualitative results
Figure 3 shows reconstruction results. Our decoder recovers sharper edges and finer textures than the baseline, particularly in high-frequency regions such as text, human faces, and fine structural details. These improvements are consistent across backbones. Figure 4 compares generation results produced using the baseline decoder vs. R EF D ECODER. As we can see, R EF D ECODER better preserves the structure and produces sharper, more consistent details across frames. 3.6
Efficiency
R EF D ECODER adds overhead only to the lightweight VAE decoding stage, whereas the DiT denoising stage dominates the diffusion pipeline. When run on Wan 2.1, the total pipeline latency is nearly equal with minimal overhead (72,037 ± 16 ms vs. 74,584 ± 40 ms). 7
Input
Reference
Ours
Ground Truth
Figure 5: Qualitative results on the style transfer task. The reconstructed images follow the reference style while preserving the structural content of the input images.
3.7
Ablation study
Effect of the number of blocks. We ablate the number of Transformer blocks on Wan 2.1 with 3, 5, 7, and 10. Reconstruction quality improves consistently with depth: the 10-blocks model achieves 34.9 dB PSNR on Inter4K, up from 34.3 for the 3-blocks variant, and is best across all benchmarks (full numbers in Appendix I). Latent token dropout and two-stage curriculum. We further ablate two training-time choices: latent token dropout and the two-stage curriculum. We find that higher latent-token dropout consistently improves reconstruction. Gains appear on both the reference and non-reference frames, indicating that stronger dropout encourages better use of reference information. The two-stage curriculum substantially improves over one-stage training (e.g., overall PSNR 30.5 → 34.3 dB), helping the model adapt from short to longer-video decoding. Full training details and results are in Appendix J. Alternative reference injection strategy. We also compare against a ControlNet-style [58] alternative that adds residual features from a parallel reference encoder to the decoder at each stage. This approach lacks temporal reasoning and converges to lower quality than our attention-based design. Details and qualitative results are in Appendix K.
4
Applications
4.1
Style transfer
To demonstrate the generality of our reference injection design, we apply R EF D ECODER to decodetime style transfer: by supplying a style image as the reference and training on OmniStyle150K [44], the R EF D ECODER architecture stylizes the input video using a reference style image at inference, transferring the reference style while preserving the structural content of the input. Figure 5 shows qualitative results. 4.2
Video editing
First-frame-guided video editors take a source video together with an edited version of its first frame, and produce a video in which the user-specified edit is applied consistently across all frames while everything else is expected to stay identical to the source. In practice, however, the editing model often noticeably degrades the non-edited regions. We experiment with video editing on LoRA-Edit [12], a first-frame-guided editor that fine-tunes the Wan 2.1 model with a mask-aware LoRA. We compare decoding the latents with the original Wan 2.1 decoder vs. R EF D ECODER. We follow LoRA-Edit’s pipeline to track the edited object across frames, 8
Table 3: Evaluation of R EF D ECODER on LoRA-Edit [12]. We decode the same latents with the original decoder vs. R EF D ECODER and score the non-edited region against the source video using a per-frame mask. Numbers in teal are gains of R EF D ECODER over the baseline. Decoder
PSNR↑
SSIM↑
LPIPS↓
Wan 2.1 [39] R EF D ECODER (Wan 2.1)
30.2 31.6
90.7 92.6
0.0477 0.0297
(+1.4)
(+1.9)
(−0.018)
Reference
Ground truth
LoRA-Edit
LoRA-Edit + R EF D ECODER
Figure 6: Qualitative comparison on LoRA-Edit [12]. Reference: first frame of the source video. The baseline decoder distorts non-edited regions, while R EF D ECODER regenerates them faithfully from the reference, preserving high-quality details such as the text. and score masked PSNR, SSIM, and LPIPS over the non-edited region, averaged over all frames. As shown in Table 3, R EF D ECODER improves the non-edited region on every metric (+1.4 dB PSNR, +1.9 SSIM (%), −0.018 LPIPS). Figure 6 further shows that R EF D ECODER preserves high-quality details that the baseline decoder blurs out, despite significant movement from the reference frame.
5
Discussion
We introduced R EF D ECODER, a reference-conditioned video VAE decoder that addresses a fundamental asymmetry in conditional generation: while the diffusion backbone is richly conditioned, the decoder remains generally unconditional. By injecting reference image tokens via self-attention, R EF D ECODER serves as a lightweight, architecture-agnostic drop-in replacement for existing decoders. Consistent improvements across several backbones, including mainstream open-source foundation video models, on both reconstruction and generation benchmarks, together with its natural extension to style transfer and video editing, demonstrate the generality and task-agnostic nature of conditional decoding. We hope this work encourages the community to rethink the decoder’s role in conditional generation. Several limitations suggest future directions: extending from a single reference to multiple references (e.g., for interpolation or multi-view tasks), exploring richer reference encoders beyond a single convolution layer, automating hyperparameter selection across architectures, and investigating scaling to longer videos where the reference becomes temporally distant. Because R EF D ECODER amplifies the perceptual fidelity of existing video generation models, it also inherits the same dual-use risks as the underlying model (e.g., deepfake misuse).
9
References [1] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021. [2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. [3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models, 2023. [4] Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. Blind video temporal consistency. ACM Transactions on Graphics (TOG), 34(6):1–9, 2015. [5] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024. [6] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets, 2019. [7] Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections, 2017. [8] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models, 2023. [9] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021. [10] Xiang Fan, Anand Bhattad, and Ranjay Krishna. Videoshop: Localized semantic video editing with noise-extrapolated diffusion inversion, 2024. [11] Xiang Fan, Xiaohang Sun, Kushan Thakkar, Zhu Liu, Vimal Bhat, Ranjay Krishna, and Xiang Hao. Reftok: Reference-based tokenization for video generation. arXiv preprint arXiv:2507.02862, 2025. [12] Chenjian Gao, Lihe Ding, Xin Cai, Zhanpeng Huang, Zibin Wang, and Tianfan Xue. Lora-edit: Controllable first-frame-guided video editing via mask-aware lora fine-tuning. arXiv preprint arXiv:2506.10082, 2025. [13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. [14] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [15] Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, Yaohui Wang, Xinyuan Chen, Ying-Cong Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench++: Comprehensive and versatile benchmark suite for video generative models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. [16] Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. [17] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhentao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zunnan Xu, Yangyu Tao, Qinglin Lu, Songtao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, and Caesar Zhong. Hunyuanvideo: A systematic framework for large video generative models, 2025. [18] Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. Learning blind video temporal consistency. In ECCV, 2018. [19] Teng Li, Guangcong Zheng, Rui Jiang, Shuigen Zhan, Tao Wu, Yehao Lu, Yining Lin, and Xi Li. Realcami2v: Real-world image-to-video generation with interactive complex camera control. arXiv preprint arXiv:2502.10059, 2025. [20] Zongjian Li, Bin Lin, Yang Ye, Liuhan Chen, Xinhua Cheng, Shenghai Yuan, and Li Yuan. Wf-vae: Enhancing video vae by wavelet-driven energy flow for latent video diffusion model, 2025.
10
[21] Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, and Sergey Tulyakov. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis, 2024. [22] Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02371, 2024. [23] NVIDIA. Cosmos tokenizer: A suite of image and video neural tokenizers, November 2024. [24] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Pattern Recognition, 2016. [25] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017. [26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. [27] Fitsum Red, Jinwei Gu, Xian Liu, Songwei Ge, Ting-Chun Wang, Haoxiang Wang, and Ming-Yu Liu. Cosmos tokenizer: A suite of image and video neural tokenizers, 2024. [28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. [29] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data, 2022. [30] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2, 2022. [31] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019. [32] Alexandros Stergiou and Ronald Poppe. Adapool: Exponential adaptive pooling for information-retaining downsampling, 2022. [33] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024. [34] Anni Tang, Tianyu He, Junliang Guo, Xinle Cheng, Li Song, and Jiang Bian. Vidtok: A versatile and open-source video tokenizer, 2024. [35] Rui Tian, Qi Dai, Jianmin Bao, Kai Qiu, Yifan Yang, Chong Luo, Zuxuan Wu, and Yu-Gang Jiang. REDUCIO! generating 1024×1024 video within 16 seconds using extremely compressed motion latents. In ICCV, 2025. [36] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation, 2017. [37] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017. [38] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022. [39] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.
11
[40] Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation, 2024. [41] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023. [42] Wenhao Wang and Yi Yang. Tip-i2v: A million-scale real text and image prompt dataset for image-to-video generation. 2024. [43] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability, 2023. [44] Ye Wang, Ruiqi Liu, Jiang Lin, Fei Liu, Zili Yi, Yilin Wang, and Rui Ma. Omnistyle: Filtering high quality style transfer data at scale. In CVPR, 2025. [45] Pingyu Wu, Kai Zhu, Yu Liu, Liming Zhao, Wei Zhai, Yang Cao, and Zheng-Jun Zha. Improved video vae for latent video diffusion model, 2024. [46] Yushu Wu, Yanyu Li, Ivan Skorokhodov, Anil Kag, Willi Menapace, Sharath Girish, Aliaksandr Siarohin, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. H3AE: High compression, high speed, and high quality autoencoder for video diffusion models. arXiv preprint arXiv:2504.10567, 2025. [47] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors, 2023. [48] Yazhou Xing, Yang Fei, Yingqing He, Jingye Chen, Jiaxin Xie, Xiaowei Chi, and Qifeng Chen. Videovae+: Large motion video autoencoding with cross-modal video vae. In ICCV, 2025. [49] Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: A high-performance long video generation method based on transformer architecture, 2024. [50] Wilson Yan, Matei Zaharia, Volodymyr Mnih, Pieter Abbeel, Aleksandra Faust, and Hao Liu. Elastictok: Adaptive tokenization for image and video, 2024. [51] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer, 2024. [52] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer, 2024. [53] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models, 2023. [54] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In CVPR, 2023. [55] Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion – tokenizer is key to visual generation, 2024. [56] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks, 2022. [57] David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation, 2023. [58] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. [59] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
12
[60] Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv-vae: A compatible video vae for latent generative video models. arXiv preprint arXiv:2405.20279, 2024. [61] Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. VBench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.21755, 2025. [62] Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2v: Camera-controlled image-to-video diffusion model. arXiv preprint arXiv:2410.15957, 2024.
13
A
Related Work
Visual tokenization for generation. Variational autoencoders (VAEs) [16] learn latent-space probabilistic models for efficient representation. VQ-VAE [37] introduces a vector-quantized discrete latent space, and VQGAN [9] extends it to high-resolution images with adversarial training. Latent diffusion models (LDMs) [28] leverage these encoded latent spaces for high-quality image generation. For video, MAGVIT [54] employs a 3D tokenizer for efficient video encoding, and MAGVIT v2 [55] introduces a causal 3D CNN. ElasticTok [50] offers adaptive tokenization for long videos, and OmniTokenizer [40] proposes joint image-video tokenization. Recent continuous video autoencoders such as Cosmos Tokenizer [27] and VideoVAE+ [48] serve as the visual backbone for state-of-the-art video generation models. Video generation. Early video generation relied on GANs [30, 36, 6, 56], but these are limited to short, low-resolution outputs. The field has since shifted to video diffusion models [2, 29, 3, 21, 5, 43, 57], which produce high-quality results at the cost of many denoising steps. Frontier open-source systems such as Wan 2.1 [39], HunyuanVideo [17], and CogVideoX [51] adopt the latent diffusion paradigm with powerful VAE backbones, achieving state-of-the-art quality. Image-to-video (I2V) models condition the generation process on a reference frame to produce temporally coherent video. Stable Video Diffusion [2] conditions a video diffusion model on a single image via concatenation in latent space. DynamiCrafter [47] animates open-domain images by injecting them as visual context into the diffusion backbone. IP-Adapter [53] introduces decoupled cross-attention to inject image prompt features into diffusion models, enabling image-guided generation with a lightweight adapter. A common thread across these approaches is that conditioning is applied exclusively to the diffusion model; the VAE decoder remains a purely unconditional reconstruction module and does not have access to the reference image. Reference-conditioned decoding. Recent attempts to introduce reference conditioning into the VAE decoder have struggled to achieve full quality recovery using the reference signal. For example, RefTok [11] achieves improved reconstruction only at low resolutions using a quantized codebook and struggles to yield a high-quality video generation model; H3AE [46] instead focuses on high compression ratios, with imperfect reference information forming blurred reconstructions; ReducioVAE [35] employs low-dimensional latent features without capturing high-fidelity reference signal. Prior methods exploring this space pervasively train their own VAE architectures from scratch and are tightly coupled to their respective systems. This further limits their quality and applicability, as their corresponding diffusion models are tied to the specific VAE models. For example, compared to the 1.5 billion training videos of Wan 2.1, H3AE and Reducio are only trained on million-scale data sources. In contrast, R EF D ECODER is designed as a plug-and-play decoder module that can be integrated with existing, pretrained frontier video model VAE decoders without modifying the encoder or retraining the diffusion backbone, making it immediately deployable in production pipelines.
B
Implementation Details
We apply R EF D ECODER to two pretrained video VAE backbones: Wan 2.1 [39] and VideoVAE+ [48]. Below we describe the architecture and training details using Wan 2.1 and VideoVAE+ as backbones. B.1
Model Architecture
The Wan 2.1 decoder operates in three stages at progressively higher spatial resolutions, with channel dimensions of 384, 192, and 192. At each stage, spatio-temporal feature tokens are extracted using 3D patch embeddings with temporal stride 1 and spatial strides of 2×2, 4×4, and 8×8, respectively. For comparison, VideoVAE+ also adopts a three-stage decoder but with channel dimensions of 512, 256, and 128. Spatio-temporal tokens are extracted via 3D patch embeddings with temporal stride 1 and spatial strides of 8×8, 16×16, and 16×16, respectively. For both backbones, Transformer blocks are shared across all three stages. Each uses H=12 heads with head dimension dh =128, resulting in a total hidden dimension of d=1536. We adopt Rotary Position Embeddings (RoPE) [33] for positional encoding and GELU activations in the feed-forward layers. 14
The reference image encoder is implemented as a lightweight tokenizer consisting of a single strided 3D convolution layer (patch size 8, Xavier-uniform initialization) that maps the reference frame into tokens matching the channel dimension of the first decoder stage (384 for Wan 2.1 and 512 for VideoVAE+). This is followed by normalization (RMSNorm for Wan 2.1 and GroupNorm for VideoVAE+). During training, the reference frame is sampled uniformly at random from the input video sequence. B.2
Training Setup
We train on 8 NVIDIA H200 GPUs with a per-GPU batch size of 2 (effective batch size 16) for 5 days. Model parameters are divided into two groups: (1) the attention block and reference image encoder, optimized at a base learning rate of 2 × 10−4 (effective learning rate 1.6 × 10−3 after linear scaling); and (2) the pretrained VAE decoder, fine-tuned at 0.1 of the effective rate. Both groups use AdamW with β1 = 0.9 and β2 = 0.999. The learning rate schedule consists of a linear warmup from 1% to 100% of the base rate over the first 1,000 steps, followed by cosine annealing over 100,000 total steps. The encoder, quantization convolutions, and post-quantization convolutions remain frozen throughout; only the decoder and attention block are updated. B.3
Loss Function
The training objective combines pixel-wise and perceptual losses: L = ∥x − x̂∥1 + LLPIPS (x, x̂) where x and x̂ denote the ground-truth and reconstructed video frames, and LLPIPS is the LPIPS perceptual loss [59]. Since the encoder is frozen, KL divergence regularization is disabled. B.4
Data and Evaluation
For Wan 2.1, training videos are resized to 480×832 pixels, with 5 frames per sample in the first training stage and 17 frames in the second stage. For VideoVAE+, training videos are resized to 216×216 pixels, with 4 frames per sample in the first stage and 16 frames in the second stage. We evaluate reconstruction quality using PSNR, SSIM, and LPIPS [59]. B.5
Reconstruction Benchmarks
We use three reconstruction benchmarks. Across all of them we use the exact video lists provided by VideoVAE+ [48] so that our numbers are directly comparable with those reported in [48]. • Inter4K (test). The test split of Inter4K [32], 500 high-quality videos. • WebVid. An in-the-wild subset of WebVid [1]. • Large Motion. 100 videos (80 from WebVid [1], 20 from Inter4K [32]) manually curated to exhibit complex motion dynamics—significant camera motion, fast-moving subjects, and large inter-frame displacements. For all three benchmarks, videos are decoded at the same resolution and frame count as the corresponding training configuration of each backbone, and PSNR / SSIM / LPIPS are averaged over all frames. B.6
Per-Category Reconstruction on Inter4K
Table 4 reports per-category PSNR on Inter4K (Wan 2.1 backbone). Gains are positive across every category. They are largest on content-rich scenes with strong high-frequency structure (urban streets, neon night, driving POVs, aerial skylines, indoor venues, sports, underwater wildlife, and nature landscapes), and smallest on content-sparse scenes that are already smooth or close-up (animation graphics, performance events, close-up macros, and people portraits), where the latent bottleneck loses comparatively less detail and the baseline already reconstructs faithfully. This pattern supports the intuition that the reference signal contributes most where the latent code is least sufficient. 15
Categorization protocol. To group the Inter4K test clips into 12 semantic categories without manual labeling, we use zero-shot CLIP [26]. Visual features. For each clip we sample three frames at evenly-spaced positions. Each frame embedding is L2 -normalized, and we mean-pool the three to obtain a single 768-d clip embedding vi . Category prompts. We define 12 categories and write 3–4 short natural-language prompts per category (42 prompts in total), encode each with CLIP’s text tower, and L2 -normalize. Example prompts include “a busy city street with cars and buildings” (urban street), “neon signs and billboards at night in a city” (neon night), “first-person view of driving down a road” (driving pov), and “an aerial drone view of a city” (skyline aerial); the remaining categories are indoor venue, sports action, nature landscape, close up macro, people portrait, underwater wildlife, performance event, and animation graphics. The full prompt list is included in the released code. Category score and assignment. For clip i and category c with prompt embeddings {tc,k }, the score P ⊤ 1 is the mean cosine similarity over prompts in that category, si,c = |c| , v t k i c,k and each clip is assigned to a single category by arg maxc si,c . Per-category counts are reported in Table 4. Table 4: Per-category reconstruction PSNR on Inter4K (Wan 2.1 backbone). n denotes the number of videos per category; ∆ is the PSNR gain of R EF D ECODER over the Wan 2.1 baseline. Categories are grouped into content-rich (top) and content-sparse (bottom) scenes.
B.7
Category
n
Wan 2.1
R EF D ECODER
∆ PSNR
Content-rich scenes Urban street Neon night Driving POV Skyline aerial Indoor venue (arena/mall) Sports action Underwater wildlife Nature landscape
43 34 36 48 6 29 37 14
29.0 31.6 31.3 31.3 30.8 32.7 32.8 32.3
31.4 33.6 33.4 33.3 32.6 34.2 34.3 33.7
+2.4 +2.1 +2.1 +1.9 +1.8 +1.6 +1.4 +1.4
Content-sparse scenes Animation graphics (CGI / fluid sim) Performance event (concert/dance) Close-up macro People portrait
164 45 23 18
33.8 34.2 37.4 38.5
35.0 35.2 38.2 39.2
+1.2 +1.0 +0.8 +0.7
VBench Evaluation Protocol
Since R EF D ECODER is a drop-in decoder, our VBench [14, 15, 61] evaluation follows a fixed-seed protocol that controls for diffusion-side stochasticity. We describe the protocol here so that any per-prompt difference reflects the choice of decoder rather than differences in noise or sampling. Fixed latent generation. For each prompt in the VBench info list we generate K = 5 samples, matching the default VBench-evaluation. For each (prompt, sample-index) pair we draw a 32-bit seed once via random.randint(0, 232 − 1), persist it to a per-GPU JSON log, and use it to instantiate a generator. The Wan 2.1 pipeline is then run with that generator at 50 inference steps, classifier-free guidance scale 5.0, 17 frames, and the same 16:9 480p reference image used by the official VBench benchmark. The pipeline is invoked with output_type="latent", so the noised input, denoising trajectory, and final latents are fully determined by the seed. The latents and the seed used to produce them are saved together to disk. Decoder swap. The same set of saved latents is then decoded twice—once with the original Wan 2.1 VAE decoder and once with our R EF D ECODER –and both decoded videos are scored with the official VBench-I2V evaluation pipeline. Because both decoders consume exactly the same latents produced from exactly the same seeds, the per-prompt comparison is paired: differences in VBench dimensions can only arise from the decoder, not from a different noise sample, prompt order, or guidance schedule. This is why we report the comparison without an explicit per-seed standard deviation; sample-level variance from the diffusion sampler is shared across the two methods. 16
Table 5: Temporal-stability metrics on VBench (Wan 2.1 backbone, fixed-seed protocol). Ewarp , EtLPIPS , and Eflicker measure temporal degradation (lower is better); CLIPcons measures temporal consistency (higher is better). R EF D ECODER reduces flickering and warping error across all degradation metrics. ∆ is the relative change vs. Wan 2.1. Model
Ewarp ↓
EtLPIPS ↓
Eflicker ↓
CLIPcons ↑
Wan 2.1 [39] R EF D ECODER (Wan 2.1)
0.03274 0.02941
0.09136 0.08367
0.02727 0.02303
0.99526 0.99588
(−10.2%)
(−8.4%)
(−15.5%)
(+0.06%)
Table 6: Reconstruction comparison with concurrent reference-conditioned VAEs, evaluated on each method’s reported benchmark: H3AE on DAVIS [24, 25] and RefTok on BAIR [7]. Best results are in bold. DAVIS
BAIR
Model
PSNR↑
SSIM (%)↑
H3AE [46] R EF D ECODER (VideoVAE+)
33.0 33.1
89.87 91.35
Model
PSNR↑
SSIM (%)↑
LPIPS↓
RefTok [11] R EF D ECODER (VideoVAE+)
28.8 36.2
95.0 99.1
0.013 0.0047
Reproducibility. The seed log files are kept alongside the saved latents, so any third party with access to the Wan 2.1 checkpoint can reproduce both the latents and the decoded videos bit-for-bit. The decoding scripts and the latent-generation script will be released together with the code.
C
Temporal Stability
Table 5 reports the temporal-stability metrics summarized in the main paper: flow warping error (Ewarp ) [18], temporal LPIPS (EtLPIPS ) [59], flicker (Eflicker ) [4], and CLIP consistency (CLIPcons ) [8]. The first three measure temporal degradation (lower is better) and the last consistency (higher is better). All metrics are computed on the VBench generations under the fixed-seed protocol of Sec. B.7, so any per-prompt difference reflects the choice of decoder rather than diffusion-side variance.
D
Comparison with Concurrent Reference-Conditioned VAEs
Table 6 reports the head-to-head reconstruction comparison against H3AE [46] and RefTok [11] referenced in the main paper. Since neither method has released code or model weights, we follow each of their reported evaluation protocols and run R EF D ECODER on the same datasets: DAVIS [24, 25] for H3AE and BAIR [7] for RefTok, both using the VideoVAE+ backbone. R EF D ECODER outperforms both baselines under their own evaluation settings.
E
Comparison with H3AE
We provide a qualitative head-to-head comparison with H3AE [46], a concurrent referenceconditioned VAE. Since H3AE has not released training or evaluation code, training data, or model weights, we cannot run it on our benchmark splits; instead, we take the examples directly from the figures in the H3AE paper and run R EF D ECODER on the same input frames so that both methods are evaluated under identical inputs. As shown in Figure 7, R EF D ECODER preserves fine structures and textures (e.g., the lizard’s dots and the spectator faces) noticeably better than H3AE.
F
Human Evaluation
Interface. Evaluations are collected through a custom web tool. Each pair is rendered as a single slider-overlay: the two videos are stacked and synchronously played, and the evaluator drags a vertical divider to reveal one model on the left half and the other on the right half. Left/right assignment is randomized per pair (and cached so reloads do not reshuffle), and model identities are hidden until 17
Example 1 Example 2
Ground truth
H3AE
Ours
Figure 7: Qualitative comparison with H3AE. The ground-truth and H3AE crops are taken directly from the figures in the H3AE paper, since H3AE’s training data, code, and weights are not publicly released. R EF D ECODER reconstructions are produced by us on the same input frames so that both methods are evaluated under identical inputs. R EF D ECODER recovers fine textures and structural details (lizard dots, background people) more faithfully than H3AE.
Table 7: Human evaluation. Evaluators compare the visual quality of R EF D ECODER against four baselines on video pairs sampled from Inter4K reconstructions and VBench generations, presented via a slider-overlay interface with randomized left/right order and hidden model identities. The blue segment is preference for R EF D ECODER and the gray tail is preference for the baseline; the two numbers next to each bar show R EF D ECODER’s win percentage versus the baseline. R EF D ECODER (Ours) vs. ...
Preference (Ours % vs. Baseline %)
Wan 2.1 Generation [39] Wan 2.1 Reconstruction [39] VideoVAE+ [48] HunyuanVAE [17] Reducio-VAE [35]
82.4% vs. 18.7% 92.3% vs. 7.7% 63.4% vs. 36.6% 69.2% vs. 30.8% 89.6% vs. 10.4%
after voting. The evaluator clicks one of two buttons, “Left wins” or “Right wins”, to register a preference. Evaluation protocol and votes. Evaluators are instructed to view and rate a random subset of the pairs assigned to a given baseline, yielding 252 votes in total: 102 for Wan 2.1 (89 generation, 13 reconstruction) and 50 each for VideoVAE+, HunyuanVAE, and Reducio-VAE (all reconstruction). Evaluators are instructed to choose the side with better overall visual quality. Results. Table 7 summarizes the outcomes; R EF D ECODER is consistently preferred over every baseline: 92.3% vs. 7.7% on Wan 2.1 reconstruction, 82.4% vs. 18.7% on Wan 2.1 generation, 89.6% vs. 10.4% vs. Reducio-VAE, 69.2% vs. 30.8% vs. HunyuanVAE, and 63.4% vs. 36.6% vs. VideoVAE+. These human preferences are consistent with the quantitative reconstruction (Table 1) and VBench (Table 2) results, as well as the temporal-stability results in Table 5. 18
Figure 8: Human-evaluation interface. Each pair of videos is rendered as a slider-overlay: the two videos are stacked and synchronously played, and the evaluator drags a vertical divider to reveal one decoder’s output on the left half and the other on the right half. Left/right assignment is randomized per pair and cached across reloads, and model identities are hidden until after voting. The evaluator registers a preference using the “Left wins” / “Right wins” buttons.
G
Latent Training
To further improve the performance of R EF D ECODER on the generation task, we additionally finetune the model using latent data. Specifically, we collect 30k images from TIP-I2V [42] and re-caption all images using CogVLM [41]. We then use the Wan 2.1 model [39] to generate corresponding latent video frames. During training, we use 30k videos from the original video dataset together with 30k latent videos generated by diffusion models. The latent videos correspond to the output of the diffusion model rather than the output of the VAE encoder. As in the main training stage, the encoder remains frozen and only the decoder is optimized. Training alternates between real videos and latent videos in successive batches. For real video data, the reconstruction loss is computed between the input video and the reconstructed video. For latent video data, the reconstruction loss is applied only to the first frame by comparing the reconstructed first frame with the input reference frame. Reconstruction loss is not applied to the remaining frames. In addition, we aim to train the model with latent sequences where the reference frame is not always the first frame, encouraging the model to learn longer temporal dependencies. Empirically, we find that when training only with forward sequences, early frames tend to preserve the reference appearance, while later frames may gradually deviate from it. To alleviate this issue, we construct reversed training sequences so that later frames are encouraged to attend to the reference frame. 19
Table 8: VBench evaluation results across 12 evaluation metrics. We compare the Wan 2.1 baseline with R EF D ECODER under different training settings, including latent fine-tuning and bidirectional training. Our chosen strategies improve performance across most dimensions and lead to higher aggregate scores. Quality Dimensions Model Wan 2.1 [39] R EF D ECODER (w/o FT) R EF D ECODER (Forward) R EF D ECODER (Bidirectional)
I2V Dimensions
Aggregate Scores
Subj.
BG
Motion
Dynamic
Aesth.
Imaging
I2V Subj.
I2V BG
Camera
0.9656 0.9675 0.9691 0.9693
0.9788 0.9820 0.9858 0.9859
0.9814 0.9828 0.9829 0.9833
0.4748 0.4764 0.4732 0.4724
0.6557 0.6598 0.6582 0.6586
0.7011 0.7006 0.7027 0.7010
0.9796 0.9830 0.9846 0.9847
0.9912 0.9930 0.9949 0.9950
0.1934 0.8127 0.9446 0.8786 0.1908 0.8156 0.9475 0.8815 0.1927 0.8167 0.9497 0.8832 0.1937 0.8167 0.9499 0.8833
Quality
I2V
Total
Since a single latent video frame encodes multiple video frames, we cannot directly reverse the latent sequence. Instead, we first decode the latent sequence using the original Wan 2.1 decoder to obtain the video frames, reverse their temporal order, and then pass the reversed video through the original Wan 2.1 encoder to obtain the corresponding latent representation used for training. This training also consists of two stages. In Forward Training, latent sequences are used in the original temporal order. In Bidirectional Training, each batch randomly uses either the forward or reversed sequence with equal probability. Table 8 presents VBench evaluation results on the Wan 2.1 backbone. Introducing R EF D ECODER already improves most evaluation dimensions compared to the original Wan 2.1 model. Applying latent fine-tuning further improves the performance, indicating that training on diffusion-generated latent videos helps the decoder better adapt to latent-space generation. Finally, the proposed bidirectional training strategy achieves the best overall performance, suggesting that exposing the model to reversed temporal sequences improves reference consistency across frames.
H
Random Reference Frame
We further study how the choice of reference frame during training affects model performance. Specifically, we train two models on Wan 2.1 backbone with different reference-frame strategies: one uses the first frame as the reference during training, while the other randomly samples a reference frame from the video. Both models use Wan 2.1 as the backbone with 5 Transformer blocks and a dropout rate of 0.7, and are evaluated on the 17 frames 480 × 832 Inter4K test set. During evaluation, we test both models under two settings: using the first frame or a randomly sampled frame as the reference. This allows us to analyze how each training strategy generalizes to different reference-frame conditions. Table 9 reports the quantitative results. When the model is trained using the first frame as the reference, it achieves the best reconstruction quality when evaluated with the same setting (first-frame reference). However, its performance drops significantly when evaluated with a randomly selected reference frame, indicating that the model overfits to the training reference strategy and generalizes poorly to other reference conditions. In particular, we observe that the model degenerates when the reference frame is fixed to frame 0, becoming overly dependent on the reference appearance. In contrast, training with randomly sampled reference frames leads to more robust performance across evaluation settings. The model trained with random references achieves the best overall performance when evaluated with random references and remains competitive when evaluated with the first frame as the reference. Notably, even under the first-frame evaluation setting, random-reference training still outperforms training that always uses the first frame as the reference. These results suggest that random reference training improves the model’s ability to adapt to varying reference-frame conditions during inference.
I
Effect of the Number of Transformer blocks
Table 10 reports the block-count ablation summarized in the main paper. We train the Wan 2.1 backbone with {3, 5, 7, 10} Transformer blocks under the same two-stage curriculum and evaluate on 20
Table 9: Effect of reference-frame selection during training and evaluation. Models are trained using either the first frame or randomly sampled frames as the reference. Each model is evaluated using both first-frame and random-frame references. Training with random references leads to more robust performance across evaluation settings. Training Ref.
Eval Ref. PSNR ↑
Overall SSIM (%) ↑
LPIPS ↓
PSNR ↑
First Frame SSIM (%) ↑
LPIPS ↓
First frame First frame
First frame Random frame
34.1 31.9
94.06 91.06
0.0371 0.0556
40.8 22.9
98.57 66.44
0.0046 0.2194
Random frame Random frame
First frame Random frame
34.2 34.6
94.14 94.64
0.0371 0.0337
40.0 37.2
98.24 96.82
0.0073 0.0215
Table 10: Effect of the number of Transformer blocks on reconstruction (Wan 2.1). Increasing the number of blocks consistently improves reconstruction quality, with the 10-blocks model achieving the best results. Inter4K (test)
WebVid
Large Motion
# Blocks
PSNR↑
SSIM (%)↑
LPIPS↓
PSNR↑
SSIM (%)↑
LPIPS↓
PSNR↑
SSIM (%)↑
LPIPS↓
3 5 7 10
34.3 34.6 34.5 34.9
94.4 94.6 94.7 94.9
0.035 0.034 0.032 0.031
33.2 33.4 33.3 33.5
92.3 92.6 92.3 92.6
0.038 0.038 0.037 0.037
31.1 31.3 31.2 31.4
89.9 90.2 90.1 90.4
0.055 0.055 0.054 0.053
all three reconstruction benchmarks. Reconstruction quality improves consistently with depth across PSNR, SSIM, and LPIPS, and the 10-blocks model attains the best results on every benchmark.
J
Latent Token Dropout and Two-Stage Curriculum
Setup. Table 11 reports the two training-time ablations summarized in the main paper. Both use the Wan 2.1 backbone with 3 Transformer blocks and are evaluated on Inter4K. The top block sweeps the maximum latent-token dropout probability (0.0, 0.3, 0.7), with each variant trained for 10,000 steps under the two-stage curriculum. The bottom block compares one-stage training (the first curriculum stage only) against the full two-stage curriculum at fixed dropout 0.7. We additionally report metrics over both the entire reconstructed video (Overall) and the reference frame alone (Reference Frame) to show whether gains come primarily from regenerating the reference or from propagating reference information to the rest of the video. Latent token dropout. Increasing the maximum dropout from 0.0 to 0.7 improves overall PSNR by +2.71 dB and the reference-frame PSNR by +3.85 dB. The reference-frame gain is larger than the overall gain, indicating that with dropout the decoder learns to lean on the reference token when the corresponding latent is missing rather than producing degraded content. Qualitative reconstructions in Figure 9 show the same trend: at dropout 0.7, reconstructions exhibit sharper edges and more faithful textures, while dropout 0.0 produces blurrier results. Two-stage curriculum. At fixed dropout 0.7, the two-stage curriculum improves overall PSNR by +3.80 dB and reference-frame PSNR by +4.64 dB compared to one-stage training. The much larger gap on reference-frame PSNR (+4.64 dB vs. +3.80 dB overall) suggests that the second stage primarily refines reference-token decoding to longer temporal contexts, supporting the choice to first train on short 5-frame clips and then fine-tune on 17-frame sequences.
K
Alternative Reference Injection Strategy
We compare our attention-based reference injection with a ControlNet-style [58] alternative that injects the reference image through a parallel encoder branch, adding residual features to the decoder at each stage. We identify two fundamental limitations of this approach: 21
Table 11: Ablations on the Wan 2.1 backbone with 3 Transformer blocks, evaluated on Inter4K. Top: effect of maximum latent token dropout probability (with two-stage curriculum). Bottom: effect of two-stage curriculum training (with dropout 0.7). Setting
Overall Reference Frame PSNR ↑ SSIM (%) ↑ LPIPS ↓ PSNR ↑ SSIM (%) ↑ LPIPS ↓
Dropout
0.0 0.3 0.7
27.4 27.5 30.1
88.3 86.7 90.1
0.083 0.077 0.060
27.3 27.6 31.1
88.6 87.1 92.1
0.082 0.075 0.054
Curriculum
One-stage Two-stage
30.5 34.3
89.7 94.4
0.058 0.035
34.9 39.6
95.7 98.2
0.018 0.008
GT
Dropout 0.0
Dropout 0.3
Dropout 0.7
Figure 9: Qualitative comparison of different dropout rates on Wan 2.1. Higher dropout encourages the decoder to rely more on reference features, leading to sharper and more detailed reconstructions. 1. No temporal reasoning. ControlNet operates via spatial addition of residual features, applying the reference signal identically to every frame at the same spatial position. Unlike joint attention, it lacks a mechanism to modulate the reference contribution based on temporal context. The decoder cannot distinguish whether a frame is temporally close to or far from the reference, nor can it selectively attend to different reference regions for different frames. 2. Suboptimal convergence. The ControlNet variant converges quickly to a suboptimal reconstruction quality and plateaus. We hypothesize that the spatially rigid injection forces the model into a local minimum where it learns to uniformly blend the reference signal rather than adaptively retrieving fine-grained details.
L
Code License
Our work is built upon the HuggingFace Diffusers [38] library, which is licensed under the Apache License 2.0 (https://github.com/huggingface/diffusers/blob/main/ LICENSE). VideoVAE+-related code is licensed under Attribution-NonCommercial-NoDerivatives 4.0 International, per the VideoVAE+ [48] license (https://github.com/VideoVerses/ VideoVAEPlus/blob/main/LICENSE).
M
More Qualitative Results
We present more qualitative comparisons for both video reconstruction and image-to-video (I2V) generation. Figure 11 shows reconstruction comparisons on the Wan 2.1 backbone. Each pair displays a cropped region from the same video frame, with the baseline decoder on the left and R EF D ECODER on the right. Figure 12 provides an analogous comparison on the VideoVAE+ backbone with cropped region pairs. In both cases, R EF D ECODER recovers finer textures, sharper edges, and more faithful structural details such as text and facial features. Figure 13 shows video generation comparisons on the Wan 2.1 backbone. For each example, the ground-truth reference image is shown alongside three generated frames (first, middle, and last) from both the baseline model and R EF D ECODER. Our method better preserves scene structure and 22
Input ControlNet Ours
First frame
Middle frame
Last frame
Figure 10: Qualitative comparison of alternative reference injection methods. Our attentionbased reference injection produces sharper and more temporally consistent reconstructions, while the ControlNet-style approach introduces ghosting artifacts and temporal inconsistency. fine-grained appearance from the reference image while maintaining temporal consistency across frames.
23
Baseline
Ours
Baseline
Ours
Figure 11: Additional Wan 2.1 reconstruction comparisons. Highlighted regions are shown for baseline vs. R EF D ECODER. Our method consistently recovers sharper textures and finer details across diverse scenes.
24
Baseline
Ours
Baseline
Ours
Figure 12: Additional VideoVAE+ reconstruction comparisons. Highlighted regions are shown for baseline vs. R EF D ECODER. Our method preserves fine-grained details such as text, facial features, and structural patterns more faithfully.
25
Wan Wan Ours Wan Ours
Wan
Wan Ours
Wan
Wan Ours
Wan
GT GT GT GT
Figure 13: Additional video generation comparisons. For each example, the GT reference image and three generated frames at different timesteps are shown. R EF D ECODER preserves scene structure and produces sharper, more consistent details across frames compared to the baseline.
26