Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation

Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation Zehua Chen1∗ , Junyou Wang1∗ , Yuxuan Jiang1 , Zhenying Fang2 , Yusheng Dai3 , Jianfei Chen1 , Ziwei Liu4 , Jun Zhu1† 1

Tsinghua University, Beijing, China 2 Hefei University of Technology, Hefei, China 3 Monash University, Australia 4 Nanyang Technological University, Singapore

arXiv:2608.04902v1 [cs.CV] 5 Aug 2026

Abstract Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or reasoning from large multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during diffusion training and sampling process, respectively. Extensive experiments on benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.

Introduction Generating audio from visual information has attracted increasing attention for its potential in augmented reality, film production, and multimodal content creation (Zhao et al. 2018; Luo et al. 2023; Dai et al. 2026). Early studies primarily focused on image-to-audio (I2A) generation (Iashin and Rahtu 2021; Guzhov et al. 2022; Sheffer and Adi 2023), where audio is synthesized from a single static image. More recently, research has progressed toward videoto-audio (V2A) generation, which aims to generate realistic and coherent audio for dynamic video inputs. Unlike I2A, V2A requires not only semantic consistency between visual content and generated audio but also temporal alignment between evolving visual events and corresponding sounds (Luo et al. 2023; Wang et al. 2024b; Cheng et al. 2025; Tian et al. ∗ Equal contribution. E-mails: zhc23thuml@tsinghua.edu.cn, ackokomi0222@gmail.com. † Corresponding author. E-mail: dcszj@tsinghua.edu.cn.

2026; Liu et al. 2026). This additional challenge stems from the temporal differences (TD) between consecutive video frames, which characterize scene evolution over time. Without TD, a video reduces to a static image, and V2A inevitably collapses to an I2A setting (Wang et al. 2025; Daithankar et al. 2026). Modern V2A systems are predominantly built upon conditional latent diffusion models, where the quality of the conditioning largely determines the fidelity of generated audio. Existing methods therefore focus on enriching the conditioning information through three main strategies: introducing auxiliary audio-oriented supervision, such as coarse mel-spectrograms or temporal audio structures (Wang et al. 2024b; Jeong et al. 2025; Comunità et al. 2024; Ren et al. 2025); incorporating separately pre-trained audio-visual representation models (Luo et al. 2023; Iashin et al. 2024); or leveraging multi-modal large language models for high-level video understanding (Liu et al. 2026; Dai et al. 2026). Although effective, these approaches rely on additional supervision, specialized pre-trained models, or strong inductive biases beyond the input video itself. In this work, inspired by recent advances in TD-based visual representation learning (Daithankar et al. 2026), we instead exploit the inherent TD between video frames to enhance visual representations. As the key factor distinguishing V2A from I2A, TD provides an intrinsic cue to enable superior V2A quality without introducing additional predictive networks or external conditioning signals. Motivated by this, we propose TD-V2A, which directly exploits TD representations derived from the input video without introducing additional supervision or dedicated extraction networks. Since TD can be computed in diverse representation spaces, we begin with an investigation of its form at two levels: frame-level TD and feature-level TD. Frame-level TD is obtained by differencing consecutive video frames before visual encoding, preserving complete visual variation while retaining the original visual input structure. This makes TD naturally compatible with pretrained visual encoders, allowing temporal cues to be incorporated with minimal architectural modification and inductive bias. In contrast, featurelevel TD is computed from the differences between encoded frame representations, where temporal variation is measured in a compressed semantic space. Although such representa-

Previous I2A & V2A Works: From Static Semantic to Temporal Alignment

❌

Caption

Static Semantic Information

❌

Image-to-Audio: What is in the scene?

Video & TD

TD-V2A

🔥

Temporal Differences

Annealed Temporal Differences Guidance Visual Guidance, High TD-Guidance

V-Encoder

Frame Feature

❄️

C

t CA

🔥

Generated Audio

Diffusion Transformer

TD-Encoder Temporal Differences ∆V

🔥

Visual Content

😐 What sound should be generated, and when? 🎵

❄️

V2A

Video

Generated Audio

Ours TD-V2A: "Generate the Right Sound at the Right Moment based on Temporal Differences."

CLIP

Textual Semantics

🤔 What sound should be generated? 🔊

Video-to-Audio: What happens and when?

🔥 Training

T2A

Generated Audio

Visual Content & Temporal Dynamics

Raw Frames V

Hierarchically Continual Learning

TD Feature

Noise

Inference

denoising steps

0

TD-strength

T

Visual Guidance, Low TD-Guidance Output

Figure 1: Overview of TD-V2A. Left: The framework exploits temporal differences (TD) for enriching visual representation to enhance video-to-audio generation. Right: A hierarchically continual learning strategy progressively learns textual semantics, visual content and TD, while annealed temporal differences guidance further exploits TD by dynamically adjusting its guidance strength across sampling steps. tions capture high-level semantic transitions, the encoding process inevitably compresses and abstracts the original visual information, potentially discarding fine-grained temporal variations. Our investigation demonstrates that preserving TD before visual encoding provides substantially more informative conditioning for V2A generation than modeling TD in the compressed feature space. When exploiting TD, the hierarchical structure of visual representations naturally motivates a progressive learning paradigm, where TD representations are progressively incorporated (Jiang et al. 2026; Dai et al. 2026). Therefore, at the model training stage, we develop a Hierarchically Continual Learning (HCL) strategy for the diffusion models developed in a latent space compressed from waveform. Specifically, HCL gradually exploits increasingly informative visual representations through three training stages. The first stage initializes the model with large-scale text-audio pre-training to acquire high-quality audio priors that facilitate subsequent V2A learning. The second stage adapts the model to V2A generation using raw visual representations. Finally, the third stage jointly optimizes the model under both raw visual conditioning and TD-augmented conditioning, enabling the conditional diffusion model to explicitly distinguish and effectively exploit TD for V2A generation. At inference time, the hierarchical visual conditioning introduced by TD further inspires an Annealed Temporal Differences Guidance (ATDG) strategy. Combined with the coarse-to-fine denoising trajectory of diffusion models, ATDG progressively adjusts the influence of TD throughout the sampling process. Specifically, stronger TD guidance is applied during the early sampling stages to encourage the

establishment of globally consistent temporal dynamics. As sampling proceeds toward finer details, the guidance strength is gradually annealed, allowing the generation process to rely increasingly on raw visual conditioning. By aligning the contribution of TD with the intrinsic denoising trajectory, ATDG enables the model to exploit TD information more effectively, resulting in more temporally coherent V2A generation. Following previous works, we develop TD-V2A on the VGGSound and AudioSet benchmarks. Without additional supervision or strong inductive biases, we introduce TD as an effective means to enhance visual representations for end-toend V2A generation. Equipped with the proposed HCL and ATDG strategies, TD-V2A achieves superior results with a commonly used DiT backbone (Evans et al. 2024a; Dai et al. 2026), validated by both objective and subjective evaluations. Notably, it even outperforms the additionally developed CAVP model (Luo et al. 2023), highlighting TD as a simple yet powerful alternative to strengthen V2A quality.

Related Work Temporal Differences. TD modeling is a fundamental learning principle that exploits temporal consistency to learn predictive representations from sequential data. Originally introduced in reinforcement learning through bootstrapping (Sutton 1988), TD enables models to refine current estimates using future predictions without requiring complete future supervision. Owing to its generality, the TD principle has been successfully extended beyond reinforcement learning to generative modeling (Farebrother et al. 2025) and visual representation learning, i.e., TDV (Daithankar et al.

2026), where temporal transitions provide intrinsic supervisory signals for learning meaningful features. These developments demonstrate that TDs encode rich structural information about the underlying dynamics, making TD a powerful framework for exploiting sequential observations. Inspired by this principle, our TD-V2A exploits temporal variations between consecutive video frames as informative cues to enhance visual representation and consequently enhance V2A generation results. Visual Representation in V2A Generation. Existing V2A generation methods typically improve generation quality by introducing additional supervision or strong inductive biases beyond raw video inputs, such as separately trained audio-visual representation models (Luo et al. 2023; Iashin et al. 2024), auxiliary networks that predict intermediate audio structures (e.g., coarse mel-spectrograms (Wang et al. 2024b) or temporal energy patterns (Jeong et al. 2025; Comunità et al. 2024; Ren et al. 2025)), or large language models for high-level video understanding (Liu et al. 2026; Dai et al. 2026). While effective, these approaches require specialized models or handcrafted conditioning signals. In contrast, our method directly exploits natural temporal differences inherent in the input video, using frame-level temporal variations as intrinsic cues to enhance video understanding without introducing additional predictive networks or external conditions, thereby reducing reliance on strong inductive biases while achieving superior V2A generation.

Preliminary Video-to-Audio Generation. V2A generation aims to synthesize audio x0 that is both semantically and temporally aligned with a given silent video input v (Luo et al. 2023). Recent V2A methods (Luo et al. 2023; Jeong et al. 2025; Cheng et al. 2025; Tian et al. 2026; Dai et al. 2026) typically adopt a conditional diffusion framework (Liu et al. 2023; Evans et al. 2024a), where audio is generated by progressively denoising a latent representation under the guidance of a video condition cvideo (v) extracted from the input video. During training, the model learns the correspondence between video and audio, and during inference, the extracted video condition controls the generation process toward aligned audio outputs. Formally, the diffusion training objective is defined as follows. h i 2 Lv2a = Ezt ,ϵ,cvideo ϵ − ϵθ (zt , t, cvideo (v)) 2 , (1) where zt is the noisy representation at time step t constructed with the diffusion forward process in a small latent space z compressed from the audio space x. As shown, given this formulation, the quality of the generated audio naturally depends on how effectively the video condition captures the underlying visual content and temporal dynamics. Therefore, improving video representation is a critical factor for achieving high-quality V2A generation. Additional Supervision. Existing V2A methods commonly enhance generation quality by introducing auxiliary supervision beyond the original video condition. Representative approaches employ separately trained audio-visual

representation models for enriched cross-modal conditioning, predict intermediate audio representations (e.g., melspectrograms or temporal energy patterns) with auxiliary networks, or leverage large language models for high-level video understanding and reasoning. Despite their different implementations, these methods share a common principle: they introduce additional modules to construct a strong auxiliary condition caux (v), which complements the original video representation during audio generation. The resulting diffusion objective can be generally formulated as follows. i h 2 Laux-v2a = Ezt ,ϵ,cvideo ,caux ϵ−ϵθ (zt , t, cvideo (v), caux (v)) 2 . (2) While effective, these approaches rely on additional predictive models, handcrafted supervision, or external reasoning modules, leading to increased engineering complexity and stronger inductive biases. In contrast, we improve the quality of the original video representation itself through enhanced video representation learning. By strengthening visual representation directly from the input video, our method improves V2A without introducing additional auxiliary networks.

Method Motivation: TD Distinguish V2A from I2A Existing V2A methods primarily improve video-audio correspondence by introducing additional supervision or auxiliary modules with stronger inductive biases. In contrast, our goal is to enhance V2A generation from a more fundamental perspective: strengthening the video representation itself. We argue that the key lies in explicitly modeling TD, which constitute the essential distinction between V2A and I2A generation. If a video contains no TD across frames, it effectively degenerates into a static image (Wang et al. 2025; Daithankar et al. 2026), reducing V2A to an I2A problem where temporal alignment is no longer a meaningful objective. This observation suggests that TD are not merely complementary cues, but the intrinsic visual information capturing temporal dynamics. Motivated by this insight, we explicitly exploit TD from the input video to enable stronger visual representations with weaker inductive bias. By improving the quality of the original video representation rather than introducing additional auxiliary conditions, we achieve more temporally coherent and semantically faithful V2A generation in a simple end-to-end framework.

Investigation of TD Representation We begin by exploring TD representations at different levels of visual abstraction. Specifically, TD can be constructed either directly from consecutive video frames (frame-level TD) or from their corresponding visual features (featurelevel TD). Since CLIP embeddings are the de facto visual representation in modern V2A systems (Wang et al. 2024b; Cheng et al. 2025; Tian et al. 2026; Dai et al. 2026), we instantiate feature-level TD as differences between consecutive CLIP embeddings, introducing temporal information with minimal architectural modification and without imposing additional inductive biases. In the following, we present

Raw Frames & CLIP Feature

CLIP-Level TD

Frame-Level TD

is kept frozen during diffusion model training, allowing TD to be seamlessly incorporated into existing V2A frameworks while preserving their original visual backbone. CLIP-Level Temporal Differences (CTD). Beyond differences between raw frames, temporal variation can also be characterized directly in the CLIP embedding space. Specifically, given the CLIP representations {h(v n )}N n=1 of video frames {v n }N , we compute their temporal differences at n=1 the CLIP level as:

Figure 2: Illustration of TD modeling at different granularities. While CLIP-level TD (middle) captures high-level semantic variations between embeddings, frame-level TD (right) explicitly characterizes temporal dynamics by computing pixel-wise residuals between raw frames. the formulations of frame-level TD (FTD) and CLIP-level TD (CTD), respectively. Frame-Level Temporal Differences (FTD). At the frame level, TD can be obtained by computing differences between consecutive raw frames. Formally, given a sequence of video frames {v n }N n=1 , the FTD computation is defined as: ∆v n = v n+k − v n ,

n n cTD video (v) = [h(v ); h(∆v )], (3)

where n = 1, 2, . . . , N − k; k is a hyperparameter with a small value controlling the window length when computing frame differences; and cTD video is the computed video condition that incorporates TD. Considering each element in ∆v n , its amplitude reflects the magnitude of local temporal variations between consecutive frames. Different from raw frames that are dominated by static appearance information, FTD naturally suppresses temporally invariant components and highlights dynamic changes (Daithankar et al. 2026). Since adjacent video frames usually exhibit strong temporal consistency, the resulting frame differences form a compact representation of visual evolution, where redundant background information is largely removed while informative motion patterns are preserved. Therefore, FTD provides a complementary visual representation that augments appearance features with explicit temporal variation cues, enabling V2A models to better capture the dynamic characteristics inherent in video inputs. To enrich visual representations with explicit temporal variations without imposing strong inductive biases, we directly encode both raw frames v n and and their temporal differences ∆v n using a shared pretrained CLIP encoder h. Since temporal differences remain visual observations that describe the evolution of scene content, they can be naturally embedded into the same representation space as raw frames, eliminating the need for dedicated motion encoders or additional architectural modifications. Following previous works (Tian et al. 2026; Dai et al. 2026), the CLIP encoder

n n ∆h(v n ) = h(v n+k )−h(v n ), cTD video (v) = [h(v ); ∆h(v )], (4) where n = 1, 2, . . . , N − k; k again denotes the temporal window length for computing the difference in the CLIP embedding space. Compared to FTD ∆v n , CTD ∆h(v n ) are computed after visual information has been projected into a high-level semantic representation. Consequently, they primarily capture semantic evolution while discarding a substantial amount of low-level visual variation. Although this abstraction improves semantic consistency, it may also attenuate fine-grained temporal changes that are preserved in FTD. Therefore, CTD provide a complementary view of temporal variation, emphasizing semantic transitions rather than detailed visual dynamics.

TD Encoder. We encode both raw frames and TD using the same pretrained CLIP encoder, treating TD as a complementary visual representation rather than introducing additional TD-specific encoders. Although it is possible to develop dedicated encoders for TD or motion representations, doing so inevitably introduces additional architectural complexity and stronger task-specific inductive biases. This design is supported by CLIP’s strong visual generalization capability (Radford et al. 2021). For example, CLIP achieves robust recognition on ImageNet-Sketch (Wang et al. 2019), indicating that its representations remain effective even when visual appearance is largely reduced to contours and structural cues (Radford et al. 2021). Since FTD similarly suppress static appearance while emphasizing structural changes, they can be naturally encoded within the same visual representation space. We therefore retain the standard frozen CLIP encoder throughout, enabling a controlled evaluation of TD under minimal architectural modification and inductive bias.

Hierarchically Continual Learning Raw frames and TD provide hierarchical visual representations from appearance to temporal variation, naturally motivating a progressive learning strategy to fully exploit visual information. Based on this observation, we develop a threestage hierarchical training framework that incrementally incorporates more detailed visual representations. Textual Semantics. The T2A pre-training stage provides a strong initialization for V2A, allowing the model to acquire advanced audio generation capabilities before introducing visual conditioning. In this stage, we adopt Stable Audio (Evans et al. 2024b) as the T2A backbone, which employs a DiT-based latent diffusion architecture operating on a compressed audio representation derived directly from waveform signals. Its powerful audio modeling capability provides

a strong foundation for subsequent video-conditioned audio generation. Formally, the T2A LDM is pretrained to generate a latent audio representation z0 conditioned on a text embedding ctext , with the objective: h i 2 LStage1-T2A = Ez0 ,ϵ,t ϵ − ϵθ (zt , t, ctext ) 2 , (5) where ctext can be extracted with either contrastive pretraining models, e.g., CLAP (Wu et al. 2023), or language models, e.g., FLAN-T5 (Chung et al. 2024). Visual Content. Starting from the pretrained T2A model, we further adapt it to V2A generation by replacing the text condition with visual representations. Since T2A and V2A share the same diffusion framework (Tian et al. 2026; Dai et al. 2026), the pretrained audio generation capability can be naturally transferred to video-conditioned generation. Formally, the fine-tuning objective replaces the text embedding ctext with a visual embedding, e.g., CLIP embeddings (Radford et al. 2021), extracted from raw video frames: h i 2 LStage2-V2A = Ez0 ,ϵ,t ϵ − ϵθ (zt , t, cvideo ) 2 . (6) Following recent V2A and VT2A generation systems (Wang et al. 2024b; Cheng et al. 2025; Tian et al. 2026; Dai et al. 2026), we use CLIP (Radford et al. 2021) to extract visual embeddings from raw video frames as the conditioning signal. The visual-conditioned fine-tuning enables the model to establish video-audio correspondence while retaining the audio generation prior learned during T2A pre-training. Temporal Differences. After obtaining the visualconditioned V2A model, we further introduce TD to enrich the visual representation. Rather than replacing raw frame features, TD serves as a complementary representation that explicitly characterizes frame-to-frame changes while remaining derived from the same visual observations. Since the model has already learned the basic visual representation through V2A fine-tuning, TD is introduced only in the third stage. Specifically, we jointly optimize both the original V2A task and the TD-enhanced V2A task, allowing the model to progressively learn the contribution of TD while preserving its original visual representation capability. The TD-enhanced objective is formulated as follows. h i 2 LStage3-TDV2A = Ez0 ,ϵ,t ϵ − ϵθ (zt , t, cTD (7) video ) 2 . where cTD video denotes the video condition that integrates TD at frame or CLIP level, computed as Equation (3) or Equation (4) defined above.

Annealed Temporal Differences Guidance During inference, diffusion models progressively reconstruct structured signals from Gaussian noise through a coarseto-fine denoising process. In our framework, TD provide a complementary visual representation that captures frame-toframe variations beyond the semantic information encoded by raw frame features. While HCL enhances the learning of TD representations during training, we further exploit these representations during inference through an adaptive

guidance strategy. Specifically, the jointly optimized model provides two complementary conditional predictions: one conditioned on raw frame representations and the other on TD-enhanced representations. The difference between them reflects the contribution of TD representations, which serves as an explicit temporal guidance signal during sampling. Let ϵθ (zt , t) denote the unconditional prediction, ϵθ (zt , t, cvideo ) the prediction conditioned on frame-level features, and ϵθ (zt , t, cTD video ) the prediction conditioned on both raw frame and temporal differences. The final guided prediction is computed as follows.   ϵATDG = ϵθ (zt , t) + wf ϵθ (zt , t, cTD θ video ) − ϵθ (zt , t)   + wTD (t) · ϵθ (zt , t, cTD video ) − ϵθ (zt , t, cvideo ) , (8) where wf is a constant guidance scale for frame-level semantics, and wTD (t) is a timestep-dependent weight for TD. To emphasize temporal guidance at early stages, we define: t min max min wTD (t) = wTD + (wTD − wTD ) · ( )γ , (9) T where T is the total number of sampling steps, t decreases from T to 0, and γ is the annealing factor. Under this formulation, TD dominate early steps and are gradually annealed, allowing semantic guidance to take over in later stages. This design follows the coarse-to-fine nature of diffusion sampling, improving temporal alignment while preserving semantic fidelity. It can be viewed as a mixture-of-guidance strategy (Wang et al. 2026), where variation and semantic cues contribute complementarily, yielding targeted alignment gains without additional inference cost, as demonstrated by our experimental results.

Experiment Experimental Setup Datasets. For T2A pre-training, we utilize AudioCaps training set (Kim et al. 2019), AudioSet (Gemmeke et al. 2017), VGGSound training set (Chen et al. 2020), FreeSound (Font, Roma, and Serra 2013), and MSD (Bertin-Mahieux et al. 2011). For visual and temporal differences fine-tuning, we use AudioSet and VGGSound training sets (detailed in Appendix C). All audio tracks are segmented into 10second clips and resampled to 16 kHz. Following previous works (Luo et al. 2023; Tian et al. 2026; Zhang et al. 2026), we evaluate on the VGGSound test set, consisting of about 15K 10-second audio clips. Model Training. During T2A pre-training, we use FLANT5 (Chung et al. 2024) as the text encoder, as it provides a stronger initialization for subsequent fine-tuning than CLIP or CLAP. We first train a Variational Autoencoder (VAE) to compress waveforms into latent representations, and pretrain the T2A model for 2M iterations with a total batch size of 64. For visual and temporal differences fine-tuning, we employ the same VAE and Diffusion Transformer (DiT) (Peebles and Xie 2023) backbone as in T2A, and each for 0.3M iterations with the same total batch size on 8 GPUs. We use the AdamW optimizer with a learning rate of 5 × 10−5 . Implementation details are further provided in Appendix D and Appendix E.

Method GT

Pre-training

Condition

Semantic

Temporal

FAD ↓

Audio Quality KL ↓

IS ↑

FD ↓

IBS ↑

AA ↑

Guidance

-

-

-

-

-

-

-

32.9

83.6

IM2WAV Diff-Foley FoleyGen VTA-LDM FoleyCrafter Frieren V2A-Mapper VAB-Encodec VATT⋄ MMAudio⋄ AudioX⋄

% T2I % Unlabeled V/A T2A % %∗ V-A pairs V-A pairs % T2A

CLIP CAVP† CLIP CLIP4CLIP CLIP+Timestamp† CAVP CLIP (CLAP)†∗∗ eva-CLIP Trained visual feature† CLIP+Syncformer† CLIP+Syncformer†

CFG CFG+CG CFG CFG % CFG CFG CFG CFG CFG CFG

6.41 5.79 1.65 2.01 2.32 1.23 0.90 2.69 2.35 0.81 1.13

2.54 3.12 2.35 2.37 2.54 2.73 2.68 2.58 2.25 2.11 2.57

10.8 10.4 9.9 11.3 12.5 11.9 12.2

21.90 12.80 18.10 12.13 8.35 5.65 8.83

19.0 20.4 26.1 26.2 27.7 21.0 22.4 28.0 26.0

74.3 89.9 73.8 77.0 83.6 78.3 82.8 -

TD-V2A (Ours)

HCL

CLIP+FTD

ATDG

0.53

2.16

16.9

3.79

33.8

89.1

Table 1: Comparison of our method with baseline models on the VGGSound test set. The best performance for each metric is in bold, while the second-best is marked with an underline. In the "Pre-training" column, "T", "I", "A", "V" stands for text, image, audio and video, respectively.⋄ : does not use text during inference. † : the feature requires additional network or training. ∗ : V2A-Mapper directly uses T2A (AudioLDM) for generation. ∗∗ : The mapper translating CLIP into CLAP needs training. OVL ↑

S-REL ↑

T-REL ↑

Main Results

GT

3.81 ± 0.49

3.92 ± 0.55

3.72 ± 0.47

Diff-Foley FoleyCrafter

2.46 ± 0.49 3.17 ± 0.47

2.59 ± 0.54 3.23 ± 0.56

2.21 ± 0.45 3.04 ± 0.50

We compare our results with GT (i.e., ground truth) and a variety of V2A systems1 , including IM2WAV (Sheffer and Adi 2023), Diff-Foley (Luo et al. 2023), FoleyGen (Mei et al. 2024), VTA-LDM (Xu et al. 2024), FoleyCrafter (Zhang et al. 2026), Frieren (Wang et al. 2024c), V2A-Mapper (Wang et al. 2024a), VAB-Encodec (Su, Liu, and Shlizerman 2024), VATT (Liu, Su, and Shlizerman 2024), MMAudio (Cheng et al. 2025) and AudioX (Tian et al. 2026). Detailed descriptions of these methods are provided in Appendix F. The quantitative results are summarized in Table 1. Our method considering HCL, FTD and ATDG outperforms the baseline models, with especially remarkable improvements in the FAD, IS, and IBS metrics. More importantly, apart from DiffFoley, which pretrains contrastive models with AA objective on VGGSound and AudioSet datasets to directly optimize this metric, our method surpasses all other baselines on AA and achieves performance comparable to Diff-Foley. In the following ablation studies, we also observe that compared with using condition signals from raw frames, both FTD and CTD achieve better generation quality, further validating the effectiveness of our methods.

Method

TD-V2A (Ours) 3.68 ± 0.46 3.85 ± 0.51 3.62 ± 0.48

Table 2: Subjective evaluation results for different methods.

Model Inference. To apply ATDG, we train with a dropout probability of 0.1 on the CLIP condition cvideo during visual fine-tuning. In the final fine-tuning with temporal differences, we apply 0.05 dropout to the TD condition and another 0.05 dropout to both conditions. The reason for using smaller probabilities is that ϵθ (zt , t) and ϵθ (zt , t, cvideo ) have already been trained in the previous visual fine-tuning stage. min max We set wf = 2.0, wTD = 0.5, wTD = 1.5 and γ = 1.2 in ATDG. During inference, we set the number of sampling steps to 67 for ATDG and 100 for pure CFG, ensuring the same number of function evaluations (NFE). Evaluation Metrics. We evaluate our models and baselines on audio quality and video-audio alignment, both semantically and temporally. Objective metrics include commonly used Fréchet Audio Distance (FAD), Kullback-Leibler (KL) divergence, Inception Score (IS), Fréchet Distance (FD) (using the PANN tagging model), Imagebind Score (IBS) (Girdhar et al. 2023) and temporal alignment accuracy (AA) employed in Diff-Foley. For subjective evaluation, we recruit 15 human raters to score three aspects: (i) overall audio quality (OVL), (ii) semantic relevance to the input video (S-REL) and (iii) temporal synchronization with the video (T-REL). All scores are rated on a 1–5 scale. More details of these metrics are introduced in Appendix G.

Ablation Studies Comparison with Different Conditions. To prove the effectiveness of our proposed auxiliary information for V2A, FTD and CTD, we compare their results with our baseline model with pure CLIP and CLIP+CAVP in Table 3. Our methods yield consistent improvements across all evaluation metrics over using only CLIP, with FTD exhibiting more substantial gains. Moreover, the FTD model even achieves 1

Some recent methods, such as ThinkSound, utilize extra text information (VT2A) and are not directly comparable to standard V2A generation. We further compare with them in Appendix H.

Dynamic Scene

Static Scene

Video

Diff-Foley

FoleyCrafter

TD-V2A

GT

Figure 3: Performance comparison across dynamic and static scenes. In the dynamic instrument scene, TD-V2A achieves precise rhythmic synchronization with GT. For the static scene, TD-V2A maintains higher audio quality and semantic fidelity compared to other baselines. We put more samples in Appendix I. FAD ↓ KL ↓ IS ↑ FD ↓ IBS ↑ AA ↑

k

FAD ↓

KL ↓

IS ↑

FD ↓

IBS ↑

AA ↑

T2A CLIP T2A CLIP+CAVP

0.64 0.57

2.20 16.6 4.33 2.16 16.4 4.06

31.8 31.9

87.2 88.5

T2A CLIP+CTD T2A CLIP+FTD HCL CLIP+FTD

0.57 0.55 0.53

2.20 17.1 3.93 2.18 16.5 3.98 2.16 16.9 3.79

33.3 33.4 33.8

88.3 88.8 89.1

1 2 3

0.58 0.53 0.59

2.21 2.16 2.17

17.2 16.9 16.9

4.11 3.79 3.94

33.4 33.8 33.8

88.6 89.1 88.8

Pret.

Condition

Table 3: Ablation studies on pre-training (denoted as Pret.) and condition. Setting

FAD ↓ KL ↓ IS ↑ FD ↓ IBS ↑ AA ↑

Low static (wTD = 0.5) High static (wTD = 1.5) ATDG (Ours)

0.57 0.62 0.53

2.18 16.7 3.85 2.21 16.5 4.01 2.16 16.9 3.79

33.6 33.0 33.8

88.2 89.1 89.1

Table 4: Ablation of TD guidance scale wTD . better results than CLIP+CAVP, which requires extra training objectives and stages. These results further validate that incorporating TD information, through capturing visual dynamics, enhances the performance of V2A generation. Effectiveness of HCL Strategy. Next, we demonstrate the effectiveness of our HCL strategy by comparing it with pure T2A pre-training. As shown in Table 3, T2A pre-training first provides a strong initialization, allowing the model to learn fundamental audio generation capabilities before incorporating video conditions. Building upon this foundation, our proposed HCL method progressively introduces TD features, enabling the model to develop more fine-grained capabilities while retaining the knowledge acquired in earlier stages. Consequently, HCL achieves further improvements over the already strong T2A-pretrained model. Impact of ATDG. We further compare our proposed ATDG with two static wTD settings in Table 4. As shown, a small constant wTD preserves the perceptual quality of the generated audio, but fails to provide sufficient visual difference conditioning, leading to worse temporal synchroniza-

Table 5: Ablation studies on window length k.

tion. In contrast, a large static wTD introduces noticeable distortion and degrades audio quality. Our method, ATDG, effectively achieves a balance by emphasizing TD in the early stages and gradually reducing it in later steps. These results highlight the importance of dynamically annealing the guidance strength for effectively exploiting TD while maintaining high audio quality throughout the diffusion process. We also ablate the frame-level guidance scale wf in Appendix B. Window Length. We also investigate the effect of our hyperparameter, namely the window length k, in the setting of CLIP+FTD. Specifically, we set k to 1, 2, and 3, and report the results in Table 5. We observe that when k = 1, the window is too short to effectively capture visual dynamics, while when k = 3, the window becomes too large and may be sensitive to distracting information in the video. In contrast, k = 2 serves as a balanced choice, as it not only captures short-term dynamics but also preserves fine-grained details in the video, thereby achieving the best performance.

Conclusion In this work, we identify TD as the fundamental cue that distinguishes V2A from I2A generation. Building upon this insight, we show that explicitly exploiting TD provides a simple yet effective paradigm for V2A, enabling strong generation results without additional supervision, strong inductive biases, or substantial architectural modifications. Together with HCL and ATDG, TD consistently improves generation quality and even surpasses the additionally developed CAVP model, demonstrating the remarkable potential of explicit TD modeling for future V2A research.

References Bertin-Mahieux, T.; Ellis, D. P.; Whitman, B.; and Lamere, P. 2011. The Million Song Dataset. In ISMIR, 591–596. Chen, H.; Xie, W.; Vedaldi, A.; and Zisserman, A. 2020. VGGSound: A Large-Scale Audio-Visual Dataset. In ICASSP, 721– 725. Cheng, H. K.; Ishii, M.; Hayakawa, A.; Shibuya, T.; Schwing, A.; and Mitsufuji, Y. 2025. MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis. In CVPR, 28901– 28911. Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, Y.; Wang, X.; Dehghani, M.; Brahma, S.; et al. 2024. Scaling Instruction-Finetuned Language Models. JMLR, 25(70): 1–53. Comunità, M.; Gramaccioni, R. F.; Postolache, E.; Rodolà, E.; Comminiello, D.; and Reiss, J. D. 2024. Syncfusion: Multimodal Onset-Synchronized Video-to-Audio Foley Synthesis. In ICASSP, 936–940. Dai, Y.; Chen, Z.; Jiang, Y.; Ke, Q.; Cai, J.; and Zhu, J. 2026. Omni2Sound: Towards Unified Video-Text-to-Audio Generation. In CVPR. Daithankar, N.; Gladstone, A.; LeCun, Y.; and Ji, H. 2026. You Don’t Need Strong Assumptions: Visual Representation Learning via Temporal Differences. arXiv:2606.15956. Evans, J.; et al. 2024a. Stable Audio: Audio generation using scalable diffusion models. In Stability AI Technical Report. Evans, Z.; Carr, C.; Taylor, J.; Hawley, S. H.; and Pons, J. 2024b. Fast Timing-Conditioned Latent Audio Diffusion. In ICML, 12652– 12665. Evans, Z.; Parker, J. D.; Carr, C.; Zukowski, Z.; Taylor, J.; and Pons, J. 2025. Stable Audio Open. In ICASSP, 1–5. Farebrother, J.; Pirotta, M.; Tirinzoni, A.; Munos, R.; Lazaric, A.; and Touati, A. 2025. Temporal Difference Flows. In ICML. Font, F.; Roma, G.; and Serra, X. 2013. Freesound Technical Demo. In ACM MM, 411–412. Gemmeke, J. F.; Ellis, D. P.; Freedman, D.; Jansen, A.; Lawrence, W.; Moore, R. C.; Plakal, M.; and Ritter, M. 2017. Audio Set: An Ontology and Human-Labeled Dataset for Audio Events. In ICASSP, 776–780. Girdhar, R.; El-Nouby, A.; Liu, Z.; Singh, M.; Alwala, K. V.; Joulin, A.; and Misra, I. 2023. ImageBind: One Embedding Space to Bind Them All. In CVPR, 15180–15190. Guzhov, A.; Raue, F.; Hees, J.; and Dengel, A. 2022. AudioCLIP: Extending CLIP to Image, Text and Audio. In ICASSP, 976–980. Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising Diffusion Probabilistic Models. In NeurIPS, 6840–6851. Iashin, V.; and Rahtu, E. 2021. Taming Visually Guided Sound Generation. In BMVC. Iashin, V.; Xie, W.; Rahtu, E.; and Zisserman, A. 2024. Synchformer: Efficient Synchronization from Sparse Cues. In ICASSP, 5325–5329. Jeong, Y.; Kim, Y.; Chun, S.; and Lee, J. 2025. Read, Watch and Scream! Sound Generation from Text and Video. In AAAI, 17590– 17598. Jiang, Y.; Chen, Z.; Ju, Z.; Dai, Y.; Dou, W.; and Zhu, J. 2026. ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling. In ACL, 1394–1413. Kim, C. D.; Kim, B.; Lee, H.; and Kim, G. 2019. AudioCaps: Generating Captions for Audios in the Wild. In NAACL-HLT, 119– 132.

Liu, H.; Chen, Z.; Yuan, Y.; Mei, X.; Liu, X.; Mandic, D. P.; Wang, W.; and Plumbley, M. D. 2023. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. In ICML, 21450–21474. Liu, H.; Luo, K.; Wang, J.; Wang, W.; Chen, Q.; Zhao, Z.; and Xue, W. 2026. ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing. In NeurIPS, 78526– 78552. Liu, H.; Yuan, Y.; Liu, X.; Mei, X.; Kong, Q.; Tian, Q.; Wang, Y.; Wang, W.; Wang, Y.; and Plumbley, M. D. 2024. AudioLDM 2: Learning Holistic Audio Generation with Self-Supervised Pretraining. TASLP, 32: 2871–2883. Liu, X.; Su, K.; and Shlizerman, E. 2024. Tell What You Hear from What You See: Video-to-Audio Generation through Text. In NeurIPS, 101337–101366. Luo, S.; Yan, C.; Hu, C.; and Zhao, H. 2023. Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models. In NeurIPS, 48855–48876. Mei, X.; Nagaraja, V.; Le Lan, G.; Ni, Z.; Chang, E.; Shi, Y.; and Chandra, V. 2024. FoleyGen: Visually-Guided Audio Generation. In MLSP, 1–6. Peebles, W.; and Xie, S. 2023. Scalable Diffusion Models with Transformers. In ICCV, 4195–4205. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning Transferable Visual Models from Natural Language Supervision. In ICML, 8748–8763. Ren, Y.; Li, C.; Xu, M.; Liang, W.; Gu, Y.; Chen, R.; and Yu, D. 2025. STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment. In ICASSP, 1–5. Shan, S.; Li, Q.; Cui, Y.; Yang, M.; Wang, Y.; Yang, Q.; Zhou, J.; and Zhong, Z. 2025. HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation. arXiv:2508.16930. Sheffer, R.; and Adi, Y. 2023. I Hear Your True Colors: Image Guided Audio Generation. In ICASSP, 1–5. Song, Y.; Sohl-Dickstein, J.; Kingma, D. P.; Kumar, A.; Ermon, S.; and Poole, B. 2021. Score-Based Generative Modeling through Stochastic Differential Equations. In ICLR. Su, K.; Liu, X.; and Shlizerman, E. 2024. From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation. In ICML, 46804–46822. Sutton, R. S. 1988. Learning to Predict by the Methods of Temporal Differences. Mach. Learn., 3: 9–44. Tian, Z.; Jin, Y.; Liu, Z.; Yuan, R.; Tan, X.; Chen, Q.; Xue, W.; and Guo, Y. 2026. AudioX: Diffusion Transformer for Anythingto-Audio Generation. In ICLR. Wang, H.; Ge, S.; Lipton, Z.; and Xing, E. P. 2019. Learning Robust Global Representations by Penalizing Local Predictive Power. In NeurIPS, 10506–10518. Wang, H.; Ma, J.; Pascual, S.; Cartwright, R.; and Cai, W. 2024a. V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models. In AAAI, 15492–15501. Wang, J.; Chen, Z.; Yuan, B.; Zheng, K.; Li, C.; Jiang, Y.; and Zhu, J. 2026. AudioMoG: Guiding Audio Generation with Mixture-ofGuidance. In ICME. Wang, X.; Wang, Y.; Wu, Y.; Song, R.; Tan, X.; Chen, Z.; Xu, H.; and Sui, G. 2024b. TiVA: Time-Aligned Video-to-Audio Generation. In ACM MM, 573–582. Wang, Y.; Chen, Z.; Chen, X.; Wei, Y.; Zhu, J.; and Chen, J. 2025. FrameBridge: Improving Image-to-Video Generation with Bridge Models. In ICML, 62596–62626.

Wang, Y.; Guo, W.; Huang, R.; Huang, J.; Wang, Z.; You, F.; Li, R.; and Zhao, Z. 2024c. Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching. In NeurIPS, 128118– 128138. Wu, Y.; Chen, K.; Zhang, T.; Hui, Y.; Berg-Kirkpatrick, T.; and Dubnov, S. 2023. Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. In ICASSP, 1–5. Xu, M.; Li, C.; Ren, Y.; Chen, R.; Gu, Y.; Liang, W.; and Yu, D. 2024. Video-to-Audio Generation with Hidden Alignment. arXiv:2407.07464. Zhang, Y.; Gu, Y.; Zeng, Y.; Xing, Z.; Wang, Y.; Wu, Z.; Liu, B.; and Chen, K. 2026. FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds. IJCV, 134(1): 46. Zhao, H.; Gan, C.; Rouditchenko, A.; Vondrick, C.; McDermott, J.; and Torralba, A. 2018. The sound of pixels. In ECCV, 570–586.

Audio Latent Diffusion Model

Given enough sampling steps, the target representation ẑ0 can be generated from zT , and then the audio waveform x̂0 can be decoded from ẑ0 with the compression network.

B

Impact of Frame-Level Guidance Scale

Besides guidance scale of TD wTD , we also analyze the impact of frame-level guidance scale wf . Specifically, we vary its value from 1.0 to 2.5, with results shown in Figure 4. In general, increasing wf within a moderate range improves audio quality; however, exceeding the optimal value of 2.0 leads to a decline in audio fidelity.

)$'



,6

























,6

Diffusion models have advanced cross-modal audio generation with the strong capability to faithfully capture the target distribution (Ho, Jain, and Abbeel 2020; Song et al. 2021). They are composed of two processes: a forward process transforming the data distribution into a known prior distribution, e.g., standard Gaussian distribution, and a reverse process gradually reconstructing the target distribution from the prior distribution with iterative sampling steps. In recent works, audio generation systems are usually built upon a latent diffusion framework. At early stage, the small space is compressed from the mel-spectrogram and the entire generation process includes latent diffusion sampling, VAE decoding, and a vocoder transforming mel-spectrogram to waveform (Liu et al. 2023, 2024; Wang et al. 2024b). In this work, we directly develop the diffusion model in the latent space compressed from audio waveform, avoiding cascaded VAE and vocoder. The details of VAE is introduced in Appendix D. Given an audio sample x0 ∈ RL sampled from the observed dataset, where L is the sample length, audio generation systems usually develop a compression network to encode it into a latent representation z0 ∈ RTd ×Fd , where Td and Fd denote the temporal and feature dimension, respectively. Then, in the compressed latent space, a forward process of the latent diffusion model injects Gaussian noise ϵ ∼ N (0, I) into z0 with a predefined noise schedule βt , constructing noisy representations zt ∈ RTd ×Fd at each time step t ∈ (0, T ] with: √ q(zt |z0 ) = N (zt ; ᾱt z0 , (1 − ᾱt )I), (10) Qt where ᾱt = s=1 (1 − βs ) indicates the noise level at time step t. When t = T , the clean latent representation z0 at time step t = 0 will be transformed into uninformative Gaussian noise zT . In generation, the reverse process starts from Gaussian noise zT , gradually reconstructing the target z0 with an iterative denoising and refinement trajectory:
pθ (zt−1 |zt ) = N zt−1 ; µθ (zt , t), σt I , (11) where the time-dependent variation σt is usually predefined according to βt and the mean function µθ (zt , t) can be parameterized with different methods and predicted by a neural network. For diffusion-based audio generation systems (Liu et al. 2023; Evans et al. 2025), noise prediction is a popular parameterization method, where the network is optimized with the objective: i h √ √ 2 Laudio = Ez0 ,ϵ,t ϵ − ϵθ ( ᾱt z0 + 1 − ᾱt ϵ, t) 2 . (12)

)$'

A













)UDPHOHYHOJXLGDQFHVFDOHwf







Figure 4: Ablation study on frame-level guidance scale wf .

C

Datasets

We present the datasets used to pre-train and fine-tune our model in Table 6. AudioCaps (Kim et al. 2019) serves as a widely used benchmark for audio captioning that comprises 44K audio segments sourced from AudioSet, each annotated with human-written textual descriptions. AudioSet (Gemmeke et al. 2017) is a large-scale, weakly labeled dataset released by Google, containing more than 2M 10-second audio clips across over 600 sound event categories. VGGSound (Chen et al. 2020) is another large-scale dataset, consisting of over 200K 10-second video clips collected from YouTube and covering 310 distinct sound categories. FreeSound (Font, Roma, and Serra 2013) is a collaborative online platform that provides a diverse collection of useruploaded audio samples, commonly utilized for environmental sound classification and retrieval tasks. The Million Song Dataset (MSD) (Bertin-Mahieux et al. 2011) provides metadata along with pre-extracted audio features for 1M popular music tracks, facilitating large-scale music analysis and recommendation research. Dataset

Task

AudioCaps T2A AudioSet T2A, V2A VGGSound T2A, V2A FreeSound T2A MSD T2A

Hours (h)

Source

109 5800 550 6246 7333

(Kim et al. 2019) (Gemmeke et al. 2017) (Chen et al. 2020) (Font, Roma, and Serra 2013) (Bertin-Mahieux et al. 2011)

Table 6: Details of training datasets in TD-V2A.

D

Compression Network

We adopt a waveform-domain VAE based on the Oobleck framework with a sampling rate of 16 kHz. Previous V2A generation approaches may rely on Mel-spectrograms as intermediate representations, typically following the AudioLDM (Liu et al. 2023) paradigm where diffusion models are developed in the continuous latent space of the Melspectrogram and a separate vocoder is required to reconstruct waveforms. In contrast, we directly compress audio

waveforms into continuous latent representations, avoiding the cascaded VAE and vocoder pipeline. To ensure fair comparison with existing audio generation systems, the waveform compression architecture and training procedure follow the widely adopted and publicly available Stable Audio Open (Evans et al. 2025) framework. The encoder and decoder are designed symmetrically with a base channel size of 128 and channel multipliers of 1, 2, 4, 8, and 16. The corresponding stride factors are 2, 2, 4, 4, and 10, resulting in an overall temporal down-sampling ratio of 640. The encoder maps the input mono-channel waveform into a 128-dimensional latent representation, while the decoder reconstructs the waveform from a 64-dimensional latent code through a variational bottleneck layer. Snake activation functions are applied throughout the network, and no final tanh activation is used in the decoder.

E

Model Configurations

Our diffusion framework is based on the Diffusion Transformer (DiT) architecture and follows the latent diffusion modeling (LDM) paradigm, which provides strong generative performance along with effective context representation. During T2A pre-training, we employ FLAN-T5 (Chung et al. 2024) as the text encoder, and adopt the VAE in Appendix D to compress the original waveform into a compact latent space. The diffusion backbone utilizes a DiT configuration with 24 layers and 24 attention heads, each with an embedding dimension of 1536. The model incorporates both cross-attention and global conditioning mechanisms: crossattention is used for all conditional inputs, while global conditioning is dedicated to handling duration-related control signals. The internal token dimension of the diffusion model is set to 64, with a conditional token dimension of 768 and a global conditioning embedding dimension of 1536. The output latent representation shares the same dimensionality as io_channels, which is 64.

F

Baseline Methods

We introduce the baseline methods used for comparison in Table 1. IM2WAV (Sheffer and Adi 2023) is an image-guided open-domain audio generation model that employs two transformer language models over discrete audio tokens derived from a VQ-VAE, with CLIP-based visual conditioning and classifier-free guidance. Diff-Foley (Luo et al. 2023) is a V2A synthesis approach based on latent diffusion models, leveraging contrastive audio-visual pretraining (CAVP) and cross-attention with double guidance to achieve strong temporal and semantic alignment. FoleyGen (Mei et al. 2024) formulates V2A generation as a language modeling task, using a neural audio codec and a single transformer conditioned on visual features to generate audio tokens. It further introduces novel visual attention mechanisms to improve temporal alignment between generated audio and video content. VTA-LDM (Xu et al. 2024) is a latent diffusion-based V2A framework that investigates key factors such as vision

encoders, auxiliary embeddings, and data augmentation. It achieves strong performance in generating semantically and temporally aligned audio through systematic design and evaluation. FoleyCrafter (Zhang et al. 2026) is a V2A framework built upon pretrained T2A models, introducing a semantic adapter and a temporal controller to enhance semantic relevance and precise audio-video synchronization, while supporting controllable generation via text prompts. V2A-Mapper (Wang et al. 2024a) is a lightweight approach that bridges visual and audio modalities by mapping CLIP embeddings to CLAP space. Conditioned on the translated embeddings, a pretrained AudioLDM generates highfidelity and well- aligned audio with minimal training cost. VAB-Encodec (Su, Liu, and Shlizerman 2024) is a unified audio-visual framework that performs both representation learning and generation in latent space. It leverages visual-conditioned masked audio token prediction and iterative decoding to produce high-quality, semantically aligned audio while supporting various downstream tasks. VATT (Liu, Su, and Shlizerman 2024) is a controllable V2A framework that generates audio from video with optional text guidance and can produce audio captions. It uses a fine-tuned LLM to map video features and a bidirectional transformer to generate audio tokens, which are decoded into waveforms, enabling text-guided audio generation and captioning. MMAudio (Cheng et al. 2025) is a V2A framework that adopts a multimodal joint training paradigm over audio-video and audio-text datasets. By jointly learning from video, audio, and text with modality masking and a conditional synchronization module, it generates high-quality audio with strong semantic alignment and temporal synchronization. AudioX (Tian et al. 2026) is a unified multimodal audio generation framework that supports flexible conditioning on text, video, and audio inputs. It employs a Multimodal Adaptive Fusion module and is trained on a large-scale dataset to achieve strong cross-modal alignment and high-quality generation across diverse tasks. For Diff-Foley, VTA-LDM and FoleyCrafter, we generate audio samples using their official implementations. For V2A-Mapper, it supplies pre-generated audio samples for evaluation. For FoleyGen, VAB-Encodec, VATT and AudioX, we report the results presented in their original papers. The IM2WAV and MMAudio results are adopted from VATT and Omni2Sound, respectively.

G

Evaluation Metrics

Objective Metrics We introduce the objective metrics employed in our evaluation, including Fréchet Audio Distance (FAD), KullbackLeibler (KL) divergence, Inception Score (IS), Fréchet Distance (FD), Imagebind Score (IBS) (Girdhar et al. 2023) and temporal alignment accuracy (AA) (Luo et al. 2023). FAD, adapted from FID (Fréchet Inception Distance), measures the distributional gap between generated and reference audio using VGGish embeddings, serving as the main

Dynamic Scene

Static Scene

Video

Diff-Foley

FoleyCrafter

TD-V2A

GT

Figure 5: Performance comparison across dynamic and static scenes. generated audio is synchronized with video by classifying correctly aligned versus temporally shifted or mismatched audio-video pairs. The percentage of true pairs correctly identified serves as a quantitative measure of temporal synchronization.

Video

Baseline

TD-V2A

Subjective Metrics GT

Figure 6: Spectrogram comparison for the "baby crying" video. TD-V2A achieves significantly better temporal synchronization and semantic alignment with the GT than the T2A+CLIP baseline. Method

FAD ↓ KL ↓ IS ↑ FD ↓ IBS ↑

ThinkSound (w/o CoT) HunyuanVideo-Foley AudioX (VT2A) MMAudio (w/ text) Omni2Sound (VT2A)

1.10 2.36 1.24 0.91 0.53

1.60 1.74 1.59 1.63 1.35

11.7 7.41 11.6 10.02 14.9 8.29 13.4 5.28 15.8 2.95

26.0 32.0 26.0 29.0 34.0

TD-V2A (Ours)

0.53

2.16

16.9

33.8

3.79

Table 7: Comparison of our method with VT2A methods in Omni2Sound. Note that these methods utilize the carefully curated text caption in VGGSound-Omni. measure of audio fidelity. KL divergence evaluates the difference between acoustic event posteriors of the generated and ground truth audio. IS captures both diversity and specificity of generated samples by computing entropy over class predictions. FD, similar in formulation to FAD, is calculated in more general embedding spaces to provide an additional assessment of generative quality. ImageBind Score (IBS) evaluates the semantic alignment between generated audio and the corresponding video by computing the cosine similarity of their embeddings in a shared multimodal space, using the ImageBind model. A higher IBS indicates stronger semantic correlation between audio and visual content. Temporal Alignment Accuracy (AA) measures how well

We randomly select 20 samples from the VGGSound test set for the subjective evaluation. Each sample group contains results from Diff-Foley, FoleyCrafter, and the ground truth (GT), with the order of items within each group randomly shuffled. Every group is evaluated by 15 human raters. Participants rate three aspects—overall audio quality (OVL), semantic relevance to the input video (S-REL), and temporal synchronization with the video (T-REL)—on a 1-to-5 scale. For OVL, raters assess perceptual audio quality, while S-REL and T-REL reflect the semantic alignment and temporal correspondence of the audio to the video, respectively. All ratings were given in integer increments.

H

Comparison with VT2A methods

Some recent methods, such as HunyuanVideo-Foley (Shan et al. 2025), ThinkSound (Liu et al. 2026), and Omni2Sound (Dai et al. 2026), incorporate additional text conditions during inference, and therefore considered as video-and-text-to-audio (VT2A) generation rather than standard V2A. In this section, we further compare our method with several representative VT2A approaches reported in Omni2Sound. As shown in Table 7, despite relying solely on video inputs, our proposed TD-V2A achieves performance comparable to or even surpassing these VT2A methods. Notably, the VGGSound-Omni dataset introduced by Omni2Sound constructs captions through an elaborate caption generation and verification pipeline, whereas our method requires no text annotations at inference. These results further demonstrate the effectiveness of TD-V2A.

I

More Generation Results

We give more generated cases in Figure 5 and Figure 6.

Related documents

• Temporal State Transport in Video Generation: Diagnosing and Correcting Spectral Imbalance#985854
• Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation#1132196
• Evaluating Design Video Generation: Metrics for Compositional Fidelity#192382
• Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation#673524
• AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation#2632