arXiv preprint
O MNI NFT: M ODALITY- WISE O MNI D IFFUSION R EIN FORCEMENT FOR J OINT AUDIO -V IDEO G ENERATION
3.326
5. 19 7
7 79 0.6 77 ↑
A
41
5.1
5.715
↑
gle-Modal Sin
y↑ alit Qu io ud
Moti on Qu ali ty
Quality ↑ Visual
0.547
LTX
2.374
2.038
From a wide, high-angle perspective, a professional basketball game unfolds in a brightly lit indoor arena. On the light wood-toned court branded with "Quicken Loans Arena\" and \"NBA 2K,\" the Golden State player wearing number 30…
0.
0.223
ore ↑
C ro
32 0.4
ss- M o d al
-S c
AP
0.445
AV H
CL
0.2 51
0. 4
12
JavisScore ↑
0.220
57
LTX-2
0.222
OmniNFT
2 0.
0.192
arXiv:2605.12480v1 [cs.CV] 12 May 2026
Guohui Zhang1 , Xiaoxiao Ma1 , Jie Huang1 , Hang Xu1 , Hu Yu1 , Siming Fu3 , Yuming Li2 , Zeyue Xue3 , Lin Song3‡ , Haoyang Huang3 , Nan Duan3 , Feng Zhao1† 1 University of Science and Technology of China, 2 Peking University, 3 JD Explore Academy
LTX-2.3
LTX-2+OmniNFT
Figure 1: OmniNFT consistently improves the performance of LTX-2 in audio and visual quality, motion quality, cross-modal alignment, and audio–video synchronization.
A BSTRACT Recent advances in joint audio-video generation have been remarkable, yet realworld applications demand strong per-modality fidelity, cross-modal alignment, and fine-grained synchronization. Reinforcement Learning (RL) offers a promising paradigm, but its extension to multi-objective and multi-modal joint audiovideo generation remains unexplored. Notably, our in-depth analysis first reveals that the primary obstacles to applying RL in this stem from: (i) multi-objective advantages inconsistency, where the advantages of multimodal outputs are not always consistent within a group; (ii) multi-modal gradients imbalance, where video-branch gradients leak into shallow audio layers responsible for intra-modal generation; (iii) uniform credit assignment, where fine-grained cross-modal alignment regions fail to get efficient exploration. These shortcomings suggest that vanilla RL fine-tuning strategy with a single global advantage often leads to suboptimal results. To address these challenges, we propose OmniNFT, a novel modality-aware online diffusion RL framework with three key innovations: (1) Modality-wise advantage routing, which routes independent per-reward advantages to their respective modality generation branches. (2) Layer-wise gradient surgery, which selectively detaches video-branch gradients on shallow audio layers while retaining those for cross-modal interaction layers. (3) Region-wise loss reweighting, which modulates policy optimization toward critical regions related to audio-video synchronization and fine-grained alignment. Extensive experiments on JavisBench and VBench with the LTX-2 backbone demonstrate that OmniNFT achieve comprehensive improvements in audio and video perceptual quality, cross-modal alignment, and audio-video synchronization. Project Page: † ‡
https://zghhui.github.io/OmniNFT/
Corresponding author Project leader
1
arXiv preprint
1
I NTRODUCTION
Recent years have witnessed significant advancements in joint audio-video generation (Low et al., 2025; Liu et al., 2026a; Seedance et al., 2025). However, achieving genuine practical utility in realworld applications demands a combination of high per-modality fidelity, robust cross-modal semantic consistency, and fine-grained audio-video synchronization. How to effectively align audio-video generative models with these multifaceted requirements remains an open and pressing challenge. Despite rapid progress, current joint audio-video generative models (Wang et al., 2025; HaCohen et al., 2026; Liu et al., 2026a) still struggle to simultaneously satisfy these multifaceted objectives. Meanwhile, Reinforcement Learning with Verifiable Rewards (RLVR), particularly Group Relative Policy Optimization (GRPO) (Guo et al., 2025), has recently emerged as a powerful post-training paradigm for generative models (He et al., 2025; Li et al., 2025), owing to its ability to optimize complex and highly subjective objectives (Liu et al., 2026b). The previous research (Xue et al., 2025; Zheng et al., 2025) has successfully leveraged RLVR to enhance text-to-image/video generation quality and semantic alignment. These successes motivate a key question: can RLVR be effectively extended to joint audio-video generation to optimize fidelity, alignment, and AV synchronization? However, the performance of directly applying vanilla RLVR to joint audio-video generation remains suboptimal. Our in-depth analysis reveals that the primary obstacles stem from three types of optimization mismatch: (i) multi-objective advantages inconsistency: the different rewards for the video and audio components of a single multimodal output are often inconsistent, as illustrated in Fig. 2(a). (ii) multi-modal gradients imbalance: intra-modal generation and cross-modal interaction mainly concentrate in the shallow and deeper layers, respectively, see Fig. 2. However, gradients from the video branch tend to dominate the update direction of the shallow audio layers in the backward process. (iii) uniform credit assignment: AV synchronization and fine-grained alignment focus on certain critical regions, yet uniform updates ignore different contributions of these regions. We address these challenges with OmniNFT (Modality-wise Omni Diffusion Negative-aware FineTuning), a novel modality-aware diffusion RL framework for joint audio-video generation. OmniNFT addresses the above issues with three coordinated designs. (1) Modality-wise advantage routing: for advantage inconsistency, we compute an independent advantage for each reward and selectively route it according to its underlying modality. (2) Layer-wise gradient surgery: for gradients imbalance, we detach the part gradient from the video stream on the shallow layers of the audio model, while preserving the effective gradients responsible for audio–video interaction. (3) Region-wise loss reweighting: for credit assignment, we introduce a critical-region reweighting strategy that strengthens the optimization on critical regions. This fine-grained credit assignment, coupled with gradient surgery, effectively circumvents optimization mismatch across modalities. We conduct extensive experiments on JavisBench (Liu et al., 2025c) and VBench (Huang et al., 2024) using LTX-2 (HaCohen et al., 2026) as the backbone. OmniNFT achieves comprehensive improvements across audio and video perceptual quality, cross-modal alignment, and audio-video synchronization. Our contributions are summarized as follows: • We first visit RL for joint audio-video generation and identify its core optimization bottleneck, including advantage inconsistency, gradients imbalance, and uniform credit assignment, which together hinder effective joint improvement. • Based on these bottlenecks, we propose OmniNFT that combines modality-wise advantage routing, layer-wise gradient surgery, and region-wise loss reweighting, effectively facilitating multi-objective and multi-modal optimization. • Extensive experiments demonstrate that OmniNFT delivers consistent gains in perceptual quality, cross-modal alignment, and audio-video synchronization over strong baselines.
2
R ELATED W ORK
2.1
T EXT- TO -V IDEO G ENERATION .
Text-to-video generation has rapidly progressed from U-Net-based diffusion to large-scale Transformer-based diffusion. Early methods (Blattmann et al., 2023a; Guo et al., 2023; Wu et al., 2
arXiv preprint
Figure 2: Advantage inconsistency and asymmetric audio-video interaction. (a) Video and audio advantages are weakly correlated: roughly half of the samples receive opposing rewards across the two modalities. (b) Blocking the V2A KV in mid-layers collapses AV synchronization to 0.41× baseline, whereas (d) the symmetric A2V ablation causes a mild degradation when applied to the later blocks. (c,e) Layer-wise gradient norms of cross-attention show that audio-video interaction is concentrated in the middle and later transformer blocks (AV Sync Zone). On the audio branch, gradients from video KV (A2V) disturb the update direction of the audio shallow blocks in (c).
2023a; Blattmann et al., 2023b) typically adapt image diffusion backbones with temporal modules to support video generation, while recent Diffusion Transformers (DiTs) (Brooks et al., 2024; Hong et al., 2022; Kong et al., 2024) scale more effectively with data and model size, demonstrating substantial gains in visual fidelity, physical plausibility, and text alignment. Among open-source lines, Wan (Wan et al., 2025) emphasizes efficient spatiotemporal attention with flow-matching training, and LTX-Video (HaCohen et al., 2024) further targets real-time inference by jointly optimizing a Video-VAE and a denoising Transformer. 2.2
J OINT AUDIO -V IDEO G ENERATION .
Building on strong text-to-video backbones, recent work increasingly focuses on generating temporally synchronized video and audio in a single pipeline. Veo3 (Google DeepMind, 2024) demonstrates joint audio-video capabilities. UniVerse-1 (Wang et al., 2025) combines Wan 2.1 (Wan et al., 2025) for video generation and ACE-Step (Gong et al., 2025) for audio generation, and introduces lightweight projection modules to align latent spaces across modalities. In parallel, the Javis series (Liu et al., 2025c; 2026a) explores unified modeling for joint generation, whereas the LTX series (HaCohen et al., 2026) adopts an asymmetric dual-stream design with bidirectional crossattention to handle modality imbalance. While these efforts have yielded promising results, there remains substantial room for improvement in generation quality and audio-visual synchronization. 2.3
R EINFORCEMENT L EARNING FOR V ISUAL G ENERATION .
Inspired by the success of GRPO (Guo et al., 2025) in large language models (LLMs), RLVR has emerged as a practical post-training strategy for improving visual quality and semantic alignment in visual generation (Luo et al., 2025; Zhang et al., 2025a;b; Jiang et al., 2025). Several methods adapt reward-driven policy optimization to image and video generation tasks. For example, T2I-R1 (Jiang et al., 2025) introduces dual-level chain-of-thought RL for autoregressive generation, while FlowGRPO (Liu et al., 2025a) and Dance-GRPO (Xue et al., 2025) extend flow matching with stochastic trajectories by reformulating Ordinary Differential Equation (ODE) as Stochastic Differential Equation (SDE) processes. DiffusionNFT (Zheng et al., 2025) instead finetunes diffusion models in the forward process through an implicit policy-improvement direction. However, effective expansion of RL into multi-objective joint audio-visual generation remains under-explored.
3
P RELIMINARY
Joint Audio-Video Flow Matching. In joint audio-video generation, an audio latent xa and a video latent xv are denoised in parallel under a shared flow matching (Lipman et al., 2022) schedule. 3
arXiv preprint
Each modality is independently perturbed by its own Gaussian prior xm 1 ∼ N (0, I), while sharing a timestep t ∈ [0, 1]: m m xm m ∈ {a, v}. (1) t = (1 − t) x0 + t x1 , A dual-stream model vθ = (vθa , vθv ) jointly predicts the velocity of both modalities, where the two streams interact through cross-modal attention layers. During the sampling phase, both streams are integrated in parallel via a deterministic ODE solver: m a v dxm t = vθ (xt , xt , t, c) dt,
m ∈ {a, v},
(2)
where c denotes the text condition. Diffusion Negative-aware Finetuning (DiffusionNFT). In contrast to existing diffusion-based GRPO frameworks (Liu et al., 2025a; Xue et al., 2025), which typically necessitate a transition from deterministic ODEs to SDEs, DiffusionNFT (Zheng et al., 2025) performs policy optimization directly on the forward diffusion process. The method leverages a reward r(x0 , c) to determine positive and negative policies, thereby defining a contrastive loss. The model’s velocity predictor, vθ , is encouraged toward the high-reward policy and away from the low-reward one. The core policy optimization loss is defined as: h i L(θ) = Ec,πold (x0 |c),t r∥vθ+ (xt , c, t) − v∥22 + (1 − r)∥vθ− (xt , c, t) − v∥22 , (3) The implicit positive and negative policies vθ+ and vθ− are combinations of the old policy v old and the training policy vθ , weighted by a hyperparameter β: vθ+ (xt , c, t) := (1 − β)v old (xt , c, t) + βvθ (xt , c, t),
(4)
vθ− (xt , c, t) := (1 + β)v old (xt , c, t) − βvθ (xt , c, t).
(5)
The optimality probability r ∈ [0, 1] is transformed from the unconstrained raw reward signal r
r(x0 , c) :=
1 1 + clip [A(x0 , c), −1, 1] , 2 2
A(x0 , c) =
raw
:
rraw (x0 , c) − Eπold (·|c) rraw (x0 , c) , (6) Zc
where A(x0 , c) denotes the group-wise normalized advantage of sample x0 under prompt c, Zc > 0 is a normalizing factor, which could take the form of a global reward std.
4
M OTIVATION
4.1
A DVANTAGE I NCONSISTENCY BETWEEN AUDIO AND V IDEO
In joint audio-video generation, the multimodal output needs to account for both video and audio quality simulta(-0.4, 0.4) (0.2, 0.5) (-0.4, 0.4) (0.2, 0.5) neously, yet the relationship between these two qualities has not been sufficiently analyzed. To this end, we ana(-0.3, -0.4) (0.1, -0.2) (-0.3, -0.4) (0.1, -0.2) lyze 1,400 generated samples (175 prompts, Group Size is 8) and calculate separate advantages for video and auvideo, audio advantage video, audio advantage dio rewards. We observe that the rewards for video and (-0.1, -0.1) (-0.4, 0.3) audio of a single multimodal output are not often consistent, as illustrated in Fig. 2(a). Specifically, high-quality videos are not necessarily accompanied by high-quality Figure 3: Advantage conflict between audio. On the contrary, nearly half of the samples exhibit audio and video modality advantage conflicts. We further enumerate four rollouts within a group, each represented as (video advantage, audio advantage), as shown in Fig. 3. The All Sum (Shared) strategy collapses both modalities into a single scalar, yielding an indistinguishable and dilute advantage of (−0.1, −0.1). In contrast, the Modality-based Sum (Separate) strategy aggregates each modality independently, yielding (−0.4, 0.3), which faithfully reflects that video should be penalized while audio should be encouraged, enabling more informative learning. All Sum (Shared)
4
Modality-based Sum (Separate)
arXiv preprint
4.2
G RADIENTS I MBALANCE ACROSS D IFFERENT M ODALITIES B RANCHES
The advantage inconsistency discussed above only reflects the issue in the output. We examine the internal information and gradients flow within the dual-stream model from both forward and backward perspectives, as illustrated in Fig. 2. Forward Analysis. We identify the functional roles of the Transformer blocks at different depths by selectively blocking the key-value (KV) information exchanged through audio-video cross-attention. As shown in Fig. 2(b) and (d), ablating the KV flow in shallow blocks (blocks 0-19) causes only marginal degradation in AV synchronization. In contrast, disrupting the KV flow in middle or deep blocks (blocks 20-32 for audio and 25-47 for video) leads to a substantial drop in AV alignment, with the audio branch exhibiting a relative decrease of up to ∆0.59. This indicates that shallow blocks are primarily responsible for intra-modal generation, while middle-to-deep blocks handle cross-modal audio-video interaction and alignment (AV-Sync Zone). Backward Analysis. Within the AV-Sync Zone, gradients flowing through the cross-modal interaction paths are dominant, indicating that the RL signal is correctly concentrated on the blocks truly responsible for cross-modal alignment. Meanwhile, the magnitudes of these interaction gradients gradually decay toward shallower layers. However, we observe an anomalous gradient spike in the shallow layers (Fig. 2(c)) on the audio branch, implying that a large portion of the RL gradient is erroneously injected into layers dedicated to intra-modal generation. We argue that this gradient misalignment disrupts the intra-modal audio generation process, resulting in sub-optimal results. 4.3
V2A C ROSS -ATTENTION AS AN I NTRINSIC P ROXY FOR C RITICAL R EGIONS
In joint audio-video generation, the local visual quality … Patrick says clearly: “Knowledge cannot replace friendship, ... SpongeBob smiles and responds: "I agree, of sound-emitting regions plays a decisive role in shap- SpongeBob!” but I also really want that Krabby Patty!" ing the subjective perceptual experience. However, uniform updates overlook their unequal contributions to the overall quality. This motivates us to localize such critical regions and apply differentiated levels of importance and exploration to them. While directly incorporating an external detection module offers a straightforward solution, it is computationally expensive. Instead, we observe an intrinsic indicator by analyzing the attention maps of the cross-modal attention layers within the audio branch, Figure 4: Visualization of V2A crossas illustrated in Fig. 4. We observe that these attention attention maps. maps effectively highlight the speaking subjects and their sound-emitting regions within video frames. This strong correlation suggests that the V2A crossattention map naturally serves as an intrinsic proxy for identifying critical regions in video frames.
5
M ETHODOLOGY
Motivated by the three issues and observations discussed above in Sec. 4, we propose OmniNFT, which performs fine-grained credit assignment at three corresponding levels: modality-wise advantage routing (Sec. 5.1), layer-wise gradient surgery (Sec. 5.2), and region-wise loss reweighting (Sec. 5.3). Fig. 5 illustrates the pipeline, and Alg. 1 summarizes the training procedure. 5.1
M ODALITY- WISE A DVANTAGE ROUTING
Reward-wise advantage computation. Instead of deriving a single advantage from all rewards, OmniNFT computes an independent advantage for each reward. During each sampling step, the (i) (i) model generates a group of G joint audio-video pairs {(xv , xa )}G i=1 conditioned on prompt c. (i) For each reward function k ∈ {v, a, av}, we evaluate the group to obtain raw scores {Rk }G i=1 and (i) compute a reward-wise advantage Ak following Eq. 6, which produces three decoupled advantage (i) (i) G (i) G sets {Av }G i=1 , {Aa }i=1 , and {Aav }i=1 5
arXiv preprint
policy A2V Cross Attention
T2V Cross Attention
V2A Cross Attention
AV reward 𝑅&"
𝑅&#
𝑅&$
𝑅&%
Audio reward
𝐴#!
r
𝐴%!
A2V Cross Attention
𝐸𝑞. (7)
Q (video)
AV advantage 𝐴$&
r
𝐴%&
Audio advantage
K
r
r
1-r
1-r
W
Region-wise Reweighting Sampling Steps
layer 𝑙
Video advantage
𝐴#&
𝑣!% 𝑥# |𝑐 − 𝑣 $$
Layer-wise Gradient Surgery 𝐴$!
𝐴"&! 𝐴#&! 𝐴$&! 𝐴%&! 𝐴"&
𝑣!% 𝑥# |𝑐 − 𝑣 $$
FFN
Video reward " # $ % 𝑅&! 𝑅&! 𝑅&! 𝑅&!
𝐴"!
𝐸𝑞. (12) FFN
𝑅!%
A2V Cross Attention
𝑅!$
Reward-wise Group Computation
𝑅!#
V2A Cross Attention
Modality-wise Advantage Routing 𝑅!"
T2V Cross Attention
Video Prompt Audio Prompt
T2A Cross Attention
+𝑣
Self Attention
SpongeBob and Patrick are sitting on a rock underwater, talking deeply. Patrick says clearly: “Knowledge cannot ……
𝑣!" 𝑥# |𝑐 − 𝑣 $$
𝑚𝑖𝑛! (') Self Attention
old
Prompt:
𝑣!" 𝑥# |𝑐 − 𝑣 $$
FFN
T2A Cross Attention
Self Attention
+𝑣 Omni Model 𝜋 &'(
FFN
Self Attention
Sampling
𝛼(")
Video latent state
𝛼(") detach factor
V2A Cross Attention
V
𝛼(")
V
𝐸𝑞. (8) detach
K
(audio) Q
Audio latent state
Q(audio)
K(video)
Mask W
𝐸𝑞. (11)
grad flow
Figure 5: Overview of OmniNFT. Given paired video and audio prompts, the Omni model first generates joint audio-video samples. Building on these samples, OmniNFT performs three coordinated operations: (i) independent advantages derived from video, audio, and cross-modal rewards are dispatched to their corresponding branches (Modality-wise Advantage Routing); (ii) the audioto-video cross-attention cached from the final sampling steps is converted into a critical-region mask that reweights the RL loss (Region-wise Reweighting); and (iii) during loss backward, the key-value gradients of A2V cross-attention in the shallow audio layers are partially detached, while all other gradient flows remain intact (Layer-wise Gradient Surgery).
Modality-decoupled advantage routing. With the reward-wise advantages, OmniNFT routes each advantage to the branch(es) whose output it evaluates. Specifically, the video advantage Av captures visual quality and motion, which are exclusively governed by the video branch. Analogously, the audio advantage Aa reflects audio fidelity determined by the audio branch. In contrast, the synchronization advantage Aav measures the synchronization alignment between the two modalities, and is thus broadcast as shared supervision to both branches. The composite routed advantages are: (i) (i) Ã(i) v = Av + Aav ,
(i) (i) Ã(i) a = Aa + Aav .
(7)
This design enforces modality-specific supervision for uni-modal rewards while preserving shared cross-modal supervision, enabling more informative and less conflicting reward assignment. 5.2
L AYER - WISE G RADIENT S URGERY (l)
For each Transformer block l, the A2V cross-attention takes its query Qv from the video hidden (l) (l) states and its key-value pair (Ka , Va ) from the audio hidden states. We apply a layer-wise partial detach on the these KV: K̃a(l) = α(l) sg(Ka(l) ) + (1 − α(l) ) Ka(l) , Ṽa(l) = α(l) sg(Va(l) ) + (1 − α(l) ) Va(l) ,
(8)
where sg(·) denotes the stop-gradient operator. This leaves the forward sampling unchanged but scales the backward gradient through the KV path by (1 − α(l) ). The detach ratio α(l) follows a simple schedule aligned with the identified layer functionality and the above observation: α(l) =
αs , l < L (shallow layers), 0, l ≥ L (deep layers),
(9)
with L defaults to 10 and αs is 0.1. In this way, RL gradients flow freely through the deep layers responsible for cross-modal alignment, while leakage into shallow audio layers is suppressed. 6
arXiv preprint
Algorithm 1 OmniNFT: Modality-wise Omni Diffusion RL Fine-Tuning Require: Pretrained dual-stream policy (vvref , varef ), reward functions {Rv , Ra , Rav }, prompt dataset {c}, detach ratio αs , shallow boundary L, reweighting strength w, later denoising steps T . old ref ref Initialize: vm ← vm , vθm ← vm for m ∈ {v, a}, data buffer D ← ∅. 1: for each iteration i do 2: for each sampled prompt c do // Sampling Stage old old Sample G joint outputs {(xv0,j , xa0,j )}G j=1 from (vv , va ). (j) (j) (j) G Evaluate rewards {Rv , Ra , Rav }j=1 and cache V2A cross-attention maps {Attn(l,t) }. (j) (j) (j) Modality-wise advantages & routing (Sec. 5.1): compute Av , Aa , Aav , then route: (j) (j) (j) (j) (j) (j) Ãv ← Av + Aav , Ãa ← Aa + Aav . (j) (j) 6: Convert Ãm to optimality probability rm via Eq. 6 for m ∈ {v, a}. 7: Region-wise weighting (Sec. 5.3): aggregate {Attn(j,l,t) } over l ≥ L, t ∈ T to obtain {w(j) } (j) (j) 8: D ← D ∪ {c, xv0,j , xa0,j , rv , ra , {w(j) }}. 9: end for 10: for each mini batch {c, xv0 , xa0 , rv , ra , {w(j) }} ∈ D do // Training Stage 11: Layer-wise gradient surgery (Sec. 5.2): replace audio KV in A2V cross-attention block l with (l) (l) (l) (l) K̃a = α(l) sg(Ka ) + (1 − α(l) )Ka , where α(l) = αs if l < L else 0, same for Ṽa . 12: Compute region-weighted video loss and standard audio loss, then update θ via Eq. 12. 13: end for 14: Update θold ← ηi θold + (1 − ηi )θ, and clear D ← ∅. // Online Update 15: end for
3: 4: 5:
Output: (vθv , vθa ) 5.3
R EGION - WISE R EWEIGHTING
Let Attn(l,t) ∈ RNv ×Na denote the V2A cross-attention map at block l and denoising step t, where Nv and Na are the numbers of video and audio tokens. Since the V2A attention becomes semantically meaningful in the deep AV-Sync Zone and at the later denoising steps, we aggregate attention only over these informative timesteps to obtain a per-token score: N
si =
a 1 XXX (l,t) Attni,j , |D| |T | j=1
i = 1, . . . , Nv ,
(10)
l∈D t∈T
where D = {l | l ≥ L} indexes the deep cross-modal blocks and T denotes the last few denoising steps. The score is then normalized and mapped into a region-wise weight: wi = 1 + λ ·
si − minj sj , maxj sj − minj sj
(11)
with λ > 0 controlling the reweighting strength. 5.4
OVERALL T RAINING O BJECTIVE
Following the DiffusionNFT formulation (Sec. 3), we convert the routed advantages into optimality probabilities rm for each branch m ∈ {v, a}. The region-wise weights wi are incorporated into the video branch loss to concentrate optimization capacity on perceptually critical regions, while the audio branch loss remains unchanged. Based on Eq. 3, the total training objective becomes: 1 Lall (θ) = P
Nv X
i wi i=1
|
6
(i)
wi · Lvideo (θ) +Laudio (θ). {z
region-weighted video loss
(12)
}
E XPERIMENTS
We first describe the experimental setup in Sec. 6.1, including reward models, evaluation metrics, and implementation details. We then present the main quantitative and qualitative results in Sec. 6.2, and conduct ablation studies to analyze the contribution of each component in Sec. 6.3. 7
arXiv preprint
Table 1: Main results on JavisBench (Liu et al., 2025c). Best results in green , second-best underlined. VQ: Visual Quality, AQ: Audio Quality. (↑: higher is better; ↓: lower is better). AV-Quality
Text-Consistency
AV-Consistency
AV-Synchrony
Model
Size
VQ ↑
AQ ↑
TV-IB ↑
TA-IB ↑
CLIP ↑
CLAP ↑
AV-IB ↑
AVHScore ↑
JavisScore ↑
DeSync ↓
− T2A+A2V TempoTkn (Yariv et al., 2024) TPoS (Jeong et al., 2023)
1.3B 1.0B
– –
– –
0.084 0.201
– –
0.205 0.229
– –
0.139 0.124
0.122 0.129
0.103 0.095
1.532 1.493
− T2V+V2A ReWaS (Jeong et al., 2025) See&Hear (Xing et al., 2024) FoleyCrafter (Zhang et al., 2026) MMAudio (Cheng et al., 2025)
0.6B 0.4B 1.2B 0.1B
– – – –
– – – –
– – – –
0.123 0.129 0.149 0.160
– – – –
0.280 0.263 0.383 0.407
0.110 0.160 0.193 0.198
0.104 0.143 0.186 0.182
0.079 0.112 0.151 0.150
1.071 1.099 0.952 0.849
− T2AV JavisDiT (Liu et al., 2025c) UniVerse-1 (Wang et al., 2025) JavisDiT++ (Liu et al., 2026a) LTX-2 (HaCohen et al., 2026)
3.1B 6.4B 2.1B 19B
1.291 1.357 1.462 2.038
4.478 4.839 5.049 5.197
0.263 0.272 0.282 0.272
0.143 0.111 0.164 0.170
0.302 0.309 0.316 0.311
0.391 0.245 0.424 0.412
0.197 0.104 0.198 0.232
0.179 0.098 0.184 0.223
0.154 0.077 0.159 0.192
1.039 0.929 0.832 0.569
LTX-2+ GDPO (Liu et al., 2026b) LTX-2+OmniNFT Our RL ∆
19B 19B –
3.209 3.326 +1.288
5.523 5.715 +0.518
0.265 0.261 -0.011
0.184 0.189 +0.019
0.308 0.310 -0.001
0.428 0.445 +0.033
0.233 0.262 +0.030
0.223 0.257 +0.034
0.185 0.220 +0.028
0.412 0.269 -0.300
6.1
E XPERIMENTAL S ETUP
Reward models. We utilize VideoAlign (Liu et al., 2025b) and HPSv3 (Ma et al., 2025) scores as rewards to evaluate video quality, while Audiobox Aesthetics (Tjandra et al., 2025) is employed as the reward for audio quality. To ensure cross-modal consistency, we adopt the CLAP (Wu et al., 2023b) score as the reward for audio-text alignment, and the synchronization score (Desync) (Iashin et al., 2024) as the reward for audio-visual synchronization. Evaluation Metrics. We report results across four complementary dimensions defined in JavisBench: (i) AV-Quality: Visual quality (VQ) and Audio quality (AQ) from VideoAlign score for uni-modal generation fidelity. Furthermore, we utilize VBench (Huang et al., 2024) as an additional benchmark to assess the quality of the generated videos. (ii) Text-Consistency: Text-Video and TextAudio ImageBind (Girdhar et al., 2023) similarity (TV-IB, TA-IB), CLIP (Radford et al., 2021) score, and CLAP score for text-to-modal alignment; (iii) AV-Consistency: Audio-Video ImageBind similarity (AV-IB) and AVHScore for semantic coherence; (iv) AV-synchronization: JavisScore (Liu et al., 2025c) and DeSync for synchronization between audio and video. 6.2
M AIN R ESULTS AND A NALYSIS
Quantitative Analysis. Tab. 1 and Fig. 6 report results on JavisBench and VBench. Compared with LTX-2 (19B) and GDPO, OmniNFT achieves the best overall performance across perceptual quality, cross-modal consistency, and temporal synchronization. For perceptual quality, VQ improves from 2.038 to 3.326 (+63.2%) and AQ from 5.197 to 5.715 (+10.0%) over LTX-2, with notable VBench gains in imaging quality (+10.5%). For cross-modal consistency, TA-IB surpasses both LTX-2 (+15.2%) and the larger LTX-2.3 (22B). For synchronization, OmniNFT reduces DeSync from 0.569 to 0.269 (-52.7%), substantially outperforming GDPO (0.412). We note that TV-IB and CLIP do not improve under either our method or GDPO, suggesting that text–video semantic alignment remains challenging.
Setting
AV-Quality
Text-Consistency
VQ ↑
CLIP ↑
CLAP ↑
AVHScore ↑
JavisScore ↑
0.310 0.312
0.445 0.427
0.257 0.242
0.220 0.204
0.308 0.310 0.310
0.429 0.445 0.438
0.249 0.257 0.260
0.212 0.220 0.223
AQ ↑
AV-Consistency
(a) Gradient Surgery Layer Selection Shallow layers (default) Deep layers
3.326 3.083
5.715 5.577
(b) Region-wise Reweighting Loss Weight λ λ = 1.25 λ = 1.50 (default) λ = 1.75
3.150 3.326 2.977
5.495 5.715 5.714
Table 2: Ablation study on detach layers and coefficient λ.
Aesthetic Quality
Imaging Quality
Subject Consistency
0.775 0.64 0.631 0.750 0.747 0.725 0.62 0.700 0.60 0.667 0.675 0.586 0.585 0.650 0.646 0.58 0.625 0.600 0.56
LTX-2
LTX-2.3
0.98 0.967 0.97 0.96 0.95 0.94 0.931 0.934 0.93 0.92 0.91
LTX-2+OmniNFT
Figure 6: Vbench Results.
Qualitative Analysis. Fig. 7 highlights four representative cases, each showcasing a distinct strength of OmniNFT: sharper frames and natural motion in the cartoon scene (visual quality), 8
arXiv preprint
SpongeBob and Patrick are sitting on a rock underwater, talking deeply. Patrick looks at Patrick and says : “Knowledge cannot replace friendship, SpongeBob!”…
A white rooster with a bright red comb and wattle stands on a weathered wooden beam that cuts diagonally across the lower frame. Its head is turned…
In a medium close-up, a young woman with blonde shoulder-length hair stands in a lavender field under a twilight sky, her gaze directed toward the camera…
A medium close-up frames two men facing each other in profile, standing noseto-nose in a tense, cinematic confrontation within a dimly lit bank. The man on…
Figure 7: Qualitative examples of joint audio-video generation by OmniNFT. The four cases illustrate improvements across different aspects: enhanced visual quality, improved audio fidelity, better speech-lip synchronization, and coherent multi-speaker scenes.
richer ambient textures in the rooster case (audio fidelity), tight waveform–lip alignment in the lavender-field close-up (speech–lip sync), and coherent identities with alternating vocal activity in the confrontation scene (multi-speaker consistency). These results confirm that OmniNFT produces temporally synchronized and semantically coherent audio–video content across diverse scenarios. 6.3
A BLATION S TUDIES
Effectiveness of Each Component. We validate the three key designs of OmniNFT by progressively incorporating them into the vanilla RL baseline, as shown in Tab. 3. Decoupling advantages across modalities alleviates reward inconsistency, bringing clear improvements in cross-modal consistency and synchrony. Shielding shallow audio layers from dominant video gradients further enhances audio fidelity and text-audio alignment, while focusing synchronization-critical regions provides fine-grained credit assignment that pushes AV-consistency and synchrony to the best results. Notably, our designs introduce only a negligible overhead. Hyperparameter Analysis. For gradient surgery layer L selection, detaching video-to-audio gradients at shallow layers (L < 10) consistently outperforms detaching at deep layers(L > 20) in Tab. 2, which aligns with our observation. For the region-wise reweighting coefficient λ, a moderate value (λ = 1.50) achieves the best trade-off: smaller values under-emphasize critical regions, while larger values influence visual quality. Table 3: Ablation results on each design (↑: higher is better; ↓: lower is better). Best results in bold. AV-Quality
Text-Consistency
AV-Consistency
AV-Synchrony
Efficiency
Model
VQ ↑
AQ ↑
TV-IB ↑
TA-IB ↑
CLIP ↑
CLAP ↑
AV-IB ↑
AVHScore ↑
JavisScore ↑
DeSync ↓
Time ↓
LTX-2 (HaCohen et al., 2026) LTX-2 + vanilla RL
2.038 3.209
5.197 5.523
0.272 0.265
0.170 0.184
0.311 0.308
0.412 0.428
0.232 0.233
0.223 0.223
0.192 0.185
0.569 0.412
– 23.9h
3.264 3.246 3.326
5.399 5.917 5.715
0.266 0.264 0.261
0.186 0.192 0.189
0.306 0.311 0.310
0.430 0.438 0.445
0.248 0.252 0.262
0.240 0.247 0.257
0.199 0.209 0.220
0.322 0.334 0.269
23.9h 24.1h 24.1h
+ Modality-wise advantage routing + Layer-wise gradient surgery + Region-wise loss reweighting (Full)
9
arXiv preprint
7
C ONCLUSION
We present OmniNFT, a modality-aware online diffusion RL framework. Through three complementary innovations: modality-wise advantage routing, layer-wise gradient surgery, and regionwise loss reweighting, OmniNFT effectively resolves advantage inconsistency, gradient imbalance, and uniform credit assignment. Extensive experiments on JavisBench and VBench demonstrate our effectiveness.
R EFERENCES Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023a. Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22563–22575, 2023b. Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Leo Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. OpenAI Blog, 2024. Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 28901–28911, 2025. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15180–15190, 2023. Junmin Gong, Sean Zhao, Sen Wang, Shengyuan Xu, and Joe Guo. Ace-step: A step towards music generation foundation model. arXiv preprint arXiv:2506.00045, 2025. Google DeepMind. Veo: A text-to-video generation system. Technical report, Google, 2024. URL https://storage.googleapis.com/deepmind-media/veo/ Veo-3-Tech-Report.pdf. Accessed: 2025-09-24. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103, 2024. Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifschitz, Dudu Moshe, Eitan Porat, et al. Ltx-2: Efficient joint audio-visual foundation model. arXiv preprint arXiv:2601.03233, 2026. Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, and Bo Zhang. Tempflow-grpo: When timing matters for grpo in flow models. arXiv preprint arXiv:2508.04324, 2025. Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 10
arXiv preprint
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807–21818, 2024. Vladimir Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Synchformer: Efficient synchronization from sparse cues. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5325–5329. IEEE, 2024. Yujin Jeong, Wonjeong Ryoo, Seunghyun Lee, Dabin Seo, Wonmin Byeon, Sangpil Kim, and Jinkyu Kim. The power of sound (tpos): Audio reactive video generation with stable diffusion. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7822–7832, 2023. Yujin Jeong, Yunji Kim, Sanghyuk Chun, and Jiyoung Lee. Read, watch and scream! sound generation from text and video. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 17590–17598, 2025. Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ann Heng, and Hongsheng Li. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint arXiv:2505.00703, 2025. Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Yiming Cheng, Miles Yang, Zhao Zhong, and Liefeng Bo. Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde. arXiv preprint arXiv:2507.21802, 2025. Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470, 2025a. Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Menghan Xia, Xintao Wang, et al. Improving video generation with human feedback. arXiv preprint arXiv:2501.13918, 2025b. Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Jiebo Luo, Ziwei Liu, Hao Fei, et al. Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization. arXiv preprint arXiv:2503.23377, 2025c. Kai Liu, Yanhao Zheng, Kai Wang, Shengqiong Wu, Rongjunchen Zhang, Jiebo Luo, Dimitrios Hatzinakos, Ziwei Liu, Hao Fei, and Tat-Seng Chua. Javisdit++: Unified modeling and optimization for joint audio-video generation. arXiv preprint arXiv:2602.19163, 2026a. Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, et al. Gdpo: Group rewarddecoupled normalization policy optimization for multi-reward rl optimization. arXiv preprint arXiv:2601.05242, 2026b. Chetwin Low, Weimin Wang, and Calder Katyal. Ovi: Twin backbone cross-modal fusion for audiovideo generation. arXiv preprint arXiv:2510.01284, 2025. Yifu Luo, Xinhao Hu, Keyu Fan, Haoyuan Sun, Zeyu Chen, Bo Xia, Tiantian Zhang, Yongzhe Chang, and Xueqian Wang. Reinforcement learning meets masked generative models: Maskgrpo for text-to-image generation. arXiv preprint arXiv:2510.13418, 2025. Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15086–15095, 2025. 11
arXiv preprint
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PmLR, 2021. Team Seedance, Heyi Chen, Siyan Chen, Xin Chen, Yanfei Chen, Ying Chen, Zhuo Chen, Feng Cheng, Tianheng Cheng, Xinqi Cheng, et al. Seedance 1.5 pro: A native audio-visual joint generation foundation model. arXiv preprint arXiv:2512.13507, 2025. Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, et al. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.05139, 2025. Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, and Gang Yu. Universe-1: Unified audio-video generation via stitching of experts. arXiv preprint arXiv:2509.06155, 2025. Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7623–7633, 2023a. Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, 2023b. Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Opendomain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7151–7161, 2024. Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818, 2025. Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-video generation via text-to-video model adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6639–6647, 2024. Guohui Zhang, Hu Yu, Xiaoxiao Ma, Yaning Pan, Hang Xu, and Feng Zhao. Maskfocus: Focusing policy optimization on critical steps for masked image generation. arXiv preprint arXiv:2512.18766, 2025a. Guohui Zhang, Hu Yu, Xiaoxiao Ma, JingHao Zhang, Yaning Pan, Mingde Yao, Jie Xiao, Linjiang Huang, and Feng Zhao. Group critical-token policy optimization for autoregressive image generation. arXiv preprint arXiv:2509.22485, 2025b. Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, Bin Liu, and Kai Chen. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. International Journal of Computer Vision, 134(1):46, 2026. Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, and Ming-Yu Liu. Diffusionnft: Online diffusion reinforcement with forward process. arXiv preprint arXiv:2509.16117, 2025.
12