MLVC: A Multi-platform Learned Video Codec for Real-World Deployment
arXiv:2606.28027v1 [eess.IV] 26 Jun 2026
Tanel Pärnamaa , Martin Lumiste , Ardi Loot , Evgenii Indenbom , Andrei Znobishchev , and Ando Saabas Microsoft Corporation {taparnam, ardiloot, eindenbom, aznobishchev, ansaaba}@microsoft.com
Abstract. Neural video codecs have surpassed classical codecs in coding efficiency but remain impractical for deployment due to cross-platform incompatibility and high computational cost. Existing quantization-based solutions fail to produce deterministic results across diverse hardware platforms, leading to catastrophic decoding failures. We introduce MLVC, a hardware-robust neural video codec designed for practical cross-platform inference. The key idea is to explicitly transmit scale parameters through the hyperprior, which guarantees entropy coding consistency across devices without requiring bit-exact arithmetic. While this increases bitrate overhead, we recover most of the coding efficiency through architectural improvements (gated memory, ReGLU activation), a long-term reference recovery mechanism, and domain-specific perceptual training. On the VCD video conferencing benchmark, MLVC achieves >70% BDrate (MOS) improvement over hardware HEVC, the strongest deployable baseline, while reaching subjective quality competitive with DCVC-RT, which cannot operate across diverse platforms. Both the encoder and decoder run at 100 FPS on average on commodity NPUs from Apple, Intel, and Qualcomm. MLVC is the first neural video codec to combine competitive compression performance, real-time speed, and crossplatform robustness across diverse consumer devices, making it suitable for widespread deployment. Code will be released. Keywords: Neural Video Compression · Cross-Platform Robustness · Real-Time Video Processing
1
Introduction
Neural video compression has matured to the point where learned codecs consistently outperform traditional standards. [17, 24, 34, 38] Recent neural codecs achieve 60-70% bitrate savings over H.265 at equivalent perceptual quality and surpass even the latest ECM standard in rate-distortion performance. [24, 34] However, despite all this research progress, neural codecs remain unused in production systems like Zoom, Teams or WebRTC. Two critical barriers prevent deployment. The first is computational efficiency: most neural codecs are evaluated on high-end datacenter GPUs. For practical impact in applications like video conferencing, neural codecs need to
2
T. Pärnamaa et al.
Original
DCVC-RT
HEVC-QSV (58.7 kbps)
MLVC (59.8 kbps)
Fig. 1: Neural codecs fail across diverse platforms. A video is encoded on an Apple M3 NPU and then decoded on another device (Intel Lunar Lake NPU). Existing stateof-the-art neural video codecs (e.g., DCVC-RT [24]) suffer from catastrophic decoding failures due to numerical divergence between platforms, even when using quantization or calibration information [50]. Classical codecs like H.265 and the proposed neural codec MLVC both decode reliably across diverse platforms, while MLVC achieves significantly higher visual quality at the same bitrate.
run in real-time on the neural processing units (NPUs) present in commodity consumer devices. The second and more fundamental barrier is cross-platform compatibility. Many applications, such as video conferencing systems, inherently require encoder and decoder to run on different devices, often with hardware from different vendors. Figure 1 illustrates the severity of this problem: when we encode a video on an Apple M3 NPU and decode on an Intel NPU, the state-of-the-art neural codec DCVC-RT [24], even in its quantized form, produces completely corrupted output. To address these barriers, we introduce MLVC, a neural video codec built for real world deployment. We specifically focus on model performance on NPUs rather than high-performance GPUs to show the practicality of our approach. Our contributions are as follows: – We develop a scale sharing mechanism and transmit scale indices within the hyperprior, enabling reliable inference across diverse hardware platforms without requiring bit-exact arithmetic. – We introduce architectural improvements (gated memory for long-term modeling, hardware-compatible ReGLU activations), training and inference-time strategies (I-frame dropout, long-term reference recovery) that recover the coding efficiency lost to increased bitrate overhead, achieving >70% BDrate improvement over HEVC-QSV in subjective quality (MOS) on a video conferencing benchmark (VCD). – We demonstrate real-time performance (100+ FPS average for both encoding and decoding) on commodity NPUs from Apple, Intel, and Qualcomm, making MLVC the first neural video codec practical for widespread deployment on diverse consumer devices.
MLVC: Cross-Platform Neural Video Codec
2
Related Work
2.1
Learned Compression
3
The hyperprior architecture introduced by Ballé et al. [11] forms the foundation of modern learned compression [13, 19,34]. This is commonly combined with autoregressive prior [40]. Since fully autoregressive priors are computationally prohibitive, recent work explores efficient approximations including checkerboard patterns [20], channel-wise autoregression [41] and hybrid spatial-channel models [32]. In learned video compression, numerous architectures since [36] have advanced the field by improving motion modeling, temporal hierarchy, and entropy estimation [8,22,35,44,47,53]. Video models often employ a third temporal prior to exploit inter-frame correlation. Li et al. [31] showed that contextual coding outperforms explicit residual coding, with subsequent advances often related to better context modelling [33, 46]. The current state-of-the-art models [24, 34] achieve improved rate-distortion performance over the reference software codec for ECM [2]. 2.2
Mobile and Low-Latency Codecs
While most learned compression research prioritizes improving rate-distortion performance, a parallel direction focuses on computational efficiency for edge development. Codecs based on overfitting such as C3 [26] and Cool-chic video [30] achieve very low decoding complexities, but require costly encoding. MobileCodec [29] achieves 720p video decoding on a mobile phone with a neural accelerator. MobileNVC [51] extends this to 1080p video. DCVC-RT achieves over an 18× FPS speed-up compared to DCVC-FM [34] on NVIDIA A100 GPUs, though edge device performance is not reported. These works employ two main strategies for efficiency. First, integer quantization to improve inference speed and also provide cross-platform consistency through bit-exact arithmetic. However, cross-platform validation is limited: MobileNVC [51] reports no cross-platform testing, and DCVC-RT [24] validates only across NVIDIA GPUs. Since NVIDIA GPUs allow for low level kernel control, their validation demonstrates only single-vendor and not true cross-platform compatibility. We will argue that these models are unlikely to work across heterogeneous NPU platforms. Secondly, one of the main performance bottlenecks on edge devices has been the expensive pixel or feature space warping operator. As a result, most mobile methods redesign motion compensation to either use convolutions [24, 29], or simplify the warp operation [51]. 2.3
Cross-platform Consistency
The cross-platform consistency problem arises because entropy coding requires encoder and decoder to use identical probability distributions. In the case of DCVC-RT, the scale parameters σ characterize these distributions. Even tiny
4
T. Pärnamaa et al.
numerical differences, for example those caused by non-deterministic floatingpoint computations, can cause the entropy decoder to select different lookuptable indices, resulting in incorrect latent values. These errors cascade through temporal prediction, causing catastrophic failures (Fig. 1). We refer readers to Tian et al. [50] for detailed exposition. Prior work addresses this through three main approaches, each with fundamental limitations. Network Quantization. The most common approach quantizes entropy model computations to integer arithmetic, aiming for bit-exact results across platforms [10,15,18,24,27,48]. In theory, the subnetwork responsible for producing σ could run in fixed-point integer arithmetic to guarantee cross-platform consistency, as traditional codecs do. In practice, this is difficult to achieve on commodity NPUs. First, many works adopt nonstandard bit-widths such as INT16 [24, 27, 28] and custom quantization scales that the NPU toolchains don’t support – attempting to convert such models typically results in a compiler error, preventing acceleration entirely. Second, even for the relatively standard INT8 convolution with INT32 accumulators, some NPUs (e.g. Apple NE) do not execute true INT8 but simulate it via FP16.1 Third, across NPUs that do implement true INT8, compiler choices (kernel selection, operator fusion, reduction order), approximations, and rounding modes prevent bit-exact parity. Unlike with traditional video codecs, there is no bit-exact reference specification for NPU vendors and quantized models can diverge more than analogous FP models due to compounding off-by-one errors. For these reasons, we use FP16 for inference, which is broadly supported across NPUs. We present detailed crossplatform divergence measurements for both floating point and integer layers in the Supplementary Material. Calibration Information. Tian et al. [50] transmit an additional side stream of calibration information to decrease the likelihood of cross-platform index mismatches. While this reduces mismatch probability, it cannot provide hard guarantees - catastrophic failures remain possible, just less frequent. This probabilistic approach is insufficient for production deployment where reliability is critical. Moreover, their experiments relied only on a single 96-frame video from the UVG dataset, encoded and decoded in FP32 precision. Neural processing units typically use FP16 rather than FP32. Since FP16 has a unit roundoff about four orders of magnitude larger, it is far more susceptible to roundingerror drift. In our cross-device experiments (see supplementary material), the method fails in FP16: we observe float-index deviations |δ| > 0.5, a situation where calibration cannot prevent index mismatches. Fixed-prior Approaches. More recently, Tian et al. [49] proposed to transmit discrete codebook indices, eliminating entropy modelling entirely. This effortlessly avoids catastrophic cross-platform failures. However, their experimental results do not effectively validate competitive compression performance. Most critically, their comparison uses mismatched GOP settings: GOP=32 for their 1
Vendor documentation indicates the int8-int8 compute path is only enabled starting from M4, released in 2024. https://apple.github.io/coremltools/docs-guides/ source/opt-overview.html
MLVC: Cross-Platform Neural Video Codec
5
model versus GOP=12 for H.264/H.265, giving their method an unfair advantage since classical codecs must encode expensive I-frames more frequently.
3
Proposed Method
We use the DCVC-RT [24] floating point model as a starting point, as it shows state-of-the-art compression ratio, while remaining lightweight and fast. An overview of the video coding framework is shown in Fig. 2. While our experiments focus on DCVC-RT, the proposed methods rely on components common to most modern video codec. The scale-sending mechanism applies to any codec using adaptive entropy priors (e.g. hyperprior, autoregressive prior, temporal prior), the memory module to any recurrent codec maintaining temporal state, and the LTR mechanism to codecs that use reference features. 3.1
Avoiding Catastrophic Failures
Catastrophic decoding failures in existing neural video codecs are caused by mismatches in entropy model parameters between the encoder and decoder, specifically the scale parameters in DCVC-RT. While eliminating all adaptive priors would trivially eliminate catastrophic decoding failures, this significantly degrades BD-rate performance in practice (see row 0-prior in Table 1). The alternative, transmitting all scale parameters explicitly, incurs prohibitive overH ×W head, since there are 16 16 × Cy scale parameters, comparable in magnitude to the encoded frame itself. However, scale parameters exhibit strong spatial and cross-channel correlation that can be exploited for compression. We reduce the parameter count by a factor s2 · r (typically 128×) through structured parameter sharing, then entropy-code the remaining parameters within the hyperlatent representation. This achieves entropy coding consistency with acceptable bitrate overhead. C ×H ×W
Scale Sending Mechanism. We derive scale indices I ∈ N0 y y y from the quantized hyperlatent ẑ ∈ ZCz ×Hz ×Wz through deterministic expansion. Since ẑ is coded with a fixed factorized entropy model, it is identical on both the encoder and decoder sides, guaranteeing that the resulting scale parameters are also identical and thus avoiding catastrophic decoding failures. This design exploits two observations: (1) neighboring spatial locations often require similar scales, and (2) groups of channels often share statistical properties. Let r ∈ N denote the channel repetition factor and s = Hy /Hz the spatial expansion factor. The generation process comprises three steps: \boldsymbol {I}^{\text {base}} &= |\hat {\mathbf {z}}_{1:C_y/r}| \in \mathbb {N}_0^{C_y/r \times H_z \times W_z} \\ \boldsymbol {I}_{c,h,w} &= \boldsymbol {I}^{\text {base}}_{\lfloor c/r \rfloor ,\, \lfloor h/s \rfloor ,\, \lfloor w/s \rfloor } \\ \boldsymbol {\sigma }_{c,h,w} &= \operatorname {lookup}(\boldsymbol {I}_{c,h,w}) (3)
T. Pärnamaa et al. e
Ft-1
yt
Entropy Model
μt
𝒩(0, σt2)
Q
ŷt
e
Ft
State Extractor
AE
AD Bits
ŷt Ft-1
Decoder
Ft-1
σt
ft
Ft
Recon.Net
xt
Entropy Parameter Network
Encoder
6
x̂ t
Fig. 2: Overview of the video coding framework adopted in this work, following modern learned codecs (e.g., DCVC-RT). The encoder maps the current frame xt and temporal context Ft−1 to the latent representation yt . The Entropy Parameter Network, cone ditioned on yt and the temporal prior context Ft−1 , produces the mean µt and scale σt parameters. The mean is subtracted from yt , and the residual is quantized (Q) and arithmetic encoded (AE) under a zero-mean Gaussian entropy model N (0, σt2 ) to obtain the bitstream. On the decoder side, arithmetic decoding (AD) recovers ŷt , and the mean µt is added back. The decoder, further conditioned on temporal context Ft−1 , produces the decoded feature ft , from which the reconstruction network (Recon. Net) yields the reconstructed frame x̂t . Finally, the state extractor derives the updated temporal contexts Ft and Fte from ft for the next time step.
where c ∈ [0, Cy ), h ∈ [0, Hy ), w ∈ [0, Wy ). The absolute value ensures nonnegative indices, the expansion maps each hyperlatent position to an s × s spatial block and replicates each channel r times, and the final lookup retrieves quantized scale values from a fixed codebook. Notably, the same scale parameters are shared across all groups in the autoregressive prior, so all groups can be arithmetic-decoded in a single pass without interleaving arithmetic decoding and neural network calls. Figure 3 shows the standard entropy model of DCVCRT, Fig. 4 shows the proposed scale-sending variant, and Fig. 5a details the individual steps. 3.2
Reducing Divergence
While our scale-sending approach eliminates catastrophic entropy decoding failures, floating-point differences still cause gradual divergence in the features and reconstructed frames. Since encoder and decoder maintain separate feature buffers that are updated recursively, even small per-frame differences compound over long prediction chains. Without a synchronization mechanism, the divergence can grow arbitrarily large. We address this through three strategies: a novel prediction chain control via long-term reference recovery inspired by traditional codecs, hardware-compatible activation functions, and periodic synchronization via I-frames. Long-Term Reference Recovery. The most straightforward method for mitigating divergence over long prediction chains is to reduce the chain length by employing a small I-frame period (e.g., 64 frames). Although this approach can effectively
MLVC: Cross-Platform Neural Video Codec
zt
Q
ẑt
AE
AD Bits
ẑt
Hyper Decoder
yt
Hyper Encoder
Factorized Entropy Model
pth
Prior Fusion
7
ŷt,1
ŷt,2
AD
AD
σt,1 μt,1
σt,2 μt,2 Spatial Prior
ptt e
Ft-1
Temporal Prior
ŷt,1
Fig. 3: Entropy Parameter Network in modern learned codecs (e.g., DCVC-RT) incorporating hyper, temporal and spatial prior. The hyper encoder transforms yt into zt , which is quantized and arithmetic coded using a factorized entropy model to yield ẑt . The hyper decoder then produces the hyperprior pht . Because the factorized entropy model is identical and fixed on both the encoder and decoder sides, coding of ẑt does not cause catastrophic decoding failures. The temporal prior maps the temporal e context Ft−1 to ptt , and the Prior Fusion module merges pht and ptt to produce the entropy parameters (σt,1 , µt,1 ) used to arithmetic code the first latent group ŷt,1 . The Spatial Prior then conditions autoregressively on the already decoded ŷt,1 to produce (σt,2 , µt,2 ) for the second group ŷt,2 . However, the σt,i can diverge between the encoding and decoding side, leading to catastrophic decoding failures.
address divergence, it incurs a substantial cost in terms of rate-distortion performance due to the use of expensive intra frames. As an alternative, a less costly mechanism is proposed, inspired by long-term reference (LTR) recovery techniques in traditional codecs. Figure 5b illustrates the core concept of this approach: rather than relying exclusively on I-frames, proactive LTR recovery frames are periodically introduced to manage the prediction chain. This strategy reduces the frequency of expensive I-frames while maintaining the same number of frames in the prediction chain, resulting in an overall improvement in BD-rate. Beyond reducing the prediction chain, proactive LTR recovery also preserves its original advantage of enhancing codec robustness against packet loss by providing periodic recovery frames without explicit requests, as in traditional codecs. It should be noted that, on a single platform with a fixed intra period, the LTR mechanism would strictly regress BD-rate due to the introduction of additional redundancy. However, in cross-platform scenarios, this reduction in prediction chain length can improve compression efficiency, as reference divergences may otherwise lead to significant quality degradation in the absence of LTR. Hardware-Compatible Activations. Many NPUs implement nonlinear activation functions through piecewise approximations rather than exact computation. Since these approximations are vendor-specific and unstandardized, even small differences compound through the deep network, causing cross-platform divergence. Furthermore, complex activations such as WSiLU often lack optimized NPU kernels, resulting in reduced inference speed. We perform a sweep over
8
T. Pärnamaa et al.
zt
Q
ẑt
AE
Abs
σt
Lookup
ẑt
AD
μt,1
Slice
AD Bits
Expand
Hyper Decoder
yt
Hyper Encoder
Factorized Entropy Model
pth
Prior Fusion
ŷt,1 ŷt,2 μt,2 Spatial Prior
ptt e
Ft-1
Temporal Prior
ŷt,1
Fig. 4: Entropy Parameter Network using the proposed scale-sending mechanism. A slice of the hyper decoder output is passed through Abs, Expand, and Lookup operations (detailed in Fig. 5a) to deterministically obtain the scale parameters σt from the hyperlatent ẑt . Since ẑt is coded with a fixed factorized entropy model, σt is identical on both the encoder and decoder sides, avoiding catastrophic decoding failures. The mean parameters µt,i are still modeled autoregressively: Prior Fusion merges the hyper prior pht and temporal prior ptt to produce µt,1 , and the Spatial Prior conditions on the already-decoded ŷt,1 to produce µt,2 . A further benefit over the standard approach (Fig. 3) is decoding speed: because σt is available before decoding any latent group, all groups can be arithmetic-decoded in a single AD pass. In contrast, the standard approach requires interleaving arithmetic decoding and neural network calls for each group.
common activation functions supported by Apple Neural Engine [6] and find that only ReLU and LeakyReLU incur no error compared to the true value, we therefore restrict our architecture to use these. While simpler activations incur a BD-rate penalty, they are necessary for reliable cross-platform operation. Regular I-Frame Period. Even with LTR recovery and simple activations, feature buffers still diverge over extended sequences due to fundamental floating-point precision limits. We therefore employ periodic I-frames to fully resynchronize encoder and decoder feature states. While our model is able to handle very long prediction chains on a single platform, regular I-frames are necessary for practical deployment across multiple hardware platforms. This periodic synchronization incurs BD-rate overhead compared to single-platform operation but remains necessary for reliable cross-platform decoding. This differs from traditional codecs, which could run cross-platform without periodic I-frames. 3.3
Unified I-Frame and P-Frame Model
Most learned video codecs employ separate models for I-frames and P-frames, even though these architectures share substantial structural overlap. Maintaining two distinct models increases both storage requirements and deployment complexity. For instance, in DCVC-RT [24], the I-frame and P-frame models require 174 MB and 79 MB respectively in FP32. To reduce this overhead, we simplify to a single unified model by defining I-frames as P-frames with a uniform gray
MLVC: Cross-Platform Neural Video Codec Intra period = 12 I0
|ẑt,:c|
0
1
2
0
slice abs
ẑt
0 -1 2
0
expand
0000 0000 1111 1111 0000 0000 1111 1111 2222 2222 0000 0000
000 0.11 111 0.24 000 0.24 111 000 000 111 111 0.37 222 0.37 000 0.11 222 0.11 000
0.11
lookup
0.11 0.11 0.24 0.24
2222 2222 0000 0000
0.37 0.37 0.11
scale indices, same shape as ŷt
scales
000 222 222 000 0.11
9
Intra period = 12, LTR period = 4 I0
P1
P4
P8
P1
P4
P8
P2
P5
P9
P2
P5
P9
P3
P6
P10
P3
P6
P10
P7
P11
P7
P11
Prediction chain=12 frames
(a) Scale sending architecture.
Prediction chain=7 frames
(b) LTR prediction structure.
Fig. 5: (a) A visualization of the steps of the scale-sending mechanism described in Sec. 3.1 (b) Comparison of prediction structures with and without long-term reference (LTR) frames. (b, left) Standard encoding with I-frame period of 12 creates prediction chains up to 12 frames long. (b, right) Using LTR period of 4 (starting from frame 1) reduces maximum chain length to 7 frames and improves robustness to packet loss. In practice, LTR period is another inference parameter, similar to intra and reset period.
reference image (0.5 in YUV colorspace). This eliminates the need for a separate I-frame model while maintaining the ability to encode frames without temporal dependencies. Although this simplification introduces a BD-rate degradation, it reduces engineering and deployment complexity. Given that practical systems prioritize simplicity and maintainability, this trade-off is well justified. 3.4
Improving Quality
Memory. While DCVC-RT employs a recurrent architecture with features serving as temporal state, this representation has limited capacity for preserving long-term information, which is important for reconstructing temporarily occluded objects or maintaining consistency across frames. Drawing inspiration from gated recurrent architectures [21], we augment the decoder with an explicit long-term memory state that enables the model to selectively retain and retrieve information over longer temporal horizons. The memory-enhanced decoder operates as follows: \begin {aligned} \mathbf {f}_{in} &= \text {Decoder}(\hat {\mathbf {y}_t}, \mathbf {F}_{t-1}) \\ [\mathbf {s}_t, \mathbf {g}_f, \mathbf {g}_o] &= \text {Split}_{3}(\mathbf {f}_{in}) \\ \mathbf {m}_t &= \sigma (\mathbf {g}_f) \odot \mathbf {m}_{t-1} + (1 - \sigma (\mathbf {g}_f)) \odot \tau (\mathbf {s}_t) \\ \mathbf {f}_{out} &= \sigma (\mathbf {g}_o) \odot \tau (\mathbf {m}_t) \cdot \mathbf {q}_{dec} \end {aligned}
(4)
where Ft−1 is the temporal context, qdec is a learnable decoder quantization step, mt is the memory state, σ(·) and τ (·) denote the gate and memory activation functions, and ⊙ denotes element-wise multiplication. The forget gate gf controls information retention from the previous memory state, while the output gate go modulates the contribution to the output features. Compared to the baseline decoder, the memory variant only triples the output channels of the final 1×1 convolution layer. The activation functions can be
10
T. Pärnamaa et al.
standard (σ=sigmoid, τ =tanh) or piecewise-linear approximations using ReLU for hardware-friendly deployment. Concurrent to our work, a recent study also introduced a memory mechanism for neural video compression [17]. In contrast, our design focuses on real-time, multi-platform deployment, emphasizing hardware compatibility and temporal robustness. ReGLU. The model performance is highly sensitive to activation function choice. Moreover, optimal activation functions differ across architectures. While crossplatform requirements restrict us to simple functions (ReLU, LeakyReLU), we can improve expressiveness through gating mechanisms that use only crossplatform-compatible operations. Inspired by [45], we find that a channel split gating variant of ReGLU improves performance for the MLVC architecture without increasing encoder–decoder divergence: \text {ReGLU}(\mathbf {x}) = \mathbf {x}_{:C/2} \odot \text {ReLU1}(\mathbf {x}_{C/2:})
(5)
where x:C/2 and xC/2: denote the first and second halves of channels, and ReLU1(·) = min(ReLU(·), 1) caps activations at 1. The capping stabilizes training by preventing large activations from the multiplicative gating. We apply ReGLU in DCVC-RT’s depth-wise convolution blocks, replacing the channel-wise addition with multiplicative gating, while using LeakyReLU elsewhere. I-Frame Dropout. To train a single model to handle both standard P-frames and I-frames without requiring a separate intra codec, we introduce I-frame dropout during training: with probability p, the model receives a constant gray frame in place of a true I-frame. We use p = 0.5 in our experiments. The effectiveness of this approach depends on the I-frame period: shorter periods benefit more, as I-frame conditions are encountered more frequently during inference. Perceptual Losses. Cross-platform architectural constraints create a cumulative BD-rate penalty, and even with the above improvements, a gap remains compared to unconstrained architectures when measured by PSNR. However, PSNR does not reflect subjective quality. For real-world applications, perceptual optimization can matter more than pixel-wise accuracy. Moreover, different applications have different perceptual priorities - video conferencing prioritizes facial detail, while screen sharing prioritizes text clarity. We can narrow the perceptual quality gap by tailoring the codec to specific use cases through perceptual losses and region-of-interest (ROI) weighting. Our subjective evaluation demonstrates that by adapting the codec to video conferencing, we achieve comparable results to reference DCVC-RT checkpoint despite lower PSNR. Specifically, we use a perceptual loss with two components. First, we use the common LPIPS [54] loss to improve texture retention. In addition, inspired by the success of [25, 37] in the CLIC [1] challenge, we use a region of interest (ROI) mask to weight the loss pixels. We use a face-segmentation model [14, 55]
MLVC: Cross-Platform Neural Video Codec
11
Table 1: Ablation study results showing PSNR-based BD-rate (%) in YUV420 colorspace relative to HEVC-QSV anchor in same- and cross-platform scenarios. Arch
I
IP
RP
DCVC-RT ✓ DCVC-RT × DCVC-RT ×
-1 -1 64
64 WSiLU × 64 WSiLU × 0 WSiLU ×
× × ×
× × ×
× × ×
-69.6 -68.6 -59.4
-55.9
∞ ∞ ∞
64 64 64 64 64 0 0 0 0
× × × ✓ ✓ ✓ ✓ ✓ ✓
× × × × × × ✓ ✓ ✓
× × × × × × × ✓ ✓
20.5 -54.5 -49.2 -52.1 -56.6 -41.1 -46.1 -52.0 -36.5
9.6 -56.1 -53.5 -58.4 -61.9 -42.7 -46.7 -60.2 -48.6
718.1 101.8 6.7 -21.9 -39.8 -41.0 -44.9 -58.7 -47.3
0-prior MLVC MLVC MLVC MLVC MLVC MLVC MLVC MLVC-S
× -1 × -1 × -1 × -1 × -1 × 64 × 64 × 1024 × 1024
Act
WSiLU WSiLU LReLU LReLU ReGLU ReGLU ReGLU ReGLU ReGLU
SS Mem I-D LTR Avg (SP) TH (SP) TH (XP)
× ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Legend: I: separate I-frame model, IP: intra period (−1: single intra frame), RP: reset period (0: no reset), SS: scale sending, Mem: memory module, I-D: I-frame dropout, LTR: long-term reference. SP: same-platform, XP: cross-platform, Avg: average BD-rate over HEVC B-E and 720p VCD sequences (NVIDIA GPU), TH: VCD TH subset at 360p (Apple M3), TH (SP): NPU encoder, NPU decoder, TH (XP): NPU encoder, GPU decoder, 0-prior: no adaptive priors, MLVC-S: small variant of MLVC.
to derive the ROI mask during training. We do not explicitly modulate latents with the mask like [37], therefore avoiding dependence on an external model in deployment. We also refrain from using a GAN loss as in [9, 17, 39], since it can lead to temporal flickering and may introduce artifacts that perturb identity.
4
Experiments
4.1
Experimental Setup
Datasets. We employ the same training schedule as DCVC-RT, with Vimeo90K [52] septuplets used for first stage training, and longer Vimeo sequences up to 64 frames used for fine-tuning the model. To assess the real-world deployability of neural codecs in video conferencing applications, we conduct a subjective test on the Video Conferencing Dataset (VCD) [43]. Objective metric results are also presented on the more common HEVC B-E [16] datasets. Training. We use the same variable rate control method as [24] and train the model for the YUV colorspace. When training on long sequences, we use gradient and frame gradient clipping for improved stability. Additionally, we use gradient checkpointing in the fine-tuning stage. Training with sequences of up to 32 frames takes 5 days on 8 NVIDIA Tesla V100s. For perceptual models, we train the first stage using standard PSNR loss, but switch to the perceptual loss in fine-tuning.
12
T. Pärnamaa et al.
Evaluation Metrics We measure the compression quality using BD-rate [12], computed from the actual encoded bitrate rather than the rate estimate from the entropy model. The distortion metric is PSNR in YUV420 colorspace or mean opinion score (MOS) in the subjective evaluation. Additional metrics are provided in the supplementary material. We primarily target 360p to 720p, as these resolutions are typical in real-time video conferencing. For the HEVC datasets, metrics are reported at native resolution. Cross-Platform Evaluation We report results in two settings: same-platform (SP), where encoder and decoder run on the same platform, and cross-platform (XP), where they run on different platforms (e.g., GPU encoder, NPU decoder). When cross-platform mismatch causes catastrophic decoding failure, we denote the BD-rate as ∞. Anchor. We use Intel Quick Sync Video (HEVC-QSV) [5] as the single anchor for all BD-rate calculations throughout the paper, as it represents a widely deployed hardware codec comparable to what neural codecs must match in practice. More details on the anchor configuration can be found in the supplementary material. While we provide rate-distortion results for the reference implementation HM [4] in the appendix, we omit VTM [7] and ECM [2], as they lack commodity hardware implementations. We do compare against DCVC-RT, which outperforms HM, VTM, and ECM in PSNR. P.910 Testing. We use the P.910 software package [42] to conduct subjective evaluations, running a 5-point ACR test as defined in ITU-T Recommendation P.910 [23]. We collect 15 votes for each encoded clip in the VCD-TH subset. All videos are displayed at 720p resolution to raters. For lower processing resolutions (360p, 540p), the input is downscaled before encoding and upscaled back to 720p after decoding using ffmpeg’s [3] Lanczos algorithm. 4.2
Experimental Results
Ablation Study. Table 1 presents an ablation study of architectural changes from DCVC-RT to MLVC. The original DCVC-RT achieves -69.6% average BD-rate in the same-platform scenario, while running at 102 FPS at 360p resolution (Table 2). Our full model reaches -52% BD-rate at 130 FPS at 360p, a gap of roughly 18 percentage points. This gap is the direct cost of cross-platform constraints: scale sending, hardware-compatible activations, periodic I-frames, and a unified I/P-frame model each contribute (Tab. 1). Importantly, DCVC-RT produces catastrophic decoding failures on heterogeneous hardware and thus cannot be deployed across diverse platforms (BD-rate = ∞ in the XP column). Among neural codecs that successfully decode across platforms, MLVC achieves the best BD-rate by a large margin (−45.2% vs. +15.0% for [49] on HEVC B, see supplementary material for details). LTR frames enable operation at much larger intra periods (IP=1024 vs. IP=64), providing additional BD-rate gains and improving robustness to packet loss. For higher resolutions, MLVC-S achieves 1080p encoding at 30 FPS while still reaching −37% BD-rate against the anchor.
13 0
-60.8 -59.0 -60.7 -60.7 -60.7 -60.7
Apple-NPU
-60.2 -58.6 -58.6 -58.4 -58.6
Intel-GPU
-60.7 -60.6 -60.6 -60.6
30
Intel-NPU
-60.8 -60.7 -60.7
40
Qualcomm-NPU
-60.8 -60.7
50
NVIDIA-GPU
-60.8
60