1
LiVeAction: a Lightweight, Versatile, and Asymmetric Neural Codec Design for Real-time Operation
arXiv:2605.06628v1 [eess.IV] 7 May 2026
Dan Jacobellis and Neeraja J. Yadwadkar University of Texas at Austin [email protected], [email protected]
Abstract—Modern sensors generate rich, high-fidelity data, yet applications operating on wearable or remote sensing devices remain constrained by bandwidth and power budgets. Standardized codecs such as JPEG and MPEG achieve efficient tradeoffs between bitrate and perceptual quality but are designed for human perception, limiting their applicability to machineperception tasks and non-traditional modalities such as spatial audio arrays, hyperspectral images, and 3D medical images. General-purpose compression schemes based on scalar quantization or resolution reduction are broadly applicable but fail to exploit inherent signal redundancies, resulting in suboptimal rate-distortion performance. Recent generative neural codecs, or tokenizers, model complex signal dependencies but are often overparameterized, data-hungry, and modality-specific, making them impractical for resource-constrained environments. We introduce a Lightweight, Versatile, and Asymmetric neural codec architecture (LiVeAction), that addresses these limitations through two key ideas. (1) To reduce the complexity of the encoder to meet the resource constraints of the execution environments, we impose an FFT-like structure and reduce the overall size and depth of the neural-network-based analysis transform. (2) To allow arbitrary signal modalities and simplify training, we replace adversarial and perceptual losses with a variance-based rate penalty. Our design produces codecs that deliver superior ratedistortion performance compared to state-of-the-art generative tokenizers, while remaining practical for deployment on lowpower sensors. We release our code, experiments, and python library at https://github.com/UT-SysML/liveaction. Index Terms—data compression, deep learning, cloud robotics, spatial audio, hyperspectral imaging, video compression
I. I NTRODUCTION Modern sensors—from wearables and medical devices to satellites—generate rich streams of high-resolution data [1], [2]. Efficient compression is critical for applications in health monitoring, remote sensing, and autonomous systems, as these deployments operate under strict power and bandwidth constraints. Standardized codecs (JPEG and MPEG) provide strong bitrate–quality trade-offs at low computational cost, but their human-centric design makes them unsuitable for machine-perception tasks and non-standard modalities where perceptual quality is not the target [3]. General-purpose methods, such as scalar quantization [4] and resolution reduction [5] remain widely used for their simplicity and universality. They apply to arbitrary signals, provide analytical guarantees on information loss, and combine easily with domain-specific approaches [6], [7]. But, being agnostic to real-world data, they fail to exploit inherent redundancies, leading to poor rate–distortion performance [8]. Recent advances in deep neural network (DNN)–based autoencoders [9], [10] and generative codecs [11], [12] show that
data-driven models can capture complex signal dependencies, greatly improving compression efficiency and realism. These tokenizer-style codecs use learned transforms and perceptual losses to reconstruct high-quality outputs at low bitrates but remain impractical for resource-constrained settings. Their deep, wide encoders dominate computational cost, and their architectures are often modality-specific. Additionally, generative codecs often depend on perceptual or adversarial losses tuned to human perception, making them ill-suited for scientific or machine-perception tasks. Such objectives are undefined for many signal types and can destabilize training, preventing these models from serving as general-purpose codecs, especially in low-power or embedded settings. To address these limitations, we propose LiVeAction, a Lightweight, Versatile, and Asymmetric neural codec designed to achieve efficient, high-fidelity compression across diverse signal modalities. LiVeAction is built to meet three primary goals: (1) extreme computational encoding efficiency, (2) competitive rate–distortion performance, and (3) versatility across signal modalities. Extreme computational encoding efficiency. Real-time sensing on mobile or remote platforms demands encoders that are computationally efficient and power-conscious. Most neural autoencoders use symmetric architectures, where analysis and synthesis transforms share nearly identical DNN layers [12], [14]. However, increasing encoder depth or width yields diminishing returns [15]. LiVeAction adopts an asymmetric design with a lightweight encoder that minimizes computation while preserving representational quality. LiVeAction improves efficiency using structured, FFT-inspired operations instead of dense projections. These impose a block-diagonal structure reminiscent of ShuffleNet [16] and Monarch matrices [17], [18], allowing multiple layers with alternating nonlinear activations at roughly the cost of one dense layer. Competitive rate-distortion performance. To enable applications with severe bandwidth limitations, the rate-distortion performance must match or exceed conventional standards like JPEG or MPEG. Existing autoencoder designs (e.g. Stable diffusion [19], Stable Audio [14], and Cosmos [12]) rely heavily on perceptual and adversarial losses, enabling the decoder to synthesize realistic, but hallucinated details. Prior work shows that removing these losses can improve compressed-domain learning by maximizing the dimension–distortion trade-off [3]. In LiVeAction, the training objective is purely to optimize the rate-distortion trade-off, similar to learned image compression systems [9]. To simplify the training process and increase
2
Rate [bits per pixel]
AVIF JPEG 2000 Cosmos WaLLoC LiVeAction
Rate [bits per pixel]
BD-Rate vs encoding speed BD-Rate vs JPEG 2000 [%]
AVIF JPEG 2000 Cosmos WaLLoC LiVeAction
DISTS vs bitrate
DISTS [dB]
SSIM
SSIM vs bitrate
LiVeAction 493K Params AVIF Cosmos 32M Params WaLLoC 9.2K Params
On-device encoding throughput [Mpix/sec]
Fig. 1: Rate-distortion-complexity trade-off for RGB images measured on the kodak dataset. BD-rate is averaged between SSIM and DISTS [13]). Throughput is measured on a low power mobile CPU (Intel Arrow Lake 255U).
accessibility for new modalities, we replace the continuouslyrelaxed probability model and auxiliary optimizer with a simplified rate penalty based on the sample variance. Compared to codecs with generative or adversarial losses, this formulation requires fewer hyperparameters and provides stable training for a wide range of signal types using thousands, rather than millions, of training examples. Versatility for use with any modality. LiVeAction is designed for architectural and loss-function generality to support diverse sensing applications. Prior autoencoders are often tied to specific modalities through custom objectives such as LPIPS [20], optical flow loss [12], or adversarial losses [21], [22]. In contrast, LiVeAction shows that a simple meansquared-error (MSE) based rate–distortion objective suffices across modalities, eliminating the need for perceptual losses. Existing DNN architecture designs also limit versatility. The convolutional and transformer-based architectures underlying previous autoencoders are meticulously engineered for specific modalities. LiVeAction’s analysis and synthesis transforms are modality-agnostic and apply to any uniformly grid-sampled signal. Additionally, simple heuristics are sufficient to choose hyperparameters, avoiding costly searches when adapting to new sensors. Together, these design choices reduce development cost while maintaining strong performance across various modalities. Contributions. Using LiVeAction, we create codecs for a wide range of signal types—spatial audio arrays, hyperspectral images, and 3D medical CT—as well as standard audio, image and video signals. Even compared to state-of the art neural tokenizers using modality-specific designs and trained with orders of magnitude more data and compute, we show improvements in the rate-distortion-complexity trade-off. For example, compared to Cosmos [12], LiVeAction provides a 34% BD-rate improvement while encoding more than 10× faster (see Fig. 1). II. BACKGROUND AND RELATED WORK We build on prior work in (1) high-throughput, trainingfree lossy compression, (2) autoencoder design for compressed
learning and generative modeling, and (3) efficiency optimizations in convolution- and attention-based neural network layers. Computationally efficient lossy compression. Transformbased standards such as JPEG and MPEG remain dominant for their strong trade-offs among rate, distortion, and computational cost. They combine energy-compacting transforms with tuned quantization matrices to minimize perceptual distortion for human observers. However, many signals fall outside standard audio, image, or video modalities, where imperceptible details may still matter. In such cases, trainingfree codecs based on scalar quantization offer high throughput and bounded error [23], [24]. While effective for scientific data, they underutilize inherent signal redundancies, yielding poor rate–distortion performance. For sensors with extreme bandwidth limits, modality-specific specialization becomes necessary, motivating learned codecs trained end-to-end from representative data. Autoencoders for compression and learning. End-toend learned compression using autoencoders has surpassed traditional audio [21], image [9], [22], and video [25] codecs in rate–distortion performance. Initially, high design and runtime complexity limited adoption, but this changed with the advent of latent generative modeling, where generative dimensionality-reducing autoencoders (GDR-AEs) accelerated high-resolution autoregressive [11] and diffusion models [19]. GDR-AEs were later repurposed for discriminative representation learning [26], [27] and now underpin state-of-the-art AI models across audio [28], [29], image [12], [30], and video [12], [31] domains. However, runtime efficiency, especially of the encoder, has received little attention, as its cost is overshadowed by the massive models it supports. Improving encoder efficiency is therefore essential for autoencoders that both compress high-resolution data at the edge and accelerate downstream models in the cloud. Network design for efficient representation learning and compression. Prior work improved the efficiency of convolutional and attention-based layers used in autoencoding high-resolution signals for both representation learning and compression. ShuffleNet [16] and Monarch [17], [18]
3
groups
Conv. with
groups
Lightweight analysis transform GELU
Rearrange
Conv. with
)
GroupNorm
Structured convolutional layers (replicated
Quantization
float int8
Synthesis transform (N-D Eff. ViT)
(training)
Entropy coding
Fig. 2: Proposed design. The analysis transform uses a lightweight DNN with block-diagonal structured operations.
replace standard convolutional and MLP layers with FFTlike structured matrix operations. Squeeze-and-Excitation networks [32] introduce lightweight channel attention, while EfficientViT [33] employs ReLU linear attention to scale to high-resolution. The computational efficiency of compressive autoencoders has since improved dramatically. Finite scalar quantization (FSQ) [34] unified earlier designs—vectorquantized VAEs [35] and soft-quantized rate–distortion autoencoders [9]. Recent models sandwich an FSQ-based bottleneck between invertible operations that trade spatial or temporal resolution for channel capacity. PatchMixer [36], ViTok [15], and DCVC-RT [37] use local patchifying or tubelet embedding, while WaLLoC [3] and Cosmos [12] employ wavelet packet transforms for additional energy compaction. Despite these advances, current methods still lag standardized codecs in the rate–distortion–complexity trade-off [37].
(g1 , g2 ) chosen to minimize MACs (Monarch/ShuffleNetstyle), yielding an FFT-like block-diagonal structure. GELU is used as the nonlinearity. The group normalization uses 8 groups. C is an Invertible power-law compander C(x) = sgn(x) (|x| + ε)γ − εγ , where γ=0.4, ε=0.1. Φ is a Non-invertible per-channel Laplacian CDF Φ(x) = 127 sgn(x) 1 − e−|x|/σc , where σc > 0 is learned; Φ ensures latents lie in [−127, 127] (strictly less than 8 bits). Q is Finite scalar quantization trained using a soft-to-hard scheme: for the first 70% of training, Q(x) = x + u, u∼U[− 12 , 21 ]; afterwards the encoder is frozen and Q(x) = round(x). GS is the synthesis transform consisting of EfficientViT linear-attention blocks (generalized to 1/2/3-D), with depth ddec .
III. P ROPOSED METHOD : DESIGN AND IMPLEMENTATION In order to enable applications of machine perception using diverse signal modalities in resource-constrained environments, Live Action is designed around three key goals: (1) extreme runtime computational encoding efficiency (2) competitive rate-distortion performance, and (3) flexibility for use with arbitrary modalities. Overview and codec workflow. LiVeAction inherits the overall architecture from WaLLoC [3] and Cosmos [12], consisting of an FSQ [34] based autoencoder sandwiched between the WPT and IWPT. However, our asymmetric design introduces several changes to the DNN-based transforms and training procedures. Fig. 2 provides an overview of the codec workflow and structured convolution layers, which we describe next. Let x ∈ RC×T1 ×···×TD signal with D spatio-temporal dimensions and C channels. The end-to-end codec is x̂ = IWPTJ ◦ C −1 ◦GS ◦Φ−1 ◦Q◦Φ◦GA ◦C ◦WPTJ (x). (1) WPTJ and IWPTJ apply J dyadic filter bank stages using the Cohen–Daubechies–Feauveau 9/7 filters to trade spatiotemporal resolution for frequency resolution. The analysis transform GA . consists of denc factorized groupconvolution residual blocks followed by a 1×1 projection to latent width Cz . A factorized convolution replaces a dense kernel by two grouped convolutions with groups
Fig. 3: Scaling behavior of linear projection (solid line) vs the proposed structured matrix pair (dotted line). Lightweight analysis transform for efficient encoding. In WaLLoC, the encoder consists solely of a learnable linear projection, trading expressiveness for high efficiency. Yet, this projection can still be costly. As an example, consider a spatiotemporal autoencoder for RGB videos. The WPT maps a 3 × 83 RGB video region to 1536 color-frequency bands; projecting these to a 12-D latent requires a 1536 × 12 matrixvector product for each local video region. At 1080p, this results in > 1.7 billion FLOPs per second of video for the projection alone. To significantly increase the computational efficiency of encoding, LiVeAction replaces this monolithic
4
projection by several grouped convolutional layers, yielding a structured pair with substantially fewer parameters and lower computational requirements compared to a dense matrix, as shown in Figure 3. This results in an FFT-like structure for the analysis transform, similar to Shufflenet [16] and Monarch [17], [18]. Even using several of these layers with alternating nonlinear activations, added channel attention [32], and group normalization, we achieve encoding throughput competitive with the fully connected linear projection used in WaLLoC. Linear attention synthesis transform for versatility across modalities. The intended applications of LiVeAction— real time sensing on resource-constrained mobile and remote sensors—place extreme demands on the encoder. However, at runtime, the decoder can be run on powerful cloud GPUs, or even discarded entirely in the case of compressed domain processing. Still, increasing accessibility for new codecs requires high-resolution training to be possible with low or moderate compute resources—not datacenter-scale GPU clusters. Thus, we adopt an EfficientViT-based design [33], leading to uncompromised expressiveness while enabling high-resolution training on a single GPU. We make two modifications to EfficientViT: (1) replacement of batch normalization with group normalization to eliminate differences between traintest behavior [38], and (2) generalization to one and three dimensions to accommodate additional signal modalities other than 2D images. Finite scalar quantization with simplified rate penalty. To achieve a high compression rate, we use finite scalar quantization (FSQ) [34] a type of learned vector quantization. Unlike standard VQ-AEs [35], which require expensive codebook lookup operations, FSQ uses of a guaranteed dimension bottleneck (typically between 32× and 128× reduction) combined with scalar quantization to achieve equally efficient coding. Existing FSQ designs typically aim for a small or moderate codebook size (typically ≤ 16 bits) to support standard crossentropy losses and increase compression ratio at the cost of objective reconstruction metrics like PSNR. To meet our goal of maximum versatility, we instead opt for much larger codebook size, but include a rate penalty during optimization, similar to the standard approach used in learned image and codecs [9], [10], [37]. To reduce the design, implementation, and “operational” [37] complexity, we introduce an extremely simplified formulation for the rate loss. Assuming that latent activations follow a distribution in the exponential family (e.g. generalized Gaussian), minimizing the rate is equivalent to minimizing the log of the sample variance. Thus, our overall training objective is to minimize 2 L = log10 x − x̂ 2 + λ log2 Var[Φ◦GA (x)] (2) with a single global hyper-parameter λ. The first term is the MSE distortion; the second approximates the latent rate under an exponential-family prior. We set λ = 3 × 10−2 for all modalities. Finally, we adopt a soft-then-hard quantization scheme [39]. During the main training phase, additive noise is used to encourage resilience to quantization [9]. Near the end of training (70 percent in our experiments), the encoder
is frozen, and the additive noise is replaced with hard quantization (rounding) for the remainder of the decoder training. After quantization, any entropy coding method can be used, including lossless media codecs (e.g. FLAC, PNG, FFV1, etc) by reshaping the latents to the appropriate dimension. In our experiments, we find that WEBP lossless and JPEGLS [40] provide the best trade-off between compression and computation efficiency for the entropy coding step, though the differences between methods are minor. We include the cost of entropy coding and file storage when measuring throughput. Heuristics for choosing hyperparameter values. Building a codec using LiVeAction requires choosing hyperparameters. The exact settings used to reproduce our results for each modality are available in the accompanying code repository. Here, we list several heuristics for choosing these hyperparameters for new modalities. 1) Dimension. The codec can operate on 1D, 2D, or 3D signals with arbitrary channel count. For many modalities (e.g. single channel audio) the choice of dimension is unambiguous. However, for modalities with high channel count (e.g. the 224 band hyperspectral AVIRIS images), the channels may be treated as an additional dimension. As a rule of thumb, we recommend treating the channels as an additional dimension if both (1) the number of channels is similar to spatiotemporal resolution of the other dimensions and (2) all of the channels have consistent units/scale. 2) Rate-distortion Lagrangian. In our experiments, all LiVeAction codecs are trained to minimize log10 ∥x − x̂∥ + λ log2 (σ̂), with the parameter λ controlling the trade-off between rate and distortion. We find that λ = 0.03 provides stable training across all codecs while cutting the average bitrate by about half (about 4 bits per latent channel instead of 8). 3) Latent dimension. In addition to λ, the main hyperparameter affecting the compression ratio is the number of latent channels. For natural signals with significant redundancy, we recommend choosing a latent dimension to be 64× lower than the original dimension. 4) Number of levels J in wavelet packet analysis. With the exception of the projection to and from the latent dimension, all hidden DNN layers operate with a hidden dimension of C2JD , where C is the number of signal channels and D is the dimension. We recommend choosing J such that the hidden dimension is between 512 and 1536. 5) Depth In our experiments, we find that an encoder depth of 4 and a decoder depth of 8 leads to a good balance between runtime encoding efficiency, decoder training cost, and rate-distortion performance. IV. E VALUATION Using LiVeAction, we train codecs across multiple signal modalities. We next describe the datasets, evaluation metrics, testbed, and baselines used. Stereo audio. We train on the lossless MUSDB18-HQ dataset [41], progressively raising clip length from 500k (11s)
5
to 2M samples (48s). Training runs for 200k steps (batch size 2). For augmentation, stems (vocals, drums, bass, other) are randomly remixed; evaluation uses the original validation mixes. Spatial audio. We train a spatial audio codec for the 7channel Aria [1] microphone array, progressively increasing clip length from 3 to 7 seconds. Training runs for 288k steps with a batch size of 2. Evaluation uses the validation split. In addition to PSNR, we measure the signal to spatial distortion ratio (SSDR) and signal to residual distortion ratio (SRDR) to isolate spatial distortion from other impairments [42]. Image. The codec is trained on LSDIR [43], with resolution increasing from 1282 to 4802 over 500k steps (batch size 16). Evaluation follows [12] on the 50k-image validation split of ImageNet, resizing all images to height 1024. We also evaluate the rate-distortion performance and top-1 classification accuracy1 on the ImageNet validation split at low resolution (224 × 224) and on the Kodak dataset. Hyperspectral. We extract 1,394 crops (1,300 training, 94 validation; ∼0.5 MP each) from 224-band AVIRIS images [44]. The codec is trained for 130k steps with a maximum resolution of 224 × 2882 . Evaluation is performed on full-size images. 3D medical images. We train a 3D codec on the MEDMNIST 3D dataset [45], with 6 categories of medical volumes: organ, adrenal, fracture, nodule, synapse, and vessel. Resolution increases from 243 to 643 voxels over 863.5k steps. Video. We train on 6,000 Vimeo90k [46] clips using two 24-frame batches, with resolution increasing from 112×64 to 640×384 over 120k steps. The model is fine-tuned on 3,000 high-resolution Vimeo90k clips (batch size 1), with resolution increasing from 680×384 to 1152×648. Evaluation uses fulllength DAVIS [47] videos at 1920×1080. Metrics and baselines. We evaluate the trade off between rate distortion, and complexity [48] using compression ratio (CR), PSNR, and per-sample throughput, and report dimensionality reduction (DR) as a proxy for downstream acceleration [3]. PSNR is computed on signals in [0, 1]; some works (e.g., Cosmos) use [−1, 1], yielding values 6.02 dB higher (20 log10 (2)). We compare with conventional and neural compression systems, including JPEG2000, Stable Audio [14], EnCodec [49], Cosmos [12], and WalloC [3].
performance while enabling practical deployment on lowpower sensor devices. Metrics Modality
Codec
Stereo music
DR
CR
Enc.
PSNR
Stable Audio LiVeAction
64 64
64.0 195
12.31 199.3
28.42 36.57
Spatial audio
EnCodec LiVeAction
5 64
455 1013
10.23 363.2
27.96 33.12
RGB Image
Cosmos DI8 Cosmos DI16 LiVeAction f16c48 LiVeAction f16c12
32 128 16 64
96.0 384 34.3 140
54.96 116.5 58.88 143.3
31.20 25.08 37.81 31.09
Hyperspectral
JPEG 2000 LiVeAction
1 64
575 575
12.47† 600.1
18.18 18.52
3D Medical
JPEG 2000 LiVeAction
1 64
95.62 209
13.60† 54.08
22.00 24.74
Video
Cosmos DV4 × 8 Cosmos DV8 × 8 LiVeAction f8c48 LiVeAction f8c12
128 256 32 128
96.0∗ 192∗ 79.6 331
7.656∗ 13.73∗ 33.61 52.94
28.96 27.43 30.24 27.60
TABLE I: Rate-distortion-complexity trade-off for each modality. DR is the degree of dimensionality reduction of. CR is the compression ratio. Encoding throughput (Enc.) is measured in megasamples per second (audio), megapixels per second (images), megavoxels per second (hyperspectral) and frames per second (video). The analysis transform is run on GPU (RTX 4090) and the entropy coding is run on CPU (Intel i9 13900k) with the exception of JPEG 2000†, where no GPU acceleration is available. ∗ Encoding the entire video in one pass using Cosmos is not possible due to memory constraints. Instead, we encode chunks of 24 frames with 50 percent overlap, resulting in reduced compression ratio and throughput. If no memory constraints were imposed, the CR would be increased by 4× and the throughput would be increased by 3×. Music (stereo). The VAE in Stable Audio generates Gaussian but high-entropy latents, requiring fp16 precision to avoid artifacts. In contrast, LiVeAction’s FSQ design with a rate penalty produces lower-entropy latents, achieving 3× higher compression. Stable Audio’s reliance on perceptual and adversarial losses causes cross-channel inconsistencies, whereas LiVeAction’s MSE loss yields better stereo fidelity and +8 dB PSNR. Its structured encoder operations are also
A. Results and Discussion. Tables I, II, and III summarize LiVeAction’s performance across modalities. Figure 4 shows downstream machineperception quality for RGB images via ImageNet classification accuracy on decoded outputs. Overall, LiVeAction establishes a superior rate–distortion–complexity frontier, particularly in encoding efficiency on resource-constrained hardware. Despite using simpler training objectives, smaller datasets, and far fewer GPU hours than prior generative tokenizers, LiVeAction remains highly competitive in rate–distortion 1 Classification accuracy is evaluated on decoded images using the pretrained EVA-CLIP vision transformer model.
Codec
BD-rate (PSNR)
BD-rate (DISTS)
BD-rate (SSIM)
Throughput (MPix/s)
Cosmos WaLLoC LiVeAction AVIF
+49.61 −27.61 −36.55 −64.03
−90.88 −61.71 −70.27 −60.56
−29.94 −57.52 −70.30 −71.10
N/A 6.12 9.95 3.01
TABLE II: BD-rate relative to JPEG 2000 and encoding throughput on low-power mobile CPU (Intel Arrow Lake 255U) for RGB images. All metrics are measured on the Kodak dataset except for Accuracy, which is measured on ImageNet. Lower BD-rate is better for all metrics.
6
Modality
Codec
Throughput Small Input
Large Input
Music (stereo)
Stable Audio LiVeAction
88.73 KSamp/s 323.76 KSamp/s
229.4 KSamp/s 5012. KSamp/s
RGB Image
JPEG 2000 Ballé18 [50] LiVeAction
6.097 Mpix/s 3.440 Mpix/s 5.252 Mpix/s
6.333 Mpix/s 5.106 Mpix/s 12.28 Mpix/s
Hyperspectral
JPEG 2000 LiVeAction
6.298 Mvox/s 13.76 Mvox/s
6.448 Mvox/s 14.93 Mvox/s
Video
LiVeAction
107.6 fps
2.386 fps
TABLE III: Encoding throughput on high-power CPU (Intel Raptor Lake i9-13900k). Cosmos models are not supported for CPU inference. Sizes for small and large inputs are 212 samples (85 ms) and 216 samples (1.3 s) [music]; 240p and 1080p [images]; 2243 voxels and 224 × 10242 voxels [hyperspectral]; 240p and 1080p [video].
Classification Accuracy
Accuracy vs bitrate [ImageNet]
AVIF Cosmos LiVeAction Bits used to represent input image
Fig. 4: Machine perceptual quality of Image codecs measured via Imagenet Classification accuracy. Note: Cosmos is trained on ImageNet, while LiVeAction is not.
duces latent dimensionality by 64× to accelerate downstream models while slightly improving rate–distortion performance. Its DNN-based design also benefits from GPU acceleration, delivering ∼70× higher throughput than CPU-only JPEG 2000 and over 2× faster encoding even on the same CPU. 3D medical images. On MEDMNIST 3D, LiVeAction surpasses JPEG 2000 across all metrics, achieving 64× dimensionality reduction, 2.1× higher compression, and 2.7dB higher PSNR for improved rate–distortion performance. Video. LiVeAction’s lightweight encoder design enables single-pass encoding of full-length 1080p videos on a single RTX 4090, avoiding the memory-intensive chunking required by Cosmos. At comparable quality, LiVeAction achieves > 1.7× higher compression ratio (330.7× vs. 192×) and > 3.8× higher GPU throughput (52.94 fps vs. 13.73 fps). Real-time encoding (> 60fps) is possible on CPU at low or moderate resolution. B. Additional experiments. Ablation of simplified rate loss. To isolate the effect of the simplified rate loss, we retrained the RGB codec using an explicit rate term. The implementation uses the EntropyBottleneck module from CompressAI [51] with an auxiliary optimizer. 2 . Results are shown in Table IV. The approximate rate model provides a 22 percent reduction in bitrate with minor quality impact. Objective
bpp
PSNR
LPIPS (dB)
Approximate rate: log2 (σ̂) Exact rate + density model
0.6456 0.8334
30.8464 31.1914
6.7503 6.8621
TABLE IV: Ablation of the simplified rate loss. The reported bitrate is the actual number of bits after entropy coding, not the rate estimated from the distribution.
Spatial audio. LiVeAction outperforms EnCodec with 12.8× greater dimensionality reduction (64× vs 5×), 2.2× higher compression, and 35.6× faster encoding while improving all distortion metrics—achieving +6.09 dB SSDR and +13.55 dB SRDR.
Perceptual quality enhancement using score-based generative model. Since LiVeAction omits adversarial and perceptual losses, its decoder does not resynthesize highfrequency details. We show that a separate score-based generative model can enhance perceptual quality post-decoding. Specifically, we use a FLUX ControlNet [52] conditioned on the decoder output. Neither model was trained on our codec outputs; instead, a generic version trained on common image corruptions (blur, JPEG, noise) was used. This approach yields modest perceptual gains (+0.5 dB DISTS) but significantly improves realism by restoring textures and fine details (Figure 5).
RGB image. On low-power mobile CPU (Intel Arrow Lake 255U), LiVeAction achieves the highest encoding throughput (9.95 MPix/s) and strong BD-rate savings relative to JPEG 2000 (-36.55% PSNR, -70.30% SSIM, -70.27% DISTS); Cosmos is not supported on this platform. Compared to prior neural tokenizers, LiVeAction provides comparable reconstruction quality at similar or higher compression ratios while enabling far greater encoding speed. Notably, despite not being trained on ImageNet (unlike Cosmos), LiVeAction matches Cosmos’ downstream ImageNet top-1 classification accuracy on decoded images while using 48% lower bitrate (Figure 4).
V. C ONCLUSION AND F UTURE W ORK We introduced LiVeAction, a neural codec design that establishes a new performance frontier and increases accessibility of learned compression for new types of signals and sensors. By improving signal-ingestion efficiency, LiVeAction lowers power and bandwidth demands while maintaining quality, enabling new mobile and remote sensing applications. Future work will explore variable-rate training and joint optimization with downstream ML tasks to better align compression with inference accuracy.
Hyperspectral. Compared to JPEG 2000, LiVeAction re-
2 https://interdigitalinc.github.io/CompressAI/tutorials/tutorial custom.html
far cheaper than Stable Audio’s CNN layers, providing over 16× higher throughput.
7
(a) Original
(b) Cosmos
(c) LiVeAction
(d) LiVeAction
FLUX
Fig. 5: Comparison of Cosmos, LiVeAction, and LiVeAction enhanced using a generative model. Best viewed zoomed in. The image was rescaled before compressing with cosmos to match the rate of two codecs (0.15 bpp).
Kodak (512 × 768 and 768 × 512) Model LiVeAction f16c12 (half res.) Cosmos di16×16 AVIF (q=10) JPEG 2000 (CR 250:1) LiVeAction f16c12 Balle 2018 Hyperprior [50] AVIF (q=25) JPEG 2000 (CR 100:1) Cosmos di8×8 LiVeAction f16c48 (λ = 0.1) WaLLoC f8c12 LiVeAction f16c48 AVIF (q=50) JPEG 2000 (CR 20:1) WaLLoC f8c48 LiVeAction f16c192
ImageNet-1k (224 × 224)
bpp
PSNR
LPIPS
DISTS
SSIM
bpp
PSNR
LPIPS
DISTS
SSIM
Acc
0.0428 0.0625 0.0916 0.0957 0.1496 0.2110 0.2316 0.2393 0.2500 0.3992 0.6171 0.6606 0.6838 1.1984 2.5436 3.9126
24.2336 21.7743 26.5260 24.3492 27.0384 27.2377 29.5176 26.4546 25.9193 29.8397 30.5576 31.1669 34.4449 32.0019 37.3370 40.2877
2.7625 5.3784 3.1129 2.4231 4.2153 3.9050 4.4152 3.1854 7.7112 5.6911 6.5138 6.5692 7.1089 5.4387 11.674 15.290
6.2970 10.619 6.9885 5.7073 8.8654 7.9973 9.1244 7.1317 13.265 12.073 13.244 13.670 12.992 11.365 18.294 25.534
0.7194 0.6449 0.8080 0.6842 0.8600 0.8258 0.9007 0.7829 0.8558 0.9303 0.9501 0.9571 0.9657 0.9262 0.9873 0.9962
0.1112 0.0625 0.3373 0.0978 0.1943 0.3338 0.5281 0.2402 0.2500 0.4868 0.7720 0.8072 1.1147 1.1952 3.0067 4.2843
23.647 19.710 24.830 20.450 25.402 28.188 27.789 23.404 24.334 27.978 29.102 29.501 32.299 29.517 35.107 36.595
3.5264 5.3340 3.7429 2.0800 4.4948 5.3061 5.3463 3.1483 8.0240 6.1277 7.0447 7.1563 8.6288 6.1171 12.120 14.870
5.4999 8.4825 5.7411 4.0033 6.3726 6.8009 7.1298 5.0364 10.808 7.9186 8.5500 8.6339 9.9068 7.7432 13.063 15.568
0.6257 0.4882 0.6982 0.4794 0.7122 0.8116 0.8106 0.6014 0.7036 0.8206 0.8529 0.8568 0.9144 0.8206 0.9529 0.9640
0.6277 0.7938 0.7299 0.2807 0.7981 0.8360 0.8262 0.6767 0.8708 0.8463 0.8569 0.8560 0.8679 0.8550 0.8838 0.8851
TABLE V: Additional results and baselines for RGB images. For ImageNet, the reported accuracy is the top-1 accuracy for 1000-way classification. The accuracy without lossy compression is 0.8979.
8
R EFERENCES [1] J. Engel et al., “Project aria,” arXiv preprint arXiv:2308.13561, 2023. [2] Bing Zhang, Y Chen, J Chanussot, et al., “Progress and challenges in intelligent remote sensing satellite systems,” IEEE Topics in Applied Earth Observations and Remote Sensing, 2022. [3] Dan Jacobellis and Neeraja J. Yadwadkar, “Learned compression for compressed learning,” in Data Compression Conference. IEEE, 2025. [4] Lee D Davisson, “The theoretical analysis of data compression systems,” P. IEEE, 1968. [5] C Kortman, “Redundancy reduction: A practical method of data compression,” Proceedings of the IEEE, 1967. [6] Mehrdad Khani, Prabakkore Ramaniharan, Burhan Hamza, Mohammed Alzayat, Amin Haghani, Saurabh Singh, Sang Klein, Arash Vahdat, and Mohammad Alizadeh, “Efficient video compression via content-adaptive sup. res.,” in CVPR, 2021. [7] Li-Heng Chen, Christos G Bampis, Zhi Li, Lukas Krasula, and Alan C Bovik, “Estimating the resize parameter in end-to-end learned image compression,” Signal Processing: Image Communication, 2025. [8] Lee D Davisson, “Rate distortion theory: A mathematical basis for data compression,” IEEE Trans. on Communications, 1972. [9] Johannes Ballé, Valero Laparra, and Eero P Simoncelli, “End-to-end optimized image compression,” in ICLR, 2017. [10] Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang, “Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding,” in CVPR, 2022. [11] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever, “Zero-shot text-to-image generation,” in ICML, 2021. [12] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al., “Cosmos world foundation model platform for physical ai,” arXiv preprint arXiv:2501.03575, 2025. [13] Keyan Ding, Keda Ma, Shiqi Wang, and Eero P Simoncelli, “Image quality assessment: Unifying structure and texture similarity,” IEEE transactions on pattern analysis and machine intelligence, 2020. [14] Zach Evans, Julian Parker, CJ Carr, Zack Zukowski, Jordan Taylor, and Jordi Pons, “Stable audio open,” in ICASSP, 2025. [15] Philippe Hansen-Estruch, David Yan, Ching-Yao Chuang, Orr Zohar, Jialiang Wang, Tingbo Hou, Tao Xu, Sriram Vishwanath, Peter Vajda, and Xinlei Chen, “Learnings from scaling visual tokenizers for reconstruction and generation,” arXiv preprint arXiv:2501.09755, 2025. [16] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” in CVPR, 2018. [17] Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré, “Monarch: Expressive structured matrices for efficient and accurate training,” in ICML, 2022. [18] Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré, “Monarch mixer: A simple sub-quadratic gemmbased architecture,” NeurIPS, 2023. [19] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer, “High-resolution image synthesis with latent diffusion models,” CVPR, 2022. [20] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018. [21] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021. [22] Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson, “High-fidelity generative image compression,” NeurIPS, 2020. [23] Sheng Di and Franck Cappello, “Fast error-bounded lossy hpc data compression with sz,” in IEEE international parallel and distributed processing symposium, 2016. [24] Kai Zhao, Sheng Di, Maxim Dmitriev, Thierry-Laurent D Tonellot, Zizhong Chen, and Franck Cappello, “Optimizing error-bounded lossy compression for scientific data by dynamic spline interpolation,” in IEEE International Conference on Data Engineering, 2021. [25] Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici, “Scale-space flow for end-to-end optimized video compression,” in CVPR, 2020.
[26] Song Park, Sanghyuk Chun, Byeongho Heo, Wonjae Kim, and Sangdoo Yun, “Seit: Storage-efficient vision training with tokens using 1% of pixel storage,” in CVPR, 2023. [27] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan, “Mage: Masked generative encoder to unify representation learning and image synthesis,” in CVPR, 2023. [28] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez, “Simple and controllable music generation,” NeurIPS, 2024. [29] Alexandre Défossez, Laurent Moulinié, Jade Copet, et al., “Moshi: a speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024. [30] A. Hurst et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024. [31] A. Polyak et al., “Movie gen: A cast of media foundation models,” arXiv preprint arXiv:2410.13720, 2024. [32] Jie Hu, Li Shen, and Gang Sun, “Squeeze-and-excitation networks,” in CVPR, 2018. [33] Han Cai, Junyan Li, Muyan Tian, Zhekai Hu, and Song Han, “Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction,” in CVPR, 2023. [34] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen, “Finite scalar quantization: Vq-vae made simple,” in ICLR, 2024. [35] Aaron Van Den Oord, Oriol Vinyals, et al., “Neural discrete representation learning,” NeurIPS, 2017. [36] Matthew J Muckley, Marton Havasi, and Jakob Verbeek, “Architecture optimizations for improving neural image compression compute complexity,” in DCC). IEEE, 2025. [37] Zhaoyang Jia, Bin Li, Jiahao Li, Wenxuan Xie, Linfeng Qi, Houqiang Li, and Yan Lu, “Towards practical real-time neural video compression,” 2025. [38] Yuxin Wu and Kaiming He, “Group normalization,” in ECCV, 2018. [39] Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen, “Rethinking the quantization in neural image compression,” in ICML, 2021. [40] Marcelo J Weinberger, Gadiel Seroussi, and Guillermo Sapiro, “The loco-i lossless image compression algorithm: Principles and standardization into jpeg-ls,” IEEE Transactions on Image processing, 2000. [41] Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner, “Musdb18-a corpus for music separation,” 2017. [42] Karn N Watcharasupat and Alexander Lerch, “Quantifying spatial audio quality impairment,” in ICASSP, 2024. [43] Yawei Li, Yulun Fan, Xiaoyu Yu, Joshua Batson, Kai Qian, Eirikur Agustsson, and Radu Timofte, “Lsdir: A large scale dataset for image restoration,” in CVPR, 2023. [44] Jet Propulsion Laboratory, California Institute of Technology, “Airborne Visible/Infrared Imaging Spectrometer (AVIRIS),” https://aviris.jpl.nasa. gov. [45] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni, “Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification,” Scientific Data, 2023. [46] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman, “Video enhancement with task-oriented flow,” International Journal of Computer Vision, 2019. [47] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung, “A benchmark dataset and evaluation methodology for video object segmentation,” in CVPR, 2016. [48] D. Minnen and N. Johnston, “Advancing the rate-distortion-computation frontier for neural image compression,” in ICIP, 2023. [49] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi, “High fidelity neural audio compression,” Transactions on Machine Learning Research, 2023. [50] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston, “Variational image compression with a scale hyperprior,” ICML, 2018. [51] Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja, “Compressai: a pytorch library and evaluation platform for end-to-end compression research,” arXiv preprint arXiv:2011.03029, 2020. [52] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, “Adding conditional control to diffusion models,” in CVPR, 2023.