F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
Dinghao Zhou2,∗
arXiv:2606.06357v1 [cs.SD] 4 Jun 2026
1
Xingchen Song2,∗ Di Wu2,∗ Pengyu Cheng∗ Shengfan Shen1,∗ Sixiang Lv1,∗
Nanjing University, China 2 WeNet Open Source Community ∗ Equal contribution
Abstract Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable. This mismatch complicates a single audio tokenizer that must support both understanding and generation. We adapt continuous autoencoder latents to this setting with two components: a noise-regularized autoencoder bottleneck and a latent-side representation encoder. The bottleneck uses channel normalization and stochastic perturbation instead of KL-based variational training, yielding scale-controlled continuous latents for reconstruction and autoregressive generation. The representation encoder is trained on frozen autoencoder latents with RQ-MTP and frozen-LLM supervision. The resulting tokenizer provides high-dimensional representations for understanding while preserving normalized continuous latents as generation targets.
1
Introduction
Audio representation learning sits at the center of recent progress in speech language models, text-tospeech synthesis, and general audio generation. A common strategy is to compress waveform into a sequence of acoustic latents and then train a language-conditioned generator over that sequence [2, 16, 15, 12]. The quality of the latent space therefore determines what the resulting generator can do: preserve intelligible speech, retain speaker and acoustic conditions, support long-form synthesis, and remain controllable by text or other high-level conditions. Existing audio representations tend to specialize in only part of this problem. Neural codecs and audio autoencoders are optimized for reconstruction and compression, producing latents that retain acoustic detail but may be poorly organized for audio understanding [22, 3]. Self-supervised audio encoders, in contrast, learn representations useful for recognition and classification [1, 5, 4], but they are not designed to be decoded into waveform. Recent continuous audio generators use patch-wise diffusion or flow heads over latent targets [10, 8, 15, 6, 12, 17], yet the target latent is often inherited from a reconstruction model and treated as fixed. Existing systems often improve semantic structure by adding another representation path. X-Codec2.0 fuses a pretrained semantic encoder with an acoustic encoder before quantization [20]; VibeVoice uses separate acoustic and semantic tokenizers for long-form continuous speech generation [12]; MiMo-Audio-Tokenizer trains a large discrete tokenizer with both semantic and reconstruction objectives [18]; and MingTok-Audio explores a continuous speech tokenizer with Whisper-distilled semantics [19]. These designs confirm the value of semantic information, but they can introduce separate encoders, external teachers, discrete codebooks, or mismatched representation formats between understanding representations, decoder inputs, and continuous generation targets. We keep the continuous autoencoder latent as the acoustic anchor: the low-dimensional latent remains Technical report. Early work.
the reconstructive and generative target, while a representation encoder built on top of it provides high-dimensional representations for understanding. Our focus is the tokenizer itself. We build on a SpectroStream-style STFT-domain audio autoencoder [9], but replace its bottleneck with a normalized, noise-regularized continuous latent inspired by continuous-token autoregressive generation work [11]. This produces normalized continuous latents without KL regularization, keeping the autoencoder latent decodable and stable as an acoustic target for reconstruction and generation. On top of frozen autoencoder latents, we train a latent-side representation encoder with RQ-MTP self-supervision and supervision from a frozen LLM. During tokenizer training, projected audio embeddings enter the frozen LLM, and the resulting audio-aligned states condition a next-patch flow head that predicts continuous autoencoder latent patches. The scope is tokenizer design rather than a full unified audio-language model. We study whether normalized autoencoder latents, representation learning, and generation-side flow supervision provide three properties: acoustic fidelity, understanding utility, and predictability for audio generation. We refer to the resulting system as F3-Tokenizer, emphasizing fidelity, high-dimensional representations, and flow-based generation. Our contributions are: • We formulate audio autoencoder latents as acoustic anchors for a tokenizer pipeline that exposes complementary outputs for reconstruction, understanding, and generation. • We use a normalized continuous bottleneck with per-token channel normalization and uniform-strength stochastic perturbation, avoiding a VAE-style KL objective while improving robustness of the latent space. • We attach a representation encoder trained with random-quantized multi-token prediction and frozen-LLM supervision, producing high-dimensional representations on top of frozen autoencoder latents. • We co-train a generation-side patch-level flow head that maps frozen-LLM states conditioned on downsampled representations to continuous autoencoder latent patches.
2
Related Work
Neural audio codecs and continuous autoencoders. Neural audio codecs compress waveform into compact latent sequences and reconstruct audio through a decoder. SoundStream [22] introduced an end-to-end codec with residual vector quantization, and EnCodec [3] demonstrated high-fidelity real-time audio compression. These reconstruction modules have become common tokenizers for speech and audio language models. More recent work has also moved toward continuous autoencoder targets for generative modeling. Continuous latent tokenizers often use VAE-style KL regularization to shape the latent distribution. LatentLM [15] further shows that autoregressive modeling over continuous latents requires careful variance control, introducing a σ-VAE variant to mitigate variance collapse. These works suggest that continuous latents should be distribution-controlled for generation, while leaving open whether such control must come from a variational posterior. Semantic-acoustic tokenizer fusion. A common strategy for improving tokenizer semantics is to add a semantic path beside the acoustic reconstruction path. AudioLM [2] explicitly separates semantic and acoustic tokens for long-range audio continuation. X-Codec [21] and X-Codec-2.0 [20] inject pretrained semantic features into codec training before quantization and add semantic reconstruction losses, improving codec tokens for audio language modeling. MiMo-Audio-Tokenizer [18] jointly optimizes semantic understanding and acoustic reconstruction in a large discrete RVQ tokenizer. MingTok-Audio [19] moves closer to continuous-token designs, but obtains speech semantics through a Whisper-initialized module and distillation from a frozen Whisper encoder, which can bias the tokenizer toward ASR-centric attributes. VibeVoice [12] uses separate acoustic and semantic tokenizers, while SemaVoice [17] refines continuous speech representations with a speech-foundationmodel guided alignment mechanism. These designs highlight the importance of semantic-acoustic fusion, but often introduce extra teachers, separate encoders, discrete codebooks, or multiple token streams. We keep the continuous autoencoder latent as the decodable acoustic target and learn high-dimensional representations on top of it. 2
Self-supervised audio representation learning. Self-supervised models such as wav2vec 2.0 [1] and HuBERT [5] learn representations from unlabeled speech that transfer well to recognition tasks. NEST-RQ [4] further studies causal next-token prediction with random-projection quantization. Such self-supervised training is effective for understanding, but the resulting representations are not usually trained to decode high-fidelity waveform. Our representation encoder borrows this RQ-style selfsupervision idea, while keeping it attached to a reconstructive autoencoder latent and decoder-facing projection. Autoregressive continuous audio generation. Flow matching [10] provides a simulation-free objective for learning continuous generative models, and Voicebox [8] applies flow matching to text-guided speech generation. Recent audio language models increasingly generate continuous representations with diffusion or flow heads rather than only discrete codec tokens. LatentLM [15] studies next-token diffusion over continuous VAE latents and introduces a σ-VAE variant to mitigate variance collapse in autoregressive latent modeling, while DiTAR [6], VibeVoice [12], and SemaVoice [17] use patch-wise diffusion or flow heads over continuous speech representations. DiTAR, for example, combines a language model over patch embeddings with a diffusion transformer that generates the next continuous speech patch. We use this next-patch view as tokenizer supervision: during tokenizer training, a flow head conditioned on frozen-LLM states predicts continuous autoencoder latent patches. This makes the target space generation-aware while keeping the paper focused on the tokenizer rather than a complete unified audio-language model.
3
Method
3.1
Normalized Autoencoder
Our autoencoder follows a SpectroStream-style STFT-domain backbone, but replaces the original bottleneck with a normalized continuous latent. Given an audio waveform x, we let Eϕ include the STFT frontend and audio encoder: it maps x to hidden states h = Eϕ (x), followed by a raw bottleneck z0 = Bϕ (h) with latent dimension D = 64 in our experiments. We then normalize each time step across channels: z0 − µc (z0 ) zn = , (1) σc (z0 ) + ϵstd where µc and σc are computed over the channel dimension. During training, we add controlled stochastic perturbation: z̃ = zn + αϵ,
α ∼ U(0, γ),
ϵ ∼ N (0, I).
(2)
The decoder reconstructs waveform from the perturbed latent during training, x̂ = Dψ (z̃),
(3)
and is optimized with spectral reconstruction losses plus GAN-style adversarial losses, following the audio reconstruction training recipes of DAC [7] and SpectroStream [9]. Deterministic encoding uses zn directly. In the rest of the paper, we write z for this normalized autoencoder latent. The latent z remains the decodable acoustic target for reconstruction and generation; a representation encoder later turns it into high-dimensional, LLM-compatible representations for understanding. 3.2
Latent-Side Representation Encoder
The autoencoder latent z is passed to a causal representation encoder Rη with a projection back to the decoder dimension. For long audio, we apply the encoder with a sliding-window schedule so that training remains causal without full-sequence computation: s, u = Rη (z).
(4)
The representation sequence u is used for self-supervised learning and language-grounded supervision. A decoder-side projection s ties the representation path back to acoustic reconstruction while leaving z as the continuous target for generation. 3
Representation and Flow Training
F3-Tokenizer
Patch DiT
Patch DiT
AE Encoder
Patch DiT
Patch DiT
LLM
norm + noise
downsample + project AE Decoder
Representation Encoder
Representation Encoder x̂ waveform
representation
text token
acoustic latent
Figure 1: Overview of the tokenizer components and training pipeline. The left panel shows the tokenizer outputs: an autoencoder exposes normalized continuous latents z for reconstruction, while a representation encoder produces high-dimensional representations u. The right panel shows how the representation encoder is trained: u is patch-projected into audio tokens for a frozen LLM, and a patch DiT head is trained to predict continuous z patches. RQ-MTP and LLM supervision. We train prediction heads over u with random-quantized multitoken prediction (RQ-MTP), following the next-token prediction formulation of NEST-RQ [4]. The main head predicts the next quantized target and auxiliary heads can predict additional future targets: Lrq-mtp =
K X
CE (gk (ut ), ct+k ) ,
(5)
k=1
where ct+k denotes the random-quantized target at a future frame. RQ-MTP encourages temporal structure useful for understanding in the latent-side representation while remaining compatible with continuous autoencoder latents. When paired text is available, the representations u are temporally downsampled and projected as audio embeddings before being fed to a frozen LLM: e = Pρ (Down(u)).
(6)
The LLM weights remain fixed; the text cross-entropy loss updates the representation encoder and its projection into the LLM input space: Llm = CE(pω (y | e), y).
(7)
We denote by qt the audio-aligned hidden state of this frozen LLM at audio patch t. Reconstruction constraint. The projected sequence s is tied back to waveform through the autoencoder decoder, so the representation encoder cannot discard acoustic information. We preserve acoustic detail with the decoder reconstruction objective: Lrecon = Lae (x, x̂).
(8)
This constraint keeps the representation path tied to waveform-relevant acoustic information. 3.3
Patch-Wise Flow Head
For generation, the continuous autoencoder latent is modeled patch by patch. Let P be the number of frame-level latents represented by one autoregressive audio-token step. For patch index t, the continuous target is at = ztP , ztP +1 , . . . , z(t+1)P −1 , (9) 4
where z is the normalized autoencoder latent. During Stage 1, a frozen LLM consumes text tokens and projected audio tokens derived from u; we denote by qt the hidden state aligned to audio patch t. A patch-wise flow head conditions on qt , optionally together with previous z patches as acoustic history, and predicts the whole continuous patch at in one autoregressive step. This follows the next-patch formulation used by continuous audio generators such as DiTAR [6], while keeping z as the decoder-facing acoustic target. The head is trained with a standard flow-matching objective [10] over each target patch. 3.4
Training Pipeline
Figure 1 summarizes the tokenizer outputs and training pipeline. We use a staged recipe: first learn a decodable autoencoder latent, then train the representation encoder and generation-side flow head, and finally evaluate the learned representations on task-specific systems. Stage 0: normalized autoencoder training. The first stage trains the autoencoder with the normalized and noise-perturbed bottleneck described in Section 3.1. This stage optimizes waveform reconstruction and exposes the continuous latent z as both the decoder input and the default generation target. Stage 1: representation and flow training. We freeze the autoencoder and the LLM, and train only the representation encoder, the projection into the LLM input space, and the generation-side patch flow head. Audio-only data provides random-quantized multi-token prediction on u. Audio-text pairs, including ASR transcripts and audio captions, provide frozen-LLM supervision after u is downsampled and projected into audio tokens. All audio examples also provide a next-patch flow loss: an audio-aligned hidden state from the frozen LLM conditions a patch head that predicts the next continuous z patch. For paired examples, the frozen LLM can additionally condition on text tokens; for audio-only examples, it conditions on audio-token history. Stage 2: task-specific training. Recent work on unified audio-language models often aims to support understanding and generation in one backbone. We do not train such a unified model; the paper isolates the tokenizer. We therefore evaluate the learned tokenizer through separate tasks such as ASR, TTS, and text-to-audio (TTA). In these experiments, the task models are trainable while the tokenizer supplies the acoustic representations. Understanding tasks use the high-dimensional representation u for task-specific prediction. Generation tasks train an autoregressive audio generator over audio tokens derived from u; its patch flow head predicts continuous z patches decoded by the autoencoder decoder.
4
Experiments
We evaluate the tokenizer from three perspectives: reconstruction, understanding, and generation. 4.1
Reconstruction Quality
We measure whether the normalized autoencoder latent preserves acoustic fidelity relative to continuous autoencoder baselines. Table 1 compares a VibeVoice-style σ-VAE tokenizer with our autoencoder variants and isolates the effect of normalization and noise injection in the bottleneck. Table 1: Reconstruction and preservation metrics across speech, music, and sound. We compare against VibeVoice and ablate normalization and noise injection in the autoencoder bottleneck. Speech metrics are evaluated on AISHELL-3 (24 kHz Chinese) and LibriTTS test-other (24 kHz English). Music metrics are evaluated on MUSDB18-HQ (24 kHz). General audio metrics are evaluated on AudioCaps. Slash-separated entries, where present, follow the speech dataset order above. Model VibeVoice (σ-VAE) Autoencoder Autoencoder w/ norm + noise
Speech
Token Rate 7.5 Hz 25 Hz 25 Hz
Music
Sound
MCD↓
M-STFT↓
PESQ↑
STOI↑
ViSQOL↑
KLPaSST ↓
FDOpenL3 ↓
ViSQOL↑
KLPaSST ↓
FDOpenL3 ↓
5.19/3.90 2.58/3.41 2.33/3.27
2.88/1.26 1.74/1.08 1.61/0.99
2.93/3.01 2.96/3.22 3.07/3.21
0.935/0.941 0.932/0.918 0.940/0.946
4.39/4.31 4.32/4.38 4.41/4.68
0.0580 0.0308 0.0269
150.99 34.20 29.02
3.68 4.32 4.44
0.6540 0.4650 0.2164
51.10 31.40 15.24
5
4.2
Audio Understanding
We test whether the learned representation is useful for audio understanding with a frozen-feature probing protocol. The autoencoder and representation encoder are fixed, and only task-specific probes are trained on top of the extracted representation. This keeps the evaluation focused on the representation rather than the capacity of a large task model. As a low-information control, we also train the same probes without the learned representation, so they can rely only on dataset priors. Table 2 reports the probing results and ablations. All scores are reported as percentages and higher is better. The results should be read within each domain rather than averaged across datasets, since task sizes and label spaces differ. Removing RQ-MTP weakens the audio-only self-supervised signal and causes broad drops on speech, sound, and music tasks. Removing frozen-LLM supervision mainly hurts speech-language tasks such as FSC, LibriSpeech-100h, and Speech Commands, indicating that language alignment complements predictive self-supervision. Combining both objectives yields the strongest representation across domains. Probe-task summary. The probing suite spans speech, environmental sound, and music understanding tasks; the compact table below lists the dataset groups used in Table 2. Domain Dataset
Task
Prediction target
Speech Speech Speech Speech Speech Speech Speech Speech Speech Sound Sound Sound Music Music
speaker verification emotion recognition spoken command understanding speaker counting speech recognition probing speaker attribute probing keyword spotting vocal sound recognition speaker identification sound event detection environmental sound classification general audio tagging music genre classification instrument recognition
trial label emotion class intent / action class number of speakers transcript-related label gender label command class vocal category speaker identity event label scene / sound class sound-event tag genre label instrument family
ASV2015 CREMA-D, RAVDESS FSC LibriCount LibriSpeech-100h LibriSpeech-MF Speech Cmds V1 Vocal Imitation, VocalSound VoxCeleb1 DESED ESC-50, UrbanSound8K FSD50K, FSD18-Kaggle FMA Small, GTZAN NSynth
6
Table 2: Frozen-representation probing on speech, sound, and music understanding datasets. “Ming-U” denotes MingTok-Audio (Unified). “No repr.” removes the learned representation, “w/o RQ” removes RQ-MTP, “w/o LLM” removes frozen-LLM supervision, and F3-Tokenizer uses all objectives.
Baselines Domain Dataset
Ours
Whisper Ming-U No repr. w/o RQ w/o LLM F3-Tokenizer
Speech
ASV2015 CREMA-D FSC LibriCount LibriSpeech-100h LibriSpeech-MF RAVDESS Speech Cmds V1 Vocal Imitation VocalSound VoxCeleb1
96.60 57.20 77.60 54.90 81.50 97.30 45.90 93.30 18.00 86.00 38.80
98.70 66.85 98.58 62.92 93.45 97.10 57.08 96.14 20.64 88.59 35.28
49.80 12.50 1.10 9.80 0.00 50.20 11.40 8.70 1.60 20.50 3.20
94.70 61.20 88.60 62.40 86.40 94.50 55.20 88.80 16.80 80.30 56.80
96.10 70.40 84.20 68.00 80.80 95.40 61.10 84.60 19.20 86.20 65.40
99.65 78.90 94.86 76.10 96.00 98.40 70.10 96.80 21.90 91.60 72.00
Sound
DESED ESC-50 FSD50K FSD18-Kaggle UrbanSound8K
12.70 52.80 26.20 24.10 68.70
39.80 62.25 24.83 40.81 72.87
9.40 12.70 2.60 6.10 28.50
34.20 57.80 21.20 63.50 62.80
43.20 69.50 28.70 75.40 70.60
50.10 75.20 32.10 81.38 74.22
Music
FMA Small GTZAN NSynth
58.10 62.20 53.20
54.45 71.17 57.00
21.60 14.20 16.40
54.40 72.60 50.10
60.50 80.40 54.60
64.84 85.00 58.12
4.3
Audio Generation
We evaluate generation by training autoregressive audio generators on top of the proposed tokenizer outputs. In both settings, the representation encoder produces high-dimensional frame-level representations u, a patch projection converts them into audio tokens, and a patch flow head maps generator hidden states back to continuous z patches for waveform decoding. Text-to-speech. For TTS, we train speech generators and evaluate them on the Seed-zh and Seed-en test sets. The token-rate column in Table 3 reports the conversion from the 25 Hz autoencoder latent stream to the audio-token stream consumed by the generator. Text-to-audio. For AudioCaps TTA, we use a two-step downstream schedule. Before training on AudioCaps, we warm up the patch projection and autoregressive backbone on several thousand hours of audio-caption pairs with a captioning objective, so that the model first learns audio-text alignment. We then train the text-to-audio generator on AudioCaps using the same tokenizer outputs and patch-flow target. Beyond final generation metrics, we also examine whether the learned representation makes TTS optimization easier. We compare a reconstruction-only autoencoder output with representation-based tokenizer outputs under the same downstream TTS setup. As shown in Figure 2, representationderived audio tokens make intelligible speech emerge earlier, while the flow head still predicts the same continuous autoencoder latent target z.
7
Table 3: Generation results for text-to-speech and text-to-audio. CER/WER is reported in percent, and SIM denotes speaker similarity. Text-to-speech on Seed-zh and Seed-en
Model
Token rate
Seed-zh CER↓
Seed-zh SIM↑
Seed-en WER↓
Seed-en SIM↑
CosyVoice 3-1.5B Qwen3-TTS-25Hz-0.6B-Base Qwen3-TTS-25Hz-1.7B-Base Qwen3-TTS-12Hz-0.6B-Base Qwen3-TTS-12Hz-1.7B-Base Ming-Flash-Omni-preview Ming-omni-tts-0.5B Ming-omni-tts-16.8B-A3B Ming-UniAudio-16B-A3B VibeVoice-1.5B
25 Hz 25 Hz 25 Hz 12 Hz 12 Hz 12.5→3.1 Hz 12.5→3.1 Hz 12.5→3.1 Hz 50→10 Hz 7.5 Hz
1.12 1.18 1.10 0.92 0.77 0.99 0.87 0.83 0.95 1.16
0.781 – – – – 0.740 0.72 0.75 0.70 0.744
2.21 1.64 1.49 1.32 1.24 1.59 2.19 2.02 1.85 3.04
0.720 – – – – 0.680 0.61 0.62 0.58 0.689
F3-Tokenizer-LLM (4B)
25→12.5 Hz
0.90
0.76
1.88
0.68
Text-to-audio on AudioCaps Model
Token rate
FDOpenL3 ↓
KLPaSST ↓
CLAP score↑
Ming-omni-tts-0.5B Ming-omni-tts-16.8B-A3B F3-Tokenizer-LLM (4B)
12.5→3.1 Hz 12.5→3.1 Hz 25→12.5 Hz
74.292 65.918 62.700
2.257 1.640 1.520
0.347 0.424 0.438
100
×
AE latent only early representation
unintelligible
WER (%) ↓
80
F3-Tokenizer
60 40 20 0 6k
12k 24k TTS training updates
50k
Figure 2: TTS convergence with reconstruction-only and representation-based tokenizer outputs. WER is measured from 6k to 50k downstream TTS updates. The 6k AE-latent-only checkpoint did not produce recognizable speech, so we mark it qualitatively rather than reporting WER. Representationderived audio tokens make speech emerge earlier and reduce WER faster under the same downstream training budget.
5
Discussion and Conclusion
We presented F3-Tokenizer, a framework for adapting audio autoencoder latents beyond reconstruction. The normalized autoencoder produces a low-dimensional, decodable acoustic latent z for reconstruction and continuous generation, while a representation encoder maps the same latent stream into high-dimensional representations u for understanding. This separates the two roles that are often in tension: preserving waveform-level acoustic detail and exposing structure that is useful to an autoregressive model. The training objectives provide complementary constraints. Reconstruction preserves local acoustic fidelity, while channel normalization and stochastic perturbation make the bottleneck robust without relying on a variational KL objective. Random-quantized prediction supplies audio-only self-supervision, frozen-LLM supervision aligns the representation with paired text, and next-patch flow training makes the continuous acoustic target predictable from autoregressive hidden states. Across reconstruction, probing, and generation experiments, these constraints produce representations that remain decodable while improving understanding and accelerating downstream TTS training. Several limitations remain. The recipe introduces extra objectives and teacher dependencies, so performance can depend on the coverage of random-quantized targets, paired text, and frozen-LLM 8
supervision. Autoencoder-only latents can also serve as direct generation targets; future work should study whether representation-derived audio tokens become more important for harder generation settings such as Any2Speech [14]. Finally, the current design keeps z and u as separate spaces. A unified high-dimensional representation that is both decodable and friendly to autoregressive modeling, similar in spirit to recent representation-autoencoder approaches in vision generation [23, 13], is a promising direction.
References [1] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477, 2020. [2] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: A language modeling approach to audio generation. arXiv preprint arXiv:2209.03143, 2022. [3] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022. [4] Minglun Han, Ye Bai, Chen Shen, Youjia Huang, Mingkun Huang, Zehua Lin, Linhao Dong, Lu Lu, and Yuxuan Wang. Nest-rq: Next token prediction for speech self-supervised pre-training. arXiv preprint arXiv:2409.08680, 2024. [5] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. arXiv preprint arXiv:2106.07447, 2021. [6] Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du, Jian Wu, Jian Cong, Xiaobin Zhuang, Chumin Li, Zhen Wei, Yuping Wang, and Yuxuan Wang. DiTAR: Diffusion transformer autoregressive modeling for speech generation. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 27255–27270. PMLR, 2025. [7] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN. In Advances in Neural Information Processing Systems, volume 36, 2023. [8] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306.15687, 2023. [9] Yunpeng Li, Kehang Han, Brian McWilliams, Zalan Borsos, and Marco Tagliasacchi. Spectrostream: A versatile neural codec for general audio. arXiv preprint arXiv:2508.05207, 2025. [10] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. [11] NextStep Team. Nextstep-1: Toward autoregressive image generation with continuous tokens at scale. arXiv preprint arXiv:2508.10711, 2025. [12] Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, Shaohan Huang, Yan Xia, and Furu Wei. Vibevoice technical report. arXiv preprint arXiv:2508.19205, 2025. [13] RAE-AR Team. RAE-ar: Replacing variational autoencoders with representation autoencoders for autoregressive image generation. arXiv preprint, 2026. [14] Xingchen Song, Di Wu, Dinghao Zhou, Pengyu Cheng, Shengfan Shen, Sixiang Lv, et al. Any2Speech: Borderless long speech synthesis. arXiv preprint arXiv:2603.19798, 2026. 9
[15] Yutao Sun, Hangbo Bao, Wenhui Wang, Zhiliang Peng, Li Dong, Shaohan Huang, Jianyong Wang, and Furu Wei. Multimodal latent language modeling with next-token diffusion. arXiv preprint arXiv:2412.08635, 2024. [16] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023. [17] Huimeng Wang, Hui Lu, Jiajun Deng, Haoning Xu, Youjun Chen, Xueyuan Chen, Zhaoqing Li, Shuhai Peng, Shiyin Kang, and Xunying Liu. Semavoice: Semantic-aware continuous autoregressive speech synthesis. arXiv preprint arXiv:2605.16964, 2026. [18] Xiaomi MiMo Team. MiMo-Audio-Tokenizer. https://huggingface.co/XiaomiMiMo/ MiMo-Audio-Tokenizer, 2025. [19] Canxiang Yan, Chunxiang Jin, Dawei Huang, Haibing Yu, Han Peng, Si Chen, et al. MingUniAudio: Speech LLM for joint understanding, generation and editing with unified representation. arXiv preprint arXiv:2511.05516, 2025. [20] Zhen Ye. X-Codec-2.0: An open source semantic-acoustic codec for speech language modeling. https://github.com/zhenye234/X-Codec-2.0, 2025. [21] Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, Yike Guo, and Wei Xue. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. arXiv preprint arXiv:2408.17175, 2024. [22] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. arXiv preprint arXiv:2107.03312, 2021. [23] Boyang Zheng, Guanhua Ma, Haotian Li, Xiaoliang Wang, Jing Yang, Jun Zhang, Yuxin Liu, Bin Lin, Quan Liu, and Lei Zhang. Diffusion transformers with representation autoencoders. arXiv preprint arXiv:2510.11690, 2025.
10