Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model Harsha Vardhan Khurdula1
Abhinav Kumar Singh2 Yoeven D Khemlani2 Vineet Agarwal2 Interfaze AI San Francisco, CA, USA {harsha, abhinav, yoeven, vineet}@interfaze.ai
arXiv:2607.13013v1 [cs.AI] 14 Jul 2026
Abstract Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio, because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.
1
Introduction
Speech recognition has been decoded the same way for a long time. An encoder reads the audio, and an attention decoder spells the transcript one token at a time, feeding each token back to itself to produce the next (Chan et al., 2016; Radford et al., 2023). The encoders have grown very strong, but the decoder is serial by construction, so a thirty second clip costs a few hundred sequential forward passes no matter how much hardware is available. Diffusion language models decode differently, starting from a canvas of noise and refining the whole
sequence in parallel over a small number of denoising steps; they now reach quality competitive with autoregressive models of similar size (Nie et al., 2025). The natural question is whether speech can be transcribed the same way, so that the cost of recognition is set by the number of denoising steps and not by the length of the transcript. Two systems have shown that diffusion can transcribe. TransFusion denoised character sequences with multinomial diffusion (Baas et al., 2022), and Whisfusion trained a masked-diffusion decoder on top of frozen Whisper features (Kwon et al., 2025). Both train a decoder specifically for the task, and both are English only. We ask a different question: can an existing, general-purpose diffusion language model be taught to hear, so that recognition runs through its own decoder with no new decoder trained at all? We take DiffusionGemma, an open-weight 26B mixture-of-experts model that generates text by discrete diffusion (does not have audio support), and give it a native audio input. We keep the backbone frozen. We add a frozen Whisper encoder as an acoustic feature extractor (Radford et al., 2023), a small trainable projector that maps those features into the model embedding space, and low-rank adapters (Hu et al., 2022) so the frozen model can learn to attend to the new modality. The projectorinto-language-model recipe follows the connectors that gave sight and hearing to text-only models (Liu et al., 2023; Chu et al., 2023; Rubenstein et al., 2023), applied here for the first time to a diffusion decoder. Only about 42M parameters train, which is 0.16 percent of the backbone. The result reaches 6.6 percent word error rate on LibriSpeech test-clean, converges in about eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages. The challenge lied in grounding, a frozen language model has never seen a spectrogram, so its embedding space has no notion of formants or
phonemes, and the two obvious training objectives both route their signal through cross-attention. If attention has already decided the audio is noise, no gradient reaches the projector to make it useful, and the projector stays noise, which in turn gives attention no reason to change. We break this deadlock with a connectionist temporal classification loss applied directly through the frozen output head, which forces the audio embeddings to be linearly predictive of the transcript without asking attention to trust them first. Our contributions are the following. • An audio-native pathway for a frozen discretediffusion language model, trained only through a projector and low-rank adapters, with no decoder trained from scratch. • A diagnosis of the grounding failure and a direct-supervision fix that breaks the projector and attention deadlock in a few hundred steps. • An empirical study of the accuracy and speed of diffusion decoding for speech, including a step count sweep that shows recognition cost decoupled from transcript length. • A single adapter trained on six languages (English, German, French, Spanish, Hindi, Mandarin) and evaluated here on English, Hindi, and Mandarin, together with an analysis of where it trails autoregressive Whisper and why.
2
Background
2.1
The diffusion backbone
DiffusionGemma is a 26B mixture-of-experts model with 128 experts, top-8 routing, and roughly 4B active parameters per token. It has 30 transformer layers, a hidden size of 2816, and a vocabulary of 262144. It generates text by discrete diffusion rather than autoregression. Most recent diffusion language models use an absorbing state, where corruption replaces tokens with a special <mask> symbol and generation fills masks in (Austin et al., 2021; Lou et al., 2024; Nie et al., 2025). DiffusionGemma instead uses uniform, random-token diffusion. It starts from a fixed-length canvas, up to 256 slots, filled with random tokens drawn from the vocabulary. At each denoising step the model reads the whole canvas at once, keeps the low-entropy predictions it is confident about, and renoises the remaining positions
Figure 1: Uniform discrete diffusion in DiffusionGemma. Generation starts from a canvas of random vocabulary tokens and, over a few steps, keeps the confident predictions and renoises the rest until the canvas anneals into text. There is no absorbing mask state.
back to fresh random tokens. After a few steps the noise anneals into text (Figure 1). Training mirrors this process. A clean sequence is corrupted by replacing a fraction γ of positions with uniform random tokens, and the model is trained to recover the originals at the corrupted positions. We formalize the process in Section 3.4. 2.2
Encoder-decoder structure
Architecturally the model is an encoder-decoder with tied transformer weights. The encoder reads the prompt causally into a read-only key-value cache. The decoder refines the canvas with bidirectional self-attention while cross-attending to that cache. Multimodal inputs are injected by scattering projected features into placeholder token positions of the encoder input embeddings. The existing vision path uses an image token identifier for this. Out of the box the model accepts text, images, and video, and produces text. It does not accept audio and providing that modality is one of our key contributions in our work. 2.3
Connecting an encoder to a language model
A frozen language model can be given a new input modality by training a small connector that maps features from a modality encoder into the model embedding space, while the backbone stays fixed or is only lightly adapted. This recipe added vision to
text-only models through a projector over a vision encoder (Liu et al., 2023), and audio to language models through speech encoders and adapters (Chu et al., 2023; Rubenstein et al., 2023), usually on top of open model families such as Gemma (Gemma Team, 2024). Our audio pathway follows the same pattern. What is new is the decoder it feeds. Every connector we are aware of feeds an autoregressive model, whereas here the connector conditions a diffusion decoder that produces the transcript in parallel, which changes both how the model must be grounded and how it decodes.
3
Method
3.1
Notation and setup
Let V be the vocabulary with V = |V| = 262,144. A transcript is a token sequence y = (y1 , . . . , yM ). We write it onto a fixed canvas of length L (we use L=256 in training and L=192 at inference) as the clean target x0 = (y, EOS, PAD, . . . ) ∈ V L . The audio is a log-mel spectrogram m ∈ R80×3000 , since Whisper pads or trims to thirty seconds. We keep the DiffusionGemma backbone and the Whisper encoder frozen and train only a projector Pϕ and low-rank adapters θ. We write στ (z) = τ tanh(z/τ ) for the final logit softcap with τ =30, and WU ∈ RV ×d for the frozen tied output head, with model width d=2816. The overall architecture is shown in Figure 2. 3.2
Audio pathway
In our first attempt, we skipped the acoustic encoder entirely. Some unified Gemma models project raw waveforms straight into the embedding space, so we fed the model short audio frames and let the backbone infer the acoustics to test this approach. And it failed completely, as a frozen language model that has never seen audio cannot build an acoustic front end from gradient signal alone, and a shallow projector cannot do feature extraction on its own. Word error rate plateaued near 94 percent while the model produced fluent text unrelated to the audio. The working design uses a frozen Whisper-small encoder (Radford et al., 2023) as a feature extractor, not as a decoder. It carries the acoustic modeling that self-supervised representations (Baevski et al., 2020) and convolution-augmented transformers (Gulati et al., 2020) established, which is the map from sound to linguistic features a frozen text model cannot learn on its own. The encoder
produces a ∈ RF ×dw with F =1500 frames and dw =768. The projector maps these to audio tokens u ∈ RT ×d , h(0) = GELU(a Win ), h
(l)
(1) (l)
= GELU Conv1d (h
(l−1)
) ,
l = 1, 2, 3, (2)
u = LN h(3) Wout ,
(3)
with Win ∈ Rdw ×dh , Wout ∈ Rdh ×d , hidden width dh =1280, and convolutions of kernel 3, stride 2, and padding 1. Each convolution maps a length n to ⌊(n − 1)/2⌋ + 1, so 1500 → 750 → 375 → 188 and T =188, about 6.25 tokens per second. The projector, together with the adapters introduced below, is the only trained part of the model. 3.3
Conditioning the diffusion decoder on audio
We build a fixed-length prompt [BOS] instr [BOA] <audio>
×T
[EOA],
with audio placeholder positions A, |A|=T . We embed the prompt and overwrite the placeholders with the projected audio tokens, ( uρ(j) j ∈ A, (4) Ej = Emb(cj ) otherwise, where ρ indexes projector rows, reusing the scatter mechanism the vision path uses. The frozen encoder, with adapters θ, reads this into a keyvalue cache K = Encθ (E), and the decoder crossattends to K while denoising, as in Eq. 6. The prompt length is constant because T is fixed. For clips shorter than thirty seconds we zero the attention mask on the silence-pad audio positions so the encoder ignores them. The canvas holds the transcript, an end-of-sequence marker, and padding, and is the decoder target x0 . 3.4
Uniform discrete diffusion
With the audio in the cache, we can state the generation process. Unlike the absorbing-state schemes used by most diffusion language models (Austin et al., 2021; Lou et al., 2024; Nie et al., 2025), DiffusionGemma corrupts toward uniform random tokens rather than a mask. A single scalar γ ∈ [0, 1] sets the corruption level and plays the role a timestep plays in continuous diffusion. Corruption acts independently at each position, γ q(xiγ = k | xi0 ; γ) = (1 − γ) 1[k = xi0 ] + , (5) V
Figure 2: The audio-native pathway. A frozen Whisper encoder produces acoustic features, a trainable projector maps them to T =188 audio tokens in the model embedding space, and these are scattered into the placeholder positions of the encoder prompt. The frozen DiffusionGemma encoder reads the prompt into a key-value cache, and the decoder denoises the transcript canvas in parallel while attending to that cache. Three losses train the projector and the low-rank adapters. The diffusion loss and the autoregressive auxiliary loss route through attention, and the connectionist temporal classification loss is applied directly through the frozen output head.
which is the cumulative uniform D3PM kernel Q̄γ = (1 − γ) I + Vγ 11⊤ (Austin et al., 2021). Equivalently, draw bi ∼ Bernoulli(γ) and ri ∼ Unif(V) and set xiγ = (1 − bi ) xi0 + bi ri . There is no special mask symbol, so a corrupted position holds a real, random vocabulary token. The model is trained in the x0 parameterization. Given the corrupted canvas and the audio conditioning c carried by the cache K, it predicts the clean token at each position, pθ,ϕ (xi0 | xγ , c) = softmax στ (WU hi ) , h = Decθ (xγ , K). 3.5
(6)
non-pad positions. Taking the expectation over γ ∼ Unif(0, 1), the data, and the corruption, " # w(γ) X Ldiff = E − log pθ,ϕ (xi0 | xγ , c) , |C| i∈C (7) with an optional importance weight w(γ) ∈ {1, 1/γ}. The loss is applied only at corrupted positions. Because the objective is bidirectional and the frozen text prior explains much of the visible clean context, the gradient asking the model to use audio is diluted (Section 3.6). We therefore force γ=1 on a fraction of batches, where the audio is the only source of information.
Training objectives
Three losses share one projector and one set of adapters.
Autoregressive auxiliary loss. The encoder is causal, so we can teacher-force the transcript through it and read a standard next-token loss,
Diffusion loss. This is the model’s own objective, the reweighted x0 -prediction surrogate of the discrete-diffusion variational bound (Austin et al., 2021; Ho et al., 2020). Let C be the corrupted,
M h1 X i Lar = E − log softmax στ (WU gi ) y , i M i=1 (8)
where gi is the encoder state that predicts yi from y<i and the audio. Because the encoder shares weights with the diffusion decoder and reads the same audio tokens, this dense per-token gradient transfers to parallel denoising. CTC loss. We decode the projector rows directly through the frozen output head, with no softcap, and align them to the transcript with connectionist temporal classification (Graves et al., 2006). With per-frame distributions qt = softmax(WU ut ) and the blank symbol at index 0, Lctc = − log
X
T Y
qt (πt ),
(9)
π∈B−1 (y) t=1
where B collapses repeats and removes blanks. Feasibility needs T ≥ M , so we cap targets to the number of real audio tokens. The full objective is L = Ldiff + λar Lar + λctc Lctc .
(10)
Low-rank adaptation. We add LoRA adapters (Hu et al., 2022) of rank 16 and scaling 32 on the query, key, value, and output projections of both the encoder and the decoder self-attention. The encoder adapters let the model fold audio into the cache, and the decoder adapters let it attend to that cache during denoising. The experts and the vision tower stay frozen. The projector and the adapters together are 42.3M trainable parameters, which is 0.16 percent of the backbone. 3.6
Why grounding fails, and how CTC fixes it
With only the diffusion and autoregressive losses, training stalled in a way that took us a while to understand. The curves fell for a few hundred steps and then went flat. The diffusion loss settled near 8 and the autoregressive loss sat around 4.5, close to chance, and neither moved for thousands of steps. Both losses reach the projector only through attention over the audio positions. Let αt be the aggregate attention mass the model places on audio token t. To first order the gradient the attentionrouted losses send to ut scales with that mass, ∂Ldiff ∼ αt WV⊤ δ t , ∂ut
(11)
where δ t is the backpropagated error at the attended positions and WV the value projection. At initialization ut is random, the frozen prior explains the visible clean context, and the model learns αt → 0,
which sends this gradient to zero. The gradient that would raise αt is also small, because ut carries nothing worth attending to. Both directions vanish at the same fixed point. The projector and the attention each wait for the other to move first. CTC removes the αt factor. Its gradient to the projector is the linear-head cross-entropy gradient composed with the CTC forward-backward occupancy ρt , the posterior probability of each label at frame t, ∂Lctc ⊤ = WU qt − ρ t , (12) ∂ut which is nonzero whatever the attention does. It pushes ut into the region of embedding space where WU ut predicts transcript tokens, with no attention required. Once ut carries content, αt grows and the gradients in Eq. 11 revive. Adding the term broke the plateau on our first attempt: the CTC loss fell from 24 to 8.6 in about 300 steps and held-out token accuracy climbed off the floor. The CTC head is a training-time scaffold, which we drop at inference so that recognition runs entirely through the diffusion decoder. 3.7
Inference as parallel denoising
Whisper’s encoder is fixed to a thirty second window, so longer audio is split at silence. We take the quietest short frame near a thirteen second target inside a three second search window and cut there, so no word is split and the non-overlapping segments concatenate with no stitching. Each segment is denoised on a canvas of L=192 slots by Algorithm 1. The sampler is the base model’s uniformdiffusion process, conditioned on audio. Each step recomputes the denoiser logits over the whole canvas, accepts the positions whose predictive entropy H(pi ) falls below a bound η, and renoises the rest to fresh uniform tokens. A linear temperature schedule anneals from 0.8 down to 0.4, self-conditioning (Chen et al., 2023) feeds the previous prediction forward, and a stability-andconfidence criterion stops early once the canvas settles. Optional classifier-free guidance (Ho and Salimans, 2022) combines a conditional pass with an unconditional pass that hides the audio cache, ℓ = ℓu + w(ℓc − ℓu ), to sharpen the effect of the audio without retraining. A short post-processing pass removes repeated words and n-grams, the canvas-fill artifacts a diffusion decoder produces when the transcript is shorter than the canvas.
Algorithm 1 Audio-conditioned parallel denoising 1: input: audio a, prompt c, steps S, guidance
w, bound η 2: K ← Encθ (scatter(Emb(c), Pϕ (a))) 3: x ← uniform random in V L 4: for s = 1 to S do 5: ℓ ← στ WU Decθ (x, K) 6: if w ̸= 1 then 7: ℓu ← στ WU Decθ (x, K\audio ) 8: ℓ ← ℓu + w (ℓ − ℓu ) 9: end if 10: p ← softmax(ℓ/temp(s)); x̂ ∼ Multinomial(p) keep x̂i where H(pi ) < η; renoise the rest 11: to fresh uniform tokens 12: if arg max ℓ stable and confident then break 13: end for 14: return arg max ℓ, decoded up to EOS
Cost. Autoregressive decoding needs M sequential decoder passes to emit M tokens (Vaswani et al., 2017). Diffusion decoding needs at most S passes over the whole canvas in parallel, plus one encoder pass, and doubles the decoder passes only when guidance is on. The cost is set by S and is independent of the transcript length M . With S ≈ 8 ≪ M , and the flat accuracy curve of Table 4, this is the source of the speedup.
4
Experimental Setup
Data. Stage one, we train the English model on the 100 hour clean subset of LibriSpeech (Panayotov et al., 2015). The multilingual model warmstarts from the English checkpoint and continues on FLEURS (Conneau et al., 2023) across six languages, English, German, French, Spanish, Hindi, and Mandarin. A further stage adds VoxPopuli parliamentary speech (Wang et al., 2021). The model has seen roughly 219 hours of audio in total. Optimization. We use AdamW with a learning rate of 1e−3, betas of 0.9 and 0.95, and weight decay of 0.01, with 100 warmup steps and a cosine decay to one tenth of the peak. We clip gradients at norm 1.0. The backbone runs in bfloat16 and the projector in float32 for stable updates. Batches hold 4 utterances with 2 gradient accumulation steps, reduced to 2 and 6 on the longer VoxPopuli material. All runs use a single H100.
Setting
Value
AdamW (0.9, 0.95), weight decay 0.01 1e−3, 100 warmup, cosine to 0.1× 1.0 4 × 2 (Libri/FLEURS), 2 × 6 (VoxPopuli) Precision bf16 backbone, fp32 projector Canvas L 256 train, 192 inference Subsample / tokens 8× / T =188 LoRA r=16, α=32, dropout 0.05, enc+dec attn Loss weights λar =1, λctc =1, w(γ)=1 Inference 8 steps, η=0.1, guidance w=1, temp 0.8 → 0.4 Optimizer Learning rate Gradient clip Batch × accum
Table 1: Training and inference configuration for the reported runs.
Evaluation. We report word error rate with the Whisper text normalizer applied to both reference and hypothesis, which is the Open-ASR and Artificial Analysis convention. For Hindi and Mandarin we report character error rate, because word error rate penalizes scripts whose word boundaries do not match the reference segmentation and overstates the error. We also report throughput as a multiple of real time. Configuration. Table 1 lists the settings used for the reported runs.
5
Results
5.1
English recognition and grounding dynamics
On LibriSpeech test-clean, English word error rate walked down over ten epochs, from roughly 90 percent early in training to about 50 percent, then to 14.6 percent, and finally to 6.6 percent at the last checkpoint (n=100 test-clean utterances). The two intermediate points come from small held-out samples and are approximate; the final number is on the full evaluation set. This is single-digit word error rate from a frozen 26B backbone, a frozen Whisper encoder, and about 42M trainable parameters. Grounding was not monotone in the metrics. Token accuracy reached 0.50 and CTC loss kept falling while the model still said nothing useful. A manual decode at 0.4 epochs showed the greedy CTC output emitting the unigram prior, the most frequent tokens in frequency order such as “the of to he”, with no relation to the audio, even though the loss curve looked healthy. Only repeated exposure over many passes brought the projector and
Step
Val AR loss
Val token acc
5.01 4.62 4.45 0.34
0.42 0.44 0.50 0.85
25 300 900 2808 (ep. 2)
Table 2: Grounding dynamics on a held-out set of about 32 utterances. The autoregressive loss stays near chance and token accuracy plateaus while the model is ungrounded, then both improve sharply. Over the same span the CTC loss falls from 24 to 8.6 and then below 0.1.
Benchmark
Metric
Score
n
FLEURS English VoxPopuli English FLEURS Hindi FLEURS Mandarin
WER WER CER CER
15.7% 18.5% 15.8% 29.6%
150 1000 300 300
Table 3: Multilingual adapter, Whisper-normalized, with evaluation set sizes n. Mandarin word error rate reads 40 percent under word segmentation, while the character error rate on the same transcripts is 29.6 percent.
Steps the attention into a configuration that actually transcribes, so we ran a manual decode every half epoch. As training progressed the outputs moved from repeated fragments to fluent text with occasional acoustic slips, such as “tents” for “tense”, the error profile of a model transcribing from the audio rather than falling back on its text prior. Table 2 shows the same run in numbers. The autoregressive loss stays near chance and token accuracy hovers around 0.4 to 0.5 through the first epoch, then both move sharply once the projector grounds. Without the CTC term the same configuration did not ground within the compute we ran, which is the failure Section 3.6 analyzes; we observed this during development rather than as a controlled ablation. 5.2
Multilingual recognition
Warm-starting from English and training on FLEURS gave one adapter covering all six languages. We report the three for which we ran heldout evaluation, English, Hindi, and Mandarin. German, French, and Spanish are in the training mix but were not separately evaluated here. Adding VoxPopuli traded a small amount of read-speech accuracy for a gain on conversational and accented audio, a movement along the accuracy frontier rather than a free improvement. Table 3 reports the adapter under the Whisper normalizer. 5.3
Parallel decoding
Because the decoder refines the entire canvas at once, transcription cost is set by the number of denoising steps and not by the length of the transcript. Table 4 sweeps the step count on FLEURS English. The curve is close to flat. Moving from 8 steps to 48 buys about half a point of word error rate and costs roughly three times the latency. The model
8 16 32 48
FLEURS-en WER
Speed
15.7% 15.6% 15.2% 15.6%
14.9× 10.3× 6.5× 4.7×
Table 4: Word error rate and throughput as a multiple of real time against the number of denoising steps (FLEURS English, n=150).
converges in about eight parallel passes. For a ten second clip this is on the order of one second of model time, and it does not grow with how much the speaker says. 5.4
Comparison with diffusion and autoregressive systems
Table 5 places our LibriSpeech test-clean result beside other diffusion and non-autoregressive systems. These are the numbers reported in the respective papers, not controlled reruns, and the systems differ in encoder, training data, tokenization, and decoding, so the table is context rather than a headto-head comparison. We place our result next to TransFusion (Baas et al., 2022) and Whisfusion (Kwon et al., 2025) for reference, not to claim a controlled win. Against autoregressive Whisper, which remains the strongest baseline, our model trails, as Table 6 shows. All numbers use the Whisper normalizer. The gap is largest on FLEURS and VoxPopuli and smallest on LibriSpeech, the one domain where we had a hundred clean hours. 5.5
What is the bottleneck
We ran ablations to locate the source of the residual error, because the answer decides whether the method is broken or simply undertrained. We ruled out one suspect at a time.
Model
Approach
WER
TransFusion Whisfusion Ours
multinomial diffusion masked diffusion frozen diffusion LM
∼6–7% 8.3% 6.6%
Table 5: Diffusion and non-autoregressive speech recognition on LibriSpeech test-clean. Baseline numbers are as reported by their authors under differing encoders, data, and decoding, and are not controlled reruns; only our row (n=100) is measured in this work. TransFusion is a character-level proof of concept. Benchmark
Ours
W-small W-large-v3
LibriSpeech clean 6.6% ∼3.4% FLEURS-en 15.7% ∼9–10% VoxPopuli-en 18.5% ∼9–11%
∼2.0% ∼4–5% ∼7–10%
Table 6: Context against autoregressive Whisper. The Whisper columns are published ranges, not controlled reruns; only the Ours column is measured here (LibriSpeech n=100, FLEURS-en n=150, VoxPopuli-en n=1000).
First, encoder quality. We swapped the Whispersmall front end for Whisper-large-v3 and retrained on the same 100 hour set. Word error rate at matched data moved from 14.75 percent at three epochs to 13.15 at six and 8.95 at eight, which is roughly even with the small-encoder run at the same data scale. A stronger encoder did not move the floor at this data budget, so the encoder is not the current limit. Second, audio resolution. We reduced the subsample factor below eight to give the projector more audio tokens per second. This regressed to 44.7 percent word error rate rather than improving, so token resolution is not the limit either, and the eight-fold subsample is not starving the decoder. Third, decode steps. The step sweep in Table 4 is flat, so the sampler is not leaving accuracy on the table. With encoder quality, resolution, and step count ruled out, the most likely remaining explanation is data scale and adapter alignment. Whisper-large-v3 was trained on a few million hours and Whispersmall on about 680 thousand. This model has seen roughly 219 hours, three orders of magnitude fewer, and it already reached 6.6 percent on the single domain where it had a hundred clean hours. The frontend ablation points the same way, since a stronger encoder did not help at this data budget, which is what one expects when the bottleneck is data rather
than features. The evidence is consistent with a data bottleneck rather than a broken method, but we cannot establish this without a scaling curve, which we leave to future work.
6
Related Work
How speech recognition decodes. End-to-end recognition grew from two ideas. Connectionist temporal classification aligns a frame sequence to a shorter label sequence without explicit segmentation (Graves et al., 2006). Attention-based sequence-to-sequence models instead let a decoder attend over the encoded audio and spell the transcript (Chan et al., 2016), a pattern the transformer generalized (Vaswani et al., 2017). Most progress since then has come from stronger encoders, including self-supervised representations (Baevski et al., 2020) and convolution-augmented transformers (Gulati et al., 2020), and from scale and weak supervision (Radford et al., 2023). In all of these the transcript is produced one token at a time. Non-autoregressive and diffusion text generation. Parallel generation began with nonautoregressive translation, which emits the whole target at once (Gu et al., 2018). Diffusion models, developed for continuous data (Sohl-Dickstein et al., 2015; Ho et al., 2020), were brought to language both by embedding tokens in a continuous space (Li et al., 2022; Gong et al., 2023) and by defining the corruption directly on discrete tokens (Austin et al., 2021; Lou et al., 2024). Self-conditioning improved discrete diffusion by feeding the previous prediction back into the next step (Chen et al., 2023), a technique our sampler uses. Large masked-diffusion language models now reach quality competitive with autoregressive models of similar size (Nie et al., 2025). DiffusionGemma sits in this family but corrupts toward uniform random tokens rather than an absorbing mask. Diffusion speech recognition and audio language models. Two systems put diffusion decoding directly on speech. TransFusion transcribed characters with multinomial diffusion as a proof of concept (Baas et al., 2022), and Whisfusion trained a masked-diffusion decoder over frozen Whisper features (Kwon et al., 2025). Both build that decoder from scratch, whereas we reuse one that already exists. Separately, a large body of work gives new modalities to frozen or lightly adapted language
models through a trained connector, for vision (Liu et al., 2023) and for audio (Chu et al., 2023; Rubenstein et al., 2023), often on top of open model families such as Gemma (Gemma Team, 2024). Our work joins these two threads. We reuse an offthe-shelf diffusion language model with no new decoder, connect a frozen speech encoder through a projector and low-rank adapters (Hu et al., 2022), ground it with connectionist temporal classification (Graves et al., 2006), decode with uniform randomtoken diffusion rather than absorbing masks, and train one adapter on six languages.
7
Limitations
The model trails autoregressive Whisper on every benchmark we tested, by the largest margin on multilingual read speech. The evidence points to data scale as the cause rather than architecture, but that remains a hypothesis until we train on substantially more audio. The Whisper encoder fixes the input window at thirty seconds, so real streaming is not possible and long audio is handled by silence segmentation, which can err when speech has no clear pauses. The diffusion decoder can produce repetition artifacts on short utterances, which we currently remove with a post-processing pass rather than at the source. Finally, our comparisons to Whisper use published ranges under the same normalizer rather than a single controlled re-run of every baseline, so the Whisper columns should be read as reference points rather than exact head-tohead numbers.
8
Conclusion
A frozen language model can learn to hear if the audio projector is supervised directly rather than only through attention. Loss curves and token accuracy will look healthy well before the model is actually grounded, so manual decoding matters during training. Once grounded, diffusion decoding makes the length of what a speaker said stop mattering for cost, because recognition converges in about eight parallel steps. The grounding recipe, the CTC unlock, and the diffusion decoder all work today; what the model still lacks is exposure to more hours of audio.
References Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. Structured
denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems. arXiv:2107.03006. Matthew Baas, Kevin Eloff, and Herman Kamper. 2022. TransFusion: Transcribing speech with multinomial diffusion. In Southern African Conference for Artificial Intelligence Research (SACAIR). arXiv:2210.07677. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems. arXiv:2006.11477. William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). arXiv:1508.01211. Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. 2023. Analog bits: Generating discrete data using diffusion models with self-conditioning. In International Conference on Learning Representations. arXiv:2208.04202. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-Audio: Advancing universal audio understanding via unified large-scale audiolanguage models. Computing Research Repository, arXiv:2311.07919. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023. FLEURS: Fewshot learning evaluation of universal representations of speech. In IEEE Spoken Language Technology Workshop (SLT). arXiv:2205.12446. Gemma Team. 2024. Gemma: Open models based on Gemini research and technology. Computing Research Repository, arXiv:2403.08295. Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2023. DiffuSeq: Sequence to sequence text generation with diffusion models. In International Conference on Learning Representations. arXiv:2210.08933. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, pages 369–376. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Nonautoregressive neural machine translation. In International Conference on Learning Representations. arXiv:1711.02281.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented transformer for speech recognition. In Interspeech. arXiv:2005.08100. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems. arXiv:2006.11239. Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. Computing Research Repository, arXiv:2207.12598. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. arXiv:2106.09685. Taeyoun Kwon, Junhyuk Ahn, Taegeun Yun, Heeju Jwa, Yoonchae Choi, Siwon Park, Jongchan Kim, Hyungon Ryu, Hyuk-Jae Lee, and Nam-Joon Kim. 2025. Whisfusion: Parallel ASR decoding with masked diffusion. Computing Research Repository, arXiv:2508.07048. Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. 2022. DiffusionLM improves controllable text generation. In Advances in Neural Information Processing Systems. arXiv:2205.14217. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems. arXiv:2304.08485. Aaron Lou, Chenlin Meng, and Stefano Ermon. 2024. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning. arXiv:2310.16834. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, JiRong Wen, and Chongxuan Li. 2025. Large language diffusion models. Computing Research Repository, arXiv:2502.09992. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning. arXiv:2212.04356.
Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, and others. 2023. AudioPaLM: A large language model that can speak and listen. Computing Research Repository, arXiv:2306.12925. Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, pages 2256–2265. arXiv:1503.03585. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. arXiv:1706.03762. Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 993–1003.