Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition

Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition Gaopeng Xu1 , Zhenyu Wang1 , Zheng Xue1 , Yinfeng Xia1 , Haitao Yao1,∗∗ 1

Qwen Business Unit of Alibaba, China

{longque.xgp, wzy497957, jeremy.xz, xiayinfeng.xyf, timmy.yht}@alibaba-inc.com

arXiv:2608.10836v1 [cs.SD] 11 Aug 2026

Abstract The signal ambiguity of whispered speech drives ASR systems toward two opposing failure modes: failing to capture whispered speech or hallucinatory transcription of noise. This paper introduces the Whisper-Aware LLM, a framework that teaches an Audio-LLM to perceive and react to this uncertainty. Our model develops an intrinsic self-awareness by learning to quantify the physical deficiencies of acoustic signals through targeted self-supervised tasks. This learned uncertainty is then operationalized via a novel Confidence-Fused Decoding mechanism, which provides both high-level instructions and framelevel attention modulation to the LLM decoder. Our experiments confirm the effectiveness of this approach. The model sets a new state-of-the-art on whispered speech with a 17% relative CER reduction on AISHELL6-Whisper. At the same time, it directly addresses the reliability trade-off, with hallucination rates dropping from over 25% to 4.5%. Index Terms: Whisper speech recognition, Audio-LLM, SelfSupervised learning, Uncertainty estimation

1. Introduction Whispered speech, a mode of phonation without vocal fold vibration, presents a formidable and long-standing challenge for Automatic Speech Recognition (ASR) systems. The acoustic manifestation of whispering characterized by the absence of a fundamental frequency (F0) and harmonic structure, and a noise-like quality [1, 2] creates a fundamental ambiguity that conventional ASR models struggle to resolve. This ambiguity gives rise to a debilitating accuracy-reliability trade-off. On one hand, models trained primarily on normal speech exhibit perceptual failure when encountering whispers, leading to high error rates. On the other hand, increasing a model’s sensitivity to capture these faint cues paradoxically heightens its susceptibility to hallucination, where it erroneously transcribes speech from ambient noise. Existing research has attempted to mitigate this trade-off via two principal avenues. The first is a data-centric approach focusing on augmentation, such as generating “pseudowhisper” data [1, 3]. While beneficial, these methods are limited by the inevitable distributional gap between synthetic and authentic whispered speech. The second avenue is modelcentric static adaptation, exemplified by the projection layer proposed by Li et al. [2]. While an improvement, such “onesize-fits-all” static solutions fail to dynamically adjust to the vast spectrum of whisper variability. Crucially, these prevailing paradigms primarily address the acoustic symptoms of whispered speech, rather than its underly** indicates the corresponding author.

ing cause: a fundamental increase in signal uncertainty. A pivotal acoustic study by Lin et al. [1] experimentally confirmed that the lack of glottal information is the single most critical factor degrading recognition performance. This key finding suggests a new direction: instead of forcing the model to make high-confidence predictions from highly uncertain signals, a more robust approach is to first teach the model to quantify the signal’s intrinsic uncertainty itself. This paper is built upon this principle. We introduce the Whisper-Aware LLM, a framework designed to endow a native Audio-LLM with an intrinsic self-awareness of signal uncertainty. We posit that a model can learn to perceive this uncertainty in a self-supervised manner by being trained to identify the very physical deficiencies that define whispered speech. To this end, we design two innovative self-supervised tasks: F0 contour prediction and masked spectrum reconstruction. The quantified uncertainty is then operationalized through our Confidence-Fused Decoding mechanism. To ensure stable learning, we introduce a disciplined three-stage training strategy. Our contributions are: 1. A novel framework integrating self-supervised uncertainty perception directly into an Audio-LLM. 2. Two physics-informed self-supervised tasks for label-free uncertainty learning. 3. A Confidence-Fused Decoding mechanism for dynamic generative control. 4. A systematic three-stage training protocol for robust representation learning.

2. The Whisper-Aware LLM Framework Our framework enhances a standard Audio-LLM by integrating a lightweight Uncertainty Perception Module (UPM) and a corresponding Confidence-Fused Decoding mechanism. 2.1. Overall Architecture As shown in Fig.1, the architecture consists of three main stages: • Acoustic Encoding: The encoder E maps an input audio waveform w into a sequence of representations Henc ∈ RT ×Dmodel . • Uncertainty Perception: Our UPM, U, takes Henc as input and produces a global uncertainty vector uglobal and a framewise confidence sequence Hconf . • Confidence-Fused Decoding: The decoder D generates the transcription Y autoregressively, guided by the UPM’s outputs.

Uncertainty Perception Module(UPM)

WI

Whisper Indicator

SR

Speech Representation

TR

Text Representation

[FP0]

Whisper-Aware LLM

Frame-wise Confidence

Confidence-Fused LLM Decoder(D)

F0 Counter Prediction

[FFN]

(��0 )

Feed-Forward

[ENC]

Input Audio W

�표 �

Acoustic Encoder(E)

[MS] Masked Spectrum Reconstruction (�푆푝 )

Attention Modulation (Additive Bais)

Global Uncertainty Vector

[SA] Self-Attention Final Transcription

[IP] Input-Processor

�푔푙표

푙

Instruction Embedding

Adapter

Acoustic Representations

WI

MLP

SR

TR

Speech Representation

Figure 1: The overall architecture of the Whisper-Aware LLM. The model first encodes the audio into acoustic representations. Our UPM then processes these representations to perceive signal quality, producing a global instruction embedding (einstruct ) and a framewise confidence sequence (Hconf ). These outputs guide the LLM decoder: the instruction acts as a system prompt, and the confidence scores modulate the attention mechanism.

2.2. Uncertainty Perception Module (UPM)

2.3. Confidence-Fused Decoding

The UPM is a small network trained to perceive signal uncertainty through two self-supervised auxiliary tasks.

2.3.1. Global Instruction Embedding

2.2.1. Task 1: F0 Contour Prediction Motivated by the finding that F0 absence is the most critical factor in whisper recognition degradation [1], we train a prediction head fF0 within the UPM to predict the F0 contour F̂0 from Henc . The training loss is the Mean Squared Error (MSE) against a ground-truth contour F0 : T

LF0 =

1 X (F̂0 (t) − F0 (t))2 T t=1

(1)

The frame-wise confidence score hconf (t) is derived from the normalized prediction error eF0 (t):  hconf (t) = 1 − clip

 eF0 (t) , 0, 1 ϵmax

(2)

where ϵmax is an error clipping hyperparameter. The resulting Hconf is then centered to have a zero mean.

The global uncertainty vector uglobal is transformed by an MLP[4], finstr , into an instruction embedding einstruct . This is prepended to the decoder’s input sequence, acting as a highlevel system prompt. 2.3.2. Attention Modulation In the LLM-based architecture, the acoustic representations Henc are treated as a prefix to the token sequence, and the LLM performs causal self-attention over this entire combined sequence. To guide this process, we introduce an attention biasing mechanism. For any query q within the LLM, when it computes an attention score with a key kt that corresponds to the t-th acoustic frame from the encoder, we inject our learned confidence as an additive bias. The modified score is: q ⊤ kt score(q, kt ) = √ + w · hconf (t) dk

(3)

where w is a learnable scalar. This simple yet effective mechanism allows the model to dynamically pay less attention to acoustic frames deemed unreliable by the UPM, preventing the decoder from being misled by uncertain evidence. 2.4. Three-Stage Training Strategy

2.2.2. Task 2: Masked Spectrum Reconstruction To capture the unstructured, noise-like nature of the whisper spectrum, a second head fSpec is trained to reconstruct randomly masked frames in Henc . The UPM’s final hidden state, averaged over time, forms the global uncertainty vector uglobal .

To stably integrate our Uncertainty Perception Module (UPM) while avoiding gradient conflicts with the large pre-trained model, we employ a disciplined three-stage training strategy: • Stage 1: UPM Pre-training. We first train the UPM exclusively on its self-supervised objectives (LF0 + LSpec ), while

the entire Audio-LLM backbone remains frozen. This isolates the UPM, allowing it to rapidly learn robust uncertainty features from the frozen encoder without destabilizing the main model. • Stage 2: Interface Adaptation. Next, we focus on connecting the UPM to the decoder. We jointly train the audio encoder, the audio-LLM adapter, the UPM, and our new decoding interfaces (the MLP finstr and scalar w), while keeping the large LLM decoder frozen. This stage aligns the modalities and teaches the model to use the uncertainty signals. • Stage 3: Full End-to-End Fine-tuning. Finally, the entire model is fine-tuned end-to-end using a composite loss (LASR + λaux (LF0 + LSpec )). To maintain efficiency, we apply LoRA[5] to the LLM decoder.

3. Experiments To validate the effectiveness of our Whisper-Aware LLM, we conducted a series of comprehensive experiments. We aimed to evaluate its performance on whispered speech recognition, its general ASR capabilities, its robustness against hallucination, and the contribution of its core components. 3.1. Experimental Setup 3.1.1. Datasets and Metrics Training Datasets. Our work leverages two strategically constructed datasets for different training phases to build a highly robust model. • UPM Pre-training Corpus (for Stage 1): To ensure the UPM learns a comprehensive model of acoustic signals, we constructed a large-scale, mixed-domain corpus. This corpus includes General-Purpose Speech (a 3,000h subset of WenetSpeech[9], a 3,000h subset of GigaSpeech[10], AISHELL-1[11] and LibriSpeech[12]), Whispered Speech (wTIMIT[13], AISHELL6-Whisper[14]), and a 1,000h collection of pure noise. • Fine-tuning Set (for Stage 2 & 3): This set consists of the training splits of AISHELL-1, LibriSpeech, wTIMIT, AISHELL6-Whisper, and a 200h subset of the pure noise data to maintain robustness. Evaluation Datasets. We evaluate our model’s performance across three key dimensions: • Whispered Speech: The official test sets of AISHELL6Whisper [2] (Chinese) and wTIMIT [15] (English). • General-Purpose ASR: The test sets of AISHELL-1 and LibriSpeech to assess general competency. • Reliability and Hallucination: To specifically evaluate hallucination under adverse conditions, we created a Noise Hallucination Set containing 1000 audio clips. These clips consist of various non-speech sounds (e.g., machine hum, wind) and segments with either no speech or very faint speech heavily masked by noise, designed to be challenging for conventional VADs and prone to trigger false transcriptions. Metrics. The primary performance metrics are Character Error Rate (CER%) for Chinese and Word Error Rate (WER%)/CER% for English. On the Noise Hallucination Set, we measure the Hallucination Rate (HR%), calculated as the percentage of audio files containing no transcribable speech that produce a non-empty transcription.

3.1.2. Implementation Details • Base Model: Our framework is built upon the Qwen2-Audio model [16]. It consists of an audio encoder, a lightweight projector as the audio-LLM adapter, and a Qwen-7B LLM [17] backbone. • Uncertainty Perception Module (UPM): Our UPM is composed of a shared feature extractor (two 1D-CNN layers with GeLU and a single Transformer[18] encoder layer) followed by two task-specific heads for F0 prediction and masked spectrum reconstruction. • Training Details: Our three-stage strategy was implemented using the AdamW[19] optimizer with a learning rate of 1×10−5 . In Stage 3, the LLM was updated via LoRA (rank=64, alpha=32). The auxiliary loss weight was set to λaux = 0.1. Stage 1 ran for 100k steps, while Stages 2 and 3 each ran for 20k steps. 3.1.3. Compared Systems We benchmark against internal baselines sharing the Qwen2Audio backbone and external state-of-the-art (SOTA) systems. The systems S1-S7 are defined as: Whisper-v3 [20] (S1) and its fine-tuned version (S2); a fine-tuned Qwen2-Audio baseline (S3); and other recent SOTA ASR systems (S4-S7). 3.2. Main Results 3.2.1. Whispered Speech Accuracy We first evaluate whispered speech accuracy on both Chinese and English benchmarks to demonstrate the cross-lingual effectiveness of our framework. As detailed in Table 2, our model achieves a new state-ofthe-art on the challenging AISHELL6-Whisper test set. With a CER of only 1.31% on whispered speech, it significantly surpasses all compared systems, including the strong Seed-ASR (S7) which scored 1.58%. This represents a relative CER reduction of 17% over the previous best. Importantly, our model also achieves the lowest error rate on normal speech (0.63%), indicating that its specialization for whispers does not compromise its general ASR capabilities. The results on English benchmarks, presented in Table 1, further reinforce these findings. Our model consistently outperforms all baselines across both normal (N) and whispered (W) conditions for US and Singaporean (SG) accents. Notably, while other models show a dramatic increase in WER when transitioning from normal to whispered speech, our model exhibits a much more graceful degradation. This demonstrates its enhanced robustness to the acoustic shift caused by whispering, a direct result of its uncertainty-aware architecture. 3.2.2. General ASR Performance To explicitly verify that our model’s specialization for whispers does not impair its general capabilities, we summarize its performance on standard, non-whispered benchmarks in Table 3. Our model demonstrates highly competitive performance against state-of-the-art systems. On the AISHELL-1 test set, it achieves a strong CER of 1.34%, placing it on par with other top-tier systems like Funasr (1.22%). On the LibriSpeech-clean set, our model obtains a robust WER of 1.91%, again showing performance in the same league as leading models. This confirms that our uncertainty-aware framework successfully enhances robustness in challenging conditions without sacrificing its proficiency on high-quality, normal speech.

Table 1: Detailed performance comparison on English benchmarks (WER%/CER%). Our model demonstrates superior performance on both normal (N) and whispered (W) speech. WER (%)

Model

CER (%)

NU S

NSG

WU S

WSG

NU S

NSG

WU S

WSG

S1: Whisper-v3 S2: Whisper-v3 fine-tuned S3: Qwen2-Audio fine-tuned S4: Aishell6-whisper AVSR S5: Qwen3-ASR[6] S6: Funasr-ASR[7] S7: Seed-ASR[8]

5.25 5.58 4.95 4.50 4.99 4.65 4.29

8.67 7.46 6.85 6.06 7.26 6.96 6.51

8.20 7.95 7.40 6.35 7.01 8.48 6.88

19.03 15.79 13.50 11.63 16.46 19.29 12.95

1.03 0.93 0.88 0.73 0.98 0.91 0.85

2.50 1.86 1.65 1.33 1.85 1.76 1.62

2.47 2.37 2.30 1.95 2.45 2.88 2.15

8.85 7.53 6.20 5.17 7.11 8.24 6.03

Ours

4.15

5.89

5.92

10.81

0.70

1.25

1.85

4.98

Table 2: Performance on the AISHELL6-Whisper set (CER%). CER (%)

Model

Normal

Whisper

S1: Whisper-v3 S2: Whisper-v3 fine-tuned S3: Qwen2-Audio fine-tuned S4: Aishell6-whisper AVSR S5: Qwen3-ASR S6: Funasr-ASR S7: Seed-ASR

3.95 1.62 1.05 1.11 0.64 0.72 0.65

18.93 6.69 3.98 4.13 3.79 19.50 1.58

Ours

0.63

1.31

Table 3: General ASR performance on standard benchmarks. Model

AISHELL-1 (CER%)

LibriSpeech-clean (WER%)

Whisper-v3 Seed-ASR Funasr

4.72 1.63 1.22

1.86 2.80 1.51

Ours

1.34

1.91

Table 4: Reliability evaluation on the Noise Hallucination Set. Model

HR(%)

S3: Qwen2-Audio fine-tuned S5: Qwen3-ASR S6: Funasr-ASR S7: Seed-ASR

35.5 25.2 29.3 25.2

Ours

4.5

3.3. Ablation of Model Components To dissect the contribution of our proposed components, we conducted an ablation study on the AISHELL6-Whisper, with results in Table 5. Starting from the S3 baseline (3.98% CER), using only the frame-wise Attention Modulation yields a clear improvement to 3.45%. The Global Instruction, however, proves to be the primary driver of performance, slashing the CER to 1.84% by providing high-level, strategic awareness of signal quality. Finally, the Full Model combines both, achieving the best performance at 1.31%. Table 5: Ablation of Confidence-Fused Decoding components on AISHELL6-Whisper (Whisper subset). Configuration

3.2.3. Reliability and Hallucination A core contribution of our work is resolving the accuracyreliability trade-off. We test this on our curated Noise Hallucination Set, where an ideal system should produce no output. As shown in Table 4, standard fine-tuned models exhibit a high Hallucination Rate (HR). For instance, even strong baselines like S7 and S5 incorrectly generate text for over 25% of the files, with the highest rate reaching 29.3% (S6). This highlights their tendency to “imagine” speech when faced with ambiguous signals. In stark contrast, our Whisper-Aware LLM achieves an HR of only 4.5%, a substantial reduction. This remarkable improvement demonstrates that by learning to perceive signal uncertainty, our model effectively learns when not to speak, drastically mitigating the risk of generating spurious text from non-speech audio. This reduction is primarily attributed to the Global Instruction mechanism, which guides the decoder, as quantified in our ablation study.

Baseline (Vanilla Finetuning) + Attention Modulation Only + Global Instruction Only + Full Model (Ours)

CER(%) 3.98 3.45 1.84 1.31

4. Conclusion In this paper, we introduced the Whisper-Aware LLM, a framework designed to handle the uncertainty inherent in whispered speech. Instead of just matching patterns, our approach teaches the model to first perceive signal quality through targeted selfsupervised tasks. This learned awareness is then used by our Confidence-Fused Decoding mechanism to guide the LLM, allowing it to selectively trust the acoustic evidence it receives. Our experiments confirm the success of this strategy. The model achieves state-of-the-art results on whispered speech benchmarks and drastically reduces hallucinations, all while remaining a highly effective general-purpose ASR system.

5. Generative AI Use Disclosure The authors used generative AI tools solely for language editing and grammar polishing. All technical content, methodology, experimental design, and analysis were conducted entirely by the authors.

6. References [1] Z. Lin, T. Patel, and O. Scharenborg, “Improving whispered speech recognition performance using pseudo-whispered based data augmentation,” in 2023 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2023, pp. 1–8. [2] C. Li, F. Su, J. Liu, H. Bu, Y. Wan, H. Suo, and M. Li, “Aishell6-whisper: A chinese mandarin audio-visual whisper speech dataset with speech recognition baselines,” arXiv preprint arXiv:2403.19833, 2024. [3] P. R. Gudepu, G. P. Vadisetti, A. Niranjan, K. Saranu, R. Sarma, M. A. B. Shaik, and P. Paramasivam, “Whisper augmented endto-end/hybrid speech recognition system—cyclegan approac,” in Proc. Interspeech 2020, 2020, pp. 2302–2306. [4] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” nature, vol. 323, no. 6088, pp. 533–536, 1986. [5] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” Iclr, vol. 1, no. 2, p. 3, 2022. [6] X. Shi, X. Wang, Z. Guo, Y. Wang, P. Zhang, X. Zhang, Z. Guo, H. Hao, Y. Xi, B. Yang et al., “Qwen3-asr technical report,” arXiv preprint arXiv:2601.21337, 2025. [7] K. An, Y. Chen, Z. Chen, C. Deng, Z. Du, C. Gao, Z. Gao, B. Gong, X. Li, Y. Li et al., “Fun-asr technical report,” arXiv preprint arXiv:2509.12508, 2025. [8] Y. Bai, J. Chen, J. Chen, W. Chen, Z. Chen, C. Ding, L. Dong, Q. Dong, Y. Du, K. Gao et al., “Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition,” arXiv preprint arXiv:2407.04675, 2024. [9] B. Zhang, H. Lv, P. Guo, Q. Shao, C. Yang, L. Xie, X. Xu, H. Bu, X. Chen, C. Zeng et al., “Wenetspeech: A 10000+ hours multidomain mandarin corpus for speech recognition,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6182–6186. [10] G. Chen, S. Chai, G. Wang, J. Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang et al., “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,” arXiv preprint arXiv:2106.06909, 2021. [11] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “Aishell-1: An opensource mandarin speech corpus and a speech recognition baseline,” in 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and assessment (O-COCOSDA). IEEE, 2017, pp. 1–5. [12] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210. [13] B. P. Lim, Computational differences between whispered and nonwhispered speech. University of Illinois at Urbana-Champaign, 2011. [14] C. Li, F. Su, J. Liu, H. Bu, Y. Wan, H. Suo, and M. Li, “Aishell6-whisper: A chinese mandarin audio-visual whisper speech dataset with speech recognition baselines,” arXiv preprint arXiv:2509.23833, 2025. [15] B. P. Lim, “Computational differences between whispered and non-whispered speech,” Ph.D. dissertation, University of Illinois at Urbana-Champaign, 2011.

[16] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin et al., “Qwen2-audio technical report,” arXiv preprint arXiv:2407.10759, 2024. [17] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang et al., “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023. [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [19] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017. [20] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, pp. 28 492–28 518.

Related documents

• Data-Free Pruning of Self-Attention Layers in LLMs#1141466
• Where the World is Quiet#45539
• Rethinking Vacuity for OOD Detection in Evidential Deep Learning#684445
• Rethinking Vacuity for OOD Detection in Evidential Deep Learning#682670
• Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference#343423