Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neuralnetworks

machine learning, deep learning, neural networks

Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models Chun-Yi Kuan♡ , Wei-Ping Huang♡ , Hung-yi Lee♡♣ ♡ Graduate Institute of Communication Engineering, National Taiwan University, Taiwan ♣ Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan

arXiv:2604.25591v1 [eess.AS] 28 Apr 2026

[email protected]

Abstract—Recent audio-aware large language models (ALLMs) have demonstrated strong capabilities across diverse audio understanding and reasoning tasks, but they still frequently produce hallucinated or overly confident outputs. While uncertainty estimation has been extensively studied in text-only LLMs, it remains largely unexplored for ALLMs, where audio-conditioned generation introduces additional challenges such as perceptual ambiguity and cross-modal grounding. In this work, we present the first systematic empirical study of uncertainty estimation in ALLMs. We benchmark five representative methods, including predictive entropy, lengthnormalized entropy, semantic entropy, discrete semantic entropy, and P(True), across multiple models and diverse evaluation settings spanning general audio understanding, reasoning, hallucination detection, and unanswerable question answering. Our results reveal two key findings. First, semantic-level and verification-based methods consistently outperform token-level baselines on general audio reasoning benchmarks. Second, on trustworthiness-oriented benchmarks, the relative effectiveness of uncertainty methods becomes notably more model- and benchmark-dependent, indicating that conclusions drawn from general reasoning settings do not straightforwardly transfer to hallucination and unanswerable-question scenarios. We further explore uncertainty-based adaptive inference as a potential downstream application. We hope this study provides a foundation for future research on reliable, uncertainty-aware audio-language systems. Index Terms—uncertainty estimation, audio-aware LLMs

I. I NTRODUCTION Recent audio-aware large language models (ALLMs) [1]– [25] have rapidly advanced across a wide range of tasks, including audio understanding [26]–[32], spoken question answering [29], [33], [34], music reasoning , and general audio-language interaction [35]–[52]. By conditioning language generation on audio inputs, these models extend the capabilities of text-only LLMs to richer multimodal scenarios. However, strong performance does not necessarily imply reliability. In practice, ALLMs still frequently produce unsupported answers [53], hallucinated content [31], [32], [53]–[55], or overly confident responses [44], [56], especially when the audio evidence is ambiguous, incomplete, or even insufficient to answer the question. This limitation highlights a fundamental gap: beyond accuracy, we need to understand whether a model knows when

it may be wrong. Uncertainty estimation [57]–[79] provides a natural framework for addressing this problem. Reliable uncertainty estimates can support error detection, selective prediction, calibration, and safer deployment. In text-only LLMs, a large body of work [62], [65]–[79] has explored uncertainty estimation through predictive entropy, semantic entropy [65], and self-verification-based measures such as P(True) [62], showing that uncertainty can serve as an effective signal for both analysis and downstream decision-making. However, it remains unclear whether these insights transfer to audio-aware LLMs. Although ALLMs inherit language modeling capabilities from their text backbones, audioconditioned generation introduces additional challenges. The model must reason over noisy and imperfect perceptual inputs, ground its predictions in acoustic evidence, and avoid overreliance on language priors when the audio signal is weak or ambiguous. As a result, uncertainty signals that are effective in text-only settings might not behave the same when correctness depends on both perception and reasoning. In this work, we present the first systematic empirical study of uncertainty estimation in audio-aware LLMs. We evaluate several representative methods, including predictive entropy, length-normalized entropy, semantic entropy [65], discrete semantic entropy [65], and P(True) [62], across multiple state-of-the-art ALLMs [9], [11] and a diverse set of benchmarks [26], [27], [29]–[31], [44]. Our evaluation spans both general audio understanding and reasoning tasks [26], [27], [29], [30], and more challenging trustworthiness-oriented settings, including hallucination detection [31], [32] and unanswerable audio question answering [44]. Our results reveal two key findings. First, on general audio understanding and reasoning benchmarks, semantic-level and verification-based methods consistently outperform token-level likelihood-based baselines. This suggests that uncertainty signals that operate at the level of meaning or explicit self-evaluation are better aligned with correctness in audio-language tasks. Second, this advantage becomes notably less stable on trustworthinessoriented benchmarks, where the relative performance of uncertainty estimation methods is more strongly dependent on the model and the task. This indicates that conclusions drawn from standard reasoning benchmarks do not directly transfer

to settings that explicitly test reliability, such as hallucination and unanswerable questions. Finally, we explore uncertaintybased adaptive inference [80]–[83] as a possible downstream application. We show that uncertainty can be used as a routing signal to selectively trigger more expensive reasoning strategies. However, its effectiveness critically depends on whether the alternative reasoning mode is itself beneficial [84]–[93]. This highlights that uncertainty alone is not sufficient, as its utility is tightly coupled with the underlying inference strategy. Overall, this work provides the first comprehensive evaluation of uncertainty estimation in audio-aware LLMs, establishes empirical baselines, and reveals key differences between general reasoning and trustworthiness-oriented settings. We hope these findings provide a foundation for developing more reliable and uncertainty-aware audio-language systems. II. R ELATED W ORK A. Uncertainty Estimation in Language Models Uncertainty estimation has become an important topic in language modeling, as large language models (LLMs) can produce fluent yet unreliable outputs. Prior work [62], [65]– [79] has explored a range of uncertainty estimation methods, which can be broadly grouped into token-level, semantic-level, and verification-based approaches. Token-level methods quantify uncertainty from the predictive distribution over generated sequences, for example through predictive entropy and related likelihood-based measures. Semantic-level methods [65] instead account for the fact that multiple generations may differ in surface form while conveying the same meaning, and therefore measure uncertainty over clusters of semantically equivalent responses. Verification-based approaches [62], such as P(True), estimate confidence by prompting the model to assess whether its own answer is correct. Beyond measuring confidence itself, prior work has shown that uncertainty estimates can support downstream decision-making, including calibration [62], [71], [94], hallucination detection [66], [95], and selective prediction [96]–[98]. In particular, semantic entropy has been shown to be effective for detecting hallucinations [66] in natural language generation by measuring uncertainty modulo semantic equivalence. Similarly, selfevaluation-based confidence estimates such as P(True) have been used to assess whether a model’s generated answer is likely to be correct, and subsequent work has shown that selfevaluation can further improve selective generation in LLMs. More broadly, uncertainty estimates have also been used in downstream settings such as hallucination detection, selective prediction, and adaptive generation. Despite this growing literature, most existing findings are established in text-only language models. Whether these uncertainty estimation methods remain effective in audio-aware LLMs is still unclear, since audio-conditioned generation introduces additional challenges such as perceptual ambiguity, cross-modal grounding, and reliance on language priors. Moreover, the use of uncertainty estimates for downstream decisionmaking remains much less explored in audio-aware LLMs. Our work builds on this line of research and systematically

investigates how uncertainty estimation methods behave in audio-aware LLMs. B. Audio-aware LLM Evaluation and Trustworthiness As audio-aware LLMs have rapidly advanced, recent work has introduced increasingly diverse benchmarks to evaluate their capabilities. Existing benchmarks cover general audio understanding and reasoning [26]–[30], [33], [34], [39]–[43], [45]–[51] across speech, environmental sounds, and music, with recent efforts expanding evaluation toward more challenging reasoning settings such as expert-level audio understanding and multi-hop reasoning [27], [28], [30], [48]. These developments have substantially improved the empirical coverage of audio-aware LLM evaluation. At the same time, recent work has begun to highlight trustworthiness issues [31], [32], [35], [40], [44], [51], [53]–[55] in audio-aware LLMs. Prior studies [31], [32] show that these models can hallucinate sound events that are not present in the input audio, revealing a gap between surface-level fluency and faithful audio grounding [31]. More recent benchmarks [44] extend this line of work by explicitly evaluating unanswerable audio questions and other trustworthiness-oriented settings. Together, these works suggest that evaluating audio-aware LLMs should not be limited to answer accuracy alone, but should also consider whether model outputs are reliable, grounded, and appropriately calibrated with respect to the input audio. Despite this growing interest in evaluating audio-aware LLM capabilities and trustworthiness, uncertainty estimation remains largely underexplored in this setting. Existing studies primarily assess what models get right or wrong, but provide limited insight into whether models can reliably indicate when they may be wrong. Our work addresses this gap by systematically studying uncertainty estimation in audio-aware LLMs across both general reasoning and trustworthiness-oriented benchmarks. Beyond benchmark evaluation, it also remains unclear whether uncertainty estimates can support trustworthiness-oriented decision making in audio-aware LLMs, such as hallucination detection or adaptive inference. III. M ETHOD A. Problem Setup Given an input x consisting of an audio clip and a corresponding instruction or question, an audio-aware LLM generates an answer y according to the conditional distribution p(y | x). Our goal is to estimate how uncertain the model is about its prediction. Therefore, we compare several representative uncertainty estimation methods, including predictive entropy, length-normalized predictive entropy, semantic entropy [65], discrete semantic entropy [65], and P(True) [62]. Predictive entropy and its length-normalized variant operate on the likelihood of generated sequences, semantic entropy measures uncertainty over semantically equivalent answer clusters, discrete semantic entropy estimates uncertainty over a discrete answer space, and P(True) measures the model’s self-assessed probability that its answer is correct.

B. Prediction and Uncertainty Estimation Protocol Following common practice [62], [65] in sampling-based uncertainty estimation for language models, we adopt a twostage protocol that decouples prediction from uncertainty estimation. Given an input x, we first obtain a prediction ŷ using low-temperature decoding, which serves as the model’s final answer for correctness evaluation. We then estimate uncertainty by sampling multiple responses from the predictive distribution p(y | x) and computing uncertainty measures over the resulting samples. Formally, let ŷ denote the prediction obtained via low-temperature decoding. We evaluate whether the model is correct based on ŷ. Separately, we draw K stochastic samples {yi }K i=1 from p(y | x) and use them to estimate uncertainty, such as predictive entropy, lengthnormalized entropy, semantic entropy, and discrete semantic entropy. For P(True), instead of estimating uncertainty from sampled responses, we directly verify the low-temperature prediction ŷ by prompting the model to judge whether ŷ is supported by the input x. C. Uncertainty Estimation Methods Predictive entropy measures the overall uncertainty of the model’s generated answer under the conditional distribution p(y | x). Formally, the sequence-level predictive entropy is defined as Hpred (x) = Ey∼p(y|x) [− log p(y | x)] .

(1)

Since the exact expectation is intractable, we approximate it with Monte Carlo sampling. Given K sampled answers {yi }K i=1 from p(y | x), we estimate K

Hpred (x) ≈

1 X − log p(yi | x). K i=1

(2)

A higher predictive entropy indicates that the model assigns lower probability to its sampled generations on average, reflecting greater uncertainty at the sequence level. Predictive entropy can be biased by answer length, since longer sequences typically accumulate larger negative log-likelihoods. To reduce this effect, we additionally consider a length-normalized predictive entropy that measures average uncertainty per token: − log p(y | x) Hnorm (x) = Ey∼p(y|x) , (3) |y| where |y| denotes the number of tokens in y. A straightforward Monte Carlo estimator is K

Hnorm (x) ≈

1 X − log p(yi | x) . K i=1 |yi |

(4)

In practice, however, we adopt a token-level estimator by normalizing the total negative log-likelihood with the total number of generated tokens: PK − log p(yi | x) tok Hnorm (x) = i=1PK . (5) i=1 |yi |

This formulation can be viewed as a ratio-of-expectations approximation, and effectively estimates the average tokenlevel uncertainty across sampled responses. Compared with per-sequence normalization, it is more stable in practice and less sensitive to variance induced by different answer lengths. Sequence-level likelihood does not distinguish between surface-form variation and genuine semantic uncertainty. Multiple sampled answers may differ lexically while expressing the same meaning. To capture uncertainty at the semantic level, we follow and cluster sampled answers by semantic equivalence. Specifically, semantic equivalence is determined via bidirectional entailment clustering, following the original implementation. Given two sampled answers s and s′ , we concatenate each answer with the original context x and use a natural language inference (NLI) model to test whether (x, s) entails (x, s′ ) and vice versa. Two answers are assigned to the same semantic cluster if and only if both entailment relations hold. Let {ck } denote the resulting semantic clusters. The probability mass of a cluster is obtained by summing the sequence probabilities of all answers assigned to that cluster: X p(ck | x) = p(s | x). (6) s∈ck

The semantic entropy is then defined as X Hsem (x) = − p(ck | x) log p(ck | x).

(7)

Intuitively, semantic entropy is low when most sampled answers collapse into the same meaning cluster, and high when the model spreads probability mass across multiple semantically distinct answers. While semantic entropy measures uncertainty over clusters of semantically equivalent free-form answers, some tasks involve a predefined discrete answer space. In such cases, semantic variation is naturally constrained by the candidate answer set, and uncertainty can be estimated directly over the model’s empirical distribution on answer choices. We refer to this variant as discrete semantic entropy. Let A denote the set of candidate answers. Given K sampled responses for the same input x, we map each response to an answer option in A and estimate the empirical answer distribution as count(a) , a ∈ A. (8) p(a | x) ≈ K The discrete semantic entropy is then defined as X Hdisc (x) = − p(a | x) log p(a | x). (9) a∈A

This quantity is low when the sampled answers concentrate on a single option, and high when the model spreads probability mass across multiple candidate answers. It can be viewed as a discrete counterpart of semantic entropy for closed-form answer spaces, avoiding the need for semantic equivalence clustering. In addition to entropy-based methods, we consider P(True), a self-verification-based confidence estimate. Given an input x and a candidate answer ŷ, the model is prompted to judge

whether the answer is supported by the input. In our setting, this corresponds to self-verification conditioned on the audio input. Formally, we define P (true | x, ŷ) = p(true | x, ŷ)

(10)

where the model predicts whether ŷ is correct or supported by the input. In practice, we instantiate P(True) with a selfverification prompt that asks the model to judge whether a candidate answer is supported by the input audio. The model is constrained to produce a binary decision (true or false), and the confidence score is computed from the normalized likelihood of these two verification tokens. The full prompt templates used in our experiments are provided in Table VII. A higher P(True) indicates that the model assigns higher confidence to the correctness of its own answer. In this work, we use self-verification only, without relying on an external verifier model. Overall, these methods quantify uncertainty at different levels: predictive entropy and length-normalized entropy measure uncertainty from sequence likelihoods, semantic entropy and discrete semantic entropy capture dispersion over meaning or answer choices, and P(True) estimates confidence through explicit self-verification. IV. E XPERIMENTAL S ETUP A. Evaluation Datasets, Models, and Setups We evaluate uncertainty estimation across diverse benchmarks and representative audio-aware LLMs. Our evaluation covers two complementary settings. The first consists of general audio understanding and reasoning benchmarks, including MMAU [26], MMAR [27], MMSU [29], and SAKURA [30]. The second focuses on trustworthiness-oriented evaluation, including audio hallucination and unanswerable question answering benchmarks such as Audio-Hallucination [31] and AQUA-Bench [44]. We follow the evaluation protocols defined in each benchmark to ensure consistency with prior work. We conduct experiments on several representative audio-aware LLMs, including Qwen2.5-Omni-3B [11], Qwen2.5-Omni7B [11], and Audio Flamingo 3 [9]. These models are chosen for their strong performance and broad use in recent audiolanguage studies. For the two-stage protocol described in Section III-B, we follow common practice in sampling-based uncertainty estimation for language models. Specifically, we use low-temperature decoding with temperature 0.1 to obtain the prediction ŷ for correctness evaluation, so that ŷ more closely reflects the model’s most likely answer. For sampling-based uncertainty estimation, we draw K = 10 stochastic samples using temperature 1.0 to better characterize the predictive distribution. The evaluation prompt template is shown in Table V. In all experiments, inference is performed using a single NVIDIA RTX 3090 GPU. B. Evaluation Metrics We evaluate uncertainty estimation using two metrics: AUROC and AURAC. We use the area under the receiver op-

erating characteristic curve (AUROC) to measure how well an uncertainty estimate separates correct predictions from incorrect ones. In our setting, each example is associated with a correctness label and an uncertainty score. A good uncertainty estimate should assign higher uncertainty to incorrect predictions and lower uncertainty to correct ones. AUROC summarizes this ranking quality across all possible decision thresholds. A higher AUROC indicates that the uncertainty score is more effective at distinguishing errors from correct answers. We also use the area under the rejection–accuracy curve (AURAC) to evaluate the usefulness of uncertainty estimates for selective prediction. The rejection–accuracy curve plots the model accuracy after progressively rejecting the most uncertain examples. If an uncertainty estimate is reliable, removing high-uncertainty predictions should lead to a faster increase in accuracy on the remaining examples. A higher AURAC therefore indicates that the uncertainty estimate is more useful for identifying predictions that should be abstained from or handled with additional care. We also report the original accuracy on each benchmark as a reference, since it reflects the model’s default performance without abstention. Comparing original accuracy with AURAC helps clarify whether the uncertainty estimate provides meaningful practical utility for selective prediction beyond the model’s base accuracy. Overall, AUROC captures the error-detection ability of an uncertainty estimate, whereas AURAC reflects its practical utility under selective prediction. V. R ESULTS AND A NALYSIS A. Overall Comparison of Uncertainty Estimation Methods Tables I and II present the overall results of uncertainty estimation across benchmarks and models. Overall, we observe a clear pattern: semantic-based and verification-based methods, including discrete semantic entropy, semantic entropy, and P(True), generally outperform token-level baselines such as predictive entropy and normalized entropy. This trend is especially consistent on general audio understanding and reasoning benchmarks, where the strongest results are usually achieved by discrete semantic entropy, semantic entropy, or P(True). Among the compared methods, semantic entropy and discrete semantic entropy are the most consistently strong performers. On MMAU, MMAR, MMSU, and SAKURA, these two methods frequently achieve the best or near-best AUROC across different models. P(True) is also competitive, and in some cases achieves the strongest results, such as on MMAR and SAKURA for Qwen2.5-Omni-3B and on AQUA for Qwen2.5-Omni-7B. These results suggest that uncertainty estimates operating at the semantic level, or through explicit self-verification, are generally more effective than methods based purely on token-level likelihood. In contrast, predictive entropy is usually the weakest method, and normalized entropy, while sometimes improving over predictive entropy, still tends to underperform relative to semantic entropy, discrete semantic entropy, and P(True). This gap indicates that token-level uncertainty alone is often insufficient to capture answer correctness in audio-aware LLMs.

TABLE I U NCERTAINTY ESTIMATION RESULTS ON U NDERSTANDING & R EASONING BENCHMARKS . T HE BEST AUROC IN EACH ROW IS HIGHLIGHTED WITH A GREEN BACKGROUND , WHILE THE BEST AURAC IS HIGHLIGHTED WITH AN ORANGE BACKGROUND . G RAY SHADING INDICATES THE MODEL’ S ORIGINAL ACCURACY ON THE CORRESPONDING BENCHMARK . Discrete Sem. Entropy

Semantic Entropy

Predictive Entropy

Normalized Entropy

Benchmark

Model

Acc.

AUROC

AURAC

AUROC

AURAC

AUROC

P(True) AURAC

AUROC

AURAC

AUROC

AURAC

MMAU [26]

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.71 0.73 0.74

0.85 0.81 0.80

0.90 0.88 0.89

0.84 0.82 0.82

0.90 0.89 0.90

0.82 0.78 0.72

0.89 0.89 0.86

0.64 0.62 0.72

0.81 0.82 0.88

0.69 0.66 0.73

0.82 0.84 0.88

MMAR [27]

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.58 0.58 0.55

0.69 0.69 0.65

0.73 0.71 0.69

0.70 0.70 0.67

0.74 0.71 0.69

0.70 0.71 0.61

0.75 0.74 0.65

0.59 0.63 0.60

0.68 0.69 0.66

0.62 0.62 0.58

0.67 0.68 0.64

MMSU [29]

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.62 0.60 0.55

0.80 0.80 0.71

0.82 0.81 0.71

0.80 0.80 0.74

0.81 0.81 0.73

0.78 0.79 0.69

0.81 0.81 0.71

0.60 0.54 0.61

0.71 0.66 0.68

0.69 0.68 0.64

0.76 0.75 0.68

SAKURA [30]

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.70 0.69 0.62

0.77 0.76 0.67

0.84 0.84 0.75

0.76 0.77 0.71

0.83 0.84 0.78

0.76 0.79 0.69

0.83 0.87 0.75

0.63 0.63 0.64

0.78 0.79 0.75

0.59 0.60 0.58

0.73 0.74 0.71

TABLE II U NCERTAINTY ESTIMATION RESULTS ON T RUSTWORTHINESS BENCHMARKS . T HE BEST AUROC IN EACH ROW IS HIGHLIGHTED WITH A GREEN BACKGROUND , WHILE THE BEST AURAC IS HIGHLIGHTED WITH AN ORANGE BACKGROUND . G RAY SHADING INDICATES THE MODEL’ S ORIGINAL ACCURACY ON THE CORRESPONDING BENCHMARK . Discrete Sem. Entropy

Semantic Entropy

Predictive Entropy

Normalized Entropy

Benchmark

Model

Acc.

AUROC

AURAC

AUROC

AURAC

AUROC

AURAC

AUROC

AURAC

AUROC

AURAC

AQUA-Bench [44]

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.68 0.76 0.18

0.70 0.71 0.82

0.80 0.87 0.34

0.71 0.72 0.83

0.80 0.87 0.34

0.79 0.52 0.89

0.85 0.80 0.40

0.55 0.72 0.81

0.73 0.88 0.34

0.62 0.75 0.81

0.77 0.89 0.34

Hallucination [31]

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.89 0.89 0.80

0.78 0.74 0.74

0.95 0.95 0.89

0.78 0.77 0.75

0.95 0.95 0.89

0.71 0.55 0.44

0.95 0.93 0.83

0.66 0.75 0.75

0.93 0.95 0.91

0.69 0.71 0.78

0.94 0.95 0.92

In many cases, generated responses may be fluent and locally likely at the token level while still being semantically incorrect or weakly grounded in the input audio. At the same time, the trustworthiness-oriented benchmarks reveal a more nuanced picture. While semantic-based and verification-based methods remain competitive overall, the relative ranking of methods becomes less stable across models and datasets. For example, on AQUA, P(True) performs best for Qwen2.5-Omni-7B, whereas normalized entropy performs best for Qwen2.5-Omni3B, and P(True) shows particularly strong AUROC for Audio Flamingo 3. These results suggest that uncertainty estimation in trustworthiness-oriented settings is more model-dependent and benchmark-dependent. B. General Audio Understanding and Reasoning Benchmarks We next examine benchmark-specific results on general audio understanding and reasoning tasks, including MMAU, MMAR, MMSU, and SAKURA. As shown in Table I, the overall trend is highly consistent across these benchmarks: semantic entropy, discrete semantic entropy, and P(True) generally outperform predictive entropy and normalized entropy. This suggests that uncertainty estimates based on semantic variation or explicit self-verification are more effective than token-level likelihood-based measures for identifying incorrect predictions in audio-aware LLMs.

P(True)

Among these methods, semantic entropy and discrete semantic entropy are the most consistently strong performers. On MMAU, semantic entropy achieves the best AUROC for Qwen2.5-Omni-3B and Audio Flamingo 3, while discrete semantic entropy performs best for Qwen2.5-Omni-7B. On MMSU, semantic entropy and discrete semantic entropy again dominate across all three models. On SAKURA, P(True) becomes more competitive, achieving the best AUROC for Qwen2.5-Omni-3B, while semantic entropy and discrete semantic entropy remain strong for the other models. Across these benchmarks, predictive entropy is typically the weakest method, and normalized entropy, although sometimes stronger than predictive entropy, still generally falls behind the semantic- and verification-based approaches. MMAR shows a slightly more mixed pattern, but still follows the same broad trend. For Qwen2.5-Omni-7B and Audio Flamingo 3, semantic entropy achieves the best AUROC, while P(True) performs best for Qwen2.5-Omni-3B. Even in this case, however, tokenlevel entropy measures remain consistently weaker. This suggests that although the relative ranking among the strongest methods can vary by benchmark and model, semantic-based and verification-based uncertainty estimates are overall more robust than token-level baselines. A more fine-grained breakdown of MMAR and MMSU into Perception and Reasoning subtasks is shown in Table III. The subtask-level results reveal that the same high-level pattern

largely holds within both categories, but also expose some interesting differences. On MMAR-Perception, P(True) is particularly competitive for the Qwen models, whereas on MMARReasoning, discrete semantic entropy and semantic entropy become relatively stronger. On MMSU, discrete semantic entropy and semantic entropy remain highly competitive across both subtasks, while P(True) becomes especially strong on the reasoning subset for Qwen2.5-Omni-7B. These results suggest that uncertainty estimation behavior is not determined solely by the benchmark as a whole, but can also depend on whether the task is more perception-driven or reasoningdriven. Overall, the results on general audio understanding and reasoning benchmarks show a consistent advantage for semantic-based and verification-based uncertainty estimates. At the same time, the subtask breakdown indicates that the relative strengths of these methods can shift depending on task structure. C. Trustworthiness-oriented Benchmarks We next examine the trustworthiness-oriented benchmarks, including AQUA-Bench [44] and Audio-Hallucination [31]. Compared with the general audio understanding and reasoning benchmarks, these tasks present a more challenging and nuanced setting for uncertainty estimation, as they more directly test whether the model can recognize unsupported, hallucinated, or unanswerable cases. As shown in Table II, the overall pattern becomes less uniform across models and benchmarks, indicating that uncertainty estimation in trustworthiness-oriented settings is more sensitive to both task structure and model behavior. On AQUA-Bench, the relative ranking of uncertainty estimation methods varies substantially across models. For Qwen2.5-Omni-7B, P(True) achieves the best AUROC, suggesting that explicit self-verification is particularly effective for detecting unanswerable or unsupported predictions in this setting. In contrast, for Qwen2.5-Omni-3B, normalized entropy achieves the best AUROC, with predictive entropy also performing competitively. For Audio Flamingo 3, P(True) again performs strongly, while semantic entropy and predictive entropy also remain highly competitive. This variability contrasts with the more consistent trends on general reasoning benchmarks, suggesting that uncertainty estimation for unanswerable audio questions is more sensitive to each model’s behavior when the audio does not provide enough evidence. The Audio-Hallucination benchmark shows a somewhat more stable pattern, but still exhibits clear model dependence. For the two Qwen models, semantic entropy and discrete semantic entropy achieve the strongest AUROC results, indicating that semantic-level uncertainty remains highly informative for detecting hallucinated audio-grounded answers. However, for Audio Flamingo 3, normalized entropy achieves the best AUROC, while predictive entropy is also competitive. This suggests that hallucination detection does not always favor the same uncertainty signal across different models, even when the benchmark itself is fixed. Overall, the trustworthinessoriented benchmarks reveal a more heterogeneous picture than

the general understanding and reasoning benchmarks. Rather than showing a single consistently dominant method, they highlight stronger benchmark- and model-dependent variation in the usefulness of uncertainty estimates. This result is itself important: it suggests that uncertainty estimation for trustworthiness-oriented audio tasks may be inherently more difficult, and that conclusions drawn from general reasoning benchmarks do not always transfer directly to hallucination and unanswerable-question settings. D. Application: Uncertainty-based Adaptive Inference Beyond evaluating uncertainty estimation as a diagnostic signal, we further explore whether it can be used to improve inference-time decision making. The key motivation is that not all examples require the same level of inference effort: some can be answered correctly with direct inference, while others are more challenging and may benefit from a more deliberate reasoning process [88], [93]. Since reasoning-based inference is typically more expensive, it is undesirable to apply it uniformly to all inputs. This makes uncertainty a natural routing signal. Let u(x) denote the uncertainty score of the direct prediction for input x, and let τ denote a predefined threshold. We define adaptive inference as ( ŷreason (x), if u(x) > τ, ŷadaptive (x) = (11) ŷdirect (x), otherwise, where ŷdirect (x) is the answer produced by direct inference, and ŷreason (x) is the answer produced by a more deliberate reasoning mode. In our experiments, the reasoning mode follows a caption-then-reason strategy: the model is first prompted to describe the audio content, and is then asked to answer the question step by step based on that description. In our experiments, the reasoning mode follows a caption-then-reason strategy. The model is first prompted to describe the audio content, and then answers the question step by step. This describe-then-reason decomposition has been explored in prior work on audio reasoning and timestamped audio captioning, where explicit intermediate descriptions are used to support downstream reasoning [32]. The full prompting templates used in reasoning mode are presented in Table VI. We set the routing threshold to τ = 0.25. In preliminary experiments, we found that performance was relatively stable within a small range around this value, so we use τ = 0.25 as a fixed threshold in the main results. Table IV compares three inference strategies: direct inference, reasoning-based inference, and uncertainty-based adaptive inference. Overall, adaptive inference is beneficial in some settings, but its gains are not universal. In particular, it tends to help when the reasoning mode itself is competitive with or better than direct inference. For example, on MMAU and SAKURA, reasoning improves over direct inference for the Qwen models, and adaptive inference often matches or further improves upon the stronger strategy. On MMAU, adaptive inference achieves the best accuracy for both Qwen2.5-Omni7B and Qwen2.5-Omni-3B, while on SAKURA it remains competitive with the best reasoning-based results. We also

TABLE III B REAKDOWN OF MMAR AND MMSU BY P ERCEPTION VS . R EASONING SUBTASKS . T HE BEST AUROC IN EACH ROW IS HIGHLIGHTED WITH A GREEN BACKGROUND , WHILE THE BEST AURAC IS HIGHLIGHTED WITH AN ORANGE BACKGROUND . G RAY SHADING INDICATES THE MODEL’ S ORIGINAL ACCURACY ON THE CORRESPONDING BENCHMARK .

Benchmark

Discrete Sem. Entropy

Semantic Entropy

Predictive Entropy

Normalized Entropy

Subtask

Model

Acc.

AUROC

AURAC

AUROC

AURAC

AUROC

AURAC

AUROC

AURAC

AUROC

AURAC

Perception

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.56 0.53 0.51

0.68 0.68 0.64

0.70 0.67 0.63

0.70 0.69 0.65

0.71 0.68 0.64

0.73 0.73 0.63

0.75 0.74 0.63

0.61 0.62 0.61

0.67 0.65 0.63

0.58 0.59 0.56

0.64 0.63 0.58

Reasoning

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.62 0.63 0.62

0.74 0.70 0.67

0.78 0.76 0.76

0.73 0.69 0.69

0.78 0.76 0.77

0.69 0.68 0.58

0.75 0.76 0.70

0.59 0.62 0.61

0.71 0.72 0.72

0.64 0.62 0.61

0.72 0.72 0.72

Perception

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.45 0.45 0.41

0.73 0.71 0.64

0.66 0.63 0.55

0.72 0.70 0.66

0.64 0.63 0.56

0.72 0.69 0.66

0.64 0.63 0.56

0.58 0.52 0.48

0.54 0.49 0.43

0.58 0.58 0.50

0.54 0.54 0.46

Reasoning

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.79 0.77 0.69

0.78 0.81 0.70

0.91 0.91 0.81

0.78 0.82 0.75

0.90 0.91 0.82

0.80 0.80 0.69

0.92 0.92 0.82

0.58 0.56 0.66

0.84 0.81 0.82

0.67 0.67 0.62

0.87 0.86 0.79

MMAR [27]

MMSU [29]

report the relative token cost of adaptive inference compared to full reasoning. Adaptive inference achieves comparable or better accuracy while using only 24–64% of the tokens required by full reasoning. Figure 1 presents a Pareto analysis comparing Reasoning and Adaptive inference, illustrating that Adaptive operating points consistently shift toward lower cost without proportional accuracy loss. In contrast, when reasoning is not helpful, adaptive inference provides limited benefit and can even underperform direct inference. This is most clearly seen on MMAR, where reasoning substantially degrades performance for all three models, and adaptive inference correspondingly fails to improve over direct inference. A similar pattern is observed for Audio Flamingo 3 on MMAU, where direct inference is strongest and adaptive inference remains slightly below it. These results suggest that uncertainty alone is not sufficient to guarantee better routing decisions: adaptive inference can only be effective when the fallback reasoning mode is genuinely useful on uncertain examples. This finding is also consistent with recent observations in the audio-aware LLM literature. Prior works [84]–[87] have shown that simply introducing chain-of-thought or reasoningstyle inference does not always improve performance in audioaware LLMs, and can even lead to degraded results when the model is not properly trained for reasoning. Our results reinforce this point from the perspective of uncertainty-aware routing: the usefulness of adaptive inference depends not only on the quality of the uncertainty estimate, but also on whether the alternative reasoning strategy is actually beneficial for the target benchmark and model. VI. D ISCUSSION AND F UTURE W ORKS Our study focuses on uncertainty estimation as an evaluation problem, but its value extends beyond benchmarking. More broadly, uncertainty estimates can serve as a useful interface between model predictions and downstream decisionmaking. Rather than treating every model output equally, a system equipped with reliable uncertainty estimates can decide when to trust its prediction, when to abstain, and when to invoke additional computation or verification. In this sense,

P(True)

TABLE IV ACCURACY UNDER DIFFERENT INFERENCE STRATEGIES . L IGHT BLUE SHADING INDICATES PERFORMANCE BETTER THAN THE D IRECT A NSWER BASELINE . VALUES IN PARENTHESES DENOTE THE RELATIVE TOKEN COST OF A DAPTIVE INFERENCE COMPARED TO R EASONING . Benchmark

Model

Direct

Reasoning

Adaptive

MMAU

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.71 0.73 0.75

0.75 0.73 0.65

0.76 (41%) 0.74 (43%) 0.74 (24%)

MMAR

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.59 0.58 0.56

0.54 0.53 0.47

0.58 (61%) 0.57 (64%) 0.53 (47%)

MMSU

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.62 0.60 0.56

0.63 0.61 0.54

0.62 (52%) 0.62 (51%) 0.56 (54%)

SAKURA

Qwen-2.5-Omni-7B Qwen-2.5-Omni-3B Audio Flamingo 3

0.70 0.69 0.63

0.77 0.73 0.70

0.75 (56%) 0.73 (56%) 0.70 (61%)

uncertainty estimation is not only a diagnostic tool, but also a potential mechanism for making audio-aware LLMs more reliable in practice. One immediate application is decisionmaking at inference time. As explored in our adaptive inference experiments (Section V-D), uncertainty can be used as a routing signal to decide whether an input should be handled by direct inference or a more expensive reasoning mode. More generally, uncertainty could be used as a threshold for abstention, fallback to external tools, or escalation to human review in high-stakes settings. This is especially relevant for audio-aware LLMs, where unsupported answers, hallucinated sound events, and failures on unanswerable questions can be difficult to detect from model outputs alone. Another promising direction is to use uncertainty as an optimization signal during training. For example, uncertainty estimates could be used to prioritize difficult or ambiguous examples, reweight training instances, or guide data selection in self-training and reinforcement learning pipelines [79]. In principle, such signals may provide a lightweight way to improve model reliability without requiring dense supervision.

Fig. 1. Cost–accuracy Pareto frontier of Reasoning vs. Adaptive inference across four benchmarks. Each point represents a model under a fixed inference mode: hollow squares (Reasoning, 100% token cost) and filled circles (Adaptive, reduced cost). Dashed arrows indicate the shift from full reasoning to adaptive inference for each model. The gray dashed line represents the Pareto frontier, which consists of operating points that are not dominated in terms of both token cost and accuracy.

This is particularly attractive because uncertainty estimation is often unsupervised: it can be computed directly from model generations or self-verification signals, without requiring additional human annotations. At the same time, our results suggest that uncertainty should not be viewed as a universally reliable solution. Its value depends on the task, the model, and the downstream application. In particular, our adaptive inference results show that uncertainty-based routing improves performance only when the alternative inference strategy is itself stronger than the default strategy. More broadly, uncertainty estimates in audio-aware LLMs may be affected by modalityspecific factors such as perceptual ambiguity, weak audio grounding, and language priors. Future work should therefore investigate not only better uncertainty estimation methods, but also better ways of integrating uncertainty into training and inference. VII. C ONCLUSION This work presents the first systematic study of uncertainty estimation in audio-aware large language models. Our findings show that semantic-level and verification-based methods generally outperform token-level baselines, but this advantage is most stable on general reasoning tasks. On trustworthinessoriented benchmarks, uncertainty estimation becomes more model- and task-dependent, suggesting that these settings pose fundamentally different challenges that deserve dedicated investigation. We hope this work provides a useful foundation for building more reliable audio-language systems. VIII. L IMITATIONS Our study has four main limitations. First, we primarily evaluate settings with relatively constrained answer spaces where correctness can be clearly defined. How well the

studied uncertainty estimation methods generalize to openended audio-language tasks, such as free-form audio question answering, captioning, or dialogue, remains an open question for future work. Second, the uncertainty estimation methods we study are largely inherited from the text LLM literature. While this design choice enables a controlled, method-agnostic comparison and helps establish baseline findings for the audiolanguage setting, these methods are not designed to explicitly model uncertainty arising from audio perception itself. Prior work in the vision-language domain [99], [100] has begun to develop modality-aware uncertainty estimation methods tailored to visual perception; analogous efforts for audiolanguage models represent a promising research direction. Third, we do not explore uncertainty signals derived from internal multimodal representations. For example, intermediatelayer attention patterns or hidden-state geometry may capture audio-specific ambiguity that token-level methods cannot surface. Investigating such representation-level signals could complement the output-level methods studied here and yield richer uncertainty estimates for audio-language models. Finally, our adaptive inference experiments employ a thresholdbased routing strategy with a fixed fallback reasoning mode. More sophisticated decision-making frameworks, such as learned routing policies, cascaded inference pipelines, or costaware selection mechanisms, could further improve the tradeoff between computational cost and prediction reliability. R EFERENCES [1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. [2] Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. R. Glass, “Listen, think, and understand,” in The Twelfth International Conference on Learning Representations, 2023. [3] Y. Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass, “Joint audio and speech understanding,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8. [4] C. Wang, M. Liao, Z. Huang, J. Lu, J. Wu, Y. Liu, C. Zong, and J. Zhang, “Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing,” arXiv preprint arXiv:2309.00916, 2023. [5] Y. Fathullah, C. Wu, E. Lakomkin, K. Li, J. Jia, Y. Shangguan, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. Seltzer, “Audiochatllama: Towards general-purpose speech abilities for llms,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024, pp. 5522–5532. [6] C.-Y. Kuan, C.-K. Yang, W.-P. Huang, K.-H. Lu, and H.-y. Lee, “Speech-copilot: Leveraging large language models for speech processing via task decomposition, modularization, and program generation,” in 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 1060–1067. [7] K.-W. Chang, H. Wu, Y.-K. Wang, Y.-K. Wu, H. Shen, W.-C. Tseng, I.-t. Kang, S.-W. Li, and H.-y. Lee, “Speechprompt: Prompting speech language models for speech processing tasks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024. [8] C. Wang, M. Liao, Z. Huang, J. Wu, C. Zong, and J. Zhang, “Blsp-emo: Towards empathetic large speech-language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 19 186–19 199. [9] S. Ghosh, A. Goel, J. Kim, S. Kumar, Z. Kong, S.-g. Lee, C.-H. H. Yang, R. Duraiswami, D. Manocha, R. Valle et al., “Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.

[10] A. H. Liu, A. Ehrenberg, A. Lo, C. Denoix, C. Barreau, G. Lample, J.-M. Delignon, K. R. Chandu, P. von Platen, P. R. Muddireddy et al., “Voxtral,” arXiv preprint arXiv:2507.13264, 2025. [11] J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang et al., “Qwen2. 5-omni technical report,” arXiv preprint arXiv:2503.20215, 2025. [12] C.-K. Yang, Y.-K. Fu, C.-A. Li, Y.-C. Lin, Y.-X. Lin, W.-C. Chen, H. L. Chung, C.-Y. Kuan, W.-P. Huang, K.-H. Lu et al., “Building a taiwanese mandarin spoken language model: A first attempt,” arXiv preprint arXiv:2411.07111, 2024. [13] C.-Y. Kuan and H.-y. Lee, “Teaching audio-aware large language models what does not hear: Mitigating hallucinations through synthesized negative samples,” in Proc. Interspeech 2025, 2025, pp. 2073–2077. [14] ——, “From alignment to advancement: Bootstrapping audio-language alignment with synthetic data,” IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 4604–4619, 2025. [15] Y. Guo, Z. Li, H. Wang, B. Li, C. Shao, H. Zhang, C. Du, X. Chen, S. Liu, and K. Yu, “Recent advances in discrete speech tokens: A review,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. [16] H. Wu, X. Chen, Y.-C. Lin, K.-w. Chang, H.-L. Chung, A. H. Liu, and H.-y. Lee, “Towards audio language modeling–an overview,” arXiv preprint arXiv:2402.13236, 2024. [17] P. Mousavi, G. Maimon, A. Moumen, D. Petermann, J. Shi, H. Wu, H. Yang, A. Kuznetsova, A. Ploujnikov, R. Marxer et al., “Discrete audio tokens: More than a survey!” arXiv preprint arXiv:2506.10274, 2025. [18] W. Cui, D. Yu, X. Jiao, Z. Meng, G. Zhang, Q. Wang, S. Y. Guo, and I. King, “Recent advances in speech language models: A survey,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 13 943– 13 970. [19] J. Peng, Y. Wang, B. Li, Y. Guo, H. Wang, Y. Fang, Y. Xi, H. Li, X. Li, K. Zhang et al., “A survey on speech large language models for understanding,” IEEE Journal of Selected Topics in Signal Processing, 2025. [20] S. Ji, Y. Chen, M. Fang, J. Zuo, J. Lu, H. Wang, Z. Jiang, L. Zhou, S. Liu, X. Cheng et al., “Wavchat: A survey of spoken dialogue models,” arXiv preprint arXiv:2411.13577, 2024. [21] A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen et al., “Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras,” arXiv preprint arXiv:2503.01743, 2025. [22] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen et al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” arXiv preprint arXiv:2507.06261, 2025. [23] S. Arora, K.-W. Chang, C.-M. Chien, Y. Peng, H. Wu, Y. Adi, E. Dupoux, H.-y. Lee, K. Livescu, and S. Watanabe, “On the landscape of spoken language models: A comprehensive survey,” Transactions on Machine Learning Research, 2025. [24] S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro, “Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities,” in Forty-second International Conference on Machine Learning, 2024. [25] K.-W. Chang, W.-C. Chen, E.-P. Hu, H.-y. Lee, and J. Glass, “Tico: Time-controllable training for spoken dialogue models,” arXiv preprint arXiv:2603.22267, 2026. [26] S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha, “Mmau: A massive multitask audio understanding and reasoning benchmark,” in The Thirteenth International Conference on Learning Representations, 2025. [27] Z. Ma, Y. Ma, Y. Zhu, C. Yang, Y.-W. Chao, R. Xu et al., “Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix,” arXiv preprint arXiv:2505.13032, 2025. [28] S. Kumar, Š. Sedláček, V. Lokegaonkar, F. López, W. Yu, N. Anand, H. Ryu, L. Chen, M. Plička, M. Hlaváček et al., “Mmau-pro: A challenging and comprehensive benchmark for holistic evaluation of audio general intelligence,” arXiv preprint arXiv:2508.13992, 2025. [29] D. Wang, J. Wu, J. Li, D. Yang, X. Chen, T. Zhang, and H. Meng, “Mmsu: A massive multi-task spoken language understanding and reasoning benchmark,” arXiv preprint arXiv:2506.04779, 2025.

[30] C.-K. Yang et al., “Sakura: On the multi-hop reasoning of large audio-language models based on speech and audio information,” in Interspeech 2025, 2025. [31] C.-Y. Kuan, W.-P. Huang, and H.-y. Lee, “Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models,” in Interspeech 2025, 2025. [32] C.-Y. Kuan and H.-y. Lee, “Can large audio-language models truly hear? tackling hallucinations with multi-task assessment and stepwise audio reasoning,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5. [33] C.-y. Huang, K.-H. Lu, S.-H. Wang, C.-Y. Hsiao, C.-Y. Kuan et al., “Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech,” in ICASSP 20242024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 136–12 140. [34] C.-y. Huang, W.-C. Chen, S.-w. Yang, A. T. Liu et al., “Dynamicsuperb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,” in The Thirteenth International Conference on Learning Representations, 2025. [35] Y.-C. Lin, T.-Q. Lin, C.-K. Yang, K.-H. Lu, W.-C. Chen, C.-Y. Kuan, and H.-y. Lee, “Listen and speak fairly: a study on semantic gender bias in speech integrated large language models,” in 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 439–446. [36] Y.-C. Lin, W.-C. Chen, and H.-y. Lee, “Spoken stereoset: on evaluating social bias toward speaker in speech large language models,” in 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 871–878. [37] Y. Tseng, L. Berry, Y.-T. Chen, I.-H. Chiu, H.-H. Lin, M. Liu, P. Peng, Y.-J. Shih, H.-Y. Wang, H. Wu et al., “Av-superb: A multitask evaluation benchmark for audio-visual representation models,” in ICASSP 2024-2024 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2024, pp. 6890–6894. [38] Y. Chen, X. Yue, X. Gao, C. Zhang, L. F. D’Haro, R. T. Tan, and H. Li, “Beyond single-audio: Advancing multi-audio processing in audio large language models,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 10 917–10 930. [39] P. He, Z. Wen, Y. Wang, Y. Wang, X. Liu, J. Huang, Z. Lei, Z. Gu, X. Jin, J. Yang et al., “Audiomarathon: A comprehensive benchmark for long-context audio understanding and efficiency in audio llms,” arXiv preprint arXiv:2510.07293, 2025. [40] K.-H. Lu, C.-Y. Kuan, and H.-y. Lee, “Speech-ifeval: Evaluating instruction-following and quantifying catastrophic forgetting in speechaware language models,” in Proc. Interspeech 2025, 2025, pp. 2078– 2082. [41] Y. Tseng, T. Parcollet, R. van Dalen, S. Zhang, and S. Bhattacharya, “Evaluation of llms in speech is often flawed: Test set contamination in large language models for speech recognition,” arXiv preprint arXiv:2505.22251, 2025. [42] S.-w. Yang, M. Tu, A. T. Liu, X. Qu, H.-y. Lee, L. Lu, Y. Wang, and Y. Wu, “Paras2s: Benchmarking and aligning spoken language models for paralinguistic-aware speech-to-speech interaction,” arXiv preprint arXiv:2511.08723, 2025. [43] W. Ren, Y.-C. Lin, W.-C. Huang, E. Cooper, R. E. Zezario, H.M. Wang, H.-y. Lee, and Y. Tsao, “Mos-bias: From hidden gender bias to gender-aware speech quality assessment,” arXiv preprint arXiv:2603.10723, 2026. [44] C.-Y. Kuan and H.-y. Lee, “Aqua-bench: Beyond finding answers to knowing when there are none in audio question answering,” in ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026, pp. 1–5. [45] C.-Y. Kuan, K.-W. Chang, and H.-y. Lee, “Aqascore: Evaluating semantic alignment in text-to-audio generation via audio question answering,” arXiv preprint arXiv:2601.14728, 2026. [46] W.-C. Chen, C.-y. Huang, and H.-y. Lee, “Causal tracing of audio-text fusion in large audio language models,” arXiv preprint arXiv:2603.13768, 2026. [47] J.-K. Dong, Y.-X. Lin, and H.-Y. Lee, “Membership inference attacks against large audio language models,” arXiv preprint arXiv:2603.28378, 2026. [48] L. Zhou, L. Yu, Y. Lyu, Y. Lin, Z. Zhao, J. Ao, Y. Zhang, B. Wang, and H. Li, “Echomind: An interrelated multi-level benchmark for evaluating empathetic speech language models,” arXiv preprint arXiv:2510.22758, 2025.

[49] T.-Q. Lin, W.-P. Huang, Y.-C. Lin, and H.-y. Lee, “How contrastive decoding enhances large audio language models?” arXiv preprint arXiv:2603.09232, 2026. [50] K.-W. Chang, Y.-C. Lin, H.-C. Chou, W. Ren, Y.-H. Huang, Y.-S. Tsai, C.-C. Chen, Y. Tsao, Y.-F. Liao, S. Narayanan et al., “Taigispeech: A low-resource real-world speech intent dataset and preliminary results with scalable data mining in-the-wild,” arXiv preprint arXiv:2603.21478, 2026. [51] Y.-C. Lin, Y. Hirota, S.-F. Huang, and H.-y. Lee, “Vibe: Voice-induced open-ended bias evaluation for large audio-language models via realworld speech,” arXiv preprint arXiv:2604.17248, 2026. [52] K.-W. Chang, E.-P. Hu, C.-Y. Kuan, W. Ren, W.-C. Chen, G.-T. Lin, Y. Tsao, S.-H. Sun, H.-y. Lee, and J. Glass, “Game-time: Evaluating temporal dynamics in spoken language models,” in ICASSP 20262026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026, pp. 1–5. [53] Y. Chen, W. Zhu, X. Chen, Z. Wang, X. Li, P. Qiu, H. Wang, X. Dong, Y. Xiong, A. Schneider et al., “Aha: Aligning large audio-language models for reasoning hallucinations via counterfactual hard negatives,” arXiv preprint arXiv:2512.24052, 2025. [54] F. Zhao, Y. Chen, W. Lu, D. Zhang, X. Yue, and J. Wei, “Halluaudio: A comprehensive benchmark for hallucination detection in large audiolanguage models,” arXiv preprint arXiv:2604.19300, 2026. [55] K. Li, C. Shen, Y. Liu, J. Han, K. Zheng, X. Zou, L. Z. Wang, S. Zhang, X. Du, H. Luo et al., “Audiotrust: Benchmarking the multifaceted trustworthiness of audio large language models,” arXiv preprint arXiv:2505.16211, 2025. [56] H. He, X. Du, R. Sun, Z. Dai, Y. Xiao, M. Yang, J. Zhou, X. Li, Z. Liu, Z. Liang et al., “Measuring audio’s impact on correctness: Audiocontribution-aware post-training of large audio language models,” arXiv preprint arXiv:2509.21060, 2025. [57] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” arXiv preprint arXiv:1610.02136, 2016. [58] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek, “Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift,” Advances in neural information processing systems, vol. 32, 2019. [59] S. Desai and G. Durrett, “Calibration of pre-trained transformers,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 295–302. [60] J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young et al., “Scaling language models: Methods, analysis & insights from training gopher,” arXiv preprint arXiv:2112.11446, 2021. [61] Z. Jiang, J. Araki, H. Ding, and G. Neubig, “How can we know when language models know? on the calibration of language models for question answering,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 962–977, 2021. [62] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson et al., “Language models (mostly) know what they know,” arXiv preprint arXiv:2207.05221, 2022. [63] S. J. Mielke, A. Szlam, E. Dinan, and Y.-L. Boureau, “Reducing conversational agents’ overconfidence through linguistic calibration,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 857–872, 2022. [64] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso et al., “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” Transactions on machine learning research, 2023. [65] L. Kuhn, Y. Gal, and S. Farquhar, “Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,” in The Eleventh International Conference on Learning Representations, 2023. [66] S. Farquhar, J. Kossen, L. Kuhn et al., “Detecting hallucinations in large language models using semantic entropy,” Nature, vol. 630, pp. 625–630, 2024. [67] A. Nikitin, J. Kossen, Y. Gal, and P. Marttinen, “Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities,” Advances in Neural Information Processing Systems, vol. 37, pp. 8901–8929, 2024.

[68] D. Nguyen, A. Payani, and B. Mirzasoleiman, “Beyond semantic entropy: Boosting llm uncertainty quantification with pairwise semantic similarity,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 4530–4540. [69] X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” in The Eleventh International Conference on Learning Representations, 2023. [70] C. Zhu, B. Xu, Q. Wang, Y. Zhang, and Z. Mao, “On the calibration of large language models and alignment,” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 9778–9795. [71] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in International conference on machine learning. PMLR, 2017, pp. 1321–1330. [72] S. Lin, J. Hilton, and O. Evans, “Teaching models to express their uncertainty in words,” arXiv preprint arXiv:2205.14334, 2022. [73] Z. Yin, Q. Sun, Q. Guo, J. Wu, X. Qiu, and X.-J. Huang, “Do large language models know what they don’t know?” in Findings of the association for Computational Linguistics: ACL 2023, 2023, pp. 8653– 8665. [74] T. Chen, X. Liu, L. Da, J. Chen, V. Papalexakis, and H. Wei, “Uncertainty quantification of large language models through multidimensional responses,” arXiv preprint arXiv:2502.16820, 2025. [75] H.-Y. Huang, Y. Yang, Z. Zhang, S. Lee, and Y. Wu, “A survey of uncertainty estimation in llms: Theory meets practice,” arXiv preprint arXiv:2410.15326, 2024. [76] M. Hu, Z. Zhang, S. Zhao, M. Huang, and B. Wu, “Uncertainty in natural language processing: Sources, quantification, and applications,” arXiv preprint arXiv:2306.04459, 2023. [77] Z. Xia, J. Xu, Y. Zhang, and H. Liu, “A survey of uncertainty estimation methods on large language models,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 21 381–21 396. [78] L. Aichberger, K. Schweighofer, and S. Hochreiter, “Rethinking uncertainty estimation in natural language generation,” arXiv preprint arXiv:2412.15176, 2024. [79] M. Chen, G. Chen, W. Wang, and Y. Yang, “Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization,” arXiv preprint arXiv:2505.12346, 2025. [80] L. Chen, M. Zaharia, and J. Zou, “Frugalgpt: How to use large language models while reducing cost and improving performance,” arXiv preprint arXiv:2305.05176, 2023. [81] P. Aggarwal, A. Madaan, A. Anand, S. P. Potharaju, S. Mishra, P. Zhou, A. Gupta, D. Rajagopal, K. Kappaganthu, Y. Yang et al., “Automix: Automatically mixing language models,” arXiv preprint arXiv:2310.12963, 2023. [82] T. Zhang, A. Mehradfar, D. Dimitriadis, and S. Avestimehr, “Leveraging uncertainty estimation for efficient llm routing,” arXiv preprint arXiv:2502.11021, 2025. [83] Y. Moslem and J. D. Kelleher, “Dynamic model routing and cascading for efficient llm inference: A survey,” arXiv preprint arXiv:2603.04445, 2026. [84] Z. Ma, Z. Chen, Y. Wang, E. S. Chng, and X. Chen, “Audio-cot: Exploring chain-of-thought reasoning in large audio language model,” arXiv preprint arXiv:2501.07246, 2025. [85] Z. Xie, M. Lin, Z. Liu, P. Wu, S. Yan, and C. Miao, “Audio-reasoner: Improving reasoning capability in large audio language models,” arXiv preprint arXiv:2503.02318, 2025. [86] J. Fan, R. Ren, J. Li, R. Pandey, P. G. Shivakumar, I. Bulyko, A. Gandhe, G. Liu, and Y. Gu, “Incentivizing consistent, effective and scalable reasoning capability in audio llms via reasoning process rewards,” arXiv preprint arXiv:2510.20867, 2025. [87] F. Patel, D. Q. Nguyen, S. T. Truong, J. Vaynshtok, S. Koyejo, and N. Haber, “The sound of syntax: Finetuning and comprehensive evaluation of language models for speech pathology,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 34 895–34 913. [88] J. Zhang, N. Lin, L. Hou, L. Feng, and J. Li, “Adaptthink: Reasoning models can learn when to think,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 3716–3730. [89] C. Snell, J. Lee, K. Xu, and A. Kumar, “Scaling llm test-time compute optimally can be more effective than scaling model parameters,” arXiv preprint arXiv:2408.03314, 2024.

[90] A. Bilal, A. Mohsin, M. Umer, A. Subhan, H. Rizwan, A. Mohsin, and D. Hougen, “What if we allocate test-time compute adaptively?” arXiv preprint arXiv:2602.01070, 2026. [91] S. S. Ghosal, S. Chakraborty, A. Reddy, Y. Lu, M. Wang, D. Manocha, F. Huang, M. Ghavamzadeh, and A. S. Bedi, “Does thinking more always help? mirage of test-time scaling in reasoning models,” arXiv preprint arXiv:2506.04210, 2025. [92] S. Qu, “Adaptive test-time compute allocation via learned heuristics over categorical structure,” arXiv preprint arXiv:2602.03975, 2026. [93] M. Li, J. Zhong, S. Zhao, Y. Lai, H. Zhang, W. B. Zhu, and K. Zhang, “Think or not think: A study of explicit thinking in rule-based visual reinforcement fine-tuning,” arXiv preprint arXiv:2503.16188, 2025. [94] J. Geng, F. Cai, Y. Wang, H. Koeppl, P. Nakov, and I. Gurevych, “A survey of confidence estimation and calibration in large language models,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024, pp. 6577– 6595. [95] P. Manakul, A. Liusie, and M. Gales, “Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,” in Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 9004–9017. [96] N. Varshney, S. Mishra, and C. Baral, “Investigating selective prediction approaches across several tasks in IID, OOD, and adversarial settings,” in Findings of the Association for Computational Linguistics: ACL 2022. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 1995–2002. [Online]. Available: https://aclanthology. org/2022.findings-acl.158 [97] S. Feng, W. Shi, Y. Wang, W. Ding, V. Balachandran, and Y. Tsvetkov, “Don’t hallucinate, abstain: Identifying llm knowledge gaps via multillm collaboration,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 14 664–14 690. [98] B. Wen, J. Yao, S. Feng, C. Xu, Y. Tsvetkov, B. Howe, and L. L. Wang, “Know your limits: A survey of abstention in large language models,” Transactions of the Association for Computational Linguistics, vol. 13, pp. 529–556, 2025. [99] V. Kostumov, B. Nutfullin, O. Pilipenko, and E. Ilyushin, “Uncertaintyaware evaluation for vision-language models,” arXiv preprint arXiv:2402.14418, 2024. [100] C. Wang, T. Chen, H. Ahmad, K. Batmanghelich, and W. Li, “Vlmuqbench: A benchmark for modality-specific and cross-modality uncertainties in vision language models,” arXiv preprint arXiv:2602.09214, 2026. [101] S.-H. Yang, C.-K. Wu, C.-Y. Lin, Y.-N. Chen, H.-y. Lee, and S.-H. Sun, “On calibration of large language models: From response to capability,” arXiv preprint arXiv:2602.13540, 2026.

IX. G ENERATIVE AI U SE D ISCLOSURE AI-assisted tools were used solely to improve the clarity and fluency of the manuscript. All experiments, analyses, and related content were conducted and verified by the authors. X. ACKNOWLEDGMENTS This work was supported by the Ministry of Education (MOE) of Taiwan under the project Taiwan Centers of Excellence in Artificial Intelligence, through the NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE). XI. AUTHOR C ONTRIBUTIONS Chun-Yi Kuan led the project, conducted the experiments, developed the analytical framework, and wrote the initial draft. Wei-Ping Huang conceived the original idea, provided technical guidance on experimental design, and contributed to drafting and revising the manuscript. Hung-yi Lee advised on research direction and experimental design, and provided critical feedback on the manuscript. All authors reviewed and approved the final version.

TABLE V P ROMPT USED TO OBTAIN THE MODEL’ S FINAL PREDICTION ON THE BENCHMARK WITH LOW- TEMPERATURE DECODING . T HIS PREDICTION IS USED FOR CORRECTNESS EVALUATION . {QUESTION} IS REPLACED BY THE INPUT QUESTION . T HE NUMBER OF CHOICES DEPENDS ON EACH QUESTION .

Basic Inference Prompt System Prompt: You are an audio assistant. Based on the given audio, select the best answer to the user’s question from the provided choices. User Prompt: Question: {QUESTION} The answer could be: (a) {OPTION1} (b) {OPTION2} (c) {OPTION3} (d) {OPTION4}

XII. P ROMPT T EMPLATES For benchmark inference, we first use the basic inference prompt shown in Table V to obtain the model’s final prediction with low-temperature decoding. This prediction is used as the model’s final answer for correctness evaluation. In the prompt, {question} is replaced by the input question, and the number of choices varies across questions. For reasoningmode inference in our uncertainty-based adaptive inference experiments, we use the prompt templates shown in Table VI. For the self-verification P(True) baseline, we then ask the model to judge whether this generated answer is likely to be correct, given the input question and the generated answer. We use the prompt template shown in Table VII, following previous work [62]. TABLE VI P ROMPT USED FOR THE REASONING MODE IN THE UNCERTAINTY- BASED ADAPTIVE INFERENCE EXPERIMENTS . QUESTION IS REPLACED WITH THE INPUT QUESTION , AND THE NUMBER OF CHOICES VARIES DEPENDING ON THE QUESTION .

Inference Prompt for Reasoning Mode System Prompt: You are an audio reasoning assistant. First, describe the audio based only on audible evidence. Summarize the main sound events, speakers, acoustic scene, and any relevant temporal order. Do not use outside knowledge or guess beyond what can be heard. Then, answer the multiple-choice question using only the audio evidence. Choose exactly one option from the provided choices. Output in the following format: Audio Caption: <your caption> Reasoning: <brief reasoning based on the audio> Answer: <one option only> User Prompt: Question: {QUESTION} The answer could be: (a) {OPTION1} (b) {OPTION2} (c) {OPTION3} (d) {OPTION4}

TABLE VII V ERIFICATION PROMPT USED FOR SELF - VERIFICATION IN P(T RUE ), WHERE { Q U E S T I O N } AND { A N S W E R } ARE REPLACED WITH THE CORRESPONDING QUESTION AND ANSWER .

Verification Prompt System Prompt: You are a careful audio verification assistant. Listen to the provided audio and judge whether the answer to the question is directly supported by audible evidence in the audio. Do not rely on plausibility, common sense, or outside knowledge. Respond with only one word: true or false. User Prompt: Question: {QUESTION} Proposed Answer: {ANSWER} Is the proposed answer true or false based on the audio? Answer with only one word: true or false. Answer: TABLE VIII S ELF - ASSESSMENT PROMPT USED IN P(T RUE ) TO ESTIMATE WHETHER THE MODEL CAN ANSWER THE INPUT QUESTION CORRECTLY, WHERE { Q U E S T I O N } IS REPLACED WITH THE INPUT QUESTION .

Self-Assessment Prompt System Prompt: You are an assistant that evaluates its own ability to answer questions. Given a question, decide whether you are confident that you can answer the question correctly. User Prompt: Question: {QUESTION} Are you able to answer the question correctly? Answer with only a single word: yes or no. Answer:

XIII. C APABILITY C ALIBRATION In this work, we primarily focus on response calibration, which evaluates whether a model’s confidence aligns with the correctness of its generated answer. A related but distinct problem is capability calibration [62], [101]: whether a model can anticipate, prior to generating an answer, whether it will respond correctly. This setting is particularly relevant for decision-making scenarios such as selective prediction and abstention, where the model must decide whether to attempt an answer at all. Since capability calibration requires a different evaluation protocol and is not the main focus of this work, we do not include it in the main experiments. Instead, we provide a preliminary analysis of the Qwen2.5-Omni 7B and 3B variants in the following subsections.

let p(y | x) denote the model’s predictive distribution over outputs. The expected correctness is defined as: Ey∼p(y|x) [1(correct(y))] , which represents the probability that a randomly sampled response from the model is correct. Let c(x) ∈ [0, 1] denote the model’s predicted confidence of answering correctly. Capability calibration then evaluates how well c(x) aligns with the expected correctness: c(x) ≈ Ey∼p(y|x) [1(correct(y))] . B. Estimation In practice, the expected correctness is intractable to compute exactly, since the full predictive distribution p(y | x) is not directly accessible. We therefore approximate it using Monte Carlo sampling. For each input x, we draw K = 100 stochastic samples {yk }K k=1 from p(y | x) using hightemperature decoding, and compute K

â(x) =

k=1

Here, â(x) is the empirical accuracy over sampled responses, which serves as a Monte Carlo estimate of the expected correctness: Ey∼p(y|x) [1(correct(y))] . As such, â(x) provides a soft target in [0, 1] for evaluating capability calibration. To estimate the model’s predicted confidence c(x), we adopt a method similar to P(True). Specifically, we prompt the model to assess whether it would answer the question correctly, and compute the confidence score from the normalized likelihood of binary outputs (true/false). In previous work [62], this is referred to as P(IK), representing the probability that the model assigns to “I know (IK)”. The prompt template used to estimate the model’s confidence in answering the question correctly is shown in Table VIII. C. Evaluation We evaluate capability calibration by measuring the alignment between the predicted confidence c(x) and the estimated expected correctness â(x) using Expected Calibration Error (ECE) and Brier score. For ECE, we adopt equal-mass binning. Specifically, we first sort all inputs by predicted confidence and partition them into M bins with approximately equal numbers of samples. Let Bm denote the set of inputs in the m-th bin. ECE is then computed as ECE =

M X |Bm | m=1

A. Formulation Following prior work on capability calibration [101], we define capability calibration as the alignment between a model’s predicted confidence and its expected correctness under the predictive distribution. Formally, given an input x consisting of an audio clip and a corresponding instruction or question,

1 X 1(correct(yk )). K

1 X 1 X c(x) − â(x) . |Bm | |Bm | x∈Bm

x∈Bm

We also report the Brier score, defined as 2 1 X Brier = c(x) − â(x) . N x To visualize calibration quality, we plot equal-mass reliability diagrams, where predictions are sorted by confidence

TABLE IX C APABILITY CALIBRATION RESULTS OF Q WEN 2.5-O MNI MODELS (7B AND 3B VARIANTS ) ACROSS BENCHMARKS . L OWER IS BETTER FOR BOTH E XPECTED C ALIBRATION E RROR (ECE) AND B RIER SCORE (B RIER ). T HE BEST ECE IN EACH ROW IS HIGHLIGHTED WITH A GREEN BACKGROUND , WHILE THE BEST B RIER SCORE IS HIGHLIGHTED WITH AN ORANGE BACKGROUND .

Benchmark