Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition Iosif Tsangko∗† , Andreas Triantafyllopoulos∗† , Björn W. Schuller∗†‡§ ∗ CHI – Chair of Health Informatics, TUM University Hospital, Munich, Germany † MCML – Munich Center for Machine Learning, Munich, Germany ‡ MDSI – Munich Data Science Institute, Munich, Germany
arXiv:2606.07309v1 [cs.SD] 5 Jun 2026
§ GLAM – Group on Language, Audio, & Music, Imperial College London, UK
discrete label, optionally accompanied by a natural language justification. Before the advent of deep learning, classic SER research relied on interpretable, handcrafted acoustic descriptors [13], [14]. Given that these features can be easily derived from the input audio and are linked to emotional expressions through decades of research, we hypothesise that they may also provide useful auxiliary cues for ALM-based SER. However, improved performance alone does not show whether an ALM uses such cues in an audio-grounded way. A model could benefit from aligned cues, ignore them entirely, or over-rely on them even when they contradict the speech signal. We therefore operationalise reasonable cue use behaviourally: an ALM should benefit from aligned acoustic concept tokens, but should remain partly anchored to the audio when those tokens are shuffled, corrupted, or contradictory. Despite increasing interest in ALMs for interpretable speech and audio tasks, this form of token-only intervention analysis remains underexplored for SER. In this work, we first introduce categorical concept tokens I. I NTRODUCTION by binning interpretable acoustic descriptors extracted from Instruction-following audio language models (ALMs) couple the target utterance. To do so, we build on the eGeMAPS an audio front-end with an autoregressive language model, feature set [13] by summarising features into six prosodic and enabling prompt-based inference and natural language ratio- voice-related categories (energy, pitch, dynamics, brightness, nales for a wide range of speech and audio tasks [1], [2], [3]. formants, and voice quality). We convert the features into However, the extent to which ALMs use prompt-provided cues, a prompt-friendly, text-based format by binning them into a rather than merely describing them post hoc, remains unclear discrete set of categories. These binned concept tokens are then [4], [5]. This is particularly important for speech emotion appended to the ALM prompt and function as auxiliary cues. recognition (SER), especially given how emerging regulatory We then evaluate audio-grounded cue use by perturbing these frameworks increasingly consider it a “high-risk” application concept tokens while holding the audio input fixed. Concretely, requiring a high degree of interpretability for decisions of we test aligned tokens, shuffled tokens, deliberately conflicting automatic systems [6]. tokens, and graded token corruption. The extent to which SER is a core topic in computational paralinguistics, where ALM predictions respond to these token-only interventions models infer affective states from vocal cues such as prosody, allows us to estimate whether the models use the symbolic spectral shape, and voice quality [7]. Over the last years, the cue channel, ignore it, or over-rely on it. Across different field has moved from task specific pipelines towards foundation datasets and ALMs, we observe that concept tokens consistently style representations, including self supervised speech encoders improve performance when aligned with the audio, while [8], [9], [10] and cross modal audio text pretraining [11], [12]. stronger corruption leads to predictable degradation without With modern ALMs, this shift also enables prompt-based SER: fully overriding the acoustic signal. one can prompt a model with an utterance and request a The remainder of this paper is organised as follows: Section 2 reviews related work; Section 3 describes the datasets, modThis work was partially funded from the DFG’s Reinhart Koselleck project No. 442218748 (AUDI0NOMOUS) and the EU H2020 project No. 101135556 els, concept-token construction, and token-only interventions; (INDUX-R). Section 4 presents results; and Section 5 concludes. Abstract—Instruction-following audio language models (ALMs) can be augmented with explicit acoustic cues, yet it remains unclear whether such cues are used in a grounded way when the raw audio is already available. We study this question in speech emotion recognition (SER) by deriving six interpretable acoustic concept tokens from the standardised eGeMAPS paralinguistic feature set. These tokens summarise energy, pitch, dynamics, brightness, formants, and voice quality, and are appended to the textual prompt while the audio input is kept unchanged. Across the widely used FAU-Aibo and IEMOCAP benchmarks, aligned tokens improve unweighted average recall (UAR), whereas shuffled, conflicting, or corrupted tokens reduce performance relative to aligned tokens and shift confusions toward neutral. Importantly, predictions do not collapse under strong token perturbations, suggesting that the models are sensitive to the symbolic cue channel but remain partly anchored to the audio signal. We argue that token-only interventions provide a practical way to probe audio-grounded cue use, robustness, and interpretability in ALM-based affective computing. Index Terms—audio language models, speech emotion recognition, acoustic concept tokens, intervention analysis, computational paralinguistics
TABLE I C ONCEPT TOKENS DERIVED FROM eGeMAPS MEAN DESCRIPTORS . W E USE µ(·) TO DENOTE THE FILE - LEVEL MEAN OF THE CORRESPONDING eGeMAPS DIMENSION . Concept
eGeMAPS mean descriptor(s) (compact)
ENERGY
µ(Loudness)
Interpretation / tokenisation
Overall intensity proxy (loudness). Discretised into 5 quantile bins: VERY_LOW–VERY_HIGH. PITCH µ(F 0) Fundamental frequency (F0) in semitone scale. Discretised into 5 quantile bins; on IEMOCAP, bin edges are computed separately per gender. BRIGHTNESS µ(AR), µ(HI), µ(Slope0Hz−500Hz ), µ(Slope500Hz−1500Hz ) Spectral tilt / spectral energy distribution proxies (brighter vs darker timbre). Discretised into 5 quantile bins: VERY_DARK–VERY_BRIGHT. DYNAMICS µ(F lux) Spectral change rate over time (more/less dynamic signal). Discretised into 5 quantile bins: VERY_STATIC– VERY_DYNAMIC. FORMANTS µ(F 1f ), µ(F 2f ), µ(F 3f ) Resonant structure of the vocal tract (formant frequencies). Discretised into 5 quantile bins; on IEMOCAP, bin edges are computed separately per gender. VOICE_QUALITY µ(Jitter), µ(Shimmer), µ(HN R), µ(H1−H2), µ(H1−A3) (and µ(F 0) as auxiliary Heuristic categorical label from perturcue) bation/noise and harmonic-structure cues: MODAL/BREATHY/PRESSED/ROUGH/CREAKY (optionally UNVOICED).
II. R ELATED W ORK
auxiliary cues. Generative ALMs instead couple an audio encoder with an instruction-following autoregressive LLM, Recent ALM evaluation is increasingly benchmark-driven, enabling audio-conditioned text generation and structured with broad task coverage and unified scoring protocols proposed outputs. Representative open-source models include Qwen2in AudioBench [15], AIR-Bench [16], and VocalBench [17]. Audio [27], Qwen2.5-Omni [28], and Audio Flamingo 3 [2]. Beyond aggregate benchmark scores, recent work focuses on Very recent work has tried to explicitly target multi-step reliability issues, such as hallucinations and errors on detecting audio reasoning (e.g., Audio-Reasoner and ECHO), motivating individual events [18], with mitigation strategies based on syn- evaluation protocols that go beyond aggregate accuracy and thesising negative samples [19]. Reasoning-oriented assessment probe how models use intermediate information [29], [30]. for ALMs has also been formalised via multi-hop reasoning Finally, although explainability for SER has advanced (often benchmarks [20] and temporal reasoning with confidence via feature attribution and post-hoc analyses), such explanations estimation via perturbation-invariance [21], aligning with our do not by themselves establish faithful use of provided cues, intervention-based consistency analysis. Finally, evidence that motivating our intervention-based tests [31], [32]. (largely) frozen LLM backbones can perceive paralinguistic Faithfulness and intervention based evaluation of exattributes through learnt speech interfaces [22] supports our use planations. A broad interpretability literature has shown that of concept/paralinguistic cues for improving the performance explanations can be plausible without being faithful to the and reliability of ALMs. underlying decision process [4], [5]. Common evaluation Audio foundation models for SER and computational strategies include perturbation-based tests, in which the basic paralinguistics. Computational paralinguistics links speech to assumption is that removing or altering purported cues should speaker states and traits, with SER as one of its central tasks change the prediction [33], [34], [35]. These ideas have [7]. Earlier SER pipelines relied on expert-designed acoustic recently been revisited in the context of large models, where descriptors, typically extracted with toolkits such as openSMILE internal interventions can reveal causal structure beyond post [14]. Widely used feature sets include eGeMAPS [13] (among hoc rationales [36]. In the audio-language setting, neuron others), which provide an interpretable interface for affective level studies have provided complementary evidence that analysis and benchmarking [23], [24]. More recently, selftargeted interventions can selectively modulate emotion-related supervised speech encoders (wav2vec 2.0, HuBERT, WavLM) behaviour [37]. Our work follows the intervention principle, have become standard backbones for downstream paralinguistic but focuses on explicit auxiliary cues that are visible to the prediction [8], [9], [10], [24]. In parallel, contrastive audiolanguage model at inference time, enabling controlled tests text pretraining (e.g., CLAP) aligns audio and text in a without modifying model weights or internal activations. shared embedding space, enabling zero-shot matching [11]. Audio-grounded cue use in audio language models. For computational paralinguistics in particular, ParaCLAP leverages feature-derived prompting to form text queries, with ALMs support flexible prompting, facilitating the inclusion of SmoothCLAP additionally introducing soft-target supervision to auxiliary context and allowing for open-ended outputs in natural better reflect graded affective similarity [25], [26]. While these language. This makes them attractive for speech tasks, where contrastive encoders provide strong retrieval-style baselines, one may ask for a discrete label, a rationale, or a structured they are not designed to consume structured, per-utterance report [38], [39]. However, the extent to which ALMs use
ENERGYVERY_LOW
LOW
MID
VERY_HIGH
HIGH
emotion
ang 46%
ang hap neu sad
sad 55%
PITCHVERY_LOW
LOW
MID
HIGH
VERY_HIGH
UNVOICED
BRIGHTNESS VERY_DARK
DARK
NEUTRAL
BRIGHT
VERY_BRIGHT
DYNAMICS VERY_STATIC
STATIC
MODERATE
DYNAMIC
VERY_DYNAMIC
LOW
MID
HIGH
VERY_HIGH
VOICE_QUALITY MODAL PRESSED
BREATHY
ROUGH
ang 42%
ang 42%
UNVOICED
Fig. 1. IEMOCAP (4-way) concept-token distributions across emotions. For each prosodic concept (row), bubbles show the within-emotion frequency (%) of each discrete token value; bubble area is proportional to frequency. Small jitter is applied only for visual separation.
the additional information provided through the prompt, rather than relying primarily on the audio representation, has not been widely quantified. Recent work on audio reasoning highlights that reasoning should not be assumed from fluent outputs: [40] shows that prompting audio LLMs to produce longer chain-ofthought can reduce accuracy and even collapse under test-time scaling, motivating behavioural probes that test how predictions respond to controlled changes in available information. We address this gap by supplying interpretable concept tokens grounded in established paralinguistic descriptors and by evaluating token use through controlled, token only interventions. III. M ETHODOLOGY Datasets. We evaluate on two established speech emotion recognition benchmarks that differ markedly in language, speaker demographics, and recording conditions. FAU-Aibo (AIBO5). The FAU-Aibo Emotion Corpus is a German dataset of spontaneous children’s speech (ages 6–
TABLE II AIBO5 UAR WITHOUT CONCEPT TOKENS (U AR− ) AND WITH APPENDED CONCEPT TOKENS (U AR+ ). ∆ = U AR+ − U AR− .
dev
Model U AR
−
U AR
test +
∆
U AR
−
U AR+
∆
Audio language models AF3 .754 qw-omni .541
.776 .582
+.022 +.041
Zero-shot contrastive baselines (no tokens) CLAP .353 Pengi .345 ParaCLAP .600 SmoothCLAP .606
ang 49%
CREAKY
U AR−
Model
sad 52%
FORMANTS VERY_LOW
TABLE III IEMOCAP ( FULL ) UAR WITH / WITHOUT APPENDED CONCEPT TOKENS . C ONTRASTIVE ZERO - SHOT REFERENCES ARE REPRODUCED FROM [26].
U AR+
∆
Zero-shot ALMs qw2-audio .209 qw-omni .262 AF3 .259
.240 .279 .268
+.031 +.017 +.009
.206 .230 .253
.234 .240 .269
+.028 +.010 +.016
Fine-tuned ALMs ft-qw-omni .260 ft-AF3 .248
.263 .268
+.003 +.020
– –
– –
– –
10 years) recorded in a naturalistic human–robot interaction setting [41]. It also served as the dataset for the INTERSPEECH 2009 Emotion Challenge [23]. Following common practice, we use the standard mapping from the original fine-grained annotations to five classes (angry, emphatic, neutral, positive, rest). We report results on the official development split for broad model sweeps and rapid prompt/token ablations, and reserve the official test split for the final comparison of the strongest ALM backbones. IEMOCAP (4-way). IEMOCAP is an English corpus of acted dyadic interactions with scripted and improvised speech [42]. To align with prior work and to ensure sufficient support per class, we adopt the widely used 4-way setup (angry, happy+excited, neutral, sad), where excited is merged into happy and the remaining categories are discarded [43]. This yields 5,531 utterances in total. Unless otherwise stated, we use the standard session-based protocol (Sessions 1–4 for training and Session 5 for testing) to avoid speaker leakage. Figure 1 visualises the per-class distributions of the mapped concept tokens for this dataset. We also exclude other standard datasets, such as MSP-Podcast, from our evaluation as they are part of the training data for some of the ALMs considered, which would confound generalisation. Evaluation protocol. All datasets are evaluated using UAR to account for class imbalance. We report U AR− for the audioonly prompt (no appended concept tokens) and U AR+ for the audio+concept-tokens prompt; ∆ = U AR+ − U AR− . On AIBO5, we first run a multi-model sweep on the development set to characterise ALM behaviour under prompting and token augmentation. We then report test performance for the base backbones (qw-omni and AF3). On IEMOCAP, we report UAR for both qw-omni and AF3, and conduct extended intervention analyses on AF3, the most reliable backbone, to probe information faithfulness under controlled perturbations of the appended concept tokens. For all IEMOCAP comparisons, we report 95% confidence intervals from a speaker-clustered bootstrap (10 speakers, 104 resamples) and test paired differences with an exact McNemar test on per-utterance correctness. Concept Tokens from eGeMAPS. We map continuous eGeMAPS descriptors into a set of categorical concept tokens that can be appended to ALM prompts as structured auxiliary
Reasoning interventions: confusion patterns (rows=GT, cols=Pred) A
A
H
H
N
N
S
S A
H
N
S
Conflict (UAR=0.758) H
H
N
N
S
S H
N
S
correct concepts
0.770 0.765 0.760 0.755
H
N
S
Corrupt p=1.0 avg (UAR=0.759) A
0.780 0.775
A
A
A
Evidence Intervention Curve (token corruption)
Shuffle avg (UAR=0.760) UAR
Correct (UAR=0.776)
0.2
0.3
0.4
0.5
0.6
0.7
Token corruption probability p
0.8
0.9
1.0
Fig. 3. Token corruption curve on AF3: UAR as a function of token corruption probability p (mean ± 95% CI across seeds). Performance degrades monotonically as token quality decreases (Spearman ρ = −.90, p < 10−4 across the 5 × 5 p/seed runs), with the correct-tokens baseline shown as a reference.
A
H
N
S
Fig. 2. Row-normalised confusion matrices (rows=GT, cols=Pred) under tokens’ interventions on AF3. Rows sum to 1; cells with value < 0.01 are not annotated. Correct (aligned tokens) yields the best UAR, while shuffle, conflict, and full corruption reduce performance and shift errors (notably toward N).
cues. The grouping follows the standard semantic definitions of the eGeMAPS descriptors (as shown in Table I). For each utterance, eGeMAPSv02 LLDs are extracted with openSMILE and summarised with a file-level mean vector. We then apply robust standardisation and compute the scalar concept scores ENERGY, PITCH, BRIGHTNESS, DYNAMICS, and FORMANTS by averaging the corresponding standardised eGeMAPS dimensions listed in Table I. We discretise each scalar score into five quantile bins using dataset-level bin edges that are fixed across samples. On IEMOCAP, PITCH and FORMANTS use gender-specific bin edges, because adult male and female pitch and formant distributions differ substantially. Because FAU-Aibo contains pre-puberty children, we do not apply gender-specific binning on this dataset. We additionally derive a categorical VOICE_QUALITY token from jitter/shimmer/HNR and harmonic-relations cues ( Table I). Finally, the six categorical labels are serialised into an ALM-friendly token string. Audio Language Models. We evaluate instruction-following, autoregressive ALMs that condition an LLM on audio embeddings produced by an audio encoder. We include three opensource ALM families: Qwen2-Audio [27], Qwen2.5-Omni [1], and Audio Flamingo 3 (AF3) [2], plus two task-adapted (finetuned) variants of Omni 1 and AF3 2 . To isolate the effect of explicit concept tokens, we use two prompt formats: audioonly and audio+concept tokens, where six categorical tokens are appended as structured auxiliary cues. We additionally report audio–text pretraining baselines (CLAP, ParaCLAP, 1 https://huggingface.co/mispeech/midashenglm-7b-0804-fp32 2 https://huggingface.co/nvidia/audio-flamingo-3# think-mode-reasoning-with-peft-adapter-af-think
SmoothCLAP) on IEMOCAP as non-generative reference points [11], [25], [26]. Pengi is an autoregressive model [44]; however, since it is not designed for instruction following, prior work adapts it to zero-shot SER by mapping its generated text to the closest emotion label [26]. Hence, we treat Pengi as a reference baseline and do not include it in our interventionbased tests. Token Interventions. We continue with AF3 since it achieves higher token-augmented performance on both datasets: on AIBO5 test U AR+ = .269 vs .240 (qw-omni), and on IEMOCAP U AR+ = .776 vs .582, making it the strongest overall backbone for follow-up analyses. Next, we test whether concept tokens measurably influence predictions by applying token-only interventions that modify token alignment and quality. Let xi denote an utterance with ground-truth emotion yi , and let ci be its associated concept-token sequence (derived from eGeMAPS). For each utterance, we construct perturbed token sequences c̃i and run the same model/prompt template, measuring changes in UAR and confusion patterns. Correct (aligned concept tokens). We append the original tokens for each utterance, i.e., c̃i = ci . This condition represents the intended, token-aligned setting. Shuffle (misaligned concept tokens, distribution preserved). We randomly reassign token sequences across utterances within the evaluation split: c̃i = cπ(i) for a random permutation π. This preserves the marginal distribution of tokens but breaks the sample-level alignment between audio and concept tokens. We repeat over multiple seeds and report the mean (and confidence intervals where applicable). Conflict (contradictory concept tokens). We construct tokens that systematically contradict the original ones by inverting the ordinal bins for each acoustic feature (e.g., VERY_LOW to VERY_HIGH, LOW to HIGH, MID to MID). Thus, we replace each concept by its opposite category and append it as c̃i . This differs from Shuffle, which preserves marginal distributions across the split but breaks sample-level alignment without enforcing contradiction. Corrupt(p) (graded token corruption). To probe sensitivity to the auxiliary cues, we corrupt concept tokens with probability
p ∈ [0, 1]. For each token position, with probability 1 − p we and (ii) robust fallback: predictions do not collapse under keep the original token and with probability p we replace it misleading cues. Even contradictory tokens stay marginally with a randomly sampled token from the same token vocabulary. above the no-token baseline (p < .05), indicating that the model Varying p yields an corruption curve: if the model relies on integrates the symbolic cue channel with the audio rather the concept tokens, performance should degrade monotonically than simply following it, while errors shift toward Neutral. as p increases. This is consistent with reliance on the audio signal and other Across all interventions, the audio input is held constant; only information pathways available to ALMs, such as their strong the appended tokens are modified. This isolates the contribution speech transcription capability and use of linguistic markers, rather than purely acoustic cueing. of structured cues to the model’s decision process. IV. R ESULTS
R EFERENCES
Tables II and III report UAR without appended concept tokens (U AR− ) and with concept tokens (U AR+ ). On AIBO5 (dev), token augmentation causes consistent gains across all evaluated ALMs; qw-omni attains the strongest overall dev performance (U AR+ = .279), and AF3 improves reliably across splits. On AIBO5 (test) and IEMOCAP (4-way), concept tokens again improve UAR for both qw-omni and AF3, with AF3 achieving the best overall IEMOCAP result (U AR+ = .776); on IEMOCAP, both token gains are significant (p < 10−5 , with bootstrap CIs on ∆ excluding zero). We next analyse token sensitivity on AF3 under controlled, token-only interventions. Aligned tokens significantly outperform every perturbation, Conflict (∆UAR= +.018, 95% CI [.011, .025], p = 4 × 10−7 ), Shuffle, and full Corruption (both p < 10−4 ), and UAR degrades monotonically as corruption increases (Spearman ρ = −.90, p < 10−4 ; Figure 3). Figure 2 shows that perturbations modestly increase Neutral confusions relative to aligned tokens, consistent with a weakening of class-specific cues. Crucially, the perturbed conditions remain close to, and slightly above, the audio-only baseline: Conflict, Shuffle, and Corruption all outperform it marginally (Conflict .758 vs. .754, p = .014; Holm-corrected p < .05). This suggests a small format prior: the structured token block itself provides some benefit, independent of whether the token content is correct. However, this effect is much smaller than the gain from aligned tokens. The fact that contradictory tokens do not drive performance below the baseline indicates that AF3 does not simply follow the symbolic cue channel, but integrates it with the audio.
[1] J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang et al., “Qwen2.5-omni technical report,” arXiv preprint, 2025. [2] A. Goel, S. Ghosh, J. Kim, S. Kumar, Z. Kong, S. G. Lee, C.-H. H. Yang, R. Duraiswami, D. Manocha, R. Valle, and B. Catanzaro, “Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,” arXiv preprint, 2025. [3] A. Triantafyllopoulos, I. Tsangko, A. Gebhard, A. Mesaros, T. Virtanen, and B. W. Schuller, “Computer audition: From task-specific machine learning to foundation models,” Proceedings of the IEEE, vol. 113, no. 4, pp. 317–343, 2025. [4] A. Jacovi and Y. Goldberg, “Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?” in Proceedings of ACL, 2020, p. no pagination. [5] J. DeYoung, S. Jain, N. F. Rajani, E. Lehman, C. Xiong, and B. C. Wallace, “Eraser: A benchmark to evaluate rationalized NLP models,” in Proceedings of ACL, 2020, p. no pagination. [6] The European Parliament, “Amendments adopted by the European Parliament on 14 June 2023 on the proposal for a regulation of the European Parliament and of the Council on laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts (COM(2021)0206 – C90146/2021 – 2021/0106(COD)),” 2023, https://www.europarl.europa.eu/ doceo/document/TA-9-2023-0236_EN.html. [7] B. Schuller and A. Batliner, Computational Paralinguistics: Emotion, Affect and Personality in Speech and Language Processing. Wiley, 2014. [8] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems, 2020, pp. 12 449– 12 460. [9] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021. [10] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “Wavlm: Large-scale self-supervised pretraining for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022. [11] B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “Clap: Learning audio concepts from natural language supervision,” in Proceedings of ICASSP, Rhodes, Greece, 2023, pp. 1–5. [12] B. Schuller, A. Mallol-Ragolta, A. P. Almansa, I. Tsangko, M. M. Amin, A. Semertzidou, L. Christ, and S. Amiriparian, “Affective computing has changed: The foundation model disruption,” npj Artificial Intelligence, vol. 2, no. 1, p. 16, 2026. [13] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan et al., “The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,” IEEE Transactions on Affective Computing, vol. 7, no. 2, pp. 190–202, 2016. [14] F. Eyben, M. Wöllmer, and B. W. Schuller, “opensmile: The munich versatile and fast open-source audio feature extractor,” in Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy, 2010, pp. 1459–1462. [15] B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen, “Audiobench: A universal benchmark for audio large language models,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Albuquerque, New Mexico: Association for
V. C ONCLUSION Recent work on latent-space reasoning argues that constraining LLM “thinking” to language tokens can be suboptimal, and that more expressive reasoning can emerge when models operate directly in continuous representations rather than producing verbose token chains [45]. This perspective is particularly natural for SER: affective features are intrinsically continuous (prosody, voice quality, spectral shape), and modern ALMs already ingest speech as continuous embeddings via an audio encoder. Our results support this view. Holding audio fixed and perturbing only concept tokens isolates the contribution of the language channel. We observe two properties: (i) token sensitivity: aligned concept tokens improve SER, and progressively corrupting them yields a monotonic degradation;
Computational Linguistics, 2025, pp. 4297–4316. [Online]. Available: https://aclanthology.org/2025.naacl-long.218/ [16] Q. Yang, J. Xu, W. Liu, Y. Chu, Z. Jiang, X. Zhou, Y. Leng, Y. Lv, Z. Zhao, C. Zhou, and J. Zhou, “AIR-bench: Benchmarking large audio-language models via generative comprehension,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics, 2024, pp. 1979–1998. [Online]. Available: https://aclanthology.org/2024.acl-long.109/ [17] H. Liu, Y. Wang, Z. Cheng, R. Wu, Q. Gu, Y. Wang, and Y. Wang, “Vocalbench: Benchmarking the vocal conversational abilities for speech interaction models,” arXiv preprint arXiv:2505.15727, 2025. [Online]. Available: https://arxiv.org/abs/2505.15727 [18] C.-Y. Kuan, W.-P. Huang, and H. yi Lee, “Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models,” in Interspeech 2024, 2024, pp. 4144– 4148. [Online]. Available: https://www.isca-archive.org/interspeech_ 2024/kuan24_interspeech.html [19] C.-Y. Kuan and H. yi Lee, “Teaching audio-aware large language models what does not hear: Mitigating hallucinations through synthesized negative samples,” in Interspeech 2025, 2025, pp. 2073– 2077. [Online]. Available: https://www.isca-archive.org/interspeech_ 2025/kuan25_interspeech.html [20] C.-K. Yang, N. Ho, Y.-T. Piao, and H. yi Lee, “Sakura: On the multi-hop reasoning of large audio-language models based on speech and audio information,” in Interspeech 2025, 2025, pp. 1788– 1792. [Online]. Available: https://www.isca-archive.org/interspeech_ 2025/yang25g_interspeech.html [21] D. Bhattacharya, A. Kulkarni, and S. Ganapathy, “Benchmarking and confidence evaluation of lalms for temporal reasoning,” in Interspeech 2025, 2025, pp. 2068–2072. [Online]. Available: https://www.isca-archive. org/interspeech_2025/bhattacharya25b_interspeech.html [22] W. Kang, J. Jia, C. Wu, W. Zhou, E. Lakomkin, Y. Gaur, L. Sari, S. Kim, K. Li, J. Mahadeokar, and O. Kalinli, “Frozen large language models can perceive paralinguistic aspects of speech,” in Interspeech 2025, 2025, pp. 4323–4327. [Online]. Available: https: //www.isca-archive.org/interspeech_2025/kang25_interspeech.html [23] B. Schuller, S. Steidl, and A. Batliner, “The interspeech 2009 emotion challenge,” in Proceedings of INTERSPEECH, Brighton, United Kingdom, 2009, p. no pagination. [24] J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller, “Dawn of the transformer era in speech emotion recognition: Closing the valence gap,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 10 745– 10 759, 2023. [25] X. Jing, A. Triantafyllopoulos, and B. Schuller, “Paraclap: Towards a general language-audio model for computational paralinguistic tasks,” in Proceedings of INTERSPEECH, Kos Island, Greece, 2024, p. no pagination. [26] X. Jing, J. Wang, A. Triantafyllopoulos, M. Gerczuk, S. Amiriparian, J. Luo, and B. Schuller, “Smoothclap: Soft-target enhanced contrastive language-audio pretraining for affective computing,” arXiv preprint, 2026. [27] J. Xu, H. Zhou, J. Wu, J. Wang, P. Zhu, H. Zhang, S. Bai, J. Lin, Z. Yu, K. Dang et al., “Qwen2-audio: Improving audio understanding in large language models,” arXiv preprint, 2024. [Online]. Available: https://arxiv.org/abs/2407.10759 [28] J. Xu et al., “Qwen3-omni technical report,” arXiv preprint, 2025. [Online]. Available: https://arxiv.org/abs/2509.17765 [29] X. Chen, W. Hu, Z. Yu, L. Xu, Z. Wang, J. Wu, Y. Ding, M. Wu, S. Wen, Y. Zhang et al., “Audio-reasoner: Improving reasoning capability in large audio language models,” arXiv preprint, 2025, accepted by EMNLP 2025. [Online]. Available: https://arxiv.org/abs/2503.02318 [30] Y. Gao, Z. Yu, S. Geng, X. Han, Z. Wu, J. Li, S. Li, S. Cao, J. Wu, X. Hu, Z. Chen, J. Wang, K. Chen, F. Huang, and J. Zhou, “Echo: Towards advanced audio comprehension via audio-interleaved reasoning,” arXiv preprint, 2026, accepted by ICLR 2026. [Online]. Available: https://arxiv.org/abs/2602.11909 [31] A. Nfissi, W. Bouachir, N. Bouguila, and B. Mishara, “Unveiling hidden factors: explainable ai for feature boosting in speech emotion recognition: N. alaa et al.” Applied Intelligence, vol. 54, no. 11, pp. 7046–7069, 2024. [32] A. Akman, Q. Sun, and B. W. Schuller, “Improving audio explanations using audio language models,” IEEE Signal Processing Letters, vol. 32, pp. 741–745, 2025.
[33] S. Jain and B. C. Wallace, “Attention is not explanation,” in Proceedings of NAACL, 2019, p. no pagination. [34] J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim, “Sanity checks for saliency maps,” in Advances in Neural Information Processing Systems, 2018, p. no pagination. [35] A. Akman, Q. Sun, and B. W. Schuller, “Improving audio explanations using audio language models,” IEEE Signal Processing Letters, 2025. [36] L. Jain, P. Mousavi, M. Ravanelli, and C. Subakan, “Investigating faithfulness in large audio language models,” arXiv preprint arXiv:2509.22363, 2025. [37] X. Zhao, B. Schuller, and B. Sisman, “Discovering and causally validating emotion-sensitive neurons in large audio-language models,” arXiv preprint, 2026. [38] J. Peng, Y. Wang, B. Li, Y. Guo, H. Wang, Y. Fang, Y. Xi, H. Li, X. Li, K. Zhang et al., “A survey on speech large language models for understanding,” IEEE Journal of Selected Topics in Signal Processing, 2025. [39] Y. Su, J. Bai, Q. Xu, K. Xu, and Y. Dou, “Audio-language models for audio-centric tasks: A survey,” arXiv preprint arXiv:2501.15177, 2025. [40] J. Fan, R. Ren, J. Li, R. Pandey, P. G. Shivakumar, I. Bulyko, A. Gandhe, G. Liu, and Y. Gu, “Incentivizing consistent, effective and scalable reasoning capability in audio llms via reasoning process rewards,” arXiv preprint arXiv:2510.20867, 2025. [41] A. Batliner, S. Steidl, and E. Nöth, “Releasing a thoroughly annotated and processed spontaneous emotional database: The FAU aibo emotion corpus,” in Proceedings of LREC, 2008, p. no pagination. [42] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” Language Resources and Evaluation, vol. 42, no. 4, pp. 335–359, 2008. [43] N. Antoniou, A. Katsamanis, T. Giannakopoulos, and S. Narayanan, “Designing and evaluating speech emotion recognition systems: A reality check case study with iemocap,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. [44] S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,” in Advances in Neural Information Processing Systems, 2024, p. no pagination. [45] S. Hao, S. Sukhbaatar, D. Su, X. Li, Z. Hu, J. Weston, and Y. Tian, “Training large language models to reason in a continuous latent space,” arXiv preprint arXiv:2412.06769, 2024.