Interpretable Singing Emotion Recognition Using Acoustic and Vocal Technique Features

1

Interpretable Singing Emotion Recognition Using Acoustic and Vocal Technique Features

arXiv:2610.32629v1 [cs.SD] 23 Oct 2026

Fotini Kiriakopoulou, Vassilis Lyberatos, Spyridon Kantarelis, Athanasios Voulodimos and Giorgos Stamou

Abstract—Singing emotion recognition relies predominantly on deep learning models that achieve strong performance but offer limited insight into the vocal characteristics driving emotion, often overlooking the singing voice as a distinct signal. We propose an interpretable singing emotion recognition framework using a compact set of singing-specific descriptors across prosody, spectral features, phonetics, and vocal technique. To decode model decisions and link predictions to measurable acoustic phenomena, we employ SHAP and apply G RAD -CAM to qualitatively assess the learned representations of vocal techniques. Evaluations show that singing-specific features reduce cross-singer variability, while vocal technique descriptors provide interpretable insights into singing expression. Furthermore, out-of-domain evaluation on VocalSet reveals interesting associations between emotional expression and ground-truth technical execution. The analysis highlights vibrato and pharyngeal techniques as salient cues of emotional delivery, with SHAP indicating that vibrato is context-dependent on articulation rate. These findings emphasize the value of domain-specific features for interpretable singing emotion analysis. Index Terms—Singing Emotion Recognition, Interpretability, Vocal Technique, Explainable AI, Feature Engineering

I. I NTRODUCTION Singing emotion recognition (SiER) has relied on deep learning models, which learn complex acoustic representations directly from audio and achieve strong recognition performance [1], [2]. However, these representations are difficult to relate to specific characteristics of vocal expression, limiting the interpretability of the recognition process. Moreover, their reliance on controlled singing data may not reflect naturally occurring expressive variation. This is particularly relevant in music applications, where understanding the acoustic cues associated with an emotional prediction can be important for analyzing vocal expression, music recommendation, and performance analysis. Explainable AI therefore provides a means of relating model decisions to human-interpretable properties of the input signal [3]. In music emotion recognition, interpretable approaches commonly extract perceptually meaningful acoustic or musical features and use them with relatively shallow classifiers [4], [5], [6]. However, existing analyses of singing rarely incorporate singing-specific descriptors that distinguish singing from speech or instrumental sound. Fotini Kiriakopoulou, Vassilis Lyberatos, Spyridon Kantarelis, Athanasios Voulodimos, and Giorgos Stamou are with the National Technical University of Athens, Athens, Greece (e-mail: fotinikrp@hotmail.com). This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

Singing exhibits acoustic properties that distinguish it from speech, including a wider pitch range, sustained harmonic structure, and distinctive resonance characteristics [7]. Moreover, emotional expression in singing is closely related to vocal technique. For example, vocal resonance has been shown to influence emotional perception causally [8], while singers may use pitch modulation, such as vibrato, differently from speakers to convey emotional information [9]. Such characteristics provide interpretable descriptors of vocal expression that are specific to singing and are not necessarily captured by generic acoustic representations. Consequently, an interpretable SiER system should explicitly incorporate singing-specific features alongside more general acoustic descriptors. Some studies have pursued interpretable, feature-based SiER. Specifically, [10] extract a set of openSMILE [11] descriptors from controlled opera recordings, spanning loudness, spectral flux and entropy, band energies, alpha ratio, Hammarberg index, and others, ranking them by importance and evaluating an SVM classifier under a leave-one-singer-out protocol. Another study extracts timbre-related spectral descriptors from RAVDESS singing samples and uses SHAP with an XGB OOST classifier to derive feature contributions [12]. Our analysis corroborates several of their selected descriptors, yet neither study explicitly takes into account vocal technique. In this work, we propose an interpretable framework for emotion recognition in singing based on explicitly measurable musical and singing-specific acoustic features (Figure 1). The framework extracts interpretable descriptors of the singing signal and uses them for emotion classification, enabling predictions to be examined in terms of identifiable characteristics of vocal expression rather than solely learned latent representations. In particular, we investigate whether incorporating domain-specific features can provide an effective and interpretable representation for SiER. Evaluations across diverse singing datasets indicate that the proposed features improve cross-singer stability, and reveal associations between emotional expression and vocal execution, with technique descriptors serving as an interpretable vocabulary for explaining expressivity in singing. II. M ETHODOLOGY Our proposed framework consists of three main stages. First, we extract a broad set of descriptors correlated with the singing voice, spanning four perceptual dimensions. Secondly, we apply a feature selection process to obtain a compact feature set that supports interpretability without sacrificing predictive power. Finally, we employ explainable

2

Fig. 1. Overview of the proposed framework.

AI techniques to interpret the model predictions and assess the learned vocal technique representations.

A. Feature extraction We extract 130 scalar features from audio resampled to 22.05 kHz, organized into four categories: prosodic, spectral, phonetic, and vocal technique features. For corpora lacking phoneme or technique annotations, these are estimated with the Montreal Forced Aligner (MFA) [13] and a dualinput CNN (Section III-B), respectively. All features are z-score normalized per singer, which reduces singer-specific variability while preserving within-singer differences and improves cross-corpus robustness [14]. Prosodic features describe pitch, intonation, and intensity, which shape perceived expressiveness in singing [15]. Using the OPEN SMILE E G E MAPS configuration [16], LI BROSA [17] and CREPE [18], we compute features such as f0 statistics, pitch-change rate (∆f0 ), jitter and shimmer. Spectral features characterize timbre, which varies with vocal effort and singing technique [7]. Using OPEN SMILE and LIBROSA, we compute the alpha ratio, formant frequencies, spectral flux and zero-crossing-rate statistics. Phonetic features, derived from phoneme boundaries, describe articulation and timing, and include the mean and standard deviation of phoneme and pause durations and the articulation rate. Vocal technique features capture expressive devices such as vibrato [19]. From the CNN’s frame-level probabilities across six techniques (vibrato, glissando, mix, pharyngeal, falsetto, and breathy), we compute four statistics (mean, standard deviation, maximum, and peak factor). We also compute overall technique entropy by aggregating the presence (ŷt,c ) of each technique c ∈ C across all T frames and normalizing by the overall technique sum: T X Sc (1) Sc = ŷt,c , pc = P k∈C Sk t=1 The recording-level entropy is then calculated via Shannon entropy: X H=− pc ln(pc ) (2) c∈C

and quantifies the diversity of vocal technique presence. The complete feature list is available in our GitHub repository 1 . 1 https://github.com/FotiniKrp/Interpretable Singing Emotion Recognition

B. Feature selection and model interpretability Feature selection uses a nested Leave-One-Singer-Out (LOSO) framework to identify features that are consistently useful across singers, rather than ones that happen to fit a particular singer split. In the inner loop specifically, treebased classifiers compute feature importance scores, and a frequency counter tracks how often each descriptor ranks among the top-N candidates across inner splits, retaining only those with high cross-singer stability. Finally, across all outer singer iterations, we aggregate fold selections to retain the most stable descriptors. We apply this procedure both per feature category (N = 10) and to the complete feature set (N = 40). To reduce redundancy and improve interpretability, highly correlated features were pruned using pairwise Pearson correlation analysis (threshold = 0.9). The resulting feature set is then used to train a classifier for the final emotion prediction. This compact set supports a clearer interpretation of model predictions, in particular with the help of SHAP [20] analysis to identify which features contribute to the classifier’s predictions. We also employ G RAD -CAM [21] to examine whether the vocal technique extractor relies on plausible acoustic cues rather than spurious patterns. III. E XPERIMENTAL S ETUP A. Dataset selection To investigate singing emotion across diverse conditions and styles, we employ three datasets. As a baseline, we use RAVDESS [22], specifically its singing subset, comprising 1,012 English clips from 23 actors portraying six discrete emotions through acted expression. To complement these recordings with more realistic and diverse singing conditions, we additionally employ GTSinger [23]. For emotion classification, we selected the English, French, and German subsets, which are Western European languages, allowing a controlled evaluation. This subset encompasses modern pop and bel canto singing from seven vocalists (3 tenor, 2 alto, 2 soprano) and contains 1,228 performances (segmented into 10,656 clips), each labeled with a global emotion (happy or sad). To train the vocal technique extractor, we utilize the Chinese, English, Japanese, and French subsets, which offer a more balanced and diverse representation of vocal technique. Finally, we hold out its Italian subset to evaluate the CNN’s ability to generalize to an unseen language. Lastly, VocalSet [24] serves as an out-ofdomain evaluation set to validate and analyze the learned vocal technique representations. Specifically, we utilize the 2,553

3

vocalizations across 20 singers from the filtered AnnotatedVocalSet subset [25].

TABLE I F EATURE S ET P ERFORMANCE ON RAVDESS AND GTSinger. Dataset

Feature Set

#

UAR (%)

SD (%)

Macro-F1

RAVDESS

Speech-Opt. Singing-Opt.

19 18

76.2 79.6

9.5 6.5

0.75 0.79

GTSinger

Speech-Opt. Singing-Opt.

20 19

60.2 61.0

5.8 3.3

0.56 0.60

B. Models The vocal technique extractor is trained on 110,000 nonoverlapping segments from GTSinger, using a 1.5 s window to provide sufficient temporal context for techniques such as vibrato while remaining a practical training unit. We use an 85:15 train/val split and the Adam optimizer with learning rate 10−4 . To address the pronounced class imbalance, we employ Focal Loss with α = 0.25 and γ = 2.0 [26]: LFL (pt ) = −α(1 − pt )γ log(pt )

TABLE II A BLATION S TUDY ON THE S INGING -O PTIMIZED F EATURE S ET. Type

Feature Set

UAR (%)

SD (%)

Single

Only Techniques Only Prosody Only Phonetic Only Spectral

35.0 66.4 32.8 53.5

8.6 11.1 9.9 11.0

All-but-1

All exc. Techniques All exc. Prosody All exc. Spectral All exc. Phonetic

79.4 62.5 73.2 76.8

7.0 7.6 10.0 5.5

(3)

where pt is the sigmoid probability of the correct label for each technique. To further mitigate the effects of class imbalance, we employ early stopping on validation Macro-F1. Our CNN combines Mel-spectrogram and pitch contour representations through a dual-input architecture, inspired by prior work [27]. The two streams are integrated through late fusion followed by a sigmoid output layer for multi-label classification, as multiple techniques can co-occur. For the emotion classification we employ XGB OOST for RAVDESS (100 estimators, depth 6, learning rate 0.1, subsample/colsample = 0.8; feature-selection variant: depth 4, no stochastic sampling) and R ANDOM F OREST for GTSinger (200 estimators, depth 12, min. leaf samples 10, balanced class weights; feature-selection variant: 50 estimators, depth 6, default min. leaf samples). We evaluate the models using a LOSO protocol [14], where each singer is held out for testing while the remaining singers are used for training. Performance is measured using Unweighted Average Recall (UAR) and Macro-F1, and reported as the mean and standard deviation across all singer folds. Confidence intervals are estimated via singer-level bootstrapping (10,000 iterations). For GTSinger, clip predictions are aggregated using majority voting to obtain the performance prediction before computing evaluation metrics. IV. R ESULTS A. Performance On RAVDESS, we compare the predictive power of two compact subsets derived via our nested LOSO feature selection: a speech-optimized set, selected from the eGeMAPS set, and a singing-optimized set, selected from our proposed feature set. As shown in Table I, the singing-optimized set achieves a higher UAR (79.6% with 95% CI: 76.9–82.2%) than the speech-optimized set (76.2% with 95% CI: 72.2– 80.0%), while also exhibiting lower cross-singer variability (singing-to-speech SD ratio = 0.68 with 95% CI: 0.452–0.927). An ablation study (Table II) shows that prosody dominates baseline accuracy, spectral and phonetic features provide complementary cues, while technique descriptors slightly reduce cross-singer deviation. We replicate the feature selection procedure on GTSinger. As shown in Table I, while the speech-optimized feature set attains a comparable UAR to the singing-optimized set,

it exhibits higher cross-singer variance and a lower macroF1 score. Notably, the speech-optimized set shows a styledependent gap (63.0% bel canto vs. 58.2% pop, dropping to near-chance 52.6% on the most difficult pop singer), whereas the singing-optimized set remains comparatively stable across styles (60.9% vs. 61.2%), though these subgroups are limited in size (n = 3 bel canto, n = 4 pop). B. Interpretability analysis When evaluated on the held-out Italian subset of GTSinger, the vocal technique extractor achieved a Macro-F1 of 0.42 and a Macro-Recall of 0.56 across multi-label frame predictions. We employ G RAD -CAM to visualize the time–frequency regions contributing most strongly to the CNN’s predictions, as a qualitative check that the model attends to acoustically meaningful regions. Figure 2 shows representative VocalSet samples. For the vibrato example, the G RAD -CAM map highlights the peaks of the periodic frequency modulations associated with the vibrato pattern (the least-represented class during training). For the belt sample, the G RAD -CAM computed with respect to the pharyngeal class shows two distinct activation patterns: continuous horizontal bands in the midfrequency mel bins, consistent with pressed phonation [28], and a localized cluster of high- activation in the highest mel bins, which may instead reflect flow phonation, commonly associated with belting technique. We further evaluate the representations learned from RAVDESS by applying both the speech and singing-optimized models to VocalSet and examining the associations between predicted emotions and ground-truth vocal techniques. As shown in Figure 3, higharousal emotions such as anger are more strongly mapped to dynamic vocal techniques, including belt, fast articulated forte, and vibrato, whereas low-arousal emotions such as sadness are associated with more subdued techniques, including breathy, slow legato piano, and straight tone. We additionally compare

4

the mappings of the speech- and singing-optimized models using mutual information (MI). The singing-optimized model achieves higher overall MI with the technique labels (0.474 vs. 0.352), suggesting that the extracted singing-specific descriptors capture genuine technique-related information.

Fig. 4. SHAP for the Singing-Opt. Feature Set: (a) beeswarm for the sad class (b) vibrato dependence for sad.

Fig. 2. G RAD -CAM on VocalSet: (a) vibrato, (b) belt.

For GTSinger, we train a classifier, with each clip inheriting the label of its parent performance. SHAP analysis (Fig. 4a) shows that predictions for happy are associated with higher pitch levels, elevated articulation rate, more frequent loudness peaks, and lower variability in phoneme duration, indicating a more stable and energetic expression. Conversely, sad predictions exhibit a more nuanced profile characterized by a higher pharyngeal quality, greater variability in breathiness, lower pitch and a reduced articulation rate. Utilizing a SHAP dependence plot (Fig. 4b), we observe that the attribution of vibrato variability is mediated by articulation rate: at slower rates, elevated vibrato variability is associated with sad predictions (consistent with a more plaintive delivery), whereas at elevated rates it drives happy predictions, consistent with a more vibrant expression. C. Discussion Vocal technique descriptors contribute to our framework’s interpretability. Although they provide limited numerical gains in raw accuracy in GTSinger (Section IV-A), they establish acoustically coherent mappings between predicted emotion and ground-truth technique on VocalSet and reveal meaningful acoustic patterns (Section IV-B). Their apparent association with low-valence emotional states suggests that they offer complementary information beyond conventional prosodic cues. In particular, pharyngeal technique emerges as the top predictor in our SHAP analysis, consistent with prior work [8] highlighting resonance manipulation as an important aspect of

expressivity in singing. While [29] demonstrates that vibrato parameters vary with emotional engagement, our analysis further suggests that vibrato’s relationship with predicted emotion depends on articulation rate. Other consistently selected descriptors, including pitch, timbre, intensity, and articulation rate, align with acoustic cues previously linked to emotional expression in vocal communication and music performance [30]. Cross-corpus comparisons also indicate that feature relevance varies with singing style, as reflected in the descriptor types favored by each corpus (Table III) and the lower cross-style stability of the speech-optimized feature set (Section IV-A). However, disentangling stylistic differences from other corpus-related factors requires further investigation. TABLE III R EPRESENTATIVE D ESCRIPTORS OF THE S INGING -O PTIMIZED S ETS . Subset

Selected Acoustic & Technique Descriptors

Common (6)

articulation rate, phoneme duration (SD), loudness peaks/s, local shimmer, alpha ratio (σnorm ), and pharyngeal (max).

GTSinger (13)

vibrato (SD, peak), falsetto (SD), breathy (SD), f0 percentiles (20th, 50th), voiced segments/s, local jitter, and σnorm of HNR, spectral slope (0–500 Hz), and harmonic differences (H1 –H2 , H1 –A3 ).

RAVDESS (12)

f0 (σnorm , 80th pct., mean deviation), mean HNR, spectral flux, alpha ratio, F3 amplitude (σnorm ), breathy (mean), falsetto/glissando (max), and technique entropy.

Several limitations warrant consideration. The observed relationships represent statistical associations rather than causal mechanisms, and generalizability is constrained by the limited number of singers, the acted emotions in RAVDESS, and the restricted emotion categories in GTSinger. Moreover, vocal technique features rely on CNN-inferred probabilities, potentially introducing label noise. V. C ONCLUSION

Fig. 3. Distribution of predicted emotion categories across VocalSet

ground-truth techniques for the RAVDESS singing-optimized model.

We presented an interpretable framework for singing emotion recognition using acoustic and vocal technique descriptors. A compact, singing-specific feature set provided a useful representation for emotion recognition while reducing variability across singers. Cross-corpus analyses revealed associations between predicted emotions, vocal techniques, and acoustic characteristics, illustrating the potential of interpretable descriptors to uncover meaningful patterns in vocal expression. Future work should evaluate the framework across more diverse languages, musical genres, and vocal traditions.

5

R EFERENCES [1] J. Dutta and D. Chanda, “Music emotion recognition and classification using hybrid cnn-lstm deep neural network,” Bangladesh Journal of Multidisciplinary Scientific Research, vol. 9, pp. 21–32, 08 2024. [2] H. Pourrostami, M. Nazari, and M. M. AlyanNezhadi, “An explainable mfcc-based deep learning framework for song emotion recognition,” in ICSPIS, 2025, pp. 158–164. [3] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi, “A survey of methods for explaining black box models,” ACM computing surveys (CSUR), vol. 51, no. 5, pp. 1–42, 2018. [4] S. Chowdhury, A. V. Portabella, V. Haunschmid, and G. Widmer, “Towards explainable music emotion recognition: The route via midlevel features,” in ISMIR, 2019, pp. 237–243. [5] V. Lyberatos, S. Kantarelis, E. Dervakos, and G. Stamou, “Perceptual musical features for interpretable audio tagging,” in IEEE ICASSPW, 2024, pp. 878–882. [6] A. Patakis, V. Lyberatos, S. Kantarelis, E. Dervakos, and G. Stamou, “Semantic-Aware Interpretable Multimodal Music Auto-Tagging,” in Interspeech 2025, 2025, pp. 236–240. [7] J. Sundberg, “The acoustics of the singing voice,” Scientific American, 1977. [8] N. Xu, Y. Feng, and X. Chen, “The resonance of emotion: How vocal resonance manipulation shapes emotional perception in singing,” Journal of Voice, 2026. [9] K. R. Scherer, J. Sundberg, L. Tamarit, and G. L. Salomão, “Comparing the acoustic expression of emotion in the speaking and the singing voice,” Computer Speech & Language, vol. 29, no. 1, pp. 218–235, 2015. [10] F. Eyben, G. L. Salomão, J. Sundberg, K. Scherer, and B. Schuller, “Emotion in the singing voice - a deeper look at acoustic features in the light of automatic classification,” EURASIP Journal on Audio Speech and Music Processing, 2015. [11] F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in Proceedings of the 18th ACM international conference on Multimedia, 2010, pp. 1459–1462. [12] L. Heng, “Emotion expression in song and speech: Determining acoustic cues through supervised machine learning techniques,” OSF Preprints, 2026. [13] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using kaldi,” in Interspeech, 2017, pp. 498–502. [14] B. Schuller, B. Vlasenko, F. Eyben, M. Wöllmer, A. Stuhlsatz, A. Wendemuth, and G. Rigoll, “Cross-corpus acoustic emotion recognition: Variances and strategies,” IEEE Transactions on Affective Computing, vol. 1, no. 2, pp. 119–131, 2010. [15] J. Sundberg, F. M. Lã, and E. Himonides, “Intonation and expressivity: A single case study of classical western singing,” Journal of Voice, vol. 27, no. 3, pp. 391.e1–391.e8, 2013. [16] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, and K. P. Truong, “The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,” IEEE Transactions on Affective Computing, vol. 7, no. 2, pp. 190–202, 2016. [17] B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, O. Nieto et al., “librosa: Audio and music signal analysis in python.” SciPy, vol. 2015, no. 18-24, p. 7, 2015. [18] J. W. Kim, J. Salamon, P. Li, and J. P. Bello, “Crepe: A convolutional representation for pitch estimation,” in ICASSP, 2018, pp. 161–165. [19] J. Sundberg, “Acoustic and psychoacoustic aspects of vocal vibrato,” STL-QPSR, vol. 35, no. 2-3, pp. 45–68, 1994. [20] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in NeurIPS, 2017, p. 4768–4777. [21] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in 2017 ICCV, 2017, pp. 618–626. [22] S. R. Livingstone and F. A. Russo, “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,” PLOS ONE, vol. 13, no. 5, pp. 1–35, 05 2018. [23] Y. Zhang, C. Pan, W. Guo, R. Li, Z. Zhu, J. Wang, W. Xu, J. Lu, Z. Hong, C. Wang, L. Zhang, J. He, Z. Jiang, Y. Chen, C. Yang, J. Zhou, X. Cheng, and Z. Zhao, “Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks.” NeurIPS, 2024, p. 1117–1140.

[24] J. Wilkins, P. Seetharaman, A. Wahl, and B. Pardo, “VocalSet: A singing voice dataset,” in ISMIR 2018, pp. 468–474. [25] B. Faghih and J. Timoney, “Annotated-vocalset: A singing voice dataset,” Applied Sciences, vol. 12, no. 18, 2022. [26] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in 2017 ICCV, 2017, pp. 2999–3007. [27] Y. Yamamoto, J. Nam, and H. Terasawa, “Primadnn’:a characteristicsaware dnn customization for singing technique detection,” in 31st EUSIPCO, 2023, pp. 406–410. [28] X. Sun, Y. Jiang, and W. Li, “Residual attention based network for automatic classification of phonation modes,” in IEEE ICME, 2020, pp. 1–6. [29] C. Dromey, S. O. Holmes, J. A. Hopkin, and K. Tanner, “The effects of emotional expression on vibrato,” Journal of Voice, vol. 29, no. 2, pp. 170–181, 2015. [30] P. N. Juslin and P. Laukka, “Communication of emotions in vocal expression and music performance: Different channels, same code?” Psychological Bulletin, 2003.

Related documents

• Interpretable Singing Emotion Recognition Using Acoustic and Vocal Technique Features#1448145
• Dual-Scale State-Space Modeling with Speaker-Wise Dynamic CRF for Speech Emotion Recognition in Conversation#927434
• Advances in speech emotion recognition using a CNN and gender-based segmentation framework with feature selection techniques#1313539
• VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale#1115972
• Deep learning-based emotion recognition using unimodal facial expressions or physiological signals: A review#363850