$TCP_α$: Margin-Controlled Confidence estimation for reliable Music Information Retrieval

T CPα: Margin-Controlled Confidence estimation for reliable Music Information Retrieval Parampreet Singh∗ , Anushka Singh‡ , Sumit Kumar† , Vipul Arora§

arXiv:2608.20326v1 [eess.AS] 20 Aug 2026

Indian Institute of Technology, Kanpur, India Email: ∗ params21@iitk.ac.in, ‡ anushkas22@iitk.ac.in, † krsumit@iitk.ac.in, § vipular@iitk.ac.in

Abstract—Deep neural networks are often overconfident, assigning high confidence even to incorrect predictions. Consequently, users lack a reliable signal for deciding when a prediction can be trusted. Post-hoc confidence estimation addresses this by training a lightweight auxiliary head over a frozen classifier. Existing targets, however, suffer from inherent ambiguity: they assign overlapping confidence values to correct and incorrect predictions, while errors near the decision boundary receive confidence scores indistinguishable from correct predictions. In this work, we propose T CPα , a novel confidence target that resolves these limitations by introducing a margin-controlled penalty for misclassified samples. We prove that T CPα guarantees complete separation between the target values of correct and incorrect predictions, with a separation margin that is independent of the number of classes and increases monotonically with the penalty parameter. Since accurate classifiers naturally produce very few errors, learning these targets results in a severely imbalanced regression problem. We therefore present a systematic study of training strategies for learning under this imbalance and identify an effective training configuration through extensive ablation studies. We evaluate the proposed approach on rāga identification, investigate its robustness under domain shift, and further validate it on frame-wise ornamentation detection without modifying the selected configuration. Across all settings, T CPα consistently outperforms existing confidence targets for failure prediction. Rejecting only the least-confident 8% of predictions improves the base model’s macro-F1 from 0.89 to 0.98, while fine-tuning the confidence head with only 5% labeled samples from a new corpus effectively restores performance under domain shift. It provides an effective and transferable framework for reliable failure prediction in practical music information retrieval systems. Index Terms—Confidence Estimation, Failure Prediction, Music Information Retrieval, Rāga Identification, Ornamentation detection

I. I NTRODUCTION

D

EEP learning has significantly advanced Music Information Retrieval (MIR), enabling state-of-the-art performance across tasks such as automatic transcription [1], [2], source separation [3], [4], instrument recognition [5], genre classification [6], rāga identification [7], and rhythmic-pattern analysis [8], [9]. Despite these advances, most deep learning models provide only a class prediction without indicating whether that prediction should be trusted. This absence of reliable confidence estimates limits their usefulness in applications such as music pedagogy [10], where an automated critique delivered with unwarranted certainty can mislead learners, and equally in recommendation systems, where an unreliable prediction shown with high confidence erodes user

trust. Labeled data is a standing bottleneck across MIR, and automatically labeling archives with tags such as rāga or genre could ease the scarcity and cost of expert annotation [11]. Such a system is useful, however, only if it can identify the predictions it is likely to get wrong and defer them for expert review. Confidence estimation [12]–[14] addresses this by appending a score to each prediction, flagging low-confidence cases for manual review, relabeling, or rejection. A range of methods have been proposed to estimate the confidence of a model’s prediction. The simplest is Maximum Class Probability (MCP) [15], which uses the predicted softmax probability as the confidence score. However, deep neural networks are systematically overconfident [12]–[14], causing MCP to assign high confidence even to incorrect predictions. Sampling- and ensemble-based approaches [16]–[18] provide improved uncertainty estimates but require multiple forward passes or retraining. Post-hoc confidence estimation methods, such as ConfidNet [12], leave the classifier frozen and train a lightweight auxiliary confidence head. Their effectiveness largely depends on the confidence target used for supervision. The original ConfidNet [12] uses the True Class Probability (TCP), which reduces overconfidence by using the probability assigned to the ground-truth class. However, its target values for correct and incorrect predictions overlap, and this ambiguity increases with the number of classes. Its normalized variant, T CPn [12], eliminates this overlap for correct predictions but assigns near-boundary errors confidence targets indistinguishable from correct predictions. These limitations arise from the design of the confidence target itself rather than from the post-hoc framework, motivating the search for improved target formulations. Additionally, learning such confidence targets results in a highly imbalanced regression problem. Since the targets are computed from the predictions of an accurate base classifier, most samples receive high confidence targets, while only a small fraction corresponds to errors. Learning from such a skewed target distribution is therefore challenging. In this work, we propose a modified target TCPα , which augments the TCPn denominator with a penalty term active only on misclassified samples. The penalty term creates separation between mis-classified and correctly classified samples with a margin that is independent of the number of classes and widens monotonically with α. The target imbalance, however, persists regardless of how the target is constructed, since it is inherited from the base classifier itself. We therefore study

a range of training strategies to address it, and identify the configuration that works best through ablations. We first validate the proposed method on rāga identification [7], where the success–error imbalance is particularly severe. We then evaluate whether the same configuration transfers to frame-wise ornamentation detection [19], a task with different temporal granularity and class distribution. Finally, we study domain shift using a second rāga corpus [20]. While our experiments focus on Indian Art Music, the proposed framework can be readily applied to other deep classification problems requiring reliable failure prediction. The contributions of this work are summarized as follows: • We propose TCPα , a novel confidence target for failure prediction that achieves complete separation between correct and incorrect predictions, independent of the number of classes. • We provide theoretical guarantees establishing complete target separation and a margin independent of the number of classes. • We identify the configuration that best handles the training imbalance on rāga identification through a systematic study of training strategies, and show that rejecting a very small fraction of samples yields large performance gains. • We validate on frame-wise ornamentation detection, a task differing in granularity and imbalance, and achieve similar gains by rejection. • We examine transfer under domain shift, and find that fine-tuning the confidence head on a few labeled target-domain samples restores well-separated confidence scores. This work substantially extends our ICASSP workshop publication [21] by (1) providing theoretical guarantees for the proposed TCPα targets; (2) analyzing a range of training strategies for learning the highly imbalanced targets; (3) formulating the problem explicitly as failure prediction rather than confidence calibration; and (4) examining the effect of domain shift. All the codes are available at: https://tinyurl.com/tcp-alpha II. R ELATED W ORKS Confidence and uncertainty estimation for deep neural networks have been studied extensively, with existing methods broadly falling into two categories. The first estimates uncertainty through the training process itself using Bayesian deep learning [16], Monte Carlo dropout [17], or deep ensembles [18]. While these methods generally improve uncertainty estimation, they require specialized training procedures or multiple stochastic forward passes. The second category comprises post-hoc confidence estimation methods, which operate on an already-trained classifier without modifying its decision boundary. Temperature scaling [14] calibrates the softmax probabilities to better match empirical accuracy, while Maximum Class Probability (MCP) uses the predicted softmax probability as a confidence score [15]. Other training-free scores include predictive entropy and the Top-2 Margin, which respectively use the full predictive distribution and the gap

between the top two class probabilities [22], as well as the energy score, which derives a confidence measure directly from the pre-softmax logits [23]. DOCTOR [24] similarly utilizes the full softmax vector for failure prediction without additional training. ConfidNet [12] instead trains a lightweight auxiliary confidence head over a frozen classifier using the True Class Probability (TCP) as the regression target, thereby reducing the overconfidence exhibited by MCP on incorrect predictions. Our work builds upon this post-hoc confidence estimation framework by proposing a new target formulation that addresses the limitations of existing TCP-based targets. An important distinction in this area is between confidence calibration and failure prediction [25]. Calibration seeks probabilistic correctness, ensuring that predicted confidence matches empirical accuracy, and is commonly evaluated using metrics such as the Expected Calibration Error (ECE) [14], [26]. However, recent studies have shown that ECE can be insensitive to important calibration failures and does not always reflect the practical usefulness of confidence estimates [27], [28]. More importantly, good calibration does not necessarily imply good failure prediction. The work [25] demonstrate that several methods which improve calibration error or out-of-distribution detection can simultaneously reduce the separability between correct and incorrect predictions, which is precisely the opposite of the objective in failure prediction. Our goal is therefore not to estimate calibrated probabilities, but to learn a confidence score that reliably distinguishes correct predictions from incorrect ones, enabling unreliable predictions to be rejected, deferred for human review, or prioritized for relabeling. Accordingly, we evaluate using failure prediction metrics such as AUPR-Error, AUROC, and FPR@95%TPR [12] rather than calibration metrics like ECE. Failure prediction is also closely related to selective prediction or confidence-based abstention, which studies using confidence or uncertainty to trade predictive risk against coverage [29]–[32]. Complementary to this line of work, we focus on learning a more reliable confidence score that can be used within such frameworks. Learning confidence targets presents another challenge, as the confidence targets are highly imbalanced. Classical imbalance mitigation techniques such as class reweighting, oversampling, and focal loss [33]–[35] are primarily designed for discrete class labels. Our setting is therefore closer to Deep Imbalanced Regression [36], which introduces Label Distribution Smoothing (LDS) and Feature Distribution Smoothing (FDS) for skewed continuous targets. However, unlike conventional regression problems, the imbalance in our setting is not an intrinsic property of the dataset but is induced by the high accuracy of the underlying classifier itself. We therefore investigate training strategies specifically tailored to this setting. Within Music Information Retrieval (MIR), uncertainty and confidence estimation have received comparatively little attention. Existing studies have explored uncertainty in music genre classification [37], [38], musical improvisation [39], music emotion recognition [40], and melody estimation [41], [42].

In Indian Art Music, previous work has primarily focused on improving predictive performance for tasks such as rāga identification [7], [43]–[45] and ornamentation detection [19], without estimating the reliability of model predictions. Our earlier work [21] took the first step toward closing this gap by introducing a modified TCP target for these two tasks. The present work substantially extends that direction through a more rigorous theoretical treatment, improved learning strategies, and broader experimental validation. III. P RELIMINARIES AND P ROBLEM F ORMULATION A. Setup Consider a labeled dataset D = {(xi , yi∗ )}N i=1 of N i.i.d. samples, where xi ∈ Rd denotes the input sample and yi∗ ∈ Y = {1, . . . , K} is its ground-truth class. Let fw denote a K-class classifier with parameters w that induces a predictive distribution P(Y | w, x) through a softmax layer, and let ŷ = arg max P(Y = k | w, x). k∈Y

(1)

Throughout this paper, we denote the probability assigned to the ground-truth class as p∗ and to the predicted class as p̂: ∗

p

∗

= P(Y = y | w, x),

p̂ = P(Y = ŷ | w, x),

(2)

respectively. We partition the dataset into the success set D+ = {i : ŷi = yi∗ } (fw makes a correct prediction) and the error set D− = {i : ŷi ̸= yi∗ } (fw makes an incorrect prediction), so that D+ ∪ D− = {1, . . . , N }. Following the post-hoc confidence estimation framework of ConfidNet [12], the base classifier fw is trained once and then frozen. As shown in Fig. 1, our objective is to attach to every prediction a scalar confidence score ĉ(x; θ) ∈ [0, 1], produced by an auxiliary head with parameters θ operating on the frozen backbone’s penultimate representation z(x) ∈ Rdf . The head is trained by regression against a confidence target c∗ (x, y ∗ ), Lconf (θ; D) =

N
2 1 X ĉ(xi ; θ) − c∗ (xi , yi∗ ) . N i=1

(3)

The confidence target is a function of the ground-truth label and is therefore available only at training time. The task is to reproduce it at inference from z(x) alone. Consequently, the effectiveness of this framework depends critically on the design of the target c∗ (x, y ∗ ), which is the primary focus of this work.

Fig. 1: Overview of the post-hoc confidence estimation module. The backbone and classifier are frozen after base training; only the confidence head is trained, against the TCPα target derived from the frozen classifier’s softmax output and the ground-truth label.

b) True Class Probability (TCP): ConfidNet [12] instead learns TCP(x, y ∗ ) = p∗ [12], which is the probability associated with the True class instead of the predicted class. Two guarantees follow immediately: 1) if p∗ > 12 then ŷ = y ∗ ; 1 2) if p∗ < K then ŷ ̸= y ∗ . Neither is vacuous, but together they leave an ambiguity band: because an error forces p∗ < p̂ and hence p∗ < 12 , the error 1 targets occupy [0, 21 ) while the success targets occupy [ K , 1]. 1 1 Any target value in [ K , 2 ) is therefore attainable by a correct and by an incorrect prediction alike, and no threshold on TCP can separate the two. The width of this band grows with K. c) Normalised TCP (TCPn ): To eliminate the ambiguity on the success side, [12] normalises the TCP score by the predicted-class probability, TCPn (x, y ∗ ) =

p∗ , p̂

(4)

which pins every success to exactly 1. The authors claim stronger theoretical guarantees compared to TCP, but this creates an additional problem for the error targets. Consider an error with p∗ = m − ε and p̂ = m + ε for some m and ε > 0. Then TCPn → 1 as ε → 0+ , even though the prediction is wrong. So, near-boundary errors are thus assigned targets indistinguishable from confident successes.

B. Existing Targets and Their Failure Modes We briefly review the existing target formulations and highlight the limitations that motivate our proposed target. a) Maximum Class Probability (MCP): The simplest confidence score is MCP(x) = p̂ [15], which is just the probability of the predicted class for the base classifier. It suffers from the problem of models being over-confident. Since p̂ ≥ 1/K always, MCP takes values in [1/K, 1] for both successes and for errors, which results in an overlap in confidence scores for both (Fig. 2).

Fig. 2: Comparison of the target distributions induced by MCP, TCPn , and the proposed TCPα for PIM [7] dataset. The xaxis represents the targets and y-axis represents the density. TCPα produces complete separation between success and error targets.

IV. P ROPOSED M ETHOD

1 1 2 − n

1 1 1 1 2 − 2n + 2 − n + α 1 1 1 n→∞ 2 − n = 3 −−−−→ 2(1 + α) . 1 + α − 2n

A. The TCPα Confidence Target

T CPαn =

The limitations of TCP and TCPn arise because the corresponding targets cannot completely separate correct and incorrect predictions. We therefore introduce TCPα , which modifies TCPn through an additive penalty that is active only for misclassified samples. We define: TCPα (x, ŷ, y ∗ ) =

p∗
, p̂ + I[ŷ ̸= y ∗ ] p∗ + α

(5)

where α ≥ 0 is a penalty hyperparameter. The proposed formulation preserves the desirable property that every correctly classified sample receives a target of one, while explicitly pushing all error targets away from the success region. When ŷ ̸= y ∗ the denominator is inflated by p∗ + α, which strictly depresses the target below its TCPn value. Fig. 2 shows a comparison of the MCP, TCPn and the proposed TCPα targets. It is worth noting that the p∗ term inside the penalty is what removes the near-boundary pathology of TCPn even at α = 0. The constant α then provides continuous, monotone control over how far below the success targets the error targets are pushed. B. Theoretical Guarantees Let T + = {TCPα (x, ŷ, y ∗ ) : ŷ = y ∗ } and T − = {TCPα (x, ŷ, y ∗ ) : ŷ ̸= y ∗ } denote the sets of achievable success and error targets respectively. proposition 1 (Success targets are exactly one). For any x with ŷ = y ∗ and any α ≥ 0, TCPα (x, ŷ, y ∗ ) = 1. Hence T + = {1}. Proof. When ŷ = y ∗ the indicator in (5) vanishes and p̂ = p∗ . So, TCPα = p∗ /p∗ = 1. ■ proposition 2 (Tight ceiling on error targets). For any x with ŷ ̸= y ∗ and any α ≥ 0, TCPα (x, ŷ, y ∗ ) <

1 . 2(1 + α)

(6)

1 The bound is tight: sup T − = 2(1+α) , approached in the limit 1 ∗ ∗ p̂ → p , p → 2 .

Proof. For an error, (5) gives TCPα = p∗ /(p̂ + p∗ + α). Strictness of the argmax gives p̂ > p∗ , hence p̂ + p∗ + α > 2p∗ + α,

so

TCPα <

p∗ 2p∗ + α

. (7)

Now, since p̂ > p∗ and p̂+p∗ ≤ 1, we have 2p∗ < p̂+p∗ ≤ 1, and hence p∗ < 21 . Eq: (7) becomes: 1/2 1 TCPα < 1+α = 2(1+α)

For tightness, fix α ≥ 0 and take the sequence p∗n = 21 − n1 , 1 p̂n = 12 − 2n , which satisfies p̂n > p∗n and p̂n + p∗n < 1 for all n ≥ 3. Thus, each pair (p∗n , p̂n ) corresponds to a valid error prediction. Then,

(8)

Thus, the target assigned to an incorrect prediction is strictly 1 bounded by 2(1+α) , and this bound is tight ■ theorem 1 (Complete target separation and separation margin). For every α ≥ 0, T + ∩ T − = ∅, and the separation margin is 1 . (9) ∆(α) ≜ inf+ v − sup v = 1 − 2(1 + α) v∈T v∈T − The margin is independent of K, satisfies ∆(0) = 12 , is strictly increasing in α, and ∆(α) → 1 as α → ∞. Proof. By Proposition 1, inf T + = 1. By Proposition 2, 1 sup T − = 2(1+α) < 1. Hence, T + ∩ T − = ∅, and ∆(α) = 1 −

1 . 2(1 + α)

Finally, d∆ 1 = > 0, dα 2(1 + α)2 so the margin increases strictly with α, with ∆(0) = 21 and ∆(α) → 1 as α → ∞. ■ proposition 3 (Monotone penalty). For any fixed error sample with p∗ > 0, TCPα is strictly decreasing in α: ∂TCPα p∗ = − < 0. ∂α (p̂ + p∗ + α)2

(10)

For success samples ∂TCPα /∂α = 0. Proof. For errors, C ≜ p̂ + p∗ > 0 is constant in α and TCPα = p∗ /(C + α); differentiating gives the stated derivative, which is negative since p∗ > 0. For successes Proposition 1 gives TCPα ≡ 1, whose derivative is zero. ■ Proposition 3 identifies α as a monotone margin-control parameter: increasing it depresses every error target while leaving every success target pinned at 1, so the two groups are driven apart uniformly and predictably. Remark 1 (Contrast with the baselines). For MCP the two 1 1 target ranges coincide, for TCP the ranges overlap on [ K , 2 ), a band that widens with the number of classes, so the guarantee degrades in exactly the many-class regime where confidence estimation is the most needed. For TCPn the ranges touch (sup T − = inf T + = 1) and the margin is zero. Theorem 1 shows that TCPα is the only member of the family with a strictly positive, K-independent margin, and it attains this already at α = 0. The role of α > 0 is therefore not to create separation but to enlarge it. Table I summarises all these differences.

TABLE I: Theoretical properties of confidence targets. T + , T − are the achievable target ranges for successes and errors; K is the number of classes; α ≥ 0 is the penalty of TCPα . Criterion MCP TCP TCPn TCPα (ours)

T+

T−

Overlap

Margin ∆

1 [K , 1] 1 [K , 1] {1} {1}

1 [K , 1] [0, 12 ) (0, 1)
1 0, 2(1+α)

always 1 1 [K , 2) at boundary none

0 0 0 1 1 − 2(1+α)

C. Learning the Confidence Head Since the base classifier is highly accurate, the target distribution induced by TCPα is extremely skewed, with only a small fraction corresponding to errors. Although the target provides complete separation between correct and incorrect predictions, the resulting regression problem is dominated by the abundant high-confidence samples, making informative low-confidence predictions difficult to learn. To address this challenge, we employ an imbalance-aware training strategy consisting of stratified mini-batch sampling and class-conditional loss weighting. We enforce a fixed errorto-success ratio in every mini-batch during training. Let r = batch batch |D− |/|D+ | be the desired ratio. Since |D− | ≪ |D+ |, error samples are exhausted faster than successes under random sampling, so we maintain a circular buffer over D− . With π (t) the t-th random permutation of D− , the buffer for epoch e concatenates successive permutations, B− (e) = π (1) ∥ π (2) ∥ · · · ,

(11)

drawing fresh permutations until every mini-batch in the epoch is filled. Each error sample thus recurs several times before all successes are seen once, without repeating within a single permutation. We combine this with class-conditional weighting, which weights each sample by the inverse frequency of its base class in the error pool. V. E XPERIMENTAL S ETUP A. Datasets 1) Rāga Identification: The Prasar Bharati Indian Music (PIM) dataset [7] contains 191 hours of polyphonic Hindustani concert recordings annotated with rāga and tonic labels. Following [7], we use the 12 most frequent rāgas, segmented into non-overlapping 30-second audio chunks. To evaluate robustness under domain shift, we use the Saraga-Hindustani dataset [20], an independently collected IAM corpus with different recording conditions and provenance. It contains recordings from 11 of the selected rāgas, with substantially fewer samples per class than PIM. 2) Ornamentation Detection: The Rāga Ornamentation Detection corpus [19] contains mono-channel expert Hindustani vocal recordings with frame-wise labels for seven ornament types recorded by 2 expert Hindustani Musicians. We use audio from Expert-2, segmented into 10-second clips with frame-wise ornamentation labels.

TABLE II: Dataset statistics and the induced imbalance seen by the confidence head. |D− | is the number of base-model errors available for training the confidence head. K is the number of classes. Base F1 shows performance of the base classifier on which the confidence head is being trained. Dataset

Unit

K

|D+ | : |D− |

Base F1

PIM ROD Saraga

30 s chunk frame 30 s chunk

12 7 11

≈8.1:1 ≈2.2:1 ≈3.2:1

0.89 0.69 0.76

B. Base Models For rāga identification, we follow [7] and use tonicnormalized chromagram features as input to a CNN–LSTM classifier trained with the cross-entropy loss on the PIM dataset. For ornamentation detection, we use the Temporal Convolutional Network proposed in [19], which operates on chromagram features using periodic padding and dilated convolutions. Both classifiers are trained to convergence and subsequently frozen. The proposed confidence head is then attached to the penultimate feature representation while leaving the backbone and the original classification layer unchanged. For the domain-shift experiments on Saraga dataset, we directly evaluate the confidence head using the classifier trained on PIM, followed by fine-tuning on a few labeled Saraga samples. Table II summarizes the datasets, the performance of the corresponding base classifiers, and the resulting success-toerror ratios. The final column highlights the central challenge addressed in this work: a classifier with a macro-F1 of approximately 0.89 produces only about one error for every eight correct predictions, resulting in a highly imbalanced confidence learning problem. Unless otherwise stated, all ablation studies are performed on the PIM dataset. For the Saraga and ornamentation datasets, we report only the primary baselines and the final selected training strategy. C. Baselines We compare the proposed method against two categories of confidence estimators. The first comprises training-free scores computed directly from the frozen classifier, while the second consists of learned confidence heads that differ only in the target used for supervision. a) MCP: MCP, described in Section III-B, serves as the simplest confidence baseline. b) Predictive Entropy: Predictive entropy [22] measures the uncertainty of the complete predictive distribution, PE(x) = −

K X k=1

P(Y = k | w, x) log P(Y = k | w, x).

(12) Lower entropy indicates higher confidence; we therefore use −PE(x) as the confidence score. Unlike MCP, it accounts for the entire predictive distribution rather than only the predicted class.

c) Top-2 Margin: The Top-2 Margin [22] measures the separation between the predicted class and its closest competitor, Margin(x) = p̂ − max P(Y = k | w, x). (13) k̸=ŷ

Larger margins correspond to higher confidence and indicate predictions that lie farther from the decision boundary. d) Energy Score: The energy score [23] is computed directly from the pre-softmax logits, E(x) = −T log

K X

exp gk (x)/T ,

(14)

k=1

where T = 1 in our experiments. Higher confidence corresponds to lower energy, so we use −E(x) as the confidence score. e) MC Dropout: It approximates Bayesian model averaging by performing Tmc stochastic forward passes with dropout enabled inference [17], yielding sampled weights wt and the averaged predictive distribution p̄k (x) =

Tmc 1 X P(Y = k | wt , x), Tmc t=1

(15)

where Tmc = 50 in our experiments. The confidence score is taken as the negative predictive entropy of the averaged predictive distribution. f) TCP and TCPn : Here we train an auxiliary confidence head, implemented as a two-layer MLP with a sigmoid output using the target formulations described in Section III-B. To ensure a fair comparison, both are trained using the same confidence head architecture and optimization strategy as the proposed TCPα model. D. Ablation Study Design In addition to the proposed imbalance-aware training strategy described in Section IV-C, we evaluate several alternative approaches for learning the confidence head. under the highly imbalanced TCPα target distribution. 1) Regression with Imbalance Correction: As a baseline, we first train the confidence head using the mean squared error loss of Eq. (3) without any imbalance handling. We then compare it with several regression strategies designed to alleviate target imbalance. Label Distribution Smoothing (LDS) [36] smooths the empirical target distribution and reweights samples according to the smoothed density, while its feature-space counterpart, Feature Distribution Smoothing (FDS) [36], performs smoothing in the learned feature representation. We also evaluate simple error upsampling, which creates duplicate copies of error samples during training. 2) Binary Discrimination: Instead of regressing continuous confidence targets, we also investigate treating confidence estimation as a binary discrimination problem by assigning each sample a success/error label ℓi as: (   1, ŷi = yi∗ , + ℓi = I i ∈ D = (16) 0, ŷi ̸= yi∗ ,

TABLE III: Performance on rāga identification (PIM). All values are percentages, best in bold. (R) denotes the target trained with vanilla regression, (P) denotes training with our proposed training strategy (Sec: IV-C). T2M: Top-2-Margin score, MC-D: MC Dropout Method

FPR@95 ↓

AUPR-E ↑

AUPR-S ↑

AUROC ↑

MCP Entropy T2M Energy MC-D

42.25 41.00 43.00 37.50 40.75

50.59 53.70 44.25 56.87 52.37

96.94 97.10 96.70 97.47 97.13

83.60 84.49 82.53 86.25 84.70

TCP (R) TCPn (R) TCPα (R) TCP (P) TCPn (P) TCPα (P)

46.02 61.74 53.68 12.24 41.24 1.60

25.35 32.64 74.30 83.13 76.01 95.96

98.85 98.42 98.89 99.76 99.20 99.95

85.11 81.74 91.26 97.37 92.74 99.46

and training the confidence head using binary cross-entropy. We additionally evaluate focal loss [33], which emphasizes difficult minority samples by down-weighting well-classified examples. During inference, we take its predicted scalar output directly as the confidence score. 3) Per-Class Models: Finally, we investigate whether replacing a single global confidence head with class-specific models improves performance. Two binary discrimination variants are considered. The first one is trained using only the successes and errors of the corresponding class given by: (k)

Dcls = {(xi , 1) : yi∗ = k, ŷi = k} ∪ {(xi , 0) : yi∗ = k, ŷi ̸= k} . | {z } {z } | class-k successes

class-k errors

(17) Hence, we train k separate models. In another configuration, we also incorporate confusing negatives, i.e., samples from other classes incorrectly predicted as the target class by the base classifier, denoted by Hk : Hk = { (xi , 0) : i ∈ D− , ŷi = k }.

(18)

(k)

So here, we work with: D(k) = Dcls ∪ Hk , giving higher weight to the confusing Hk in the regression loss. Additionally, we evaluate per-class regression models trained on D(k) using the proposed TCPα targets in place of binary targets ℓi . During inference, each sample is routed to the confidence head corresponding to its predicted class. E. Evaluation Metrics We evaluate using the standard failure prediction metrics: AUPR-Error (AUPR-E), FPR@95%TPR, AUROC, and AUPR-Success (AUPR-S). Given the severe class imbalance, AUPR-E and FPR@95%TPR are the primary metrics because they evaluate performance on the rare error class. AUROC measures overall separability, while AUPR-S is reported only for completeness.

VI. R ESULTS AND D ISCUSSION A. Comparison with Existing Baselines Table III compares the proposed method against both training-free confidence scores and learned confidence estimators on the PIM dataset. Among the training-free methods, all confidence scores perform similarly, which is expected since MCP, predictive entropy, Top-2 Margin, and the energy score are deterministic functions of the same classifier output, while MC Dropout estimates uncertainty from multiple stochastic forward passes of the same model. Consequently, they induce similar rankings over correct and incorrect predictions. The energy score is the strongest training-free baseline, achieving an AUPR-E of 56.87 and an AUROC of 86.25, whereas MC Dropout offers no noticeable improvement despite requiring 50 forward passes. Among the learned confidence models, TCPα consistently outperforms both TCP and TCPn under identical training settings. Even with vanilla regression, TCPα substantially improves failure prediction, confirming that the proposed target formulation itself is more informative. Applying the proposed imbalance-aware training strategy further improves all three learned targets, with TCPα achieving the best overall performance (AUPR-E = 95.96, AUROC = 99.46 and FPR@95%TPR = 1.60). TCPn consistently performs the worst among the learned targets because its formulation assigns high target values to many near-boundary errors, making them difficult for the confidence head to distinguish from correct predictions. The rejection curves in Fig. 4 demonstrate the practical impact of this improvement: rejecting only the leastconfident 8% of predictions increases the retained macro-F1 from 0.89 to approximately 0.98. The confidence density plots (Fig. 3) further show that TCPα produces substantially cleaner separation between correct and incorrect predictions than the existing targets.

Fig. 3: Distribution of predicted confidence scores for TCPα on the PIM test set. The proposed target produces clear separation between correct and incorrect predictions. B. Ablation Studies 1) Training Strategy: Table IV compares different strategies for learning the TCPα confidence head under severe target

Fig. 4: Macro-F1 of the retained predictions as increasingly low-confidence samples are rejected using confidence scores from baselines and proposed method.

imbalance. All methods use the same confidence head and TCPα target, differing only in the optimization strategy. Among the regression-based methods, LDS substantially improves over vanilla regression, but the confidence distributions remain considerably more diffuse than those produced by the proposed method. Instead of concentrating the correctly classified samples near confidence one, LDS spreads them over a broad high-confidence region while simultaneously shifting many error samples toward lower confidence. Consequently, the overlap between the two populations is reduced but not eliminated, with a noticeable number of successes still assigned relatively low confidence. The proposed training strategy, in contrast, produces a much sharper separation between the two distributions, resulting in substantially better failure prediction performance. FDS provides no benefit and error upsampling performs poorly, likely because repeatedly sampling the same small set of error examples increases overfitting without improving diversity. Treating confidence estimation as a binary discrimination problem using binary cross-entropy or focal loss is consistently inferior to confidence regression, indicating that the continuous TCPα target contains richer supervision than hard success-error labels. Treating confidence estimation as a binary discrimination problem using binary cross-entropy or focal loss is consistently inferior to confidence regression, indicating that the continuous TCPα target contains richer supervision than hard successerror labels. Since binary models learn only two target values, even a few errors predicted as successes or a few successes assigned low confidence significantly degrade failure prediction metrics, as observed in their density plots. Among the per-class models, incorporating the confusing negatives Hk improves performance over the basic per-class discriminator, confirming that hard negatives are beneficial. However, neither the per-class classifiers nor the per-class regressors outperform a single global confidence model. For the regressors in particular, the small number of error samples available within each class provides insufficient supervision to learn reliable class-specific confidence functions. Such approaches

TABLE IV: Ablation studies for training TCPα target on PIM. All rows use the same target and the same confidence head, and differ only in how the imbalance is handled. EU: Error Upsampling, TCPα -P: proposed (Sec IV-C) Strategy

FPR@95 ↓

AUPR-E ↑

AUPR-S ↑

AUROC ↑

reg FDS EU LDS TCPα (P) Binary Focal P-C P-C-W K-reg

53.68 54.71 66.14 21.98 1.60 76.08 66.67 68.79 63.65 77.26

74.30 65.92 29.91 85.70 95.96 25.92 24.28 50.61 45.92 29.52

98.89 98.89 98.34 98.94 99.95 97.13 97.62 97.95 98.52 96.99

91.26 90.07 81.67 94.42 99.46 76.75 78.52 82.10 86.17 79.14

TABLE V: Effect of the success-to-error ratio r in each training batch on PIM. Batch size is fixed at 64. r

FPR@95 ↓

AUPR-E ↑

AUROC ↑

5.4 3.3 2.2 1.6 1.1 0.6 0.3

39.36 10.62 1.60 7.53 7.63 10.17 13.28

91.09 97.41 95.96 89.21 82.19 73.69 52.81

95.08 97.50 99.46 98.14 98.25 97.31 95.19

may become more effective on datasets with substantially larger numbers of error samples per class. The proposed imbalance-aware training strategy achieves the best performance by a large margin, improving AUPR-E from 74.30 to 95.96 while reducing FPR@95%TPR from 53.68 to only 1.60. The corresponding confidence distributions (Fig. 3) show a clear separation between correct and incorrect predictions, and we use this configuration for all subsequent experiments. 2) Effect of the Success:Error Batch Ratio: Table V studies the influence of the success-to-error ratio r within each training mini-batch while keeping all other settings fixed. Very small numbers of error samples bias learning toward the dominant success class, leading to high FPR@95%TPR despite good average ranking. Conversely, increasing the proportion of error samples gradually degrades AUPR-E. For a ratio of 0.3:1, we observe that the error samples are concentrated below a confidence value of 0.5, but the confidence assigned to correctly classified samples becomes increasingly dispersed in the whole range of (0,1) instead of remaining concentrated near one. As a result, the overlap between the two populations increases, making them less distinguishable. A ratio of approximately 2.2:1 (20 error samples in a batch of 64) provides the best trade-off and is therefore used in all subsequent experiments. 3) Effect of the Penalty Parameter α: Table VI evaluates the influence of the penalty parameter α. Although Theorem 1 guarantees complete target separation even for α = 0, introducing a positive penalty provides an additional margin

TABLE VI: Effect of the penalty α on T CPα performed on PIM dataset. α 0 1/2K 1/K 2/K 0.5 1 5

FPR@95 ↓

AUPR-E ↑

AUROC ↑

11.02 3.48 1.60 6.31 8.10 13.09 8.38

88.10 91.15 95.96 90.20 87.49 89.33 90.40

97.42 97.29 99.46 96.90 96.77 98.02 96.71

TABLE VII: Performance on Saraga dataset. The base classifier is the PIM model in all rows and is never retrained. reg represents vanilla regression training. TCPα (Ps ): best model trained on PIM fine-tuned on Saraga using s% labeled training samples, test set being fixed for all. Model MCP reg TCPα (P0 ) TCPα (P5 ) TCPα (P10 ) TCPα (P20 ) TCPα (P40 )

FPR@95 ↓

AUPR-E ↑

AUPR-S ↑

AUROC ↑

73.38 52.83 87.5 5.0 3.12 2.50 2.25

60.39 80.14 40.16 97.90 97.28 97.92 98.77

89.85 92.05 78.15 99.50 99.46 99.37 99.70

80.61 88.56 62.61 98.21 98.78 98.60 99.36

between the success and error targets, making the regression problem easier to learn. Accordingly, all positive values of α outperform α = 0, with the best performance obtained at α = 1/K. Beyond this point, the performance varies only slightly. We use α = 1/K throughout the experiments. C. Transfer to Domain Shift Table VII evaluates the proposed confidence model under domain shift using the Saraga dataset. The base rāga classifier trained on PIM is kept fixed throughout, and only the confidence head is adapted. Directly applying the confidence head learned on PIM to Saraga (TCPα (P0 )) performs poorly, even below MCP, indicating that the learned confidence distribution is dataset-specific and does not transfer reliably across recording conditions. Encouragingly, fine-tuning the confidence head using only 5% labeled samples from the target domain restores almost all of the performance, improving AUPR-E from 40.16 to 97.90 while reducing FPR@95%TPR from 87.5 to 5.0. Increasing the adaptation set beyond 5% yields only marginal improvements, suggesting that a small amount of labeled target-domain data is sufficient to adapt the confidence model to the new domain. D. Ornamentation Detection Table VIII evaluates the proposed method on frame-wise ornamentation detection using the same configuration selected for rāga identification. TCPα again achieves the best failure prediction performance across all metrics, improving AUPR-E to 86.01 while reducing FPR@95%TPR to 19.13. Similar to

TABLE VIII: Failure prediction on ornamentation detection. The configuration is the one selected on rāga identification; no additional search was performed. Method

FPR@95 ↓

AUPR-E ↑

AUPR-S ↑

AUROC ↑

MCP TCP TCPn TCPα Binary Focal

68.66 33.46 47.88 19.13 35.30 35.10

43.34 76.62 71.21 86.01 81.58 81.41

91.98 97.60 96.77 98.70 97.74 97.91

77.14 92.31 89.22 95.80 92.90 92.96

Fig. 5: Rejection Curves for Ornamentation detection task

VII. C ONCLUSION AND F UTURE W ORK In this work, we propose TCPα , a novel confidence target for post-hoc failure prediction that provides complete separation between correct and incorrect predictions with a margin independent of the number of classes. We study several training strategies for the resulting imbalanced regression problem and identify an imbalance-aware configuration that transfers across tasks. Experiments on rāga identification and ornamentation detection demonstrate that the proposed approach consistently outperforms existing confidence targets and commonly used confidence scores. Rejecting predictions using the proposed confidence scores yields substantial improvements in the retained predictions, showing that a small amount of expert review directed by TCPα is sufficient to make the remaining predictions highly reliable. We also observe that only a small amount of labeled target-domain data is sufficient to adapt the confidence head to a new domain. A promising direction for future work is to develop classconditional confidence estimation that not only identifies unreliable predictions but also indicates plausible alternative classes. This is particularly useful for music analysis, where confusing two closely related rāgas is fundamentally different from confusing unrelated ones. Such information can provide more meaningful feedback in downstream applications such as music analysis and pedagogy. R EFERENCES

the rāga identification task, the binary discrimination models improve upon MCP but remain inferior to confidence regression, confirming that the continuous TCPα target provides richer supervision than hard success-error labels. The rejection curves in Fig. 5 further demonstrate the practical benefit of the proposed confidence estimator. Rejecting approximately 20% of the least-confident frames increases the macro-F1 from 0.69 to approximately 0.95, after which the curve saturates, indicating that most erroneous predictions have already been removed. In contrast, TCP and the binary models peak earlier and then decline, reflecting their weaker separation between correct and incorrect predictions.

Fig. 6: Density plot for the best performing model for Ornamentation detection

[1] A. Saha, H.-U. Berendes, M. Müller, and B. Maman, “Snapping matters: Context-aware onset refinement for automatic music transcription,” arXiv preprint arXiv:2606.11903, 2026. [2] T. Kwon, D. Jeong, and J. Nam, “Towards efficient and real-time piano transcription using neural autoregressive models,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), vol. 32, pp. 5106–5116, 2024. [3] B. Torres, G. Peeters, and G. Richard, “The inverse drum machine: Source separation through joint transcription and analysis-by-synthesis,” IEEE TASLP, vol. 34, pp. 84–95, 2026. [4] S. Araki, N. Ito, R. Haeb-Umbach, G. Wichern, Z.-Q. Wang, and Y. Mitsufuji, “30+ years of source separation research: Achievements and future challenges,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025, pp. 1–5. [5] S. Xu, Y. Wang, Z. Li, F. Yu, and W. Li, “Leveraging timbre semantic descriptors for cross-dataset instrument recognition,” IEEE TASLP, vol. 33, pp. 1196–1207, 2025. [6] Y. Zhang and T. Li, “Music genre classification with parallel convolutional neural networks and capuchin search algorithm,” Scientific reports, vol. 15, no. 1, p. 9580, 2025. [7] P. Singh and V. Arora, “Explainable deep learning analysis for raga identification in indian art music,” IEEE TASLP, vol. 33, pp. 2302–2311, 2025. [8] C.-W. Wu, C. Dittmar, C. Southall, R. Vogl, G. Widmer, J. Hockman, M. Müller, and A. Lerch, “A review of automatic drum transcription,” IEEE TASLP, vol. 26, no. 9, pp. 1457–1483, 2018. [9] R. B. Kodag and V. Arora, “Meta-learning-based percussion transcription and tala identification from low-resource audio,” IEEE TASLP, vol. 33, pp. 2749–2758, 2025. [10] S. Kumar, S. Jaiswal, P. Singh, and V. Arora, “Automatic detection and analysis of singing mistakes for music learning by imitation,” IEEE TASLP, 2026. [11] Q. Kong, C. Yu, Y. Xu, T. Iqbal, W. Wang, and M. D. Plumbley, “Weakly labelled audioset tagging with attention neural networks,” IEEE TASLP, vol. 27, no. 11, pp. 1791–1802, 2019. [12] C. Corbière, N. Thome, A. Bar-Hen, M. Cord, and P. Pérez, “Addressing failure prediction by learning model confidence,” Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019.

[13] C. Corbiere, N. Thome, A. Saporta, T.-H. Vu, M. Cord, and P. Perez, “Confidence estimation via auxiliary models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 6043– 6055, 2021. [14] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in International Conference on Machine Learning (ICML), 2017, pp. 1321–1330. [15] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in International Conference on Learning Representations (ICLR), 2017. [16] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural networks,” in ICML, 2015, pp. 1613–1622. [17] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in ICML, 2016, pp. 1050–1059. [18] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in NeurIPS, vol. 30, 2017. [19] S. Kumar, P. Singh, and V. Arora, “Recognizing ornaments in vocal indian art music with active annotation,” IEEE TASLP, vol. 34, pp. 220– 229, 2026. [20] A. Srinivasamurthy, S. Gulati, R. Caro Repetto, and X. Serra, “Saraga: Open datasets for research on indian art music,” Empirical Musicology Review, vol. 16, no. 1, p. 85–98, Dec. 2021. [21] S. Kumar, P. Singh, and V. Arora, “Confidence-enhanced models for indian art music analysis,” in IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), 2025, pp. 1–5. [22] V.-L. Nguyen, M. H. Shaker, and E. Hüllermeier, “How to measure uncertainty in uncertainty sampling for active learning,” Machine Learning, vol. 111, no. 1, pp. 89–122, 2022. [23] W. Liu, X. Wang, J. Owens, and Y. Li, “Energy-based out-of-distribution detection,” NeurIPS, vol. 33, pp. 21 464–21 475, 2020. [24] F. Granese, M. Romanelli, D. Gorla, C. Palamidessi, and P. Piantanida, “DOCTOR: A Simple Method for Detecting Misclassification Errors,” in NeurIPS, 2021. [25] F. Zhu, Z. Cheng, X.-Y. Zhang, and C.-L. Liu, “Rethinking confidence calibration for failure prediction,” in European Conference on Computer Vision (ECCV). Springer, 2022, pp. 518–536. [26] M. P. Naeini, G. F. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” in AAAI Conference on Artificial Intelligence, 2015. [27] M. Chidambaram, H. Lee, C. McSwiggen, and S. Rezchikov, “How flawed is ECE? an analysis via logit smoothing,” in ICML, 2024. [28] J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, “Measuring calibration in deep learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019. [29] Y. Geifman and R. El-Yaniv, “Selective classification for deep neural networks,” in NeurIPS, vol. 30, 2017. [30] R. El-Yaniv and Y. Wiener, “On the foundations of noise-free selective classification,” Journal of Machine Learning Research, vol. 11, no. 53, pp. 1605–1641, 2010. [31] V. Franc, D. Prusa, and V. Voracek, “Optimal strategies for reject option classifiers,” Journal of Machine Learning Research, vol. 24, no. 11, pp. 1–49, 2023. [32] Y. Geifman and R. El-Yaniv, “Selectivenet: A deep neural network with an integrated reject option,” in ICML. PMLR, 2019, pp. 2151–2159. [33] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980–2988. [34] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie, “Class-balanced loss based on effective number of samples,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 9268– 9277. [35] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: Synthetic minority over-sampling technique,” Journal of Artificial Intelligence Research, vol. 16, pp. 321–357, 2002. [36] Y. Yang, K. Zha, Y.-C. Chen, H. Wang, and D. Katabi, “Delving into deep imbalanced regression,” in ICML, 2021, pp. 11 842–11 851. [37] H. Lukashevich, S. Grollmisch, and J. Abeßer, “Quantifying uncertainty in music genre classification,” in Proceedings of The 49th Annual Conference on Acoustics DAGA. Hamburg, Germany, 2023, pp. 1378– 1381.

[38] T. Ye, S. Si, J. Wang, N. Cheng, and J. Xiao, “Uncertainty Calibration for Deep Audio Classifiers,” in Interspeech, 2022, pp. 1556–1560. [39] T. Daikoku, “Temporal dynamics of uncertainty and prediction error in musical improvisation across different periods,” Scientific Reports, vol. 14, no. 1, p. 22297, 2024. [40] J. Han, Z. Zhang, Z. Ren, and B. Schuller, “Exploring perception uncertainty for emotion recognition in dyadic conversation and music listening,” Cognitive Computation, vol. 13, pp. 231–240, 2021. [41] A. Jaiswal, P. Singh, and V. Arora, “Improving active learning for melody estimation by disentangling uncertainties,” in IEEE ICASSP, 2026, pp. 3401–3405. [42] K. R. Saxena and V. Arora, “Interactive singing melody extraction based on active adaptation,” IEEE TASLP, vol. 32, pp. 2729–2738, 2024. [43] S. Chowdhuri, “Phononet: multi-stage deep neural networks for raga identification in hindustani classical music,” in ICMR, 2019. [44] P. Singh, A. Mishra, A. Raina, and V. Arora, “Ontology-driven hierarchical learning for raga identification,” in 2025 National Conference on Communications (NCC), 2025, pp. 1–6. [45] P. Singh, A. Gupta, A. Mishra, and V. Arora, “Identification and clustering of unseen ragas in indian art music,” in Proceedings of the 26th International Society for Music Information Retrieval Conference, 2025, pp. 811–818.

Related documents

• Retrieving Individual Stems from Music Mixtures with Slot Embeddings#1130325
• TuneJury: An Open Metric for Improving Music Generation Preference Alignment#280190
• Usul and Makam Driven Automatic Melodic Segmentation for Turkish Music#1027266
• MUUNRiver-Bench: Diagnosing Relation-Dependent Music Retrieval with Multimodal Instructions#1110477
• Precision-weighting across domains: A predictive coding framework for language-music integration under uncertainty#297813