arXiv:2605.03820v1 [cs.CV] 5 May 2026
Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration Xun Jiang1,2 , Yufan Gu2 , Disen Hu2 , Yuqing Hou3 , Yazhou Yao4 , Fumin Shen2 , Heng Tao Shen1 , Xing Xu1 * 1 School of Computer Science and Technology, Tongji University 2 School of Computer Science and Engineering, University of Electronic Science and Technology of China, 3 Independent Researcher 4 School of Computer Science and Engineering, Nanjing University of Science and Technology
Abstract Multimodal learning often grapples with the challenge of low-quality data, which predominantly manifests as two facets: modality imbalance and noisy corruption. While these issues are often studied in isolation, we argue that they share a common root in the predictive uncertainty towards the reliability of individual modalities and instances during learning. In this paper, we propose a unified framework, termed Conformal Predictive Self-Calibration (CPSC), which leverages conformal prediction to equip the model with the ability to perform self-guided calibration onthe-fly. The core of our proposed CPSC lies in a novel selfcalibrating training loop that seamlessly integrates two key modules: (1) Representation Self-Calibration, which decomposes unimodal features into components, selectively fuses the most robust ones identified by a conformal predictor to enhance feature resilience. (2) Gradient SelfCalibration, which recalibrates the gradient flow during backpropagation based on instance-wise reliability scores, steering the optimization towards more trustworthy directions. Furthermore, we also devise a self-update strategy for the conformal predictor to ensure the entire system co-evolves consistently throughout the training process. Extensive experiments on six benchmark datasets under both imbalanced and noisy settings demonstrate that our CPSC framework consistently outperforms existing stateof-the-art methods. Our code is available at https: //github.com/XunCHN/CPSC.
1. Introduction The proliferation of multimodal sensors has catalyzed remarkable progress in multimodal learning [4–6]. However, * Corresponding author.
Implicit Imbalance
Explicit Noise
Ideal Optimization
Highly Entangled
Corrupted Representation
NYU Depth V2 Dataset
Misleading Optimization
CREMA-D Dataset
High Predictive Uncertainty
Figure 1. Implicit imbalance and explicit noise corruption would increase the model predictive uncertainty. Here we illustrate the statistical log entropy of predicted distributions on NYU Depth V2 [1] and CREMA-D [2] datasets with LFM method [3].
practical systems frequently grapple with low-quality data [7–9], which primarily manifests in two detrimental forms: (1) Implicit modality imbalance [9, 10] with skewed data distributions, causing the model to be biased towards dominant modalities. (2) Explicit noisy corruption [8, 11], where one or more modalities are contaminated by dynamic noise, misleading the learning process. Traditionally, these two challenges have been addressed in isolation. A suite of techniques, such as weighted losses [12, 13], gradient modulation [9, 10], and data re-sampling [14, 15], has been developed to handle modality imbal-
ance. Meanwhile, another line of research focuses on learning from noisy modalities via robust fusion [8, 11, 16]. While effective within their specific domains, this divided approach overlooks a fundamental commonality: As illustrated in Fig. 1, both imbalance and noise exacerbate models’ predictive uncertainty, which is defined as the model’s inherent doubt about the correct prediction for a given sample. Imbalance leads to under-represented modalities being ignored, while noise injects misinformation. Both of these result in overconfident but unreliable predictions. Consequently, the lack of a unified framework that explicitly quantifies and mitigates model uncertainty limiting the generalization and robustness of existing methods. Following this observation, we argue that multimodal learning on low-quality data can be unified by addressing its root cause, i.e., insufficient awareness and calibration of predictive uncertainty. However, as multimodal data can be corrupted by both explicit and implicit factors, existing methods are specialized for particular settings. Hence, how to devise a model-agnostic, unified predictive uncertainty quantification approach becomes an essential problem. To address this problem, we draw inspiration from Conformal Prediction (CP) [17], a statistical framework providing prediction sets with finite-sample and distribution-free guarantees. Unlike Bayesian methods [18, 19] that require priors, CP offers a model-agnostic way to quantify predictive uncertainty. For multimodal learning on low-quality data, we propose a novel unified framework termed Conformal Predictive Self-Calibration (CPSC). As illustrated in Fig. 2, CPSC leverages a dynamically maintained CP model to instill a self-calibration mechanism into the training loop. It enables the model to continuously diagnose its uncertainty and correctify its learning trajectory. The core of our CPSC framework is a novel training paradigm that integrates three key innovations: (1) Representation Self-Calibration that decomposes unimodal features into orthogonal components and employs the CP model to identify and retain the most reliable components for fusion. (2) Gradient Self-Calibration that uses the CP model to recalibrate the gradient during backpropagation, steering the optimization towards more trustworthy directions and mitigating the influence of ambiguous or corrupted samples. (3) Conformal Predictor Updating strategy where the model parameters and the CP model co-evolve by refreshing the CP model with optimized model. Extensive experiments on six benchmark datasets under both imbalanced and noisy settings demonstrate that our proposed CPSC consistently outperforms state-of-the-art methods. Overall, our contributions in this work are threefold: • We propose the CPSC framework, which addresses the multimodal learning from a unified and model-agnostic perspective, linking modality imbalance and noisy corruption through the lens of model predictive uncertainty.
• We propose the Representation Self-Calibration and Gradient Self-Calibration modules, which effectively integrates Conformal Prediction into the core of multimodal training, introducing self-calibration mechanisms for feature representation and gradient optimization. • We provide extensive empirical and theoretical analysis, showing that CPSC achieves new state-of-the-art performance across diverse low-quality data scenarios.
2. Related Work Multimodal Learning on Low-Quality Data. Practical multimodal learning often contends with imperfect data [20–22], which has been primarily studied on the imbalanced and noisy multimodal data. Early methods often employed static fusion strategies, such as weighted averaging [23] or gated mechanisms [24]. Recent approaches dynamically modulate the influence of each modality. For instance, some methods [9, 13, 25, 26] use gradient manipulation to balance the learning pace across modalities, preventing one from dominating the training. Meanwhile, other researchers aim to improve the multimodal learning robustness towards dynamic noise. In this paradigm, a common strategy is to design robust fusion architectures that can selectively focus on reliable information [8, 16]. Other methods [27, 28] explicitly model the reliability of modalities to down-weight untrustworthy signals during fusion. However, existing works above are developed as specialized solutions, thus lacking a unified mechanism to address these low-quality multimodal learning jointly. Predictive Uncertainty. Predictive uncertainty is a crucial concept for reliable machine learning, which quantifies models’ lack of confidence in their outputs for a given input. It has been widely used in massive computer vision or multimodal tasks, such as video understanding [6, 29], cross-modal retrieval [30, 31], and autonomous driving [32– 34]. In multimodal learning, predictive uncertainty estimation has also been prevalent to improve fusion robustness. Typically, a group of methods [8, 27, 34–36] use uncertainty to weight modalities in late fusion [27, 34] or to guide the fusion process in intermediate layers [8, 36]. However, these approaches relied on specified model designs with specific architectural choices or prior assumptions. In contrast, our proposed CPSC approach employs conformal prediction, which address the predictive uncertainty estimation in a model-agnostic and distribution-free manner. Conformal Prediction. The CP [17] is a statistical framework that produces prediction sets with guaranteed coverage probabilities, meaning the true label is contained within the set with a user-specified probability. Its model-agnostic nature has led to applications in risk control [37], anomaly detection [38, 39], large language models [40, 41]. While CP has seen growing diverse applications, its application in multimodal learning theory remains nascent. Existing
Representation Self-Calibration
New Training Set Target Model
Original Initialized Training Warm-Up Training Model Set Random Sampling
Gradient Self-Calibration
Unimodal Classifier (CP Model)
Probability Probability
Unimodal Classifier (CP Model)
… Unimodal Classifier (CP Model)
…
Next Inference on Training Set
Frequency
Probability
Decompose
Unimodal Classifier (CP Model)
Overview of Our CPSC Framework
Class
Decompose
…
Conformal Predictor Updating
Calibration Set
y
Modality V
Training
CP Model
Updating Quantile on Calibration Set
Modality A
…
Probability
Split
Multimodal Classifier (Target Model)
Class
Components with Top-K Reliability
Modality-wise Reliability Estimation Class
Representation Self-Calibration
Conformal Predictor Updating
Gradient Self-Calibration
Figure 2. The overall architecture of our CPSC framework, illustrating the self-calibration training loop with Representation SelfCalibration (RSC), Gradient Self-Calibration (GSC), and Conformal Predictor Updating.
works have primarily applied CP in a post-hoc manner, for instance, to create predictive sets for multimodal outputs after training is completed [42–44]. A few works [45–47] have begun to integrate CP into the training process, such as loss function. Compared to these works, our proposed CPSC pioneeringly builds a self-calibrating training framework that dynamically adjusts both feature-level representation and optimization-level gradients.
3. Our Proposed Method 3.1. Preliminary The Conformal Prediction [17] is a distribution-free framework that provides predictive sets with statistical guarantees. Consider a classification task with label space Y = {1, 2, ..., K}. Given a trained model f and a calibration set Dcal = {(xi , yi )}ni=1 drawn from the same distribution as the test data, CP quantifies the model’s uncertainty for a new test sample xtest . The core of CP is the nonconformity score s(x, y), which measures how “strange” the pair (x, y) is relative to the model’s predictions. Let f (x)y represent the predicted probability for the true label y, the nonconformity score can be calculated as: s(x, y) = 1 − f (x)y ,
(1)
Hence, for a predefined significance level α ∈ (0, 1), i.e., risk factor, the conformal prediction set is constructed as: C(xtest ) = {y ∈ Y : s(xtest , y) ≤ q̂},
(2)
where q̂ is the ⌈(n + 1)(1 − α)⌉/n-th quantile of the nonconformity scores {s(xi , yi )}ni=1 on the calibration set. In this way, the predictions of model on test cases can be guaranteed within marginal coverage: P(ytest ∈ C(xtest )) ≥ 1 − α.
(3)
3.2. Conformal Predictor Construction An Overview of Our CPSC Method. As illustrated in Fig. 2, our CPSC introduces a self-calibration mechanism into the standard multimodal training pipeline. Given a multiN modal training dataset D = {(xm i , yi )}i=1 with M modalities, we first split it into two sets randomly with the indentical distributions, i.e., training set Dtrain and calibration set Dcal . The framework operates through the following two key phases: Warm-up Training Phase for initial training of the multimodal model fθ on Dtrain and Self-Calibration Training Loop. The overall framework creates a closedloop system where the model continuously self-calibrates with its evolving understanding of predictive uncertainty. Specifically, in the self-calibration training loop, for each training iteration t, the training loop follows: (1) Extract unimodal features and apply Representation SelfCalibration (RSC) using the current CP model. (2) Fuse calibrated features and compute predictions. (3) Apply Gradient Self-Calibration using CP reliability scores before parameter update. (4) Update model parameters θt → θt+1 . (5) Execute the Conformal Predictor Updating to refresh the CP model for the next iteration. Warm-up Training Phase. Before starting the selfcalibration training process, we conduct warm-up training process to initialize the CP model. Let fθ = {Eθ11 , Eθ22 , ..., EθMM , Fθf } denote our multimodal model, where E m are modality-specific encoders and F is the fusion classifier, the overall training objective is: θ = arg min E(x,y)∼Dtrain [LCE (fθ (x), y)], θ
(4)
where LCE is the standard cross-entropy loss and θ is the model parameters. We set the warm-up epoch to t0 , and the initialized CP model parameters can be denoted as θt0 After warm-up, we further initialize the CP model using the
calibration set Dcal . For each (xi , yi ) ∈ Dcal , we compute the nonconformity score si = s(xi , yi ) using Eq. 1 and obtain the initial quantile q̂t0 .
The conformal prediction set for component cm k follows:
3.3. Representation Self-Calibration
where q̂t is the current conformal quantile obtained from the calibration set (Please refer to Sec. 3.1). The reliability score rkm for component cm k is then computed based on the position of the ground truth y in the sorted prediction set:
Feature Decomposition. To enable fine-grained reliability assessment, we first decompose the original feature into multiple components. Through the decomposition, we can examine different aspects of the representation separately and identify which components contribute most to reliable predictions. Specifically, we first project the original feature hm into a higher-dimensional space using a modalitym specific fully-connected layer Wdec ∈ Rl×d where l = n × d, followed by a ReLU activation: m m l×1 hm . high = ReLU(Wdec h ) ∈ R
(5)
The high-dimensional feature is then split into n compon m d nents {cm k }k=1 , where each component ck ∈ R : m m m {cm 1 , c2 , ..., cn } = Split(hhigh ).
(6)
d cm k ∈ R is considered as a feature component that captures different aspects of the original representation. To ensure the components are both representative and diverse, we impose constraints using KL divergence:
(7)
where P (·) denotes the probability distribution obtained by applying softmax to the feature vector. λ1 and λ2 are balance factors. By applying the training objective above, we encourage each component to be close to the original feature while maintaining diversity among components. Reliability Scoring. Not all feature components contribute equally to reliable predictions, especially in the presence of noisy or imbalanced data. Hence, we leverage conformal prediction to quantitatively assess the reliability of each component, providing a principled way to distinguish between robust and noisy features with statistical guarantees. For each component cm k , we compute its reliability score using the current CP model. Since the CP model shares parameters with the current main model fθt at iteration t, we directly feed each component to the corresponding unimodal classifier F to obtain classification probabilities. Specifically, for component cm k , we compute: (8)
where pm k represents the predicted probability distribution with modality m over classes. We then calculate the nonconformity scores for all classes using Eq. 1: m s(cm k , y) = 1 − pk [y],
∀y ∈ Y.
rank [y, C(cm k )] . m |C(ck )|
(10)
(11)
rank [y, C(cm k )] denotes the rank position of the true label y when the labels in C(cm k ) are sorted by their nonconformity scores in ascending order, and rkm = 0 if y ∈ / C(cm k ). Robust Feature Reconstruction. After identifying the most reliable components, we need to reconstruct a robust feature representation that emphasizes these components while suppressing less reliable ones. Specifically, n we sort all components {cm k }k=1 by their reliability scores m n {rk }k=1 in descending order and select the top-K components with the highest scores. Let S m denote the set of indices of these selected components, the calibrated feature h̃m is obtained by averaging the selected components: 1 X m ck . K m
(12)
k∈S
i̸=j
m |Y| pm , k = Fm (ck ) ∈ R
rkm = 1 −
h̃m =
n
λ1 X DKL (P (hm )∥P (cm Lm k )) div = n k=1 X λ2 m DKL (P (cm − i )∥P (cj )), n(n − 1)
m C(cm k ) = {y ∈ Y : s(ck , y) ≤ q̂t },
(9)
Here we give a formulated proposition as follows: Proposition 1 The expected deviation between the calibrated representation h̃m and an ideal robust representation hm ∗ is bounded by: E[∥h̃m − hm ∗ ∥2 ] ≤
1 X m E[∥cm k − h∗ ∥2 ], K m
(13)
k∈S
where the selection of components into S m ensures that m m E[∥cm k − h∗ ∥2 ] is minimized for k ∈ S . The theorem demonstrate the effectiveness of our calibrated representation, and the proof is available in our supplementary materials. Hence, the overall training objective of our CPSC framework comprises of conventional classification loss andPdiversity loss for RSC across all modalities M m L = Lmul CE + m=1 Ldiv .
3.4. Gradient Self-Calibration After obtaining the final prediction ŷ = F ({h̃m }M m=1 ), we compute the multimodal cross-entropy loss LCE (ŷ, y) and m unimodal Lm CE (ŷ , y) synchronously. Before backpropagation, we apply gradient calibration to guide the optimization process toward more trustworthy directions. Modality-wise Reliability Estimation. To enable modality-wise gradient calibration, we first need to quantify the reliability of each training instance across different
modalities. By leveraging the conformal prediction framework, we obtain statistically grounded reliability scores that reflect how well each sample aligns with the model’s current understanding of multimodal synergy. Specifically, for each training sample with multiple modalities in the current batch B, we compute the reliability score of the outputs of each unimodal classifier. Rather than using the ground truth label y, we regard the multimodal predicted label y ′ as a synergistic recognition of the current model. Following that, we conduct a similar process with current CP model inference in Reliability Scoring (See Sec. 3.3). Finally, we obtain the unimodal reliability ρm that describes the synergistic discrepancy of m modality: ρm = 1 −
rank(y ′ , C(h̃m )) , |C(h̃m )|
(14)
Gradient Re-weighting. Once we have quantified the reliability of each modality, we can modulate their contributions to the gradient updates for this sample. Specifically, we modulate the loss from each modality based on its reliability using a linear weighting w(ρm ) = a · ρm + b, where a and b are hyperparameters that control the intensity and baseline of the gradient calibration. The gradient for parameter θ with respect to the calibrated modality becomes: |B|
∇θ Lm GSC =
1 X m w(ρm ) · ∇θ Lm CE (ŷ , y), |B| i=1
(15)
where ŷ is the predicted classification results. Here we give a formulated proposition to show the GSC module is stable: Proposition 2 For a convex loss function L, the GSC module with linear weighting reduces the effective variance of the stochastic gradient estimate when w(ρ) is positively correlated with ∥∇L∥2 . The proof can be available in our supplementary materials. Hence, our re-weighting mechanism reduces the influence of unreliable samples with low ρm during optimization while amplifying the contribution of highly reliable samples, effectively steering the model toward more confident and accurate predictions.
3.5. Conformal Predictor Updating After updating model parameters to θt+1 , we refresh the CP model to maintain accurate predictive uncertainty quantification. Specifically, we recompute nonconformity scores for the calibration set using the updated model: st+1 = 1 − fθt+1 (xi )yi , i
∀(xi , yi ) ∈ Dcal ,
(16)
where st+1 is the updated nonconformity score. The coni formal quantile is updated accordingly: q̂t+1 = Quantile({st+1 }N i=1 , ⌈(N + 1)(1 − α)⌉/N ), (17) i
Algorithm 1 Self-Calibration Training Loop 1: for epoch = 1 to E do 2: for batch B in Dtrain do 3: % Representation Self-Calibration 4: Extract and decompose features {hm } → {cm k }
Compute reliability scores {rkm } via CP model 6: Reconstruct h̃m from top-K components 7: % Gradient Self-Calibration 8: Compute predictions ŷ m = Fm ({h̃m }) 9: Estimate modality-wise reliability ρm m 10: Compute classification loss Lmul CE and LCE m 11: Apply the gradient modulation ∇θ LGSC 12: end for 13: % Model Parameter Update 14: Update θ ← θ − η(∇θ L + ∇θ Lm GSC ) 15: % CP Model Update 16: Recompute q̂ using Dcal and updated fθ 17: end for 5:
where N = |Dcal | and Quantile(·) represent the operation of extraction the quantile at the specified location. α is a manual risk factor. This updating strategy ensures that the CP model remains synchronized with the current state of fθ , providing relevant guidance for the next training iteration. After updating the conformal predictor, the overall training loop start the next iteration. For clarity, we summarized the self-calibration training loop of our CPSC approach in Alg. 1. During testing, we use the trained model without RSC to obtain robust predictions. The features from each unimodal encoders are not required to be decomposed during inference. In summary, our CPSC approach only performs during the training stage, thus providing a modelagnostic and unified framework for multimodal learning.
4. Experiments 4.1. Experimental Settings Datasets and Metrics. We evaluate our CPSC method on six multimodal datasets, three imbalanced multimodal datasets: CREMA-D [2], AVE [52], Kinetics Sounds [53], and three noisy multimodal datasets: SUN RGB-D [54], NYU Depth V2 [1] and MVSA-Single [55]. In sum, these datasets cover four modalities including RGB image, depth, audio, and text. We evaluate the model under two settings. For the imbalanced settings [26], we report the multimodal and unimodal accuracy on CREMA-D, AVE, and KineticsSounds to assess the balanced use of modalities. For the robustness settings [11], we report the multimodal accuracy on SUN RGB-D, NYU Depth V2, and MVSA under synthetic noise corruptions like Gaussian and Salt-Pepper. Implementation Details. We followed the former works [3, 11] on imbalanced or robust multimodal learning for
Table 1. Comparisons with recent state-of-the-art methods under the Imbalanced Multimodal Learning settings. Note that Acc{m, a, v} denote multimodal, audio, and visual classification performance.
Method ReconBoost (ICML’24) [48] MMPareto (ICML’24) [26] LFM (NeurIPS’24) [3] InfoReg (CVPR’25) [25] IPRM (IJCAI’25) [49] ARL (ICCV’25) [50] DGL (ICCV’25) [51] CPSC (Ours)
Accm 70.85 70.13 72.53 72.00 74.82 74.38 74.78 76.08
Kinetics Sounds Acca Accv 56.23 50.27 56.40 53.05 57.98 56.43 57.21 53.57 59.76 58.34 58.26 59.74 52.89 60.11 61.54 61.83
Avg 59.12 59.86 62.31 60.93 64.31 64.12 62.59 66.48
fairness. Concretely, for audio-visual datasets, the acoustic modality is transformed into 257 × 1004 spectrograms, and we randomly sample multiple frames from 10-frame video clips. The balance factors λ1 and λ2 are set to 0.8 and 0.2. For the RGB-Depth datasets SUN RGB-D and NYU Depth V2, we similarly adopted ResNet18 as the backbone to extract features from both the RGB image and the depth image. For the text-image dataset MVSA, ResNet152 was utilized for image feature extraction, while the textual content was encoded using a pre-trained BERT model to obtain sentence-level representations. Please refer to our supplementary materials for more details.
4.2. Model Comparisons We compare CPSC against recent state-of-the-art methods in both imbalanced and robust multimodal learning settings. For imbalanced settings, we compare our proposed CPSC method with recent state-of-the-art works including ReconBoost [48], MMPareto [26], LFM [3], InfoReg [25], DGL [51], ARL [50], and IPRM [49]. For robust settings, we compare our method with baseline methods EAU [8], ECML [56], and NLC [57]. Particularly, as a few methods [25, 49–51] did not report their complete performance, we reproduce these methods with their official code under the same experimental settings for fair comparisons. Evaluation on Imbalanced Multimodal Learning. As shown in Table 1, our proposed method outperforms all compared approaches on all three audio-visual datasets in terms of both multimodal and unimodal accuracy. In particular, compared to the recent state-of-the-art method ARL [50], our proposed CPSC method shows superior performance across all three benchmarks. On the AVE dataset, our CPSC method achieves a 5% improvement in multimodal accuracy over ARL, while on CREMA-D, the advantage reaches 3%. These results confirm that CPSC can identify and select the most reliable feature components from each modality via the conformal predictor for fusion. This is because when dealing with modality imbalance, our CPSC method preserves informative yet scarce patterns from the weaker modalities, preventing useful signals from being overwhelmed by the dominant modality.
Accm 79.82 78.53 86.02 76.28 85.35 83.79 82.52 87.83
CREMA-D Acca Accv 60.23 73.01 67.38 70.26 66.53 75.27 64.19 70.62 65.28 76.41 65.92 72.18 65.36 74.84 67.74 80.38
Avg 71.02 72.06 75.94 70.36 75.68 73.96 74.24 78.65
Accm 71.35 75.81 68.58 74.19 74.61 70.42 73.89 77.66
AVE Acca Accv 61.20 39.06 64.34 45.39 64.35 44.89 63.78 42.54 65.11 43.89 63.78 39.61 64.30 42.16 66.93 45.65
Avg 57.20 61.85 59.27 60.17 61.20 57.94 60.11 63.41
Table 2. Comparative results under the Robust Multimodal Learning settings. Note that ϵ is the noise strength. MVSA-Single Gaussian@ϵ Method Clean 5.0 10.0 EAU (CVPR’24) [8] 79.15 73.34 61.78 ECML (AAAI’24) [56] 76.83 71.28 61.03 NLC (AAAI’25) [57] 73.79 65.39 58.98 IPRM (IJCAI’25) [49] 75.84 71.25 60.69 ARL (ICCV’25) [50] 75.76 70.89 60.74 CPSC (Ours) 80.07 74.12 63.32 NYU Depth V2 Gaussian@ϵ Method Clean 5.0 10.0 ECML (AAAI’24) [56] 71.72 62.08 54.58 EAU (CVPR’24) [8] 72.05 62.54 56.23 NLC (AAAI’25) [57] 67.33 54.90 45.04 IPRM (IJCAI’25) [49] 70.13 58.16 52.69 ARL (ICCV’25) [50] 68.72 56.93 49.81 CPSC (Ours) 73.12 64.15 57.32 SUN RGB-D Gaussian@ϵ Method Clean 5.0 10.0 ECML (AAAI’24) [56] 59.82 52.46 46.71 EAU (CVPR’24) [8] 55.68 49.39 44.23 NLC (AAAI’25) [57] 52.75 43.57 38.49 IPRM (IJCAI’25) [49] 58.67 50.82 45.16 ARL (ICCV’25) [50] 56.29 50.78 45.92 CPSC (Ours) 62.12 54.11 49.10
Salt-Pepper@ϵ 5.0 10.0 73.69 60.46 72.13 61.04 66.64 57.28 70.13 58.26 70.49 59.82 73.95 61.27 Salt-Pepper@ϵ 5.0 10.0 57.57 44.93 58.44 46.21 56.02 44.66 56.82 43.37 57.13 44.25 61.22 47.40 Salt-Pepper@ϵ 5.0 10.0 51.92 39.29 50.38 38.38 45.07 37.25 49.98 37.29 50.34 38.46 53.37 41.28
Evaluation on Robust Multimodal Learning. According to the results in Table 2, we list following observations: (1) our proposed CPSC method demonstrates consistent superiority over existing approaches under both clean and noisy conditions. On clean data without corruption, CPSC achieves the best performance on all three datasets. This indicates that the self-calibration mechanism not only enhances robustness but also significantly improves fundamental fusion performance on high-quality data by learning more discriminative feature representations. (2) Under noisy conditions, as the noise intensity increases, all baseline methods exhibit performance degradation, while CPSC maintains stronger stability. We speculate the reason is our RSC module selectively suppresses contaminated feature
components, while the GSC module automatically reduces the optimization weight of unreliable samples, collectively enhancing the model robustness in noisy environments. Table 3. Ablation study on Imbalanced Multimodal Learning and Robust Multimodal Learning settings. Experiments were conducted on CREMA-D and NYU Depth V2 datasets respectively.
Module RSC GSC × × × ✓ ✓ × ✓ ✓
Imbalanced Multimodal Learning Multiple Audio Video Average 78.23 65.41 75.66 73.10 86.29 66.68 76.28 75.32 86.35 66.55 76.72 76.20 87.83 67.74 80.38 78.65
Module
Robust Multimodal Learning Gaussian Salt-Pepper Clean 5.0 10.0 5.0 10.0 69.16 59.63 51.99 56.27 41.22 70.14 59.94 50.17 55.37 41.36 71.36 63.91 53.82 61.16 45.94 73.12 64.15 57.32 61.22 47.40
RSC
GSC
× × ✓ ✓
× ✓ × ✓
Analysis on Representation Self-Calibration. To deeply analyze the individual contributions and synergistic effects of the consistency and diversity constraints in our RSC module, we conduct a dedicated ablation study. Concretely, we consider four configurations through controlled variable testing: the baseline without constraints (w/o Ldiv ), consistency constraint without diversity term (w/o Diversity), diversity constraint without consistency term (w/o Consistency), and full constraints (Full Model). According to Fig. 3, we can see that both individual constraints improve model performance, while the full constraints achieve the optimal results. This demonstrates that consistency and diversity constraints enhance feature representations through distinct mechanisms: the consistency constraint preserves semantic coherence by preventing feature variants from deviating from the original feature core, while the diversity constraint encourages comprehensive feature space coverage by reducing redundancy among variants. Table 4. Performances with different optimizers on Kinetics Sounds (KS) and CREMA-D (CD) datasets.
Dataset
80
w/o Ldiv w/o Consistency w/o Diversity Full Model
70 60 Multi Audio Vision Modality
70 Accuracy (%)
Accuracy (%)
90
60
w/o Ldiv w/o Consistency w/o Diversity Full Model
CD
50
Avg
40 GS5
GS10 SP5 Noise Type
SP10
Figure 3. Analysis of the RSC module on the CREMA-D (Left) and NYU Depth V2 (Right) datasets. GS and SP denoted Gaussian and Salt-Pepper noise respectively.
4.3. Further Analysis Ablation Studies. To validate the effectiveness of our proposed RSC and GSC modules, we conduct systematic ablation studies. From the experimental results in Table 3, we can observe that: (1) On the CREMA-D dataset, the full model incorporating both two modules achieves the best performance, significantly outperforming the baseline model and variants with individual modules. It demonstrates that the RSC and GSC modules effectively alleviate the modality imbalance problem through distinct yet complementary mechanisms. (2) Under noisy conditions, the model incorporating RSC achieves a 2-5% gain over the baseline, whereas the variant with GSC alone shows negligible improvement. It indicates that feature-level reliability filtering plays a more crucial role than gradient-level reweighting when explicit noise interference is present. As the RSC module can directly suppress noise-contaminated feature components, thus preserving model performance.
KS
Optimizer SGD SGD+Ours Adam Adam+Ours AdaGrad AdaGrad+Ours SGD SGD+Ours Adam Adam+Ours AdaGrad AdaGrad+Ours
Accm 78.23 87.83 83.78 85.29 68.31 76.68 67.82 76.08 72.18 75.13 59.48 65.63
Acca 65.41 67.74 61.86 65.38 56.49 59.63 51.98 61.54 53.96 59.82 46.58 53.27
Accv 75.66 80.38 73.78 76.97 34.81 42.18 42.37 61.83 59.31 62.17 27.96 34.69
Avg. 73.10 78.65 73.14 75.88 53.20 59.50 54.06 66.48 61.82 65.71 44.67 51.20
Analysis on Gradient Self-Calibration. To verify that our proposed GSC module is not dependent on a specific optimization algorithm, we conduct experiments using multiple adaptive optimizers, including SGD [58], Adam [59], and AdaGrad [60]. According to the comparative results in Table 4, our method yields significant performance gains across all optimizers. On the CREMA-D dataset, SGD integrated with our approach achieves over 10% improvement in multimodal accuracy, while Adam and the weaker AdaGrad optimizer exhibit gains of approximately 2% and over 8%, respectively. These results strongly demonstrate that our method effectively mitigates modality imbalance, leading to a more stable optimization process. The consistent improvements indicate that our approach successfully guides the optimization trajectory away from local minima, enabling convergence to superior solutions. Analysis on Conformal Predictor Updating. To investigate how the update frequency and initialization strategy of the conformal predictor affect the framework’s perfor-
60 50 Multi Audio Vision Avg Modality
Warm-up w/o Warm-up
5 10 15 20 25 Updating Interval
Figure 4. Analysis on the conformal predictor updating frequency on Kinetics Sounds datasets.
Audio Features t-SNE - Baseline Video Features t-SNE - Baseline Angry Disgust Fear Happy Neutral Sad
Audio Features t-SNE - CPSC Angry Disgust Fear Happy Neutral Sad
Angry Disgust Fear Happy Neutral Sad
Video Features t-SNE - CPSC Angry Disgust Fear Happy Neutral Sad
Figure 5. Comparative visualizations of audio and visual feature representations on CREMA-D datasets.
mance, we conduct ablation studies. By observing the Fig. 4, we can see that: (1) Regarding update frequency, experimental data clearly indicate that model performance monotonically decreases as the update interval increases. One probable reason is that excessively long update intervals cause the conformal predictor to generate calibration signals based on outdated model states. (2) The ablated model w/o Warm-Up performs significantly worse than our complete approach, revealing the importance of calibration timing. The warm-up mechanism allows the model to establish basic representational capacity first, ensuring subsequent calibration is built on a relatively reliable foundation. Visualizations of Feature Representation. We also conduct t-SNE visualization on the visual and acoustic features learned by the baseline method LFM [3] and our CPSC approach on the CREMA-D dataset, with results shown in Fig. 5. The visualization demonstrates a notable advantage of our CPSC method in feature learning compared to the baseline. In the acoustic modality, features learned by CPSC exhibit more compact intra-class distribution and clearer interclass separation, while the baseline features show relatively
scattered distributions with noticeable class overlap. This advantage becomes more pronounced in the visual modality. This visualization result provides feature-level evidence that our method effectively filters reliable feature components through our RSC module. Probability Density
70
77 75 73 71 69 1
Probability Density
Accuracy (%)
Interval 20 Interval 10 Interval 5 Interval 1
Accuracy (%)
80
8 7 6 5 4 3 2 1 0 6 5 4 3 2
Initialized Model Trained Model CREMA-D
0.4
0.5
0.6
0.7
0.8
0.9
Reliability Score
1.0
1.1
Initialized Model Trained Model Kinetics Sounds
1 0
0.2
0.4
0.6
Reliability Score
0.8
1.0
Figure 6. Analysis of reliability of initialized and trained models on the KineticsSounds and CREMA-D datasets..
Analysis on Reliability Scoring. To validate the enhancement of predictive reliability, we also compare the reliability score distributions of initialized and trained models on both the CREMA-D and Kinetics-Sounds datasets. As illustrated in Fig. 6, experimental results indicate the models trained with our CPSC method exhibit significant improvements in reliability on both datasets. The post-training reliability distribution shows a pronounced shift toward higher scores with a more compact profile, indicating higher consistency in predictions across individual samples. The significant refinement in the reliability distribution confirms that our method effectively addresses the issue of overconfident yet unreliable predictions, enabling the model to produce more accurate and trustworthy outcomes. Additionally, to show the effectiveness of our proposed CP-based strategy, we also provide more analysis regarding the prediction coverage rate and average set size along training epochs in our supplementary materials.
5. Conclusion In this paper, we presented CPSC, a novel unified framework for multimodal learning on low-quality data that leveraged conformal prediction to enable self-calibration during training. It consists of two key components: the Representation Self-Calibration module which enhances model robustness by identifying and preserving reliable features, and the Gradient Self-Calibration module steering optimization toward trustworthy directions. Extensive experiments revealed that our proposed CPSC effectively improved the multimodal learning performance on low-quality data. For future work, we will explore extending CPSC to other challenging multimodal scenarios.
6. Acknowledgment This work was supported in part by the National Natural Science Foundation of China under Grant No.62476201, the Fundamental Research Funds for the Central Universities, the Central Guidance on Local Science and Technology Development Fund of Shanghai City (No. YDZX20253100002004), and the New Cornerstone Science Foundation through the XPLORER PRIZE.
References [1] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, pages 746–760. Springer, 2012. 1, 5 [2] Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE TAFFC, 5(4):377–390, 2014. 1, 5 [3] Zhangyi Hu, Bin Yang, and Mang Ye. Empowering visibleinfrared person re-identification with large foundation models. NeurIPS, 37:117363–117387, 2024. 1, 5, 6, 8 [4] Yuan Yuan, Zhaojian Li, and Bin Zhao. A survey of multimodal learning: Methods, applications, and future. ACM Comput. Surv., 57(7):1–34, 2025. 1 [5] Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Comput. Surv., 56(10):1–42, 2024. [6] Xun Jiang, Xing Xu, Huimin Lu, Lianghua He, and Heng Tao Shen. Joint objective and subjective fuzziness denoising for multimodal sentiment analysis. IEEE TFS, 33(1):15–27, 2024. 1, 2 [7] Qingyang Zhang, Yake Wei, Zongbo Han, Huazhu Fu, Xi Peng, Cheng Deng, Qinghua Hu, Cai Xu, Jie Wen, Di Hu, et al. Multimodal fusion on low-quality data: A comprehensive survey. arXiv preprint arXiv:2404.18947, 2024. 1 [8] Zixian Gao, Xun Jiang, Xing Xu, Fumin Shen, Yujie Li, and Heng Tao Shen. Embracing unimodal aleatoric uncertainty for robust multimodal fusion. In CVPR, pages 26876–26885, 2024. 1, 2, 6 [9] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In CVPR, pages 8238–8247, 2022. 1, 2 [10] Yake Wei and Di Hu. Mmpareto: Boosting multimodal learning with innocent unimodal assistance. ICML, 2024. 1 [11] Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. In ICML, pages 41753–41769, 2023. 1, 2, 5 [12] Yuwei Zhou, Xin Wang, Hong Chen, Xuguang Duan, and Wenwu Zhu. Intra-and inter-modal curriculum for multimodal learning. In ACM MM, pages 3724–3735, 2023. 1 [13] Disen Hu, Xun Jiang, Zhe Sun, Hao Yang, Chong Peng, Peng Yan, Heng Tao Shen, and Xing Xu. Geometric gradient divergence modulation for imbalanced multimodal learning. In ACM MM, pages 1337–1345, 2025. 1, 2
[14] Meng Shen, Yizheng Huang, Jianxiong Yin, Heqing Zou, Deepu Rajan, and Simon See. Towards balanced active learning for multimodal classification. In ACM MM, pages 3434–3445, 2023. 1 [15] Xiaoyu Ma, Hao Chen, and Yongjian Deng. Improving multimodal learning balance and sufficiency through data remixing. ICML, 2025. 1 [16] Shicai Wei, Yang Luo, Yuji Wang, and Chunbo Luo. Robust multimodal learning via representation decoupling. In ECCV, pages 38–54. Springer, 2024. 2 [17] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. JMLR, 9(3), 2008. 2, 3 [18] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. NeurIPS, 32, 2019. 2 [19] Violet Yinuo Han, Tianyi Wang, Hyunsung Cho, Kashyap Todi, Ajoy Savio Fernandes, Andre Levi, Zheng Zhang, Tovi Grossman, Alexandra Ion, and Tanya R Jonker. A dynamic bayesian network based framework for multimodal contextaware interactions. In IUI, pages 54–69, 2025. 2 [20] Xun Jiang, Xing Xu, Zailei Zhou, Yang Yang, Fumin Shen, and Heng Tao Shen. Zero-shot video moment retrieval with angular reconstructive text embeddings. IEEE TMM, 26:9657–9670, 2024. 2 [21] Zheng Wang, Xing Xu, Lei Zhu, Yi Bin, Guoqing Wang, Yang Yang, and Heng Tao Shen. Evidence-based multifeature fusion for adversarial robustness. IEEE TPAMI, 2025. [22] Xun Jiang, Xing Xu, Zheng Wang, Jingkuan Song, Fumin Shen, and Heng Tao Shen. Generalizable egocentric task verification via cross-modal hybrid hypergraph matching. IEEE TPAMI, 2026. 2 [23] Xiao-Yuan Jing, Xinyu Zhang, Xiaoke Zhu, Fei Wu, Xinge You, Yang Gao, Shiguang Shan, and Jing-Yu Yang. Multiset feature learning for highly imbalanced data classification. IEEE TPAMI, 43(1):139–156, 2019. 2 [24] Cheng Chen, Qi Dou, Yueming Jin, Quande Liu, and Pheng Ann Heng. Learning with privileged multimodal knowledge for unimodal segmentation. IEEE TMI, 41(3):621–632, 2021. 2 [25] Chengxiang Huang, Yake Wei, Zequn Yang, and Di Hu. Adaptive unimodal regulation for balanced multimodal information acquisition. In CVPR, pages 25854–25863, 2025. 2, 6 [26] Yake Wei and Di Hu. Mmpareto: Boosting multimodal learning with innocent unimodal assistance. ICML, 2024. 2, 5, 6 [27] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification with dynamic evidential fusion. IEEE TPAMI, 45(2):2551–2566, 2022. 2 [28] Ying Liu, Lihong Liu, Cai Xu, Xiangyu Song, Ziyu Guan, and Wei Zhao. Dynamic evidence decoupling for trusted multi-view learning. In ACM MM, pages 7269–7277, 2024. 2 [29] Xun Jiang, Xing Xu, Liqing Zhu, Zhe Sun, Andrzej Cichocki, and Heng Tao Shen. Resisting noise in pseudo labels: Audible video event parsing with evidential learning. IEEE TNNLS, 36(6):10874–10888, 2024. 2
[30] Shenshen Li, Xing Xu, Chen He, Fumin Shen, Yang Yang, and Heng Tao Shen. Cross-modal uncertainty modeling with diffusion-based refinement for text-based person retrieval. IEEE TCSVT, 35(3):2881–2893, 2024. 2 [31] Shenshen Li, Chen He, Xing Xu, Fumin Shen, Yang Yang, and Heng Tao Shen. Adaptive uncertainty-based learning for text-based person retrieval. In AAAI, volume 38, pages 3172–3180, 2024. 2 [32] Carl-Johan Hoel, Krister Wolff, and Leo Laine. Ensemble quantile networks: Uncertainty-aware reinforcement learning with applications in autonomous driving. IEEE TITS, 24(6):6030–6041, 2023. 2 [33] Sanbao Su, Yiming Li, Sihong He, Songyang Han, Chen Feng, Caiwen Ding, and Fei Miao. Uncertainty quantification of collaborative detection for self-driving. In ICRA, 2023. [34] Luke Chen, Junyao Wang, Trier Mortlock, Pramod Khargonekar, and Mohammad Abdullah Al Faruque. Hyperdimensional uncertainty quantification for multimodal uncertainty fusion in autonomous vehicles perception. In CVPR, pages 22306–22316, 2025. 2 [35] Disen Hu, Xun Jiang, Xiaofeng Cao, Zheng Wang, Jingkuan Song, Heng Tao Shen, and Xing Xu. Hyper-opinion vagueness quantification for robust multimodal learning. In AAAI, volume 40, pages 21831–21839, 2026. [36] Zixian Gao, Xun Jiang, Hua Chen, Yujie Li, Yang Yang, and Xing Xu. Uncertainty-debiased multimodal fusion: Learning deterministic joint representation for multimodal sentiment analysis. In ICME, pages 1–6, 2024. 2 [37] William Overman, Jacqueline Vallon, and Mohsen Bayati. Aligning model properties via conformal risk control. NeurIPS, 37:110702–110722, 2024. 2 [38] Arya Saboury and Mustafa Kemal Uyguroglu. Uncertaintyaware real-time visual anomaly detection with conformal prediction in dynamic indoor environments. IEEE RAL, 2025. 2 [39] Xixun Lin, Yanan Cao, Nan Sun, Lixin Zou, Chuan Zhou, Peng Zhang, Shuai Zhang, Ge Zhang, and Jia Wu. Conformal graph-level out-of-distribution detection with adaptive data augmentation. In WWW, pages 4755–4765, 2025. 2 [40] Huanxin Sheng, Xinyi Liu, Hangfeng He, Jieyu Zhao, and Jian Kang. Analyzing uncertainty of llm-as-a-judge: Interval evaluations with conformal prediction. EMNLP, 2025. 2 [41] Yu Gui, Ying Jin, and Zhimei Ren. Conformal alignment: Knowing when to trust foundation models with guarantees. NeurIPS, 37:73884–73919, 2024. 2 [42] Shayan Kiyani, George J Pappas, and Hamed Hassani. Conformal prediction with learned features. In ICML, pages 24749–24769, 2024. 3 [43] Po-han Li, Yunhao Yang, Mohammad Omama, Sandeep Chinchali, and Ufuk Topcu. Any2any: Incomplete multimodal retrieval with conformal prediction. arXiv preprint arXiv:2411.10513, 2024. [44] Alexis Bose, Jonathan Ethier, and Paul Guinand. Conformal prediction for multimodal regression. arXiv preprint arXiv:2410.19653, 2024. 3
[45] Alex C Stutts, Danilo Erricolo, Sathya Ravi, Theja Tulabandhula, and Amit Ranjan Trivedi. Mutual informationcalibrated conformal feature fusion for uncertainty-aware multimodal 3d object detection at the edge. In ICRA, pages 2029–2035, 2024. 3 [46] Domenico Parente, Nastaran Darabi, Alex C Stutts, Theja Tulabandhula, and Amit Ranjan Trivedi. Conformalized multimodal uncertainty regression and reasoning. In ICASSP, pages 6985–6989, 2024. [47] Erfan Hajihashemi and Yanning Shen. Multi-model online conformal prediction with graph-structured feedback. TMLR. 3 [48] Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: Boosting can achieve modality reconcilement. ICML, 2024. 6 [49] Yang Yang, Xixian Wu, and Qing-Yuan Jiang. Towards equilibrium: An instantaneous probe-and-rebalance multimodal learning approach. In IJCAI, pages 3552–3560, 2025. 6 [50] Shicai Wei, Chunbo Luo, and Yang Luo. Improving multimodal learning via imbalanced learning. In ICCV, pages 2250–2259, 2025. 6 [51] Shicai Wei, Chunbo Luo, and Yang Luo. Boosting multimodal learning via disentangled gradient learning. In ICCV, pages 22879–22888, 2025. 6 [52] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In ECCV, pages 247–263, 2018. 5 [53] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017. 5 [54] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, pages 567–576, 2015. 5 [55] Teng Niu, Shiai Zhu, Lei Pang, and Abdulmotaleb El Saddik. Sentiment analysis on multi-view social data. In MMM, pages 15–27. Springer, 2016. 5 [56] Cai Xu, Jiajun Si, Ziyu Guan, Wei Zhao, Yue Wu, and Xiyue Gao. Reliable conflictive multi-view learning. In AAAI, volume 38, pages 16129–16137, 2024. 6 [57] Shilin Xu, Yuan Sun, Xingfeng Li, Siyuan Duan, Zhenwen Ren, Zheng Liu, and Dezhong Peng. Noisy label calibration for multi-view classification. In AAAI, volume 39, pages 21797–21805, 2025. 6 [58] Léon Bottou. Stochastic gradient descent tricks. In Neural networks: tricks of the trade: second edition, pages 421–436. 2012. 7 [59] Kingma DP Ba J Adam et al. A method for stochastic optimization. ICLR, 1412(6), 2014. 7 [60] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12(7), 2011. 7