Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification Francisco Mena1[0000−0002−5004−6571] ( ), Dino Ienco2,3[0000−0002−8736−3132] , Roberto Interdonato4,3[0000−0002−0536−6277] , Cassio F. 2,3[0000−0002−1934−0625] Dantas , and Simon Besnard1[0000−0002−1137−103X] 1
GFZ Helmholtz Center for Geosciences, Potsdam, Germany [email protected] INRAE, UMR TETIS, University of Montpellier, Montpellier, France 3 INRIA, EVERGREEN, University of Montpellier, Montpellier, France 4 CIRAD, UMR TETIS, University of Montpellier, Montpellier, France
arXiv:2607.24683v1 [cs.CV] 27 Jul 2026
2
Abstract. Multi-modal classification leverages complementary information across diverse data sources to enhance predictive performance. However, real-world scenarios subject to operational constraints, such as sensor failures or privacy restrictions, lead to inconsistent modality availability between training and inference times. To handle missing modalities, prior studies have mainly covered bimodal data setups and focused on designing robust fusion processes. Instead, we adopt a multi-modal colearning framework that prioritizes inter-modal collaboration rather than multi-modal fusion. Specifically, we consider that any subset of modalities may be absent, without assuming predefined missing-modality patterns, an inference scenario we refer to as missing arbitrary modalities. To address this challenge, we introduce two alternative approaches that leverage information at both feature- and decision-level. Experiments on two multi-modal classification benchmarks demonstrate significant robustness gains in various missing modality conditions. The first method shows more robust behavior under minimal missing conditions, where a single modality is absent, whereas the second performs better under extreme missing conditions, where all-but-one modalities are missing. Our code is available at https://github.com/fmenat/Co4Miss. Keywords: Multi-modal classification · Co-learning · Missing modalities · Robustness.
1
Introduction
Multi-modal classification aims to fuse complementary data sources to enhance predictive performance and support reliable decision-making [22]. Although related research primarily focuses on standard multi-modal setups such as image–text–audio [22], real-world applications often involve heterogeneous modalities beyond this setup. Nowadays, modern sensing technologies collect a vast amount of data modalities with varying characteristics, e.g. acquisition processes, resolutions, and physical meaning. In the Earth Observation (EO) domain, satellite imagery, radar, LiDAR, and in-situ measurements provide structurally distinct yet complementary views of the Earth’s surface. In human-centered studies,
2
Mena et al.
wearable and biometric sensors produce diverse physiological and motion signals of a subject. Whereas such diversity enriches representation, it also poses significant challenges if training modalities are absent at inference time. In real-world applications, data modalities cannot be assumed to be consistently accessible across the same phenomenon and time span. This is because data acquisition can be constrained by environmental, operational, and cost factors. As a result, modalities used at training time may be partially observed or entirely missing at inference [21]. This challenge is evident in EO, where optical imagery may be affected by cloud coverage, satellite missions can be discontinued, or may operate only over restricted geographic areas, leading to abrupt gaps in multimodal data sources [18]. Similar issues arise in human-centered systems relying on wearable sensors, where motion signals may be corrupted, interrupted by battery depletion, or unavailable due to user non-compliance. These situations underscore the need for multi-modal models capable of remaining robust when one or more modalities are missing at deployment. The challenge of missing modalities has been addressed with simple data processing techniques (e.g. imputation) to more advanced DL approaches [21]. Most DL methods enforce the fusion process to be robust, e.g. by randomly dropping modalities during training. This Modality Dropout (ModDrop) technique [16] can be implemented in various ways in the literature. For instance, data modalities are replaced with zero at input- [16], or feature-levels [8], filled up with a learnable parameter [21] masked out from the attention [10], or completely ignored during fusion [13]. To recover the full set of training modalities before the fusion time, models like SMIL [11], and ActionMAE [20] perform cross-modal reconstruction. Beyond designing robust fusion processes, Ma et al. [10] introduce a sharing weights mechanism across modality-dedicated components. Moreover, McKinzie et al. [12] harness the knowledge distillation framework with a fullmodal teacher who guides a student with partial modalities available. Recently, the multi-modal co-learning framework has emerged for addressing the challenge of missing modalities. In the co-learning paradigm, multiple models are trained to cooperate (share knowledge), aiming to enhance their individual performance [17]. Beyond its common usages for domain adaptation, noisy labels, and knowledge distillation, few works have harnessed it in multi-modal data setups. This involves the collaboration between modality-dedicated models or components, either in a model-, feature-, or decision-based approach. For instance, MLA [25] shares the last layers of modality-specific models (modelbased) and adjusts gradient directions to avoid overwriting modality-specific knowledge. MDiCo [15] enforces the learning of shared and specific features in modality-dedicated models (feature-based) for unimodal boosting in a bimodal setup. MV-HFMD [1] uses mutual distillation, previously introduced in [26], to guide individual per-modality predictions toward a full-modality consensus. Despite efforts to handle missing modalities, important challenges remain. Most existing methods focus on bimodal setups [21], where models are trained with two modalities, and one is missing at inference time. This limits the applicability of methods such as SMIL [11], DisCoM [6], MDiCo [15], and hallucination-
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification Training time
Audio (
)
Full-modal availability
Elevation (
)
Ground-sensor (
)
Weather (
Inference time
Audio (
)
Elevation (
)
)
Ground-sensor (
)
Weather (
Elevation (
)
Remote optical (
)
RGB optical (
)
Text (
)
Remote radar (
)
)
Remote optical (
)
RGB optical (
)
Text (
)
Remote radar (
)
)
RGB optical (
)
Text (
)
Remote radar (
Moderate missing
)
Ground-sensor (
)
Weather (
Inference time
Audio (
)
Minimal missing
Inference time
Audio (
3
)
Remote optical (
)
Extreme missing
Elevation (
)
Ground-sensor (
)
Weather (
)
Remote optical (
)
RGB optical (
)
Text (
)
Remote radar (
)
Fig. 1: Illustration of three missing modality conditions at inference time: minimal (a missing case when a single modality is missing), moderate (various modalities are missing), and extreme (all-but-one modalities are missing).
based approaches [5] to broader multi-modal setups. In addition, some works specifically address the missing of all-but-one modalities, like EmbrNet [2], distillation [21], and cross-reconstruction-based methods [4]. In contrast, we address the more general scenario of missing arbitrary modalities, where any subset of the training modalities may be available at inference time. As illustrated in Fig. 1, this encompasses a broad spectrum of missing modality conditions, from minimal (single modality) to moderate and extreme (all-but-one modalities). To address missing arbitrary modalities, we formulate a framework grounded in the principles of multi-modal co-learning and knowledge distillation. Rather than focusing the model design on robust fusion processes as previous works have done [16, 2, 10, 20, 4, 13], we prioritize inter-modal collaboration to encourage each modality to contribute to and benefit from the others. Concretely, we introduce two decision-level fusion methods that leverage information at both feature- and decision-level to enhance the classification robustness: Co-learning for Missing (Co-Miss) and Full Co-learning (FullCo). Each method is co-guided to learn modality-shared and modality-specific features with volume-based contrastive loss and a modality discriminant classifier. The Co-Miss method also adopts ModDrop, enforcing missing modality predictions to imitate a consensus derived from full-modality data, a strategy we refer to as missing distillation. In contrast, the FullCo method employs the mutual distillation strategy to guide individual per-modality predictions toward the full-modality consensus. We validate our methods via two multi-modal classification benchmarks. The Multi-CropHarvest dataset for crop-type recognition with four sensor modalities, and HL-Opportunity for human activity recognition with nineteen sensor
4
Mena et al.
modalities. The results demonstrate the classification robustness of our methods against three baselines and seven recent state-of-the-art approaches. Furthermore, the results reveal complementary strengths between our approaches: the Co-Miss method proves particularly effective under minimal missing conditions, while the FullCo method stands out in moderate to extreme conditions. Taken together, these findings highlight the potential of our co-learning strategies for handling missing arbitrary modalities in real-world multi-modal classification scenarios.
2
Related Work
Multi-modal Co-learning The co-learning paradigm has been applied to multi-modal data to handle noisy modalities and weak supervision [17]. This consists of unimodal models employing a feature-, decision-, or model-based collaboration. The feature-based approach usually involves the design of a shared space across modalities. For instance, contrastive learning frameworks maximize the cosine-similarity among modalities [23], while Ienco et al. [6] align representations with a modality classifier trained adversarially. In decision-based colearning, the objective is to exchange the unimodal predictive knowledge. In image classification, Black et al. [1] use the mutual distillation strategy to guide per-modality predictions into a full-modality consensus. In contrast, the modelbased approach exchanges components across unimodal networks. For example, Zhang et al. [25] propose a modality-shared prediction head with orthogonal gradients among modalities. Moreover, Zadeh et al. [24] provide theoretical evidence that multi-modal training can improve performance even if a single modality is available at inference time, suggesting the benefit from auxiliary modalities. Missing Modalities The literature has addressed the challenge of missing modalities with diverse strategies [21]. A common technique corresponds to imputing missing data with a zero placeholder. However, this carries strong bias with a drop in predictive performance [10]. An alternative consists of designing a robust fusion process in multi-modal learning. Recent models like ShaSpec [19] and MissModal [8] use the ModDrop technique [16] to make the multi-modal fusion invariant to missing modalities. Additionally, missing-modality predictions can be regularized to remain less confident than their full-modality counterparts [9], or optimized such that their loss is explicitly higher than that of the fullmodality case [7]. To encourage robustness, Choi et al. [2] propose to select a random modality (for each feature) in the fusion process. Data-driven recovery has also been explored by reconstructing the modalities that are missing from the available ones. Models like SMIL [11] and SimMMDG [4] reconstruct the data modalities for the underlying full-modality fusion process. Beyond the focus on robust fusion design, knowledge- and self-distillation frameworks have been employed to increase robustness to missing modalities [12, 8]. A standard distillation setup consists of a full-modal teacher who guides student models trained with a subset of all training modalities.
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification
5
Co-learning for Missing Modalities The collaboration of modality-dedicated components has been leveraged to improve robustness when modalities are missing. For instance, Hoffman et al. [5] use a hallucination approach that enforces an unimodal model mimicking the behavior of a potential missing modality. Mena et al. [15], use a feature-based co-learning strategy to enforce unimodal models learning modality-shared and -specific features in a bimodal setup. Zhang et al. [25] use a model-based co-learning strategy that exchanges parameters between unimodal models. As decision-based co-learning, Mena et al. [14] introduced mutual distillation in unimodal models tailored for EO data classification. These works have shown the potential of co-learning for missing modalities. Among these works, only a limited number address the challenge of missing arbitrary modalities and extend beyond bimodal setups. For instance, EmbrNet [2], FCoM [13], and MissModal [8] rely on random modality masking to improve the fusion robustness, yet their performance with missing modalities remains limited. To this end, we formulate two methods grounded in the principles of multi-modal co-learning and knowledge distillation to enhance the classification robustness in inference scenarios with missing arbitrary modalities.
3
Methodology
Multi-Modal Setup Let us consider M as the set of M = |M| training modalities, and X = {Xm }m∈M the multi-modal input data with label y ∈ {1, . . . , K}. At inference time, any arbitrary subset of these modalities M̃ ⊆ M (with M̃ = |M̃|) may be accessible, expressed by X̃ = {Xm }m∈M̃ . We define this scenario, encompassing any missing modality case, as missing arbitrary modalities. To address this challenge, we introduce two multi-modal methods employing decision-level fusion, also known as late fusion. In concrete, we employ a simple fusion process where per-modality probability predictions ŷm ∈ [0, 1]K are averaged to Pyield the full-modality estimation (called consensus), given by ŷfull = M −1 · m∈M ŷm . Thus, if modalities are missing at inference time, they P are just disregarded from the aggregation, expressed by ỹ = M̃ −1 · m∈M̃ ŷm . For optimization, we use a standard cross-entropy loss to guide the main prediction with full-modality data, defined as Lmain = LCE (y, ŷfull ) ,
(1)
P where LCE (p, q) = − k 1(p = k) · log qk is the loss function between the true label p and probabilities q, and 1(·) is the indicator function. We introduce additional loss terms at the feature-level (via co-learning) and at the decisionlevel (via knowledge distillation) as follows. 3.1
Feature-level Learning Criteria
We assume that training modalities have shared (invariant among modalities) and specific (unique to each modality) information among them relevant for the
6
Mena et al.
classification. The shared space corresponds to class-relevant features that can be extracted from either modality, while the specific space contains class-relevant features that can only be extracted from a specific modality. For instance, a highresolution and a low-resolution optical satellite image have common data (the optical part) and specific data (related to the differences in spatial resolutions). In concrete, we use modality-dedicated encoders that learn to extract both the d spe d shared zsha m ∈ R and specific zm ∈ R features explicitly, given by spe zsha m , zm = Em (Xm ) .
(2)
For learning the specific features per modality, we use a modality discriminant loss function (based on the cross-entropy), expressed by Lmod =
1 X LCE (m, P spe (zspe m )) , M
(3)
m∈M
where P spe (·) is an auxiliary linear classifier that is fed with the specific features from either modality and has to predict the correct one. In this way, the specific features per sample have to be linearly distinguished among modalities. For learning the shared features among modalities, we use a volume-based contrastive loss function [3], defined by Lcont =
1 sha sha sha LGRAM zsha , a , Za ; γ + LGRAM Za , za ); γ 2
(4)
where LGRAM (·, ·; γ) is the GRAM multi-modal contrastive loss defined in [3] = and parametrized by a scalar γ, a ∈ M is the anchor modality, and Zsha a : m ∈ M/a}. We use γ = 0.07, following [15]. The minimization of the {zsha m Gramian volume among the shared features is applied over all modalities for each sample, enforcing them to be closer in space. Individual Prediction To obtain per-modality predictions, our model concatenates the modality-shared and modality-specific features (from Eq. (2)) followed spe by a modality-dedicated linear head Pm (·). This is given by ŷm = Pm (zsha m ||zm ). 3.2
Decision-level Learning Criteria
We consider two knowledge distillation variants used at the decision-level to improve the predictive robustness to missing arbitrary modalities. Missing Distillation Expecting any missing modality condition (see Fig. 1), we incorporate the ModDrop technique at the decision-level to expose the model to missing modalities during training. This missing modality prediction is ex−1 P pressed by ŷ miss = Mdrop · m∈M (1−dm )·ŷm , where dm ∼ P Bern(α) the randomly drawn decision if modality m is masked out, and Mdrop = m∈M dm the number of modalities available. However, instead of drawing a single random case per
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification
7
Modality label Modality 1 Modality 2 Modality
Main predictive loss Shared features
Logits
Modality-dedicated encoder ( ) Modality 1
C
Loss function
Specific features
Modality-dedicated head ( )
C
C1 C2 C3
/
A
Contrastive loss
Concat / Average
Modality discrim. loss
Main prediction
Missing distillation loss
Modality label Modality 1 Modality 2 Modality
Fused logits
Logits
Modality-dedicated encoder ( )
C
Modality-dedicated head ( )
C1 C2 C3
A
Ground truth
C1 C2 C3
C1 C2 C3
Modality 2 Volume-based (GRAM)
Fused logits
A
C1 C2 C3
Case 1
Case 2
Logits
Modality-dedicated encoder ( ) Modality
C
Modality-dedicated head ( )
C1 C2 C3
ModDrop Case 3
Case 4
Case 5
Case 6
Case 7
Modality label Modality 1 Modality 2 Modality
Fig. 2: Illustration of the Co-Miss method with four loss functions. sample, we exhaustively simulate all missing modality cases, i.e. 2M − 1 combinations. Then, we enforce this missing modality prediction ŷmiss to imitate the consensus ŷfull , as well as the ground truth y, by the following loss function Lmiss = LCE (y, ŷmiss ) + λ · LKD (ŷfull , ŷmiss ; τ ) ,
(5)
where λ is a weighting factor (we use λ = τ 2 following [14]), and LKD (·, ·; τ ) is the knowledge distillation function parametrized by the temperature τ (we use τ = 0.5 following [14]). We refer to this strategy as missing distillation. This Co-learning for Missing (Co-Miss) method is optimized via an unweighted sum of all afore-mentioned loss terms, expressed by Ltotal = Lmain + Lmod + Lcont + Lmiss .
(6)
This method, illustrated in Fig. 2, follows a feature-based co-learning (in Lmod and Lcont ) combined with a distillation process (in Lmiss ). Mutual Distillation To address extreme missing conditions (See Fig. 1), we follow a decision-based co-learning strategy called mutual distillation, where each per-modality prediction ŷm has to imitate both the consensus ŷfull and ground truth y. The loss function of this mutual distillation strategy considers the average of the individual per-modality distillations, given by 1 X (m) Lmut = (7) Lmut M m∈M
(m) Lmut = LCE (y, ŷm ) + λ · LKD (ŷfull , ŷm ; τ )
,
(8)
where λ = τ 2 and τ = 0.5, following the same criteria as for the Co-Miss method. This Full Co-learning (FullCo) method is learned by optimizing an
8
Mena et al. Specific features
Modality label
Shared features C
/
Logits
Modality-dedicated encoder ( )
C
Modality 1
Modality-dedicated head ( )
Loss function Main predictive loss
Modality 1 Modality 2 Modality
A
Contrastive loss
Concat / Average
Modality discrim. loss
Main prediction
Mutual distillation loss
C1 C2 C3
Modality label Modality 1 Modality 2 Modality
Fused logits
Logits
Modality-dedicated encoder ( )
C
Modality-dedicated head ( )
C1 C2 C3
A
Ground truth
C1 C2 C3
C1 C2 C3
Modality 2 Volume-based (GRAM)
Logits
Modality-dedicated encoder ( ) Modality
C
Modality-dedicated head ( )
C1 C2 C3
Modality label Modality 1 Modality 2 Modality
Fig. 3: Illustration of the FullCo method with four loss functions.
unweighted sum of the following loss terms: Ltotal = Lmain + Lmod + Lcont + Lmut .
(9)
This method, illustrated in Fig. 3, follows a full co-learning strategy, with a feature-based (in Lmod and Lcont ) and decision-based (in Lmut ) collaboration. Our methods differ in how the decision-level learning is carried out: through the missing distillation (Lmiss ) in Co-Miss or mutual distillation (Lmut ) in FullCo.
4
Experiments
4.1
Experimental Setup
Datasets We use the following multi-modal classification benchmarks. Multi-CropHarvest 5 We consider a crop-type recognition problem by using the CropHarvest dataset. This benchmark contains 29 642 samples around the globe between 2016 and 2021. The label covers ten different crop-type groups (other, beverage spice, cereals, leguminous, fruits nuts, root tuber, oilseeds, vegetables melons, sugar, non crop). The input data consist of three multi-temporal sensor modalities, at 10[m] spatial resolution: multi-spectral optical data from Sentinel2, radar data from Sentinel-1, and weather variables. In addition, topographic features (mono-temporal modality) are available for each sample. Following previous research [14, 15], we perform a 10-fold cross-validation. 5
https://github.com/nasaharvest/cropharvest (Accessed 27.07.2026).
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification
9
HL-Opportunity 6 We consider a high-level activity recognition problem by using the Opportunity dataset. This benchmark consists of data from six subjects performing different activities (five high-level: relaxing, coffee time, early morning, cleanup, sandwich time). There are 19 modalities, corresponding to various sensors positioned on the body of the subjects (e.g. food, back, arms, hip). Following the evaluation setup by Choi et al. [2], a total of 264 279, 6 528, and 12 688 samples are available for training, validation, and testing, respectively. Comparison We compare our methods against the following approaches. Baselines We consider unimodal models trained exclusively for each modality (i.e. no multi-modal interaction during training). As multi-modal baselines, we include two models employing feature- (Feat) and decision-level (Dec) fusion, respectively, where missing modalities are imputed with zeros at inference time. Competitors We include seven methods from the recent literature, general enough to accommodate multi-modal beyond bimodal setups. Five methods use featurelevel fusion: EmbrNet [2], a method that randomly selects a single modality per feature, and the following ModDrop-based approaches: MissModal [8], ShaSpec [19], FCoM [13], and SimMLM [7]. In addition, two decision-level fusion methods that leverage the mutual distillation strategy: DML [26], and DSensD+ [14]. Evaluation details For the classification assessment, we calculate the weighted F1 (F1) score with full-modality data and under every missing modality case. For each missing modality condition (i.e. minimal, moderate, and extreme), we average the F1 values of the corresponding missing modality cases. We also consider the macro missing performance as the average between the following settings: full-modality data, minimal missing, moderate missing, and extreme missing conditions. In addition, the method rank (according to performance, i..e rank 1= best) is calculated and averaged across all missing modality cases. Implementation We apply a z-score normalization on the input data. We use standard encoder architectures for the selected datasets [14, 2]. This corresponds to a 1D CNN for multi-temporal modalities, and an MLP for mono-temporal ones. For each encoder, we use two layers with 128 units and 20% of dropout on all encoders in the Multi-CropHarvest dataset, and four layers with 64 units for the HL-Opportunity. We apply a layer normalization to the per-modality predictions to scale the logits magnitudes. For optimization, we use AdamW with a learning rate of 10−3 , batch-size of 128, and early stopping. The stopping considers the F1 score (macro missing) calculated on the validation set. We use weights in the main predictive loss function Lmain (inverse to the number of samples per class) to cope with class imbalance. For all competitors, we retain the default hyperparameter settings as reported in their original works. 6
www.opportunity-project.eu/challengeDataset.html (Accessed 27.07.2026).
10
4.2
Mena et al.
Experimental Results
We report the crop-type recognition results in Table 1. Here, the lowest drop in performance occurs under minimal missing conditions, varying whether the optical, radar, weather, or topographic modality is missing. In the Dec baseline, this ranges from 0.6 points when the topographic is missing to 17.1 points when the optical is missing. In contrast, the Co-Miss method reduces these drops to 0 and 12.1 points, respectively. Overall, the drop in performance is more evident in the extreme missing conditions. In the baselines, there is around 40 points of performance drop in the extreme (average) condition. In this case, our Co-Miss method has a drop of 25.2, while FullCo has 23.9 points. In addition, our methods outperform all competitors with significant differences in each missing condition, due to different advantages. The Co-Miss method outperforms all approaches in full-modality data, as well as in minimal missing conditions, except when missing the weather modality. The FullCo method achieves the best results under moderate and extreme missing conditions. This is expected, as the FullCo method is explicitly trained to handle unimodal data (via the distillation of per-modality predictions). Besides, FullCo achieves the second-best results in full-modality data and minimal missing conditions. These complementary advantages make our methods tie as the best average ranking with a 1.5 rank.
Table 1: F1 score in the crop-type recognition task (Multi-CropHarvest). The sensor modalities are optical (O), radar (R), weather (W), and topographic (T). Bold indicates the best mean performance. Underlined indicates values that are not statistically different from the best method (Welch’s t-test, p > 0.05). ∗
Averaged over available cases.
O R W T Uni- Feat Dec Embr- Miss- Sha- FCoM Sim- DML DSen- Co- Fullmodal Net Modal Spec MLM sD+ Miss Co ✓ ✓ ✓ ✓ 71.1 73.3 73.6 72.1
70.6
67.1
75.1
73.9 66.5 74.3
79.9
79.7
×✓ ✓ ✓ ✓× ✓ ✓ ✓✓ × ✓ ✓✓ ✓ ×
57.5 56.5 66.1 69.4 65.0 59.7 70.3 73.0
56.6 69.6 65.6 72.5
63.7 68.1 69.5 70.5
56.4 64.9 65.9 67.0
64.4 73.2 74.8 75.2
62.3 71.7 73.2 73.8
65.1 73.1 74.5 74.7
67.8 79.0 79.3 79.9
67.7 78.9 79.4 79.7
minimal
64.7 64.7 66.1
67.9
63.6
71.9
70.3 64.8 71.9
76.5
76.4
moderate
51.8 51.4 56.2
62.0
55.6
65.1
62.5 59.5 65.7
69.3
69.6
57.8 56.4 37.1 31.8 30.9 34.8 10.0 3.1
58.7 64.2 69.0 67.2
✓× × × ×✓ × × ×× ✓ × ×× × ✓
72.5 55.6 46.5 21.8
64.4 38.8 39.9 8.3
67.4 54.2 47.2 22.9
63.4 43.5 39.7 12.9
73.3 54.2 46.1 21.5
70.8 51.0 42.7 18.3
73.6 56.9 48.2 28.4
78.5 57.9 51.3 31.2
78.8 59.2 52.0 33.5
extreme
49.1 34.0 31.5 37.8
47.9
39.9
48.8
45.7 49.8 51.8
54.7
55.8
55.8 55.4 58.0
62.2
56.3
65.4
63.2 61.2 65.8
70.1
70.4
9.7 9.6
6.5
9.2
3.9
5.8
1.5
1.5
Macro Ranking
60.1
∗
8.0
77.8 56.6 47.4 23.9
6.8
3.5
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification
11
Table 2: F1 score in the high-level activity recognition task (HL-Opportunity). Bold indicates the best mean performance. Underlined indicates values that are not statistically different from the best method (Welch’s t-test, p > 0.05). ∗
Averaged over available cases
Inference Uni- Feat Dec Embr- Miss- Sha- FCoM Sim- DML DSen- Co- Fullmodal Net Modal Spec MLM sD+ Miss Co full-modal 77.4 71.9 69.6 71.0 minimal 70.6 68.7 70.7 moderate 57.5 56.4 62.2 extreme 59.1 28.0 27.1 35.1
78.7 78.5 72.2 49.6
77.0 76.6 67.6 32.3
78.8 78.5 69.9 41.1
78.2 77.9 69.7 42.6
76.0 76.0 73.5 60.5
79.5 79.4 74.9 57.0
78.1 77.7 70.0 43.3
79.0 78.7 74.0 56.4
Macro
68.2∗ 57.0 55.5 59.7
69.8
63.4
67.0
67.1 71.5 72.7
67.0
72.0
Ranking
10.1 10.6 8.9
3.7
7.5
5.4
5.6
6.1
2.4
4.3
1.4
Finally, the macro missing results in the Multi-CropHarvest dataset indicate that, regardless of the modalities missing at inference time, the expected F1 performance is 70.1 and 70.4 for Co-Miss and FullCo methods, respectively. This value improves around 4 points compared to the closest competitor (DSensD+ ), and up to 15 points compared to the Feat and Dec baselines. Moreover, the improvements achieved by our methods are statistically significant in all cases. Besides, no significant differences are observed among our methods across the inference scenarios, except when radar or topographic modalities are available. The results in the activity recognition task are displayed in Table 2. We only report the average of the three missing conditions, as with 19 sensor modalities, there are 219 − 1 missing modality cases possible. Similarly to previous results, the performance degrades the least under minimal missing conditions and more noticeably under extreme missing ones. The potential information redundancy among the 19 sensor modalities explains the limited performance drop when a single modality is missing (i.e., 18 modalities are still available). In our methods, the performance decreases by just 0.3 points with FullCo and 0.4 with Co-Miss. Overall, the best results are achieved by the DSensD+ method in all missing conditions except for the extreme missing one, where DML obtains the best results. Among our methods, FullCo competes with DSensD+ and DML methods, obtaining the second-best results across all missing conditions (except extreme), and being statistically similar to the best results in both full-modal and moderate conditions. Moreover, the average ranking shows FullCo as the second-best method overall with a 2.4 rank. At last, the macro missing results indicate that, regardless of the modalities missing at inference time, the expected F1 performance of the FullCo method remains 72.0. This represents a marginal difference of only 0.7 points compared to DSensD+, which is not statistically significant. Increasing missing modalities Figure 4 shows that the classification performance decreases non-linearly as the number of missing modalities increases. This drop is more pronounced for the baseline methods, which lack any dedicated mech-
12
Mena et al.
Fig. 4: F1 score by increasing the number of modalities missing at inference time. The 95% confidence interval is included as shaded colors for each method.
anism to handle missing modalities. The class-relevant redundancy among the 19 modalities is noted in the HL-Opportunity, where the performance remains mostly unchanged until five modalities are missing. Moreover, the performance difference and variability between the methods increase considerably as more modalities are missing. This trend highlights the growing robustness challenge as we move from minimal to moderate and extreme missing conditions. Similar to previous results, the performance relates to the considered dataset. In MultiCropHarvest, FullCo and Co-Miss methods achieve the best robustness curves along missing modalities, while in HL-Opportunity, the FullCo has comparable behavior to the best competing approaches (i.e. DSensD+ and DML). 4.3
Ablation Results
In this subsection, we vary individual components to isolate the key factors characterizing the behavior of our methods. These results are reported in Table 3 for the Multi-CropHarvest dataset, while similar results are observed in HL-Opportunity. We observe that the most relevant loss term is the contrastive one (Lcont ), followed by the decision-level term, either mutual (Lmut ) or missing (Lmiss ) distillation. On the other hand, the loss term that least affects performance is the modality discriminant one. This indicates that for our co-learning method, the modality-shared information is more crucial than the modalityspecific counterpart. Furthermore, we compare the volume-based contrastive loss against its standard pair-wise version (computed over all pairs [23]). These results suggest that the volume-based formulation is more effective for missing conditions, yielding a more structured modality-shared space within our approach.
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification
13
Table 3: F1 score in the Multi-CropHarvest dataset under different configurations in our methods. The best and second-best values are highlighted. Ablation case
Full-modal
Minimal
Moderate
Extreme
Macro
Co-Miss w/o Lmod w/o Lcont w/o Lmod and Lcont pair-wise contrastive
79.9 79.9 79.2 79.2 77.5
76.5 76.5 75.7 75.6 75.4
69.3 69.3 68.5 68.5 68.5
54.7 54.4 54.0 54.1 53.4
70.1 70.0 69.4 69.3 68.7
FullCo w/o Lmod w/o Lcont w/o Lmod and Lcont pair-wise contrastive
79.7 79.5 78.8 78.9 77.4
76.4 76.3 75.5 75.7 74.1
69.6 69.5 68.7 68.9 67.5
55.8 55.5 55.0 55.2 54.0
70.4 70.2 69.5 69.7 68.2
w/o Lmut or Lmiss
79.8
76.1
68.7
53.4
69.5
Fig. 5: F1 gap between the default version of our methods and different variants.
We display the performance gap (difference between the F1 score of a variant and the default version of our approaches) in Fig. 5. In this case, removing the feature-level loss terms (Lmod and Lcont ) causes the largest performance drop when few modalities are missing. Conversely, removing the decision-level loss term (either Lmut or Lmiss ) degrades performance most when more modalities are missing. This reflects the complementarity of our feature- and decision-level learning criteria: feature-level collaboration is key to robustness when facing moderate to minimal missing conditions, while decision-level distillation is crucial when facing moderate to extreme missing conditions at inference time. 4.4
Model Comparison & Limitations
The Co-Miss method has a scalability limitation that future work should address, as its training complexity scales as O(2M ). For example, with M = 10 modalities, it must simulate 1023 missing-modality combinations. Given this limitation and its weaker performance under severe missing conditions, we recommend FullCo, particularly for moderate to extreme missing inference scenarios. When a few
14
Mena et al.
modalities are available for training (i.e., M < 10), Co-Miss remains a suitable choice, especially for full-modality and minimal missing inference scenarios. Both of our methods share similarities to MDiCo [15]. Despite the learning of a modality-shared space by contrastive learning (Sec. 3.1), we use a volumebased version instead of the pairwise loss as in [15]. However, MDiCo is a purely multi-modal co-learning approach, i.e. no fusion is performed among modalities. Thus, MDiCo does not employ any decision-level learning criteria as us (Sec. 3.2). We validate our approaches in two applications, while conventional multimodal benchmarks (e.g., image-text-audio) can demonstrate their generalization beyond sensor-based domains. Future work should also consider adaptive modeling for varying amounts of modality-shared and modality-specific information.
5
Conclusion
Missing arbitrary modalities at inference time is expected in real-world applications, degrading the classification performance of multi-modal models. In this work, we present two multi-modal co-learning methods to improve the classification robustness under inference scenarios with missing arbitrary modalities. Our methods learn modality-shared and -specific feature spaces through feature-based co-learning, while promoting predictive robustness via decisionlevel knowledge distillation. Evaluated on two benchmarks, our methods consistently outperform (with significant differences) all competing approaches on one dataset, and achieve competitive (significantly similar) results in the other. The evidence highlights the potential and synergies of co-learning and knowledge distillation for enhancing the robustness of multi-modal classification models. Acknowledgments. Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 414984028 – SFB 1404 FONDA.
References 1. Black, S., Souvenir, R.: Multi-view classification using hybrid fusion and mutual distillation. In: Proceedings of the IEEE/CVF WACV. pp. 270–280 (2024) 2. Choi, J.H., Lee, J.S.: EmbraceNet: A robust deep learning architecture for multimodal classification. Information Fusion 51, 259–270 (2019) 3. Cicchetti, G., Grassucci, E., Sigillo, L., Comminiello, D.: Gramian multimodal representation learning and alignment. In: ICLR (2025) 4. Dong, H., Nejjar, I., Sun, H., Chatzi, E., Fink, O.: SimMMDG: A simple and effective framework for multi-modal domain generalization. NeurIPS 36, 78674– 78695 (2023) 5. Hoffman, J., Gupta, S., Darrell, T.: Learning with side information through modality hallucination. In: Proceedings of the IEEE CVPR. pp. 826–834 (2016) 6. Ienco, D., Dantas, C.F.: DisCoM-KD: Cross-modal knowledge distillation via disentanglement representation and adversarial learning. In: The 35th BMVC (2024) 7. Li, S., Chen, C., Han, J.: SimMLM: A simple framework for multi-modal learning with missing modality. In: Proceedings of the IEEE ICCV. pp. 24068–24077 (2025)
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification
15
8. Lin, R., Hu, H.: MissModal: Increasing robustness to missing modality in multimodal sentiment analysis. Transactions of the Association for Computational Linguistics 11, 1686–1702 (2023) 9. Ma, H., Zhang, Q., Zhang, C., Wu, B., Fu, H., Zhou, J.T., Hu, Q.: Calibrating multimodal learning. In: ICML. pp. 23429–23450 (2023) 10. Ma, M., Ren, J., Zhao, L., Testuggine, D., Peng, X.: Are multimodal transformers robust to missing modality? In: Proceedings of the IEEE/CVF CVPR. pp. 18177– 18186 (2022) 11. Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: SMIL: Multimodal learning with severely missing modality. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 2302–2310 (2021) 12. McKinzie, B., Shankar, V., Cheng, J.Y., Yang, Y., Shlens, J., Toshev, A.T.: Robustness in multimodal learning under train-test modality mismatch. In: ICML. pp. 24291–24303 (2023) 13. Mena, F., Arenas, D., Dengel, A.: Missing data as augmentation in the Earth observation domain: A multi-view learning approach. Neurocomputing 638 (2025) 14. Mena, F., Ienco, D., Cassio, D.F., Interdonato, R., Dengel, A.: Multi-sensor model for Earth observation robust to missing data via sensor dropout and mutual distillation. IEEE Access 13, 83930 – 83943 (2025) 15. Mena, F., Ienco, D., Dantas, C.F., Interdonato, R., Dengel, A.: Multi-modal colearning for Earth observation: Enhancing single-modality models via modality collaboration. Machine Learning 114(12), 279 (2025) 16. Neverova, N., Wolf, C., Taylor, G., Nebout, F.: ModDrop: Adaptive multi-modal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(8), 1692–1706 (2015) 17. Rahate, A., Walambe, R., Ramanna, S., Kotecha, K.: Multimodal co-learning: Challenges, applications with datasets, recent advances and future directions. Information Fusion 81, 203–239 (2022) 18. Shen, H., Li, X., Cheng, Q., Zeng, C., Yang, G., Li, H., Zhang, L.: Missing information reconstruction of remote sensing data: A technical review. IEEE Geoscience and Remote Sensing Magazine 3(3), 61–85 (2015) 19. Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., Carneiro, G.: Multi-modal learning with missing modality via shared-specific feature modelling. In: Proceedings of the IEEE/CVF CVPR. pp. 15878–15887 (2023) 20. Woo, S., Lee, S., Park, Y., Nugroho, M.A., Kim, C.: Towards good practices for missing modality robust action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 2776–2784 (2023) 21. Wu, R., Wang, H., Chen, H.T., Carneiro, G.: Deep multimodal learning with missing modality: A survey. Transactions on Machine Learning Research (2026) 22. Yan, X., Hu, S., Mao, Y., Ye, Y., Yu, H.: Deep multi-view learning methods: A review. Neurocomputing 448, 106–129 (2021) 23. Yuan, X., Lin, Z., Kuen, J., Zhang, J., Wang, Y., Maire, M., Kale, A., Faieta, B.: Multimodal contrastive training for visual representation learning. In: Proceedings of the IEEE/CVF CVPR. pp. 6995–7004 (2021) 24. Zadeh, A., Liang, P.P., Morency, L.P.: Foundations of multimodal co-learning. Information Fusion 64, 188–193 (2020) 25. Zhang, X., Yoon, J., Bansal, M., Yao, H.: Multimodal representation learning by alternating unimodal adaptation. In: Proceedings of the IEEE/CVF CVPR. pp. 27456–27466 (2024) 26. Zhang, Y., Xiang, T., Hospedales, T.M., Lu, H.: Deep mutual learning. In: Proceedings of the IEEE CVPR. pp. 4320–4328 (2018)