Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study
Hao Dong1
arXiv:2605.06643v1 [cs.CV] 7 May 2026
1
Hongzhao Li2∗ Shupan Li2 Muhammad Haris Khan3 Eleni Chatzi1 Olga Fink4
ETH Zürich
2
Zhengzhou University
3
MBZUAI
4
EPFL
Abstract Despite the growing popularity of Multimodal Domain Generalization (MMDG) for enhancing model robustness, it remains unclear whether reported performance gains reflect genuine algorithmic progress or are artifacts of inconsistent evaluation protocols. Current research is fragmented, with studies varying significantly across datasets, modality configurations, and experimental settings. Furthermore, existing benchmarks focus predominantly on action recognition, often neglecting critical real-world challenges such as input corruptions, missing modalities, and model trustworthiness. This lack of standardization obscures a reliable assessment of the field’s advancement. To address this issue, we introduce MMDG-Bench, the first unified and comprehensive benchmark for MMDG, which standardizes evaluation across six datasets spanning three diverse tasks: action recognition, mechanical fault diagnosis, and sentiment analysis. MMDG-Bench encompasses six modality combinations, nine representative methods, and multiple evaluation settings. Beyond standard accuracy, it systematically assesses corruption robustness, missing-modality generalization, misclassification detection, and outof-distribution detection. With 7, 402 neural networks trained in total across 95 unique cross-domain tasks, MMDG-Bench yields five key findings: (1) under fair comparisons, recent specialized MMDG methods offer only marginal improvements over ERM baseline; (2) no single method consistently outperforms others across datasets or modality combinations; (3) a substantial gap to upper-bound performance persists, indicating that MMDG remains far from solved; (4) trimodal fusion does not consistently outperform the strongest bimodal configurations; and (5) all evaluated methods exhibit significant degradation under corruption and missing-modality scenarios, with some methods further compromising model trustworthiness. We release MMDG-Bench to enable more rigorous, reproducible, and directly comparable evaluation, addressing current limitations in evaluation practices and providing a stronger foundation for future progress in multimodal domain generalization.1
1
Introduction
Machine learning (ML) models often suffer substantial performance degradation when deployed in dynamic real-world environments due to distribution shifts between training and testing data [40]. Consequently, generalizing to unseen domains has become a central challenge for building reliable ML systems. Multimodal learning, which integrates complementary signals such as video, audio, and optical flow, is widely regarded as a promising approach to improve robustness. While multimodal models achieve strong in-distribution performance across applications including egocentric action ∗ Corresponding author. 1 https://github.com/lihongzhao99/MMDG_Benchmark
Preprint.
MMDG-Bench Methods
ERM, RNA-Net, SimMMDG, MOOSA, CMRF, NEL, JAT, MBCD, GMP
Key Observations Evaluations
Tasks
Modalities
Scale
Datasets
Action Recognition, Fault Diagnosis, Sentiment Analysis
Limited gains of MMDG methods.
Standard Accuracy, Corruption Robustness, Missing-modality, MisD & OOD Detection
No single method consistently dominates. Large gap to upper bound.
Video, Optical Flow, Audio, Text, Vibration, Acoustic
Trimodal fusion does not always win. 7,402 Trained Models, 95 Cross-Domain Tasks
EPIC-Kitchens, HAC, HUST motor, MOSI, MOSEI, SIMS
Severe degradation under realistic challenges.
Figure 1: An overview of the MMDG-Bench and a summary of our key observations. recognition [6], mechanical fault diagnosis [15, 14], and affective computing [47, 48, 46], they remain brittle under domain shifts caused by environmental changes, operating conditions, or cultural variations. Moreover, multimodal systems introduce unique challenges such as modality imbalance, unreliable fusion, and sensitivity to missing or corrupted inputs [11, 12]. These challenges have driven increasing interest in multimodal domain generalization (MMDG), with a growing body of work proposing specialized methods that report consistent empirical gains [37, 11, 8, 12, 49, 30, 45, 29]. Despite this apparent progress, it remains unclear to what extent current MMDG methods yield genuine improvements in cross-domain generalization, as opposed to benefiting from inconsistent evaluation protocols. In unimodal domain generalization, DomainBed [17] revealed that carefully tuned empirical risk minimization (ERM) can match or outperform many specialized methods, fundamentally reshaping the field’s understanding of progress. In contrast, MMDG lacks a comparable, rigorous benchmark. Existing evaluations vary widely in datasets, modality configurations, training protocols, and metrics, often focusing narrowly on action recognition while overlooking realistic challenges such as missing modalities, input corruptions, and model trustworthiness. Consequently, this lack of standardization hinders reliable assessment and raises a fundamental question: are we measuring genuine progress, or simply overfitting to biased evaluation protocols? To answer this question, we introduce MMDG-Bench, a comprehensive and standardized benchmark for evaluating multimodal domain generalization (Figure 1). MMDG-Bench unifies evaluation across six datasets spanning three tasks: egocentric action recognition (EPIC-Kitchens [6], HAC [11]), mechanical fault diagnosis (HUST Motor [51]), and multimodal sentiment analysis (CMU-MOSI [47], CMU-MOSEI [48], CH-SIMS [46]). It covers six modality combinations and evaluates nine representative methods across 95 cross-domain tasks under both multi-source and single-source settings. Beyond standard accuracy, we systematically assess corruption robustness, missing-modality generalization, misclassification detection, and out-of-distribution (OOD) detection, capturing both predictive performance and model reliability. To ensure fair comparison, we standardize data splits, hyperparameter search, optimization protocols, and model selection criteria. With 7, 402 neural networks trained in total, MMDG-Bench provides a comprehensive evaluation and yields critical insights to guide future research: • Under fair evaluation, specialized MMDG methods offer only marginal gains over strong baselines, with ERM frequently matching or outperforming recent approaches. • No single method consistently dominates across datasets or modality configurations. • A substantial gap relative to the Oracle model remains, confirming that MMDG is far from solved. • Trimodal fusion does not consistently surpass the strongest bimodal configurations, challenging the assumption that additional modalities inherently improve generalization. • All methods remain highly vulnerable to corruptions and missing modalities, with some degrading model trustworthiness despite improving raw accuracy. These results suggest that progress in MMDG may be partially overestimated due to inconsistencies in evaluation protocols, underscoring the need for rigorous and standardized benchmarking. 2
2
A Comprehensive Benchmark for Multimodal Domain Generalization
This section outlines the design and scope of MMDG-Bench. We first formalize the relevant MMDG paradigms (Sec. 2.1), then describe the representative methods included (Sec. 2.2), and finally detail the datasets, modality configurations, backbone architectures, evaluation protocols, and hyperparameter search procedures utilized (Sec. 2.3). 2.1
Multimodal Domain Generalization Paradigms
Let M = {m1 , . . . , mK } denote a set of K modalities (e.g., video, audio, optical flow). A multimodal sample (xm1 , . . . , xmK , y) is drawn from a joint distribution PD associated with domain D, where xmk represents the input from modality mk , and y ∈ Y is the corresponding label. s Definition 1 (Multi-source MMDG). Given Ns labeled source domains {Dis }N i=1 sharing a common label space and modality set, multi-source MMDG seeks to learn a model f : X m1 ×· · ·×X mK → Y that generalizes effectively to an unseen target domain Dt , without access to any target-domain data during training. Definition 2 (Single-source MMDG). Given a single labeled source domain Ds and an unseen target domain Dt sharing the same label space and modality set, single-source MMDG seeks to train a model that transfers robustly from Ds to Dt without target-domain access during training. Definition 3 (Corruption Robustness). Given a source-trained MMDG model, corruption robustness evaluates performance when one or more target-domain modalities undergo realistic perturbations (e.g., audio wind noise, video defocus blur). It is quantified by the performance degradation between clean and corrupted target conditions. Definition 4 (Missing-modality Generalization). Given a source-trained MMDG model, this setting measures generalization when modalities present during training are absent during target-domain inference, reflecting real-world scenarios such as sensor failures or incomplete observations. 2.2
Multimodal Domain Generalization Methods
MMDG-Bench evaluates nine representative MMDG methods alongside an Oracle reference. ERM [41] serves as our foundational baseline, pooling all source domains to minimize empirical risk without explicit MMDG objectives. RNA-Net [37] aligns the average feature norms across modalities using a Relative Norm Alignment objective, mitigating modality-induced domain bias without requiring domain annotations. SimMMDG [11] decomposes representations into modality-shared and modality-specific components. It uses supervised contrastive learning to extract domain-invariant shared features and incorporates a cross-modal translation module to improve missing-modality robustness. MOOSA [8] utilizes masked cross-modal translation and multimodal jigsaw puzzles as selfsupervised auxiliary tasks, combined with entropy-guided modality balancing. Though designed for open-set MMDG, it remains highly competitive in standard closed-set settings. CMRF [12] addresses modality competition and inconsistent unimodal flatness in sharpness-aware minimization. It flattens the cross-modal representation landscape by interpolating between modalityspecific minima, followed by feature distillation into individual modality branches. NEL [49] mitigates representation polarization, where one modality dominates the shared embedding space, via a nonpolarized learning objective that encourages balanced, domain-invariant multimodal representations. JAT [30] performs adversarial training using gradient reversal layers on both modality-specific and fused representations, enforcing domain invariance across multiple representation levels. MBCD [45] observes that asynchronous modality convergence limits conventional weight averaging and introduces a collaborative distillation framework utilizing adaptive modality dropout, gradient consistency regularization, and an EMA teacher for cross-modal knowledge transfer. GMP [29] revisits gradient modulation under domain shift by decomposing modality gradients into classification-oriented and domain-invariant components. By dynamically modulating and projecting these gradients based on semantic and domain confidence, it resolves optimization conflicts. 3
Action Recognition (Video+Flow+Audio)
Training:
Sentiment Analysis
Fault Diagnosis
(Video+Text+Audio)
Training:
(Vibration+Acoustic)
Training: 中文：你说的那些 都是我的业余爱好
Testing:
Testing:
Testing: ENGLISH: BUT IT WAS A GOOD MOVIE
Figure 2: Illustration of three core tasks included in the MMDG-Bench. Finally, our Oracle model is trained directly on target-domain data. While not a valid domain generalization method, it provides an empirical performance ceiling to quantify the remaining gap between current MMDG approaches and ideal target-domain performance. 2.3
Experimental Setups
Datasets. MMDG-Bench unifies six datasets across three task families for diverse evaluation (Figure 2). For action recognition, we include EPIC-Kitchens [6] (eight classes across three kitchen environments) and HAC [11] (seven classes performed by humans, animals, and cartoons). Both provide video (V), audio (A), and optical flow (F). For mechanical fault diagnosis, we adopt HUST motor [51], comprising four operating-condition domains with vibration and acoustic signals. For sentiment analysis, we evaluate CMU-MOSI [47], CMU-MOSEI [48], and CH-SIMS [46] (video, audio, text); each acts as a distinct domain for cross-dataset MMDG. Detailed statistics, preprocessing, and splits are in the Appendix C. Modality combinations. We assess six modality configurations: four for action recognition (V+A, V+F, A+F, V+A+F), one for fault diagnosis (vibration+acoustic), and one for sentiment analysis (video+audio+text), enabling systematic evaluation of both bimodal and trimodal fusion. Backbone architectures. For action recognition, we build on MMAction2 [5]: video via Kinetics400 pretrained SlowFast [13], audio via VGGSound pretrained ResNet-18 [20], and optical flow via a Kinetics-initialized SlowFast slow-only pathway. For fault diagnosis, we employ a four-layer 1D CNN for vibration and acoustic signals [51]. For sentiment analysis [19], we extract 768-dimensional text embeddings via pretrained BERT [7], audio features via LibROSA [34], and visual facial features via OpenFace 2.0 [2], fused by a Transformer encoder [42]. Evaluation protocols. Multi-source MMDG follows a leave-one-domain-out protocol, while singlesource evaluates all source-target pairs. For sentiment analysis, we report binary accuracy (ACC2), F1 score, and mean absolute error (MAE). To ensure fair comparisons, all methods use identical data splits, optimizers, and training-domain validation for model selection [17]. Hyperparameter search. For each algorithm-dataset pair, we evaluate the default hyperparameters alongside 10 random-search trials (detailed in the Appendix D). The optimal configuration, selected via training-domain validation, is retrained with two additional random seeds to mitigate variance from random initialization and stochastic optimization, and the final performance is reported as the average across all seeds to provide a more reliable estimate. This rigorous protocol requires training 7, 402 neural networks, making MMDG-Bench the most comprehensive MMDG benchmark studies to date.
3
Multimodal Domain Generalization Under Fair Comparison
Experimental setup. This section examines whether recent MMDG algorithms still outperform strong baselines once major confounding factors are removed. To ensure a fair and rigorous comparison, we standardize all key pipeline components, including data splits, batch sizes, optimizers, and model selection strategies. All methods are selected using training-domain validation, thereby isolating algorithmic contributions rather than evaluation artifacts. 4
Table 1: Multimodal multi-source DG with different modality combinations on EPIC-Kitchens and HAC datasets for action recognition task. Modality Method ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
EPIC-Kitchens dataset
HAC dataset
Video Audio Flow D2, D3 → D1 D1, D3 → D2 D1, D2 → D3 Mean A, C → H H, C → A H, A → C Mean
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
57.47 57.24 58.62 59.31 57.01 54.63 57.98 59.38 57.62 60.23
61.20 60.40 66.40 65.33 69.47 66.75 66.82 69.60 65.39 76.13
60.68 60.47 65.30 66.63 64.37 62.55 64.14 65.63 64.88 76.80
59.78 59.37 63.44 63.76 63.62 61.31 62.98 64.87 62.63 71.05
75.91 75.20 78.59 79.38 77.94 76.33 78.16 78.12 77.36 92.75
77.48 77.48 78.04 78.70 78.26 76.42 77.99 78.91 76.47 97.16
53.40 53.58 55.79 54.78 51.84 51.07 53.11 53.49 52.33 88.53
68.93 68.75 70.81 70.95 69.35 67.94 69.75 70.17 68.72 92.81
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
59.77 60.00 60.69 61.84 61.61 59.00 61.88 63.36 60.37 65.52
66.13 67.47 69.33 69.20 69.33 67.02 68.79 71.06 67.21 80.00
62.73 64.58 64.07 64.89 65.81 63.99 65.82 67.18 65.82 81.21
62.88 64.02 64.70 65.31 65.58 63.34 65.50 67.20 64.47 75.58
76.93 77.58 78.95 80.46 81.47 80.29 78.39 81.39 77.92 93.48
77.59 76.71 75.94 76.71 76.38 76.45 77.38 77.08 76.35 96.59
49.82 52.85 54.60 56.71 52.30 51.16 52.17 53.67 52.56 85.78
68.11 69.05 69.83 71.29 70.05 69.30 69.31 70.71 68.94 91.95
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
52.18 52.41 55.86 58.16 53.56 56.24 56.83 56.78 55.38 59.77
61.47 59.47 69.20 68.27 68.40 63.33 65.26 66.57 64.92 74.13
58.31 62.53 63.04 62.42 61.81 61.09 62.17 65.36 62.77 73.61
57.32 58.14 62.70 62.95 61.26 60.22 61.42 62.90 61.02 69.17
55.66 56.67 58.83 59.55 58.54 58.80 59.32 61.60 57.31 81.52
63.90 64.13 65.45 66.11 65.34 64.08 65.12 66.07 65.13 90.91
47.24 46.42 45.96 46.88 46.42 45.95 45.07 48.71 46.84 68.35
55.60 55.74 56.75 57.51 56.77 56.28 56.50 58.79 56.43 80.26
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
56.78 57.24 63.91 59.77 62.76 60.46 61.38 61.29 59.77 65.52
66.67 66.00 71.47 72.93 70.40 68.48 69.96 71.24 68.39 79.47
65.61 67.97 68.89 69.82 68.17 65.02 66.37 69.50 66.33 78.64
63.02 63.74 68.09 67.51 67.11 64.65 65.90 67.34 64.83 74.54
73.32 73.68 78.15 75.70 79.02 77.26 77.32 79.06 78.26 92.75
76.49 76.16 75.39 78.37 80.35 78.10 77.59 79.21 77.35 96.02
53.86 54.41 54.60 56.43 54.87 55.88 54.88 55.64 53.97 86.24
67.89 68.08 69.38 70.17 71.41 70.41 69.93 71.30 69.86 91.67
Results on action recognition. Table 1 summarizes multi-source MMDG results on EPIC-Kitchens and HAC. Crucially, no single method consistently dominates across datasets, modality combinations, or domain shifts. Performance rankings fluctuate substantially, and gains over strong baselines (e.g., ERM, SimMMDG) are often modest, indicating that reported MMDG progress remains highly context-dependent. Furthermore, the Audio+Flow configuration consistently yields the weakest results across both benchmarks, confirming that video remains the most informative modality for action recognition. Results on fault diagnosis. Table 2 presents results multi-source MMDG on HUST motor. The performance gap across methods is larger than that observed in action recognition. MOOSA achieves the highest mean accuracy (78.23%), followed by GMP and CMRF, significantly outperforming ERM (69.90%). However, the ranking of methods differs from that in action recognition. MBCD performs strongly on EPIC-Kitchens but drops to the lowest rank on HUST, while GMP improves from a mid-tier position in action recognition to second place here. These drastic ranking shifts reveal that current methods fail to generalize reliably across task families, highlighting the risk of drawing broad conclusions from limited benchmark settings. Results on sentiment analysis. Table 3 reports performance multi-source MMDG on sentiment analysis datasets, further highlighting the limitations of current methods. The strongest specialized method (MOOSA, 66.60% ACC2) outperforms ERM (65.63%) by less than one percentage point. In half of the scenarios, ERM matches or exceeds specialized approaches. Moreover, several prominent methods (SimMMDG, MBCD, GMP) underperform ERM on mean ACC2, indicating potential 5
Table 2: Multimodal multi-source DG on HUST motor dataset with vibration and acoustic modalities for fault diagnosis task. Method
D2, D3, D4 → D1
D1, D3, D4 → D2
D1, D2, D4 → D3
D1, D2, D3 → D4
Mean
42.25 43.50 42.33 51.08 47.42 46.97 44.22 42.89 47.45 99.83
83.92 84.58 88.50 93.00 87.92 80.50 82.36 83.72 91.66 99.83
76.25 73.25 82.42 84.92 83.67 76.53 77.36 79.31 89.17 100.00
77.17 79.58 82.08 83.92 80.75 78.19 79.58 70.64 81.61 99.83
69.90 70.23 73.83 78.23 74.94 70.55 70.88 69.14 77.47 99.87
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
Table 3: Multimodal multi-source DG on MOSI, MOSEI, and SIMS datasets with video, audio, and text modalities for sentiment analysis task. Method ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
MOSI, MOSEI → SIMS
MOSI, SIMS → MOSEI
MAE↓
F1↑
ACC2↑
MAE↓
F1↑
ACC2↑
MAE↓
F1↑
ACC2↑
1.82 1.83 1.84 1.89 1.83 1.91 1.85 1.84 1.93 1.32
69.00 66.71 64.39 71.76 72.12 51.79 67.16 58.12 58.68 76.80
63.90 64.55 61.71 66.30 65.21 52.44 64.40 57.84 57.54 76.80
1.02 0.92 1.00 0.96 0.89 0.99 0.98 1.03 1.09 0.58
67.35 67.22 67.87 67.17 67.75 67.55 67.90 67.47 67.32 73.89
67.35 67.22 67.65 66.90 67.74 67.52 67.87 66.83 67.16 73.63
1.42 1.38 1.42 1.43 1.36 1.45 1.42 1.44 1.51 0.95
68.18 66.97 66.13 69.47 69.94 59.67 67.53 62.80 62.00 75.35
65.63 65.89 64.68 66.60 66.48 59.98 66.14 62.34 62.35 75.22
Mean
negative transfer in text-centric tasks. Moreover, most methods perform poorly on regression tasks, as reflected by high MAE. Ultimately, these results show that current MMDG techniques are highly task-dependent and lack broad cross-domain robustness. Single-source DG. Single-source DG results largely reinforce the trends observed in the multi-source setting. On EPIC-Kitchens (Table 4 and Table 8), MBCD achieves the best average performance across modality combinations, with SimMMDG and MOOSA closely following. On HAC (Table 9), SimMMDG leads in the trimodal V+A+F setting (63.60%), while MBCD remains highly competitive (63.53%). HUST Motor (Table 10) provides a particularly challenging evaluation, where limiting training to a single source domain substantially reduces performance for all methods. In severe transfer scenarios (e.g., D1 → D4), accuracy declines sharply to 1.75% − 18.14%, indicating that existing methods depend heavily on source-domain diversity. This suggests that much of the improvement in multi-source DG may arise from broader source coverage rather than fundamental algorithmic advances. For sentiment analysis (Table 11), SimMMDG achieves the strongest average classification performance (F1 and ACC2), while CMRF performs best on MAE. Trimodal fusion does not consistently improve generalization. Multimodal learning is often assumed to improve robustness by incorporating additional modalities. However, the trimodal (V+A+F) results in Table 1 present a more complex picture. On HAC, V+A+F outperforms V+F in only five of nine methods. For several approaches, including ERM, RNA-Net, SimMMDG, and MOOSA, adding a third modality yields minimal benefit or even degrades performance (e.g., MOOSA declines from 71.29% to 70.17%). Methods explicitly designed to address modality competition, such as CMRF, MBCD, and GMP, demonstrate more consistent gains from trimodal integration (+1.36%, +0.59%, +0.92%, respectively), supporting the view that modality competition is a key optimization bottleneck. Nevertheless, current solutions remain only partially effective and fail to deliver substantial, reliable improvements across datasets. Massive gap to Oracle model. Across all datasets, Oracle results reveal a substantial gap between current MMDG performance and achievable target-domain accuracy. For example, on HAC (V+A), the Oracle reaches 92.81% mean accuracy, surpassing the best-performing method (MOOSA, 70.95%) by nearly 22 percentage points. These results demonstrate that MMDG remains an open and challenging problem and highlight the need for fundamentally new approaches to close this large generalization gap. 6
Table 4: Multimodal single-source DG with video and audio on EPIC-Kitchens dataset. Source: D1 Method ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
ERM RNA-Net SimMMDG MOOSA CMRF NEL JAT MBCD GMP
Source: D2
D1→ D3
D2 → D1
D2 → D3
D3→ D1
D3→ D2
Mean
51.07 52.53 53.33 53.60 58.67 54.66 55.32 56.22 53.17 76.13
54.72 51.85 51.54 51.23 51.33 54.07 50.08 55.30 49.82 76.80
43.45 51.03 51.72 47.82 49.66 47.81 50.12 53.41 48.97 60.23
55.44 56.26 60.16 61.91 62.01 59.13 59.23 61.17 59.65 76.80
46.67 53.79 55.63 56.55 50.11 48.50 50.18 53.64 49.81 60.23
56.13 55.60 58.93 58.80 57.73 57.51 56.22 62.26 57.33 76.13
51.25 53.51 55.22 54.98 54.92 53.61 53.52 57.00 53.12 71.05
Wind on audio: accuracy per task (vs. clean V+A) +1.52 -3.09 -0.73 +0.58 -3.97 -4.22 +1.37 -1.55 +0.46 -1.73 -2.99 +0.00 -1.88 -6.63 -4.14 -0.50 -2.46 -4.87 -1.74 -3.72 -3.29 -0.19 -2.92 -1.87 -0.88 -1.55 -3.64 A,C H
Source: D3
D1→ D2
H,C A
H,A C
ERM RNA-Net 4 SimMMDG 2 MOOSA 0 CMRF NEL 2 JAT 4 MBCD 6 GMP 6
Defocus on video: accuracy per task (vs. clean V+A) -10.37 -4.41 -11.58 -13.27 -3.64 -8.91 -17.09 -9.50 -11.86 -11.25 -8.61 -6.34 -15.65 -7.84 -6.25 -15.59 -7.21 -10.78 -16.81 -9.33 -10.36 -12.24 -6.26 -6.13 -12.63 -5.52 -5.76 A,C H
H,C A
H,A C
4 6 8 10 12 14 16 18
Figure 3: Multimodal multi-source DG with corruptions on HAC dataset. Values show the change relative to the clean Video+Audio setting. Detailed results are in Table 12.
Takeaways 1. Under standardized and fair evaluation, specialized MMDG methods yield only marginal improvements over strong baselines. 2. No single method consistently dominates across different datasets, modality combinations, or task families. 3. Trimodal fusion delivers inconsistent gains; while methods targeting modality competition benefit more reliably, current solutions remain incomplete. 4. A substantial performance gap relative to the Oracle persists across all datasets, confirming that MMDG remains far from a solved problem.
4
Robustness under Corruptions and Missing Modalities
Real-world deployments frequently expose multimodal systems to corrupted inputs and missing modalities, yet these critical scenarios remain largely underexplored in MMDG research. To evaluate robustness under realistic sensor failures, we adopt two representative corruptions commonly studied in the literature [9]: wind noise in the audio stream and defocus blur in the video stream. We further assess missing-modality generalization by removing either video or audio during inference. Robustness under corruptions. Figure 3 reports multi-source DG performance on HAC under both corruptions, with subscripts indicating deviations from the clean V+A baseline. Under audio corruption, degradation is modest but widespread: all methods except SimMMDG decline by 0.77 − 4.22 percentage points. Conversely, video corruption proves substantially more severe, causing accuracy drops of 7.97 − 12.82 points. Crucially, performance rankings under corruption deviate markedly from clean-data rankings: MOOSA rises to first place, while SimMMDG drops from second to seventh. This rank inversion yields a critical takeaway: clean benchmark performance does not reliably predict deployment robustness under corruption. This suggests that methods optimized for clean-domain alignment may overfit to modality-specific statistics, making them brittle when modality quality degrades. Notably, the most robust methods under defocus blur all incorporate explicit modality-balancing or competition-aware objectives, suggesting that these strategies inherently improve corruption robustness. 7
Table 5: Multimodal misclassification detection on HAC with video and audio modalities. A, C → H
Method
H, C → A
H, A → C
Mean
AURC↓ AUROC↑ FPR95↓ AURC↓ AUROC↑ FPR95↓ AURC↓ AUROC↑ FPR95↓ AURC↓ AUROC↑ FPR95↓ ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29]
ERM RNA-Net SimMMDG MOOSA CMRF NEL JAT MBCD GMP
75.02 84.62 58.94 63.00 69.36 64.43 62.33 67.54 66.50
84.62 82.95 86.06 85.18 85.89 85.34 85.96 85.12 85.83
73.95 73.84 68.01 61.19 66.88 68.30 63.65 63.80 67.49
73.26 75.73 67.61 65.33 83.75 73.29 68.46 79.35 69.83
Video only: accuracy per task (vs. V+A) +2.97 -2.76 -5.70 +2.59 -2.20 -3.86 +2.23 -1.00 -2.21 -0.65 -2.21 -1.29 +1.87 -6.85 -4.60 +3.45 -3.28 -3.98 +1.13 -3.67 -5.32 +2.06 -2.17 -0.87 +0.81 -0.98 -1.27
A,C H
H,C A
H,A C
84.67 83.13 85.19 84.92 81.93 83.78 84.47 82.24 84.97
59.31 63.73 68.84 57.51 77.21 59.44 66.37 69.87 58.39
ERM RNA-Net 4 SimMMDG 2 MOOSA 0 CMRF NEL 2 JAT 4 MBCD 6 GMP 6
271.82 266.91 237.51 264.25 359.01 289.59 268.85 270.52 304.57
74.22 74.54 76.42 73.14 69.07 74.62 74.18 74.08 69.87
85.40 81.78 84.20 81.91 86.43 79.89 84.22 84.35 86.47
140.03 142.42 121.35 130.86 170.71 142.44 133.21 139.14 146.97
81.17 80.21 82.56 81.08 78.96 81.25 81.54 80.48 80.22
72.89 73.12 73.68 66.87 76.84 69.21 71.41 72.67 70.78
Audio only: accuracy per task (vs. V+A) -49.95 -40.06 -31.16 -47.30 -46.13 -35.29 -47.95 -50.45 -26.56 -52.34 -39.63 -32.72 -45.93 -39.74 -23.81 -53.75 -47.98 -30.06 -50.65 -44.15 -31.46 -46.79 -40.95 -24.80 -47.95 -42.27 -28.47
A,C H
H,C A
H,A C
20 25 30 35 40 45 50 55
Figure 4: Multimodal multi-source DG with missing modalities on HAC dataset. Values show the change relative to the full Video+Audio setting. Detailed results are in Table 13. Missing modalities. Figure 4 evaluates robustness when a modality is unavailable at inference. We observe a striking asymmetry: removing audio causes only minor degradation (0.32 − 3.20 point drops), whereas removing video results in severe failures (36.50 − 43.93 point drops). For example, SimMMDG loses merely 0.33 points when transitioning from V+A to V-only, but drops by 41.66 points under A-only inference. Furthermore, in the A, C → H transfer setting, removing audio actually improves performance in most cases. This reveals a modality hierarchy under domain shift, where dominant modalities (e.g., video) govern robustness, while auxiliary modalities can introduce instability when not properly integrated. Takeaways 1. Clean benchmark performance does not reliably predict real-world robustness: methods that perform strongly under standard DG settings can degrade substantially under realistic input corruptions. 2. Video remains the dominant modality in action recognition, while audio often provides only auxiliary benefits and can even reduce performance when multimodal integration is not carefully optimized. 3. Current MMDG methods remain highly vulnerable to sensor failures, highlighting the urgent need for future approaches that explicitly address corruption robustness and missing-modality resilience.
5
Trustworthiness: Misclassification and Out-of-Distribution Detection
Beyond classification accuracy, multimodal systems are also expected to identify when their predictions are likely to be incorrect (misclassification detection [32]) and to detect inputs that are semantically novel (out-of-distribution detection [31, 33]). This is the first standardized evaluation of trustworthiness in MMDG. We evaluate both capabilities on HAC using the V+A combination. For OOD detection, HAC serves as the in-distribution dataset, while EPIC-Kitchens is used as the OOD dataset. For misclassification detection (MisD), we report AURC (Area Under the Risk-Coverage Curve), AUROC, and FPR95 (false positive rate at 95% true positive rate). For OOD detection, we report AUROC and FPR95. Misclassification detection. Table 5 presents the MisD results. SimMMDG achieves the strongest overall performance (best mean AURC and AUROC), suggesting that its explicit decomposition of modality-shared and modality-specific features yields better-calibrated uncertainty estimates. 8
Table 6: Multimodal out-of-distribution detection with video and audio modalities, where HAC is the ID dataset and EPIC-Kitchens as OOD dataset. Method ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29]
A, C → H
H, C → A
H, A → C
Mean
AUROC↑
FPR95↓
AUROC↑
FPR95↓
AUROC↑
FPR95↓
AUROC↑
FPR95↓
70.63 68.21 77.19 67.23 77.61 69.32 72.71 76.44 71.54
62.00 67.63 53.42 75.13 52.34 65.81 61.82 53.46 61.48
53.64 57.68 73.12 64.65 60.04 63.20 63.62 65.87 55.37
88.08 83.77 62.14 69.43 79.58 78.62 70.24 68.16 86.46
46.05 38.56 35.18 34.29 40.70 37.21 41.20 34.11 39.28
90.53 97.61 96.50 98.25 95.59 96.65 95.33 98.54 96.79
56.77 54.82 61.83 55.39 59.45 56.58 59.18 58.81 55.40
80.20 83.00 70.69 80.94 75.84 80.36 75.80 73.39 81.58
Meanwhile, MOOSA achieves the best mean FPR95, indicating that its self-supervised pretext tasks generate confidence scores that effectively separate correct from incorrect predictions. In contrast, while CMRF maintains competitive classification accuracy, it ranks last across all MisD metrics. This discrepancy exposes a critical disconnect between predictive accuracy and model trustworthiness, a vulnerability largely overlooked in prior MMDG research. Out-of-distribution detection. Table 6 reports the OOD detection results, where SimMMDG again achieves the strongest overall performance. Interestingly, CMRF, which ranks last in MisD, achieves the second-highest mean OOD AUROC. This confirms that these two trustworthiness dimensions are non-redundant: mechanisms that improve OOD separation can simultaneously degrade confidence calibration for in-distribution errors. The inverse pattern is also observed: MOOSA attains the best MisD FPR95 but falls to the bottom in OOD AUROC. Furthermore, despite its exceptional classification accuracy across EPIC-Kitchens and HAC, MBCD yields only moderately on OOD AUROC and MisD metrics. Ultimately, these findings demonstrate that high predictive accuracy does not guarantee model trustworthiness, and even trust-oriented metrics may favor different methods depending on whether the focus is on misclassification calibration or OOD detection. Takeaways 1. Predictive accuracy does not guarantee model trustworthiness. High classification performance does not inherently translate to reliable confidence estimation. 2. Misclassification detection and OOD detection are non-redundant capabilities. A model’s ability to separate in-distribution from OOD samples does not predict its ability to properly calibrate confidence on in-distribution errors. 3. Current MMDG methods exhibit substantial limitations in uncertainty calibration, highlighting the need for future approaches that jointly optimize accuracy and trustworthiness.
6
Conclusion
We introduce MMDG-Bench, the first unified benchmark for multimodal domain generalization, providing standardized evaluations across six datasets, three task families, six modality configurations, and nine representative methods in both multi- and single-source settings. Beyond clean-domain accuracy, MMDG-Bench systematically assesses corruption robustness, missing-modality generalization, misclassification detection, and out-of-distribution detection to rigorously evaluate real-world deployment capability. Our large-scale study reveals five key findings: (1) under fair evaluation, specialized methods yield only marginal gains over strong baselines; (2) no single method consistently dominates across datasets, modalities, or task families; (3) a substantial gap relative to the target-trained Oracle confirms that MMDG is far from solved; (4) trimodal fusion does not reliably outperform the strongest bimodal configurations; and (5) all methods remain highly vulnerable to corruptions and missing modalities, with some degrading model trustworthiness despite clean accuracy gains. Collectively, these results demonstrate that evaluating clean cross-domain performance alone is insufficient. Future MMDG research must prioritize modality competition, corruption resilience, and trustworthy uncertainty estimation as first-class objectives. We hope MMDG-Bench serves as a rigorous, reproducible foundation to drive the development of robust, deployment-ready multimodal systems. 9
References [1] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019. [2] Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. Openface: an open source facial behavior analysis toolkit. In WACV, 2016. [3] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In NeurIPS, 2011. [4] Xin Chen, Huanjie Tao, and Benran Li. Towards robust incomplete multimodal open-set domain generalization with uncertain missing modalities. Knowledge-Based Systems, page 115777, 2026. [5] MMAction2 Contributors. Openmmlab’s next generation video understanding toolbox and benchmark. https://github.com/open-mmlab/mmaction2, 2020. [6] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018. [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019. [8] Hao Dong, Eleni Chatzi, and Olga Fink. Towards multimodal open-set domain generalization and adaptation through self-supervision. In ECCV, 2024. [9] Hao Dong, Eleni Chatzi, and Olga Fink. Towards robust multimodal open-set test-time adaptation via adaptive entropy-aware optimization. In ICLR, 2025. [10] Hao Dong, Moru Liu, Kaiyang Zhou, Eleni Chatzi, Juho Kannala, Cyrill Stachniss, and Olga Fink. Advances in multimodal adaptation and generalization: From traditional approaches to foundation models. arXiv preprint arXiv:2501.18592, 2025. [11] Hao Dong, Ismail Nejjar, Han Sun, Eleni Chatzi, and Olga Fink. SimMMDG: A simple and effective framework for multi-modal domain generalization. In NeurIPS, 2023. [12] Yunfeng Fan, Wenchao Xu, Haozhao Wang, and Song Guo. Cross-modal representation flattening for multi-modal domain generalization. In NeurIPS, 2024. [13] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019. [14] Olga Fink, Ismail Nejjar, Vinay Sharma, Keivan Faghih Niresi, Han Sun, Hao Dong, Chenghao Xu, Amaury Wei, Arthur Bizzi, Raffael Theiler, et al. From physics to machine learning and back: Part ii-learning and observational bias in prognostics and health management (phm). Reliability Engineering & System Safety, page 112376, 2026. [15] Olga Fink, Vinay Sharma, Ismail Nejjar, Leandro Von Krannichfeldt, Sergei Garmaev, Zepeng Zhang, Amaury Wei, Gaetan Frusque, Florent Forest, Mengjie Zhao, et al. From physics to machine learning and back: Part i-learning with inductive biases in prognostics and health management. Reliability Engineering & System Safety, page 112213, 2026. [16] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015. [17] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020. [18] Cagri Gungor and Adriana Kovashka. Integrating audio narrations to strengthen domain generalization in multimodal first-person action recognition. In ICASSP, 2025. [19] Zirun Guo, Tao Jin, Wenlong Xu, Wang Lin, and Yangyang Wu. Bridging the gap for test-time multimodal sentiment analysis. In AAAI, 2025. [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
10
[21] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021. [22] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019. [23] Hai Huang, Yan Xia, Sashuai Zhou, Hanting Wang, Shulei Wang, and Zhou Zhao. Bridging domain generalization to multimodal domain generalization via unified representations. In ICCV, 2025. [24] Hyeonbin Ji, Juyeob Lee, and Eunil Park. Alignment and distillation: A robust framework for multimodal domain generalizable human action recognition. In WACV, 2026. [25] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In ICML, 2021. [26] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In ICML, 2021. [27] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI, 2018. [28] Hongzhao Li, Hao Dong, Hualei Wan, Shupan Li, Mingliang Xu, and Muhammad Haris Khan. Towards multimodal domain generalization with few labels. arXiv preprint arXiv:2602.22917, 2026. [29] Hongzhao Li, Guohao Shen, Shupan Li, Mingliang Xu, and Muhammad Haris Khan. Balancing multimodal domain generalization via gradient modulation and projection. In AAAI, 2026. [30] Hongzhao Li, Hualei Wan, Liangzhi Zhang, Mingyuan Jiu, Shupan Li, Mingliang Xu, and Muhammad Haris Khan. Towards robust multimodal domain generalization via modality-domain joint adversarial training. In Proceedings of the 33rd ACM International Conference on Multimedia, 2025. [31] Shawn Li, Huixian Gong, Hao Dong, Tiankai Yang, Zhengzhong Tu, and Yue Zhao. Dpu: Dynamic prototype updating for multimodal out-of-distribution detection. arXiv preprint arXiv:2411.08227, 2024. [32] Moru Liu, Hao Dong, Olga Fink, and Mario Trapp. Adaptive confidence regularization for multimodal failure detection. arXiv preprint arXiv:2603.02200, 2026. [33] Moru Liu, Hao Dong, Jessica Kelly, Olga Fink, and Mario Trapp. Extremely simple multimodal outlier synthesis for out-of-distribution detection and segmentation. arXiv preprint arXiv:2505.16985, 2025. [34] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, Oriol Nieto, et al. librosa: Audio and music signal analysis in python. SciPy, 2015(18-24):7, 2015. [35] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML, 2013. [36] Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition. In CVPR, 2020. [37] Mirco Planamente, Chiara Plizzari, Emanuele Alberti, and Barbara Caputo. Domain generalization through audio-visual relative norm alignment in first person action recognition. In WACV, 2022. [38] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019. [39] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016. [40] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, 2011. [41] Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 10(5):988–999, 1999. [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
11
[43] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurIPS, 2018. [44] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip S Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE transactions on knowledge and data engineering, 35(8):8052–8072, 2022. [45] Xiaohan Wang, Zhangtao Cheng, Ting Zhong, Leiting Chen, and Fan Zhou. Modality-balanced collaborative distillation for multi-modal domain generalization. In AAAI, 2026. [46] Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In ACL, 2020. [47] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259, 2016. [48] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACL, 2018. [49] Baoqiang Zhang, Kunze Huang, Luyao Luyao, Xiaotong Tu, and Xiaolu Li. Nonpolarized embedding learning in multimodal domain generalization. Neurocomputing, page 131754, 2025. [50] Xingxuan Zhang, Yue He, Renzhe Xu, Han Yu, Zheyan Shen, and Peng Cui. Nico++: Towards better benchmarking for domain generalization. In CVPR, 2023. [51] Chao Zhao, Enrico Zio, and Weiming Shen. Domain generalization for cross-domain fault diagnosis: An application-oriented perspective and a benchmark study. Reliability Engineering & System Safety, 245:109964, 2024. [52] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE transactions on pattern analysis and machine intelligence, 45(4):4396–4415, 2022.
12
A
Related Work
A.1
Domain Generalization
Domain generalization (DG), formalized by [3] and named by [35], aims to learn models that transfer to unseen target distributions using only labeled source data, without target access during training. Comprehensive surveys [52, 44] categorize prior methodologies into four broad families. Domain alignment reduces source-domain feature divergence via moment matching [39], adversarial learning [16], or invariant risk minimization [1], positing that source-invariant representations will generalize to unseen targets. Meta-learning simulates domain shift by partitioning sources into pseudo-train and pseudo-test sets to optimize held-out performance [27]. Data augmentation [43] diversifies the training distribution through adversarial examples, mixup, or generative perturbations to cover potential test-domain shifts. Finally, regularization enforces solution properties conducive to out-of-distribution generalization, such as cross-domain gradient agreement [26] or worst-case group robustness [38]. Despite this methodological diversity, [17] demonstrated that under standardized evaluation, a carefully tuned ERM baseline matches or outperforms prominent DG algorithms across multiple benchmarks. This pivotal finding recentered the field on evaluation rigor, directly motivating our parallel investigation in the multimodal setting. A.2
Multimodal Domain Generalization
Multimodal domain generalization (MMDG) extends DG to inputs comprising heterogeneous modalities (e.g., video, audio, text) [18, 23, 4, 24]. This setting is uniquely challenging because modalities exhibit distinct statistical properties, converge at varying rates [45], and establish spurious crossmodal correlations that fracture under distribution shift [12]. The canonical protocol originated with MM-SADA [36], which defined the cross-kitchen action recognition task for domain adaptation, establishing the de facto MMDG benchmark. For DG specifically, RNA-Net [37] introduced Relative Norm Alignment to rebalance audio-visual feature norms across source domains. SimMMDG [11] subsequently decomposed representations into modality-shared and -specific components, concurrently introducing the Human-Animal-Cartoon (HAC) dataset to stress-test cross-style generalization. MOOSA [8] later extended this approach to open-set MMDG via self-supervised pretext tasks. More recent methods have targeted increasingly specific bottlenecks in the multimodal optimization landscape: CMRF [12] flattens cross-modal representation spaces to address discrepant modality sharpness; NEL [49] mitigates embedding polarization; JAT [30] jointly applies adversarial training across modality and domain axes; MBCD [45] replaces weight averaging with collaborative distillation and adaptive modality dropout; and GMP [29] modulates gradients to resolve cross-modal conflicts. While adjacent domains like semi-supervised MMDG [28] and comprehensive surveys unifying multimodal adaptation and foundation models [10] have recently emerged, no prior work systematically consolidates MMDG evaluation across diverse task families, modality configurations, and robustness axes. MMDG-Bench directly addresses this critical gap. A.3
Domain Generalization Benchmarks
The maturation of the DG field has been largely driven by community benchmarks. DomainBed [17] standardized the evaluation of 14 algorithms across seven image datasets, revealing that prior reported gains largely stemmed from inconsistent evaluation protocols rather than algorithmic innovation. WILDS [25] extended benchmarking to 10 real-world datasets (e.g., satellite imagery, histopathology), demonstrating that substantial performance gaps persist on natural distribution shifts even for methods excelling on synthetic tasks. NICO++ [50] introduced quantitative metrics for covariate and concept shift, showing that prior datasets occupied a narrow shift spectrum, and released a 200,000-image benchmark to expand this scope. Additionally, benchmarks like ImageNet-C [22] and ImageNetR [21] have emerged to target specific failure modes, such as visual corruptions. MMDG-Bench serves as the multimodal analogue to these foundational efforts. It provides a consolidated testbed that rigorously standardizes backbones, data splits, hyperparameters, and model selection across nine MMDG methods, six modality combinations, and six datasets. Furthermore, it introduces systematic evaluation axes for corruption robustness, missing modalities, and trustworthiness, critical dimensions completely absent from prior MMDG evaluations. 13
B
Limitations, Broader Impacts, and Future Work
B.1
Limitations
MMDG-Bench currently focuses on discriminative and regression tasks and does not yet cover other important settings such as multimodal retrieval or generative modeling. Additionally, our robustness evaluation is limited to two representative perturbations; extending this to broader, modality-specific corruption suites and adversarial attacks remains an important direction for future work. B.2
Broader Impacts
Promoting Safe and Reliable AI Deployment: By systematically exposing the vulnerabilities of current multimodal models to real-world noise, missing modalities, and out-of-distribution data, MMDG-Bench incentivizes the development of much safer AI systems. This is particularly crucial for high-stakes domains, such as industrial safety and predictive maintenance, where model failures can lead to physical harm or severe economic loss. Enhancing Model Transparency and Trust: Our findings emphasize that high predictive accuracy does not guarantee reliable confidence estimation. By evaluating misclassification and out-ofdistribution detection, our benchmark encourages the community to build AI systems that "know what they do not know." This transparency is essential for fostering meaningful human-AI collaboration and trust. B.3
Future Work
Based on the comprehensive evaluations and findings from MMDG-Bench, it is evident that MMDG remains far from a solved problem. We identify several critical directions for future research to address the limitations of current approaches: Developing Beyond-Marginal Algorithms: Current specialized MMDG methods offer only marginal improvements over strong baselines like ERM and fail to consistently dominate across diverse datasets or task families. Furthermore, a substantial gap to upper-bound performance persists. Future work must focus on discovering novel training paradigms or architectural innovations that genuinely generalize across task families, rather than overfitting to specific modality combinations or datasets. Addressing Modality Competition and Adaptive Fusion: Our findings indicate that simply adding more modalities, such as through trimodal fusion, produces inconsistent benefits. In tasks like action recognition, dominant modalities (e.g., video) often overshadow auxiliary modalities (e.g., audio), which can even reduce performance when not properly integrated. These results highlight the need for dynamic and adaptive fusion mechanisms that explicitly address modality competition and optimally balance modality contributions based on context. Building Resilience to Real-World Corruptions and Sensor Failures: Clean benchmark performance has proven to be a poor predictor of real-world robustness. Existing methods degrade substantially under realistic input corruptions and exhibit high vulnerability to missing modalities (sensor failures). Future MMDG frameworks must explicitly incorporate corruption robustness and missing-modality resilience into their optimization objectives, moving beyond idealized training environments. Jointly Optimizing Accuracy and Trustworthiness: High predictive accuracy does not inherently translate to reliable confidence estimation. We observed that current models struggle with uncertainty calibration, and that misclassification detection and OOD detection represent non-redundant challenges. Future research should prioritize trustworthy MMDG by jointly optimizing for predictive accuracy and robust uncertainty quantification, ensuring models are safe and reliable in open-world deployments.
C
Introduction of Datasets
We provide detailed information on the datasets included in MMDG-Bench, including action recognition, mechanical fault diagnosis, and sentiment analysis. 14
Human
Animal
Kitchen 1
Cartoon
(a) HAC
Kitchen 2
Kitchen 3
(b) EPIC-Kitchens
Figure 5: Examples from action recognition datasets. Kitchen 3
ns
Figure 6: Examples from fault diagnosis dataset. C.1
Action Recognition
Human-Animal-Cartoon (HAC) [11]. The HAC dataset consists of seven actions (“sleeping,” “watching TV,” “eating,” “drinking,” “swimming,” “running,” and “opening door”) performed by humans, animals, and cartoon characters, forming three distinct domains: Human (H), Animal (A), and Cartoon (C). The dataset contains a total of 3, 381 video clips, including 1, 387 human samples, 906 animal samples, and 1, 088 cartoon samples. Each sample includes video, audio, and pre-computed optical flow modalities. EPIC-Kitchens [36]. Following the experimental protocol of prior work [36], we use a subset of EPIC-Kitchens containing eight actions (“put,” “take,” “open,” “close,” “wash,” “cut,” “mix,” and “pour”) recorded across three different kitchens, which define three domains: D1, D2, and D3. The dataset comprises 10, 094 video clips in total, with 1, 978 samples from D1, 3, 245 from D2, and 4, 871 from D3. Each sample includes video, audio, and pre-computed optical flow modalities. C.2
Mechanical Fault Diagnosis
HUST Motor [51]. HUST Motor is a public motor fault diagnosis dataset that provides synchronized vibration and acoustic signals collected from a Spectra-Quest Mechanical Fault Simulator, distinguishing it from the predominantly vibration-only datasets commonly used in this field. The dataset covers six motor health states: healthy, bearing fault, bowed rotor, broken rotor bars, rotor misalignment, 15
(a) HAC
(b) EPIC-Kitchens
Figure 7: Examples from sentiment analysis datasets.
and voltage unbalance, with all faults artificially introduced to ensure controlled ground-truth labels. Each health condition is recorded under four steady-state rotational speeds (5, 10, 20, and 30 Hz), forming four distinct domains. Both vibration and acoustic signals are sampled at 25.6 kHz, with 163,840 samples collected for each configuration. The combination of complementary modalities, multiple operating conditions, and diverse fault categories makes HUST Motor a valuable benchmark for multimodal domain generalization in fault diagnosis. C.3
Sentiment Analysis
CMU-MOSI [47]. CMU-MOSI is a foundational dataset for English-language multimodal sentiment analysis. It contains 2,199 short opinion video clips collected from YouTube monologue reviews, with each utterance annotated for sentiment intensity on a continuous scale from -3 (highly negative) to +3 (highly positive). The dataset provides three temporally aligned modalities: text (transcribed speech), acoustic features (e.g., pitch and energy), and visual features (e.g., facial expressions and gestures). Despite its relatively modest scale, CMU-MOSI remains a widely used benchmark for sentiment regression and classification, and serves as a standard evaluation dataset for multimodal fusion methods. CMU-MOSEI [48]. CMU-MOSEI extends CMU-MOSI and is one of the largest publicly available datasets for multimodal sentiment and emotion analysis. It includes over 23,500 sentence-level video utterances from more than 1,000 distinct YouTube speakers across diverse topics. Each utterance is annotated with both a sentiment intensity score in [-3, +3] and six Ekman-style emotion categories (happiness, sadness, anger, fear, disgust, and surprise) with corresponding intensity labels. Like CMU-MOSI, it provides temporally aligned text, acoustic, and visual modalities. Its large scale, speaker diversity, and comprehensive annotations make CMU-MOSEI a standard benchmark for multimodal fusion, transfer learning, and generalization research. Since CMU-MOSEI is a broader and larger-scale extension of CMU-MOSI, we only consider the MOSI → MOSEI generalization direction in our experiments. CH-SIMS [46]. CH-SIMS is a Chinese-language multimodal sentiment analysis dataset designed to address limitations of prior datasets that provide only unified multimodal sentiment labels. It consists of 2,281 refined video segments collected from real-world sources such as movies, TV series, and variety shows. In addition to an overall multimodal sentiment label, CH-SIMS provides independent sentiment annotations for each modality—text, audio, and visual—using a five-point scale ranging from negative to positive in [-1, +1]. This modality-specific labeling enables more detailed analysis of inter-modality consistency and disagreement, while also supporting unimodal, multimodal, and multi-task learning research. Since sentiment scales vary across datasets, we formulate sentiment classification as a binary task (negative vs. positive) and normalize regression targets to the range [-3, +3]. 16
Table 7: Hyperparameters, their default values and distributions for random search.
D
Condition
Parameter
Default value
Random distribution
RNA-Net
alpha_RNA
1.0
Uniform(0, 3)
SimMMDG
alpha_trans explore_loss_coeff alpha_contrast
0.1 0.7 3.0
Uniform(0, 1) Uniform(0, 1) Uniform(0, 5)
MOOSA
entropy_min_weight jigsaw_ratio mask_ratio
0.001 1.0 0.3
Uniform(0, 1) Uniform(0, 3) Uniform(0, 1)
CMRF
distill_coef mix_coef
3.0 2.0
Uniform(0, 5) Uniform(0, 5)
NEL
alpha beta temp_s temp_u k
0.7 1/bsz 0.1 0.25 8
Uniform(0, 1) Uniform(0, 1) Uniform(0, 1) Uniform(0, 1) Choice({4, 8, 12, 16})
JAT
alpha_rev alpha_rev2 domain_adv_loss modal_adv_loss cls_loss
0.1 0.3 0.5 0.1 3.0
Uniform(0, 1) Uniform(0, 1) Uniform(0, 1) Uniform(0, 1) Uniform(0, 5)
MBCD
ema_beta kl_mm_coeff kl_um_coeff modality_drop_base
0.999 1.0 1.0 0
Uniform(0.9, 1.0) Uniform(0, 2) Uniform(0, 2) Uniform(0, 1)
BMP
alpha_rev alpha_k alpha_p cls_loss
0.3 0.5 0.1 3.0
Uniform(0, 1) Uniform(0, 1) Uniform(0, 1) Uniform(0, 5)
Hyperparameter Spaces
We list all hyperparameters, their default values, and the corresponding search distributions used in our random hyperparameter sweeps in Table 7.
E
Detailed Experimental Results
We present detailed experimental results for single-source DG (Table 8 to Table 11), as well as under corruption(Table 12) and missing-modality settings (Table 13).
F
Compute Resources
All experiments were conducted on servers equipped with NVIDIA RTX 3090 and RTX 4090 GPUs. Each model was trained using standard deep learning frameworks with GPU acceleration. In total, 7,402 neural networks were trained across 95 cross-domain tasks, reflecting the large computational scale of MMDG-Bench.
17
Table 8: Multimodal single-source DG with different modalities on EPIC-Kitchens dataset. Modality Method
D2 → D3
D3→ D1
D3→ D2
Mean
51.07 52.53 53.33 53.60 58.67 54.66 55.32 56.22 53.17 76.13
54.72 51.85 51.54 51.23 51.33 54.07 50.08 55.30 49.82 76.80
43.45 51.03 51.72 47.82 49.66 47.81 50.12 53.41 48.97 60.23
55.44 56.26 60.16 61.91 62.01 59.13 59.23 61.17 59.65 76.80
46.67 53.79 55.63 56.55 50.11 48.50 50.18 53.64 49.81 60.23
56.13 55.60 58.93 58.80 57.73 57.51 56.22 62.26 57.33 76.13
51.25 53.51 55.22 54.98 54.92 53.61 53.52 57.00 53.12 71.05
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
58.93 56.40 59.07 57.07 58.27 54.75 55.32 56.31 54.83 80.00
55.24 54.93 51.13 50.51 49.79 47.43 48.18 53.18 50.67 81.21
49.43 53.56 56.55 54.25 52.64 52.79 53.16 55.55 51.67 65.52
56.98 58.01 59.14 62.22 60.27 60.95 59.02 62.28 59.19 81.21
55.40 56.78 57.93 54.94 56.09 54.40 55.78 56.55 55.82 65.52
64.40 62.27 64.27 66.00 64.00 63.11 63.55 67.58 64.19 80.00
56.73 56.99 58.01 57.50 56.84 55.57 55.84 58.57 56.06 75.58
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
47.20 50.93 53.47 53.07 50.93 49.91 50.09 52.62 49.83 74.13
49.38 54.00 51.33 54.11 53.59 50.34 52.11 53.73 50.25 73.61
42.53 42.07 47.13 45.75 43.22 44.13 43.29 54.35 44.32 59.77
52.57 54.72 56.06 55.75 52.87 57.46 54.66 52.94 53.88 73.61
47.13 48.51 52.64 54.48 49.89 50.19 50.17 54.25 49.64 59.77
57.33 57.87 63.33 63.07 62.40 60.08 59.89 66.02 58.71 74.13
49.36 51.35 53.99 54.37 52.15 52.02 51.70 55.65 51.10 69.17
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
55.47 59.07 58.27 60.27 59.47 58.40 58.61 60.04 57.38 79.47
52.87 56.06 53.49 57.39 56.37 54.07 54.35 55.91 53.59 78.64
52.64 53.10 51.49 50.57 51.72 49.19 50.16 55.78 50.59 65.52
58.52 60.16 63.35 62.53 61.29 62.25 61.33 64.81 61.55 78.64
55.86 52.64 58.16 61.15 57.01 55.25 56.38 56.78 54.82 65.52
63.60 64.80 70.93 66.27 66.40 66.04 63.24 72.00 65.79 79.47
56.49 57.64 59.28 59.70 58.71 57.53 57.34 60.89 57.29 74.54
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Source: D3
D2 → D1
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
Source: D2
D1→ D3
Audio
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
Source: D1 D1→ D2
Video
Flow
18
Table 9: Multimodal single-source DG with different modalities on HAC dataset. Modality Method
H→ C
A→H
A→C
C→ H
C→ A
Mean
49.36 52.11 45.86 43.38 45.59 46.32 44.15 42.93 48.71 88.53
65.83 67.84 75.34 72.39 73.54 74.90 70.31 71.52 72.43 92.75
50.00 53.13 50.64 49.45 54.96 45.52 45.28 43.35 44.48 88.53
64.67 60.27 69.00 70.87 74.55 69.50 65.51 65.97 64.65 92.75
72.74 71.30 73.18 72.08 71.52 69.31 70.82 69.57 69.13 97.16
61.54 61.76 63.71 62.69 64.49 62.35 60.49 60.34 61.12 92.81
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
65.78 64.90 68.87 67.99 66.78 65.85 66.37 67.88 66.74 96.59
45.31 45.40 43.84 45.31 47.61 41.14 40.91 40.50 41.94 85.78
75.78 72.24 74.33 76.42 77.22 73.61 71.37 79.23 69.21 93.48
48.35 50.09 53.13 54.04 51.93 36.58 41.66 48.77 41.91 85.78
69.79 59.63 71.23 70.37 69.72 69.14 62.86 71.32 63.54 93.48
64.13 65.01 65.12 68.10 66.56 65.12 59.30 62.62 60.49 96.59
61.52 59.54 62.75 63.71 63.30 58.57 57.08 61.72 57.30 91.95
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
57.73 54.86 61.81 59.38 58.06 57.43 56.80 60.15 57.26 90.91
40.35 38.05 40.07 39.98 39.98 34.98 36.80 36.33 37.88 68.35
52.49 50.32 56.02 58.54 58.69 56.16 50.73 58.80 54.37 81.52
39.61 44.21 41.73 44.49 43.38 36.76 37.31 41.69 38.61 68.35
38.57 41.17 41.89 40.88 43.76 38.50 38.55 50.03 40.15 81.52
49.34 49.67 50.00 52.87 45.92 44.44 46.06 51.28 48.17 90.91
46.35 46.38 48.59 49.36 48.30 44.71 44.38 49.71 46.07 80.26
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
68.10 64.35 66.45 66.11 67.99 65.92 65.27 70.42 63.42 96.02
44.67 46.23 45.13 47.98 49.54 43.72 45.58 44.45 48.26 86.24
70.44 67.27 73.90 72.03 69.36 70.22 70.75 77.21 71.18 92.75
50.83 48.99 52.30 52.67 55.70 47.70 45.77 49.63 47.30 86.24
63.30 61.93 70.44 66.33 65.10 65.54 61.06 70.17 63.97 92.75
68.43 65.45 73.40 72.74 65.12 66.15 64.09 69.28 62.51 96.02
60.96 59.04 63.60 62.98 62.13 59.88 58.75 63.53 59.44 91.67
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Source: C
66.67 65.89 68.21 67.99 66.78 68.57 66.84 68.69 67.29 97.16
Audio
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
Source: A
H→ A
Video
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
Source: H Flow
Table 10: Multimodal single-source DG on HUST dataset with vibration and acoustic modalities. Source: D1 Method
Source: D2
Source: D3
Source: D4
Target:
D2
D3
D4
D1
D3
D4
D1
D2
D4
D1
D2
D3
Mean
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
45.08 56.08 51.25 49.50 49.50 57.39 58.20 63.33 58.67 99.83
12.00 12.83 16.00 18.58 17.58 20.92 16.39 23.19 17.36 100.00
4.92 1.75 14.67 11.50 16.67 17.86 10.95 18.14 14.19 99.83
50.00 51.58 44.75 55.92 51.42 53.39 44.19 38.25 50.11 99.83
58.92 52.08 68.83 66.83 60.42 56.94 58.78 66.81 63.78 100.00
50.67 48.75 63.17 58.00 46.33 47.17 57.70 53.94 59.33 99.83
21.42 27.00 24.50 26.67 21.17 24.86 22.06 17.69 17.00 99.83
58.25 62.92 71.75 63.17 51.25 52.00 58.42 66.58 58.03 99.83
80.17 80.92 81.83 82.83 76.50 75.89 80.67 66.72 82.92 99.83
18.00 21.08 17.00 14.67 17.25 18.39 18.17 15.25 17.97 99.83
37.83 40.83 48.00 46.92 42.25 32.36 42.64 43.55 39.05 99.83
74.25 74.33 76.67 75.25 78.83 72.28 76.36 70.11 77.61 100.00
42.63 44.18 48.20 47.49 44.10 44.12 45.38 45.30 46.34 99.87
19
Table 11: Multimodal single-source DG on MOSI, MOSEI, and SIMS datasets for sentiment analysis with video, audio, and text modalities. Method
MOSEI → SIMS
MOSI → SIMS
MOSI → MOSEI
SIMS → MOSI
SIMS → MOSEI
Mean
MAE↓ F1↑ ACC2↑ MAE↓ F1↑ ACC2↑ MAE↓ F1↑ ACC2↑ MAE↓ F1↑ ACC2↑ MAE↓ F1↑ ACC2↑ MAE↓ F1↑ ACC2↑ ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29] Oracle
1.79 1.81 1.80 1.83 1.84 1.79 1.88 1.98 1.84 1.32
67.92 72.41 75.94 75.93 74.07 64.64 70.64 51.28 65.18 76.80
63.68 66.30 66.96 66.52 65.65 63.09 64.99 49.22 61.77 76.80
1.81 1.85 1.89 1.81 1.86 1.85 1.89 1.82 1.82 1.32
67.74 66.69 75.55 63.30 70.91 64.78 69.11 73.74 71.15 76.80
62.80 61.71 68.05 60.61 63.02 61.39 62.80 66.07 64.47 76.80
0.99 0.93 0.98 0.89 0.94 1.03 1.00 0.85 1.06 0.58
67.55 68.12 68.86 67.33 66.96 67.17 67.64 65.74 66.08 73.89
66.96 68.10 68.60 67.22 66.96 66.87 67.43 64.81 65.30 73.63
1.46 74.61 1.49 73.49 1.46 74.91 1.59 74.06 1.38 74.87 1.47 73.87 1.51 74.83 1.53 70.90 1.56 69.99 0.97 78.37
60.03 59.29 59.88 60.18 60.03 59.53 59.83 58.06 57.86 78.47
1.60 1.43 1.42 1.39 0.94 1.42 1.39 1.12 1.23 0.58
66.39 65.86 66.65 66.69 67.11 61.03 56.78 53.03 57.01 73.89
50.55 50.51 51.02 50.59 50.92 50.51 46.95 48.66 50.84 73.63
1.53 1.50 1.51 1.50 1.39 1.51 1.53 1.46 1.50 0.95
68.84 69.31 72.38 69.46 70.78 66.30 67.80 62.94 65.88 75.95
60.80 61.18 62.90 61.02 61.32 60.28 60.40 57.36 60.05 75.87
Table 12: Multimodal multi-source DG with corruptions on HAC dataset. Subscripts show the change relative to the clean Video+Audio setting. HAC dataset Corruption
Method
A, C → H
H, C → A
H, A → C
Mean
Wind on audio
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29]
77.43+1.52 75.78+0.58 79.96+1.37 77.65−1.73 76.06−1.88 75.83−0.50 76.42−1.74 77.93−0.19 76.48−0.88
74.39−3.09 73.51−3.97 76.49−1.55 75.71−2.99 71.63−6.63 73.96−2.46 74.27−3.72 75.99−2.92 74.92−1.55
52.67−0.73 49.36−4.22 56.25+0.46 54.78+0.00 47.70−4.14 46.20−4.87 49.82−3.29 51.62−1.87 48.69−3.64
68.16−0.77 66.22−2.53 70.90+0.09 69.38−1.57 65.13−4.22 65.33−2.61 66.84−2.91 68.51−1.66 66.70−2.02
Defocus on video
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29]
65.54−10.37 61.93−13.27 61.50−17.09 68.13−11.25 62.29−15.65 60.74−15.59 61.35−16.81 65.88−12.24 64.73−12.63
73.07−4.41 73.84−3.64 68.54−9.50 70.09−8.61 70.42−7.84 69.21−7.21 68.66−9.33 72.65−6.26 70.95−5.52
41.82−11.58 44.67−8.91 43.93−11.86 48.44−6.34 45.59−6.25 40.29−10.78 42.75−10.36 47.36−6.13 46.57−5.76
60.14−8.79 60.15−8.60 57.99−12.82 62.22−8.73 59.43−9.92 56.75−11.19 57.59−12.16 61.96−8.21 60.75−7.97
Table 13: Multimodal multi-source DG with missing modalities on HAC dataset. Subscripts show the change relative to the full Video+Audio setting. Modality
HAC dataset
Video
Audio
A, C → H
H, C → A
H, A → C
Mean
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29]
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
78.88+2.97 77.79+2.59 80.82+2.23 78.73−0.65 79.81+1.87 79.78+3.45 79.29+1.13 80.18+2.06 78.17+0.81
74.72−2.76 75.28−2.20 77.04−1.00 76.49−2.21 71.41−6.85 73.14−3.28 74.32−3.67 76.74−2.17 75.49−0.98
47.70−5.70 49.72−3.86 53.58−2.21 53.49−1.29 47.24−4.60 47.09−3.98 47.79−5.32 52.62−0.87 51.06−1.27
67.10−1.83 67.60−1.15 70.48−0.33 69.57−1.38 66.15−3.20 66.67−1.27 67.13−2.62 69.85−0.32 68.24−0.48
ERM RNA-Net [37] SimMMDG [11] MOOSA [8] CMRF [12] NEL [49] JAT [30] MBCD [45] GMP [29]
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
25.96−49.95 27.90−47.30 30.64−47.95 27.04−52.34 32.01−45.93 22.58−53.75 27.51−50.65 31.33−46.79 29.41−47.95
37.42−40.06 31.35−46.13 27.59−50.45 39.07−39.63 38.52−39.74 28.44−47.98 33.84−44.15 37.96−40.95 34.20−42.27
22.24−31.16 18.29−35.29 29.23−26.56 22.06−32.72 28.03−23.81 21.01−30.06 21.65−31.46 28.69−24.80 23.86−28.47
28.54−40.39 25.85−42.90 29.15−41.66 29.39−41.56 32.85−36.50 24.01−43.93 27.67−42.08 32.66−37.51 29.16−39.56
Method
20