Residual-Guided Expert Specialization for Incomplete Multimodal Learning

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Residual-Guided Expert Specialization for Incomplete Multimodal Learning

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

Residual-Guided Expert Specialization for Incomplete Multimodal Learning

arXiv:2606.30355v1 [cs.CV] 29 Jun 2026

Seunghun Baek , Jihwan Park , Jaeyoon Sim , Minjae Jeong , Hoseok Lee , and Won Hwa Kim Pohang University of Science and Technology (POSTECH), South Korea Code: https://github.com/seunghub/MARS

Abstract. As real-world prediction systems often face missing modalities at inference, incomplete multimodal learning (IML) remains a practical challenge. While prior methods aim to learn representations robust to missing inputs, representations from incomplete modalities inevitably deviate from their full-modality counterparts due to missing evidence. To explicitly leverage these deviations, we propose MARS (MissingnessAware Residual-guided Specialization), a mixture-of-experts framework that guides expert specialization based on how representations are reshaped by missingness. By contrasting task representations derived from incomplete inputs with their complete counterparts during training, we derive a privileged residual signal that captures this representational gap. The residual signal guides a residual router to assign samples to the experts specialized for the corresponding deviation patterns. In parallel, a feature router learns to imitate this routing behavior using only incomplete inputs, enabling deployment without access to full modalities. To mitigate this train–test router gap, we develop a discrepancy-aware noise regularization that adaptively perturbs the residual router’s decisions when the feature router deviates, enhancing the expert robustness under imperfect imitation. Experiments on multimodal classification (CASIASURF, CREMA-D, UPMC Food-101) and segmentation (MCubeS) under missing scenarios show that MARS consistently surpasses baselines, while remaining efficient and extensible to diverse backbones and tasks. Keywords: Incomplete Multimodal Learning · Mixture-of-Experts

Introduction

Leveraging information from multiple sources (i.e., modalities) has become a mainstream topic in computer vision across diverse tasks (e.g., classification [27, 39, 42], segmentation [16, 41]) and domains (e.g., medical imaging [1, 2, 15], remote sensing [10, 19]). By integrating complementary cues from diverse sources, multimodal learning enables a deeper understanding of the target task. However, utilizing all modalities, which were available during training, may be infeasible during inference due to practical reasons such as sensor malfunction or acquisition cost. To make multimodal systems practically reliable, it is crucial to make prediction models robust to test-time missingness. This incomplete multimodal

S. Baek et al.

(a) Router

This seems plausible, but I'm not certain.

(b) Full-modality

Representation

Partial-modality Representation

−

Residual Representation

With direct comparison, I know how this is distorted.

Shift induced by missingness

Fig. 1: Motivation of our residual-guided routing strategy. (a) In conventional Mixtureof-Experts (MoE) frameworks, the router bases its decision on the same input representation as the experts, which may lack sufficient task evidence and be distorted under missing modalities. (b) By contrasting the partial-modality representation with its full-modality counterpart, the residual captures how missingness reshapes the task representation. (c) Our router conditions expert specialization on this residual, enabling routing that accounts for such deviations and yields more reliable predictions.

learning (IML) setting is distinct from scenarios [18,38] that assume missingness during training, whose objective is to fully leverage incomplete multimodal data. Under the IML scenario, various approaches have been proposed to handle potential modality missingness. Imputation-based methods [3, 6] aim to reconstruct the missing modalities from the observed ones. However, these approaches often suffer from data hallucination [28, 30] and heavy computational overhead [36], which limit their practicality. Recent efforts have therefore shifted toward learning representations that are inherently robust to potential modality missingness. Early approaches, such as LCR [43] and RFNet [8], focus on learning modality-invariant features, but they unavoidably discard useful complementary cues from individual modalities. Subsequent studies have attempted to retain the complementary information via feature-level imputation [33], uncertainty modeling [35], or expert-based architectures [14,37]. But still, they rely solely on simulated input with missingness without explicitly exploring the direct effect of the absence, which causes systematic representation deviation. Key insight. Unlike previous works, we examine how modality removal reshapes the task representation. The representation computed from all modalities serves as the most task-sufficient reference, and when some modalities are absent, the resulting representation is inevitably altered as supportive evidence is removed. Although MMANet [34] attempts to align representations from partial inputs with their full-modality counterparts via knowledge distillation, the partial-modality representations still remain unreliable due to inevitable information loss. Rather than relying solely on this altered representation, we consider the difference (i.e., residual) between representations obtained with and without missing modalities. This residual directly reflects how the task representation shift due to modality removal, as illustrated in Fig. 1. By leveraging such deviation patterns captured by the residual, the model can better account for deviations in the input representation instead of making blind decisions. In this regime, we introduce MARS (Missingness-Aware Residual-guided Specialization), a mixture-of-experts (MoE) framework [25] equipped with a residual router that inputs the residual. Exposing such residual during training allows the router to directly assign samples to experts specialized for the corresponding deviation pattern (Sec. 3.2). Notice that this residual router cannot

MARS: Missingness-Aware Residual-guided Specialization

operate without access to complete information. Therefore, for test-time deployment with missing modalities, a feature router is introduced to approximate the residual router’s behavior without relying on residual signals (Sec. 3.3). This dual-router scheme thus enables a similar routing strategy at inference. Moreover, to mitigate the inherently inevitable discrepancy between the residual and feature routers, we introduce a discrepancy-aware noise regularization (Sec. 3.4) that adaptively increases the residual router’s stochasticity for experts with larger discrepancies between the feature and residual routers. This mechanism enhances expert robustness by encouraging exploration of alternative experts that may be preferable at test time under imperfect imitation. In addition, a discrepancy-guided sampling (Sec. 3.5) dynamically prioritizes modality combinations where the routers disagree the most, facilitating balanced learning under incomplete conditions. Together, these components yield robust expert behaviors and consistent performance gains across diverse domains and tasks. Overall, our contributions are summarized as threefold: – Residual perspective. We reformulate IML through a characterization of representation deviations in incomplete inputs, where routing is conditioned on such deviations while experts operate on incomplete features. This separation enables explicit expert specialization to missing-modality patterns rather than relying solely on unreliable incomplete representation. – Dual-router design. We introduce a dual-router framework that disentangles expert specialization and deployment. A deployable feature router learns to replicate the privileged residual router’s behavior at inference. Beyond knowledge distillation, we further guide the routing noise to mitigate train–test routing discrepancy and promote robust expert specialization. – Efficiency and generality. Through extensive experiments on four datasets across diverse domains and tasks, MARS consistently improves performance across modality combinations while remaining computationally efficient.

Related Work

Existing IML methods for test-time modality missingness can be broadly categorized into imputation-based, representation-based, and expert-based approaches. Imputation-based methods. Early works attempt to impute missing inputs from existing modalities. Generative models [3, 6] have been employed to reconstruct the absent modalities. However, retrieving signals from partial observations often leads to quality issues [28,30], while the computational overhead and reliance on generative models at inference make real-time applications infeasible. Representation learning under incompleteness. A subsequent line of work aims to learn representations that are robust to simulated modality missingness. HeMIS [12], LCR [43] and RFNet [8] learn modality-invariant representations, whereas mmFormer [40] and ShaSpec [33] retain modality-specific cues. While MMANet [34] distills model knowledge from a full-modality teacher to a partialmodality student, DMRNet [35] enhances robustness through probabilistic uncertainty modeling. However, the estimated uncertainty itself remains unreliable, as it is inferred solely from partial observations without explicit supervision.

S. Baek et al.

⋮

𝑧𝑧𝑖𝑖𝑟𝑟𝑒𝑒𝑒𝑒

𝜙𝜙 (𝑀𝑀)

⋮

𝜙𝜙 (1)

(𝒎𝒎)

𝒙𝒙𝒊𝒊

𝑓𝑓𝑢𝑢𝑢𝑢𝑢𝑢

(𝑴𝑴) 𝜹𝜹𝒊𝒊

𝐹𝐹𝑢𝑢𝑢𝑢𝑢𝑢𝑢𝑢𝑢𝑢 𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿

𝑝𝑝𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎

𝑧𝑧𝑖𝑖

: 𝑚𝑚𝑡𝑡𝑡 Modality Input

𝑝𝑝𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎

𝑧𝑧𝑖𝑖

𝑓𝑓𝑢𝑢𝑢𝑢𝑢𝑢

(𝒎𝒎)

𝑁𝑁

Residual Router

Routing Logits

Feature Router

𝑁𝑁

: Binary Mask

𝓛𝓛𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅

(Sec 3.3)

Routing Logits

𝑧𝑧𝑖𝑖

𝑁𝑁

(Sec 3.4)

Expert 1 Expert 11 Expert

Expert Networks Expert 1 Expert 11 Expert

𝑁𝑁

Routing Logits

𝝓𝝓(𝒎𝒎) : Modality Encoder

Learning from incomplete modalities 𝓛𝓛𝑳𝑳𝑳𝑳

Top-K Probability

Expert Networks

Feature Router

𝜹𝜹𝒊𝒊

(Sec 3.4)

𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝐻𝐻𝐻𝐻𝐻𝐻𝐻𝐻

⋮

𝐹𝐹𝑢𝑢𝑢𝑢𝑢𝑢𝑢𝑢𝑢𝑢 𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿

𝑧𝑧𝑖𝑖

(𝟏𝟏) 𝜹𝜹𝒊𝒊

𝓛𝓛𝒏𝒏𝒏𝒏𝒏𝒏𝒏𝒏𝒏𝒏

𝑁𝑁

Routing Noise

𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝐻𝐻𝐻𝐻𝐻𝐻𝐻𝐻

Random Modality Missing Simulation (Sec 3.1 & 3.5) (𝑀𝑀) (1) 𝑥𝑥𝑖𝑖 𝑥𝑥𝑖𝑖

𝒚𝒚𝑖𝑖 : Task Label

Top-K Probability

: Weighted Aggregation

𝑦𝑦�𝑖𝑖 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑦𝑦𝑖𝑖

𝓛𝓛𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕

(Sec 3.3)

𝑦𝑦�𝑖𝑖 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓

Learning from complete modalities : Weight Shared

Fig. 2: Overview of MARS. During training, complete and incomplete features (i.e., zif ull and zipartial ) are used to compute residuals that guide the residual router to specialize experts in distinct representation deviation patterns. The feature router learns to imitate this routing, enabling deployment without access to residuals. Discrepancyaware noise and load-balancing further enhance expert robustness and diversity.

Expert-based specialization. MoMKE [37] first introduced modality-specific experts, trained on unimodal data, and aggregated them through a soft router during a second training stage. However, all available inputs are fed into every expert during aggregation, which leads to unnatural cross-feeding (e.g., using RGB input in an IR-trained expert) and high computational overhead. SimMLM [14] alleviated this by routing each modality only to its corresponding expert, thereby achieving more efficient inference. Nevertheless, its aggregation operates at the logit level, which limits fine-grained specialization and increases computational cost. Moreover, both methods require two-stage training and impose a structural constraint that ties the number of experts to the number of modalities. In contrast, Flex-MoE [38] defines experts at the modality-combination level, assigning a dedicated expert to each combination while allowing additional experts to be selected per sample. However, this design still requires at least 2M -1 experts for M modalities. Moreover, defining experts strictly by modality combinations restricts specialization to predefined modality combinations. In practice, the task-specific contribution of each modality may vary across samples. Samples within the same combination may require distinct expertise, while different combinations often exhibit similar representation deviation patterns. Our perspective. Our model neither i) imposes structural constraints or predefined roles on experts, nor ii) relies solely on unreliable embeddings with missing modalities. By comparing task embeddings from partial observations with their complete counterparts, the router conditions expert specialization on representation deviation patterns, enabling adaptive handling of incomplete inputs.

Method

The goal of MARS is i) to train experts to specialize in task-specific deviation caused by modality missingness, and ii) to ensure consistent utilization of this

MARS: Missingness-Aware Residual-guided Specialization

specialization during inference. Fig. 2 illustrates the overview, and following sections describe how its components and training procedure are designed. 3.1

Problem Definition

Following recent works [14,35,37], we consider the general setting of IML, where some modalities may be missing at inference while remaining fully available dur(m) ing training. Given a dataset, each sample is denoted as xi = {xi }M m=1 , where (m) M indicates the number of modalities. Each sample with the m-th modality xi (m) (m) is encoded by a modality-specific encoder ϕ(m) , which yields ei =ϕ(m) (xi ). To simulate missing-modality scenarios during training, each modality is stochasti(m) (m) cally deactivated using a binary indicator δi ∈ {0, 1}, where δi = 0 masks its (m) corresponding embedding ei . Masked embeddings are concatenated and fused by a shared fusion layer ffuse to obtain a unified multimodal representation as (m) (m) zi = ffuse Concat {δi ei }M . (1) m=1 This task representation zi is later used to compute task-specific predictions. In previous methods, models process only one masked input (i.e., a single modality combination) per sample, from which a single task representation zi is extracted and used for prediction. In contrast, MARS jointly operates on two fused features of the same sample, one obtained from the complete input and the other from the masked input. By directly comparing these features, MARS learns an explicit notion of how representations change under missing modalities. Preliminaries. A sparse Mixture-of-Experts model [25] replicates some parts of the network as multiple expert modules. A router assigns scores to experts based on the input representation and aggregates the top-scoring expert outputs through weighted average. The router’s decision encourages experts to specialize across diverse input patterns with a modest increase in computational cost. 3.2

MoE with Privileged Residual Router

When all modalities are available, we refer to resulting task representation zi as a complete feature zifull , otherwise as an incomplete feature zipartial : ( (m) zifull , if ∀m, δi = 1 zi = (2) partial zi , otherwise. The MoE module is trained under a privileged setting where both zifull and zipartial are provided. Their difference zires provides a residual signal that explicitly characterizes missingness-induced representation deviation patterns: zires = zifull − zipartial .

(3)

This zires is fed into a residual router Rres , which guides each expert to specialize in how the routed feature zipartial deviates. The Rres yields a clean logit vector

S. Baek et al.

lires ∈ RN and an estimated noise standard deviation σi ∈ RN , where N denotes the number of experts. To encourage diverse expert exploration, we adopt noisy top-K routing [25] where Gaussian noise ϵ is injected to make logits lires noisy as ˜lres = lres + ϵσi , i i

ϵ ∼ N (0, I).

(4)

The top-K experts with the highest noisy logits ˜lires are selected to promote specialization while keeping the computation efficient. Their routing probabilities pres i are then computed using top-K softmax operator SoftmaxK (·), which applies the softmax over the selected top-K logits while assigning zero to the rest: N ˜res pres i = SoftmaxK (li ) ∈ R .

(5)

Let (pres i )j denote the routing probability of expert j for sample i. Each expert Ej (·) processes zipartial , and its output is weighted by (pres i )j . The aggregated output is then passed to a task head h(·), which produces a prediction ŷipartial :   N X partial partial res ŷi = h (pi )j Ej (zi ) . (6) j=1

This residual routing directly supervises deviation-aware expert specialization. Remarks. The formulation above faces two practical issues: 1) zires is unavailable at inference when complete modalities are missing. 2) When all modalities are present, zires becomes zero, resulting in identical routing across samples. We therefore introduce a feature router that operates without privileged information. 3.3 Preparing for Unprivileged Inference To enable practical deployment without residual signal, we introduce a feature router Rfea that performs routing using only the available embedding zi ∈ {zifull , zipartial }. It is designed as the deployable router at inference, capable of handling both complete and incomplete modality conditions. Under full modality condition, residual zires inherently becomes zero. Routfull ing is thus entirely handled by Rfea as pfea i = SoftmaxK (Rfea (zi )). The task full prediction ŷi follows the same aggregation process in Eq. (6) as ŷifull = P N fea full h j=1 (pi )j Ej (zi ) . With a task-specific objective ℓ (e.g., cross-entropy) and target label yi , the task-loss Ltask combines both full and partial cases as Ltask = ℓ(ŷifull , yi ) + ℓ(ŷipartial , yi ).

(7)

For incomplete inputs, Rres performs privileged routing using zires to obtain partial ŷi . In parallel, Rfea processes zipartial but is optimized only through a distillation loss Ldistill that aligns its logits lifea to those of Rres as Ldistill = DKL Softmax(GradStop(lires )) ∥ Softmax(lifea ) ,

(8)

where DKL denotes the Kullback–Leibler divergence, and GradStop(·) detaches gradients to prevent updates. This enables Rfea to learn from the privileged Rres , making it the sole deployable router with similar expert selection at inference.

MARS: Missingness-Aware Residual-guided Specialization

3.4

Discrepancy-Aware Noise Regularization

With noisy routing (Sec. 3.2), MoE frameworks typically incorporate a load–importance balancing loss LLB [25] for balanced expert utilization. Let B denote the mini-batch size, and define the importance and load of expert j as importancej =

B X

(pres i )j ,

i=1

loadj =

B X

I[(pres i )j > 0],

(9)

i=1

where I(·) is an indicator function that equals 1 if the condition holds, and 0 otherwise. The load-importance balancing loss LLB is then computed as 2 LLB = CV2 {importancej }N {loadj }N j=1 + CV j=1 ,

(10)

where squared coefficient of variation CV2 (x) = (σ(x)/µ(x))2 measures the uniformity across experts. This LLB prevents expert collapse and ensures balanced participation, while the injected routing noise ϵσi (Eq. (4)) promotes stochastic expert exploration. Beyond this conventional role, we further leverage the noise to mitigate the unavoidable routing discrepancy between training and inference. The residual router Rres provides an optimal supervision by leveraging both zipartial and zifull . However, despite the distillation loss Ldistill , Rfea cannot fully reproduce the behavior of Rres since it operates only with zipartial . This inevitably widens the train–test routing gap which causes inconsistent expert assignments. To mitigate this gap, we introduce a discrepancy-aware noise regularization Lnoise , which scales the noise variance (σi2 )j with the routing discrepancy of expert j. Higher noise variance induces the stochasticity of expert selection, allowing experts to remain robust even when the Rfea replaces Rres at inference. Let lires and lifea denote the clean logits from Rres and Rfea , respectively. We extract the top-K expert indices Tires , Tifea ∈ RK from each router as Tires = TopK(lires , K),

Tifea = TopK(lifea , K),

(11)

and define their union Ui =Tires ∪ Tifea . For each j ∈ Ui , we measure the routing discrepancy (mi )j as the absolute difference between the normalized clean logits: (mi )j = Softmax(lires )j − Softmax(lifea )j ,

j ∈ Ui .

(12)

We then sort experts in Ui in descending order of (mi )j , which yields a permutation πi such that (mi )πi (1) ≥. . . ≥(mi )πi (|Ui |) . To promote larger noise for experts with greater discrepancy, we penalize cases where the ordering of (σi2 )j contradicts that of (mi )j . Thus, the noise regularization Lnoise is formulated as Lnoise =

|Ui |−1 X 1 Softplus log(σi )2πi (j+1) − log(σi )2πi (j) , |Ui | − 1 j=1

(13)

which softly enforces (σi2 )πi (1) ≥ · · · ≥ (σi2 )πi (|Ui |) . As experts with greater discrepancy (mi )j are guided to exhibit higher routing noise (σi2 )j , this encourages

S. Baek et al.

exploration of alternative experts where the routers disagree. By being stochastically activated, these experts become more robust to test-time routing mismatch. Finally, the overall training objective Ltotal is defined as Ltotal = λtask Ltask + λLB LLB + λdistill Ldistill + λnoise Lnoise ,

(14)

where each λ controls the balance between individual loss terms L. 3.5

Top-K Discrepancy-Guided Modality Sampling

We further introduce an adaptive sampling strategy to balance the learning progress across different incomplete modality configurations. At the beginning of training, configurations are sampled uniformly, while the full-modality case is always included to provide complete supervision. After several warm-up epochs, the sampling probabilities are updated according to the routing discrepancy. Let C = {c1 , . . . , c|C| } denote all feasible incomplete modality combinations, where each cj∈{1,...,|C|} is represented by a binary vector composed of the masking indicators {δ (m) ∈ {0, 1}}M m=1 . We denote qj as the sampling probability of P|C| configuration cj satisfying j=1 qj = 1. For each training epoch, we measure the average top-K routing disagreement dj between the routers over all samples whose masked configuration corresponds to cj as dj = 1 −

1 X |Tires ∩ Tifea | , |Dcj | K

(15)

i∈Dcj

where Dcj denotes the set of training samples assigned to cj throughout the current epoch. At the end of the t-th iteration, discrepancies are normalized by (t+1) a softmax to update the next sampling probabilities qj as (t+1)

exp(dj /τ ) , = P|C| k=1 exp(dk /τ )

(16)

where the temperature τ adjusts the smoothness. Combinations showing higher router disagreement dj are sampled more frequently in subsequent epochs, allowing the model to focus on modality settings that are harder to align.

Experiments

We evaluate our method on 1) multimodal spoof classification, 2) material segmentation, 3) emotion recognition, and 4) food classification using CASIASURF [39], MCubeS [16], CREMA-D [4], and UPMC Food-101 [9]. For all datasets, missing-modality conditions follow the common protocols (Sec. 3.1) used in previous IML works [14, 34, 35, 37]. Due to space limitations, detailed dataset descriptions and result analyses for CREMA-D [4] (Tab. 3) and UPMC Food-101 [9] (Tab. 4) are provided in Sec. B.

MARS: Missingness-Aware Residual-guided Specialization

4.1

Datasets

CASIA-SURF [39] is a large-scale multimodal face anti-spoofing benchmark that contains RGB, Depth, and Infrared (IR) modalities. Each modality provides complementary cues for distinguishing live faces from spoofing attacks. The dataset offers binary labels indicating live or spoof, with N =29k, 1k, and 57k samples for training, validation, and testing data. Performance is evaluated using the Average Classification Error Rate (ACER, %) [39], which represents the percentage of misclassified samples. CASIA-SURF has been adopted as a standard benchmark for incomplete multimodal classification [34, 35]. MCubeS [16] is a multimodal material segmentation dataset composed of 500 samples captured from 42 diverse outdoor scenes. Each sample includes four modalities: RGB, Degree of Linear Polarization (DoLP), Angle of Linear Polarization (AoLP), and Near-Infrared (NIR) reflectance. Every pixel is annotated with one of 20 material categories such as concrete, water, and metal. The official split includes 302 training, 96 validation, and 102 test samples. Performance is measured by the mean Intersection-over-Union (mIoU), where higher values indicate better segmentation accuracy. We employ this recent dataset in IML scenarios to assess segmentation robustness under missing-modality conditions. CREMA-D [4] and UPMC Food-101 [9] are multimodal classification datasets comprising audio–visual and image–text modalities, respectively. They enable evaluation beyond vision-centric datasets. We follow the experimental setups of DMRNet [35] for CREMA-D and SimMLM [14] for UPMC Food-101. 4.2

Implementation Details

For CASIA-SURF [39] and MCubeS [16], we adopt a sparse MoE [25] framework with 16 experts, where the top five are activated per sample. The residual router Rres employs a single linear projection layer, and the feature router Rf ea consists of a two-layer MLP. All experiments are run on a single NVIDIA RTX 6000 ADA. CASIA-SURF. Following DMRNet [35], we employ a ResNet-18 [13] backbone with a binary indicator δ (m) (Sec. 3.1). The final convolution block is used as the expert set. Training is performed for 100 epochs using SGD [22] with a learning rate of 5×10−4 , weight decay of 5×10−4 , and a batch size of 64. Loss weights are λtask =1, λLB =0.05, λdistill =1, and λnoise =0.01. Sampling begins at epoch 20. MCubeS. We build on DeepLab v3+ [5] with the same Bernoulli indicator scheme. The final layer before the task head serves as the expert set. Models are trained for 500 epochs using SGD with a learning rate of 5×10−3 , weight decay of 5×10−4 , and a batch size of 8. Loss weights are set to λtask =1, λLB =0.01, λdistill =0.05, and λnoise =0.1, and our sampling strategy starts from epoch 30. CREMA-D and UPMC Food-101. Refer to Sec. B. 4.3

Baselines

We compare our method with recent state-of-the-art approaches. For classification, we report the DMRNet [35] results from the original paper and reimplement Flex-MoE [38], MoMKE [37], and SimMLM [14] using a ResNet-

S. Baek et al.

Table 1: Classification results (ACER ↓) on CASIA-SURF [39] under all modality combinations. MARS consistently outperforms baselines and achieves the lowest error rate across all settings, demonstrating strong robustness to missing modalities. The best and second-best results are highlighted with bold and underlined, respectively. ( : Presence, #: Absence) Modality

Methods

RGB Depth IR ResNet-18 HeMIS LCR RFNet mmFormer ShaSpec MMANet DMRNet Flex-MoE MoMKE SimMLM MARS #

# #

# # #

Average

11.75 5.87 16.62

14.36 13.44 12.43 4.70 4.40 4.17 16.21 15.26 14.69

11.15 3.67 13.99

11.57 6.25 10.71

8.57 2.27 10.04

8.23 2.01 8.98

9.10 2.63 7.25

10.04 1.95 12.57

7.82 2.42 9.05

6.92 1.87 3.96

4.61 6.68 4.95

3.23 6.27 3.68

3.32 5.16 3.53

2.23 4.27 3.22

1.93 4.77 3.10

3.11 4.23 2.52

1.61 3.01 1.18

1.21 3.00 0.80

1.27 3.58 1.19

1.20 5.09 1.06

1.13 3.38 1.15

1.06 2.09 0.66

2.21

1.97

1.88

1.18

1.94

1.79

0.87

0.66

0.84

0.79

0.63

0.45

7.52

7.18

6.71

6.02

5.93

5.59

3.94

3.58

3.69

4.67

3.66

2.43

Table 2: Segmentation results (mIoU ↑) on Table 3: Results on CREMA-D [4]. MCubeS [16] with all RGB combinations. Audio Visual ShaSpec MMANet DMRNet MARS ab t lexDeepL MRNe F E MoMKE SimMLM MARS RGB AoLP DoLP NIR v3+ D Mo

# # # #

Average

0.4249

0.4647 0.4490 0.4150

0.4462 0.4730

# #

0.4238 0.4249 0.4247

0.4682 0.4528 0.4241 0.4670 0.4529 0.4326 0.4672 0.4515 0.4008

0.4473 0.4767 0.4420 0.4753 0.4355 0.4769

0.4269 0.4269 0.4261

0.4694 0.4559 0.4173 0.4701 0.4554 0.4325 0.4691 0.4558 0.4238

0.4427 0.4777 0.4286 0.4808 0.4224 0.4776

0.4271

0.4712 0.4586 0.4376

0.4220 0.4808

0.4257

0.4683 0.4540 0.4273

0.4367 0.4773

Average

59.86 47.17 66.80 57.94

58.89 47.31 67.87 57.66

58.87 55.10 70.10 61.35

61.04 59.43 76.10 65.52

Table 4: Results on Food-101 [9]. Image Text ShaSpec MoMKE SimMLM MARS # 69.22 70.46 72.20 91.86 86.55 86.59 87.20 90.18 # 92.73 92.71 94.99 92.72 Average 82.83 83.25 84.81 91.59

18 [13] backbone for fair comparison. Since the original Flex-MoE assumes incomplete modalities during training, we adapt it by providing both complete and randomly dropped modalities via δ (m) . For segmentation, all baselines are re-implemented on DeepLab v3+ [5]. All hyperparameters are tuned for optimal performance. Further experimental details are given in Sec. D. 4.4

Spoof Classification Results

Tab. A1 summarizes the spoof classification results on the CASIA-SURF dataset. Among the baselines, DMRNet [35] achieves the lowest average ACER of 3.58, while MARS achieves 2.43 reducing it by 1.15. The model consistently attains the lowest ACER across all modality combinations, demonstrating strong robustness to missing modalities. The most challenging case occurs when only the IR is available. In this setting, MARS reduces the ACER from 8.98 (in DMRNet) to 3.96, effectively reducing the error by more than half. These results confirm that MARS robustly handles representation deviations induced by modality removal. 4.5

Material Segmentation Results

Tab. 2 reports the segmentation results on the MCubeS [16] dataset. Following the initial work [16], we focus on combinations that include the RGB, since nonRGB modalities, such as NIR, AoLP, or DoLP, alone lack sufficient radiometric

MARS: Missingness-Aware Residual-guided Specialization Water

Concrete

Plastic

Asphalt

Wood

Leaf

Metal

RGB Image

Ground Truth

DeepLab v3+ (RGB Only)

DeepLab v3+ (RGB+NIR)

MARS (RGB Only)

MARS (RGB+NIR)

RGB Image

Ground Truth

DeepLab v3+ (RGB Only)

DeepLab v3+ (RGB+AoLP+DoLP)

MARS (RGB Only)

MARS (RGB+AoLP+DoLP)

Fig. 3: Qualitative results on multimodal material segmentation (MCubeS [16]). Top: When near-infrared (NIR) information is missing, DeepLab v3+ [5] degrades in recognizing water and reflected objects, whereas MARS remains relatively consistent. Bottom: In the absence of polarization cues (AoLP/DoLP), MARS correctly separates metallic and dielectric materials even under incomplete inputs.

diversity to discern all classes. Consistent with the classification results, DMRNet [35] achieves the best baseline performance with an average mIoU of 0.4683. MARS improves it to 0.4773, surpassing baselines across all combinations. Fig. 3 presents qualitative comparisons on test samples from the MCubeS dataset. When compared to DeepLab v3+ [5], MARS provides clearer and more consistent segmentation results. When the NIR is missing, which helps identify water, DeepLab v3+ often misinterprets reflected surfaces over water as separate objects, while MARS remains robust to such misleading cues. Similarly, polarization cues such as AoLP and DoLP are important for distinguishing metallic and dielectric materials. Without them, the baseline predictions frequently confuse concrete, metal, and plastic, whereas MARS produces more stable segmentation results. These observations suggest that MARS better handles regions that are ambiguous under the given modalities, as residual guidance encourages experts to focus on reliable cues rather than misleading artifacts.

Model Behavior Analysis

We analyze the MARS framework to better understand the effect of its key components and the behavior of expert routing under missing-modality scenarios. 5.1

Ablation Study

Tab. 5 presents the ablation results on CASIA-SURF and MCubeS. Starting from each baseline, ResNet-18 and DeepLab v3+, the introduction of the MoE structure improves performance by diversifying strategies on inputs. Adding residual routing further enhances those MoE by leveraging the representational deviations for expert specialization. Variance ordering brings an additional gain by guiding the routing noise to account for train-test gap induced from router

S. Baek et al.

Table 5: Ablation study. Performance Table 6: Error rate (ACER, %) under expert is averaged over all modality configura- deactivation on CASIA-SURF. Changes in (). tions. “+” denotes cumulative inclusion. Modality Deactivated Expert CASIA-SURF MCubeS (ACER ↓) (mIoU ↑)

Method

7.52 4.12 3.78 2.72 2.43

0.4257 0.4327 0.4661 0.4750 0.4773

Oracle Routing

0.94

0.4768

# # #

# # # Average

E12

# 6.92 7.46 (0.54↑) 7.54 (0.62↑) 6.92 (0.00–) # 1.87 1.87 (0.00–) 1.74 (0.13↓) 1.87 (0.00–) 3.96 3.96 (0.00–) 7.58 (3.62↑) 3.96 (0.00–) # 1.06 0.99 (0.07↓) 2.09 1.52 (0.57↓) 0.66 0.66 (0.00–)

0.93 (0.13↓) 1.06 (0.00–) 1.73 (0.36↓) 2.09 (0.00–) 0.48 (0.18↓) 0.66 (0.00–)

0.45 0.31 (0.14↓)

0.36 (0.09↓)

2.43 2.40 (0.03↓)

2.91 (0.48↑) 2.50 (0.07↑)

0.93 (0.48↑)

0.4 0.3 0.2 0.1 0.0 E0 E1 E2 E3 E4 E5 E6 E7 E8 E E190 E11 E12 E13 E14 E15 E0 E1 E2 E3 E4 E5 E6 E7 E8 E E190 E11 E12 E13 E14 E15

Modality Combination [RGB, Depth, IR]

ResNet-18 / DeepLab v3+ + MoE + Residual Routing + Noise Regularization + Discrepancy-guided sampling

RGB Depth IR None

Fig. 4: Top-K routing probabilities of the feature router at inference on CASIASURF [39]. Probabilities are averaged over modality combination. Our method shows diverse expert utilization, unlike the baseline MoE [25] activating similar experts.

change. Finally, discrepancy sampling, which selectively emphasizes large gap between routers cases during training, yields the best overall performance. We add an experiment to solidify our key idea that residual signal promotes optimal expert specializations. The Oracle routing experiment assumes access to full modalities even at test time, where the true residuals are fed to the residual router for inference. This setting achieves near-perfect classification accuracy (ACER 0.94), confirming that the experts themselves are already well specialized to modality missingness. Interestingly, in segmentation, our model slightly surpasses the Oracle (mIoU 0.4773 vs. 0.4768). We attribute this to the noise regularization, which prevents overreliance on the residual routing and encourages the experts to learn complementary decision boundaries for the feature routing. 5.2

Effectiveness of Residual Routing

To show the effectiveness of residual routing in expert specialization, we analyze the routing distribution (Fig. 4) and visualizations of feature attribution (Fig. 5). Routing Distribution. Fig. 4 shows the routing probabilities of the feature router at test time on CASIA-SURF [39]. For each modality combination, we collect the top-K routing probabilities and average them across all samples to depict the expert activation distribution. To ensure a fair comparison, both the original MoE and our model share identical weight of hyperparameters including load-balancing weight λLB . In the baseline MoE, routing remains biased toward a few dominant experts irrespective of the modality combination. In contrast, our residual routing exhibits distinct yet structured expert utilization among different combinations. Most combinations activate at most two dominant experts,

MARS: Missingness-Aware Residual-guided Specialization RGB+Depth

RGB+IR

RGB+Depth+IR

RGB Only

RGB+Depth

RGB+IR

RGB+Depth+IR

MARS

Sparse MoE

RGB Only

Live Sample

Spoof Sample

8 16

Top-K

3 3

(a) Sensitivity analysis.

DeepLab v3+ DMRNet Flex-MoE MoMKE SimMLM MARS

#Params FLOPs 235M 238M 672M 245M 247M 244M

639G 639G 716G 1297G 955G 716G

(b) Complexity analysis.

RGB Depth IR RGB+Depth RGB+IR Depth+IR 40

0.2 0.3

# of Experts

0.1

4 ACER

Method

ACER

Fig. 5: Grad-CAM [23] visualization on CASIA-SURF [39]. Heatmaps are generated by backpropagating the predicted logit of the target class through zi and then overlaid on the RGB image. Compared to the baseline MoE [25], which inconsistently attends to superficial artifacts (e.g., around the mouth corner), MARS produces stable and semantically meaningful activations (e.g., around the nose) across diverse combinations. Epoch (Training Progress ) 4

80 100

Fig. 6: Comprehensive analyses of MARS. (a) Effect of the number of experts N and top-K on CASIA-SURF [39]. (b) Parameter and FLOPs comparison on MCubeS [16]. (c) Sampling probabilities across modality combinations on CASIA-SURF [39].

which rarely overlap to other combinations. Especially, we identify strong associations between specific modality combinations and experts (e.g., RGB–E12, IR–E0, Full–E5). This implies that the router effectively operates on missingness. To further verify whether the experts act responsible for the combination, we deactivate certain experts at inference and summarize the corresponding performance drops for each combination in Tab. 6. The combinations most strongly associated with the expert consistently incur the largest error increase. This confirms that experts with strong affinity to particular modality combinations learn disentangled specializations aligned with distinct missing-modality patterns. Feature Attribution. Fig. 5 visualizes Grad-CAM [23] results on CASIASURF [39]. For test samples xi , we backpropagate the predicted target logit yˆi through the fused embedding zi , and overlay the activation map on its RGB image xRGB . Each selected configuration includes one spoof and one live sample. i In the baseline MoE, attention regions vary largely across modality combinations, often highlighting superficial artifacts (e.g., mouth corner or forehead in RGB-only input). In contrast, our method consistently focuses on stable facial regions, particularly around the nose, which serves as a key discriminative cue for spoof detection. It also adapts its attention to secondary areas, such as the eyes or cheeks, depending on the available signals. This coherent attention pattern indicates that residual routing encourages reasoning based on reliable features rather than overfitting to spurious modality-specific artifacts.

5.3

S. Baek et al.

Comprehensive Analyses

Sensitivity Study. We analyze the sensitivity on the CASIA-SURF dataset [39]. In Fig. 6a, the top plot varies the number of experts with topK fixed to 5, and the bottom plot varies top-K with the number of experts fixed to 16. We observe that the performance degrades when either the number of experts or top-K increases beyond the optimal setting. This suggests that employing too many experts leads to redundancy among them, whereas activating more experts per sample could reduce their specialization. Complexity Study. We evaluate model complexity on the MCubeS dataset [16] in terms of FLOPs and parameter count, as summarized in Fig. 6b. We use segmentation to assess scalability, since its backbone (DeepLab v3+ [5]) and resulting feature maps are heavier than those in classification tasks, providing a more realistic view of computational extensibility. While Flex-MoE [38] introduces substantial parameter overhead due to its modality bank, MARS remains compact by adding only a lightweight residual router. Our FLOPs exhibit a moderate increase compared to DeepLab v3+, which is expected from adopting the sparse MoE structure. In contrast, other MoE-based frameworks, i.e., MoMKE [37] and SimMLM [14], show considerably higher computational costs, i.e., FLOPs, due to cross-feeding encoders and logit-level aggregation, respectively. Sampling Study. Fig. 6c visualizes the sampling probabilities q (in Eq. (16)) across various modality combinations on the CASIA-SURF dataset [39]. During the first 20 epochs, sampling is performed uniformly for all combinations. After activating the proposed discrepancy-aware sampling strategy (Sec. 3.5), the IR-only configuration exhibits the highest probability (∼0.43), indicating a large discrepancy d between the feature and residual routers resulting in more emphasis during training. This aligns with the observation in Sec. 4.4 that prior methods performed poorly under the IR-only setting, while our model achieved significant improvement. The next highest probabilities are observed for the RGB-only and RGB+IR combinations, confirming that the sampling mechanism prioritizes modality configurations requiring further refinement. Additional rationale for the residual is provided in Sec. A.

Conclusion

In this paper, we present MARS, a MoE framework that specializes experts by modeling missingness-induced representation deviations. By comparing full- and partial-modality representations, the residual router exploits residual cues that reflect how missingness affects task reasoning, specializing experts for the resulting deviations. The dual-router design and noise regularization together bridge the train–test routing gap, enabling benefits of residual-inspired routing even at inference. Furthermore, the proposed sampling balances training across modality configurations. Extensive incomplete multimodal experiments across domains and tasks demonstrate consistent superiority with minimal computational overhead. Thus, jointly leveraging observation and missingness offers a principled path toward robust multimodal systems under test-time incompleteness.

MARS: Missingness-Aware Residual-guided Specialization

Residual-Guided Expert Specialization for Incomplete Multimodal Learning Supplementary Material Overview ▶ Appendix Structure. To build intuition for the residuals, we provide further justification in Sec. A. The following sections (Sec. B–D) present additional experimental results and details that were omitted from the main paper due to space constraints. We conclude the appendix with clarification of the technical novelty (Sec. E) and a discussion of limitations and future work (Sec. F). ▶ Code Availability. We conduct experiments on CASIA-SURF [39], MCubeS [16], CREMA-D [4], and UPMC Food-101 [9] datasets. The code and pretrained checkpoints are available at https://github.com/seunghub/MARS.

Table of Contents A Additional Justification for Using Residual . . . . . . . . . . . . . . . . . . . . . . . .

A.1 Residual Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.2 Interpreting Residuals in Deep Feature Space . . . . . . . . . . . . . . . . .

A.3 Empirical Analysis of Representation Deviation . . . . . . . . . . . . . . .

A.4 Comparative Analysis of Representation Deviation Modeling . . . .

B Additional Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.1 CREMA-D [4] and UPMC Food-101 [9] Dataset . . . . . . . . . . . . . . .

B.2 Experimental Setups on CREMA-D [4] . . . . . . . . . . . . . . . . . . . . . . .

B.3 Experimental Setups on UPMC Food-101 [9] . . . . . . . . . . . . . . . . . .

B.4 Result Analysis (Tab. 3 and Tab. 4 in the main paper) . . . . . . . . .

C Additional Qualitative Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D Additional Experimental Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.1 Experimental Setups in CASIA-SURF [39] . . . . . . . . . . . . . . . . . . . .

D.2 Experimental Setups in MCubeS [16] . . . . . . . . . . . . . . . . . . . . . . . .

E Technical Novelty Clarification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

F Limitation and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

S. Baek et al.

Additional Justification for Using Residual RGB

Depth

MARS (zres)

RGB+Depth

RGB+IR

MARS (zfea)

Depth+IR

RGB+Depth+IR

Spoof

Live

Sparse MoE (zres) Sparse MoE (zfea)

Fig. A1: Embedding visualization from MARS and sparse MoE.

A.1

Residual Analyses

Fig. A1 visualizes feature z fea and residual embeddings z res across all configurations of 2,000 test samples using t-SNE. MARS: z res clusters by configuration with partial overlap, supporting our assumption. z fea follows this structure, where overlapping RGB/Depth/IR samples use same expert (e.g., E10 in Fig. 4), indicating shared deviation patterns. Sparse MoE: While z fea shows no clear clustering, constructing z res enables partial configuration-wise grouping. A.2

Interpreting Residuals in Deep Feature Space

The linear difference (i.e., residual) between z partial and z full reflects the elementwise deviations of z partial from the most informative anchor z full . This allows the router to infer relative reliability without requiring explicit scaling or normalization, since such adjustments can be learned implicitly during training. Although directly visualizing the residual z res is not straightforward, we’ve empirically demonstrated that this signal is effective in guiding expert specialization. Moreover, linear operations in learned representation spaces have been widely used in prior principled works. For instance, Word2Vec [20] demonstrates that vector offsets encode relational semantics, as illustrated by the classic example king - man + woman ∼ queen. Similarly, latent editing in GANs [11, 26] shows that adding or subtracting latent directions produces interpretable semantic transformations such as pose, age, or expression. These indicate that linear operations within a shared manifold can induce meaningful changes in model behavior. In our setting, both z full and z partial are generated by the same encoder and optimized for the same downstream objective. They therefore reside in a shared task-specific representation space. Their difference z res naturally reflects how the absence of modalities perturbs the evidence used by the model. From this perspective, the residual is not interpreted as a geometric distance but as a functional direction that isolates the task-relevant information removed by missing inputs. This interpretation aligns with a broad body of work showing

MARS: Missingness-Aware Residual-guided Specialization

that differences between deep representations can encode meaningful semantic or causal changes, even when the underlying feature space is highly entangled.

RGB

ACER

6 4

RGB+IR RGB+Depth Depth IR+Depth

RGB+IR+Depth

0.0

0.2

0.4

0.6

0.8

Normalized Residual 2

1.0

Fig. A2: Relationship between task performance and the normalized residual magnitude on CASIA-SURF [39] at inference. Larger residuals, which reflect stronger representation deviations induced by missingness, generally correspond to higher error rates.

A.3

Empirical Analysis of Representation Deviation

As an additional post-hoc examination, we investigate the empirical relation between the residual and prediction accuracy. While the magnitude of the residual alone does not fully characterize the effect of modality missingness, since directional and feature-wise factors also contribute, it provides a simple scalar summary that allows a coarse inspection of representation deviation. As shown in Fig. A2, larger residual magnitudes generally correspond to lower model accuracy across modality configurations. This trend suggests that residuals capture representation shifts induced by modality missingness, and that their magnitude offers a coarse yet informative indicator of such deviations. A.4

Comparative Analysis of Representation Deviation Modeling

A natural question is whether subtraction is the most suitable way to express the representation deviation introduced by modality missingness. To examine this, we compare it with several alternative formulations, including element-wise summation, product, ratio, cosine similarity, concatenation, and attention-based fusion. These alternatives capture different relationships between complete and

S. Baek et al.

Table A1: Ablation study of privileged router inputs on multimodal classification (ACER ↓) using CASIA-SURF [39]. Alternative ways of combining complete and incomplete features consistently underperform compared to the residual formulation. Modality

Methods

RGB Depth IR Sum Product Concat Ratio Cos Sim Attention MARS # # #

# # # Average

# 10.31 10.62 # 2.67 3.82 7.90 8.50

8.64 1.72 9.40

25.74 5.22 21.92

10.81 4.02 9.29

11.86 2.34 8.59

6.92 1.87 3.96

# 1.98 5.26 1.47

2.38 4.98 2.79

1.08 3.66 1.17

4.01 19.91 3.74

1.98 4.69 2.58

2.01 6.26 1.18

1.06 2.09 0.66

1.27

2.25

1.13

3.54

1.22

1.55

0.45

4.41

5.05

3.83

12.01

4.94

4.83

2.43

incomplete representations, while concatenation additionally provides the router with both embeddings without imposing a predefined structure. As shown in Table A1, none of these alternatives outperforms the residual formulation. In the incomplete-modality setting, such operations often introduce additional interactions between the two representations, which can make the resulting signal more sensitive to the uncertainty present in incomplete embeddings. As a result, the router may receive less stable cues for expert selection. Subtraction, in contrast, provides a direct characterization of the representation deviation caused by modality missingness. Its simplicity isolates the change between complete and incomplete embeddings without introducing additional interactions, yielding a clearer signal for routing decisions. This likely explains why the residual formulation consistently achieves the best performance.

Additional Experiment

Audio

Visual

…

Don’t forget a jacket.

Emotion: Angry

…

Don’t forget a jacket.

Emotion: Happy

Fig. A3: Visualization of a sample from the CREMA-D [4] dataset. The same actor speaks the same sentence while expressing different emotions. Audio and visual modalities provide complementary cues, such as vocal tone and facial expressions.

MARS: Missingness-Aware Residual-guided Specialization

Text

“Caesar Salad Recipe | Savoring Today LLC”

“12 Months of Martha - Fancy Shanty”

“Milk and Honey: Spaghetti Carbonara”

Caesar salad

Apple pie

Spaghetti carbonara

Image

Label

Fig. A4: Visualization of samples from the UPMC Food-101 [9] dataset. The text modality often provides strong cues by explicitly mentioning the food category (first and third samples), but in some cases it may be unrelated to the label (second sample).

B.1

CREMA-D [4] and UPMC Food-101 [9] Dataset

As visualized in Fig. A3, CREMA-D [4] is a multimodal emotion recognition dataset consisting of audio and visual recordings of acted emotional expressions. It provides six categories (happy, sad, anger, fear, disgust, and neutral ). Following the benchmark [4], we use 6,698 samples for training and 744 samples for testing. UPMC Food-101 [9] is a multimodal food classification dataset composed of paired images and textual descriptions collected in uncontrolled environments. It contains 101 food categories with 67,988 training samples and 22,716 test samples. In Fig. A4, the textual descriptions often provide strong cues by directly mentioning the corresponding food category. However, in some cases they contain only task-irrelevant information, as shown in the second sample of Fig. A4. For both datasets, performance is evaluated using classification accuracy. B.2

Experimental Setups on CREMA-D [4]

We extend the reported performance of DMRNet [35] and apply the same task setting. OGM-MD is a variant of OGM [21], proposed as a baseline in DMRNet [35], augmented with a binary indicator δ after each modality encoder. ShaSpec [33], MMANet [33], and DMRNet [35] are included as recent baselines. For consistency, we follow the model structure and data configuration of OGM [21]. All remaining data preprocessing and training procedures adhere to the settings described in DMRNet [35]. As in our other experiments, we activate the top-5 experts out of 16. The residual router generates clean logits and noise through two separate linear projection layers, while the feature router is implemented as a 2-layer MLP. The weights λtask , λLB , λdistill , and λnoise are set to 1, 1, 5, and 1, respectively. Our sampling strategy starts from epoch 20. B.3

Experimental Setups on UPMC Food-101 [9]

For UPMC Food-101, we reproduce the experimental setting of SimMLM [14]. Since the official implementation is not publicly available, we follow the model description provided in their paper. The image is encoded using Inception V3 [29],

S. Baek et al.

while text is processed with BERT [7]. Image and text representations are fused using the concatenation scheme proposed in the original dataset work [9]. In this setup, we employ a smaller MoE configuration with eight experts and top-3 routing. Each expert is implemented as a two-layer MLP operating on the concatenated representation. The residual router generates routing logits and noise using two linear projection layers, and the feature router is implemented as a two-layer MLP. The loss weights λtask , λLB , λdistill , and λnoise are set to 1, 0.05, 0.5, and 0.01, respectively. The model is trained for 100 epochs, and the proposed sampling mechanism starts from epoch 3. We use the AdamW [17] optimizer with a learning rate of 1×10−3 and a weight decay of 1×10−4 . A smaller learning rate is applied to the Inception V3 [29] and BERT [7] encoders. The learning rate is reduced by a factor of 0.1 when the loss plateaus. B.4

Result Analysis (Tab. 3 and Tab. 4 in the main paper)

On the CREMA-D [4] dataset, DMRNet [35] achieves the strongest performance among the baselines, consistent with observations from our main experiments. MARS further improves the average accuracy by a clear margin of ∼4.17%p over DMRNet. These consistent gains in the audio–visual setting demonstrate the generality and effectiveness of MARS across different multimodal domains. On the UPMC Food-101 [9] dataset, MARS attains the best average accuracy of 91.59%, with its advantage concentrated in the missing-modality configurations rather than the full-modality setting. This improvement is particularly meaningful for this dataset, where the textual modality exhibits highly variable informativeness. Descriptions may either explicitly reveal the food category or contain largely task-irrelevant or even misleading content. The results suggest that residual-guided routing enables the model to account for task-specific representational deviations rather than relying on predefined expert roles (i.e., modality configurations). As a result, the model maintains stable performance even when certain modalities provide noisy or unreliable cues, remaining above 90% regardless of which modality is missing in this benchmark.

MARS: Missingness-Aware Residual-guided Specialization Wood

Leaf

Cobblestone

Concrete RGB Only

Plastic

Asphalt RGB+NIR

Metal RGB+AoLP+DoLP

Brick

21 sky

RGB+NIR+AoLP+DoLP

DeepLab V3+ Sparse MoE DMRNet

RGB Image

Flex-MoE MoMKE

Ground Truth

SimMLM Ours

Fig. A5: Qualitative results on multimodal segmentation using MCubeS [16]. Compared to the baselines, MARS better separates concrete from cobblestone (red box) and wood from leaves (yellow box), even when only the RGB modality is available.

Additional Qualitative Result

In Fig. A5, we provide additional qualitative examples from the MCubeS dataset [16], comparing our method with all baselines. Since Near-Infrared (NIR) and polarization cues (AoLP and DoLP) offer complementary information be-

S. Baek et al.

yond RGB, leveraging more modalities generally leads to improved segmentation performance (see RGB Only vs. RGB+NIR+AoLP+DoLP). When only RGB is available, however, differentiating concrete from cobblestone (red box) and woods from leaves (yellow box) remains difficult. Although certain baselines occasionally perform well on one of these regions, none of them consistently handle both. In contrast, our method produces reliable predictions across both challenging areas, demonstrating stronger robustness under incomplete modalities.

ResNet18.conv1

ResNet18.bn1

ResNet18.relu

ResNet18.maxpool

ResNet18.layer1

ResNet18.layer2

ResNet18.se_layer

(b)

ResNet18.conv1

ResNet18.bn1

ResNet18.relu

ResNet18.maxpool

ResNet18.layer1

ResNet18.layer2

ResNet18.se_layer

ResNet18.conv1

ResNet18.bn1

ResNet18.relu

ResNet18.maxpool

ResNet18.layer1

ResNet18.layer2

ResNet18.se_layer

Concatenation

nn.AvgPool2d

ResNet18.layer4

Top-K Aggregation

nn.AvgPool2d nn.Linear

(e)

ResNet18.conv1

ResNet18.bn1

nn.Conv2d nn.ReLU

ResNet18.relu ResNet18.maxpool

ResNet18.layer1

ResNet18.layer2

nn.Linear

ResNet18.se_layer

nn.ReLU

nn.AvgPool2d

nn.Linear

nn.Conv2d

Aggregation

Concatenation

ResNet18

nn.Parameter

ResNet18.conv1

ResNet18.bn1

ResNet18.relu

ResNet18.maxpool

ResNet18.layer1

ResNet18.layer2

ResNet18.se_layer

Concatenation Fusion Layer

ResNet18.layer3

nn.AvgPool2d

ResNet18.layer4

nn.Linear

Top-K Aggregation

Residual Router

nn.AvgPool2d nn.Linear

nn.Linear G-Router

nn.AvgPool2d

ResNet18.conv1

Expert

nn.ReLU

nn.Conv2d nn.AvgPool2d

nn.AvgPool2d

ResNet18.layer4 ResNet18.layer4 ResNet18.layer4 Top-K Aggregation

Modality Bank

Aggregation

Task Head

nn.Conv2d

nn.ReLU

(f)

ResNet18.layer3

nn.Linear

nn.Linear Task Head Fusion Layer

nn.Conv2d

nn.ReLU

S-Router

nn.AvgPool2d

Modality Encoder

ResNet18.relu ResNet18.maxpool

Router

ResNet18.relu

nn.Conv2d

nn.Linear

Concatenation

ResNet18.maxpool

ResNet18.conv1

Concatenation Fusion Layer

ResNet18.layer3

Expert

Fusion Layer

ResNet18.layer3

Task Head

Task Head Modality Encoder

(d)

ResNet18.conv1

Task Head

Concatenation Fusion Layer

ResNet18.conv1

Modality Encoder

(a)

ResNet18.layer3 ResNet18.layer4 ResNet18.layer4 ResNet18.layer4 Top-K Aggregation nn.AvgPool2d nn.Linear

nn.AvgPool2d nn.Linear nn.ReLU nn.Linear Feature Router

Fig. A6: Architectural overview of the baseline models and the proposed method on CASIA-SURF [39]: (a) ResNet-18 [13], (b) DMRNet [35], (c) Flex-MoE [38], (d) MoMKE [37], (e) SimMLM [14], and (f) MARS (ours). Table A2: Comprehensive training configurations for all compared methods on CASIA-SURF [39] (left) and MCubeS [16] (right). CASIA-SURF

MCubeS

Setting

DMRNet Flex-MoE MoMKE SimMLM MARS DMRNet Flex-MoE MoMKE SimMLM MARS

Learning rate

1×10−3

Weight decay Epochs Batch size Optimizer Backbone Augmentation

5×10−4 100 64 SGD ResNet-18 Horizontal Flip, Random Crop, Random Rotation

1×10−2

1×10−5

5×10−3

5×10−4

5×10−2

5×10−3

5×10−2

5×10−3

5×10−4 500 8 SGD DeepLab V3+ Horizontal Flip, Random Crop, Gaussian Blur

MARS: Missingness-Aware Residual-guided Specialization

Additional Experimental Details

As illustrated in Fig. A6, our model and baselines are implemented on top of the same backbone architecture. Model-specific variations follow their official codebases or papers. We conduct an extensive hyperparameter search for all baselines to obtain their best performance, and the selected configurations are summarized in Tab. A2. Detailed setups for each baseline are provided below. D.1

Experimental Setups in CASIA-SURF [39]

DMRNet [35]. We report the presented results in the paper. Flex-MoE [38]. Flex-MoE is originally designed for training scenarios with incomplete modalities. To align it with our setting and achieve its best performance, we provide full-modality inputs for training the G-router, and maskedmodality inputs for training the S-router. This configuration offers a more privileged setting than their original formulation but matches our IML setup. Both routers use a single projection layer, and the missing-modality bank operates on z (m) i. We found that selecting Top-5 experts out of 16 yielded the best performance within the search ranges of K ∈ {3, 4, 5, 6} and N ∈ {8, 16, 32}. The optimal balancing-loss weight was LLB =0.1 (searched over {0.5, 0.1, 0.05, 0.01}), and the optimal top-1 guiding loss weight was Lce =0.1 (searched over {1, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001}). MoMKE [37]. We first train an individual ResNet-18 [13] for each modality for 50 epochs with a learning rate of 1×10−3 . We then train the full model with the experts and a projection router for 100 epochs with a learning rate of 5×10−4 . We search learning rates over [10−4 , 10−2 ] and training epochs over [50, 200]. SimMLM [14]. Following MoMKE, we train modality-specific ResNet-18 models for 50 epochs with a learning rate of 1×10−3 . We then attach a lightweight router network that produces aggregation weights for modality-specific outputs. The second stage is trained for 100 epochs with a learning rate of 5×10−3 , using the same hyperparameter search space as MoMKE. For the MoFe regularization, the best performance is achieved with weight 0.1 (searched over 1, 0.5, 0.1, 0.05, 0.01). D.2

Experimental Setups in MCubeS [16]

Unless otherwise specified, we reuse the configurations and hyperparameter search ranges described in Section D.1. DMRNet [35]. Following their formulation, we replace zi with µi + ϵσi , where µi and σi are learned in the same manner as zi , and ϵ ∼ N (0, I). We set the distribution-regularization weight to LDR =1×10−4 (searched over {10−2 , 10−3 , 10−4 , 10−5 }) and the hard-combination regularization weight to LHCR =0.1 (searched over {1, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001}). Flex-MoE [38]. We adopt the same strategy as in CASIA-SURF [39] (Section D.1), but the optimal balancing-loss weight was LLB = 0.01 and the top-1 guiding loss weight was Lce = 0.005 within the same search range.

S. Baek et al.

MoMKE [37]. We train the first and second stages for 200 and 500 epochs with learning rates of 1 × 10−4 and 5 × 10−3 , respectively. We use a 2-layer CNN cross-modal encoder and a 2-layer MLP router with average pooling. SimMLM [14]. We use the same first-stage setup as in MoMKE [37]. In the second stage, we train for 500 epochs with a learning rate of 5×10−2 . The router consists of a 2-layer CNN followed by pooling and a linear layer, and LMoFe =0.1.

Technical Novelty Clarification

To the best of our knowledge, the technical novelty of MARS lies in three aspects. – Modeling Representation Deviations. Most prior IML methods focus on making better use of the simulated inputs, for example through modality imputation, distillation, or robust feature fusion. MARS instead focuses on explicitly modeling how the task representation changes when modalities are missing. Our key idea is that incomplete modalities inherently produce imperfect task representations compared to full-modality inputs, even with sophisticated modeling. When modalities contain complementary information, their absence inevitably introduces representational deviations. By explicitly capturing this deviation through the residual between complete and incomplete embeddings, our framework shifts the perspective of IML from missing-data compensation to representation-deviation modeling. – Privileged Signals for Routing Specialization. MARS is related to the Learning Using Privileged Information (LUPI) paradigm [24, 31, 32] in that additional information is available during training but not at inference. In MARS, privileged signals (i.e., z full ) are used to compute residual representations that guide expert specialization. This design decouples expert specialization from deployment routing. It introduces routing mechanisms with different inputs for specialization and deployment. Such a routing-oriented use of privileged signals has not been explored in prior studies. – Task-Oriented Routing Noise. Conventional noisy routing in mixtureof-experts models mainly serves to improve expert diversity. In contrast, MARS leverages routing noise to address the discrepancy between the privileged residual router and the deployable feature router. Expert activation is therefore influenced through two complementary channels as deterministic routing logits and discrepancy-aware stochastic modulation. This perspective expands the role of routing noise from a generic regularizer to a controllable signal that stabilizes routing under train–test modality gaps.

Limitation and Future Work

MARS is designed for the standard IML setting where all modalities are available during training and missingness occurs only at inference. Under this assumption, the residual signals are reliably constructed from complete inputs, which enables effective expert specialization. However, in scenarios where the

MARS: Missingness-Aware Residual-guided Specialization

training data themselves contain incomplete modalities, the residual supervision may not be directly obtainable. Adopting our approach to such settings would require redefining how to estimate representation deviations without consistent full-modality references. We leave this as an interesting direction for future work.

Acknowledgements This research was supported by RS-2025-02216257 (65%), RS-2022-II220290 (30%), and RS-2019-II1091906 (AI Graduate Program at POSTECH, 5%).

References 1. Baek, S., Choi, I., et al.: Learning covariance-based multi-scale representation of neuroimaging measures for alzheimer classification. In: ISBI. pp. 1–5. IEEE (2023) 2. Baek, S., Sim, J., Wu, G., Kim, W.H.: Ocl: Ordinal contrastive learning for imputating features with progressive labels. In: MICCAI. pp. 334–344. Springer (2024) 3. Cai, L., Wang, Z., Gao, H., et al.: Deep adversarial learning for multi-modality missing data completion. In: ACM SIGKDD. pp. 1158–1166 (2018) 4. Cao, H., Cooper, D.G., Keutmann, M.K., et al.: Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing 5(4), 377–390 (2014) 5. Chen, L.C., Zhu, Y., Papandreou, G., et al.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV. pp. 801–818 (2018) 6. Chen, Z., Li, H., Wang, F., et al.: Rethinking the diffusion models for missing data imputation: A gradient flow perspective. NeurIPS 37, 112050–112103 (2024) 7. Devlin, J., Chang, M.W., Lee, K., et al.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL. pp. 4171–4186 (2019) 8. Ding, Y., Yu, X., Yang, Y.: Rfnet: Region-aware fusion network for incomplete multi-modal brain tumor segmentation. In: ICCV. pp. 3975–3984 (2021) 9. Gallo, I., Ria, G., Landro, N., La Grassa, R.: Image and text fusion for upmc food-101 using bert and cnns. In: 2020 35th International conference on image and vision computing New Zealand (IVCNZ). pp. 1–6. IEEE (2020) 10. Gómez-Chova, L., Tuia, D., Moser, G., et al.: Multimodal classification of remote sensing images: A review and future directions. Proceedings of the IEEE 103(9), 1560–1584 (2015) 11. Härkönen, E., Hertzmann, A., Lehtinen, J., et al.: Ganspace: Discovering interpretable gan controls. NeurIPS 33, 9841–9850 (2020) 12. Havaei, M., Guizard, N., Chapados, N., Bengio, Y.: Hemis: Hetero-modal image segmentation. In: MICCAI. pp. 469–477. Springer (2016) 13. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016) 14. Li, S., Chen, C., Han, J.: Simmlm: A simple framework for multi-modal learning with missing modality. In: ICCV (2025) 15. Li, Y., Daho, M.E.H., Conze, P.H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., Quellec, G.: A review of deep learning-based information fusion techniques for multimodal medical image classification. Computers in Biology and Medicine 177, 108635 (2024)

S. Baek et al.

16. Liang, Y., Wakaki, R., Nobuhara, S., et al.: Multimodal material segmentation. In: CVPR. pp. 19800–19808 (2022) 17. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. ICLR (2017) 18. Ma, M., Ren, J., Zhao, L., et al.: Are multimodal transformers robust to missing modality? In: CVPR. pp. 18177–18186 (2022) 19. Ma, X., Zhang, X., Pun, M.O., et al.: A multilevel multimodal fusion transformer for remote sensing semantic segmentation. IEEE Transactions on Geoscience and Remote Sensing 62, 1–15 (2024) 20. Mikolov, T., Chen, K., Corrado, G., et al.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013) 21. Peng, X., Wei, Y., Deng, A., et al.: Balanced multimodal learning via on-the-fly gradient modulation. In: CVPR. pp. 8238–8247 (2022) 22. Ruder, S.: An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 (2016) 23. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., et al.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: ICCV. pp. 618–626 (2017) 24. Sharmanska, V., Quadrianto, N., et al.: Learning to rank using privileged information. In: ICCV. pp. 825–832 (2013) 25. Shazeer, N., Mirhoseini, A., Maziarz, K., et al.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR (2017) 26. Shen, Y., Yang, C., Tang, X., et al.: Interfacegan: Interpreting the disentangled face representation learned by gans. TPAMI 44(4), 2004–2018 (2020) 27. Sleeman IV, W.C., Kapoor, R., Ghosh, P.: Multimodal classification: Current landscape, taxonomy and future directions. ACM Computing Surveys 55(7), 1– 31 (2022) 28. Sun, Y., Sheng, D., Zhou, Z., et al.: Ai hallucination: towards a comprehensive classification of distorted information in artificial intelligence-generated content. Humanities and Social Sciences Communications 11(1), 1–14 (2024) 29. Szegedy, C., Vanhoucke, V., Ioffe, S., et al.: Rethinking the inception architecture for computer vision. In: CVPR. pp. 2818–2826 (2016) 30. Tivnan, M., Yoon, S., Chen, Z., et al.: Hallucination index: An image quality metric for generative reconstruction models. In: MICCAI. pp. 449–458. Springer (2024) 31. Vapnik, V., Izmailov, R.: Learning using privileged information: similarity control and knowledge transfer. JMLR 16(1), 2023–2049 (2015) 32. Vapnik, V., Vashist, A.: A new learning paradigm: Learning using privileged information. Neural networks 22(5-6), 544–557 (2009) 33. Wang, H., Chen, Y., Ma, C., et al.: Multi-modal learning with missing modality via shared-specific feature modelling. In: CVPR. pp. 15878–15887 (2023) 34. Wei, S., Luo, C., Luo, Y.: Mmanet: Margin-aware distillation and modality-aware regularization for incomplete multimodal learning. In: CVPR. pp. 20039–20049 (2023) 35. Wei, S., Luo, Y., Wang, Y., et al.: Robust multimodal learning via representation decoupling. In: ECCV. pp. 38–54. Springer (2024) 36. Wu, R., Wang, H., Chen, H.T., et al.: Deep multimodal learning with missing modality: A survey. ACM Computing Surveys (2024) 37. Xu, W., Jiang, H., Liang, X.: Leveraging knowledge of modality experts for incomplete multimodal learning. In: ACM MM. pp. 438–446 (2024) 38. Yun, S., Choi, I., Peng, J., et al.: Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts. NeurIPS 37, 98782–98805 (2024)

MARS: Missingness-Aware Residual-guided Specialization

39. Zhang, S., Wang, X., Liu, A., et al.: A dataset and benchmark for large-scale multi-modal face anti-spoofing. In: CVPR. pp. 919–928 (2019) 40. Zhang, Y., He, N., Yang, J., et al.: mmformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation. In: MICCAI. pp. 107–117. Springer (2022) 41. Zhang, Y., Sidibé, D., Morel, O., et al.: Deep multimodal fusion for semantic image segmentation: A survey. Image and Vision Computing 105, 104042 (2021) 42. Zheng, X., Tang, C., Wan, Z., et al.: Multi-level confidence learning for trustworthy multimodal classification. In: AAAI. vol. 37, pp. 11381–11389 (2023) 43. Zhou, T., Canu, S., Vera, P., et al.: Brain tumor segmentation with missing modalities via latent multi-source correlation representation. In: MICCAI. pp. 533–541. Springer (2020)

Record · ID 321885 · SHA-256 d7b91ca25ae4a2a3
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.