SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits Adis Alihodzic1* and Selma Skopljakovic Hubljar1 1*
Department of Mathematical and Computer Sciences, Faculty of Science, University of Sarajevo, Sarajevo, Bosnia and Herzegovina.
*Corresponding author(s). E-mail(s): [email protected]; Contributing authors: [email protected];
arXiv:2607.08573v1 [cs.AI] 9 Jul 2026
Abstract Multimodal emotion and sentiment recognition commonly relies on either early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors. Early fusion is often accurate but monolithic, whereas late fusion is modular but can lose cross-modal interactions. This paper revisits XAI-guided adaptive fusion (XGAF), a tree-based mixture of unimodal and cross-modal experts whose sample-level weights are derived from TreeSHAP attribution magnitudes. The study focuses on a methodological issue that becomes important when experts have unequal feature dimensionalities: reducing feature attributions by mean absolute SHAP values can suppress high-dimensional cross-modal experts, while reducing them by summed absolute SHAP values preserves total attribution mass. On MELD 7-class emotion recognition, the proposed sum-abs reduction closes the gap between the cross-modal expert mixture and early fusion across three face-sequence aggregators, with the Transformer variant reaching 0.5983 weighted-F1 compared with 0.6018 for early fusion and 0.4598 for probability-average late fusion. McNemar testing shows no significant difference between sum-abs XGAF and early fusion on MELD (p = 1.000), while XGAF remains significantly better than late fusion (p < 0.0001). On CMU-MOSEI 3-class sentiment recognition, sum-abs XGAF reaches 0.6519 weighted-F1 compared with 0.6485 for early fusion and 0.5696 for late fusion, with a small but statistically significant improvement over early fusion (p = 0.0452). An expert-pool ablation indicates that most of the gain comes from adding crossmodal experts, especially the trimodal expert, rather than from rich per-sample routing. A focused three-way ablation further confirms that median-abs SHAP behaves similarly to mean-abs SHAP (0.5682 versus 0.5669 weighted-F1 on MELD), while sum-abs SHAP reaches early-fusion-level performance (0.5957 versus 0.5955). Diagnostic analysis shows that mean-abs and median-abs weights are nearly uniform, whereas sum-abs weights become strongly concentrated on the trimodal expert. The main contribution is therefore not a new state-of-the-art recognition model, but a transparent empirical study of how SHAP attribution reduction, expert dimensionality, and cross-modal expert design affect modular multimodal fusion. Keywords: multimodal fusion, emotion recognition, sentiment analysis, explainable artificial intelligence, SHAP, XGBoost, MELD, CMU-MOSEI, mixture of experts
1
1 Introduction
maximum benchmark accuracy but also modularity, reproducibility, and interpretable behaviour of modality-specific components. This paper studies a deliberately simple alternative: a tree-based expert pool composed of unimodal and cross-modal XGBoost classifiers [13]. The experts are combined by a sample-dependent gate derived from TreeSHAP attribution magnitudes [14]. The motivation is straightforward. If a model’s explanation suggests that a given expert has strong attribution mass on a sample, then that expert should receive a higher fusion weight. This idea produces a modular architecture in which text-only, audio-only, face-only, bimodal, and trimodal experts can be inspected separately while still being combined into a single decision. The central finding of the paper is more nuanced than the original “adaptive fusion” motivation. When feature attributions are reduced by the mean of their absolute values, experts with different input dimensionalities are implicitly placed on a per-feature scale. This can normalise away the total attribution mass of high-dimensional cross-modal experts and lead to almost uniform weights. Replacing mean absolute SHAP by summed absolute SHAP restores the scale of crossmodal experts and substantially improves performance. A third natural statistic is the median of the absolute SHAP values. It is more robust to extreme individual attributions and measures a typical feature contribution, but it can also understate experts whose predictive evidence is concentrated in a small number of highly informative features. For this reason, the paper formalises and empirically evaluates mean-, median-, and sum-absolute reductions. The median-abs ablation is important because it is a statistically natural robust alternative, yet the results show that it behaves much closer to mean-abs than to sum-abs. Diagnostic analysis also shows that the improved sum-abs gate does not produce rich persample routing on the evaluated datasets. Instead, the gain is largely explained by the availability of cross-modal experts, especially the trimodal expert. This limitation is important: it prevents the method from being overstated, but it also turns the experiment into a useful empirical lesson about attribution-based gating. The contributions of this paper are as follows:
Multimodal affective computing attempts to infer human emotion, sentiment, or related affective states from several complementary channels, most commonly text, speech, and visual cues. A sentence may look positive in text but sound sarcastic in speech, or a neutral phrase may become emotionally informative when facial expression is considered. This simple observation explains why fusion is not a minor implementation detail but one of the central design choices in multimodal emotion recognition. Benchmark datasets such as MELD [1] and CMU-MOSEI [2] have made it possible to compare modelling strategies under controlled experimental protocols, but the question of how modalities should be fused remains central. Recent surveys emphasise that modern MER systems must be evaluated not only by accuracy but also by fusion design, robustness to missing or noisy modalities, generalisation across users and datasets, and interpretability of multimodal decisions [3]. In practice, two simple strategies continue to be widely used. Early fusion concatenates modality-specific feature vectors and trains a single predictor on the joint representation. Late fusion trains separate predictors and combines their output probabilities. Early fusion can exploit cross-modal feature interactions, but it is monolithic and less transparent at the modality level. Late fusion is modular and easy to extend, but it often underuses cross-modal interactions. A large body of work has proposed more sophisticated fusion mechanisms for multimodal language and affect recognition, including tensor fusion [4], low-rank multimodal fusion [5], cross-modal translation [6], multimodal Transformers [7], and representation factorisation into modality-invariant and modality-specific components [8]. In emotion recognition in conversations, dialogue-aware models further exploit speaker state, discourse structure, graphs, or commonsense knowledge [9–12]. In the taxonomy of recent MER surveys, these methods mainly occupy hybrid and model-level fusion branches, whereas feature concatenation and probability averaging correspond to early feature-level and decision-level fusion, respectively [3]. These methods are powerful, but they can be harder to diagnose and deploy when the operational goal is not only
2
Fusion methods for multimodal language analysis range from simple concatenation to explicit modelling of high-order interactions. Tensor Fusion Network captures unimodal, bimodal, and trimodal interactions through an outerproduct representation [4]. Low-rank multimodal fusion reduces the computational burden of such interactions [5]. Multimodal Transformer uses cross-modal attention for unaligned multimodal sequences [7], while MISA separates modalityinvariant and modality-specific factors [8]. SelfMM further introduces self-supervised multi-task representation learning for multimodal sentiment analysis [15]. These models usually target strong end-to-end representation learning, whereas the present paper focuses on a modular tree-based expert pool with explicit diagnostic analysis. The same survey also highlights three trends that directly motivate the design choices and limitations of this paper: stronger cross-modal fusion, robustness under missing or degraded modalities, and explainability for trustworthy affective computing [3]. XGAF addresses the explainability and modularity side by using TreeSHAP scores as gate signals, but it does not yet address the missingmodality and noisy-modality scenarios that are increasingly treated as central deployment challenges. This distinction is important because the experiments in this paper use clean pre-extracted features; therefore, the results should be interpreted as a controlled analysis of attribution-based expert weighting rather than as a complete robust MER system. For conversation emotion recognition, DialogueRNN [9], DialogueGCN [10], COSMIC [11], and M2FNet [12] illustrate the importance of dialogue context and speaker interactions. Because the present work treats each utterance independently after feature extraction, it is not expected to surpass such dialogue-aware methods. This design choice makes the fusion analysis cleaner, but it also defines an important limitation.
1. A reproducible SHAP-weighted expert-fusion framework is formulated for multimodal emotion and sentiment recognition using unimodal, bimodal, and trimodal XGBoost experts. 2. The expert-score construction is clarified by formalising and evaluating three natural SHAP reductions: mean-abs, median-abs, and sumabs. The experiments show that median-abs provides a useful robust baseline, but that preserving total attribution mass with sum-abs is critical when experts have unequal feature dimensionalities. 3. Experiments on MELD and CMU-MOSEI show that the sum-abs variant matches early fusion on MELD and slightly outperforms it on CMU-MOSEI, while consistently outperforming probability-average late fusion. 4. Expert-pool ablations identify a compact fourexpert variant, denoted XGAF-Lite, as sufficient to match the full seven-expert pool on MELD. 5. A diagnostic interpretability analysis is reported, showing that the current SHAP gate mainly produces expert dominance rather than diverse per-sample routing. This negative result is explicitly discussed as a limitation and as guidance for future work. The paper is intentionally positioned as a fusion-method study rather than as a state-ofthe-art dialogue-emotion model. Dialogue context is not modelled, and the goal is to analyse modular expert fusion under a transparent and reproducible classical-machine-learning pipeline.
2 Related Work 2.1 Multimodal emotion and sentiment recognition MELD extends the EmotionLines corpus with multimodal information from multi-party conversations and is widely used for emotion recognition in conversations [1]. CMU-MOSEI is a large-scale multimodal sentiment and emotion dataset based on opinion videos and aligned language, acoustic, and visual signals [2]. The two datasets differ in task formulation, label structure, and modality quality, making them useful for testing whether a fusion mechanism is dataset-specific.
2.2 Mixture of experts and explainable gating Mixture-of-experts models combine several specialised predictors through a gating mechanism, and sparse or conditional computation has been used successfully in large-scale neural models [16]. In multimodal settings, expert specialisation can 3
3.2 SHAP-derived expert scores
arise naturally because different modalities have different noise patterns and missingness profiles. However, a gate trained end-to-end can be difficult to interpret, especially when the final model is a deep architecture. TreeSHAP provides consistent feature attributions for tree ensemble models and is widely used to explain individual predictions [14]. This paper uses TreeSHAP not only for post-hoc explanation but also as a gate signal. The idea is simple: an expert receives a higher weight when its absolute attribution mass is larger on the current sample. The experiments show, however, that a post-hoc attribution signal is not automatically a reliable routing signal. The way attributions are reduced to one scalar per expert has a strong influence on the resulting gate.
For a sample x and expert e, TreeSHAP produces feature-attribution values. For multi-class classification, the implementation aggregates absolute attribution magnitudes over classes and features. Let de denote the number of input features of expert e and let ϕe,j (x) denote the attribution contribution associated with feature j after the class-level aggregation. Three natural scalar expert scores can be defined: d
amean (x) = e
amedian (x) = medianj=1,...,de |ϕe,j (x)|, e asum (x) = e
de X
|ϕe,j (x)|.
(2) (3) (4)
j=1
3 Method
The mean-abs score measures the average attribution per feature. The median-abs score measures the typical attribution and is robust to unusually large individual feature contributions. The sum-abs score measures the total attribution mass carried by the whole expert. This distinction matters because expert-level gating is not the same as ranking individual features. A median score can be useful when a few outlier attributions should not dominate the gate, but it may also undervalue an expert whose evidence is sparse and concentrated in only a few highly informative dimensions. A sum score has the opposite bias: it preserves total evidence, but it can favour high-dimensional cross-modal experts. When experts have strongly different dimensions, the reduction choice is therefore not merely cosmetic. In the MELD setup, text and voice experts have 768 features, the face expert has 512 features, bimodal experts have 1280 or 1536 features, and the trimodal expert has 2048 features. Mean-abs and median-abs reductions place experts on a typical-per-feature scale, whereas sum-abs reduction compares total attribution mass. The empirical tables in this preprint report mean-, median-, and sum-abs reductions. This makes the reduction analysis more informative: mean-abs and median-abs both operate on a typical-feature scale, whereas sum-abs operates on the total-evidence scale of the whole expert.
3.1 Expert pool Let each sample be represented by three modalityspecific feature vectors: text xt , voice xv , and face xf . The full modality set is M = {t, v, f }. For each non-empty subset S ⊆ M , an expert hS may be trained on the concatenated feature vector xS . In the full configuration, the expert pool is
E = {ht , hv , hf , htv , htf , hvf , htvf }.
e 1 X |ϕe,j (x)|, de j=1
(1)
The paper also evaluates smaller pools, including a unimodal-only pool and a compact pool containing the three unimodal experts plus the trimodal expert. All experts are implemented as XGBoost classifiers. Each expert returns a class-probability vector pe (y | x). Early fusion is the special case in which only the trimodal classifier htvf is used. Late fusion is implemented as the unweighted average of unimodal output probabilities. The proposed framework generalises both by allowing a pool of unimodal and cross-modal experts to be combined through SHAP-derived weights. From the perspective of recent MER taxonomies, this design is best viewed as an interpretable expert-level fusion layer rather than an end-to-end modellevel fusion architecture: it keeps the representations and experts modular, but it uses sampledependent attribution evidence to combine their predictions [3].
4
4 Experimental Setup
Algorithm 1 SHAP-weighted cross-modal expert fusion Require: Training, validation, and test splits with modality features xt , xv , xf ; expert pool E ; temperature grid T ; SHAP reduction mode r ∈ {mean, median, sum}. 1: for each expert e ∈ E do 2: Train XGBoost classifier he on its modality subset. 3: Compute validation probabilities and TreeSHAP attributions. 4: end for 5: for each τ ∈ T do 6: Convert SHAP attributions to expert scores using reduction r. 7: Compute softmax expert weights and fused validation probabilities. 8: Record validation weighted-F1. 9: end for ∗ 10: Select τ with the best validation weightedF1. 11: Apply the same experts, reduction mode, and τ ∗ to the test set. 12: return fused test predictions and diagnostic expert weights.
4.1 Datasets and feature representations Table 1 summarises the two experimental settings. MELD is used for seven-class emotion recognition, and CMU-MOSEI is used for three-class sentiment recognition. The filtering counts refer to samples for which all required modality features were available. For MELD, text features are BERT-base [CLS] embeddings [17], voice features are wav2vec 2.0based embeddings [18], and face features are obtained from a face-emotion pipeline based on frame sampling, face detection, and an EmoAffectNet-style visual backbone [19]. Fifteen faceframe embeddings are aggregated by mean pooling, a BiLSTM aggregator, or a Transformer aggregator. These three variants test whether the fusion conclusions depend on the temporal face aggregator. For CMU-MOSEI, the experiments use preprocessed aligned features with 768-dimensional text vectors, 74-dimensional acoustic vectors, and 35-dimensional visual vectors. The original sentiment labels are mapped to a three-class setting. The purpose of this dataset is not to claim a new state-of-the-art result, but to test whether the SHAP-reduction behaviour observed on MELD transfers to a second multimodal benchmark.
3.3 Temperature-scaled fusion Given expert scores ae (x), the gate assigns a softmax weight
we (x; τ ) = P
exp(ae (x)/τ ) , exp(ae′ (x)/τ )
(5)
4.2 Baselines and evaluated variants
e′ ∈E
The following methods are compared: where τ > 0 is selected on the validation set. The final probability vector is
p(y | x) =
X
we (x; τ )pe (y | x).
• Text-only XGBoost: a strong unimodal baseline using only text features. • Early fusion: one XGBoost classifier trained on the concatenation of all available modality features. • Late fusion: probability averaging of unimodal XGBoost classifiers. • XGAFv1: a unimodal-only Random Forest variant retained as a historical baseline. • XGAFv2 mean-abs: a seven-expert XGBoost pool gated by mean absolute SHAP scores. • XGAFv2 sum-abs: a seven-expert XGBoost pool gated by summed absolute SHAP scores. • XGAFv2 median-abs: a robust variant gated by median absolute SHAP scores, included to test whether a typical-feature statistic
(6)
e∈E
A low temperature produces concentrated weights, while a high temperature approaches uniform averaging. The temperature is tuned by maximising validation weighted-F1.
5
Table 1: Datasets and feature dimensions used in the experiments. Dataset MELD CMU-MOSEI
Task
Train
Val.
Test
Text
Voice / Face
7-class emotion 3-class sentiment
9660 16326
1067 1871
2525 4659
768 768
768 / 15 × 512 74 / 35
improves over mean-abs without allowing highdimensional experts to dominate as strongly as sum-abs. • XGAF-Lite: a compact four-expert pool with three unimodal experts and one trimodal expert.
the same seven-expert pool. This experiment was added to test whether the median of absolute SHAP values is a useful robust alternative to the mean and the sum. The result is clear: median-abs improves only marginally over meanabs (+0.13 percentage points in weighted-F1), while sum-abs improves over median-abs by 2.74 percentage points and matches the early-fusion reference. The entropy diagnostics also separate the reductions. Mean-abs and median-abs produce almost maximum-entropy weights across the seven experts, indicating nearly uniform weighting. Sum-abs produces much lower entropy and assigns a dominant expert on every test sample. Thus, median-abs is a sensible statistical baseline, but it does not solve the expert-scale problem in this architecture.
Performance is reported using accuracy, weighted-F1, macro-F1, and Cohen’s κ. Because MELD is class-imbalanced, weighted-F1 is treated as the main metric. Pairwise prediction differences are evaluated with the exact-binomial version of McNemar’s test using the same test instances.
5 Results 5.1 MELD results Table 2 reports the main MELD results for the Transformer face aggregator. Text-only XGBoost is already a strong baseline, reaching 0.5935 weighted-F1. Early fusion improves this to 0.6018. Probability-average late fusion is much weaker, reaching only 0.4598 weighted-F1, which confirms that naive late fusion loses useful cross-modal structure. The mean-abs version of XGAFv2 reaches 0.5714 weighted-F1, while the sum-abs version improves to 0.5983. The compact XGAFLite variant reaches 0.6013 weighted-F1, essentially matching early fusion. Table 3 shows that the main conclusion is stable across face-sequence aggregators. The sumabs variant consistently reduces the gap to early fusion and improves over mean-abs by 2.52 to 2.90 percentage points in weighted-F1. The three face aggregators are close to each other, suggesting that the fusion behaviour is not driven by a specific face-sequence model. Table 4 then adds the explicit median-abs ablation for the Transformer setup.
5.3 Statistical testing on MELD Table 5 reports pairwise McNemar tests on the MELD test set for the Transformer aggregator. Early fusion and sum-abs XGAFv2 are statistically indistinguishable (p = 1.000). Both early fusion and sum-abs XGAFv2 are significantly better than probability-average late fusion (p < 0.0001). The small difference between text-only and the two strongest multimodal methods is not statistically significant under this test, which again indicates that MELD is strongly textdominated under the present per-utterance setup.
5.4 CMU-MOSEI cross-dataset validation Table 6 reports the corresponding CMU-MOSEI results. Sum-abs XGAFv2 reaches 0.6519 weighted-F1, compared with 0.6485 for early fusion and 0.5696 for late fusion. The advantage over early fusion is small, but McNemar testing gives p = 0.0452. Since the effect size is only 0.34 percentage points in weighted-F1, it should be interpreted as evidence that the method is competitive with early fusion rather than as a large practical improvement.
5.2 Mean-, median-, and sum-abs SHAP ablation Table 4 reports the focused SHAP-reduction ablation for the Transformer face aggregator using 6
Table 2: MELD test results with the Transformer face aggregator. Method
Accuracy
weighted-F1
macro-F1
κ
Text-only XGB Early fusion XGB Late fusion avg. XGAFv1 unimodal RF XGAFv2 mean-abs, all cross XGAFv2 sum-abs, all cross XGAF-Lite sum-abs
0.6257 0.6313 0.5541 0.5743 0.6222 0.6317 0.6309
0.5935 0.6018 0.4598 0.4862 0.5714 0.5983 0.6013
0.4091 0.4090 0.2511 0.2593 0.3747 0.4051 0.4087
0.4292 0.4392 0.2083 0.2535 0.3910 0.4349 0.4384
Table 3: MELD comparison of mean-abs and sum-abs SHAP reductions across face aggregators. Aggregator
Early fusion
XGAF mean-abs
XGAF sum-abs
Gain over mean
Mean pooling BiLSTM Transformer
0.6029 0.6007 0.6018
0.5686 0.5749 0.5714
0.5976 0.6001 0.5983
+0.0290 +0.0252 +0.0269
6 Interpretability and Diagnostic Analysis 6.1 Why mean-abs and sum-abs behave differently Figure 2 summarises the source of the reduction difference. The experts have unequal feature dimensionalities, but mean-abs reduction places each expert on an average-per-feature scale. Median-abs goes further toward a robust typicalfeature interpretation by ignoring extreme attribution values more strongly than the mean. This is attractive statistically, but the explicit ablation in Table 4 shows that it is too conservative for expert gating in the present architecture: it remains close to mean-abs and far below sum-abs in weighted-F1. In the MELD run, the trimodal expert has the largest feature count but a relatively low mean attribution value. The sum-abs reduction preserves total attribution mass and therefore gives larger scores to cross-modal experts when their aggregate contribution is large.
Fig. 1: Expert-pool ablation on MELD. Positive values relative to late fusion show that crossmodal experts recover much of the performance lost by naive late fusion. The four-expert XGAFLite variant is sufficient to match early fusion.
5.5 Expert-pool ablation Table 7 and Fig. 1 show the MELD expertpool ablation. The unimodal-only pool reaches 0.5736 weighted-F1. Adding only the trimodal expert increases performance to 0.6013 weightedF1, which is almost identical to early fusion. Adding all three bimodal experts without the trimodal expert reaches 0.5972, and the full sevenexpert pool reaches 0.5983. These results indicate that the main improvement comes from adding cross-modal experts, especially the trimodal expert, rather than from increasing the number of experts.
6.2 Expert weights do not show rich adaptive routing The most important diagnostic result is shown in Figs. 3 and 4. With the original mean-abs reduction, per-class average weights are almost uniform across the seven experts. The median-abs ablation confirms the same behaviour quantitatively: its average weight entropy is 1.9459, essentially the same as mean-abs (1.9454), and no expert dominates. With the sum-abs reduction, weights 7
Table 4: Focused SHAP-reduction ablation on MELD with the Transformer face aggregator. The table compares mean-, median-, and sum-absolute reductions using the same XGAFv2 expert pool. Method
Accuracy
weighted-F1
macro-F1
κ
Val. weighted-F1
τ∗
Entropy
Dom. rate
Early fusion (reference) XGAFv2 mean-abs XGAFv2 median-abs XGAFv2 sum-abs
0.6265 0.6190 0.6198 0.6269
0.5955 0.5669 0.5682 0.5957
0.4022 0.3696 0.3713 0.4022
0.4299 0.3851 0.3866 0.4304
– 0.5276 0.5266 0.5611
– 0.7 2.5 1.5
– 1.9454 1.9459 0.3912
– 0.0 0.0 1.0
Table 5: Pairwise McNemar tests on MELD, Transformer aggregator. Here b denotes samples on which method A is wrong and method B is correct, while c denotes samples on which A is correct and B is wrong. Comparison Early vs. XGAF sum-abs Early vs. late fusion XGAF sum-abs vs. late fusion Early vs. text-only XGAF sum-abs vs. text-only
Fig. 2: Normalised diagnostic comparison of feature counts, mean-abs SHAP scores, and sumabs SHAP scores on MELD. Mean-abs reduction can make high-dimensional cross-modal experts appear weak on an average-per-feature scale, while sum-abs reduction preserves total attribution mass.
∆ acc.
b
c
pexact
+0.0004 -0.0772 -0.0776 -0.0055 -0.0059
36 146 125 131 110
35 341 321 145 125
1.0000 < 0.0001 < 0.0001 0.4340 0.3611
Fig. 3: Average expert weights per MELD emotion class using mean-abs SHAP reduction. The weights are close to uniform, so the gate behaves similarly to simple averaging.
7 Discussion 7.1 What the results support
become strongly concentrated on the trimodal expert, with only a small residual weight assigned to the text-voice expert and other experts. The per-sample entropy analysis in Fig. 5 leads to the same conclusion. Sum-abs reduces entropy to 0.3912 and assigns the argmax expert to the trimodal expert for all MELD test samples. This diagnostic result changes the interpretation of the method. The sum-abs gate is empirically useful, but the present formulation should not be described as a strongly adaptive per-sample router. It is better described as a SHAP-weighted expert mixture in which cross-modal expert dominance is selected automatically under the evaluated conditions.
The results support three conclusions. First, late fusion by probability averaging is not sufficient in this setting. It underperforms early fusion by 14.2 percentage points in weighted-F1 on MELD with the Transformer face aggregator and by 7.89 percentage points on CMU-MOSEI. Second, adding cross-modal experts to a modular expert pool recovers early-fusion-level performance. Third, if SHAP attributions are used as a gate signal across experts of unequal dimensionality, the reduction rule must be stated and justified. Mean-abs asks how strong the average feature is, median-abs asks how strong the typical robust feature is, and sum-abs asks how much total evidence the expert
8
Table 6: CMU-MOSEI 3-class sentiment results. Method
Accuracy
weighted-F1
macro-F1
κ
Text-only XGB Early fusion Late fusion avg. XGAFv2 mean-abs XGAFv2 sum-abs
0.6649 0.6729 0.6280 0.6622 0.6798
0.6396 0.6485 0.5696 0.6203 0.6519
0.5871 0.5984 0.4995 0.5604 0.5996
0.4369 0.4497 0.3350 0.4124 0.4592
Table 7: Expert-pool ablation on MELD, Transformer aggregator, sum-abs reduction. Variant
Experts
weighted-F1
∆ early
∆ late
3 4 4 4 4 6 7
0.5736 0.6013 0.5924 0.5870 0.4571 0.5972 0.5983
-0.0282 -0.0005 -0.0095 -0.0149 -0.1448 -0.0046 -0.0036
+0.1138 +0.1415 +0.1326 +0.1272 -0.0027 +0.1374 +0.1385
Unimodal only + trimodal (XGAF-Lite) + text-voice only + text-face only + voice-face only + all three bimodal + trimodal + all bimodal
Fig. 5: Per-sample entropy of the expert-weight vector on MELD. Mean-abs reduction collapses to near-uniform weights. Sum-abs reduction lowers entropy, but the class distributions still overlap strongly and the trimodal expert remains dominant.
Fig. 4: Average expert weights per MELD emotion class using sum-abs SHAP reduction. The gate becomes concentrated on the trimodal expert, which explains the recovery of earlyfusion-like performance but also shows limited routing diversity.
expert. Therefore, the performance gain over late fusion should not be attributed primarily to a sophisticated per-sample routing mechanism. It is more accurately attributed to the inclusion of cross-modal experts combined with a reduction rule that allows those experts to receive adequate weight.
carries. The new median-abs ablation shows that robustness to outlying attributions is not enough: median-abs behaves almost like mean-abs and remains about 2.74 percentage points below sumabs in weighted-F1. For the present architecture, preserving total attribution mass with sum-abs is therefore more suitable than either mean-abs or median-abs.
7.3 Practical implication For deployment, the four-expert XGAF-Lite configuration is the most attractive variant in the current experiments. It keeps the modular unimodal predictors, includes a trimodal expert to capture cross-modal interactions, and matches early-fusion performance on MELD. The full seven-expert pool adds complexity without improving weighted-F1 in the reported MELD ablation. This suggests that, in settings where computational cost and
7.2 What the results do not support The results do not support a strong claim of rich adaptive routing across emotions or samples. The diagnostic logs show that mean-abs and median-abs weighting are almost uniform and that sum-abs weighting is dominated by the trimodal 9
to incomplete, degraded, or asynchronous modalities as a core open problem [3]. Fifth, the current SHAP gate does not produce diverse per-sample routing. Sixth, the median-abs ablation has now been evaluated only for the MELD Transformer setting; it should still be repeated for the other aggregators, CMU-MOSEI, and additional seeds before broader claims about robust SHAP reductions are made. Future work should therefore test calibrated gates, hard routing, learned temperature functions, per-class priors, entropy regularisation, contrastive alignment losses, uncertaintyaware decision layers, and explicit robustness objectives. A particularly important next experiment is to evaluate whether unimodal and bimodal experts become useful when one modality is deliberately masked, corrupted, or made unavailable at test time.
interpretability matter, a small expert pool may be preferable to a larger one.
7.4 Position relative to current MER trends Recent MER surveys describe a shift toward transformer-based cross-modal attention, contrastive alignment, foundation-model encoders, dynamic modality weighting, and explicit robustness mechanisms for incomplete or noisy modalities [3]. The present work does not compete with those end-to-end neural architectures. Its role is complementary: it asks whether a lightweight, tree-based, and explanation-derived fusion layer can recover the benefit of cross-modal modelling while preserving expert modularity. The results suggest that this is possible under clean benchmark conditions, but they also show that explanation-derived weights may collapse to crossmodal expert dominance. This observation gives a practical diagnostic criterion for future adaptive MER systems: performance should be accompanied by weight-diversity, entropy, and missingmodality analyses, not by aggregate accuracy alone.
9 Conclusion This paper revisited SHAP-weighted cross-modal expert fusion for multimodal emotion and sentiment recognition. The main empirical finding is that the choice of SHAP attribution reduction matters. Mean-abs reduction can collapse the gate toward nearly uniform weights when experts have unequal feature dimensionalities, medianabs provides a robust typical-feature alternative but empirically behaves similarly to mean-abs on MELD, and sum-abs reduction restores crossmodal expert influence and closes the performance gap to early fusion. On MELD, sum-abs XGAFv2 is statistically indistinguishable from early fusion and significantly better than naive late fusion. On CMU-MOSEI, it produces a small statistically significant improvement over early fusion and a large improvement over late fusion. At the same time, diagnostic analysis shows that the current gate is not a rich adaptive router: its useful behaviour is largely cross-modal expert dominance. The paper therefore argues for a restrained interpretation of the method. Its value lies in showing how attribution reduction and expert-pool design affect modular multimodal fusion, and in identifying the conditions under which the method behaves like early fusion rather than a genuinely adaptive routing system. In the broader MER landscape, this supports a more diagnostic style of evaluation in which accuracy is reported together with fusion
8 Limitations and Future Work Several limitations should be emphasised. First, the experiments use pre-extracted features rather than end-to-end fine-tuning. This makes the comparison reproducible and computationally manageable, but it limits direct comparison with the strongest neural architectures, including transformer- and foundation-model-based MER pipelines. Second, the MELD setup is perutterance and does not model dialogue context, speaker state, or conversation graphs. Dialogueaware models may achieve higher benchmark scores because they solve a richer temporal problem. Third, the CMU-MOSEI improvement over early fusion is statistically significant but numerically small. It should be replicated with additional seeds, bootstrap confidence intervals, and additional datasets such as IEMOCAP before being interpreted as a robust practical improvement. Fourth, the present experiments do not include missing-modality or noisy-modality stress tests, although recent MER literature treats robustness
10
behaviour, attribution scale, expert diversity, and robustness-oriented tests.
[4] Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L.-P.: Tensor fusion network for multimodal sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103–1114 (2017)
Acknowledgements. The authors should insert funding, supervision, institutional, and computational-resource acknowledgements here, if applicable. Use of generative AI. An initial draft of this manuscript was prepared with assistance from a large language model. The present version was substantially revised by the authors, and the authors take responsibility for the experimental results, interpretation, references, and final scientific claims.
[5] Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A.B., Morency, L.-P.: Efficient low-rank multimodal fusion with modality-specific factors. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2247–2256 (2018)
Data and code availability. The experiments use publicly available benchmark datasets or preprocessed representations derived from them. The project code and reproducibility scripts should be released with the preprint or made available in a public repository, subject to the licenses of the underlying datasets.
[6] Pham, H., Liang, P.P., Manzini, T., Morency, L.-P., Poczos, B.: Found in translation: Learning robust joint representations by cyclic translations between modalities. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 6892–6899 (2019)
Competing interests. The authors declare that they have no competing interests. Replace this statement if needed.
[7] Tsai, Y.-H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.-P., Salakhutdinov, R.: Multimodal transformer for unaligned multimodal language sequences. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6558–6569 (2019)
References [1] Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., Mihalcea, R.: MELD: A multimodal multi-party dataset for emotion recognition in conversations. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 527–536 (2019)
[8] Hazarika, D., Zimmermann, R., Poria, S.: MISA: Modality-invariant and -specific representations for multimodal sentiment analysis. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 1122– 1131 (2020)
[2] Zadeh, A.B., Liang, P.P., Poria, S., Cambria, E., Morency, L.-P.: Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2236–2246 (2018)
[9] Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., Cambria, E.: DialogueRNN: An attentive RNN for emotion detection in conversations. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 6818–6825 (2019)
[3] Yazici, A., Kucukyilmaz, T., Dokeroglu, T., Sharipbay, A., Lee, M.-H., Tyler, B.: State-of-the-art multimodal emotion recognition: A comprehensive survey and taxonomy. Intelligent Systems with Applications 30, 200642 (2026) https://doi.org/10.1016/j. iswa.2026.200642
[10] Ghosal, D., Majumder, N., Poria, S., Chhaya, N., Gelbukh, A.: DialogueGCN: A graph convolutional neural network for emotion recognition in conversation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pp. 154–164 (2019) 11
In: Advances in Neural Information Processing Systems, vol. 33, pp. 12449–12460 (2020)
[11] Ghosal, D., Majumder, N., Gelbukh, A., Mihalcea, R., Poria, S.: COSMIC: COmmonSense knowledge for eMotion identification in conversations. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2470–2481 (2020)
[19] Ryumina, E., Dresvyanskiy, D., Karpov, A.: In search of a robust facial expressions recognition model: A large-scale visual crosscorpus study. Neurocomputing 514, 435–450 (2022)
[12] Chudasama, V., Kar, P., Gudmalwar, A., Shah, N., Wasnik, P., Onoe, N.: M2FNet: Multi-modal fusion network for emotion recognition in conversation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 4652–4661 (2022) [13] Chen, T., Guestrin, C.: XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016) [14] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems, vol. 30 (2017) [15] Yu, W., Xu, H., Yuan, Z., Wu, J.: Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 10790–10797 (2021) [16] Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In: International Conference on Learning Representations (2017) [17] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 4171–4186 (2019) [18] Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: A framework for selfsupervised learning of speech representations.
12