VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation Linan Zhu1 , Zihao Zhai1 , Xiao Han1,* Yuqian Fu2 , Xiangfan Chen1 , Xiangjie Kong1 , Guojiang Shen1
arXiv:2605.18547v1 [cs.AI] 18 May 2026
1
Zhejiang University of Technology 2 ETH Zurich, Switzerland
Abstract Emotion Recognition in Conversation (ERC) is essential for effective humanmachine interaction, aiming to identify speakers’ emotional states in multi-turn dialogues. Early text-based methods struggle with complex scenarios like sarcasm because they inherently neglect vital non-verbal information. While recent Vision-Language Models (VLMs) address this by analyzing video directly, they are not inherently tailored for ERC and often focus on emotionally irrelevant background regions or passive listeners rather than the active speaker. Furthermore, fine-tuning these large models incurs prohibitive computational costs. Additionally, isolated visual signals are frequently ambiguous or technically compromised without the context of linguistic content and vocal prosody. To address these challenges, we propose VISAFF, a speaker-centered VISual AFFective feature learning framework for ERC. VISAFF consists of two stages: SpeakerCentered Affective Grounding and Reliability-Guided Affective Complementation. VISAFF utilizes a tuning-free approach to unlock the reasoning capabilities of frozen VLMs, efficiently steering them to focus on the active speaker’s emotional visual cues without heavy training overheads. In the second stage, we introduce a reliability-guided affective complementation mechanism that dynamically leverages textual and acoustic modalities to compensate for visual uncertainty. Experiments on two real-world datasets demonstrate that VISAFF achieves highly competitive performance compared to state-of-the-art methods in a tuningfree setting, significantly enhancing computational efficiency by eliminating the need for expensive fine-tuning of large VLMs. The source code is available at https://anonymous.4open.science/r/speaker-2365/.
1
Introduction
With the development of visual sensing platforms such as smart glasses, in-vehicle cameras, and embodied agents, machines can continuously observe facial expressions, body movements, and interactive behaviors in real-world interactions [35, 7]. However, effective human-machine interaction requires not only understanding observable behaviors, but also inferring human affective states to support contextually appropriate responses and decisions [3, 31]. Consequently, Emotion Recognition in Conversation (ERC) has emerged to address this need, aiming to identify the emotional state of each speaker at each utterance in multi-turn dialogues [27, 4]. Early ERC methods rely on textual signals to capture contextual semantics and emotional expressions [10, 16]. While these approaches are straightforward, often inferring emotional states directly from key lexical cues [26, 22], they inherently neglect vital non-verbal information. Crucial emotional indicators such as facial expressions, vocal prosody (tone of voice), and body gestures are completely overlooked. This limitation becomes particularly evident in complex scenarios where a speaker
(b) Context-Dependent Visual Cues
(a) Need for Task-Guided Visual Grounding Unguided VLM Visual Focus
Visual cue : Smile
Background region
Context : Fake smile
Non-target speaker
Real emotion : Neutral
Irrelevant gesture
A smile may not reflect the true emotion without context.
Salient regions may not match the speaker's affective cues.
Figure 1: Motivation of speaker-centered visual affective feature learning for ERC. (a) Need for task-guided visual grounding: without explicit ERC-oriented guidance, visually salient regions may not match the speaker’s affective cues. (b) Context-dependent visual cues: visual expressions can be ambiguous without dialogue and acoustic context. deliberately masks their true intentions or uses sarcasm [5, 28], causing text-only methods to fail in accurately extracting the underlying affective state. Existing research has begun to directly analyze videos for Emotion Recognition in Conversation (ERC), especially with the emergence of Vision-Language Models (VLMs) [35, 7, 3]. These models break the limitations of traditional approaches that either suffer from information loss by narrowly focusing solely on localized facial expressions [6, 34] or struggle with background noise and nontarget speaker interference when using global video representations [12, 8, 31]. However, VLMs are not inherently tailored for ERC. They do not automatically fixate on the current speaker’s emotionspecific regions [3, 35, 7]. As illustrated in Figure 1(a), their attention is often distracted by visually salient but emotionally irrelevant areas, such as dynamic backgrounds or passive listeners. While fine-tuning or parameter-efficient adaptations (e.g., LoRA) can steer VLMs toward relevant visual cues [13, 11], these training paradigms incur prohibitive computational costs and demand extensive annotated resources. This brings us to our first core challenge: How can we unlock the latent ERC reasoning capabilities of VLMs using a tuning-free approach, enabling them to efficiently and precisely attend to the active speaker’s emotional visual cues? Beyond isolated visual extraction, affective cues must be contextualized within a unified multimodal framework to ensure robust interpretation. As demonstrated in Figure 1(b), a facial expression such as a smile is not strictly indicative of positive affect. Contextual nuances may reveal it to be a superficial facade, mapping instead to a neutral or negative underlying state. This ambiguity underscores that visual signals are highly sensitive to accompanying linguistic content, vocal prosody, and conversational history. Furthermore, real-world video streams are frequently compromised by technical artifacts like motion blur, rendering purely visual inference unreliable. Addressing these multimodal dependencies introduces our second challenge: How can we design an adaptive complementation mechanism that dynamically leverages textual and acoustic modalities to compensate for the uncertainty and unreliability of visual cues? To address these challenges, we propose VISAFF, a two-stage speaker-centered VISual AFFective feature learning framework for ERC. Specifically, we leverage a tuning-free approach in the first stage to activate the inherent ability of the VLM to associate the current speaker with emotion-related visual cues, seamlessly circumventing huge training overheads. Moreover, to enhance the visual stream, we design a reliability-guided affective complementation mechanism in the second stage that adaptively integrates auxiliary text and audio information. By gating these modalities based on visual reliability, VISAFF significantly improves the model’s overall robustness. This innovative approach provides a highly efficient framework for multimodal reasoning, making it an effective solution for dynamic dialogue scenarios. The main contributions of this paper are summarized as follows: • We propose VISAFF, a speaker-centered visual affective feature learning framework for ERC that activates frozen VLMs for visual affective reasoning without ERC-specific fine-tuning, and further enhances visual representations with textual and acoustic context. • We introduce SCAG and RGAC: SCAG performs zero-shot speaker-centered affective grounding with frozen VLMs, while RGAC adaptively complements visual affective features with text and audio according to visual reliability. • Experiments on two real-world ERC datasets demonstrate the effectiveness of VISAFF, supported by ablation studies and confidence-binned analyses. 2
2
Related Work
ERC has been widely studied through contextual modeling and multimodal learning. Early methods mainly focus on textual context modeling, including sequential utterance modeling, speaker-state tracking, graph-based speaker relation modeling, and pretrained language representations [26, 22, 10, 14, 19, 29, 17, 16]. Later multimodal methods introduce acoustic and visual signals through dynamic fusion, self-distillation, hypergraph modeling, adaptive graph learning, and evidence-cause reasoning [12, 8, 20, 37, 33, 1, 38]. Recent large-model-based methods further explore LLMs for ERC. DialogueLLM adapts LLMs for conversational emotion reasoning [39], LaERC-S introduces speaker-related cues for zero-shot ERC [9], and SpeechCueLLM converts acoustic cues into textual prompts for speech-aware reasoning [36]. These methods demonstrate the potential of large models for ERC, but they mainly rely on language-centered or speech-aware reasoning. For visual modeling, existing methods usually rely on face-centric pipelines, such as active speaker localization, face cropping, action unit extraction, and facial affective encoding [31, 6, 34, 2, 30, 32, 40], or extract global video features with pretrained visual/video encoders [12, 8, 20, 1, 38]. Face-centric methods provide fine-grained local affective cues, while global video features preserve broader scene information. Recent vision-language models, such as Qwen2-VL and InternVL, provide stronger full-frame understanding, visual-language alignment, and instruction-following abilities [35, 7], making them promising for visual affective modeling in ERC. Different from prior multimodal fusion and large-model-based ERC methods, VISAFF focuses on speaker-centered visual affective grounding with a frozen VLM, and uses textual and acoustic information as reliability-aware complements for uncertain visual cues.
3
Method
We propose VISAFF, a two-stage speaker-centered visual affective feature learning framework for ERC. Given a conversation U = {u1 , . . . , uN }, each utterance ui is associated with a video clip Vi , a transcript Ti , and an audio segment Ai . The goal is to predict an emotion label yi ∈ Y for each utterance. As shown in Figure 2, VISAFF consists of two stages: Speaker-Centered Affective Grounding and Reliability-Guided Affective Complementation. The first stage uses a frozen VLM as a zero-shot inference-time feature extractor and guides it with two input groups to extract speakercentered visual affective features: Prompt-Guided VLM Inputs (PGVI) and Affective Semantic Guidance Inputs (ASGI). The second stage uses visual features to retrieve textual and acoustic affective references, and adaptively injects their residual complements according to visual reliability. 3.1
Speaker-Centered Affective Grounding
VLMs possess strong visual understanding capabilities and can recognize persons and salient visual regions from video frames. However, due to the lack of explicit speaker constraints and affective task guidance, their general visual understanding ability cannot be directly translated into the visual affective reasoning required by ERC. To avoid costly task-specific VLM training, we propose SCAG. First, Prompt-Guided VLM Inputs (PGVI) provide visual inputs and task formulation for speakercentered affective grounding. For each utterance video, we sample several frames from the current clip, denoted as Fi , and use the target speaker reference image Ri as a localization constraint. The task prompt Ptask,i explicitly instructs the frozen VLM to perform speaker-centered visual affective analysis rather than generic scene description, while Ri helps the model identify the current speaker in full video frames. Thus, PGVI is defined as: IPGVI,i = {Fi , Ri , Ptask,i }.
(1)
While IP GV I,i provides the identity and spatial constraints to locate the speaker, isolated visual frames lack the necessary emotional context for fine-grained reasoning. To address this, we then introduce Affective Semantic Guidance Inputs (ASGI) to provide semantic cues that steer the VLM’s attention toward emotion-relevant visual patterns. PASGI,i = Pctx (T≤i ) ⊕ Paud (Ai ) ⊕ Pvad (Ti ), 3
(2)
Stage 1: Speaker-Centered Affective Grounding Prompt-Guided VLM Inputs
Video Frames
Frozen VLM-Embedding-Model
Affective Semantic Guidance Inputs Dialogue Context
A: What’s wrong? B: Nothing.
Speaker-Centered Visual Affective Feature
Step 1: Speaker Grounding
Visual Context Encoder
Speaker Reference Task Setting
Stage 2 : Reliability-Guided Affective Complementation
Speaker
Speaker Region
𝐻𝑣
Speaker Reference Reference
Head
Body
Hand
VAD Priors valence: 0 ~ 1 arousal: 0 ~ 1 dominance: 0 ~ 1
Q(𝐻𝑣 )
Key(𝐻𝑡 )
𝐻𝑡
Value
෪𝑡 𝐻
Audio Feature
Text Context Encoder 𝐻𝑡
Audio Context Encoder 𝐻𝑎
Reliability Gate
Speaker-Centered Visual Affective Feature
Q(𝐻𝑣 )
Key(𝐻a )
𝐻𝑎
Value
෪𝑎 𝐻
Residual Affective Update ( Δ𝑣 )
Visual Confidence (𝑐𝑣 )
Audio Description pitch: low timbre: tense
Text Feature
Visual-Guided Cross-Modal Retrieval
Step 2: Affective Cue Extraction
Face
External Affective References
𝐻𝑣∗
Concat ෪𝑡 ; ෪ [ 𝐻𝑣∗ ; 𝐻 𝐻𝑎 ]
MLP
Figure 2: Overall architecture of VISAFF. Stage 1, Speaker-Centered Affective Grounding, uses a frozen VLM to extract speaker-centered visual affective features under the joint guidance of PGVI and ASGI. Stage 2, Reliability-Guided Affective Complementation, retrieves textual and acoustic affective references using visual features, and adaptively injects residual complements according to visual reliability for emotion prediction. where Pctx , Paud , and Pvad denote the dialogue context prompt, audio description prompt, and lexical VAD prompt, respectively [23]. The complete visual affective analysis prompt is composed of the task prompt and ASGI: Pi = Ptask,i ⊕ PASGI,i ,
(3)
where ⊕ denotes prompt-level concatenation. Given PGVI and ASGI, we use a frozen VLM-based embedding model as an inference-time visual affective feature extractor: vi = Poollast VLMfrozen (Fi , Ri , Pi ) , vi ∈ R4096 , (4) θ where θ denotes the fixed parameters of the frozen VLM. No gradient is back-propagated to VLMfrozen , and no LoRA or ERC-specific adaptation is used. θ Through this design, PGVI preserves the complete visual scene while reducing distractions from emotion-irrelevant regions such as backgrounds and non-target speakers. ASGI further helps the model more accurately localize visual cues related to the current affective state. In this way, SCAG activates the latent ERC-oriented visual affective reasoning ability of the frozen VLM without training it. 3.2
Reliability-Guided Affective Complementation
After SCAG, the model obtains speaker-centered visual affective features. However, visual cues may still be ambiguous or unreliable in conversational contexts. The same visual expression may convey different emotions depending on linguistic content, vocal prosody, and dialogue history; meanwhile, occlusion, motion blur, or subtle movements may further weaken purely visual inference. Therefore, visual affective features still need to be interpreted with textual and acoustic context. To address this challenge, we propose RGAC. First, RGAC retrieves textual and acoustic affective references under the guidance of the visual state. We project visual, textual, and acoustic features into a shared hidden space. The visual branch uses the speaker-centered visual affective feature vi obtained from Stage 1, the text branch uses 4
RoBERTa-large features [16], and the audio branch uses emotion2vec-large features [21]. For each modality, a lightweight context encoder models only the past dialogue history up to the current utterance, avoiding future utterance leakage. (v)
(t)
(a)
Let hi , hi , and hi denote the tokenized visual, textual, and acoustic states, respectively. Using the visual state as the query, we retrieve textual and acoustic affective references by (m) (v) (m) (m) h̃i = CrossAttn Q = hi , K = hi , V = hi , m ∈ {t, a}. (5) In this way, text and audio are not treated as independent dominant modalities, but are retrieved as visual-guided external affective references for interpreting the contextual meaning of current visual cues. Second, RGAC controls the strength of textual and acoustic complementation according to visual reliability. We compute a residual affective complement from the discrepancy between the retrieved external references and the current visual state: (t) (v) (a) (v) ∆i = MLPδ [h̃i − hi ; h̃i − hi ] . (6) This residual represents the complementary affective information that text and audio can provide to the current visual state. Let ci denote the visual reliability score. The complemented visual representation is computed as: (v)∗
hi
(v)
= hi
+ (1 − ci )∆i .
(7)
Thus, when visual cues are reliable, the external residual complement is suppressed; when visual cues are uncertain, textual and acoustic references provide stronger complementation. The complete reliability-guided affective complementation process consists of visual-guided affective reference retrieval and reliability-aware residual complementation. Finally, we concatenate the complemented visual representation with the retrieved textual and acoustic affective references for emotion prediction: (v)∗ (t) (a) oi = MLPcls (hi ). (8) hi = [hi ; h̃i ; h̃i ], 3.3
Learning Objectives
The framework is optimized with three objectives. The classification loss Lcls supervises the final emotion prediction. The auxiliary visual loss Laux supervises the video-only classifier, stabilizing the visual reliability score ci and preserving the emotion-discriminative ability of the visual branch. Specifically, ci is obtained as the maximum softmax probability of the video-only classifier. The semantic alignment loss Lcl regularizes the multimodal feature space before cross-modal interaction using InfoNCE and supervised contrastive learning [25, 15]. The overall objective is: L = Lcls + λcl Lcl + λaux Laux ,
(9)
where λcl and λaux control the weights of the semantic alignment loss and the auxiliary visual loss, respectively. Theorem 1 (Generalization Error Bound for Reliability-Guided Affective Complementation). Assume that the loss function ℓ : RK × Y → [0, M ] is bounded, convex, and L-Lipschitz continuous. Let hv ∈ Hv denote the visual predictor, and let haux ∈ Haux denote the externally complemented predictor induced by textual and acoustic residuals. Let c : Xv → [0, 1] be the visual reliability function. For the reliability-aware fused predictor hfuse (X) = c(X)hv (Xv ) + (1 − c(X))haux (X), the expected risk R(hfuse ) admits the following decomposition: R(hfuse ) ≤ E[c(X)]R(hv ) + E[1 − c(X)]R(haux ) + Cov(c(X), lv (X)) − Cov(c(X), laux (X)) (10) where lv (X) = l(hv (Xv ), y) and laux (X) = l(haux (X), y). Furthermore, for any δ ∈ (0, 1), over an i.i.d. training set S ∼ Dn , with probability at least 1 − δ, the expected risk is bounded by its empirical risk R̂S (hfuse ) and the Rademacher complexity Rn (Hfuse ): r R(hfuse ) ≤ R̂S (hfuse ) + 2LRn (Hfuse ) + 3M 5
ln(2/δ) , 2n
(11)
where ℓv (X) = ℓ(hv (Xv ), y) denotes the visual prediction loss, ℓaux (X) = ℓ(haux (X), y) denotes the loss of the externally complemented predictor, and Rn (Hfuse ) is the Rademacher complexity of the fused hypothesis class. Theorem 1 supports the reliability-guided design of RGAC. When visual confidence is negatively correlated with visual loss, the term Cov(c(X), ℓv (X)) helps tighten the bound. Meanwhile, when textual and acoustic references are more useful under low visual confidence, Cov(c(X), ℓaux (X)) becomes positive, and the negative covariance term further reduces the upper bound. This dynamic theoretically supports the strategy of leveraging external affective modalities primarily when visual cues lack certainty. 3.4
Training Procedure
The training procedure of VISAFF is summarized in Algorithm 1. Given a frozen VLM VLMfrozen , the VLM parameters θ are θ fixed throughout the whole procedure, and no gradient is backpropagated to the VLM. In practice, VISAFF first performs an offline Stage-1 feature extraction step. For each utterance ui ∈ D, we sample video frames Fi from Vi and construct the Stage-1 inputs, including IPGVI , PASGI , and the complete prompt P . These inputs guide VLMfrozen to extract the speaker-centered θ visual affective feature vi , which is cached for subsequent training (Lines 1–5). After this offline extraction, the downstream modules are trained on the cached visual features together with the textual and acoustic features. For each mini-batch B ⊂ D, the trainable encoders produce contextual states H v , H t , and H a , and the visual-guided cross-modal module retrieves textual and e t and H e a . Based on these referacoustic affective references H ences, VISAFF computes the residual complement ∆, the visual reliability score c, and the complemented visual representation H v∗ (Lines 11–12). Finally, the classifier predicts emotion logits O, the training objective L is computed, and only the downstream parameters Φ are updated by back-propagation with respect to L, while θ remains frozen (Lines 13–14).
4
Experiments
4.1
Experimental Setup
Algorithm 1: Offline Feature Extraction and Downstream Training of VISAFF Input: Frozen VLMfrozen , training set D, epochs E θ Output: Trained downstream parameters Φ 1: Freeze θ; initialize Φ 2: for each utterance ui ∈ D do 3: Fi ← Sample(Vi ) 4: IPGVI , PASGI , P ← Eqs. (1)–(3) 5: vi ← Poollast VLMfrozen (IPGVI , P ) ; cache vi θ 6: end for 7: for e = 1 to E do 8: for each mini-batch B ⊂ D do 9: Load cached vB , TB , and AB 10: H v , H t , H a ← Enc(vB , TB , AB ) e t, H e a ← Eq. (5) 11: H 12: ∆, c, H v∗ ← Eqs. (6)–(7) 13: O ← Eq. (8); L ← Eq. (9) 14: Φ ← Update(Φ, ∇Φ L) 15: end for 16: end for 17: return Φ
We evaluate the proposed framework on two widely used ERC benchmarks: MELD [27] and IEMOCAP [4]. We follow the official training, validation, and test splits. Besides, we use the weighted F1 score (W-F1) as the primary metric. We additionally report per-class F1 in the main comparison. For our method, we use Qwen3-VL-Embedding as the backbone VLM. The VLM is kept frozen and is not fine-tuned. It is used only as an inference-time feature extractor to obtain target-speaker visual affective representations. All experiments of our method are conducted on a single NVIDIA RTX 4090 GPU, and the final results are averaged over five random seeds. 4.2
Main Results
Table 1 compares the proposed method with representative ERC methods, including context modeling methods, multimodal ERC methods, and large-model-based ERC methods. For large-model-based methods, we focus on settings where the large model is not fine-tuned or LoRA-adapted for ERC, since VISAFF keeps the large VLM frozen and uses it only as an inference-time visual feature extractor. On both IEMOCAP and MELD, VISAFF achieves the best W-F1 among traditional ERC methods and large-model-based methods without ERC-specific fine-tuning or LoRA adaptation. Notably, this performance is obtained without training or LoRA-adapting the large VLM for ERC. These results indicate that SCAG can effectively elicit visual affective representations from frozen VLMs 6
Table 1: Main results on IEMOCAP and MELD. Best results are in bold, and second-best results are underlined. IEMOCAP Method Name
MELD
LM FT? Happy Sad Neutral Angry Excited Frustrated W-F1 Neutral Surprise Fear Sadness
Joy
Disgust Anger W-F1
Context Modeling ERC Methods bc-LSTM (2017) [26] DialogueGCN (2019) [10] EmoBERTa (2021) [16]
– – –
34.40 60.80 42.75 84.54 56.40 83.00
51.80 63.54 61.50
56.70 64.19 69.60
57.90 63.08 78.00
58.90 66.99 68.70
54.90 64.18 69.90
73.80 72.10 82.50
47.70 41.70 50.20
5.40 2.80 1.90
25.10 21.80 31.20
51.30 44.20 61.70
5.20 6.70 2.50
38.40 36.50 46.40
55.80 52.80 63.30
Multimodal ERC Methods MM-DFN (2022) [12] M2FNet (2022) [8] SDT (2024) [20] HAUCL (2024) [37] AdaGIN (2024) [33] DER-GCN (2024) [1] ECERC (2025) [38]
– – – – – – –
42.22 60.00 66.19 54.30 53.00 58.80 60.86
78.98 82.11 81.84 81.85 81.50 79.80 79.28
66.42 65.88 74.62 68.24 71.30 61.50 71.95
69.77 68.21 69.73 65.90 65.90 72.10 66.27
75.56 72.60 80.17 77.03 76.30 73.30 78.29
66.33 68.31 68.68 64.52 67.80 67.80 68.25
68.18 69.86 74.08 69.56 70.70 68.80 71.78
77.38 80.06 80.19 79.11 79.80 80.60 79.80
57.19 58.66 59.07 59.27 60.50 51.00 58.98
14.29 3.45 17.88 19.18 15.20 10.40 26.12
41.36 47.03 43.69 41.11 43.70 41.50 40.95
63.25 65.50 64.29 62.93 64.50 64.30 64.95
20.41 25.24 28.78 22.00 29.30 10.30 31.43
51.44 55.25 54.33 52.89 56.20 57.40 53.89
64.04 66.71 66.60 65.35 66.80 65.50 66.46
Large-Model-Based ERC Methods OmniVox (2025) [24] SpeechCueLLM (2025) (zero-shot) [36] LaERC-S (2025) (zero-shot) [9] DialogueLLM (2025) (w/o LoRA) [39]
No No No No
– – – –
– – – –
– – – –
– – – –
– – – –
– – – –
47.60 59.23 40.07 63.78
– – – –
– – – –
– – – –
– – – –
– – – –
– – – –
– – – –
61.70 – 54.37 64.42
Ours VISAFF
No
82.47
75.94
79.93
70.70
77.30∗
80.63
58.84
25.26
47.44
64.68
22.06
61.13 86.61
54.32 67.12∗
Note. “LM FT?” indicates whether a large model is fine-tuned or LoRA-adapted for ERC. “–” indicates that the corresponding result is not reported. ∗ denotes significant improvement over the best baseline (p < 0.05, paired t-test).
for conversational emotion recognition, while RGAC further supplements visual information when visual cues are uncertain. Overall, the results show that VISAFF achieves strong performance on two widely used ERC benchmarks while avoiding ERC-specific large-model training. The ablation study in Table 2 and the visual feature analysis in Table 3 further support the contributions of SCAG and RGAC. 4.3
Component Analysis
We analyze the key components of VISAFF in Tables 2 and 3. VISAFF consists of two stages: SCAG as the first-stage speaker-centered affective feature extraction module, and RGAC as the second-stage reliability-guided affective complementation module. In SCAG, PGVI contains the sampled video frames, the speaker reference, and the task setting, while ASGI provides affective semantic guidance from dialogue context, acoustic descriptions, and lexical VAD priors. It is important to note that the sampled video frames are kept in all visual-related ablations. Therefore, removing PGVI or SCAG does not mean removing the video input; instead, it removes the corresponding non-frame guidance signals while still feeding the sampled video frames into the frozen VLM. Table 2 evaluates the complete framework, stage-level ablations, and component-level ablations of both SCAG and RGAC. For the first stage, w/o SCAG directly feeds the sampled video frames into the frozen VLM without the speaker reference, task setting, or ASGI. This setting evaluates whether directly using frozen VLM representations from video frames is sufficient for ERC. The performance drop from VISAFF to w/o SCAG shows that simply feeding video frames into a frozen VLM is less effective than using SCAG to extract speaker-centered visual affective representations. Within SCAG, w/o ASGI removes affective semantic guidance while keeping the sampled video frames, speaker reference, and task setting. This evaluates the contribution of dialogue context, acoustic descriptions, and lexical VAD priors in guiding visual affective feature extraction. In contrast, w/o PGVI removes the speaker reference and task setting while keeping the sampled video frames as the visual input. This evaluates whether ASGI alone is sufficient without the speaker reference and task setting. The performance drops caused by these variants show that both PGVI and ASGI are important for extracting emotion-relevant visual affective representations. For the second stage, w/o RGAC disables the entire reliability-guided complementation process, including textual and acoustic affective reference retrieval, residual complementation, and visual reliability gating. This reduces the framework to the first-stage SCAG visual affective features and leads to a clear performance drop, confirming the overall contribution of reliability-guided affective complementation. Within RGAC, removing the textual reference, acoustic reference, or visual reliability gate consistently hurts performance, showing that these components all contribute to the final prediction. 7
Table 2: Ablation study of VISAFF on MELD and IEMOCAP. MELD
Method
IEMOCAP
Accuracy (%)
W-F1 (%)
Accuracy (%)
W-F1 (%)
67.62
67.12
77.31
77.30
Stage-level and component ablations w/o SCAG 66.41 w/o ASGI 66.82 w/o PGVI 66.97 w/o RGAC 63.52 w/o textual reference 61.53 w/o acoustic reference 66.21 w/o visual reliability gate 66.32
66.02 66.31 66.29 62.34 60.34 65.98 65.72
75.65 76.14 76.70 70.47 75.40 71.21 75.40
75.39 76.26 76.76 70.44 75.20 71.17 75.84
Full framework VISAFF
Table 3 further focuses on the visual representations extracted before the second-stage complementation. It compares existing visual-feature ERC methods with our frozen-VLM-based visual representations under different Stage-1 configurations. The results show that directly using generic frozen VLM representations from video frames leads to weak visual feature performance, indicating that video-frame-only VLM features are insufficient for ERC. Removing ASGI while keeping the sampled video frames, speaker reference, and task setting also causes clear drops, especially on IEMOCAP, showing that affective semantic guidance helps the frozen VLM extract more emotionrelevant visual affective representations. The full SCAG feature setting achieves the best performance among our visual representation variants on both datasets. This further confirms that PGVI and ASGI jointly improve the discriminative ability of speaker-centered visual affective representations. Table 3: Visual feature analysis of SCAG on MELD and IEMOCAP. Method
MELD
IEMOCAP
Accuracy (%)
W-F1 (%)
Accuracy (%)
W-F1 (%)
Existing visual-feature ERC methods MM-DFN (2022) [12] M2FNet (2022) [8] SDT (2024) [20] DER-GCN (2024) [1] ECERC (2025) [38]
– 45.63 48.05 60.50 48.33
32.34 32.44 32.01 60.60 43.24
– 20.39 41.47 57.80 30.72
27.46 13.10 42.71 57.10 27.57
Ours: visual features w/o ASGI and PGVI visual features w/o ASGI visual features w/o PGVI visual features SCAG visual features
47.49 61.92 57.62 63.52
41.58 60.77 54.75 62.34
38.47 61.04 63.50 70.47
36.99 60.93 63.28 70.44
4.4
Visualization and Reliability Analysis
Figure 3 presents qualitative examples drawn according to visual cue descriptions generated by Qwen3-VL. Since Qwen3-VL-Embedding is designed to produce unified multimodal representations rather than explicit attention maps for visualization [18], we do not directly visualize internal attention weights. Instead, we use a question-answering procedure aligned with the visual embedding extraction process: we keep the sample order consistent with visual feature extraction and ask Qwen3-VL to describe the visual cues related to the target speaker. The visual elements in Figure 3 are then drawn according to these model-generated descriptions. Figure 3 shows that semantic guidance changes the visual cues described by the model. Without affective semantic prompting, the model mainly focuses on salient hand-related actions and facial movements, and the resulting cues are closer to anger-related descriptions. After adding the prompt, the described cues shift toward fear-related details, such as terrified eyes, eye avoidance, body 8
withdrawal, face covering, and alert eyes. This suggests that semantic guidance helps the model describe visual cues that are more consistent with the target emotion.
Frowning eyebrows aggressive stance
tight lips continuous pointing
Raise eyebrows wave with hand
Cover face shrugs and furrows Eyes slightly closed brow
Fear Prompt
Terrified eyes Mouth slightly open
Eye avoidance body withdrawal
panicked eyes Arm retraction
Cover face Eyes slightly closed
Eyes alert
Figure 3: Qualitative visualization of visual cues described by the model. Figure 4 analyzes how the proposed reliability-aware affective complementation behaves under different levels of initial visual confidence. Test samples are grouped according to the confidence estimated by the auxiliary visual classifier. Figures 4(a) and 4(b) compare the Stage-1 SCAG visual features with the full framework on IEMOCAP and MELD, respectively. The full framework consistently improves over the Stage-1 visual features, especially in low- and medium-confidence intervals. Figures 4(c) and 4(d) further show the corresponding W-F1 gains. On IEMOCAP, the gain is most prominent in the 0.2–0.4 confidence bin and gradually decreases as visual confidence increases, indicating that textual and acoustic affective references are mainly used to complement uncertain visual cues. MELD shows a similar but milder trend, where the gains remain positive across confidence intervals but are less concentrated in the lowest-confidence region. This is consistent with the stronger textual bias and noisier visual signals in MELD. Overall, these results suggest that the second stage does not simply overwrite the visual representation with external modalities; instead, it provides residual affective complementation when visual cues are unreliable, while preserving reliable visual features when the initial visual confidence is high.
800
60
600
48.2
80 60
40
400
20
200
20
0
0
0
0.2–0.4
0.4–0.6
0.6–0.8
0.8–1.0
Initial Visual Confidence Bin
82.383.2
67.2 62.3
1250 1000
40
750 500
0.4–0.6
0.6–0.8
0.8–1.0
17.5
15
15.0 12.5
10
10.0
+7.0% +5.0%
5
250
0.2–0.4
20.0
20
1500
54.0 48.4
6
22.5
1750
7.5
+0.9%
0
0
Initial Visual Confidence Bin
0.2–0.4
0.4–0.6
0.6–0.8
0.8–1.0
Initial Visual Confidence Bin
5.0
5.5
+5.6% +4.9%
5
5.0
4
+3.2%
4.5
3 4.0
2 +1.0%
3.5
1 0
0.2–0.4
0.4–0.6
0.6–0.8
0.8–1.0
RGAC W-F1 Gain (%)
1000
(d) MELD gain
25 +24.1%
2000
W-F1 Gain (%)
84.785.6
(c) IEMOCAP gain 94.7 91.6
Sample Count
79.3 74.2
SCAG Only (w/o RGAC) VISAFF (w/ RGAC) Sample Count
W-F1 Gain (%)
72.0 65.0
W-F1 (%)
72.3
(b) MELD 100
Sample Count
W-F1 (%)
80
SCAG Only (w/o RGAC) VISAFF (w/ RGAC) Sample Count
RGAC W-F1 Gain (%)
(a) IEMOCAP 100
3.0
Initial Visual Confidence Bin
Figure 4: Effect of RGAC under Different Initial Visual Confidence Levels.
5
Conclusion
In this work, we revisit the role of video in Emotion Recognition in Conversation from a targetspeaker-centered perspective. We show that the limited and unstable contribution of video can be effectively improved by grounding visual affective cues on the target speaker and guiding frozen VLMs with affective semantic information. To this end, we propose VISAFF, a two-stage framework that first extracts speaker-centered visual affective representations with a frozen VLM, and then complements uncertain visual cues with textual and acoustic affective references under the control of visual reliability. Experiments on MELD and IEMOCAP demonstrate that VISAFF achieves strong performance without ERC-specific VLM fine-tuning. The results validate the effectiveness of starting from target-speaker visual information for conversational emotion recognition, and show that frozen VLMs can provide useful visual affective representations when properly guided. Further analyses confirm that affective semantic guidance substantially improves the discriminative ability of visual representations, while reliability-guided complementation brings consistent gains when visual cues are ambiguous or unreliable. Overall, this work provides a visual-oriented perspective for ERC and highlights the potential of using frozen VLMs as inference-time visual affective feature extractors. We hope this study can encourage future research to further explore speaker-centered visual reasoning, reliability-aware multimodal complementation, and more effective use of visual cues in conversational emotion understanding. 9
References [1] Wei Ai, Yuntao Shou, Tao Meng, and Keqin Li. DER-GCN: Dialogue and event relation-aware graph convolutional neural network for multimodal dialog emotion recognition. arXiv preprint arXiv:2312.10579, 2024. [2] Juan León Alcázar, Fabian Caba, Long Mai, Federico Perazzi, Joon-Young Lee, Pablo Arbeláez, and Bernard Ghanem. Active speakers in context. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12465–12474, 2020. [3] Sree Bhattacharyya and James Z Wang. Evaluating vision-language models for emotion recognition. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1798–1820, 2025. [4] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359, 2008. [5] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an _obviously_ perfect paper). In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4619–4629, 2019. [6] Di Chang, Yufeng Yin, Zongjian Li, Minh Tran, and Mohammad Soleymani. Libreface: An open-source toolkit for deep facial expression analysis. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 8205–8215, 2024. [7] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024. [8] Vishal Chudasama, Purbayan Kar, Ashish Gudmalwar, Nirmesh Shah, Pankaj Wasnik, and Naoyuki Onoe. M2fnet: Multi-modal fusion network for emotion recognition in conversation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4652–4661, 2022. [9] Yumeng Fu, Junjie Wu, Zhongjie Wang, Meishan Zhang, Lili Shan, Yulin Wu, and Bingquan Liu. Laerc-s: Improving llm-based emotion recognition in conversation with speaker characteristics. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6748–6761, 2025. [10] Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 154–164, 2019. [11] Chenyuan He, Senbin Zhu, Hongde Liu, Fei Gao, Yuxiang Jia, Hongying Zan, and Min Peng. Dialoguemmt: Dialogue scenes understanding enhanced multi-modal multi-task tuning for emotion recognition in conversations. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2497–2512, 2025. [12] Dou Hu, Xiaolong Hou, Lingwei Wei, Lianxin Jiang, and Yang Mo. Mm-dfn: Multimodal dynamic fusion network for emotion recognition in conversations. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7037–7041. IEEE, 2022. [13] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022. 10
[14] Taichi Ishiwatari, Yuki Yasuda, Taro Miyazaki, and Jun Goto. Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7360–7370, 2020. [15] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673, 2020. [16] Taewoon Kim and Piek Vossen. Emoberta: Speaker-aware emotion recognition in conversation with roberta. arxiv 2021. arXiv preprint arXiv:2108.12009, 2021. [17] Joosung Lee and Wooin Lee. CoMPM: Context modeling with speaker’s pre-trained memory tracking for emotion recognition in conversation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5669–5679, 2022. [18] Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, et al. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720, 2026. [19] Zheng Lian, Bin Liu, and Jianhua Tao. CTNet: Conversational transformer network for emotion recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 985–1000, 2021. [20] Hui Ma, Jian Wang, Hongfei Lin, Bo Zhang, Yijia Zhang, and Bo Xu. A transformer-based model with self-distillation for multimodal emotion recognition in conversations. IEEE Transactions on Multimedia, 26:776–788, 2023. [21] Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15747–15760, 2024. [22] Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. DialogueRNN: An attentive RNN for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6818–6825, 2019. [23] Saif Mohammad. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 174–184, 2018. [24] John Murzaku and Owen Rambow. Omnivox: Zero-shot emotion recognition with omni-llms. arXiv preprint arXiv:2503.21480, 2025. [25] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. [26] Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 873–883, 2017. [27] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 527–536, 2019. [28] Anupama Ray, Shubham Mishra, Apoorva Nunna, and Pushpak Bhattacharyya. A multimodal corpus for emotion recognition in sarcasm. In Proceedings of the thirteenth language resources and evaluation conference, pages 6992–7003, 2022. 11
[29] Minjie Ren, Xiangdong Huang, Wenhui Li, Dan Song, and Weizhi Nie. Lr-gcn: Latent relationaware graph convolutional network for conversational emotion recognition. IEEE Transactions on Multimedia, 24:4422–4432, 2021. [30] Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi, et al. Ava active speaker: An audio-visual dataset for active speaker detection. In ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4492–4496. IEEE, 2020. [31] Tao Shi and Shao-Lun Huang. Multiemo: An attention-based correlation-aware multimodal fusion framework for emotion recognition in conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14752–14766, 2023. [32] Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection. In Proceedings of the 29th ACM international conference on multimedia, pages 3927–3935, 2021. [33] Geng Tu, Tian Xie, Bin Liang, Hongpeng Wang, and Ruifeng Xu. Adaptive graph learning for multimodal conversational emotion detection. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19089–19097, 2024. [34] Michel F Valstar, Enrique Sánchez-Lozano, Jeffrey F Cohn, László A Jeni, Jeffrey M Girard, Zheng Zhang, Lijun Yin, and Maja Pantic. Fera 2017-addressing head pose in the third facial expression recognition and analysis challenge. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pages 839–847. IEEE, 2017. [35] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. [36] Zehui Wu, Ziwei Gong, Lin Ai, Pengyuan Shi, Kaan Donbekci, and Julia Hirschberg. Beyond silent letters: Amplifying llms in emotion recognition with vocal nuances. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2202–2218, 2025. [37] Zijian Yi, Ziming Zhao, Zhishu Shen, and Tiehua Zhang. Multimodal fusion via hypergraph autoencoder and contrastive learning for emotion recognition in conversation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 4341–4348, 2024. doi: 10.1145/3664647.3681633. [38] Tao Zhang and Zhenhua Tan. ECERC: Evidence-cause attention network for multi-modal emotion recognition in conversation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2064–2077, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10. 18653/v1/2025.acl-long.102. URL https://aclanthology.org/2025.acl-long.102/. [39] Yazhou Zhang, Mengyao Wang, Youxi Wu, Prayag Tiwari, Qiuchi Li, Benyou Wang, and Jing Qin. Dialoguellm: Context and emotion knowledge-tuned large language models for emotion recognition in conversations. Neural Networks, page 107901, 2025. [40] Yuanhang Zhang, Susan Liang, Shuang Yang, Xiao Liu, Zhongqin Wu, Shiguang Shan, and Xilin Chen. Unicon: Unified context network for robust active speaker detection. In Proceedings of the 29th ACM international conference on multimedia, pages 3964–3972, 2021.
12