Low-Cost Video--Time Priors as a Strong Baseline for EEG--fNIRS Emotion Regression on Familiar Videos

arXiv:2610.03618v1 [cs.AI] 2 Oct 2026

Low-Cost Video–Time Priors as a Strong Baseline for EEG–fNIRS Emotion Regression on Familiar Videos Minghao Kong

Jiurun Chen

Ying Gao

School of Electronics and Communication Engineering Sun Yat-sen University, Shenzhen Campus Shenzhen, China Department of Network Intelligence Pengcheng Laboratory Shenzhen, China

Department of Network Intelligence Pengcheng Laboratory Shenzhen, China

Department of Network Intelligence Pengcheng Laboratory Shenzhen, China

Xiangbin Meng

Rongjie Wang∗

Department of Network Intelligence Pengcheng Laboratory Shenzhen, China

Department of Network Intelligence Pengcheng Laboratory Shenzhen, China wangrj@pcl.ac.cn

Abstract

Keywords

Continuous emotion regression estimates moment-to-moment valence and arousal while a viewer watches a video. In familiar-video deployment, responses from training participants can form a group trajectory without collecting physiology from each new viewer. We use mean absolute error (MAE) as the primary objective for new-viewer prediction on familiar, time-aligned videos and construct a leakage-controlled, fold-wise video–time prior from training participants. An EEG–fNIRS branch provides a participant-specific estimate, and prior-dominating fixed fusion tests whether physiology adds residual correction. In five-fold subject-held-out evaluation on 24 participants and 15 videos, fusion achieved 29.01 MAE and remained best in an external subject-disjoint evaluation on four participants, where it reached 27.72. The prior was within 0.05 and 0.32 MAE of fusion in the internal and external evaluations, respectively. Source-explicit ablations showed that video identity and within-video time accounted for most of the reduction, while EEG–fNIRS gains were smaller and varied across participants and videos. These results identify the video–time prior as a strong, lowcost baseline. Because the physiological branch uses 𝑡 + 1 context, its reported contribution is offline and does not establish real-time decoding. We position EEG–fNIRS as an optional residual signal for familiar-video emotion regression.

EEG–fNIRS, continuous emotion regression, mean absolute error, video–time prior, familiar-video prediction

CCS Concepts • Information systems → Multimedia information systems; • Computing methodologies → Neural networks; • Applied computing → Psychology.

∗

Corresponding author.

ACM MM ’26, Rio de Janeiro, Brazil 2026. 1

ACM Reference Format: Minghao Kong, Jiurun Chen, Ying Gao, Xiangbin Meng, and Rongjie Wang. 2026. Low-Cost Video–Time Priors as a Strong Baseline for EEG–fNIRS Emotion Regression on Familiar Videos. In Proceedings of the 34th ACM International Conference on Multimedia (ACM MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 9 pages.

1 Introduction Continuous emotion regression estimates moment-to-moment valence and arousal while a viewer watches a video. Accurate grouplevel affect trajectories can support video understanding and provide baselines for video editing, advertising placement, content recommendation, or coarse initialization of a new viewer’s expected response. This paper focuses on the predictive prerequisite for these uses: lowering mean absolute error (MAE) for new viewers watching familiar, time-aligned videos. EEG provides millisecond-scale electrophysiological measurements, whereas functional near-infrared spectroscopy (fNIRS) captures slower haemodynamic changes. Their complementary properties motivate EEG–fNIRS affect models [1, 2, 21]. Cross-participant prediction nevertheless remains difficult because anatomy, sensor contact, baseline state, and response style can produce shifts larger than the affective variation of interest [12, 13]. Collecting both modalities from every new viewer also carries substantially greater sensing and processing cost than using information already associated with the video. The familiar-video setting changes what information is available. Naturalistic videos can induce reliable, time-locked responses across participants, suggesting that familiar videos contain shared response structure aligned to playback time [5, 7, 9]. When training participants and a new viewer watch the same familiar video on an

ACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil

Minghao Kong et al.

aligned playback timeline, their matched labels permit a group affect trajectory indexed by video identity and within-video time. A valid predictor must reconstruct this trajectory inside each training split so that held-out labels cannot enter the estimate. We operationalize the shared response as a fold-wise video– time prior, pair it with an EEG–fNIRS branch, and combine the two through prior-dominating fixed fusion. The study is organized around three research questions:

not typically operationalized as explicit continuous affect predictors. In our repeated-stimulus setting, matched labels from training participants permit a different operationalization: a fold-wise group trajectory indexed by video identity and within-video time. The video–time prior turns this trajectory into a low-cost regression baseline and directly compares it with an EEG–fNIRS branch.

RQ1. What MAE do the video–time prior and fixed fusion achieve when predicting new viewers’ affect on familiar videos? RQ2. Which information source, global level, video identity, within-video time, or EEG–fNIRS, accounts for the MAE reduction? RQ3. How do the source contributions and fusion gains vary across participants, videos, and within-video time? Our contributions are threefold:

• We formulate MAE-oriented continuous emotion prediction for new viewers of familiar videos and construct a fold-wise video–time prior that excludes labels from evaluated participants and requires only video identity and playback time at inference. • We show empirically that the video–time prior is a strong predictor, while prior-dominating fusion with an EEG–fNIRS branch achieves the lowest evaluated MAE in both the internal subject-held-out and external subject-disjoint evaluations. • We separate global level, video identity, within-video time, and EEG–fNIRS correction through source-explicit ablations and paired residual analysis, revealing a small and heterogeneous physiological contribution.

Evaluation, priors, and leakage control

Evaluation protocols determine which form of generalization a reported score supports. Grouped cross-validation prevents direct participant overlap, but repeated videos across folds still evaluate new participants on familiar content [18, 20]. Video-held-out and crossed participant-by-video designs answer different questions. Model selection within evaluation folds can further bias estimates [23, 24]. For familiar-video prediction, a video–time prior is legitimate only when it is built from training participants and applied to the corresponding held-out participant–video–time samples. We therefore reconstruct the prior within every internal fold and keep all external targets outside training and prior construction. Sourceexplicit variants then separate the information supplied by global affect level, video identity, within-video time, and EEG–fNIRS measurements.

3 Method 3.1 Problem formulation and pipeline The task studied here is participant generalization conditional on known stimulus identity: every evaluated viewer watches one of the same 15 familiar videos, and playback is aligned to the development timeline. Video identity and playback second are therefore available at inference. We do not evaluate video-held-out transfer or extrapolation to unseen content, and no result below should be interpreted as such. For participant 𝑠 , familiar video 𝑣 , and second 𝑡 , the target is 𝑦𝑠,𝑣,𝑡 ∈ [1, 255]2 , containing valence and arousal. Our primary endpoint is sample-pooled MAE over both affect dimensions,

2 Related Work 2.1 Physiological emotion regression Physiological emotion recognition commonly represents affect in categorical or valence–arousal spaces [11, 19]. EEG work has identified informative spectral and spatial patterns [26], while fNIRS studies use oxygenation-related statistics to model affective state [1]. Multimodal systems combine the two signals through graph convolution, attention, or representation alignment [2, 6, 21]. These mechanisms motivate our EEG–fNIRS branch. Cross-participant variation remains a persistent obstacle [12, 13]. Resting recordings and transfer learning can align participants or personalize affective decoders [16, 27]. Such methods address participant shift, but they do not quantify the low-cost information already contained in a familiar video’s identity and playback time. A fair analysis of this setting therefore requires an explicit video– time baseline beside the physiological model.

2.2

2.3

𝑁

MAE =

1 ∑ ∑ |𝑦 𝑑 − 𝑦𝑖̂ 𝑑 | . 2𝑁 𝑖=1 𝑑∈{𝑉 ,𝐴} 𝑖

(1)

Here, 𝑁 denotes the number of evaluated one-second participant– video–time samples, and 𝑖 indexes each sample tuple (𝑠, 𝑣, 𝑡). The pipeline produces two complementary estimates before fū retrieves the group trajectory assosion. The video–time prior 𝑝𝑣,𝑡 ciated with the familiar video and second. The EEG–fNIRS branch 𝑚𝑠,𝑣,𝑡 extracts one-second channel features, encodes each modality with a graph network, exchanges cross-modal information through attention, and regresses valence and arousal from pooled tokens. A fixed convex combination returns the final prediction 𝑦𝑠,𝑣,𝑡 ̂ . This design makes the low-cost shared response and the participantspecific physiological correction separately measurable. Figure 1 summarizes this leakage-controlled, source-explicit pipeline.

Naturalistic stimuli and shared responses

Shared-stimulus studies have demonstrated inter-subject synchronization in BOLD and EEG responses during naturalistic viewing [5, 7, 9]. Related work has linked the reliability of such responses to emotion, attention, and audience preference [4, 10, 17]. Together, these findings characterize shared neural processing, but they are 2

Low-Cost Video–Time Priors as a Strong Baseline for EEG–fNIRS Emotion Regression on Familiar Videos

A

B

INPUT SCENE

ACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil

C FUSION AND OUTPUT

TWO PARALLEL BRANCHES

Video-Time Prior (Low-Cost, Label-Derived)

fNIRS optode patches

Radius-3 within-video smoothing

Fold-wise median over participants

Familiar Video Training participants

EEG electro-cap

Video-Time Prior

Held-out participant labels excluded

Prior-Dominating Fixed Fusion

EEG-fNIRS Branch (Participant-Specific)

Predicted affect trajectory

Graph Encoder (EEG)

New viewer

EEG (64ch)

1-sec features

Regressor

(t-1, t, t+1)

Cross-modal

video time t

Attention

fNIRS (���� × �)

Graph

Encoder (����)

Physiological Prediction

Valence over time

Arousal over time

Figure 1: Method overview. (A) A new viewer contributes EEG–fNIRS at second 𝑡 of a familiar video. (B) The video–time prior branch (blue) aggregates training-participant labels via a fold-wise cross-participant median and radius-three within-video ̄ ; held-out labels are excluded. The EEG–fNIRS branch (orange) extracts one-second features from 𝑡 − 1, smoothing to yield 𝑝𝑣,𝑡 𝑡 , and 𝑡 + 1, applies graph encoders and bidirectional cross-modal attention, and regresses 𝑚𝑠,𝑣,𝑡 . (C) Prior-dominating fixed fusion with valence and arousal prior weights 𝑤 = [0.99, 0.92] produces 𝑦𝑠,𝑣,𝑡 ̂ .

3.2

Fold-wise video–time prior

relative log power and differential entropy in six bands from 1 to 45 Hz, together with Hjorth activity, mobility, and complexity [8]. Concatenating the features from 𝑡 − 1, 𝑡 , and 𝑡 + 1 yields a 64 × 45 EEG tensor. For each fNIRS channel and each of six signal types, we compute mean, standard deviation, slope, skewness, and kurtosis. The same three-segment concatenation yields a 51 × 90 fNIRS tensor. Edge padding is used at trial boundaries. Because both tensors include the 𝑡 + 1 segment, the branch is an offline estimator rather than a strictly causal real-time decoder. Each modality is encoded with a learned-adjacency, multi-order graph network using identity, first-order, and second-order propagation supports. The encoder produces 32-dimensional channel embeddings. Four-head attention exchanges information in the EEGto-fNIRS and fNIRS-to-EEG directions [22, 25]. Mean and maximum pooling over modality-specific and cross-modal tokens feed a multilayer regressor. Targets are min–max scaled from [1, 255] to [0, 1] as 𝑦̃ = (𝑦 −1)/254, and a sigmoid produces predictions on the same interval. Models trained on participant-grouped folds minimize mean squared error on these scaled targets plus 0.01 times

Within each training fold, we aggregate labels at every video–time coordinate using the cross-participant median: 𝑑 = median 𝑑 𝑝𝑣,𝑡 𝑠∈𝒮tr 𝑦𝑠,𝑣,𝑡 ,

𝑑 ∈ {𝑉 , 𝐴}.

(2)

We then smooth each trajectory with a radius-three moving average, 𝑑̄ = 𝑝𝑣,𝑡

1 ∑ 𝑝𝑑 , |ℐ𝑣,𝑡 | 𝑢∈ℐ 𝑣,𝑢

ℐ𝑣,𝑡 = {𝑢 ∶ |𝑢 − 𝑡| ≤ 3, 0 ≤ 𝑢 < 𝑇𝑣 }. (3)

𝑣,𝑡

Smoothing never crosses video boundaries. The median limits the influence of individual trajectories, and fold-wise reconstruction prevents held-out labels from determining their own predictions. At inference, the prior is a lookup by video identity and second, giving it low marginal input and computation cost once the training trajectories have been built.

3.3

EEG–fNIRS branch

The EEG–fNIRS branch proceeds through channel-level feature extraction, graph encoding within each modality, cross-modal attention in both directions, and pooled regression. Signals are centred using the corresponding trial baseline, and EEG is resampled to 200 Hz. For each EEG channel and one-second segment, we compute 3

ACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil

Minghao Kong et al.

Table 1: Descriptive fusion-weight sensitivity on a 0.01 grid. The near-minimum range contains weights with target MAE no more than 0.10 above the grid minimum. Internal results use pooled out-of-fold predictions and are not nested selection estimates; external results are post hoc diagnostics that were not used to change the reported fixed weights. Δ is the fixed-weight MAE minus the grid-minimum MAE.

Eval. Internal

Target

Valence Arousal External Valence Arousal

Grid min. Near-min. 𝑤 ∗ /MAE range .99 / 26.664 .92 / 31.365 .83 / 23.748 .82 / 30.073

overall, valence, and arousal MAE. This protocol evaluates new viewers responding to the same 15 familiar videos.

4.3

The external cohort contains four participants, 60 trials, and 6,143 one-hertz targets. These participants are disjoint from the development set but viewed the same 15 familiar videos at aligned playback times. Their targets are excluded from prior construction, checkpoint training, and prediction generation; they are joined to the already-generated predictions only for retrospective scoring and diagnostic analyses. The external EEG–fNIRS estimate averages predictions from six checkpoints. For the final checkpoint, EEG and fNIRS features are standardized using statistics computed from all 24 development participants. Each of the five participant-grouped fold checkpoints uses feature-standardization statistics computed from its training participants. Fixed fusion combines this ensemble with a video–time prior built from all 24 development participants.

Fixed 𝑤/Δ

.93–1.00 .99 / +.000 .85–1.00 .92 / +.000 .79–.86 .99 / +1.449 .77–.89 .92 / +.180

a symmetric contrastive alignment loss [3]. Predictions are transformed back using 254𝑦̂̃ +1 and clipped to [1, 255] before MAE evaluation. Hereafter, 𝑚𝑠,𝑣,𝑡 denotes this transformed and clipped prediction on the original [1, 255] scale. The alignment term regularizes paired EEG and fNIRS representations from the same sample without dominating the regression objective. All evaluated EEG– fNIRS checkpoints use a dropout rate of 0.7. Models trained on participant-grouped folds use AdamW optimization and early stopping on fold validation MSE.

3.4

4.4

Table 2: Information used by each source variant at inference. “Label lookup” means inference-time retrieval of a cached statistic derived only from labels in the corresponding training split; evaluated-participant labels never enter the statistic. Supervised models still use training targets. “Time key” means explicit lookup by within-video playback time, not temporal context inside the EEG–fNIRS branch.

Prior-dominating fixed fusion 𝑤 = [0.99, 0.92].

(4)

The fixed weights intentionally keep the video–time prior dominant for valence and arousal. Our objective is to test whether EEG– fNIRS can provide residual correction to an already strong low-cost estimate, rather than to optimize a complex adaptive gating model. We retain the numerical pair as a fixed, descriptive operating point rather than an adaptively estimated gate. Because its selection was not fully nested, we do not interpret it as an unbiased or universal optimum. The same pair is used for every split and is transferred unchanged to the external evaluation, which keeps the incremental correction directly interpretable.

Variant Global constant Video identity Video–time prior EEG–fNIRS branch Fixed fusion

Participant phys.

Video key

Time key

Label lookup

N N N Y Y

N Y Y N Y

N N Y N Y

Y Y Y N Y

The global constant is the two-dimensional median over all development labels. The video-identity variant uses the temporal median of each development video trajectory and remains constant within that video. The video–time prior uses the full median trajectory with radius-three within-video smoothing. The EEG–fNIRS and fixed-fusion variants follow Section 3 and the external ensemble definition above. The internal evaluation reports only the video– time prior, EEG–fNIRS branch, and fixed fusion; the external evaluation reports all five variants.

4 Experimental Setup 4.1 Dataset and participants The MER-PS 2026 training and validation data contain 24 participants, 15 familiar videos viewed by all participants, 360 trials, and 36,864 one-hertz valence–arousal targets [14, 15]. Each trial includes a five-second resting baseline. EEG contains 64 channels, and fNIRS contains six signal types over 51 channels. All reported metrics are computed after predictions are mapped back to the original [1, 255] scale.

4.2

Source variants

Table 2 summarizes the information used by each variant at inference.

The final estimate is

̄ + (1 − 𝑤) ⊙ 𝑚𝑠,𝑣,𝑡 , 𝑦𝑠,𝑣,𝑡 ̂ = 𝑤 ⊙ 𝑝𝑣,𝑡

External subject-disjoint evaluation

4.5

Metrics and reporting

We report sample-pooled MAE overall and separately for valence and arousal on identical participant–video–time samples. We also computed overall MSE as a secondary external diagnostic; MAE remains the primary endpoint. For external heterogeneity analysis, paired errors are also aggregated by participant, by familiar video, and by ten equal bins of normalized within-video time. For each video–time cell, MAE equally pools the valence and arousal absolute errors of all one-second samples assigned to that cell. These

Internal subject-held-out evaluation

Participants are partitioned into five folds. For each fold, the video– time prior is reconstructed only from training participants, and the matching EEG–fNIRS checkpoint predicts the held-out participants. The full-data checkpoint is excluded. Predictions and targets from all folds are concatenated before computing sample-pooled 4

Low-Cost Video–Time Priors as a Strong Baseline for EEG–fNIRS Emotion Regression on Familiar Videos

Figure 2 shows the same internal pattern visually. The large reduction from the EEG–fNIRS branch to the final predictor was almost entirely recovered by the video–time prior, while fusion supplied the smallest final step. Table 1 places that step in context. On the pooled internal out-offold predictions, the target-wise grid minima retrospectively coincided with the reported pair: 𝑤𝑉 = 0.99 gave 26.664 valence MAE and 𝑤𝐴 = 0.92 gave 31.365 arousal MAE. Overall MAE was 47.3509 for branch only, 29.0147 for the reported pair, and 29.0633 for prior only, but the pair improved fold-level overall MAE in only three of five folds. This retrospective coincidence does not reconstruct the pair’s original selection procedure. Applied unchanged externally, the pair reached 27.7246 overall MAE versus 42.7486 for branch only and 28.0410 for prior only. The post hoc external target-wise minima shifted to 0.83 and 0.82; their combination gave 26.9102 overall MAE. This external value is diagnostic only, and the shift shows that the exact weights are not universal. Figure 3 complements the scalar external scores with a descriptive view of calibration and error structure. Fixed fusion achieved valence MAE of 25.20 and arousal MAE of 30.25. Both densities showed prediction-range compression toward the middle of the scale, with the effect more pronounced for arousal. These plots identify score ranges that contributed to error, but they do not treat the temporally correlated second-level observations as independent evidence.

Internal MAE comparison

a

Prior accounts for 99.7%

47.35

EEG–fNIRS branch

Video–time prior

29.06

Fixed fusion

29.01 0

10

20

30

40

50

Mean absolute error

Target-level increment

b

3/3 targets improve +0.049

Overall

+0.004

Valence

+0.093

Arousal 0.00

0.05

ACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil

0.10

Prior MAE − fusion MAE

Figure 2: Internal subject-held-out MAE. (a) Overall MAE for the EEG–fNIRS branch, video–time prior, and fixed fusion. (b) Additional MAE reduction from the prior to fusion by target.

5.2

RQ2: Source decomposition

The staged external variants identify the source of the reduction. Replacing the global constant with video identity lowered overall MAE by 10.97 points; adding within-video time lowered it by a further 5.32 points. The shared response therefore contains both a video-specific affect level and a substantial time-resolved trajectory. Using unrounded overall MAE, the video–time prior accounted for 99.73% and 97.89% of the total internal and external reductions from EEG–fNIRS to fixed fusion, respectively. Each percentage was calculated as the MAE improvement from EEG–fNIRS to the prior divided by the corresponding improvement from EEG–fNIRS to fusion. The fixed fusion remained best, but most of its advantage over the EEG–fNIRS branch was already available from video identity and within-video time. Figure 4a shows the complete external ordering of the five variants.

grouped summaries and the predicted-versus-observed densities are descriptive because the external cohort is small and adjacent one-second targets are temporally correlated. The 6,143 one-second observations are therefore not treated as independent replicates. For the internal evaluation, only aggregate component metrics are available, so paired sample-level residual analysis is performed on the external cohort. For the descriptive sensitivity analysis in Table 1, we independently varied each target’s prior weight from 0 to 1 in increments of 0.01 while holding the prior and EEG–fNIRS predictions fixed. Thus, 𝑤 = 0 is branch only and 𝑤 = 1 is prior only. The pooled internal out-of-fold sweep is a retrospective characterization, not a nested model-selection estimate; the external sweep is strictly post hoc and serves only as a robustness diagnostic. All scores involving the EEG–fNIRS branch or fixed fusion are offline because the branch uses the 𝑡 + 1 segment. The label-derived global, videoidentity, and video–time variants do not require future physiology.

5.3

RQ3: Heterogeneity of source contributions and fusion gains

We define signed fusion gain as Δ = MAEprior − MAEfusion , so positive values favour fusion. For S1–S4, Δ was −0.062, +0.399, +0.235, and +0.694 MAE, respectively. Fusion improved V1, V4–V9, V14, and V15; the largest gain occurred for V7 (+1.310), whereas the largest loss occurred for V11 (−0.297). Across the 60 participant– video trials, 38 improved and 22 worsened. Concrete trials illustrate both signs. For S2–V7, overall MAE fell from 24.586 with the prior to 22.422 with fusion, driven mainly by arousal MAE falling from 27.016 to 23.066. For S1–V11, overall MAE rose from 38.689 to 41.016, with arousal MAE rising from 45.967 to 49.918. These cases localize behaviour retrospectively;

5 Results and Analysis 5.1 RQ1: Main MAE performance Table 3 combines the internal and external results. Fixed fusion achieved the lowest overall MAE in both evaluations, reaching 29.01 internally and 27.72 externally. The video–time prior stayed within 0.05 and 0.32 MAE, respectively, whereas the EEG–fNIRS branch was substantially less accurate. Valence and arousal followed the same ordering. 5

ACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil

Minghao Kong et al.

Table 3: Internal and external MAE on the original label scale. Lower is better; dashes mark source variants not reported for the internal evaluation. Target

Global

Video ID

Video–time prior

EEG–fNIRS

Fusion

Internal

Overall Valence Arousal

– – –

– – –

29.06 26.67 31.46

47.35 51.35 43.35

29.01 26.66 31.37

External

Overall Valence Arousal

44.33 47.46 41.20

33.36 32.56 34.17

28.04 25.34 30.74

42.75 45.29 40.21

27.72 25.20 30.25

b

Valence (MAE = 25.20)

Arousal (MAE = 30.25)

255

255

192

192

Predicted arousal

Predicted valence

102

128

128

64

101 64

1

100

1 1

64

128

192

255

1

True valence

Samples per hexagon (log scale)

a

Evaluation

64

128

192

255

True arousal

Figure 3: External fixed-fusion predicted-versus-observed densities. (a,b) Hexbin densities compare predictions with observed valence and arousal, respectively, for 6,143 one-second samples. The dashed diagonal marks exact prediction; dotted horizontal and vertical lines mark the neutral score of 128. Colour gives sample count on a logarithmic scale, and panel titles report MAE on the original [1, 255] scale. Both panels show prediction-range compression toward the middle of the scale, particularly for arousal. with four participants and temporally correlated observations, they are neither stable subgroup effects nor a prospective selection rule. The physiological correction varied across playback. Fusion worsened normalized bins 0–2, the first 30% of playback, with Δ from −0.129 to −0.052 MAE. It improved bins 3–9, with Δ from +0.070 to +0.779. This comparison between fusion and the prior differs from the contribution of the time key itself, examined next. The contribution of within-video time also varied over playback. Figure 4d provides a one-dimensional marginal after pooling errors across videos within each normalized time bin. In the first bin, video identity reached 45.53 MAE and the video–time prior reached 13.40. This advantage narrowed over time, and video identity was slightly better in each of the final five bins. The aggregate temporal gain was therefore concentrated in specific phases rather than distributed uniformly.

Figure 5 restores the two-dimensional video–time structure hidden by that marginal summary. Overall MAE was 28.04 for the video–time prior and 27.72 for fixed fusion. Across the 150 cells, fusion had lower MAE in 76, equal MAE in 7, and higher MAE in 67. The two methods shared 9 of their 10 highest-error cells. Their main hotspot occurred in the latter half of V1. Thus, the small aggregate fusion gain reflected many local changes of both signs rather than a uniform reduction across the grid.

6 Discussion and Limitations 6.1 Why the video–time prior is effective The video–time prior is effective because the deployment setting preserves two strong coordinates: familiar video identity and playback time. Participants respond differently, but a shared audiovisual sequence still induces a population-level trajectory. Median 6

Low-Cost Video–Time Priors as a Strong Baseline for EEG–fNIRS Emotion Regression on Familiar Videos

a

Participant-level increment

b

External MAE comparison

3/4 improve

33.36

Video identity

+0.694

S4

44.33

Global constant

ACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil

+0.399

S2

28.04

Video–time prior

+0.235

S3

42.75

EEG–fNIRS branch 27.72

Fixed fusion

−0.062

S1 0

10

20

30

40

50

0.0

Mean absolute error

Video-level increment

c

0.5

1.0

Prior MAE − fusion MAE 9/15 improve

d

Error varies across video time 55

V7 V1

50

V15

Mean absolute error

V4 V14 V6 V9 V8 V5 V12 V3 V13

45 40 35 30 25 20

V2 V10

15

V11 0.0

0.5

1.0

0

Prior MAE − fusion MAE Global constant

20

40

60

80

100

Normalized video time (%) Video identity

Video–time prior

Fixed fusion

Figure 4: External MAE and residual analysis. (a) Overall MAE for five source variants. (b,c) Participant- and video-level priorminus-fusion increments; positive values favour fusion. (d) MAE over normalized video-time bins; this panel focuses on labelderived variants and fusion. aggregation limits individual extremes, and within-video smoothing reduces local variation without crossing video boundaries. The resulting predictor gains its strength from the data structure rather than from model complexity. This property gives the prior practical value. Once previous viewers have established a trajectory, it provides a low-cost group-response curve for a new viewer of the same time-aligned video without EEG or fNIRS acquisition. The curve may support retrospective content analysis or initialize an expected trajectory; downstream decision utility was not evaluated here.

6.2

When EEG–fNIRS adds value

EEG–fNIRS is not a competitive replacement for the video–time prior here; it tests whether participant-specific physiology retains residual value after familiar video identity and playback time are 7

known. The incremental value is small and heterogeneous: fusion offers the lowest aggregate MAE, whereas the prior supplies most of the benefit without new physiological measurement. The fixed correction lowers error only when its scaled adjustment moves the prior toward the unknown participant-specific target. It increases error when the adjustment has the wrong direction or overshoots. The paired analysis can localize this behaviour only retrospectively and provides no inference-time criterion for knowing when physiology will help. A prospective reliability rule would require independently validated gating or uncertainty estimation in a larger crossed participant-by-video design. Several mechanisms could limit the correction, including sensor noise, handcrafted one-second features, imperfect haemodynamic timing, cross-participant shift, and a model that does not explicitly isolate residuals from the prior. The present analysis locates

ACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil

a

Video–time prior

Minghao Kong et al.

Overall MAE = 28.04

b

Fixed fusion

Overall MAE = 27.72

V1 60

V2

V4 V5

Video

V6

40

V7 V8 V9 V10 20

V11 V12

Overall MAE (lower is better)

V3

V13 V14 V15

0 5

15

25

35

45

55

65

75

85

95

5

Normalized video time (%)

15

25

35

45

55

65

75

85

95

Normalized video time (%)

Figure 5: External video–time MAE. (a) Video–time prior and (b) fixed fusion across 15 familiar videos and ten equal bins of normalized playback time. Each cell equally pools valence and arousal absolute errors over all one-second samples assigned to that video–time cell. Both panels use the same fixed 0–65 MAE scale, with deep blue denoting lower error and orange–red denoting higher error. Overall MAE was 28.04 for the prior and 27.72 for fusion.

6.4

gains and losses but does not distinguish among these explanations. Future work should test stronger residual modelling and explicit haemodynamic alignment.

6.3

Limitations and future work

The conclusions are bounded by the familiar-video setting: all participants viewed the same 15 videos, so transfer to unseen content was not measured. The external cohort contains only four participants, and adjacent one-second targets are correlated; its heterogeneity analysis is therefore descriptive rather than stable participant-level inference. Anonymous identifiers prevent demographic or subgroup analysis. From a modelling and evaluation perspective, fusion weights and checkpoint choices were not selected through a fully nested procedure. Although the retrospective internal sweep minimized at the reported pair, the pair improved only three of five folds and the post hoc external minima shifted to 0.83/0.82. The exact weights are therefore not universal. Internal and external fusion increments are not directly comparable because their physiological ensembles and held-out groups differ. The EEG–fNIRS branch uses one-second future context and no explicit fNIRS haemodynamic shift, so all physiology-assisted scores are offline; no causal feature variant was evaluated. The study measures prediction error rather than downstream utility and compares one physiological architecture. Crossed participant-by-video evaluation, nested selection, causal features, and stronger residual modelling are the main priorities for broader validation.

Implications for practice and evaluation

The results also clarify the evaluation question. Subject-held-out evaluation with familiar videos measures generalization to new participants, whereas video-held-out evaluation measures transfer to new content. Future studies should separate these axes through crossed participant-by-video splits and rebuild the prior, feature standardization, checkpoint selection, and fusion weights within each outer training split [23, 24]. Participant-macro, video-macro, and paired residual summaries would further preserve the hierarchy hidden by a single sample-pooled score. For deployment, the choice between the prior and fusion depends on the value of the residual improvement relative to acquisition cost. The prior provides a simple group baseline for familiar content, while EEG–fNIRS can be reserved for settings in which participant-specific correction is worth the additional instrumentation. In the current feature construction, however, physiologyassisted fusion supports retrospective or offline estimation only. A real-time version must remove the 𝑡 + 1 segment, use causal features, and be evaluated anew; the present scores cannot be extrapolated to that setting. 8

Low-Cost Video–Time Priors as a Strong Baseline for EEG–fNIRS Emotion Regression on Familiar Videos

7

Conclusion

ACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil

in Neuroscience 12 (2018), 162. doi:10.3389/fnins.2018.00162 [13] Xiang Li, Yazhou Zhang, Prayag Tiwari, Dawei Song, Bin Hu, Meihong Yang, Zhigang Zhao, Neeraj Kumar, and Pekka Marttinen. 2023. EEG Based Emotion Recognition: A Tutorial and Review. Comput. Surveys 55, 4, Article 79 (2023), 57 pages. doi:10.1145/3524499 [14] Zheng Lian, Xiaojiang Peng, Kele Xu, Ziyu Jia, Xinyi Che, Zebang Cheng, Fei Ma, Laizhong Cui, Yazhou Zhang, Xin Liu, Liang Yang, Jia Li, Fan Zhang, Liumeng Xue, Erik Cambria, Guoying Zhao, Bjorn W. Schuller, and Jianhua Tao. 2026. MER 2026: From Discriminative Emotion Recognition to Generative Emotion Understanding. arXiv:2604.19417 [cs.HC] https://arxiv.org/abs/2604.19417 [15] MER-PS. 2026. MER-PS/MER-PS-trainval. Hugging Face Datasets. https : / / huggingface . co / datasets / MER - PS / MER - PS - trainval / tree / 776227330531a705a55bfd52f74ea5b537f89e75 Dataset repository, accessed 23 July 2026. [16] Hiroshi Morioka, Atsunori Kanemura, Jun-ichiro Hirayama, Manabu Shikauchi, Takeshi Ogawa, Shigeyuki Ikeda, Motoaki Kawanabe, and Shin Ishii. 2015. Learning a Common Dictionary for Subject-Transfer Decoding with Resting Calibration. NeuroImage 111 (2015), 167–178. doi:10.1016/j.neuroimage.2015.02.015 [17] Lauri Nummenmaa, Enrico Glerean, Mikko Viinikainen, Iiro P. Jääskeläinen, Riitta Hari, and Mikko Sams. 2012. Emotions Promote Social Interaction by Synchronizing Brain Activity Across Individuals. Proceedings of the National Academy of Sciences 109, 24 (2012), 9599–9604. doi:10.1073/pnas.1206095109 [18] David R. Roberts, Volker Bahn, Simone Ciuti, Mark S. Boyce, Jane Elith, Gurutzeta Guillera-Arroita, Severin Hauenstein, José J. Lahoz-Monfort, Boris Schröder, Wilfried Thuiller, David I. Warton, Brendan A. Wintle, Florian Hartig, and Carsten F. Dormann. 2017. Cross-Validation Strategies for Data with Temporal, Spatial, Hierarchical, or Phylogenetic Structure. Ecography 40, 8 (2017), 913–929. doi:10.1111/ecog.02881 [19] James A. Russell. 1980. A Circumplex Model of Affect. Journal of Personality and Social Psychology 39, 6 (1980), 1161–1178. doi:10.1037/h0077714 [20] Sohrab Saeb, Luca Lonini, Arun Jayaraman, David C. Mohr, and Konrad P. Kording. 2017. The Need to Approximate the Use-Case in Clinical Machine Learning. GigaScience 6, 5, Article gix019 (2017), 9 pages. doi:10.1093/gigascience/gix019 [21] Yanjia Sun, Hasan Ayaz, and Ali N. Akansu. 2020. Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression. Brain Sciences 10, 2 (2020), 85. doi:10.3390/brainsci10020085 [22] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal Transformer for Unaligned Multimodal Language Sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 6558–6569. doi:10.18653/v1/P19-1656 [23] Sudhir Varma and Richard Simon. 2006. Bias in Error Estimation When Using Cross-Validation for Model Selection. BMC Bioinformatics 7, 1 (2006), 91. doi:10. 1186/1471-2105-7-91 [24] Gaël Varoquaux, Pradeep Reddy Raamana, Denis A. Engemann, Andrés HoyosIdrobo, Yannick Schwartz, and Bertrand Thirion. 2017. Assessing and Tuning Brain Decoders: Cross-Validation, Caveats, and Guidelines. NeuroImage 145 (2017), 166–179. doi:10.1016/j.neuroimage.2016.10.038 [25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc., Red Hook, NY, USA, 5998–6008. https://proceedings.neurips. cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html [26] Wei-Long Zheng and Bao-Liang Lu. 2015. Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks. IEEE Transactions on Autonomous Mental Development 7, 3 (2015), 162–175. doi:10.1109/TAMD.2015.2431497 [27] Wei-Long Zheng and Bao-Liang Lu. 2016. Personalizing EEG-Based Affective Models with Transfer Learning. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. AAAI Press / International Joint Conferences on Artificial Intelligence, Palo Alto, CA, USA, 2732–2738. https://www.ijcai.org/Proceedings/16/Papers/388.pdf

This study establishes a fold-wise video–time prior as a strong, low-cost baseline when new participants view the same 15 familiar, time-aligned videos. Fixed offline fusion achieved the lowest pooled MAE, but the prior supplied most of the improvement and physiological correction was small and heterogeneous. These results support retrospective estimation or initialization for repeated familiar content only. They do not establish real-time decoding, prospective identification of reliable physiological corrections, or generalization to unseen videos. Those claims require causal features and crossed participant-by-video evaluation with fully nested selection.

Data and Code Availability Complete implementations of all algorithms evaluated in this study are open source under the Apache License 2.0 and are available at https://github.com/rudykon/MER2026track4-EEG-fNIRS-AffectRegression/. The MER-PS training and validation data are available at https://huggingface.co/datasets/MER-PS/MER-PS-trainval. The participant-disjoint held-out evaluation data are available at https: //huggingface.co/datasets/MER-PS/MER-PS-public-leaderboardevaluation-data. Access to and reuse of each dataset are subject to the terms specified by the corresponding repository.

References [1] Danushka Bandara, Senem Velipasalar, Sarah Bratt, and Leanne Hirshfield. 2018. Building Predictive Models of Emotion with Functional Near-Infrared Spectroscopy. International Journal of Human-Computer Studies 110 (2018), 75–85. doi:10.1016/j.ijhcs.2017.10.001 [2] Guijun Chen, Yue Liu, and Xueying Zhang. 2024. EEG–fNIRS-Based Emotion Recognition Using Graph Convolution and Capsule Attention Network. Brain Sciences 14, 8 (2024), 820. doi:10.3390/brainsci14080820 [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119). PMLR, Virtual, 1597–1607. https: //proceedings.mlr.press/v119/chen20j.html [4] Jacek P. Dmochowski, Matthew A. Bezdek, Brian P. Abelson, John S. Johnson, Eric H. Schumacher, and Lucas C. Parra. 2014. Audience Preferences Are Predicted by Temporal Reliability of Neural Processing. Nature Communications 5, 1 (2014), 4567. doi:10.1038/ncomms5567 [5] Jacek P. Dmochowski, Paul Sajda, Joao Dias, and Lucas C. Parra. 2012. Correlated Components of Ongoing EEG Point to Emotionally Laden Attention—A Possible Marker of Engagement? Frontiers in Human Neuroscience 6 (2012), 112. doi:10. 3389/fnhum.2012.00112 [6] K. Fang, J. Qu, Z. Ding, J. Ding, and L. Bu. 2026. ASAC-Net: A Novel Multimodal Alignment-Complementary Fusion Framework for EEG-fNIRS Emotion Recognition. Information Fusion 134 (2026), 104329. doi:10.1016/j.inffus.2026.104329 [7] Uri Hasson, Yuval Nir, Ifat Levy, Galit Fuhrmann, and Rafael Malach. 2004. Intersubject Synchronization of Cortical Activity During Natural Vision. Science 303, 5664 (2004), 1634–1640. doi:10.1126/science.1089506 [8] Bo Hjorth. 1970. EEG Analysis Based on Time Domain Properties. Electroencephalography and Clinical Neurophysiology 29, 3 (1970), 306–310. doi:10.1016/ 0013-4694(70)90143-4 [9] Jukka-Pekka Kauppi, Iiro P. Jääskeläinen, Mikko Sams, and Jussi Tohka. 2010. Inter-Subject Correlation of Brain Hemodynamic Responses During Watching a Movie: Localization in Space and Frequency. Frontiers in Neuroinformatics 4 (2010), 5. doi:10.3389/fninf.2010.00005 [10] Jason J. Ki, Simon P. Kelly, and Lucas C. Parra. 2016. Attention Strongly Modulates Reliability of Neural Responses to Naturalistic Narrative Stimuli. Journal of Neuroscience 36, 10 (2016), 3092–3101. doi:10.1523/JNEUROSCI.2942-15.2016 [11] Sander Koelstra, Christian Mühl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. 2012. DEAP: A Database for Emotion Analysis Using Physiological Signals. IEEE Transactions on Affective Computing 3, 1 (2012), 18–31. doi:10.1109/TAFFC.2011.15 [12] Xiang Li, Dawei Song, Peng Zhang, Yazhou Zhang, Yuexian Hou, and Bin Hu. 2018. Exploring EEG Features in Cross-Subject Emotion Recognition. Frontiers 9

Related documents

• Low-Cost Video--Time Priors as a Strong Baseline for EEG--fNIRS Emotion Regression on Familiar Videos#1188329
• Improving ToM in Children With ASD Through VPT Training: Behavioral and fNIRS Study#1320090
• NeuroECG: ECGFounder-Based Deep ECG Representation for EEG-Free Neurological Prognostication After Cardiac Arrest#1319163
• NeuroECG: ECGFounder-Based Deep ECG Representation for EEG-Free Neurological Prognostication After Cardiac Arrest#1013048
• Predicting Illness Trajectories In Fully Remitted Major Depression Using Concurrent TBS/fNIRS#901659