arXiv CSopen access

STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition Nandani Sharmaa,∗ , Varun Sharmaa,b,∗ and Dinesh Singha a Vision Intelligence and Machine Learning (VIML) Group, School of Computing and Electrical Engineering, Indian Institute of Technology

arXiv:2606.28083v1 [cs.CV] 26 Jun 2026

Mandi, Mandi, 175005, Himachal Pradesh, India b Indian Institute of Information Technology Bhagalpur, Sabour Rd, Bhagalpur, 813210, Bihar, India

ARTICLE INFO

ABSTRACT

Keywords: Micro-expression recognition action unit transformer graph attention networks spatio-temporal representations

Micro-expression recognition is challenging due to extremely subtle and short-lived facial muscle movements. Existing approaches depend heavily on apex–onset frames and often ignore fine-grained inter-frame dynamics. They also treat spatial and temporal information separately, failing to jointly model “where” facial actions occur and “when” they evolve. Fixed ROI connectivity and loosely fused action unit (AU) cues further limit structured temporal alignment and reduce cross-dataset generalization. As a result, the learned representations become highly specialized to the training dataset and fail to generalize when tested on datasets containing different recording conditions, ethnicities, camera setups, illumination variations, frame rates, and expression distributions. To address these limitations, we propose STAG, a dynamic ROI–AU-coupled spatial–temporal network that jointly models motion flow and adaptive facial connectivity. The framework first extracts optical flow from discriminative frames using magnitude-based selection and temporal attention to capture the most informative motion patterns. A dual-branch architecture is then employed, consisting of an enhanced graph attention network for structured spatial reasoning over facial regions and a transformer encoder for full-sequence temporal modeling. A bidirectional cross-attention module enables mutual refinement between spatial and temporal features, tightly coupling motion flow with evolving ROI relationships. Additionally, AU-guided dynamic connectivity allows facial region interactions to adapt to muscle activation patterns. The transformer preserves subtle inter-frame dynamics missed by apex-based approaches, improving semantic consistency and interpretability for XAI-based microexpression recognition. The fused representation is optimized using focal loss and evaluated on benchmark datasets: CASME II, 4DME, DFME, NaME, SAMM, and SMIC-HS. Extensive qualitative and quantitative experiments demonstrate the robustness, generalization capability, interpretability, and computational efficiency of the proposed framework. The results confirm the effectiveness of adaptive relational reasoning, AU-guided dynamic connectivity, and deep spatial–temporal feature fusion for accurate and explainable micro-expression recognition across diverse datasets.

Graph-based MER methods [13, 69, 66, 4, 36, 35] further model facial regions as structured ROIs using GNNs, Micro-expressions are involuntary, extremely brief faimproving relational reasoning and interpretability. Howcial muscle movements that reveal hidden emotions [62]. ever, most approaches rely on static adjacency matrices, Despite their importance in deception detection, security, loosely integrate AU cues [53, 13, 49], and independently healthcare, and human–computer interaction, micro-expression optimize spatial and temporal learning, limiting their ability recognition (MER) remains highly challenging due to their to model dynamic facial interactions and generalize across low intensity, short duration, localized facial activations, and datasets. limited annotated data [28, 30]. These characteristics make To overcome the limitations of local convolutional modrobust and generalized MER difficult, requiring models that eling, transformer-based MER frameworks have recently can effectively capture subtle spatial-temporal dynamics. attracted increasing attention due to their ability to capture With deep learning, CNN, RNN, and 3D CNN-based long-range temporal dependencies through self-attention methods [38, 47, 6] and later hybrid and transformer-based mechanismss [41]. MMTNet [43] introduced multi-modal models [48, 27, 62] significantly improved performance. multi-scale transformer learning for joint motion and appearHowever, CNN-based approaches have limited temporal reance modeling, while HLoRA-MER [33] utilized parameterceptive fields and weak interpretability, while transformerefficient LoRA fine-tuning over DinoV2 features to improve based methods [41, 43, 25, 34] better capture long-range subtle facial representation learning with reduced compudependencies but often decouple spatial and temporal modtational cost. MER-CLIP [25] further integrated visualeling. language alignment by associating action unit (AU) descrip∗ Corresponding author tions with visual facial dynamics. In addition, MOL [31] [email protected] (N. Sharma); jointly optimized MER, optical flow estimation, and [email protected] (V. Sharma) mark detection within a unified transformer-graph-style https://sites.google.com/view/nandanisharma/home/ (N. Sharma) convolutional framework. More recently, SODA4MER [59] ORCID (s): 0000-0003-0096-9754 (N. Sharma); 0000-0001-8889-9847 proposed an apex-free self-supervised framework based (D. Singh) on oriented deformation estimation, while CausalNet [67]

1. Introduction

N. Sharma et al.: Preprint submitted to Elsevier

Page 1 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

improved robustness against key-frame annotation errors using causal motion reasoning. ME-NAS further explored neural architecture search for automated MER network optimization. Despite these advances, many transformerbased methods still treat spatial structure modeling and temporal sequence learning independently without explicitly coupling facial region interactions with temporal evolution. To explicitly model relationships among facial regions, graph neural networks (GNNs) have emerged as an effective paradigm for MER. By representing facial landmarks or regions of interest (ROIs) as graph nodes and their anatomical relationships as edges, graph-based methods can perform structured relational reasoning among facial muscle movements [10]. Early graph-based MER methods such as AU-GCN [13] and STA-GCN [69] incorporated facial action units into graph learning to capture local muscle interactions. Subsequently, GCL [12] introduced graph contrastive learning to model temporal variation in ME sequences, while AGT [66] dynamically constructed adaptive ROI graphs through graph attention mechanisms. ATM-GCN [60] further improved clip-level temporal motion aggregation using adaptive graph convolution. More recent graph-based approaches such as SGCN [39], GTS-GN+AU [49], OFVIGNet [61], GTA [45], ADSS [15], DGAT [46], and AU-GCNCUR [4] enhanced graph learning through stochastic graph structures, geometric graph reasoning, transformer integration, and adaptive discriminative region selection. These methods demonstrated that graph reasoning can effectively capture structured spatial dependencies among facial regions and improve interpretability. Nevertheless, existing graph-based MER methods still face several fundamental challenges. First, most methods rely on predefined or static adjacency matrices derived from facial geometry priors, which fail to capture dynamically evolving muscle interactions during micro-expression progression. Second, AU information is often integrated in a loosely coupled manner without temporally aligned relational reasoning. Third, spatial graph learning and temporal sequence modeling are commonly optimized independently, preventing mutual refinement between where facial motions occur and when they evolve over time. Consequently, existing methods struggle to jointly model adaptive spatial connectivity and fine-grained temporal evolution in a unified framework. The facial action coding system (FACS) [3] provides valuable physiological priors for understanding facial expressions through AUs. Although several MER methods incorporate AU cues [53, 13, 49], they gen- erally fail to exploit dynamic AU-driven connectivity evolu- tion across temporal sequences. Furthermore, most existing approaches exhibit limited cross-dataset generalization due to overfitting on small-scale MER datasets and insufficient modeling of adaptive relational dependencies. To address these limitations, we propose STAG (SpatioTemporal Evolving Structural Representation of Action Units for Micro-expression RecoGnition), a unified spatial–temporal framework that jointly models motion flow N. Sharma et al.: Preprint submitted to Elsevier

and adaptive ROI connectivity. It extracts discriminative optical flow using magnitude-based selection and temporal attention, followed by dual-branch graph attention and transformer-based modeling. A bidirectional cross-attention mechanism tightly couples spatial and temporal features, while AU-guided dynamic graphs enable evolving facial connectivity. This unified design improves interpretability, captures fine-grained motion evolution, and enhances crossdataset generalization for robust MER. To tightly couple spatial and temporal representations, we introduce a bidirectional cross-attention module that enables mutual refinement between graph-based ROI features and temporal motion representations. This design allows the framework to simultaneously learn where subtle facial motions occur and when they evolve across the full sequence. Furthermore, dynamic ROI connectivity is adaptively modulated through AU-guided attention, allowing the graph structure to evolve according to facial muscle activation patterns instead of relying on fixed adjacency matrices. Compared with existing graph-based MER methods, the proposed dynamic AU-guided connectivity provides stronger semantic consistency, improved interpretability, and enhanced explainability for explainable MER (XAI-MER). Extensive experiments on six benchmark datasets (CASME II [55], 4DME [18], DFME [68], NaME [24], SAMM [2], SMICHS [19]) under LOSO, Group K-Fold, and Stratified KFold protocols demonstrate that STAG achieves state-ofthe-art (SOTA) performance, robustness, and strong crossdataset generalization. The main contributions of this work are summarized as follows: • We propose STAG, a unified dynamic ROI–AU- coupled spatial-temporal framework for MER that jointly models adaptive facial region connectivity and fullsequence motion evolution within a single learning architecture. • We design an enhanced graph attention network (EGAT) as the core spatial reasoning module, which dynamically captures AU-guided facial region dependencies and enables adaptive graph learning that evolves with muscle activation patterns. • STAG uses a bidirectional cross-attention mechanism between the enhanced GAT and a transformer-based temporal encoder, enabling fine-grained mutual refinement of spatial and temporal representations while preserving subtle inter-frame motion dynamics. The remainder of this paper is organized as follows. Section 2 reviews related work and discusses SOTA methods in micro-expression recognition. Section 3 presents the proposed STAG framework and its methodology. Section 4 describes the experimental setup and reports the results along with comprehensive analyses. Finally, section 5 concludes the paper and outlines potential directions for future work.

Page 2 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

2. Related Work Early MER methods primarily relied on optical-flowguided CNN architectures to capture subtle facial motion patterns. OFF-ApexNet employed handcrafted optical-flow features and GAN-based augmentation to improve motion representation learning [22], while Graph-TCN incorporated motion magnification and temporal convolution to enhance local motion perception [14]. DSTAN and ConvLSTM further explored dual-stream attention mechanisms and recurrent temporal modeling for spatio-temporal feature extraction [47, 38]. Although these approaches achieved promising performance, they primarily depend on onset-apex representations and local receptive fields, limiting their ability to capture complete temporal evolution and long-range dependencies throughout the micro-expression sequence. Furthermore, the learned CNN features often lack physiological interpretability because facial muscle interactions are not explicitly modeled. To address these limitations, the proposed STAG framework leverages a transformer encoder to model the entire temporal sequence rather than relying solely on apex-based information. By capturing long-range temporal dependencies and integrating an E-GAT for ROI-level relational reasoning, STAG learns more discriminative and interpretable spatio-temporal representations. Subsequent research focused on multi-scale feature extraction and dual-branch fusion strategies. LRCN-STAN integrated CBAM and LSTM modules for attentive spatiotemporal learning, while TFT combined CNN and Swin Transformer features to capture hierarchical facial representations [65, 48]. DSBICNet, DBDE-Net, and SFML-Net further enhanced motion-sensitive feature learning through dual-stream architectures and local-global feature refinement [37, 44, 63]. Similarly, MARNet, FDP, MADV-Net, and MDMO improved MER performance through motion enhancement, dynamic ranking, self-supervised learning, and efficient optical-flow estimation [54, 32, 11, 64]. Additional methods such as SCFRAM, MPFNeT, DLRRF-MER, 3DCNN, Concat-CNN, and LSSNet investigated supervised contrastive learning, multi-prior feature fusion, long-video spotting, and lightweight feature extraction strategies [17, 27, 7, 6, 56, 58]. Despite their effectiveness, most of these methods learn spatial and temporal information independently and perform feature fusion only at later stages, which restricts effective interaction between facial spatial structures and temporal dynamics. In contrast, STAG introduces a bidirectional graphtransformer cross-attention mechanism that enables continuous information exchange between graph-based spatial reasoning and transformer-based temporal modeling. Combined with dynamic ROI-AU graph construction, temporally smoothed adjacency updates, and AU-guided dynamic connectivity, the proposed framework learns adaptive facial relationships and unified spatio-temporal representations, effectively overcoming the limitations of existing CNNbased and dual-branch MER approaches.

N. Sharma et al.: Preprint submitted to Elsevier

2.1. Graph-based MER Graph-based MER methods model facial muscle dependencies and AU relationships using graph convolution and relational learning. AU-GCN pioneered facial graph representation learning by integrating landmark-aware node learning and AU fusion [13]. However, graph connectivity remains largely predefined and static, limiting adaptation to changing facial muscle interactions, STA-GCN models spatio-temporal AU dependencies but still relies on predefined graph structures and cannot dynamically update facial relationships during expression evolution [69]. GCL introduced graph contrastive learning for temporal variation modeling [12]. However, graph connectivity remains weakly adaptive and does not explicitly incorporate evolving AU relationships. AGT introduces adaptive graph attention but still lacks tight temporal coupling and AU-guided dynamic evolution [66]. Although these methods improved structured facial representation learning, most rely on pre-defined or weakly adaptive graph connectivity. Recent graph-based frameworks focused on dynamic graph construction and temporal dependency modeling. ATM-GCN improves temporal motion aggregation but spatial and temporal learning remain relatively independent [60], while SGCN improves generalization through stochastic graph learning but lacks physiologically meaningful AUdriven connectivity evolution [39]. Although AU information is included, the AU integration remains loosely coupled and not temporally aligned [49]. OFVIG-Net directly learned graph representations from optical-flow patches using Vision-GNN [61], whereas GTA combined graph convolution and transformers for AU-aware relational modeling [45]. ADSS selected discriminative graph regions adaptively for subtle motion representation [15]. More recently, SOFP exploited bilateral facial symmetry and region-aware attention fusion for robust local-global motion learning [57], while AU-GCN-CUR extended graph reasoning for digitalhuman micro-expression rendering [4]. These methods improve graph learning using Vision GNNs, transformers, or discriminative region selection. However, most still depend on static or weakly adaptive graph structures and fail to jointly optimize spatial-temporal interaction. To overcome the limitations of existing graphbased MER methods, STAG introduces a dynamic ROI-AU graph construction mechanism that adaptively models facial muscle interactions. The framework further employs temporally smoothed adjacency updates and an E-GAT to capture evolving spatial relationships. In addition, a bidirectional graph-transformer cross-attention module enables effective integration of spatial and temporal information, while AU-guided dynamic connectivity ensures physiologically meaningful feature learning. Together, these components provide adaptive relational reasoning and robust spatiotemporal representation learning, effectively addressing the shortcomings of prior graph-based approaches.

Page 3 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition Table 1 Comparison of representative graph-based, transformer-based, and AU-guided micro-expression recognition methods with the proposed STAG framework. “Partial” indicates that the capability is only indirectly or partially incorporated. Component Graph-based Spatial Modeling Dynamic Graph Construction AU Guidance Transformer Encoder Temporal Sequence Modeling Graph–Transformer Fusion Bidirectional Cross-Attention AU-driven Adaptive Connectivity Temporal Graph Evolution Full-Sequence Modeling (Beyond Apex) Explainability-Oriented Design Cross-Dataset Generalization Focus

AU-GCN ✓ × ✓ × Partial × × Partial × × ✓ ×

STA-GCN ✓ × ✓ × ✓ × × Partial Partial Partial ✓ ×

GCL ✓ Partial × × ✓ × × × Partial Partial Partial ×

AGT ✓ ✓ × ✓ Partial Partial × × × Partial ✓ ×

ATM-GCN ✓ Partial × × ✓ × × × Partial Partial Partial ×

2.2. Transformer and Foundation-Model-based MER Transformer-based MER frameworks have demonstrated superior capability in modeling long-range spatio-temporal dependencies. Captures hierarchical global dependencies but does not explicitly model dynamic facial region interactions [62], while MMTNet employed multi-modal multiscale transformers with cross-modal contrastive learning but treats spatial structure and temporal evolution separately [43]. HLoRA-MER and DGAT leveraged DinoV2 and parameter-efficient LoRA fine-tuning to improves efficiency through LoRA fine-tuning but remains dependent on appearance features and apex-centered learning [33, 46]. Although these methods achieve strong representation learning performance, most still rely heavily on apex-based motion extraction and weak graph-temporal interaction. Several recent approaches integrated self-supervised learning, causal reasoning, and multimodal learning. MERCLIP aligned AU-guided textual semantics with visual motion representations through CLIP-based cross-modal learning but does not explicitly learn evolving graph structures [25]. MOL jointly optimized MER, optical flow estimation, and landmark detection using transformer-graphstyle convolution but lacks explicit dynamic graph-temporal interaction [31]. SODA4MER proposed an apex-free selfsupervised deformation learning framework but does not incorporate structured graph reasoning [59], while CausalNet improved robustness against key-frame annotation errors through causal motion reasoning but lacks dynamic ROI connectivity modeling [67]. ME-NAS introduced a neural architecture search strategy specialized for MER [42]. In summary, existing CNN, graph, and transformerbased MER methods have achieved promising performance; however, they often model spatial and temporal information independently, rely on static ROI relationships, or employ apex-centered representations that overlook subtle interframe dynamics. Moreover, the interaction between facial muscle activations, spatial dependencies, and temporal evolution remains insufficiently explored. To address these limitations, the proposed STAG framework introduces a unified and interpretable architecture that jointly models dynamic ROI connectivity, AU-guided relational reasoning, and fullsequence temporal evolution. By integrating graph-based spatial learning with transformer-based temporal encoding N. Sharma et al.: Preprint submitted to Elsevier

GTS+GN+AU ✓ Partial ✓ × Partial × × Partial × Partial ✓ ×

OFVIG-Net ✓ Partial × × Partial × × × × Partial Partial ×

GTA ✓ ✓ ✓ ✓ ✓ ✓ × Partial Partial ✓ Partial ×

DGAT ✓ ✓ Partial ✓ ✓ ✓ × Partial × ✓ Partial Partial

MER-CLIP × × ✓ ✓ ✓ × × ✓ × ✓ ✓ ✓

MOL ✓ Partial × ✓ ✓ Partial × × × ✓ Partial Partial

STAG ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

through bidirectional cross-attention, STAG enables mutual refinement between ‘where” and ‘when” micro-expressions occur. Furthermore, AU-modulated graph attention dynamically adapts facial relationships according to underlying muscle activations, enhancing both representation learning and interpretability. Consequently, STAG provides more discriminative spatial-temporal features, improved crossdomain generalization, and stronger explainability for robust micro-expression recognition across diverse benchmark datasets. Table 1 summarizes the methodological characteristics of representative graph-based, transformer-based, and AUguided micro-expression recognition approaches. While existing methods incorporate subsets of these capabilities, none simultaneously integrate dynamic graph construction, temporal graph evolution, bidirectional cross-attention, AUdriven adaptive connectivity, and unified ROI–AU coupled learning. The proposed STAG framework is designed to bridge these gaps by jointly modeling evolving spatial relationships and temporal dynamics within a unified graphtransformer architecture.

3. Methodology The proposed framework processes a micro-expression video through a unified pipeline consisting of geometric feature extraction, optical flow modeling, graph-based spatial reasoning, temporal transformer modeling, and cross-modal fusion. Given an input video, the model first converts raw frames into structured motion representations, which are then encoded using a joint Graph-Transformer architecture with AUs conditioning for final classification (shown in the Figure 1).

3.1. Problem Formulation Let 𝐗 ∈ ℝ𝑇 ×𝑅×2 denote a temporally-standardized optical flow sequence, where 𝑇 is the fixed number of temporal frames, 𝑅 is the number of facial ROIs, and the trailing dimension of 2 captures the horizontal and vertical flow components (𝑢𝑡,𝑟 , 𝑣𝑡,𝑟 ) for frame 𝑡 and ROI 𝑟. The goal is to predict the micro-expression class 𝑦̂ ∈  via: ( ) 𝑦̂ = arg max 𝑓𝜃 𝐗, 𝐯au , (1) 𝑐∈

Page 4 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

INPUT: VIDEO SEQUENCE (T’)

OPTICAL FLOW REPRESENTATION & MOTION EXTRACTION

FACE ALIGNMENT & ROI CONSTRUCTION

FLOW VECTORS

Pt,r Xt,r X

H(0)

Proposed STAG Framework

E-GAT

A T P

Pool MLP layers

Graph Branch (Spatial Reasoning) C L S

TE

AUs Localization

T ok en

Transformer Branch (Temporal Reasoning)

Pool Cross Attention C2

AUs Features

CN

Concatenation

Trainable

C1

MER Class Prediction

Figure 1: Overview of the proposed STAG framework for MER. Given an input facial video sequence, facial landmarks are detected and ROIs are constructed around key facial components, including the eyes, eyebrows, and mouth. Dense optical flow is then computed to capture subtle facial motions, producing motion intensity maps and flow vectors represented as feature tensor 𝐗. These features are projected into a latent embedding space through a MLP, yielding the initial representation 𝐇(0) . Simultaneously, facial AUs are localized and corresponding AU features are extracted. The proposed STAG architecture consists of a graph branch for spatial reasoning using E-GAT and a transformer branch for temporal reasoning. The outputs of both branches are integrated through cross-attention and pooling operations to learn complementary spatial-temporal representations. The fused features are further concatenated with AU features and passed through trainable MLP layers to perform micro-expression classification, producing the final expression class predictions. Flame icons denote trainable modules within the framework.

where 𝑓𝜃 (⋅) is the proposed network parameterized by the set of all trainable parameters 𝜃, 𝑐 indexes over the set of microexpression classes , and 𝐯au ∈ {0, 1}𝑁AU is the optional binary AU indicator vector of length 𝑁AU .

3.2. Face Alignment and ROI Construction A micro-expression video is represented as a sequence of frames  = {𝐼𝑡 }𝑇𝑡=1 , where each frame 𝐼𝑡 ∈ ℝ𝐻×𝑊 ×3 is N. Sharma et al.: Preprint submitted to Elsevier

an RGB image. A face-centred crop of size 256 x 256 is extracted using the eye-corner landmarks (points 39 and 42) to define scale and center. Six anatomical ROI bounding boxes are derived from landmark subsets: left eyebrow (points 17– 21), right eyebrow (22–26), left eye (36–41), right eye (42– 47), nose tip (30–35), and mouth (48–67). Within each ROI a set of key landmark positions defines finer sub-windows, giving 𝑅 = 18 motion channels in total (4 left-eyebrow, Page 5 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

4 right-eyebrow, 6 mouth, 2 nose-tip, 1 left-eye, 1 righteye) [50, 17]. For the initial frame, a frontal-face detector and a 68-point facial landmark predictor [9] are applied to extract a set of 2D facial keypoints, and landmarks detected in the first frame are tracked across the sequence: { }68 𝑡 = 𝐥(𝑖) , 𝑡 𝑖=1

(

)

(𝑖) (𝑖) 𝐥(𝑖) ∈ ℝ2 . 𝑡 = 𝑥𝑡 , 𝑦𝑡

(2)

To minimize inter-subject variation and ensure scale invariance across subjects, we compute the inter-ocular distance using predefined eye-corner landmarks indexed by 𝑎 and 𝑏: ‖ ‖ 𝑑𝑒 = ‖𝐥(𝑎) − 𝐥(𝑏) . 𝑡 ‖ ‖𝑡 ‖2

(3) 𝑑

The normalization scaling coefficient is defined as 𝑔 = 2𝑒 , and the stable geometric center of the face is computed as the midpoint between the two eye landmarks: ( ) 1 (𝑎) (𝑏) 𝐜𝑡 = 𝐥𝑡 + 𝐥𝑡 . (4) 2 Using 𝐜𝑡 and 𝑔, each frame is cropped and resized to a uniform target spatial resolution. The face is segmented into 𝑅 = 18 finer motion channels derived from landmark subsets 𝑟 corresponding to key facial muscle groups. For each ROI 𝑟 ∈ {1, … , 𝑅} the bounding box boundaries at frame 𝑡 are bounded by: (𝑖) 𝑥min 𝑡,𝑟 = min 𝑥𝑡 ,

(𝑖) 𝑥max 𝑡,𝑟 = max 𝑥𝑡 ,

(5)

(𝑖) 𝑦min 𝑡,𝑟 = min 𝑦𝑡 ,

(𝑖) 𝑦max 𝑡,𝑟 = max 𝑦𝑡 .

(6)

𝑖∈𝑟

𝑖∈𝑟

𝑖∈𝑟

𝑖∈𝑟

The localized rectangular spatial region 𝑡,𝑟 in pixel space is formulated as: min max max 𝑡,𝑟 = [𝑥min 𝑡,𝑟 , 𝑥𝑡,𝑟 ] × [𝑦𝑡,𝑟 , 𝑦𝑡,𝑟 ].

(7)

3.3. Optical Flow Representation The dense optical flow field between consecutive frames is defined over the localized grid space as: ( ) 𝐅𝑡,𝑟 (𝑝) = 𝑢𝑡,𝑟 (𝑝), 𝑣𝑡,𝑟 (𝑝) ∈ ℝ2 , (8) where 𝑝 ∈ 𝑡,𝑟 represents a specific pixel location within the ROI boundary. The local motion intensity magnitude is evaluated by: √ 𝑚𝑡,𝑟 (𝑝) = 𝑢2𝑡,𝑟 (𝑝) + 𝑣2𝑡,𝑟 (𝑝). (9) To suppress structural noise and background movement, we apply a threshold 𝜏𝑡,𝑟 to filter out low-motion intensities, retaining only the dominant motion pixel set 𝑡,𝑟 : 𝑡,𝑟 = {𝑝 ∈ 𝑡,𝑟 ∣ 𝑚𝑡,𝑟 (𝑝) ≥ 𝜏𝑡,𝑟 },

(10)

where 𝜏𝑡,𝑟 is a percentile-based threshold engineered to filter the top-𝜌 proportion of high-motion components. The final aggregated dominant motion vector 𝐱𝑡,𝑟 for ROI 𝑟 at time N. Sharma et al.: Preprint submitted to Elsevier

step 𝑡 is calculated via average pooling over the high-motion pixel set: [ ] 1 ∑ 𝑢𝑡,𝑟 (𝑝) ∈ ℝ2 . (11) 𝐱𝑡,𝑟 = |𝑡,𝑟 | 𝑝∈ 𝑣𝑡,𝑟 (𝑝) 𝑡,𝑟

Compiling this operation over all sequences constructs the global standardized optical flow input tensor 𝐗 ∈ ℝ𝑇 ×𝑅×2 .

3.4. ROI Embedding To project the low-dimensional flow coordinates into a richer latent space, each regional motion vector 𝐱𝑡,𝑟 is processed via a two-layer multi-layer perceptron (MLP) mapping: ( ) 𝐡𝑡,𝑟 = GELU 𝑊2 GELU(𝑊1 𝐱𝑡,𝑟 + 𝑏1 ) + 𝑏2 ∈ ℝ𝐷𝑒 , (12) where 𝑊1 ∈ ℝ𝐷ℎ ×2 maps the inputs to a hidden dimension 𝐷ℎ with bias 𝑏1 ∈ ℝ𝐷ℎ , and 𝑊2 ∈ ℝ𝐷𝑒 ×𝐷ℎ projects the hidden features to the target embedding dimension 𝐷𝑒 with bias 𝑏2 ∈ ℝ𝐷𝑒 . This step transforms the sequence into the initial node feature tensor 𝐇(0) ∈ ℝ𝑇 ×𝑅×𝐷𝑒 .

3.5. Evolving Graph Formulation (E-GAT) The spatial structural relationships among the 𝑅 facial regions are modeled as an evolving fully-connected graph sequence 𝑡 = (, ). The node features are initialized from 𝐇(0) . To establish an initial reference topology, a datadriven cosine-similarity adjacency matrix 𝐀(0) ∈ [0, 1]𝑅×𝑅 is computed across the first 𝑇0 = min(5, 𝑇 ) initialization frames: ( ⊤ ) ̃𝐡 ̃𝐡𝑗 𝑖 (13) 𝐀(0) = sof tmax 𝑗 𝑖𝑗 𝜏 ̄𝐡 ∑𝑇0 (0) where ̄𝐡𝑖 = 𝑇1 𝑡=1 𝐡𝑡,𝑖 and ̃𝐡𝑖 = ‖̄𝐡 𝑖‖ , and 𝜏 = 0.1 0 𝑖 2 is a sharpening temperature parameter. The spatial node properties are subsequently refined through 𝐿𝑔 stacked EGAT layers.

3.5.1. Attention Score Computation For an attention head 𝑘 ∈ {1, … , 𝐾}, the edge correlation score 𝑒𝑡𝑖𝑗𝑘 at frame 𝑡 from node 𝑖 to node 𝑗 merges scaled dot-product structures with parametric additive GAT branches: ⎞ ⎛ ⎟ ⎜ ⎜𝐐 𝐊 ⊤ ⎟ 𝑡𝑖𝑘 𝑡𝑗𝑘 𝑒𝑡𝑖𝑗𝑘 = LReLU⎜ √ + (𝐐𝑡𝑖𝑘 ⊙ 𝜶𝑘𝑙𝑒𝑓 𝑡 )𝟏 + (𝐊𝑡𝑗𝑘 ⊙ 𝜶𝑘𝑟𝑖𝑔ℎ𝑡 )𝟏 +𝑏𝑖𝑗 ⎟ ⎜ ⎟ 𝑑 ⎜⏟⏞⏞⏞⏟ℎ⏞⏞⏞⏟ ⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟ ⎟ additive (GAT) ⎜ ⎟ ⎝ dot-product ⎠ + 𝛿𝑖𝑗 𝛾loop , (14) (𝑘) (𝓁) (𝑘) where 𝐐𝑡𝑖𝑘 = 𝐡(𝓁) 𝑡,𝑖 𝐖𝑄 and 𝐊𝑡𝑗𝑘 = 𝐡𝑡,𝑗 𝐖𝐾 represent the query and key transformations using projections 𝐖(𝑘) , 𝐖(𝑘) ∈ ℝ𝐷𝑒 ×𝑑ℎ (𝑑ℎ = 𝐷𝑒 ∕𝐾). 𝟏 ∈ ℝ𝑑ℎ denotes an 𝑄 𝐾

Page 6 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

all-ones vector used to sum the element-wise products into a right scalar. The parameter vectors 𝜶𝑘lef t , 𝜶𝑘 ∈ ℝ𝑑ℎ govern the additive attention weights, 𝑏𝑖𝑗 is a learnable spatial structural bias, 𝛿𝑖𝑗 acts as the Kronecker delta, and 𝛾loop controls selfloop information flow.

3.5.2. Temporally-Smoothed Dynamic Adjacency To maintain temporal continuity across frames, the adjacency transitions are smoothed using an autoregressive update controlled by a learnable gating parameter 𝜆 = 𝜎(𝛾𝑠 ) ∈ (0, 1). The recursion is initialized using the cosinê (0) = 𝐀(0) . The head-aggregated attention similarity graph 𝐀 (𝑡) 𝑅×𝑅 matrix 𝐄 ∈ ℝ at frame 𝑡 is obtained by averaging the raw attention scores across all 𝐾 heads: 1 ∑ 𝑒 . 𝐾 𝑘=1 𝑡𝑖𝑗𝑘 𝐾

𝐄(𝑡) 𝑖𝑗 =

(15)

̂ (𝑡) ∈ The temporally-smoothed dynamic adjacency matrix 𝐀 𝑅×𝑅 ℝ is then recursively updated via: ( ) ̂ (𝑡−1) , ̂ (𝑡) = 𝜆 sof tmax 𝐄(𝑡) + (1 − 𝜆) 𝐀 (16) 𝐀 where the sof tmax(⋅) operator normalizes the aggregated matrix row-wise. Node representations are then updated using residual connections and Layer Normalization (LN). 𝐇(𝓁) = 𝐇(𝓁) [𝑡, ∶, ∶] ∈ ℝ𝑅×𝐷𝑒 denotes the node-feature 𝑡 matrix of frame 𝑡: ( ( )) ⊤ ̂ (𝑡) 𝐇(𝓁+1) = LN 𝐇(𝓁) , (17) 𝑡 𝑡 + Dropout 𝐀 ⋅ 𝐕𝑡 𝐖𝑂 where 𝐕𝑡 = 𝐇(𝓁) 𝑡 𝐖𝑉 uses value mapping weights 𝐖𝑉 , 𝐖𝑂 ∈ ℝ𝐷𝑒 ×𝐷𝑒 .

3.5.3. Temporal Attention Pooling & Graph-Level Representation To condense the frame representations along the temporal axis, frame salience metrics 𝜔𝑡 are evaluated using a localized attention network: ( ) 𝑒𝑠𝑡 , 𝑠𝑡 = 𝐰⊤ tanh 𝐖3̄𝐡𝑡 + 𝐛3 , (18) 𝜔𝑡 = ∑𝑇 2 𝑠 ′ 𝑡 𝑡′ =1 𝑒 ∑ (𝐿𝑔 ) where ̄𝐡𝑡 = 𝑅1 𝑅 ∈ ℝ𝐷𝑒 , with parameters 𝐖3 ∈ 𝑟=1 𝐡𝑡,𝑟 𝐷 ×𝐷 𝐷 ℝ 𝑎 𝑒 , 𝐛3 ∈ ℝ 𝑎 , and context vector 𝐰2 ∈ ℝ𝐷𝑎 . Concurrently, to yield the token sequence for the subsequent fusion module, the node features after the final E-GAT layer 𝐇(𝐿𝑔 ) ∈ ℝ𝑇 ×𝑅×𝐷𝑒 are structurally aggregated along the time dimension using the computed weights 𝜔𝑡 to construct a clean spatial node matrix 𝐆:

aggregated, and projected into a spatial vector 𝐟graph : ( (𝐿 ) ) 𝐳𝑡 = vec 𝐇𝑡 𝑔 ∈ ℝ𝑅⋅𝐷𝑒 ,

𝐳=

𝐓 ∑

𝜔𝐭 𝐳𝐭 ∈ ℝ𝐑⋅𝐃𝐞 , (20)

𝐭=𝟏

(21)

𝐟graph = 𝐳𝐖𝑔 + 𝐛𝑔 ∈ ℝ𝐷𝑔 ,

where 𝐖𝑔 ∈ ℝ(𝑅⋅𝐷𝑒 )×𝐷𝑔 and 𝐛𝑔 ∈ ℝ𝐷𝑔 project the unified array into the global graph dimension 𝐷𝑔 .

3.6. Temporal Transformer Branch Simultaneously, a standard transformer encoder captures long-range sequence context. The initial embedding space is flattened across spatial( dimensions to yield ) temporal tokens 𝐇f lat = Reshape 𝐇(0) , [𝑇 , 𝑅 ⋅ 𝐷𝑒 ] ∈ ℝ𝑇 ×(𝑅⋅𝐷𝑒 ) . These tokens are projected to the transformer dimension 𝐷𝑡 , prepended with a learnable classification token 𝐜 ∈ ℝ1×𝐷𝑡 , and injected with learnable positional encodings 𝐏 ∈ ℝ(𝑇 +1)×𝐷𝑡 : ] [ ̂ (0) = 𝐜; 𝐇f lat 𝐖inp + 𝐏, (22) 𝐇 where 𝐖inp ∈ ℝ(𝑅⋅𝐷𝑒 )×𝐷𝑡 . The sequence is processed through 𝐿𝑡 Pre-LN transformer blocks: ( ( )) ̂ (𝓁) + MHA LN 𝐇 ̂ (𝓁) , 𝐳(𝓁) = 𝐇 (23) ( ( )) ̂ (𝓁+1) = 𝐳(𝓁) + FFN LN 𝐳(𝓁) . 𝐇 (24) The final transformer output token sequence is defined ̂ (𝐿𝑡 ) ∈ ℝ(𝑇 +1)×𝐷𝑡 . To support cross-modal token as 𝐇 matching, the pure frame embeddings are extracted as 𝐓 = ̂ (𝐿𝑡 ) ∈ ℝ𝑇 ×𝐷𝑡 . The global temporal [𝐭1 , 𝐭2 , … , 𝐭𝑇 ]⊤ = 𝐇 1∶,∶ summary feature 𝐟trans is obtained by applying layer normalization to the isolated classification token at position 0: ( ) ̂ (𝐿𝑡 ) ∈ ℝ𝐷𝑡 . 𝐟trans = LN 𝐇 (25) 0,∶

3.7. Bidirectional Cross-Attention Fusion To exchange complementary spatial and temporal information before global reduction, cross-attention operations interact at the token level between graph arrays 𝐆 ∈ ℝ𝑅×𝐷𝑒 and temporal arrays 𝐓 ∈ ℝ𝑇 ×𝐷𝑡 . Both sequences are mapped into a shared projection fusion space of dimension 𝐷𝑓 : 𝐆𝑓 = 𝐆𝐖𝑔→𝑓 ∈ ℝ𝑅×𝐷𝑓 ,

𝐓𝑓 = 𝐓𝐖𝑡→𝑓 ∈ ℝ𝑇 ×𝐷𝑓 , (26)

where 𝐖𝑔→𝑓 ∈ ℝ𝐷𝑒 ×𝐷𝑓 and 𝐖𝑡→𝑓 ∈ ℝ𝐷𝑡 ×𝐷𝑓 . The cross-attention blocks perform bidirectional aggregation using residual connections: 𝑇 ∑ (𝐿𝑔 ) ⊤ 𝑅×𝐷𝑒 𝐷𝑒 ( ) 𝐆 = [𝐠1 , 𝐠2 , … , 𝐠𝑅 ] ∈ ℝ , where 𝐠𝑟 = 𝜔𝑡 𝐡𝑡,𝑟 ∈ ℝ . 𝐆∗ = LN 𝐆𝑓 + MHA(𝑄 = 𝐆𝑓 , 𝐾 = 𝐓𝑓 , 𝑉 = 𝐓𝑓 ) , 𝑡=1 (27) (19) ( ) ∗ 𝐓 = LN 𝐓𝑓 + MHA(𝑄 = 𝐓𝑓 , 𝐾 = 𝐆𝑓 , 𝑉 = 𝐆𝑓 ) , For the global graph feature representation, the frame state (28) matrices are flattened via vectorization (vec), temporally N. Sharma et al.: Preprint submitted to Elsevier

Page 7 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

where 𝐆∗ ∈ ℝ𝑅×𝐷𝑓 and 𝐓∗ ∈ ℝ𝑇 ×𝐷𝑓 . The enhanced spatial and temporal sequences are averaged via global pooling and projected back to their native feature dimensions: 1 ∑ ∗ 𝐆 ∈ ℝ𝐷𝑓 , 𝑅 𝑟=1 𝑟 𝑅

𝐟𝑔 =

𝐷𝑔

𝐟̂𝑔 = 𝐟𝑔 𝐖𝑓 →𝑔 ∈ ℝ

,

1∑ ∗ 𝐓 ∈ ℝ𝐷𝑓 , (29) 𝑇 𝑖=1 𝑖 𝑇

𝐟𝑡 =

(30)

𝐟̂𝑡 = 𝐟𝑡 𝐖𝑓 →𝑡 ∈ ℝ , 𝐷𝑡

Table 2 Distribution of emotional labels for different classification tasks on micro-expression datasets used in this work. Dataset SMIC-HS

Class 3

Sample 164

CASME II

3

255

CASME II

5

255

SAMM

3

133

SAMM

5

159

4DME

3

215

4DME

5

267

DFME

3

474

DFME

5

474

DFME

7

474

NaME

3

506

NaME

5

318

NaME

6

506

where 𝐖𝑓 →𝑔 ∈ ℝ𝐷𝑓 ×𝐷𝑔 and 𝐖𝑓 →𝑡 ∈ ℝ𝐷𝑓 ×𝐷𝑡 . The final fused cross-modal presentation is formed via concatenation: [ ] 𝐟f usion = cat 𝐟̂𝑔 , 𝐟̂𝑡 ∈ ℝ𝐷𝑔 +𝐷𝑡 .

(31)

3.8. Action Unit Embedding Branch To inject domain-specific anatomical constraints, parsed binary Action Unit strings are mapped to an indicator vector 𝐯au ∈ {0, 1}𝑁AU . This vector is processed through an auxiliary 2-layer MLP to construct a dense embedding 𝐟au : ) ) ( ( 𝐟au = LN 𝐖au,2 GELU 𝐖au,1 𝐯au + 𝐛au,1 + 𝐛au,2 ∈ ℝ𝐷au , (32) ′

′

where 𝐖au,1 ∈ ℝ𝐷ℎ ×𝑁AU and 𝐖au,2 ∈ ℝ𝐷au ×𝐷ℎ . The final composite representation vector 𝐟f inal is conditionally assembled depending on data availability: { [ ] cat 𝐟f usion , 𝐟au if AU annotations are available, 𝐟f inal = 𝐟f usion otherwise. (33)

3.9. Classification and Class-Aware Focal Loss The feature vector 𝐟f inal is passed through a dense linear layer classification head to output the class logits. To combat severe dataset label imbalance, a multi-class label-smoothed focal loss is employed. For a given sample with true class index 𝑦𝑖 , the target probabilities 𝑦̃𝑖𝑐 are smoothed via parameter 𝜀𝑦𝑖 : { 1 − 𝜀𝑦𝑖 if 𝑐 = 𝑦𝑖 , (34) 𝑦̃𝑖𝑐 = 𝜀𝑦𝑖 if 𝑐 ≠ 𝑦𝑖 . 𝐶−1 Given the predicted softmax probability 𝑝𝑖𝑐 = sof tmax(𝐲̂ 𝑖 )𝑐 for sample 𝑖 and class 𝑐, the full batch optimization objective over batch size 𝐵 is minimized via: 1 ∑∑ 𝛼 𝑦̃ (1 − 𝑝𝑖𝑐 )𝛾𝑐 log(𝑝𝑖𝑐 ), 𝐵 𝑖=1 𝑐=1 𝑐 𝑖𝑐 𝐵

=−

𝐶

(35)

where 𝛼𝑐 balances inverse-frequency class distribution scales and 𝛾𝑐 controls the class-specific hard-sample focusing power. For clarity, Tables 11 and 12 summarize the symbols, notations, and tensor dimensions used throughout the proposed STAG framework. These tables provide a convenient reference for understanding the mathematical formulations, feature representations, and data flow across different components of the architecture. N. Sharma et al.: Preprint submitted to Elsevier

Emotion Distribution Positive (51), Negative (70), Surprise (43) Positive (32), Negative (198), Surprise (25) Positive (32), Surprise (25), Negative (63), Others (27), Repression (108) Positive (26), Negative (92), Surprise (15) Anger (57), Happiness (26), Contempt (12), Surprise (15), Others (49) Positive (48), Negative (138), Surprise (29) Positive (56), Negative (142), Surprise (29), Repression (9), Others (31) Positive (63), Negative (310), Surprise (101) Anger (39), Contempt (34), Happiness (63), Surprise (101), Others (237) Happiness (63), Disgust (129), Fear (62), Anger (39), Sadness (46), Surprise (101), Contempt (34) Positive (15), Negative (329), Surprise (162) Happiness (15), Anger (26), Disgust (69), Sadness (46), Surprise (162), others-excluded (188) Happiness (15), Anger (26), Disgust (69), Sadness (46), Surprise (162), others (188)

4. Experimental Setup We evaluate the proposed STAG framework on six widely used benchmark datasets for micro-expression recognition, whose class distributions are summarized in Table 2. Since different MER datasets adopt varying expression taxonomies and annotation protocols, we report the class distributions under multiple evaluation settings, including the original 5-class, 6-class, and 7-class categorizations used in previous benchmark studies. Furthermore, to facilitate direct and fair comparisons with existing SOTA methods, we additionally adopt the unified 3-class protocol (Positive, Negative, and Surprise) recommended by Li et al. [20]. This unified setting mitigates inconsistencies in label definitions across datasets and has become the de facto standard for Page 8 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition Table 3 Comparison with state-of-the-art methods on NaME, 4DME, and DFME datasets using UF1 and UAR metrics. The best results for each benchmark are highlighted in bold. Dataset

Method

NaME

OFF-ApexNet [5] RCN-A [52] FR [70] AMAN [51] SRMCL [1] MixFormer [24] CIT [21] STAG (ours) STAG (ours) STAG (ours)

Class Type

UF1

UAR

SP19 TIP20 PR21 ICASSP24 TAFFC24 ICMM25 AIHCIR25 – – –

3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 5-class 6-class

0.2054 0.1721 0.1882 0.1426 0.1765 0.2703 0.2949 0.8983 0.7562 0.6561

0.2000 0.1951 0.1889 0.1584 0.1840 0.2598 0.2898 0.8981 0.7777 0.6667

4DME

CCDN [18] TSFmicro [23] STAG (ours) STAG (ours)

TAFFC22 PR26 – –

5-class 5-class 3-class 5-class

0.6760 0.6852 0.9410 0.7245

– – 0.9613 0.7744

SMIC-HS

GCL [12] AGT [66] TFT [48] SGCN [40] SODA4MER [59] MMTNet [43] MADV-Net [11] CausalNet [67] DBDE-Net [44] MPFNet [27] MDMO [64] ADSS [15] OFVIG-Net [61] MOL [31] HFA-Net [62] ME-NAS [42] SCFRAM [17] DSBICNet [37] MARNet [54] LRCN-STAN [65] STAG (ours)

ICPR22 ICME23 Sensors24 NN24 CVPR25 JVCIR25 Sensors25 ICCV25 NC25 ICASSP25 AI25 TBIOM25 NC25 TPAMI25 CIS25 TIST26 SP26 MM26 TMM26 ICMLC26 –

3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class 3-class

0.7560 0.8073 0.8140 0.8510 0.8855 0.7799 0.6900 0.8405 0.8366 0.7860 0.8870 0.7384 0.6435 0.7881 0.7785 0.6717 0.7670 0.8249 0.8296 0.7950 0.7370

0.8220 0.8010 0.8881 0.8433 0.8676 0.8670 0.6400 0.7786 0.7123 0.7610 0.8297 0.8220 0.8010 0.7497

FearRef [70] Wang et al. [68] He et al. [68] MER-CLIP [25] STAG (ours)

PR22 TAFFC24 TAFFC24 TAFFC25 –

Test A Test A Test A Test A Test A

0.3410 0.4067 0.4123 0.5024 0.5110

0.3686 0.4074 0.4210 0.5115 0.5196

FearRef [70] Wang et al. [68] He et al. [68] FED-PsyAU [16] MER-CLIP [25] STAG (ours)

PR22 TAFFC24 TAFFC24 ICCV25 TAFFC25 –

Test B Test B Test B Test B Test B Test B

0.2875 0.3534 0.4016 0.3853 0.5128 0.4776

0.3228 0.3661 0.4008 0.3978 0.5120 0.4710

DFME

Ref.

cross-dataset MER evaluation. CASME II and SAMM contain high-resolution spontaneous micro-expression videos captured under controlled laboratory settings with detailed emotion and facial action annotations. SMIC-HS is one of the earliest MER datasets and provides spontaneous microexpression clips recorded at high frame rates for evaluating subtle facial motion analysis. 4DME extends conventional N. Sharma et al.: Preprint submitted to Elsevier

MER datasets by incorporating multi-view and temporal facial dynamics for robust spatial-temporal learning. DFME focuses on challenging real-world facial variations and subtle expression changes, making MER evaluation more practical and diverse. NaME further introduces naturalistic spontaneous micro-expressions collected in unconstrained

Page 9 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition Table 4 Comparison with state-of-the-art methods on CASME-II and SAMM datasets under 3-class and 5-class protocols (UF1/UAR).

Method

Ref.

CASME-II (3-class)

CASME-II (5-class)

SAMM (3-class)

SAMM (5-class)

UF1

UAR

UF1

UAR

UF1

UAR

UF1

UAR

Graph-TCN [14] STA-GCN [69] ConvLSTM [38] GCL [12] GCNs [29] AGT [66] TFT [48] ATM-GCN [60] GTS+GN+AU [50] SGCN [40] AU-GCN-CUR [4] FDP [32] SODA4MER [59] MER-CLIP [25] MMTNet [43] MADV-Net [11] CausalNet [67] DBDE-Net [44] MPFNet [27] MDMO [64] ADSS [15] OFVIG-Net [61] GTA [45] HLoRA-MER [33] MOL [31] HFA-Net [62] ME-NAS [42] SCFRAM [17] DSBICNet [37] MARNet [54] LRCN-STAN [65] DGAT [46] TSFmicro [23]

MM20 PRCV21 ACCV22 ICPR22 ICMLA23 ICME23 Sensors24 ICME24 TAFFC24 NN24 Biomimetics25 2025 CVPR25 TAFFC25 JVCIR25 Sensors25 ICCV25 NC25 ICASSP25 AI25 TBIOM25 NC25 ICUS25 VC25 TPAMI25 CIS25 TIST26 SP26 MM26 TMM26 ICMLC26 Sensors26 PR26

– 0.7096 – 0.7660 0.8597 0.9024 0.9070 0.9048 0.8120 0.7920 0.7793 0.9071 0.8870 0.9409 0.8007 0.8400 0.9748 0.9600 – 0.7040 0.9034 0.7129 0.8854 0.9094 0.8891 0.9738 0.8913 0.8970 0.9928 0.9433 0.8190 – 0.9430

– – – – 0.8520 0.9221 0.9090 0.9042 – – – – 0.8809 0.9487 – – 0.9782 0.9619 – 0.7740 – 0.7195 – – – 0.9754 0.8942 0.8980 0.9896 0.9325 0.8390 – –

0.7206 – – – – – – – – – – 0.8705 0.8141 0.8233 – 0.7800 – 0.8691 0.8190 – 0.8309 – – 0.8424 0.7585 0.8410 – 0.7590 0.8566 0.8316 – – 0.8617

– – – – – – – – – – – – 0.8418 0.8378 – – – 0.8912 – – – – – – – 0.8406 – 0.7630 0.8637 0.8252 – – –

– – 0.7892 0.7650 0.8807 0.7928 0.7090 0.7920 0.7110 0.7440 – 0.8667 – 0.8321 0.6736 – 0.8708 0.8813 – – 0.7890 0.6066 0.8207 0.8671 0.8272 0.9002 0.8635 0.8150 0.8670 0.8427 0.7300 0.8293 0.8567

– – – – 0.8618 0.7643 0.6560 0.8049 – – – – – 0.8434 – – 0.8522 0.8860 – – – 0.5787 – – – 0.8938 0.8875 0.8270 0.8641 0.8331 0.7730 0.8377 –

0.6985 – – – – – – – – – – 0.8529 0.7893 0.7721 – 0.8800 – 0.8114 0.6950 – 0.7297 – – 0.8307 0.7190 0.7868 – 0.7250 0.7576 0.8093 – 0.8116 0.7700

– – – – – – – – – – – – 0.8030 0.7414 – – – 0.8318 – – – – – – – 0.6950 – 0.7310 0.6720 0.8309 – 0.8537 –

STAG (Ours)

–

0.8742

0.9296

0.7776

0.8413

0.9175

0.9372

0.8225

0.8362

environments, improving the assessment of model generalization and robustness in real-world scenarios. The model was trained for 200 epochs using AdamW [26] with a learning rate of 1.5 × 10−4 , weight decay of 0.08, batch size of 64, and dropout of 0.3. Early stopping (patience = 100), 12 warm-up epochs, and stochastic weight averaging (SWA) starting at 75% of training with a learning rate of 5 × 10−5 were employed. MixUp (𝛼 = 0.5), CutMix (𝛼 = 0.3), testtime augmentation (TTA= 10), and class oversampling (factor = 6) were used to mitigate class imbalance arising from highly variable class distributions. Performance was evaluated under three rigorous validation protocols: Stratified KFold (SKF), Group K-Fold (GKF), and Leave- One-SubjectOut (LOSO) cross-validation. All models are implemented in PyTorch and trained on a NVIDIA RTX A6000 GPU with 48 GB memory. SWA [8] is activated during the final phase of training, and test-time augmentation (TTA) is applied at inference. N. Sharma et al.: Preprint submitted to Elsevier

To maintain consistency with prior work, all unspecified hyperparameters and implementation settings are inherited from the respective benchmark papers associated with each dataset. This includes dataset-specific preprocessing, training schedules, and evaluation configurations where applicable. By following the original experimental protocols, we ensure a fair comparison with existing methods while isolating the contribution of the proposed STAG architecture.

4.1. Results and Discussions Table 3 summarizes the performance comparison on NaME, 4DME, SMIC-HS, and DFME benchmarks. The proposed STAG consistently achieves the best results on NaME and 4DME across all evaluated class settings, demonstrating the effectiveness of the Dynamic ROI-AU Graph Transformer in capturing subtle spatial-temporal facial dynamics. On DFME Test-A, STAG achieves the highest UF1 and UAR scores, while remaining competitive on Page 10 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

Test-B against recent foundation-model-based approaches. Although the performance on SMIC-HS is comparatively lower than some SOTA methods, this can be attributed to the limited dataset size, lower expression in- tensity, and reduced sample diversity, which restrict the learning of robust graph-based representations. Overall, the results validate the strong generalization capability and effectiveness of STAG across multiple micro-expression recognition benchmarks and evaluation protocols. Despite the lower performance on SMIC-HS, STAG demonstrates superior results on the larger and more diverse CASME-II and SAMM datasets, indicating stronger generalization capability when sufficient facial variation is available. These findings suggest that the proposed framework is particularly effective in realistic micro-expression recognition scenarios where richer spatial-temporal information can be exploited. Table 4 provides a comprehensive comparison of STAG with recent SOTA methods on CASME-II and SAMM under both 3-class and 5-class settings. Overall, the results demonstrate that STAG achieves strong and competitive performance across all settings, with particularly notable improvements on the more challenging SAMM dataset. On CASME-II (3-class), STAG attains a UF1 of 0.8742 and UAR of 0.9296, which, while not the absolute highest in UF1, still reflects robust performance compared to many recent methods. This suggests that STAG maintains a balanced classification capability without overfitting to specific expression categories. In the more fine-grained CASME-II (5class) setting, STAG achieves 0.7776 UF1 and 0.8413 UAR, indicating stable generalization even as the classification task becomes more complex. Although some recent methods such as DBDE-Net and HFA-Net show higher values in this dataset, the performance gap is relatively small, suggesting that STAG remains competitive in dense-class scenarios. The most significant results are observed on the SAMM dataset, where STAG achieves 0.9175 UF1 and 0.9372 UAR (3-class) and 0.8225 UF1 and 0.8362 UAR (5-class). In both settings, STAG clearly outperforms all competing methods, including strong recent baselines such as CausalNet, DBDENet, and HFA-Net. This consistent superiority indicates that STAG is particularly effective in handling the challenging characteristics of SAMM, such as subtle motion variations, subject diversity, and class imbalance. From a methodological perspective, the overall improvements can be attributed to STAG’s ability to better capture spatiotemporal dependencies and discriminative micro-expression dynamics, which are crucial for subtle facial motion analysis. The stronger gains on SAMM further suggest enhanced cross-subject generalization and robustness, which are key limitations in prior approaches.

4.2. Ablation Study The ablation study on the SAMM dataset (Table 5) clearly validates the effectiveness of each component in the proposed STAG framework. The full model achieves the best performance (UF1 = 0.9112, UAR = 0.9335), confirming N. Sharma et al.: Preprint submitted to Elsevier

that the integration of graph modeling, transformer-based temporal learning, and AU-guided connectivity leads to optimal representation learning. Individually, both graphonly and transformer-only variants show strong but limited performance, while their combined use provides more stable and balanced results, indicating their complementary roles in capturing spatial and temporal dependencies. Further analysis shows that increasing graph depth enhances discriminative power, with the best results obtained using deeper E-GAT layers, highlighting the importance of richer relational modeling. Similarly, a 3-layer transformer provides the most stable temporal representation, while deeper configurations do not yield consistent gains, suggesting an optimal balance between capacity and overfitting. Data augmentation strategies such as MixUp, CutMix, and test-time augmentation contribute to improved generalization, with their combination producing the most reliable results. However, the most significant impact comes from AU-guided connectivity, where its removal causes a drastic performance drop (UF1: 0.9112 → 0.6231, UAR: 0.9335 → 0.7101), clearly demonstrating its critical role in guiding facial region dependencies and enhancing micro-expression discrimination. Overall, the results confirm that each component contributes meaningfully, while their unified integration is essential for achieving robust and SOTA performance in micro-expression recognition. Table 6 presents the cross-dataset evaluation results under the 3-class protocol. The proposed STAG framework demonstrates strong generalization capability when trained on large and diverse datasets such as NaME and DFME. Among all source datasets, NaME achieves the best transfer performance on SAMM, obtaining a UF1 of 0.9093 and a UAR of 0.9212, while DFME exhibits similarly robust performance with a UF1 of 0.8991 and a UAR of 0.9026. These results indicate that the learned spatial-temporal representations are highly transferable across datasets with similar annotation schemes and facial motion distributions. Furthermore, both NaME and DFME maintain competitive performance when evaluated on CASME II and 4DME, suggesting that the proposed framework effectively captures datasetinvariant micro-expression characteristics However, a notable performance decline is observed when transferring to SMIC-HS regardless of the source dataset. For example, NaME→SMIC-HS and DFME→SMICHS achieve UF1 scores of only 0.2131 and 0.3532, respectively. This degradation can be attributed to the substantial domain discrepancy between SMIC-HS and the other datasets, including differences in recording conditions, image resolution, subject demographics, and emotion labeling protocols. Interestingly, CASME II and 4DME exhibit moderate cross-dataset performance, indicating that smaller datasets can still learn transferable representations but are more sensitive to domain shifts. Overall, the results confirm that the proposed STAG framework achieves strong crossdataset robustness under the 3-class setting, particularly when trained on larger and more diverse datasets. Page 11 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition Table 5 Comprehensive ablation study of the proposed STAG framework on the SAMM dataset using stratified 5-fold validation. Category Architecture

Graph Depth

Transformer Depth

Data Augmentation

Test-Time Augmentation AU Guidance

Configuration Transformer Only Graph Only Graph + Transformer Full Model (STAG) 1 E-GAT Layer 2 E-GAT Layers 4 E-GAT Layers 2 Layers 3 Layers 5 Layers 7 Layers No Augmentation Without MixUp Without CutMix MixUp + CutMix Without TTA With TTA Without AU Guidance With AU Guidance

GL 0 3 3 3 1 2 4 3 3 3 3 3 3 3 3 3 3 3 3

TL 5 0 5 5 5 5 5 2 3 5 7 5 5 5 5 5 5 5 5

UF1 0.9104 0.9024 0.9006 0.9112 0.9086 0.9156 0.9175 0.9024 0.9175 0.9112 0.9137 0.9086 0.9011 0.9040 0.9112 0.9024 0.9112 0.6231 0.9112

Std-UF1 0.0502 0.0505 0.0693 0.0524 0.0710 0.0509 0.0540 0.0505 0.0540 0.0524 0.0717 0.0693 0.0684 0.0500 0.0524 0.0505 0.0524 0.1150 0.0524

UAR 0.9335 0.9150 0.9054 0.9335 0.9239 0.9335 0.9372 0.9150 0.9372 0.9335 0.9239 0.9239 0.9152 0.9298 0.9335 0.9150 0.9335 0.7101 0.9335

WAR 0.9399 0.9399 0.9399 0.9396 0.9402 0.9399 0.9476 0.9399 0.9476 0.9396 0.9402 0.9399 0.9399 0.9322 0.9396 0.9399 0.9396 0.6823 0.9396

F1-Pos 0.9331 0.9483 0.9414 0.9357 0.9260 0.9199 0.9483 0.9483 0.9483 0.9357 0.9128 0.9263 0.9636 0.9203 0.9357 0.9483 0.9357 0.6864 0.9357

F1-Neg 0.9554 0.9560 0.9574 0.9550 0.9568 0.9554 0.9614 0.9560 0.9614 0.9550 0.9568 0.9568 0.9567 0.9489 0.9550 0.9560 0.9550 0.7515 0.9550

F1-Surp 0.8429 0.8029 0.8029 0.8429 0.8429 0.8714 0.8429 0.8029 0.8429 0.8429 0.8714 0.8429 0.7829 0.8429 0.8429 0.8029 0.8429 0.4314 0.8429

Table 6 Cross-dataset evaluation results under the 3-class protocol. The STAG model is trained on a source dataset and directly evaluated on unseen target datasets. Source

Target

UF1

NaME NaME NaME NaME

CASME II SAMM SMIC-HS 4DME

0.7884 0.9093 0.2131 0.7450

DFME DFME DFME DFME DFME

NaME CASME II SAMM SMIC-HS 4DME

0.8046 0.8519 0.8991 0.3532 0.7703

4DME 4DME 4DME

CASME II SAMM SMIC-HS

0.7570 0.4880 0.5101

CASME II CASME II CASME II

DFME SAMM SMIC-HS

0.7608 0.6032 0.5304

UAR WAR Samples NaME as Source 0.7677 0.8863 255 0.9212 0.9323 133 0.3399 0.4329 164 0.6882 0.8194 216 DFME as Source 0.8229 0.7925 318 0.8315 0.8682 129 0.9026 0.9323 133 0.4075 0.4695 164 0.7767 0.8148 216 4DME as Source 0.8494 0.8246 171 0.5313 0.6241 133 0.5463 0.5183 164 CASME II as Source 0.8260 0.7932 474 0.5713 0.7444 133 0.5407 0.5366 164

Table 7 reports the more challenging cross-dataset evaluation under the 5-class protocol. As expected, all performance metrics decrease compared with the 3-class setting due to the increased class granularity and severe class imbalance. Nevertheless, the proposed framework maintains reasonable generalization performance across several target datasets. DFME consistently achieves the strongest overall transferability, reaching a UF1 of 0.6704 on CASME II and 0.6550 on NaME, while NaME attains its best performance on SAMM with a UF1 of 0.5524. Analysis of the class-wise F1 scores reveals that Surprise and Happiness are recognized reliably across most dataset pairs, frequently achieving scores above 0.70, which suggests that these expressions exhibit more consistent facial motion patterns across datasets. N. Sharma et al.: Preprint submitted to Elsevier

F1-Pos

F1-Neg

F1-Surp

0.7541 0.9020 0.0385 0.6667

0.9303 0.9508 0.6009 0.8746

0.6809 0.8750 0.0000 0.6939

0.8387 0.8276 0.9020 0.5000 0.8000

0.8070 0.8947 0.9565 0.5596 0.8551

0.7681 0.8333 0.8387 0.0000 0.6557

0.6197 0.4872 0.5816

0.8926 0.7625 0.4200

0.7586 0.2143 0.5287

0.6012 0.5200 0.4286

0.8307 0.8351 0.5442

0.8505 0.4545 0.6186

In contrast, Anger, Disgust, and Sadness remain challenging, often producing near-zero F1 scores in several transfer scenarios. This behavior is largely attributable to the limited number of samples available for these categories and the substantial variation in their appearance across datasets. The additional experiments using SAMM as the source dataset further highlight the impact of data scale and class distribution on cross-domain generalization. Despite containing high-quality recordings, SAMM yields considerably lower UF1 and UAR values when evaluated on NaME, CASME II, and 4DME, indicating that models trained on relatively smaller datasets struggle to learn sufficiently diverse representations. Collectively, the 5-class results demonstrate that the proposed STAG framework remains effective unPage 12 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition Table 7 Cross-dataset evaluation results under the 5-class protocol. The model is trained on a source dataset and evaluated on target datasets. Source Target UF1 UAR WAR Samples F1-Surp F1-Hap F1-Ang F1-Dis F1-Sad NaME as Source NaME CASME II 0.5023 0.4768 0.7244 127 0.7600 0.8000 0.0000 0.8403 0.0000 NaME SAMM 0.5524 0.5960 0.5133 113 0.8667 0.9231 0.3944 0.4706 0.5806 NaME 4DME 0.4401 0.4123 0.7824 216 0.6909 0.6667 0.0000 0.0000 0.2727 DFME as Source DFME NaME 0.6550 0.6567 0.7893 318 0.9908 0.8387 0.2703 0.5946 0.1111 DFME CASME II 0.6704 0.6571 0.7132 129 0.8000 0.8125 0.0000 0.7963 0.1071 DFME SAMM 0.5937 0.6325 0.5489 133 0.7407 0.8000 0.4474 0.4286 0.8424 DFME 4DME 0.6482 0.6115 0.6713 216 0.6557 0.5217 0.0000 0.0000 0.0000 SAMM as Source SAMM CASME II 0.2210 0.1991 0.6510 255 0.0000 0.2927 0.0000 0.8122 0.0000 SAMM NaME 0.1856 0.2057 0.4960 506 0.0119 0.1905 0.0580 0.6676 0.0000 SAMM 4DME 0.1995 0.1798 0.498 247 0.1071 0.000 0.1622 0.7284 0.0000 Table 8 Performance comparison of 14-ROI and 18-ROI configurations on the DFME 3-class setting.

ROI

AU

UF1

UAR

WAR

Fold UF1

Positive

Negative

Surprise

14 14 18 18

Yes No Yes No

0.8221 0.6458 0.7070 0.7172

0.8354 0.7245 0.7777 0.7739

0.8847 0.7080 0.7829 0.7948

[0.8669, 0.8146, 0.7860, 0.7721, 0.8711] [0.6895, 0.6183, 0.6318, 0.5944, 0.6949] [0.6726, 0.7556, 0.7288, 0.7481, 0.6298] [0.6687, 0.7574, 0.7348, 0.7674, 0.6576]

0.7334 0.4106 0.5443 0.5462

0.9235 0.7867 0.8424 0.8513

0.8094 0.7401 0.7342 0.7541

der challenging cross-dataset conditions, while also revealing that dataset diversity and balanced class distributions are critical factors for achieving robust fine-grained microexpression recognition. Tables 8-10 demonstrate that the proposed 14-ROI configuration consistently outperforms the 18-ROI configuration across the 3, 5, and 7-class settings, particularly when AU guidance is incorporated. The 14-ROI model achieves substantial improvements in UF1, UAR, and class-wise F1 scores, with the largest gain observed in the 3-class setting, where UF1 increases from 0.6458 to 0.8221 after integrating AU information. Similar performance gains are evident in the 5-class and 7-class experiments, indicating that AU guid-

ance effectively enhances the discriminative representation of subtle facial muscle movements. In contrast, the 18ROI configuration shows only marginal improvements with AU guidance and even slightly underperforms the non-AU model in the 3-class setting, suggesting that additional facial regions introduce redundant information rather than complementary features. These results indicate that a compact and anatomically meaningful 14-ROI design provides more effective feature representation and allows AU guidance to contribute more effectively, resulting in improved recognition performance and stable generalization across different classification complexities.

Table 9 Performance comparison of 14-ROI and 18-ROI configurations on the DFME 5-class setting. ROI

AU

UF1

UAR

WAR

Fold UF1

Anger

Disgust

Happiness

Sadness

Surprise

14 14 18 18

Yes No Yes No

0.5712 0.3874 0.4421 0.4106

0.5912 0.4577 0.5039 0.4941

0.5749 0.4170 0.4364 0.4634

[0.5668, 0.5783, 0.5720, 0.5325, 0.6064] [0.3931, 0.3712, 0.3798, 0.3779, 0.4150] [0.4411, 0.4367, 0.4590, 0.4543, 0.4193] [0.4371, 0.4105, 0.4318, 0.4071, 0.3665]

0.2451 0.2131 0.2102 0.1833

0.5839 0.5246 0.3882 0.5410

0.7347 0.4177 0.5327 0.5085

0.4807 0.1017 0.3395 0.0987

0.8115 0.6799 0.7398 0.7216

Table 10 Performance comparison of 14-ROI and 18-ROI configurations on the DFME 7-class setting. ROI

AU

UF1

UAR

WAR

Fold UF1

Anger

Contempt

Disgust

Fear

Happiness

Sadness

Surprise

14 14 18 18

Yes No Yes No

0.5211 0.2946 0.4066 0.4044

0.5381 0.3567 0.4312 0.4396

0.5425 0.4009 0.4612 0.4671

[0.5738, 0.5157, 0.5132, 0.4972, 0.5058] [0.2695, 0.2876, 0.3050, 0.2914, 0.3194] [0.3999, 0.3912, 0.4171, 0.4423, 0.3827] [0.3908, 0.4042, 0.4050, 0.4348, 0.3874]

0.2443 0.1928 0.1885 0.1839

0.5492 0.1876 0.3960 0.3953

0.5795 0.5594 0.5584 0.5596

0.3008 0.0190 0.1382 0.1413

0.7557 0.4041 0.5324 0.5391

0.4185 0.0108 0.2876 0.2781

0.7998 0.6885 0.7455 0.7338

N. Sharma et al.: Preprint submitted to Elsevier

Page 13 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

Dataset

Original +68 LMKs

GradCAM (ROI Embb)

Perturbation

Optical flow (Mean Vectors)

E-GAT (Last Layer)

Combined

CASME-II

4DME

DFME

NaME

SMIC-HS

SAMM

Figure 2: Explainability analysis of the proposed STAG framework across multiple datasets using Grad-CAM, perturbation maps, optical flow, and E-GAT attention visualizations. Results are shown for positive classes (SMIC-HS and SAMM) and negative classes (remaining datasets), highlighting the model’s focus on discriminative facial action regions and motion patterns.

Figure 2 clearly validate the effectiveness of the proposed STAG framework. The consistent and concentrated activations across Grad-CAM, perturbation maps, optical flow, and E-GAT demonstrate that the model reliably attends to micro-expression relevant facial regions, particularly the eyes and mouth, while suppressing irrelevant background information. The sharper and more localized responses in the combined model further confirm that integrating appearance, motion, and relational (AU-based graph) cues leads to more discriminative and stable representations. This alignment between visual explanations and known facial action patterns provides strong evidence that STAG not only N. Sharma et al.: Preprint submitted to Elsevier

achieves high recognition performance but also learns meaningful and interpretable spatio-temporal features, thereby validating its effectiveness for micro-expression recognition. Figure 3 illustrates the qualitative comparison of feature attention maps produced by different SOTA MER methods. Existing approaches such as MER-CLIP, HFA- Net, ADSS, GCL, and SCFRAM often generate dispersed or noisy activations across irrelevant facial regions, which may weaken the representation of subtle micro-expression cues. In contrast, the proposed STAG framework focuses more precisely on action-unit-related regions, particularly around the eyes and mouth, while suppressing background interference. Page 14 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

STAG

MER-CLIP

HFA-NeT

ADSS

GCL

LRCN-STAN

DBDE-NeT

MARNeT

ME-NAS

SFML-NeT

MDMO

SCFRAM

Figure 3: Qualitative comparison of heat-maps generated by different SOTA MER methods for the Figure 2 illustration of the CASME-II dataset.

Figure 4: Visualization of ROI-based optical flow analysis for a micro-expression sample predicted as mentioned in the Figure 2 for SAMM dataset. The figure illustrates temporal optical-flow magnitudes across facial regions, average motion intensity per ROI, mean motion vectors, and the temporal evolution of the five most active facial regions.

Figure 4 provides an interpretable visualization of the facial motion patterns captured by the proposed framework. The heatmap (top-left) shows the temporal distribution of optical-flow magnitudes across 18 facial ROIs, highlighting localized motion primarily around the mouth, eyebrows, and eyelids. The bar chart (top-right) summarizes the average motion intensity of each ROI, revealing that the mouth and eye regions contribute most strongly to the predicted exN. Sharma et al.: Preprint submitted to Elsevier

pression. The mean flow vector plot (bottom-left) illustrates the dominant direction and magnitude of facial movements, while the temporal curves of the top-5 active ROIs (bottomright) show the evolution of motion intensity across frames. Together, these visualizations demonstrate that the model focuses on meaningful facial muscle activations associated with the positive micro-expression.

Page 15 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition (a) 4DME

(b) CASME-II

(d) SAMM

(e) SMIC-HS

(c) DFME

(f) NaME

Figure 5: t-SNE visualization of learned feature embeddings. The proposed STAG method produces well-separated clusters for different emotion classes, indicating strong discriminative capability. Compared to baseline representations, STAG reduces class overlap and improves intra-class compactness, particularly for subtle expressions. E-GAT layer 1/2

E-GAT layer 2/2

Figure 6: Spatial connection and message-passing visualization across the two sequential layers of the E-GAT. Node positions correspond structurally to facial ROIs, where edge colors and thicknesses signify the relative attention weights dynamically assigned to spatial dependencies during micro-expression processing of the Figure 2 illustration of the SAMM dataset.

Figure 5 shows the 3D t-SNE feature embeddings for the 4DME, CASME-II, DFME, SAMM, SMIC-HS, and NaME datasets. Clear clustering of Positive, Negative, and Surprise classes demonstrates the discriminative power of the proposed framework. NaME and CASME-II exhibit the most compact and well-separated clusters, while SMIC-HS shows greater overlap due to subtle expression variations. N. Sharma et al.: Preprint submitted to Elsevier

Moderate overlap is also observed in DFME and 4DME. Overall, the results confirm effective feature learning and class separability across diverse micro-expression datasets. To evaluate the structural dependencies captured by the spatial graph network, we visualize the learned attention distributions across the sequential layers of the E-GAT in Figure 6. The node topologies are organized to reflect the physiPage 16 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition UF1

UAR

WAR

Class

100

SFK

SKF

SKF

GKF

SFK

GKF

GKF

GKF

SFK

SKF

GKF

GKF

SKF

SKF

SFK

LOSO

GKF

LOSO

LOSO

SFK

LOSO

GKF

SKF

GKF

LOSO

GKF

SKF

50

LOSO

75

4DME

SAMM

SAMM

4DME

CASME-II

CASME-II

SAMM

DFME

CASME-II

DFME

SAMM

CASME-II

SAMM

SMIC-HS

4DME

SMIC-HS

SMIC-HS

4DME

DFME

CASME-II

CASME-II

DFME

DFME

4DME

SAMM

DFME

4DME

0

DFME

25

Figure 7: Performance comparison of the proposed STAG framework across multiple MER benchmark datasets under different evaluation protocols (LOSO, SKF, and GKF). The plot illustrates the variation of UF1, UAR, and WAR scores with respect to dataset complexity and class settings (3-class, 5-class, and 7-class protocols).

cal anatomical distribution of the 18 facial Regions of Interest (ROIs), grouping upper-facial features (e.g., eyebrows) and lower-facial features (e.g., mouth regions) respectively. In the first layer (E-GAT Layer 1/2), the network establishes dense, cross-regional attention maps, indicated by the pronounced red and purple interconnected edges, capturing broad, global synchronized movements across disparate facial muscle regions during a micro-expression event. Conversely, the second layer (E-GAT Layer 2/2) demonstrates a clear structural refinement; the attention weights become sparser and highly localized, focusing predominantly on fine-grained, intra-region localized dynamics (such as local eye or mouth micro-movements) and critical structural boundaries. This layer-wise progression confirms that the EGAT effectively shifts from extracting global coarse-grained facial context to isolating local, highly discriminative localized muscular variations necessary for precise microexpression classification. Figure 7 summarizes the performance of the proposed framework across multiple datasets and evaluation protocols (LOSO, SKF, and GKF) under 3, 5, and 7-class settings. The model consistently achieves strong results, with the best performance obtained on 4DME under the 3-class SKF protocol (UF1 = 94.10%, UAR = 96.13%, WAR = 95.80%). SAMM and CASME II also show competitive performance, confirming the effectiveness of the proposed graph-temporal representation. Although performance decreases in the 5class and 7-class settings due to increased class complexity and imbalance, the framework maintains stable results across datasets. Overall, the findings demonstrate the robustness, generalization capability, and effectiveness of the proposed approach for micro-expression recognition. Figure 8 presents the confusion matrices of STAG under 3-class and 5-class settings. The model achieves strong recognition performance with high diagonal dominance across datasets, particularly on NaME, SAMM, and CASME-II. Although increased class complexity introduces confusion in 4DME and DFME, especially among visually resents the confusion matrices of STAG under 3-class and 5-class settings. The model achieves strong recognition performance with high diagonal domi- nance across datasets, N. Sharma et al.: Preprint submitted to Elsevier

particularly on NaME, SAMM, and CASME-II. Although increased class complexity introduces confusion in 4DME and DFME, especially among visually Overall, the results confirm that the proposed framework effectively integrates graph attention learning, transformerbased temporal modeling, and AU-guided spatial connectivity to achieve robust and generalized micro-expression recognition performance across multiple datasets and evaluation settings.

5. Conclusion This paper proposed STAG, a unified spatial–temporal framework for micro-expression recognition that effectively models subtle facial dynamics. The framework integrates AU-guided dynamic ROI connectivity, graph attentionbased spatial reasoning, and transformer-based temporal modeling to jointly capture fine-grained appearance, motion, and relational cues. In addition, a cross-attention mechanism is introduced to enhance interaction between spatial and temporal representations, leading to more discriminative feature learning. Extensive experiments on six benchmark datasets under LOSO, group K-fold, and stratified K-fold protocols demonstrate that STAG consistently achieves competitive and often superior performance compared to existing stateof-the-art methods. Further cross-dataset evaluations confirm its strong generalization ability in unseen domain settings, while ablation and explainability analyses validate the contribution of each component and the interpretability of the learned representations. Overall, the results indicate that dynamic AU-guided relational modeling combined with spatio-temporal attention is highly effective for micro-expression recognition. The proposed STAG framework provides a robust and interpretable solution for XAIbased, advancing the reliability and generalization of microexpression analysis in real-world scenarios. Future work will investigate explainable vision-language graph learning frameworks that align evolving ROI–AU interactions with textual facial behavior descriptions, improving both interpretability and generalization across diverse microexpression datasets. Page 17 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition (b) CASME-II

(a) 4DME

(d) SAMM

(a) 4DME

(c) DFME

(e) SMIC-HS

(b) CASME-II

(f) NaME

(c) DFME

(d) SAMM

Figure 8: Confusion matrices of STAG on six benchmark micro-expression datasets (4DME, CASME-II, DFME, SAMM, SMIC-HS, and NaME) for 3-class (top) and 5-class (bottom) classification tasks. Strong diagonal patterns indicate high recognition accuracy, while off-diagonal entries highlight confusion among visually similar and low-intensity micro-expressions. Despite increased difficulty in the 5-class setting, STAG maintains robust discriminative performance across datasets.

6. Acknowledgment

References

The authors express their profound appreciation to the Vision Intelligence and Machine Learning (VIML) Group for their important help, guidance, and collaborative attitude. Their proficiency and support have significantly enhanced the success of this endeavor. Gratitude is extended to Dr. Dinesh Singh for his guidance, unwavering support, and for facilitating access to high-performance GPUs and edge computing devices. The authors express their gratitude for the utilization of CHEETAH, a GPU-based computational facility established under research grant No. IITM/SG/DISROS-SPA/111 at the Indian Institute of Technology Mandi, Department of Higher Education, Ministry of Education, Government of India, to fulfill the computational needs of this research endeavor. We acknowledge the utilization of Grammarly (Grammarly Inc.) for rectifying grammatical inaccuracies and improving the readability of this paper.

[1] Bao, Y., Wu, C., Zhang, P., Shan, C., Qi, Y., Ben, X., 2024. Boosting micro-expression recognition via self-expression reconstruction and memory contrastive learning. IEEE Transactions on Affective Computing 15, 2083–2096. [2] Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H., 2018. SAMM: A spontaneous micro-facial movement dataset. IEEE Transactions on Affective Computing 9, 116–129. [3] Ekman, P., Friesen, W.V., 1978. Facial action coding system. Consulting Psychologists Press. [4] Fang, L., Yang, F., Lin, Y., Zhang, J., Whang, M., 2025. Graph-driven micro-expression rendering with emotionally diverse expressions for lifelike digital humans. Biomimetics 10, 587. [5] Gan, Y.S., Liong, S.T., Yau, W.C., Huang, Y.C., Tan, L.K., 2019. Offapexnet on micro-expression recognition system. Signal Processing: Image Communication 74, 129–139. [6] Irawan, B., Utama, N.P., Munir, R., Purwarianti, A., 2023. Spontaneous micro-expression recognition using 3dcnn on long videos for emotion analysis, in: 2023 10th international conference on advanced informatics: concept, theory and application (ICAICTA), IEEE. pp. 1–6.

N. Sharma et al.: Preprint submitted to Elsevier

Page 18 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition Table 11 List of symbols and notations used in the proposed STAG framework. Symbol  𝑇 𝑡 𝐥(𝑖) 𝑡 𝑡,𝑟 𝑔 𝐅𝑡,𝑟 (𝑝) 𝑚𝑡,𝑟 (𝑝) 𝑡,𝑟 𝐗 𝐇(0) 𝐷ℎ 𝑡 ̂ (𝑡) 𝐀 𝐾 𝑒𝑡𝑖𝑗𝑘 𝜆 𝜔𝑡 𝐟𝑔𝑟𝑎𝑝ℎ 𝐇𝑓 𝑙𝑎𝑡 𝐿𝑡 𝐓 𝐆∗ 𝐷𝑓 𝑁𝐴𝑈 𝐟𝑎𝑢 𝐟𝑓 𝑖𝑛𝑎𝑙  𝛼𝑐 𝜀𝑐 𝜃

Description Input micro-expression video Number of temporal frames Facial landmarks at frame 𝑡 Landmark coordinate ROI region at frame 𝑡 Face scaling factor Optical flow vector Motion magnitude Dominant motion pixels Optical flow tensor Initial ROI embedding tensor MLP hidden dimension Dynamic graph Dynamic adjacency matrix Number of attention heads Attention score Temporal smoothing coefficient Temporal attention weight Graph representation Flattened ROI sequence Transformer layers Transformer token sequence Cross-attended graph tokens Fusion projection dimension Number of Action Units AU embedding Final representation Set of classes Class balancing weight Label smoothing factor Trainable parameters

Symbol 𝐼𝑡 𝐻, 𝑊 , 𝐶 𝑁𝐿 𝑅 𝑑𝑒 𝐜𝑡 𝑢𝑡,𝑟 (𝑝), 𝑣𝑡,𝑟 (𝑝) 𝜏𝑡,𝑟 𝐱𝑡,𝑟 𝐡𝑡,𝑟 𝐷𝑒 𝐵 𝐀(0) 𝐿𝑔 𝑑ℎ 𝑏𝑖𝑗 𝐄(𝑡) 𝐆 𝐷𝑔 𝐏 𝐷𝑡 𝐟𝑡𝑟𝑎𝑛𝑠 𝐓∗ 𝐟𝑓 𝑢𝑠𝑖𝑜𝑛 𝐯𝑎𝑢 𝐷𝑎𝑢 𝐲̂ 𝐶 𝛾𝑐  𝑓𝜃 (⋅)

Description RGB frame at time step 𝑡 Image height, width, and channels Number of facial landmarks (68) Number of facial ROIs Inter-ocular distance Facial center coordinate Horizontal and vertical flow Motion threshold Aggregated ROI motion vector ROI embedding feature ROI embedding dimension Mini-batch size Initial adjacency matrix Number of E-GAT layers Attention-head dimension Learnable edge bias Aggregated attention matrix Graph token sequence Graph feature dimension Positional encoding Transformer dimension Transformer representation Cross-attended transformer tokens Fused representation Binary AU vector AU embedding dimension Predicted logits Number of classes Focal-loss parameter Training loss Proposed network

Table 12 Tensor dimensions used in the proposed STAG framework. Variable 𝐗 𝐡𝑡,𝑟 ̂ (𝑡) 𝐀 𝐇(𝓁) 𝑡 𝐠𝑟 𝐟𝑔𝑟𝑎𝑝ℎ ̂ (0) 𝐇 𝐆𝑓 𝐓∗ 𝐯𝑎𝑢 𝐲̂

Dimension ℝ𝑇 ×𝑅×2 ℝ𝐷𝑒 ℝ𝑅×𝑅 ℝ𝑅×𝐷𝑒 ℝ𝐷𝑒 ℝ𝐷𝑔 ℝ(𝑇 +1)×𝐷𝑡 ℝ𝑅×𝐷𝑓 ℝ𝑇 ×𝐷𝑓 {0, 1}𝑁𝐴𝑈 ℝ𝐶

Variable 𝐱𝑡,𝑟 𝐇(0) 𝐐𝑡𝑖𝑘 , 𝐊𝑡𝑗𝑘 𝐕𝑡 𝐳𝑡 𝐇𝑓 𝑙𝑎𝑡 𝐓 𝐓𝑓 𝐟𝑔 , 𝐟𝑡 𝐟𝑎𝑢 𝐲̃

[7] Islam, M.S., Sang, Y., Mohammed, A.A., Yuan, L., Lv, J., 2023. Highly effective end-to-end single-to-multichannel feature fusion and ensemble classification to decode emotional secretes from smallscale spontaneous facial micro-expressions. Journal of King Saud University-Computer and Information Sciences 35, 101653. [8] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G., 2018. Averaging weights leads to wider optima and better generalization, in: UAI. [9] King, D.E., 2009. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research 10, 1755–1758.

N. Sharma et al.: Preprint submitted to Elsevier

Dimension ℝ2 ℝ𝑇 ×𝑅×𝐷𝑒 ℝ𝑑ℎ ℝ𝑅×𝐷𝑒 ℝ𝑅𝐷𝑒 ℝ𝑇 ×(𝑅𝐷𝑒 ) ℝ𝑇 ×𝐷𝑡 ℝ𝑇 ×𝐷𝑓 ℝ𝐷 𝑓 ℝ𝐷𝑎𝑢 ℝ𝐶

Variable 𝐅𝑡,𝑟 (𝑝) 𝐀(0) 𝐄(𝑡) 𝐆 𝐳 𝐏 𝐟𝑡𝑟𝑎𝑛𝑠 𝐆∗ 𝐟𝑓 𝑢𝑠𝑖𝑜𝑛 𝐟𝑓 𝑖𝑛𝑎𝑙 𝑝𝑖𝑐

Dimension ℝ2 ℝ𝑅×𝑅 ℝ𝑅×𝑅 ℝ𝑅×𝐷𝑒 ℝ𝑅𝐷𝑒 ℝ(𝑇 +1)×𝐷𝑡 ℝ𝐷𝑡 ℝ𝑅×𝐷𝑓 ℝ𝐷𝑔 +𝐷𝑡 ℝ𝐷𝑔 +𝐷𝑡 +𝐷𝑎𝑢 ℝ

[10] Kipf, T.N., Welling, M., 2017. Semi-supervised classification with graph convolutional networks, in: International Conference on Learning Representations (ICLR). [11] Kong, W., You, Z., Lv, X., 2025. 3d micro-expression recognition based on adaptive dynamic vision. Sensors 25, 3175. [12] Lao, L., Li, Y., Liu, M.L., Xu, C., Cui, Z., 2022. Temporal discriminative micro-expression recognition via graph contrastive learning, in: 26th International Conference on Pattern Recognition (ICPR), IEEE. pp. 1033–1040.

Page 19 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition [13] Lei, L., Chen, T., Li, S., Li, J., 2021. Micro-expression recognition based on facial graph representation learning and facial action unit fusion, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1571–1580. [14] Lei, L., et al., 2020. A novel graph neural network method for microexpression recognition. Information Sciences . [15] Li, C., Ba, R., Wang, X., Yu, M., Li, X., Huang, D., 2025a. Structure representation with adaptive and compact facial graph for microexpression recognition. IEEE Transactions on Biometrics, Behavior, and Identity Science 7, 256–269. [16] Li, J., Qian, Y., Zhao, L., Wang, S.J., 2025b. Fed-psyau: Privacypreserving micro-expression recognition via psychological au coordination and dynamic facial motion modeling, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14453– 14463. [17] Li, S., Li, M., Sun, J., Lu, S., 2026. Micro-expression recognition through feature enhancement and region weighted fusion based on supervised contrastive learning. Signal Processing 238, 110171. [18] Li, X., Cheng, S., Li, Y., Behzad, M., Shen, J., Zafeiriou, S., Pantic, M., Zhao, G., 2022a. 4dme: A spontaneous 4d micro-expression dataset with multimodalities. IEEE Transactions on Affective Computing 14, 3031–3047. [19] Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M., 2013. A spontaneous micro-expression database: Inducement, collection and baseline, in: Face and Gesture 2013, IEEE. pp. 1–6. [20] Li, X., et al., 2022b. Micro-expression recognition: A survey. IEEE Transactions on Affective Computing . [21] Liang, H., Liu, J., Wang, Y., Shi, H., 2025. Cit: Context interaction transformer for micro-expression recognition in natural scene, in: 2025 4th International Conference on Artificial Intelligence, HumanComputer Interaction and Robotics (AIHCIR), IEEE. pp. 1–6. [22] Liong, S.T., Gan, Y.S., Zheng, D., Li, S.M., Xu, H.X., Zhang, H.Z., Lyu, R.K., Liu, K.H., 2020. Evaluation of the spatio-temporal features and gan for micro-expression recognition system. Journal of Signal Processing Systems 92, 705–725. [23] Liu, F., Nan, B., Qian, X., Fu, X., 2026. Temporal-spatial crossfusion for dynamic micro expression recognition. Pattern Recognition , 113715. [24] Liu, J., Shi, H., Liang, H., Xu, X., Zong, Y., Wang, Y., Zheng, W., 2025a. Name: A natural micro-expression dataset for microexpression recognition in the wild, in: Proceedings of the 33rd ACM International Conference on Multimedia, pp. 5617–5626. [25] Liu, S., Mao, X., Zhao, S., Li, P., Xu, T., Chen, E., 2025b. Mer-clip: Au-guided vision-language alignment for micro-expression recognition. IEEE Transactions on Affective Computing . [26] Loshchilov, I., Hutter, F., 2019. Decoupled weight decay regularization, in: ICLR. [27] Ma, C., Zhao, S., Pei, Y., Xie, L., Yin, E., Yan, Y., 2025. A multiprior fusion network for video-based micro-expression recognition, in: ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 1–5. [28] Malik, P., Singh, J., Ali, F., Kwak, D., 2026. A comprehensive review of microexpression recognition, classification, and datasets. IEEE Transactions on Computational Social Systems . [29] Matharaarachchi, N., Pasha, M.F., Coleman, S., Kerr, D., 2023. Time efficient micro-expression recognition using weighted spatiotemporal landmark graphs, in: International Conference on Machine Learning and Applications (ICMLA), IEEE. pp. 347–354. [30] Shangguan, Z., Dong, Y., Guo, S., Leung, V.C., Deen, M.J., Hu, X., 2025. Facial expression analysis and its potentials in iot systems: A contemporary survey. ACM Computing Surveys 58, 1–39. [31] Shao, Z., Cheng, Y., Li, F., Zhou, Y., Lu, X., Xie, Y., Ma, L., 2025a. Mol: Joint estimation of micro-expression, optical flow, and landmark via transformer-graph-style convolution. arXiv preprint . [32] Shao, Z., Cheng, Y., Zhang, F., Shi, X., Li, C., Ma, L., Yeung, D.Y., 2025b. Micro-expression recognition via fine-grained dynamic perception. ACM Transactions on Multimedia Computing, Communications and Applications 21, 1–23.

N. Sharma et al.: Preprint submitted to Elsevier

[33] Shao, Z., Cheng, Y., Zhou, Y., Xiang, X., Li, J., Liu, B., Yeung, D.Y., 2025c. High-level lora and hierarchical fusion for enhanced microexpression recognition. The Visual Computer 41, 4533–4546. [34] Sharma, N., Banothu, S., Shah, M., Singh, D., 2026. Improved singlestage facial landmarks for real-time driver drowsiness detection, in: International Conference on Distributed Computing and Intelligent Technology, Springer. pp. 189–204. [35] Sharma, N., Singh, D., 2025. Exp-graph: How connections learn facial attributes in graph-based expression recognition. arXiv preprint arXiv:2507.14608 . [36] Sharma, N., Singh, K., Singh, D., 2025. Para-x: Graph-based facial paralysis detection using structural deformations of facial expression, in: 2025 International Joint Conference on Neural Networks (IJCNN), IEEE. pp. 1–8. [37] Shou, Z., Huang, L., Yuan, X., Yu, Y., Xu, X., Wu, Z., 2026. Dsbicnet: dynamic-static bidirectional interaction collaborative network for micro-expression recognition. Multimedia Systems 32, 122. [38] Shukla, S., Rai, P.K., Verlekar, T.T., 2022. Micro-expression recognition using a shallow convlstm-based network, in: Proceedings of the Asian Conference on Computer Vision, pp. 17–28. [39] Tang, H., Chai, L., 2024a. Facial micro-expression recognition using stochastic graph convolutional network and dual transferred learning. Neural Networks 178, 106421. [40] Tang, H., Chai, L., 2024b. Facial micro-expression recognition using stochastic graph convolutional network and dual transferred learning. Neural Networks 178, 106421. URL: https://doi.org/10.1016/j. neunet.2024.106421, doi:10.1016/J.NEUNET.2024.106421. [41] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017. Attention is all you need. Advances in Neural Information Processing Systems 30. [42] Verma, M., Vipparthi, S.K., Murala, S., Abdel-Mottaleb, M., 2026. Me-nas: A micro expression feature adaptive neural architecture search. ACM Transactions on Intelligent Systems and Technology . [43] Wang, F., Li, J., Qi, C., Wang, L., Wang, P., 2025a. A multi-modal multi-scale network based on transformer for micro-expression recognition. Journal of Visual Communication and Image Representation , 104537. [44] Wang, H., Wang, L., Xu, L., Li, Y., 2026a. Dbde-net: Dual-branch detail-enhanced network for micro-expression recognition. Neurocomputing , 133167. [45] Wang, K., Xu, H., Wu, M., Yao, X., Zhang, P., 2025b. Gta: A graphtransformer architecture for micro-expression recognition, in: IEEE International Conference on Unmanned Systems (ICUS), pp. 1611– 1616. [46] Wang, M., Tang, X., Wang, B., Ren, J., 2026b. Micro-expression recognition via lora-enhanced dinov2 and interactive spatio-temporal modeling. Sensors 26, 625. [47] Wang, Y., Huang, Y., Liu, C., Gu, X., Yang, D., Wang, S., Zhang, B., 2021. Micro expression recognition via dual-stream spatiotemporal attention network. Journal of Healthcare Engineering 2021, 7799100. [48] Wang, Z., Yang, M., Jiao, Q., Xu, L., Han, B., Li, Y., Tan, X., 2024. Two-level spatio-temporal feature fused two-stream network for micro-expression recognition. Sensors 24, 1574. [49] Wei, J., Peng, W., Lu, G., Li, Y., Yan, J., Zhao, G., 2024a. Geometric graph representation with learnable graph structure and adaptive au constraint for micro-expression recognition. IEEE Transactions on Affective Computing 15, 1343–1357. [50] Wei, J., Peng, W., Lu, G., Li, Y., Yan, J., Zhao, G., 2024b. Geometric graph representation with learnable graph structure and adaptive AU constraint for micro-expression recognition. IEEE Trans. Affect. Comput. 15, 1343–1357. URL: https://doi.org/10.1109/TAFFC.2023. 3340016, doi:10.1109/TAFFC.2023.3340016. [51] Wei, M., Zheng, W., Zong, Y., Jiang, X., Lu, C., Liu, J., 2022. A novel micro-expression recognition approach using attention-based magnification-adaptive networks, in: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 2420–2424.

Page 20 of 21

Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition [52] Xia, Z., Peng, W., Khor, H.Q., Feng, X., Zhao, G., 2020. Revealing the invisible with model and data shrinking for composite-database micro-expression recognition. IEEE Transactions on Image Processing 29, 8590–8605. [53] Xie, X., et al., 2020. Assisted macroscopic expression recognition based on graphs. Pattern Recognition Letters . [54] Xu, S., Chen, C.P., Zhang, T., 2026. Motion-aware representation network for micro-expression recognition. IEEE Transactions on Multimedia . [55] Yan, W.J., Li, X., Wang, S.J., Zhao, G., Liu, Y.J., Chen, Y.H., Hao, X., 2014. CASME II: An database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Transactions on Affective Computing 5, 3–14. [56] Yang, B., Wu, J., Ikeda, K., Hattori, G., Sugano, M., Iwasawa, Y., Matsuo, Y., 2023. Deep learning pipeline for spotting macro-and micro-expressions in long video sequences based on action units and optical flow. Pattern Recognition Letters 165, 63–74. [57] Yu, K., Zhang, Z., Hu, C., Luo, J., 2026. Sofp: Capturing subtle facial dynamics with symmetric optical flow perception for microexpression recognition. Pattern Recognition , 113199. [58] Yu, W.W., Jiang, J., Li, Y.J., 2021. Lssnet: A two-stream convolutional neural network for spotting macro-and micro-expression in long videos, in: Proceedings of the 29th ACM International Conference on Multimedia, pp. 4745–4749. [59] Zhang, B., Wang, X., Wang, C., He, G., 2025a. Dynamic stereotype theory induced micro-expression recognition with oriented deformation, in: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10701–10711. [60] Zhang, F., Huang, Z., Zhang, X., Jin, Q., 2024. Adaptive temporal motion guided graph convolution network for micro-expression recognition, in: IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. [61] Zhang, L., Zhang, Y., Sun, X., Tang, W., Wang, X., Li, Z., 2025b. Micro-expression recognition based on direct learning of graph structure. Neurocomputing 619, 129135. [62] Zhang, M., Yang, W., Wang, L., Wu, Z., Chen, D., 2025c. Hfanet: hierarchical feature aggregation network for micro-expression recognition. Complex & Intelligent Systems 11, 169. [63] Zhang, M., Zhu, T., Gong, H., Yang, W., Wei, F., Yao, L., 2026a. Sfml-net: Spatial feature-guided motion learning network for microexpression recognition. Journal of King Saud University Computer and Information Sciences . [64] Zhang, Y., Fei, Z., Zhou, W., Fei, M., 2025d. Fast micro-expression recognition method based on bi-directional optical flow. Applied Intelligence 55, 865. [65] Zhang, Y., Guo, X., Wu, S., Xu, W., Zheng, Y., 2026b. Lrcnstan: micro-expression recognition via dual-stream lrcn-enhanced spatiotemporal attention network. International Journal of Machine Learning and Cybernetics 17, 54. [66] Zhang, Y., Wang, H., Xu, Y., Mao, X., Xu, T., Zhao, S., Chen, E., 2023. Adaptive graph attention network with temporal fusion for micro-expressions recognition, in: IEEE International Conference on Multimedia and Expo (ICME), pp. 1391–1396. [67] Zhang, Z., Tang, W., Chen, H., 2025e. Rethinking key-framebased micro-expression recognition: A robust and accurate framework against key-frame errors, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12274–12283. [68] Zhao, S., Tang, H., Mao, X., Liu, S., Zhang, Y., Wang, H., Xu, T., Chen, E., 2023. Dfme: A new benchmark for dynamic facial microexpression recognition. IEEE Transactions on Affective Computing 15, 1371–1386. [69] Zhao, X., Ma, H., Wang, R., 2021. Sta-gcn: Spatio-temporal au graph convolution network for facial micro-expression recognition, in: Chinese Conference on Pattern Recognition and Computer Vision, Springer. pp. 80–91. [70] Zhou, L., Mao, Q., Huang, X., Zhang, F., Zhang, Z., 2022. Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition. Pattern Recognition 122, 108275.

N. Sharma et al.: Preprint submitted to Elsevier

Miss. Nandani Sharma is a doctoral candidate at IIT Mandi in the SCEE department, working under the supervision of Dr. Dinesh Singh. She earned her B.Tech. (Hons.) and M.Tech. in Computer Science and Engineering from Dr. A.P.J. Abdul Kalam Technical University (APJAKTU), where she was awarded the Gold Medal for securing the top position in her branch across Uttar Pradesh, India, under the supervision of Dr. Peeyush Kumar Pathak. Her work contributes to advancements in humancomputer interaction, affective computing, and medical diagnostics. As a visual intelligence and machine learning (VIML) research group member at IIT Mandi, she contributes to computer vision, face analysis, deep learning, human-computer interaction, surveillance video analytics.

Mr. Varun Sharma is a B.Tech. student in the Department of Electronics and Communication Engineering at the Indian Institute of Information Technology (IIIT) Bhagalpur. His research interests include artificial intelligence, machine learning, computer vision, representation learning, large language models, and generative AI systems. His work focuses on multimodal AI, image and video analysis, intelligent virtual assistants, and human-computer interaction. He is actively exploring modern AI technologies and their applications in interactive and creative computing systems, with particular interest in AI-driven research, visual intelligence, and advanced deep learning methodologies for real-world intelligent systems.

Dr. Dinesh Singh is an Assistant Professor in the School of Computing and Electrical Engineering, Indian Institute of Technology Mandi. Prior to joining IIT Mandi, He worked in the Highdimensional Statistical Modeling Team as a Postdoctoral Researcher at the RIKEN Center for Advanced Intelligence Project (AIP), Kyoto University Office, Japan. He completed his Ph.D. degree in Computer Science and Engineering from IIT Hyderabad on Scalable and Distributed Methods for Large-scale Visual Computing. He received his M.Tech degree in Computer Engineering from NIT Surat on machine-learning approaches for network anomaly and intrusion detection in the domain of cybersecurity and cloud security. His research interests include machine learning, big data analytics, visual computing, and cloud computing.

Page 21 of 21