arXiv CSopen access

HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessment

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessment Changdao Chen [email protected] School of Computer Science and Technology, Xi’an Jiaotong University Xi’an, Shaanxi, China

arXiv:2604.08435v1 [cs.CV] 9 Apr 2026

Abstract It remains challenging to assess driver fatigue from untrimmed videos under constrained computational budgets, due to the difficulty of modeling long-range temporal dependencies in subtle facial expressions. Some existing approaches rely on computationally heavy architectures, whereas others employ traditional lightweight pairwise graph networks, despite their limited capacity to model high-order synergies and global temporal context. Therefore, we propose HST-HGN, a novel Heterogeneous Spatial-Temporal Hypergraph Network driven by Bidirectional State Space Models. Spatially, we introduce a hierarchical hypergraph network to fuse pose-disentangled geometric topologies with multi-modal texture patches dynamically. This formulation encapsulates high-order synergistic facial deformations, effectively overcoming the limitations of conventional methods. In temporal terms, a Bi-Mamba module with linear complexity is applied to perform bidirectional sequence modeling. This explicit temporal-evolution filtering enables the network to distinguish highly ambiguous transient actions, such as yawning versus speaking, while encompassing their complete physiological lifecycles. Extensive evaluations across diverse fatigue benchmarks demonstrate that HST-HGN achieves state-of-the-art performance. In particular, our method strikes a balance between discriminative power and computational efficiency, making it wellsuited for real-time in-cabin edge deployment.

Keywords Driver Fatigue Detection, Hypergraph Neural Networks, State Space Models, Multimodal Fusion, Spatial-Temporal Modeling

Introduction

Fine-grained facial dynamic analysis is a cornerstone of humancomputer interaction (HCI) . With its assistance, information from raw videos containing faces can be effectively extracted, thereby providing new solutions to challenging problems like driver fatigue assessment. Fatigued driving poses a significant threat to the safety of both drivers and pedestrians.[13]. However, distinct from explicit action recognition, driver fatigue—characterized by behaviors like recurrent yawning and microsleep—is difficult to detect and typically deepen gradually over extended periods.[1, 33]. It takes a novel paradigm, which is capable of capturing subtle local facial deformations and encoding long-range temporal dependencies to accurately estimating global fatigue states and pinpointing critical temporal segments from untrimmed video sequences. To this end, extensive work has explored diverse spatial-temporal modeling strategies. Spatially, appearance-based CNNs remain

Figure 1: Qualitative comparison between traditional spatial/temporal methods and our HST-HGN.

the dominant paradigm for extracting discriminative texture cues such as eye closure and mouth opening from RGB frames[36, 51]. Geometry-driven GCNs extend this line by representing facial landmarks as graph nodes linked via anatomical or kinematic priors, improving robustness to pose variation and occlusion while reducing input redundancy[9, 46]. Temporally, sliding-window RNN/LSTM pipelines are widely adopted to encode the evolution of blinking and yawning patterns over short segments[47, 50]. More recently, studies have shifted toward Transformer to directly encode feature or explicit fatigue parameter sequences, exploiting global selfattention to uncover long-range dependencies across extended temporal horizons[18]. Nevertheless, these established paradigms encounter inherent bottlenecks in global fatigue assessment. Spatially, CNN-based backbones are computationally heavy and remain sensitive to large head poses and motion blur[51]. While GCN-driven models alleviate efficiency concerns, they often struggle to perceive critical texture semantics—such as eye-closure status—leading to

sub-optimal multimodal fusion[1, 46]. Temporally, sliding-windowbased RNNs/LSTMs suffer from local myopia and long-term forgetting, failing to capture the global context of extended sequences[23, 25, 44]. Furthermore, although Transformers excel in global modeling, their quadratic computational complexity, 𝑂 (𝑇 2 ), impedes efficient deployment on resource-constrained edge devices for long video processing[6, 32]. To overcome these bottlenecks, we propose HST-HGN, a novel heterogeneous spatial-temporal hypergraph architecture driven by bidirectional State Space Models for robust and efficient global fatigue assessment. Rather than dense local windows, a global sparse sampling strategy is adopted to extract heterogeneous representations from extended sequences. Spatially, we introduce a 3D canonical alignment mechanism integrated with a heterogeneous super-node topology. Through star-topology hyperedges, our model broadcast local texture semantics into the global geometric framework without causing dimensionality explosion. Temporally, Bi-Mamba layers are applied instead of self-attention mechanism. Leveraging selective state spaces, HST-HGN achieves global temporal context modeling with linear complexity, effectively capturing the frequency differences between yawning and speaking. The primary contributions of this work are summarized as follows: • We propose HST-HGN, an efficient global fatigue assessment framework that pioneers the integration of heterogeneous hypergraphs with Bi-SSMs, enabling robust longrange video modeling with linear complexity. • We design a novel hierarchical super-node topology featuring 3D canonical alignment, which effectively decouples rigid pose variations and facilitates the high-order fusion of local texture semantics and global geometric structures via hyperedges. • We demonstrate the inherent interpretability of our framework, enabling weakly supervised temporal localization of fatigue events and significantly mitigating action ambiguity between behaviors like speaking and yawning. • Extensive evaluations on benchmark datasets validate that HST-HGN demonstrates state-of-the-art (SOTA) performance in both accuracy and computational efficiency, highlighting its substantial potential for real-world edge deployment.

2 Related Work 2.1 Spatial Representation for Facial Behavior Robust spatial representation is the foundation for assessing driver fatigue. Conventional deep learning paradigms primarily leverage 2D convolutional neural networks, such as ResNet and VGG, to extract dense texture features from facial regions of interest (ROIs) [17, 28, 36, 52]. To capture short-term motion, architectures such as C3D and I3D extended these convolutions into the temporal domain[8, 39]. Nevertheless, such grid-based models impose a prohibitive computational burden for continuous edge deployment and exhibit great sensitivity to unconstrained environments, suffering from severe feature degradation under head-pose variations and illumination shifts[19, 43].

To overcome the computational bottlenecks of dense pixel processing, Graph Convolutional Networks (GCNs) emerged to model the non-Euclidean topologies of structural joints. Building upon foundational skeleton-based architectures like ST-GCN [5] and 2sAGCN, dynamic GCNs have been further adapted to model facial topologies over structural landmarks [9] and relational patches [21]. However, standard GCNs remain restricted to pairwise edges, making it hard to encapsulate high-order synergistic deformations of facial muscle groups during physiological expressions such as yawning [9]. To break these pairwise restrictions, spatio-temporal hypergraphs have recently emerged to capture high-order structural correlations in fine-grained physiological tasks, such as motor symptom assessment [4] and spasms detection [42]. While recent advances further explore multi-modal hypergraph fusion [10] and Transformer-integrated temporal reasoning [41], combining highorder hypergraphs with self-attention inevitably incurs quadratic computational overhead. This prohibits their deployment on edge devices for long-video processing, thereby motivating our exploration of a more efficient temporal engine.

2.2

Global Temporal Modeling & SSMs

To distinguish the frequency difference between talking and yawning from untrimmed videos, it is essential to model longrange temporal dependencies. Historically, Recurrent Neural Networks—particularly LSTMs—have been extensively employed to aggregate sequential frame features through gating mechanisms[3, 7, 19]. However, when processing long sequences, LSTMs suffer from gradient vanishing during Backpropagation Through Time (BPTT), causing severe long-term forgetting. As a remedial paradigm, Vision Transformers (ViTs) and their video-centric variants, such as TimeSformer, leverage global self-attention to establish unfettered long-range dependencies[20, 23, 40]. Regrettably, the computational complexity of the attention matrix grows quadratically with sequence length 𝑇 , which is unacceptable on edge devices. More recently, State Space Models (SSMs) have surfaced as a transformative alternative for sequence modeling. The S4 layer parameterizes continuous-time dynamics using HiPPO projections, enabling efficient linear-time handling of long sequences and serving as a backbone in long-form video models such as ViS4mer and S5-based selective token frameworks[20, 34, 40]. Based on S4, Mamba introduces a hardware-aware selective scan algorithm, allowing the model to dynamically filter noise while retaining critical contextual memory, thereby achieving a global receptive field with linear complexity, 𝑂 (𝑇 )[27, 34, 37]. VideoMamba further adapts this architecture to video by scanning spatial-temporal tokens with forward–backward selective SSMs[23].

Methodology

Given a sparsely sampled long video sequence, the proposed framework is designed to accurately assess the global fatigue state by jointly capturing high-order spatial facial synergies and long-range temporal dynamics. An overview of our method is illustrated in Figure 2. We first detail the Global Sparse Sampling strategy, which extracts a sequence of 𝑇 frames from the raw video (Sec. 3.1). Following this, we introduce the Heterogeneous Feature Construction process, which extracts pose-disentangled geometric nodes and

HST-HGN: Spatio-Temporal Hypergraph for Fatigue Detection

Stage 2: Spatial Dynamic Hypergraph Modeling

Stage 1: Hierarchical Feature Construction

Micro-CNN Encoder

Texture Super Nodes

Geometric KNN

TextureGeometry Broadcasting

Adaptive Attention Pooling

HGNN Conv Layer

Frame-level Weighted Sum Features

Softmax Attention

Geo-Nodes

Stage 3: Temporal Evolution Modeling Bi-Mamba Block Forward SSM Frame-level Features

Backward SSM

Temporal Aggregation Fusion & Normaliz ation

Texture Stream

Linear Layer

Concatenation & Stacking

T Frames

Geometry Nodes

...

MediaPipe Face Mesh

3D Canonical Alignment

Raw Video

Tex-Geo Hypergraph

Geo-Geo Hypergraph

Geometry Stream

Max Pooling Video Vector

Classification Head

Normal/ Talking / Yawning

Figure 2: An overview of HST-HGN framework. The architecture consists of three core stages. Stage 1 extracts pose-invariant geometry nodes and texture super-nodes from sparsely sampled frames. Stage 2 constructs a spatial dynamic hypergraph to fuse heterogeneous features, followed by adaptive attention pooling to obtain frame-level representations. Stage 3 employs a Bidirectional Mamba (Bi-Mamba) block to efficiently capture long-range temporal dependencies, concluding with temporal aggregation and a classification head for the final tri-class prediction. appearance-based texture super-nodes to form robust dual-stream representations (Sec. 3.2). Then, we present the core component of our spatial representation: a Spatial Dynamic Hypergraph Modeling module. This module fuses multimodal cues through a hierarchical Tex-Geo hypergraph and utilizes an adaptive attention pooling mechanism to obtain frame-level features (Sec. 3.3). Finally, we elaborate on the Temporal Evolution Modeling stage, where a Bidirectional Mamba (Bi-Mamba) block is employed to efficiently encapsulate both forward and backward temporal contexts with linear complexity for the final tri-class prediction (Sec. 3.4).

3.1

Global Sparse Sampling

Traditional methods often employ dense sliding windows, which not only incur massive computational overhead but also suffer from restricted local receptive fields, failing to capture the global context of a prolonged behavior. Therefore, we introduce a global sparse sampling strategy. Given an untrimmed raw video 𝑉 consisting of 𝑁 frames, we uniformly sample a fixed-length sequence to represent the entire video dynamically. Mathematically, the raw video consists of several frames 𝑆𝑟𝑎𝑤 = {𝑠 1, 𝑠 2, ..., 𝑠 𝑁 }, the sampled frame sequence 𝑆 = {𝑠 1, 𝑠 2, ..., 𝑠𝑇 } is obtained by selecting frames at specific indices 𝑖𝑡 ∈ {1, 2, ..., 𝑁 }, formulated as:

𝑖𝑡 = ⌊

𝑁 · 𝑡⌋, 𝑇

𝑡 ∈ {1, 2, ...,𝑇 }

(1)

where 𝑇 is the predefined sequence length. In our implementation, we set 𝑇 = 128. This sparse paradigm reduces the spatial redundancy of adjacent frames while granting the subsequent Bi-Mamba module a global temporal receptive field covering the entire video sequence.

3.2

Heterogeneous Feature Construction

Upon obtaining the sparsely sampled frame sequence 𝑆, we extract dual-stream representations—geometry and texture—to capture complementary facial cues. Geometry Stream and 3D Canonical Alignment. Facial landmarks provide highly compressed geometric topology. Although tools like MediaPipe[26] excel at predicting dense 3D facial meshes in real-time, directly constructing a graph with hundreds of nodes introduces severe computational redundancy. Therefore, we strategically sample a semantic subset of 68 keypoints from the MediaPipe mesh, rigorously aligned with the standard Dlib 68-point protocol[22]. This mapping preserves crucial structural semantics while maintaining a lightweight node dimension. Let 𝑃𝑡 ∈ R68×3 denote the raw 3D coordinates of the 68 landmarks at frame 𝑠𝑡 . In driving scenarios, 𝑃𝑡 is highly entangled with rigid head movements, which can easily overwhelm the subtle nonrigid deformations caused by fatigue. Therefore, we introduce a 3D canonical alignment mechanism. Given a pre-defined frontal canonical face template 𝑃𝑟𝑒 𝑓 ∈ R68×3 , we estimate the optimal scale 𝑐𝑡 , rotation matrix 𝑅𝑡 ∈ R3×3 , and translation vector 𝑡𝑡 ∈ R3 by

minimizing the Procrustes distance: arg min ||𝑐𝑡 𝑃𝑡 𝑅𝑡 + 𝑡𝑡 − 𝑃𝑟𝑒 𝑓 || 2𝐹

s.t. 𝑅𝑇𝑡 𝑅𝑡 = 𝐼

(2)

𝑐𝑡 ,𝑅𝑡 ,𝑡𝑡

By resolving this optimization, we transform the raw landmarks into a pose-invariant canonical space. The aligned coordinates, 𝑔𝑒𝑜 denoted as 𝑋𝑡 ∈ R68×3 , exclusively reflect pure facial muscular 𝑔𝑒𝑜 deformations. These canonical coordinates 𝑋𝑡 serve as the initial features for the geometric nodes in our subsequent spatial hypergraph. Texture Stream and Micro-CNN Encoder. While geometric landmarks can capture structural deformations, they intrinsically lack the visual semantics to distinguish specific physiological states, such as the subtle textural difference between an open and a closed eye. Therefore, a parallel texture stream is introduced. We select three localized Regions of Interest (ROIs)—corresponding to the left eye, right eye, and mouth—from the raw RGB frames. To guarantee computational efficiency, each ROI is resized to a compact resolution of 32 × 32 pixels. Subsequently, these patches are fed into a lightweight MicroCNN encoder and projected into a high-dimensional feature space independently. The resulting sequence of embeddings establishes three distinct "Texture Super-Nodes", denoted as 𝑋𝑡𝑡𝑒𝑥 , which reflects facial appearance states such as eye closure degrees. Finally, 𝑔𝑒𝑜 the geometry nodes 𝑋𝑡 and the texture super-nodes 𝑋𝑡𝑡𝑒𝑥 are concatenated across the feature dimension, formulating a hierarchical multimodal node set.

3.3

Spatial Dynamic Hypergraph Modeling

To ensure semantic alignment across modalities, the raw 3D coor𝑔𝑒𝑜 dinates of the geometry nodes 𝑋𝑡 are first mapped into a shared 𝑑-dimensional latent space via a linear projection layer. Aiming to model the high-order synergistic deformations of facial muscles, we formulate the hierarchical nodes into a heterogeneous hypergraph G = (V, E, W). Let V = V𝑔𝑒𝑜 ∪ V𝑡𝑒𝑥 denote the multimodal node set containing 𝑁𝑡𝑜𝑡𝑎𝑙 = 71 vertices (i.e., 68 geometric nodes and 3 texture super-nodes). Their concatenated feature matrix is denoted as X = [X𝑔𝑒𝑜 ∥ X𝑡𝑒𝑥 ] ∈ R𝑁𝑡𝑜𝑡𝑎𝑙 ×𝑑 . The core of our spatial modeling is constructing a heterogeneous incidence matrix H ∈ R𝑁𝑡𝑜𝑡𝑎𝑙 ×𝐸 that represents the topological connections. Unlike standard graphs restricted to pairwise edges, a hyperedge 𝑒 ∈ E can connect an arbitrary number of vertices. Our incidence matrix consists of two complementary sub-matrices: H = [H𝑔𝑒𝑜 ∥ H𝑡𝑒𝑥 ]. H𝑔𝑒𝑜 represents Geo-Geo Hyperedges, which aims to capture local geometric structures. We compute pairwise Euclidean distances 𝑔𝑒𝑜 based on the canonical coordinates 𝑋𝑡 . For each node, a hyperedge is constructed by connecting it to its 𝑘-nearest neighbors, forming 𝐸𝑔𝑒𝑜 = 68 spatial hyperedges. H𝑡𝑒𝑥 denotes Tex-Geo Hyperedges. To achieve cross-modal fusion, we establish a star-topology broadcasting mechanism and define 𝐸𝑡𝑒𝑥 = 3 hyperedges corresponding to the left eye, right eye, and mouth regions. The incidence value ℎ(𝑣, 𝑒) is formally defined as: ( 1, 𝑣 ∈ R𝑒 ℎ(𝑣, 𝑒) = (3) 0, 𝑣 ∉ R𝑒

where R𝑒 ⊂ V𝑔𝑒𝑜 denotes the predefined subset of geometric nodes associated with the localized related region of super-node 𝑒. By cascading H𝑔𝑒𝑜 and H𝑡𝑒𝑥 , the resulting incidence matrix H bridges localized visual semantics with global structural dynamics without triggering feature dimension explosion. With the incidence matrix H established, we propagate the heterogeneous node features to capture high-order intra-modal and cross-modal interactions. Given the input node features X, the hypergraph convolution layer updates the representations through a symmetric normalized Laplacian propagation[12]: X𝑜𝑢𝑡 = 𝜎 D𝑣−1/2 HWH𝑇 D𝑣−1/2 XΘ (4) where Θ ∈ R𝑑 ×𝑑𝑜𝑢𝑡 is the learnable weight matrix for linear feature transformation. W ∈ R𝐸×𝐸 is a learnable diagonal matrix representing the weight of each hyperedge, enabling the network to dynamically emphasize crucial spatial synergies. We define the weighted adjacency matrix as A = HWH𝑇 . Consequently, D𝑣 is the diagonal node degree matrix where its diagonal element is comÍ puted as the row sum of A (i.e., D𝑣 (𝑖, 𝑖) = 𝑗 A𝑖,𝑗 ). Finally, 𝜎 (·) denotes the LeakyReLU activation function. Through this layer, geometric landmarks can obtain collaborative information from each other and visual semantics from texture super nodes. Following the hypergraph convolution, the geometric nodes 𝑔𝑒𝑜 X𝑜𝑢𝑡 ∈ R𝑁𝑔𝑒𝑜 ×𝑑𝑜𝑢𝑡 are highly enriched with multimodal context. Since the texture super-nodes have fulfilled their broadcasting mission, we safely discard them to focus exclusively on the facial topology. To compress these node-level features into a compact, framelevel representation, we introduce an Adaptive Attention Pooling module. We deploy a Multi-Layer Perceptron with a tanh activation to estimate the distinct contribution of each node dynamically. The attention score 𝑒𝑖 and the normalized weight 𝛼𝑖 for the 𝑖-th node are calculated as: 𝑒𝑖 = w𝑇𝑎 tanh(W𝑎 x𝑖 + b𝑎 ) 𝛼 𝑖 = Í𝑁

exp(𝑒𝑖 )

𝑔𝑒𝑜

𝑗=1

(5) (6)

exp(𝑒 𝑗 )

where W𝑎 , b𝑎 , and w𝑎 are learnable parameters of the attention network. Finally, the frame-level feature vector z𝑠𝑒𝑞 ∈ R𝑑𝑜𝑢𝑡 is Í𝑁𝑔𝑒𝑜 obtained via a weighted sum: z𝑠𝑒𝑞 = 𝑖=1 𝛼𝑖 x𝑖 . This mechanism inherently filters out irrelevant localized noise and selectively forces the model to attend to salient fatigue indicators, formulating a robust temporal sequence for the subsequent Bi-Mamba module.

3.4

Temporal Evolution Modeling

After spatial modeling, the untrimmed video is abstracted into a sequence of highly refined frame-level features Z = [z1, z2, . . . , z𝑇 ] ∈ R𝑇 ×𝑑𝑜𝑢𝑡 . To capture the long-range temporal dependencies of fatigue behaviors, traditional RNNs suffer from long-term forgetting, while Transformers incur an heavy 𝑂 (𝑇 2 ) computational complexity. To resolve this dilemma, we introduce a Temporal Evolution Modeling module driven by a Bidirectional State Space Model (BiMamba). The core of Mamba originates from the continuous-time State Space Model (SSM)[16], which maps a 1D input sequence 𝑧 (𝑡) to

HST-HGN: Spatio-Temporal Hypergraph for Fatigue Detection

an output response 𝑦 (𝑡) via a latent state ℎ(𝑡). Mathematically, it is formulated as a linear Ordinary Differential Equation: ℎ ′ (𝑡) = Aℎ(𝑡) + B𝑧(𝑡),

𝑦 (𝑡) = Cℎ(𝑡)

B̄ = (ΔA) −1 (exp(ΔA) − I) · ΔB

(8)

Crucially, standard SSMs utilize time-invariant parameters. In contrast, our module leverages the selective scan mechanism[15], parameterizing B, C, and Δ as data-dependent functions of the input z𝑡 . This input-awareness empowers the model to dynamically filter out redundant information (e.g., prolonged periods of normal driving) and memorize salient occurrences (e.g., the onset of a yawn) into the hidden state. Standard mamba has a causal, unidirectional nature—it processes sequences strictly forward. However, assessing complex behaviors often requires offline temporal localization where future context is necessary for understanding current actions. Therefore, we deploy a Bi-Mamba block comprising a forward SSM and a backward SSM. Given the input sequence, the forward scan processes it chronolog→ − ically to yield Y , while the backward scan processes the reversed ← − sequence to yield Y . The bidirectional temporal representation is then obtained via feature summation: → − ← − Y𝑏𝑖 = Y + Y ∈ R𝑇 ×𝑑𝑜𝑢𝑡 (9) Finally, since fatigue events may occur sparsely within the 128frame sequence, mean pooling might dilute these critical signals. Thus, we perform a global temporal max pooling across the temporal dimension to extract the most discriminative global video vector v 𝑓 𝑖𝑛𝑎𝑙 : (𝑡 ) v 𝑓 𝑖𝑛𝑎𝑙 = max (Y𝑏𝑖 ) 𝑇

𝑡 =1

(10)

This vector is subsequently fed into a fully connected classification head with a Softmax activation to predict the ultimate behavior category (Normal, Talking, or Yawning).

3.5

Optimization and Loss Function

For the reason of two inherent challenges in unconstrained driving datasets: severe class imbalance and subtle intra-class variations, We tackle this by jointly optimizing a multi-class Focal Loss L 𝑓 𝑜𝑐 and a Center Loss L𝑐𝑒𝑛 . To counteract class imbalance and heavily penalize hard-todistinguish boundary samples such as an onset of yawning versus talking, L 𝑓 𝑜𝑐 is formulated as: L 𝑓 𝑜𝑐 = −

𝐶 ∑︁

𝛼𝑐 (1 − 𝑝𝑐 )𝛾 𝑦𝑐 log(𝑝𝑐 )

𝐵

(7)

where A acts as the evolution matrix, and B, C are projection parameters. To process discrete video frames, we apply the zero-order hold rule with a timescale parameter Δ to discretize the continuous matrices into Ā and B̄: Ā = exp(ΔA),

between the temporal feature v 𝑓 𝑖𝑛𝑎𝑙 and its corresponding learnable class center c𝑦𝑖 : L𝑐𝑒𝑛 =

1 ∑︁ (𝑖 ) ||v − c𝑦𝑖 || 22 2 𝑖=1 𝑓 𝑖𝑛𝑎𝑙

(12)

where 𝐵 is the batch size. The overall objective function is optimized as L𝑡𝑜𝑡𝑎𝑙 = L 𝑓 𝑜𝑐 + 𝜆L𝑐𝑒𝑛 , where the hyperparameter 𝜆 balances the two terms.

4 Experiments 4.1 Datasets and Implementation Details Dataset. To evaluate the robustness and generalizability of our proposed HST-HGN framework, we conduct experiments on a diverse collection of real-world datasets. Our primary training and evaluation benchmark is derived from the widely adopted YawDD dataset[2]. To construct a high-fidelity benchmark, we split and relabel sequences containing overlapping talking and yawning instances. Then, we recursively truncate long sequences (duration > 30s) and discard overly short segments. This curation yields a clean, non-overlapping corpus of 427 pure video clips. To fundamentally prevent data leakage and assess true cross-identity generalization, we strictly enforce a subject-independent (driver-level) split. The dataset is partitioned into training (70%), validation (15%), and testing (15%) sets based on unique subject IDs, intrinsically preserving the real-world class imbalance. To further establish a comprehensive cross-domain generalization analysis, our evaluation uses four additional fatigue datasets: UTA-RLDD[14], FatigueView[48], and DMD[30]. These datasets introduce varied camera perspectives, diverse illumination conditions, and heterogeneous behavioral annotations. Due to the missing Talking labels and samples in some datasets, all sequence annotations are roughly mapped into a binary classification of Yawning and Normal. Implementation Details. All experiments are conducted using PyTorch(v2.4.0) and accelerated via CUDA 3.11 on a system equipped with a single NVIDIA RTX 3060 GPU. This hardware constraint deliberately underscores the lightweight and edge-deployment feasibility of our architecture. For spatial modeling, the spatial hypergraph is constructed dynamically with 𝐾 = 4 nearest neighbors. The entire network is trained from scratch for 100 epochs with a batch size of 8. The main network parameters are optimized using the Adam optimizer with an initial learning rate of 1×10−3 and a weight decay of 1 × 10−4 to prevent overfitting. Concurrently, the learnable class centers formulated in the Center Loss are updated using Stochastic Gradient Descent (SGD) with a significantly larger learning rate of 0.5. For the loss formulation, the Focal Loss parameters are set to 𝛼 = 0.25 and 𝛾 = 2.0 to heavily penalize hard-to-distinguish boundary samples, while the Center Loss weight is set to 𝜆 = 0.001 to enforce intra-class compactness.

(11)

𝑐=1

where 𝐶 = 3, 𝑦𝑐 is the one-hot label, and the modulating factor (1−𝑝𝑐 )𝛾 dynamically scales the loss based on prediction confidence. Simultaneously, to enforce intra-class compactness regardless of individual identity or head pose, L𝑐𝑒𝑛 minimizes the distance

4.2

Comparison with SOTA Methods

We compare our proposed HST-HGN with a comprehensive suite of state-of-the-art (SOTA) methods on the YawDD dataset. To ensure a rigorous evaluation, our baselines include foundational generic video understanding architectures (SlowFast[11]

Table 1: Comparison with state-of-the-art generic video models and recent domain-specific fatigue detection networks on the YawDD dataset. The best and second-best results are highlighted in bold and underlined, respectively.

Table 2: Cross-dataset generalization performance (Binary classification). The classification head is adapted while the backbone remains unchanged to evaluate domain transferability.