DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models
arXiv:2604.15016v1 [cs.LG] 16 Apr 2026
Jingyuan Wang1 , Meiyan Xu2 , Zhihao Jia1 , Chenyu Liu3 , Xinliang Zhou3 , Ziyu Jia4 , Yong Li5 , Fang Li3 , Junfeng Yao1,∗ , Yi Ding3,∗ 1 Xiamen University 2 Zhangzhou Normal University 3 Nanyang Technological University 4 Institute of Automation, Chinese Academy of Sciences 5 Southeast University ∗ Corresponding authors [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] Abstract EEG foundation models (FMs) achieve strong cross-subject and cross-task generalization but impose substantial computational and memory costs that hinder deployment on embedded BCI systems. Knowledge distillation is a natural solution; however, conventional methods fail for EEG FMs because task-relevant semantics are often distributed across intermediate layers, and aggressive dimensionality reduction can distort oscillatory structure via representational collapse and aliasing. To address these challenges, we propose DLink (Distilling Layer-wise and Dominant Knowledge), a unified framework for transferring knowledge from large EEG FMs to compact students with three key innovations: (1) a dynamic Router that adaptively aggregates teacher layers to capture dominant intermediate representations; (2) an EEG MiC student with a Mimic-then-Compress pipeline, which inherits high-dimensional teacher features and then applies structured spatio-temporal compression to avoid a heavy classification head; and (3) spectral distillation that aligns teacher–student representations in the frequency domain to regularize compression and mitigate aliasing and temporal jitter. Experiments on four EEG benchmarks show that DLink enables compact students to outperform lightweight baselines while approaching fully finetuned FM performance at substantially lower model size and inference cost.
1
Introduction
Electroencephalogram (EEG) signals provide a non-invasive window into neural dynamics [Pan et al., 2023; Jia et al., 2025]. Recently, EEG foundation models (FMs) have set new benchmarks by learning generalizable representations [Wu
teacher Layer-wise Linear Probe Acc layers 0
FM
1
NO.1
...
best
...
last
features
our distillation method Acc
classifier
Spectral
Parameters Transform
logits
distill
compressor
Traditional distill
router
Mimicker
student(MiC)
student
Figure 1: DLink vs. Traditional KD. Unlike rigid last-layer distillation, DLink utilizes a dynamic Router for multi-layer aggregation and Spectral Transformation for efficient distillation into the EEG MiC student, optimizing the accuracy-parameter trade-off.
et al., 2025]. However, their deployment on resourceconstrained embedded systems is hindered by substantial memory requirements and limited on-device adaptability, motivating the need for model compression and knowledge distillation to enable efficient deployment. While knowledge distillation (KD) has successfully compressed large models in other domains [Ko et al., 2024], the distillation of EEG FMs remains largely unexplored and presents several domain-specific challenges. A fundamental question in distilling EEG FMs is which representations should be transferred to compact students. As illustrated in Fig. 1, conventional KD typically treats the final layer as the supervision source [Kan et al., 2025], whereas intermediate layers often encode richer and more task-relevant information [Liang et al., 2023]. Our empirical analysis confirms this phenomenon in EEG FMs: linear probing across all hidden layers (Fig. 5) shows that intermediate representa-
tions consistently outperform the final classification-oriented layer. This suggests that EEG FMs organize critical rhythmic and spatio-temporal patterns hierarchically [Zhou et al., 2025], and static reliance on the last layer risks discarding dominant knowledge essential for downstream decoding. However, transferring such high-dimensional intermediate representations introduces a second challenge. Our Mimicker aligns with the teacher’s wide latent space, producing high-dimensional feature maps. Directly flattening these features would require an overly parameterized classifier, while naive dimensionality reduction risks collapsing informative structure. [Dubois et al., 2020]. To balance fidelity and efficiency, we design the EEG MiC student architecture following a staged Mimic-then-Compress paradigm. The student first mimics teacher features to inherit rich semantic structure, and then applies structured dimensionality reduction to meet strict computational constraints. Two variants—MiC-S (Small) and MiC-M (Medium)—enable systematic evaluation across different efficiency budgets. Importantly, aggressive structural compression is not merely a capacity issue but also a signal processing problem. When high-dimensional representations are reduced, highfrequency components may be folded into lower-frequency bands, leading to aliasing artifacts that distort key spectral characteristics of EEG signals [Ribeiro and Schön, 2021; Zhang, 2019]. Such distortion undermines the faithful transfer of oscillatory and rhythmic information that is central to neural decoding. To address this issue, DLink aligns teacher–student representations in the frequency domain, explicitly regularizing the compression process to suppress aliasing. Beyond its anti-aliasing effect, spectral alignment leverages the EEG frequency domain as a physiologically grounded and robust representation space, which is inherently less sensitive to temporal shifts and better preserves oscillatory patterns [Wang et al., 2024; Geirnaert et al., 2022; Li et al., 2022]. Our main contributions are summarized as follows: • A Unified Distillation Framework: We propose DLink, a distillation framework for compressing large EEG FMs into compact students, integrating the teacher, a dynamic Router, and the MiC student to coordinate layer-wise knowledge transfer. • Dynamic Selection of Dominant Knowledge: The Router identifies and aggregates the most informative representations across all teacher layers, enabling the student to focus on critical hierarchical features. • Efficient Student Learning with Spectral Alignment: The MiC student follows a Mimic-then-Compress paradigm, and frequency-domain alignment is employed as an anti-aliasing regularizer to faithfully distill highdimensional features into a compact architecture (see Section 3.4). • Strong Performance Across EEG Tasks: Experiments on four EEG datasets show that DLink surpasses established small models, and the MiC-M student achieves competitive performance compared with fully fine-tuned FMs while maintaining minimal computational cost,
demonstrating the framework’s potential for deployment on resource-constrained BCI devices.
2
Background
2.1
EEG Foundation Models
EEG foundation models (FMs) learn hierarchical spatiotemporal representations from large, heterogeneous brain signal datasets [Wang et al., 2024]. Representative architectures such as LaBraM [Jiang et al., 2024], EEG-DINO [Wang et al., 2025b], and CBraMod [Wang et al., 2025a] leverage deep Transformer backbones and specialized modules (e.g., CrissCross Transformers) to integrate cross-channel and temporal information. While these models achieve strong performance, their depth and parameter size (often >15M) hinder deployment on resource-constrained BCI devices. Moreover, knowledge is distributed across layers, with intermediate representations capturing rich rhythmic and spatio-temporal patterns critical for downstream decoding [Zhou et al., 2025]. This hierarchical structure motivates distillation strategies that can selectively inherit informative features rather than relying solely on final-layer outputs.
2.2
Knowledge Distillation and Spectral Motivation
Knowledge distillation (KD) [Park et al., 2021; Gou et al., 2021] transfers knowledge from a high-capacity teacher to a compact student, but for hierarchical EEG foundation models, identifying informative intermediate representations is non-trivial [Liang et al., 2023; Ramtoula et al., 2025]. Direct feature-space alignment [Liu et al., 2023] is particularly fragile for EEG due to strong non-stationarity and temporal shifts, where small misalignments can lead to large time-domain discrepancies. From a signal processing perspective, dimensionality reduction and implicit subsampling across layers may introduce aliasing, as high-frequency components can fold into lower-frequency bands and distort neural representations [Zhang, 2019]. The frequency domain offers a principled alternative: spectral representations are more robust to temporal shifts and emphasize physiologically meaningful rhythmic patterns [Ding et al., 2024; Wang et al., 2024; Ding et al., 2023]. Aligning teacher and student spectra therefore provides a natural mechanism for transferring taskrelevant oscillatory structures while mitigating spectral distortion under compression [Zhang et al., 2024]. Taken together, EEG FMs provide hierarchical, multi-scale features that are critical for decoding, conventional KD struggles with both layer selection and temporal variability, and spectral representations offer a robust medium for preserving essential information under compression.
3
Methodology
3.1
Problem Formulation and Framework Overview
Transferring knowledge from large EEG foundation models to compact students is challenging due to limited computational resources and the hierarchical, multi-layered nature of teacher representations. Our goal is to distill the most
layer 1
C
S
C
layer 2
× S
t
Input
t
layer
foundation
Spectral Transform
teacher
Mag
phase
Mag
phase
× Mag
phase
model
PSD
scores
concatenate
mag
student (miC)
router
training
C conv
Element-wise
Compressor
S
Product
Norm
Mimicker frozen
phase
Gelu
logits
t mag
transformer
classifier
Spectral Transform
phase
Figure 2: Overview of the DLink pipeline. The framework distills knowledge from a multi-layer EEG Foundation Model (FM) teacher into the EEG MiC student (comprising a Mimicker and Compressor). A dynamic Router adaptively aggregates these hierarchical FM layers guided by PSD scores, while frequency-domain alignment of magnitude and phase ensures efficient EEG decoding.
Mimicker
Compressor Gelu
classifier
conv
Norm
conv
S
Norm
Input
C CNN
Gelu
t
where α is learnable. Here fmimic retains the original structure (C×S×T) . Compressor: Applies structured spatio-temporal downsampling Ds , Dt followed by a bottleneck projection: fcomp = C(fmimic ) = Dt (Ds (fmimic )) .
transformer
summation
Element-wise Product
Learnable factor
feature
Figure 3: EEG MiC architecture comprising a Mimicker, a Compressor, and an MLP Classifier. (l)
task-relevant features {fT }L l=1 from a high-capacity teacher (l) T to a student S, where fT has a channel–segment–time (C×S×T) structure, while preserving spectral information critical for EEG decoding. To achieve this, we propose DLink (Distilling Layer-wise and Dominant Knowledge), a framework where three components work together, as illustrated in Fig. 2. The Router decides what layers of the teacher to emphasize, guiding the student toward task-salient representations. Spectral Distillation ensures how the knowledge is transferred by aligning student and teacher features in the frequency domain, making the process robust to temporal jitter and aliasing. The EEG MiC student then how to implement this knowledge: it first absorbs the selected information in a high-dimensional latent space (Mimic) and then compresses it for task-specific predictions (Compress).
3.2
EEG MiC Student: Mimic-then-Compress
As shown in Fig. 3, the student manages high-dimensional knowledge transfer via two stages followed by a classification head. Mimicker: Combines depthwise separable convolutions fcnn for local temporal features and a Transformer encoder ftrans for global context: fmimic = α · fcnn + (1 − α) · ftrans ,
(1)
(2)
This reduces all three dimensions, yielding a compact representation with smaller channel, segment, and time sizes. Classifier: Maps compressed features to the label space with an MLP: ŷ = LinearNcls Dropout ELU(Lineardh (vec(fcomp ))) , (3) where vec(·) flattens the feature map, and Lineardh represents a linear transformation to a dh -dimensional hidden space.
3.3
Adaptive Layer-wise Knowledge Aggregation
DLink employs a lightweight Router R to adaptively select knowledge from all L teacher layers. To capture both structural and spectral cues, the Router takes a concatenated input zin defined as: zin = [f̄mimic ∥ M̄S ], (4) where f̄mimic and M̄S represent the channel-wise averaged temporal features and their corresponding magnitude spectra, respectively. Subsequently, the Router maps zin to layer-wise logits a ∈ RL via a streamlined policy network: a = Hhead Tenc σ(Conv1D(zin )) . (5) This architecture sequentially employs a Channel Fuser (Conv1D with GELU σ) for dimensionality reduction, a Transformer-based Sequence Processor (Tenc ) to capture temporal dependencies, and a Routing Head (Hhead ) that applies global pooling and linear projection to predict layer importance.
Dynamic Weight Generation The logits are normalized through a temperature-scaled softmax function to produce the aggregation weights w = [w1 , . . . , wL ]: exp(al /τ )
wl = PL
k=1 exp(ak /τ )
MS (ω) ≈ ,
(6)
where τ is a temperature hyperparameter. These weights represent the importance of each teacher layer on a per-sample basis and guide the subsequent distillation objective. Spectral Energy Supervision To guide the Router toward task-relevant layers, we introduce spectral energy scores {sl }L l=1 as a supervisory signal. For each teacher layer l, the score sl is computed from its power spectral density (PSD): X 1 (l) sl = |MT (c, s, f )|2 , (7) |Ω| (c,s,f )∈Ω
where Ml is the magnitude spectrum and Ω indexes channel, segment, and frequency dimensions. The Router is optimized via KL divergence: s a ∥ softmax , (8) Lpsd = DKL softmax τ τ where s = [s1 , . . . , sL ].
3.4
DLink introduces spectral distillation to align MS with a dynamically weighted combination of teacher spectra: L X
(l)
wl · MT (ω),
(10)
l=1
where wl is provided by the Router. Rather than indiscriminately suppressing high-frequency components, this constraint penalizes spectral deviations unsupported by the teacher representation. As shown in Fig. 6, DLink preserves more high-frequency details before compression, indicating higher fidelity to the teacher spectrum, while exhibiting a significantly reduced high-frequency tail and preserved lowfrequency energy after compression, providing direct evidence of suppressed aliasing. Therefore, spectral distillation acts as an implicit antialiasing regularizer: it shapes the student spectrum such that high-frequency components remain structured and do not collapse into aliasing artifacts under downsampling. A secondary benefit is improved robustness to temporal jitter, since magnitude-based alignment satisfies |F{fmimic (t − δ)}| = |F{fmimic (t)}|,
(11)
making the distilled representation invariant to small temporal shifts.
3.5
Optimization Objective
Spectral Distillation with Anti-Aliasing Regularization
The overall training objective balances task performance, spectral distillation fidelity, and adaptive layer aggregation:
DLink performs knowledge transfer by aligning student and teacher representations in the frequency domain. Given a feature map f , we compute its Fourier representation using real FFT: F(f ) = M⊙ejΦ , where M and Φ denote the magnitude and phase spectra. To inherit task-salient expertise, the total distillation loss is formulated as a weighted sum of the spectral discrepancies across all teacher layers, modulated by the weights w from the Router:
Ltotal = Lcls + λ1 Ldistill + λ2 Lpsd ,
Ldistill =
L X
(l)
(l)
wl · ∥MS − MT ∥2F + ∥Φenc,S − Φenc,T ∥2F ,
l=1
(9) where Φenc = (cos Φ, sin Φ) is the encoded phase representation, providing a continuous and rotation-invariant embedding that avoids the 2π discontinuity of raw phase val(l) (l) ues, and MT , ΦT denote the spectral components of the l-th teacher layer. Anti-Aliasing Regularization The MiC student employs strided convolutions in the Compressor to achieve aggressive temporal compression, which inevitably risks aliasing artifacts when high-frequency components are folded into lower frequencies [Zhang, 2019]. According to the Nyquist– Shannon principle [Shannon, 1949], downsampling by stride k requires the signal spectrum to be sufficiently band-limited; otherwise, spectral replicas induced by striding may distort the compressed representation. Let fmimic denote the student features before compression and MS (ω) = |F{fmimic }(ω)| their magnitude spectrum.
(12)
where Lcls denotes the task-specific classification loss (binary cross-entropy or categorical cross-entropy), and λ1 , λ2 control the relative importance of each component.
4
Experiments
4.1
Datasets and Preprocessing
We evaluate DLink on four diverse BCI benchmarks covering distinct neural decoding tasks: (1) FACED [Chen et al., 2023] for 9-class emotion recognition (32-ch, 250 Hz); (2) Mumtaz2016 [Mumtaz, 2016] for binary depression diagnosis (19-ch, 256 Hz); (3) PhysioNet-MI [Goldberger et al., 2000; Schalk et al., 2004] for 4-class motor imagery (64ch, 160 Hz); and (4) SHU-MI [Ma et al., 2022] for binary motor imagery (32-ch, 250 Hz). Following the preprocessing pipeline established in CBraMod [Wang et al., 2025a], all EEG signals are resampled to 200 Hz and segmented into 1-second patches to align with the foundation model’s input format. For example, FACED adopts a cross-subject split, and all datasets are partitioned into training, validation, and testing sets according to their standard paradigms. See Appendix for details.
4.2
Baselines
To validate DLink, we compare the MiC student against three standalone lightweight models: EEGNet [Lawhern et al., 2018], Conformer [Song et al., 2023], and Deformer [Ding et al., 2025]. Three fine-tuned foundation models (CBraMod [Wang et al., 2025a], LaBraM-Base [Jiang
Model
ACC-B
FACED (9-class) F1-W
Kappa
ACC-B
Mumtaz2016 (2-class) AUROC
AUC-PR
0.2281 ± 0.0265 0.4324 ± 0.0188 0.3440 ± 0.0126 0.4118 ± 0.0206 0.5038 ± 0.0172
0.1771 ± 0.0353 0.4405 ± 0.0188 0.3408 ± 0.0145 0.4109 ± 0.0192 0.5063 ± 0.0158
0.1294 ± 0.0306 0.3772 ± 0.0086 0.2592 ± 0.0123 0.3355 ± 0.0225 0.4390 ± 0.0190
0.8958 ± 0.0056 0.8923 ± 0.0075 0.9003 ± 0.0247 0.8853 ± 0.0175 0.8975 ± 0.0171
0.9616 ± 0.0128 0.9502 ± 0.0060 0.9760 ± 0.0137 0.9634 ± 0.0103 0.9648 ± 0.0078
0.9678 ± 0.0073 0.9526 ± 0.0080 0.9806 ± 0.0156 0.9692 ± 0.0089 0.9702 ± 0.0069
0.4915 ± 0.0050 0.4793 ± 0.0032 0.4461 ± 0.0071
0.4888 ± 0.0050 0.4805 ± 0.0033 0.4451 ± 0.0059
0.4240 ± 0.0059 0.4116 ± 0.0034 0.3735 ± 0.0080
0.9011 ± 0.0076 0.8968 ± 0.0018 0.8851 ± 0.0041
0.9753 ± 0.0070 0.9706 ± 0.0047 0.9735 ± 0.0042
0.9785 ± 0.0048 0.9732 ± 0.0038 0.9739 ± 0.0038
0.4239 ± 0.0068 0.4125 ± 0.0125 0.4330 ± 0.0132
0.4217 ± 0.0067 0.4117 ± 0.0127 0.4251 ± 0.0107
0.3486 ± 0.0083 0.3367 ± 0.0141 0.3548 ± 0.0113
0.8801 ± 0.0042 0.8491 ± 0.0234 0.9010 ± 0.0045
0.9631 ± 0.0069 0.9325 ± 0.0161 0.9612 ± 0.0044
0.9697 ± 0.0056 0.9449 ± 0.0130 0.9683 ± 0.0034
0.4858 ± 0.0259 0.5122 ± 0.0077 0.5221 ± 0.0052
0.4889 ± 0.0234 0.5117 ± 0.0071 0.5202 ± 0.0074
0.4191 ± 0.0281 0.4483 ± 0.0078 0.4581 ± 0.0066
0.8963 ± 0.0131 0.8937 ± 0.0230 0.9024 ± 0.0082
0.9674 ± 0.0080 0.9464 ± 0.0377 0.9695 ± 0.0074
0.9731 ± 0.0064 0.9633 ± 0.0212 0.9738 ± 0.0060
ACC-B
PhysioNet-MI (4-class) F1-W
Kappa
ACC-B
SHU (2-class) AUROC
AUC-PR
0.5890 ± 0.0062 0.5879 ± 0.0057 0.6056 ± 0.0042 0.5828 ± 0.0085 0.5806 ± 0.0119
0.5968 ± 0.0133 0.5878 ± 0.0056 0.6065 ± 0.0052 0.5855 ± 0.0080 0.5818 ± 0.0148
0.4853 ± 0.0076 0.4504 ± 0.0077 0.4768 ± 0.0106 0.4437 ± 0.0113 0.4408 ± 0.0160
0.5714 ± 0.0221 0.6014 ± 0.0236 0.5829 ± 0.0185 0.5806 ± 0.0221 0.5846 ± 0.0364
0.6484 ± 0.0125 0.6651 ± 0.0222 0.6676 ± 0.0077 0.6104 ± 0.0324 0.6243 ± 0.0438
0.6305 ± 0.0115 0.6463 ± 0.0213 0.6503 ± 0.0070 0.6050 ± 0.0359 0.6190 ± 0.0431
0.6188 ± 0.0072 0.6150 ± 0.0087 0.5753 ± 0.0080
0.6205 ± 0.0079 0.6152 ± 0.0088 0.5757 ± 0.0089
0.4917 ± 0.0096 0.4866 ± 0.0117 0.4337 ± 0.0108
0.6166 ± 0.0165 0.5997 ± 0.0163 0.5969 ± 0.0161
0.6647 ± 0.0266 0.6442 ± 0.0241 0.6461 ± 0.0214
0.6599 ± 0.0322 0.6464 ± 0.0269 0.6455 ± 0.0313
0.5934 ± 0.0066 0.5609 ± 0.0232 0.5979 ± 0.0043
0.5942 ± 0.0094 0.5666 ± 0.0238 0.6013 ± 0.0042
0.4579 ± 0.0089 0.4145 ± 0.0310 0.4638 ± 0.0057
0.5848 ± 0.0197 0.5623 ± 0.0152 0.6073 ± 0.0133
0.6210 ± 0.0281 0.6274 ± 0.0163 0.6488 ± 0.0234
0.6148 ± 0.0303 0.6295 ± 0.0194 0.6455 ± 0.0203
0.5831 ± 0.0048 0.5640 ± 0.0122 0.6060 ± 0.0049
0.5858 ± 0.0072 0.5652 ± 0.0136 0.6037 ± 0.0052
0.4441 ± 0.0065 0.4186 ± 0.0162 0.4666 ± 0.0076
0.5920 ± 0.0088 0.6051 ± 0.0132 0.6150 ± 0.0099
0.6218 ± 0.0132 0.6573 ± 0.0154 0.6782 ± 0.0071
0.6115 ± 0.0185 0.6566 ± 0.0170 0.6676 ± 0.0137
Small Models
EEGNet EEGConformer EEGDeformer MiC-S (Ours) MiC-M (Ours) Teachers (Fine-tuned)
CBraMod LaBraM EEG-DINO KD (Student: MiC-S)
FitNets Logit-std DLink (Ours) KD (Student: MiC-M)
FitNets Logit-std DLink (Ours) Model Small Backbones
EEGNet EEGConformer EEGDeformer MiC-S (Ours) MiC-M (Ours) Teachers (Fine-tuned)
CBraMod LaBraM EEG-DINO KD (Student: MiC-S)
FitNets Logit-std DLink (Ours) KD (Student: MiC-M)
FitNets Logit-std DLink (Ours)
Table 1: Main results across four datasets (bold/underline: top two results among lightweight models only; teachers serve as reference upper bounds). For fairness, all teachers and MiC-M share a consistent 200-unit hidden layer in their classification MLP heads.
et al., 2024], and EEG-DINO [Wang et al., 2025b]) serving as performance upper bounds. For teacher features, we apply simple reshaping to a unified space to enable direct layerwise aggregation by the Router without additional learnable projections. Furthermore, we benchmark DLink against two representative distillation paradigms: FitNets [Romero et al., 2015] and Logit-std [Sun et al., 2024]. These baselines represent traditional feature-level and logit-level distillation, respectively.
4.3
Results
The empirical evaluations across four EEG datasets, summarized in Table 1 and Table 2, demonstrate the robust performance of the DLink framework. A detailed description of the evaluation metrics is provided in Appendix. Efficacy of Student Scaling and Distillation. We vali-
dated DLink on two student configurations: MiC-S and MiCM. As shown in Table 1, both models exhibit significant performance gains post-distillation compared to their standard supervised counterparts. Notably, the distilled MiC-S effectively outperforms almost all established lightweight baselines. Furthermore, DLink consistently surpasses conventional distillation paradigms, including the feature-based FitNets and the output-based Logit-std. This confirms that our adaptive layer-wise aggregation and spectral-domain alignment capture more discriminative knowledge than static feature or logit transfer strategies. For the larger MiC-M variant, the distillation process enables it to reach parity with or even surpass fine-tuned foundation models, proving that our router effectively filters teacher redundancy to extract task-salient expertise. Impact of Teacher Expertise. Table 2 shows that DLink is
Configuration
FACED (9-class)
Mumtaz2016 (2-class)
Student
Teacher
ACC-B
F1-W
Kappa
ACC-B
AUROC
AUC-PR
MiC-S
CBraMod LaBraM EEG-DINO
0.4330 ± 0.0132 0.4248 ± 0.0063 0.4258 ± 0.0171
0.4251 ± 0.0107 0.4233 ± 0.0080 0.4227 ± 0.0172
0.3548 ± 0.0113 0.3498 ± 0.0074 0.3500 ± 0.0190
0.8970 ± 0.0094 0.9010 ± 0.0045 0.9008 ± 0.0069
0.9571 ± 0.0083 0.9612 ± 0.0044 0.9664 ± 0.0074
0.9655 ± 0.0065 0.9683 ± 0.0034 0.9722 ± 0.0057
MiC-M
CBraMod LaBraM EEG-DINO
0.5221 ± 0.0052 0.5191 ± 0.0096 0.5246 ± 0.0074
0.5202 ± 0.0074 0.5194 ± 0.0110 0.5236 ± 0.0075
0.4581 ± 0.0066 0.4563 ± 0.0109 0.4616 ± 0.0085
0.9015 ± 0.0041 0.9024 ± 0.0082 0.8999 ± 0.0143
0.9671 ± 0.0082 0.9695 ± 0.0074 0.9628 ± 0.0115
0.9727 ± 0.0063 0.9738 ± 0.0060 0.9686 ± 0.0096
Configuration
PhysioNet-MI (4-class)
SHU (2-class)
Student
Teacher
ACC-B
F1-W
Kappa
ACC-B
AUROC
AUC-PR
MiC-S
CBraMod LaBraM EEG-DINO
0.5915 ± 0.0009 0.5979 ± 0.0043 0.5937 ± 0.0080
0.5950 ± 0.0028 0.6013 ± 0.0042 0.5961 ± 0.0089
0.4554 ± 0.0013 0.4638 ± 0.0057 0.4582 ± 0.0107
0.6073 ± 0.0133 0.6054 ± 0.0150 0.5924 ± 0.0165
0.6488 ± 0.0234 0.6487 ± 0.0165 0.6303 ± 0.0171
0.6455 ± 0.0203 0.6492 ± 0.0193 0.6289 ± 0.0231
MiC-M
CBraMod LaBraM EEG-DINO
0.5954 ± 0.0094 0.6060 ± 0.0049 0.5977 ± 0.0041
0.5970 ± 0.0102 0.6037 ± 0.0052 0.5990 ± 0.0046
0.4605 ± 0.0126 0.4666 ± 0.0076 0.4635 ± 0.0055
0.6150 ± 0.0099 0.6068 ± 0.0155 0.5988 ± 0.0172
0.6782 ± 0.0071 0.6571 ± 0.0267 0.6402 ± 0.0295
0.6676 ± 0.0137 0.6506 ± 0.0299 0.6370 ± 0.0261
Table 2: Cross-dataset ablation analysis of DLink performance using diverse foundation model teachers. The table is organized into two sections, each covering two datasets, to ensure legibility. The best result within each student group is highlighted in bold.
Variant
Kappa
0.3933 ± 0.0049 0.4224 ± 0.0226 0.3968 ± 0.0123
0.3134 ± 0.0053 0.3459 ± 0.0250 0.3186 ± 0.0134
0.3897 ± 0.0042 0.4166 ± 0.0232 0.3948 ± 0.0119
Route: Fixed Last Route: Fixed Avg
0.4007 ± 0.0069 0.4248 ± 0.0114
0.3234 ± 0.0080 0.3502 ± 0.0118
0.3995 ± 0.0069 0.4235 ± 0.0095
Metric: Mean Power Metric: Max Amp Metric: PSD (Ours)
0.4273 ± 0.0058 0.4212 ± 0.0094 0.4330 ± 0.0132
0.3533 ± 0.0059 0.4191 ± 0.0101 0.3548 ± 0.0112
0.4259 ± 0.0042 0.3436 ± 0.0106 0.4251 ± 0.0107
Student: MiC-M w/o CNN w/o Trans w/o Lpsd
0.5086 ± 0.0068 0.4913 ± 0.0082 0.4917 ± 0.0107
0.4447 ± 0.0061 0.4245 ± 0.0098 0.4257 ± 0.0125
0.5096 ± 0.0034 0.4898 ± 0.0090 0.4923 ± 0.0146
Route: Fixed Last Route: Fixed Avg
0.4273 ± 0.0093 0.5171 ± 0.0054
0.4285 ± 0.0100 0.4530 ± 0.0067
0.4945 ± 0.0088 0.5163 ± 0.0077
Metric: Mean Power Metric: Max Amp Metric: PSD (Ours)
0.5167 ± 0.0045 0.5036 ± 0.0071 0.5220 ± 0.0051
0.4530 ± 0.0065 0.4819 ± 0.0069 0.4581 ± 0.0066
0.5166 ± 0.0073 0.4157 ± 0.0076 0.5202 ± 0.0073
Table 3: Ablation study on FACED.
robust across different teacher architectures and tasks. While the choice of teacher affects final performance, DLink consistently improves over standalone training across all FM teachers and datasets. Different teachers exhibit strengths on different classification settings, reflecting complementary domain expertise. Overall, these results indicate that the DLink router can adaptively identify and transfer task-salient knowledge from diverse teachers to compact students, independent of the specific teacher architecture. Accuracy-Efficiency Trade-offs. We analyze the tradeoff between accuracy, computation (FLOPs), and parameters in resource-constrained BCI settings (Fig. 4). On FACED, MiC-M achieves superior accuracy with only 1.25M parameters and 27M FLOPs, corresponding to a nearly 50× FLOPs reduction compared to CBraMod (1288M) and LaBraM (1480M), while MiC-S reaches 43.30% accuracy using merely 20M FLOPs, outperforming EEGNet under
45 40
(17.72) CBraMod
(1.25) Mic-M
50
(0.2) Mic-S
(0.22) Conformer
35
(1.31) Deformer
30 25
(17.24) EEG-Dino
(18.80) LaBraM
Parameters(M)
F1-W
Accuracy (%)
ACC-B
Student: MiC-S w/o CNN w/o Trans w/o Lpsd
EEGNet (0.1)
20
30
40
100 150 200 800 1000 1200 1400 1600 FLOPs (M)
Figure 4: Accuracy vs. efficiency on FACED. Bubble size and color denote parameter volume.
lower budgets. The proposed Router adds negligible overhead (0.04M parameters, 0.22M FLOPs) and does not affect the overall efficiency profile; its learned routing decisions are distilled into the student model, rendering it inactive during inference. Notably, directly flattening the mimicked features and applying a standard MLP classifier would require 12.8M parameters and 13M FLOPs for the classifier alone, exceeding the total budget of both MiC-S and MiC-M. In contrast, MiC enables structured compression that preserves discriminative power, allowing MiC variants to occupy the upper-left frontier of the accuracy–efficiency spectrum.
4.4
Implementation Details
Optimization and Training. All models are implemented in PyTorch and optimized using the AdamW optimizer (β1 = 0.9, β2 = 0.999) with a weight decay of 0.05. We train each configuration for 100 epochs with a batch size of 64. For the DLink objective (Eq. 12), we fix λ2 = 1.0 and set the distillation weight λ1 and learning rate (LR) according to datasetspecific requirements: FACED (LR: 2e-3, λ1 : 0.5), Mumtaz2016 (LR: 8e-3, λ1 : 0.2), PhysioNet-MI (LR: 2e-3, λ1 : 0.8), and SHU (LR: 5e-4, λ1 : 0.2). Configurations and Evaluation. We evaluate two stu-
Trajectory of Layer Selection
Layer-wise Linear Probe Acc
via MLP Reduction Baseline (Temporal KD)
1.0
Dataset: Mumtaz2016
DLink (Spectral KD)
normal controls
Area (Baseline - DLink) Log Magnitude(dB)
major depressive disorder
0.8
0.6
0.4
High Freq 0.2 0.0
0.1
0.2
0.3
0.4
0.5
Normalized Frequency (0→DC, 0.5→Nyquist Limit)
Linear Probe Best Layer (11) Router Policy Evolution Student Best Matching Layer
Visualization of Features
Feature Specturm (After Compression)
via Proposed Compressor
1.2
Aliasing Suppression Area
1.0
20
...
100 Epochs
23
24
25
26 Acc(%)
Figure 5: Interpretability analysis of the Router policy on the FACED dataset. The synchronization between autonomous policy discovery and objective probing results validates the Router’s effectiveness in identifying dominant knowledge.
dent scales, MiC-S and MiC-M, which are defined by distinct configurations of convolution kernels, output channels, and the number of stacked Transformer blocks. For instance, the hidden dimension dh in Eq. 3 is set to 100 for MiC-S and 200 for MiC-M. Additionally, MiC-S implements more aggressive spatio-temporal downsampling (e.g., larger temporal strides) than MiC-M to further minimize computational overhead. To ensure fairness, all foundation model teachers and the MiC-M student share a consistent dh = 200. Distillation baselines (FitNets and Logit-std) are implemented using identical teacher-student configurations as DLink. All reported metrics are averaged across five independent runs with different random seeds.
Ablation Study and Analysis
We conduct a comprehensive ablation study on the FACED dataset to evaluate the structural components, routing strategies, and saliency metrics of DLink (Table 3). Structural and Supervision Components. Removing either the CNN or Transformer modules from MiC-S/M leads to significant performance degradation, confirming the necessity of our hybrid spatio-temporal design. Crucially, the exclusion of Spectral Saliency Supervision (Lpsd ) causes a clear drop in accuracy. The performance loss without this term proves that frequency-domain alignment is vital for regularizing the Router to identify task-relevant teacher expertise. Routing Strategies and Saliency Metrics. Our dynamic routing consistently outperforms static strategies like using the teacher’s last layer (Fixed Last) or an unweighted average (Fixed Avg). This confirms that teacher knowledge importance is sample-dependent and varies across the representational hierarchy. Finally, we evaluate different metrics for the Router’s policy discovery, including Mean Power (average squared amplitude) and Max Amplitude. Our PSD-based metric outperforms these alternatives, demonstrating that Power Spectral Density serves as a more precise proxy for physiological saliency in EEG signals, enabling the student to more
0.8
0.6
0.4 High Freq
0.0
0.1
0.2
0.3
0.4
0.5
Normalized Frequency (0→DC, 0.5→Nyquist Limit)
(a)
4.5
Feature Specturm (Before Compression)
Visualization of Features
Log Magnitude(dB)
Layer 13 12 11 10 9 8 7 6 5 4 3 2 1 0 0
(b)
Figure 6: Feature visualization. (a) t-SNE illustrates preserved class separability. (b) Spectral analysis shows reduced high-frequency artifacts, indicating anti-aliasing behavior.
inherit dominant knowledge from FM teachers.
4.6
Interpretability and Visualization
Interpretability of the Router. The Router autonomously selects teacher layers for distillation. To validate the rationality of these selections, we performed a layer-wise linear probing analysis on the teacher model (see Fig.5). As shown in the probing results, Layer 11 yields the highest linear accuracy, identifying it as the most representative feature space for the task. During training, the Router’s policy evolution and the Student’s best-matching layer both converge to Layer 11. This synchronization suggests that the Router identifies the optimal knowledge source without manual intervention, bridging the representational gap between the largescale teacher and the compact student. Feature Visualization. Fig. 6 provides complementary evidence for the effectiveness of spectral distillation. Fig. 6 (a) shows t-SNE visualizations of features before the final classifier on Mumtaz2016, where the proposed compressor preserves clear class separation comparable to a full-dimensional MLP, indicating that discriminative structure is maintained after compression. Fig. 6 (b) further analyzes the feature spectra. Before compression, the baseline exhibits higher energy in the low-frequency region, while DLink allocates more energy to higher frequencies. After compression, however, the low-frequency spectra of the two methods largely overlap, whereas DLink shows a clearly reduced high-frequency tail compared to the baseline. These results indicate that spectral distillation preserves discriminative structure.
5
Conclusion
In this paper, we propose DLink, a knowledge distillation framework that compresses EEG foundation models into de-
ployable architectures for resource-constrained BCI systems. Unlike conventional distillation based on static last-layer supervision, DLink introduces a dynamic Router to adaptively aggregate task-salient intermediate representations, whose learned routing policy aligns well with independent layer-wise probing results. Combined with the Mimic-thenCompress design of the MiC student and frequency-domain alignment as a regularizer during compression, DLink enables aggressive spatio-temporal reduction while preserving discriminative structure, as supported by visualization and ablation analyses. Overall, our results demonstrate the importance of principled layer selection and signal-aware regularization for effective EEG FM distillation. Extensive experiments further show that DLink allows students to outperform lightweight baselines and approach fully fine-tuned foundation models with substantially reduced computational cost.
References [Chen et al., 2023] Jingjing Chen, Xiaobin Wang, Chen Huang, Xin Hu, Xinke Shen, and Dan Zhang. A large finer-grained affective computing EEG dataset. Scientific Data, 10(1):740, oct 2023. [Ding et al., 2023] Yi Ding, Neethu Robinson, Su Zhang, Qiuhao Zeng, and Cuntai Guan. TSception: Capturing temporal dynamics and spatial asymmetry from EEG for emotion recognition. IEEE Transactions on Affective Computing, 14(3):2238–2250, 2023. [Ding et al., 2024] Yi Ding, Chengxuan Tong, Shuailei Zhang, Muyun Jiang, Yong Li, Kevin Lim Jun Liang, and Cuntai Guan. EmT: A novel transformer for generalized cross-subject EEG emotion recognition, June 2024. [Ding et al., 2025] Yi Ding, Yong Li, Hao Sun, Rui Liu, Chengxuan Tong, Chenyu Liu, Xinliang Zhou, and Cuntai Guan. EEG-Deformer: A dense convolutional transformer for brain-computer interfaces. IEEE Journal of Biomedical and Health Informatics, 29(3):1909–1918, 2025. [Dubois et al., 2020] Yann Dubois, Douwe Kiela, David J. Schwab, and Ramakrishna Vedantam. Learning optimal representations with the decodable information bottleneck. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc. [Geirnaert et al., 2022] Simon Geirnaert, Tom Francart, and Alexander Bertrand. Time-adaptive unsupervised auditory attention decoding using eeg-based stimulus reconstruction. IEEE Journal of Biomedical and Health Informatics, 26(8):3767–3778, 2022. [Goldberger et al., 2000] Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. PhysioBank, PhysioToolkit, and PhysioNet: Components of a New Research Resource for Complex Physiologic Signals. 101(23), 2000. [Gou et al., 2021] Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A
survey. Int. J. Comput. Vision, 129(6):1789–1819, June 2021. [Jia et al., 2025] Zhihao Jia, Meiyan Xu, Jingyuan Wang, Ziyu Jia, Yong Li, Xinliang Zhou, Chenyu Liu, Junfeng Yao, and Yi Ding. Sera: Separated coarse-to-fine representation alignment for cross-subject eeg-based emotion recognition. In Proceedings of the 33rd ACM International Conference on Multimedia, MM ’25, page 5509–5518, New York, NY, USA, 2025. Association for Computing Machinery. [Jiang et al., 2024] Wei-Bang Jiang, Liming Zhao, and Baoliang Lu. Large brain model for learning generic representations with tremendous eeg data in bci. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Representation Learning, volume 2024, pages 16405–16426, 2024. [Kan et al., 2025] Siyuan Kan, Huanyu Wu, Zhenyao Cui, Fan Huang, Xiaolong Xu, and Dongrui Wu. CMCRD: cross-modal contrastive representation distillation for emotion recognition. CoRR, abs/2504.09221, 2025. [Ko et al., 2024] Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. DistiLLM: Towards streamlined distillation for large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 24872–24895. PMLR, 21–27 Jul 2024. [Lawhern et al., 2018] Vernon J Lawhern, Amelia J Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. Journal of Neural Engineering, 15(5):056013, jul 2018. [Li et al., 2022] Rui Li, Yiting Wang, Wei-Long Zheng, and Bao-Liang Lu. A multi-view spectral-spatial-temporal masked autoencoder for decoding emotions with selfsupervised learning. In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, page 6–14, New York, NY, USA, 2022. Association for Computing Machinery. [Liang et al., 2023] Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: task-aware layer-wise distillation for language model compression. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023. [Liu et al., 2023] Yucheng Liu, Ziyu Jia, and Haichao Wang. EmotionKD: A cross-modal knowledge distillation framework for emotion recognition based on physiological signals. In Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, page 6122–6131, New York, NY, USA, 2023. Association for Computing Machinery. [Ma et al., 2022] Jun Ma, Banghua Yang, Wenzheng Qiu, Yunzhe Li, Shouwei Gao, and Xinxing Xia. A large EEG
dataset for studying cross-session variability in motor imagery brain-computer interface. Scientific Data, 9(1):531, September 2022. [Mumtaz, 2016] Wajid Mumtaz. MDD patients and healthy controls EEG data (new). 11 2016. [Pan et al., 2023] Bei Pan, Kaoru Hirota, Zhiyang Jia, and Yaping Dai. A review of multimodal emotion recognition from datasets, preprocessing, features, and fusion methods. Neurocomputing, 561:126866, 2023. [Park et al., 2021] Dae Young Park, Moon-Hyun Cha, changwook jeong, Daesin Kim, and Bohyung Han. Learning student-friendly teacher networks for knowledge distillation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 13292–13303. Curran Associates, Inc., 2021. [Ramtoula et al., 2025] Benjamin Ramtoula, Pierre-Yves Lajoie, Paul Newman, and Daniele De Martini. Fantastic features and where to find them: A probing method to combine features from multiple foundation models. In NeurIPS, 2025. [Ribeiro and Schön, 2021] Antônio H. Ribeiro and Thomas B. Schön. How convolutional neural networks deal with aliasing. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2755–2759, 2021. [Romero et al., 2015] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. [Schalk et al., 2004] G. Schalk, D.J. McFarland, T. Hinterberger, N. Birbaumer, and J.R. Wolpaw. Bci2000: a general-purpose brain-computer interface (bci) system. IEEE Transactions on Biomedical Engineering, 51(6):1034–1043, 2004. [Shannon, 1949] C.E. Shannon. Communication in the presence of noise. Proceedings of the IRE, 37(1):10–21, 1949. [Song et al., 2023] Yonghao Song, Qingqing Zheng, Bingchuan Liu, and Xiaorong Gao. EEG conformer: Convolutional transformer for EEG decoding and visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:710–719, 2023. [Sun et al., 2024] Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15731–15740, 2024. [Wang et al., 2024] Guangyu Wang, Wenchao Liu, Yuhong He, Cong Xu, Lin Ma, and Haifeng Li. EEGPT: Pretrained transformer for universal and reliable representation of EEG signals. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[Wang et al., 2025a] Jiquan Wang, Sha Zhao, Zhiling Luo, Yangxuan Zhou, Haiteng Jiang, Shijian Li, Tao Li, and Gang Pan. CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Representation Learning, volume 2025, pages 75310– 75346, 2025. [Wang et al., 2025b] Xujia Wang, Xuhui Liu, Xi Liu, Qian Si, Zhaoliang Xu, Yang Li, and Xiantong Zhen. EEGDINO: Learning EEG Foundation Models via Hierarchical Self-Distillation . In proceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2025, volume LNCS 15960. Springer Nature Switzerland, September 2025. [Wu et al., 2025] Jiamin Wu, Zichen Ren, Junyu Wang, Pengyu Zhu, Yonghao Song, Mianxin Liu, Qihao Zheng, Lei Bai, Wanli Ouyang, and Chunfeng Song. Adabrainbench: Benchmarking brain foundation models for braincomputer interface applications. CoRR, abs/2507.09882, 2025. [Zhang et al., 2024] Yuan Zhang, Tao Huang, Jiaming Liu, Tao Jiang, Kuan Cheng, and Shanghang Zhang. FreeKD: Knowledge distillation via semantic frequency prompt. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15931– 15940, June 2024. [Zhang, 2019] Richard Zhang. Making convolutional networks Shift-Invariant again. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7324–7334. PMLR, 09–15 Jun 2019. [Zhou et al., 2025] Xinliang Zhou, Chenyu Liu, Zhisheng Chen, Kun Wang, Yi Ding, Ziyu Jia, and Qingsong Wen. Brain foundation models: A survey on advancements in neural signal processing and brain discovery. IEEE Signal Processing Magazine, 42(5):22–35, 2025.