Research Article
Bouke et al., 2026.
Multi-Level Distributional Entropy for Explainable Network Intrusion Detection Mohamed Aly Bouke
#1,*,
Md Shohel Sayeed
#1,
Swee-Huay Heng 2,3 Othman #
# 1,
Azizol Abdullah
# 2,
Mohamed
1 Centre for Intelligent Cloud Computing, CoE for Advanced Cloud,
Faculty of Information Science and Technology, Multimedia University, Jalan Ayer Keroh Lama, Bukit Beruang, 75450, Melaka, Malaysia 2 Department of Communication Technology and Networking, Faculty of Computer Science and Information Technology, Universiti Putra Malaysia, Serdang 43400, Malaysia 3 Laboratory of Computational Science and Mathematical Physics, Institute for Mathematical Research, Universiti Putra Malaysia, Serdang, Malaysia *[email protected], [email protected]
arXiv:2606.29797v1 [cs.CR] 29 Jun 2026
Research Article, June 30, 2026
Abstract Machine learning network intrusion detection systems (IDS) rely on aggregate flow statistics that discard distributional structure, while established entropy measures require raw packet sequences unavailable in pre-aggregated flow datasets. We propose Multi-Level Distributional Entropy (MDE), an analytical framework that derives interpretable entropy features directly from flow-level summary statistics at three levels: within-flow Gaussian differential entropy, crossdirectional Jensen-Shannon divergence (JSD), and Transmission Control Protocol (TCP) flag-pattern Shannon entropy, without raw packet access or training data. Across four benchmarks (NSL-KDD, CICIDS-2017, CICIDS-2018, UNSW-NB15) under a leakage-free fold-local pipeline, entropy-only features achieve weighted F1 of 0.708–0.989, matching conventional features without degrading performance. Full operational metric reporting then exposes failure modes that aggregate F1 conceals. On CICIDS-2018, F1=0.74 hides a detection rate (DR) of 0.48, and on held-out attack families F1 exceeds 0.998 while DR falls to zero. Under temporal shift, a pseudo-live replay of 703K flows reveals a threshold-ranking divergence in which score ranking is preserved (AUC=0.87) but fixed thresholds collapse (DR=0.082) and recalibration offers no recovery. SHapley Additive exPlanations (SHAP) fold-stability analysis (Spearman ρ = 0.80–0.95) confirms that entropy attributions are reproducible and domain-coherent across heterogeneous environments. Keywords: Intrusion detection systems, Entropy-based feature engineering, Explainable AI, SHAP, Network traffic analysis.
(SampEn) [5] requires ordered sequences of raw packet measurements that are unavailable in pre-aggregated flow datasets, The proliferation of networked infrastructure across enterprise, the dominant format in the IDS benchmark literature [6], [7], [8]. cloud, and embedded environments has made automated intru- No prior work derives entropy analytically from the summary sion detection an operational necessity. Machine learning-based statistics that flow records already contain. IDS have matured substantially over the past decade, with en- Second, interpretability remains limited. High-performing ensemble classifiers, particularly gradient-boosted trees and ran- semble classifiers are opaque, and in operational security endom forests, consistently achieving strong detection accuracy vironments analysts must audit model decisions before acting on standard benchmarks [1], [2]. Despite this progress, three on them; in regulated contexts this opacity creates compliance gaps in current IDS practice constrain broader applicability. obstacles [9], [10]. SHAP provides a theoretically grounded
1
Introduction
First, standard IDS pipelines operate on flow-level feature vectors: packet counts, byte volumes, connection duration, and statistical summaries extracted by tools such as CICFlowMeter. These aggregate descriptors discard the distributional and sequential structure of traffic that is known to differ systematically between benign sessions and attacks [3]. Information-theoretic measures quantify precisely this kind of structural difference, yet their integration into supervised IDS pipelines has been limited by a practical barrier. Classical entropy computation such as approximate entropy (ApEn) [4] and sample entropy
attribution framework [11], [12], but its application to entropyenriched IDS pipelines has not been studied, leaving open whether entropy features receive domain-coherent SHAP explanations or produce consistent attributions across environments. Third, entropy-based detectors are typically evaluated on a single dataset, so the cross-dataset transferability of entropy signatures and the factors governing it remain unstudied. This paper addresses these gaps through the MDE framework. The statistics already present in flow records (means, standard
Page 1 of 20
Research Article
Bouke et al., 2026.
deviations, minima, maxima, and packet counts) implicitly characterize the underlying distributions of packet sizes and interarrival times; MDE computes entropy analytically from those statistics, eliminating the need for raw packet access while producing features grounded in information theory and directly interpretable via SHAP. MDE differs from three categories of related work. Conventional flow statistics (byte counts, packet rates, duration) capture aggregate magnitudes but not distributional structure, and are susceptible to the labeling artifacts documented by Engelen et al. [13]. Statistical feature engineering approaches (e.g. skewness, kurtosis, higher-order moments) are empirically motivated but lack a principled basis for feature selection; MDE features are grounded in information theory with analytic definitions and known ranges (Propositions 1 and 2). Representation learning methods (convolutional neural networks (CNNs), long shortterm memory networks (LSTMs), autoencoders) extract features through training and produce non-interpretable representations; MDE features require no training data, apply to any flow-level schema, and are natively interpretable via SHAP. The main contributions of this paper are as follows: 1. We propose MDE, a method that constructs 7–12 interpretable entropy features analytically from pre-aggregated network flow statistics at three complementary levels, within-flow Gaussian differential entropy (L1), crossdirectional JSD (L2), and flag-pattern Shannon entropy (L3), requiring no raw packet access or training data for feature construction. 2. We develop a leakage-free, fold-local protocol that reports the full operational metric suite (DR, false alarm rate (FAR), Matthews Correlation Coefficient (MCC), and precision-recall AUC (PR-AUC) alongside F1) and applies it across cross-validation, temporal-split, pseudo-live replay, cross-dataset transfer, and unseen-attack-family settings, designed to surface failure modes that aggregate scores conceal. 3. We analyze SHAP attributions for MDE-augmented classifiers across five cross-validation folds, quantifying the rank-stability and domain-coherence of analytical differential entropy (ADE) and JSD attributions across structurally distinct environments. The remainder is organized as follows. Section 2 reviews related work. Section 3 covers entropy and SHAP foundations. Section 4 presents the MDE framework. Section 5 describes the experimental setup. Section 6 reports the results. Sections 7 and 8 discuss and conclude.
2
Related Work
2.1
ML-Based IDS: Recent Advances
ML-based IDS has evolved from early shallow classifiers through gradient-boosted ensembles to transformer, graph neural network (GNN), and federated architectures [1], [14], [15]. Gradient-boosted trees and random forests remain consistently dominant on tabular flow data [14], while deep learning adds value in sequential and raw-packet settings. Transformer archi-
tectures have been applied to flow-level sequences [16], and a recent comprehensive survey covering attention-based and large language model approaches [17] identifies cross-dataset generalization as the principal unsolved challenge. GNN-based approaches model network topology explicitly and show promise for lateral-movement and Advanced Persistent Threat (APT) detection [18], though they require graph construction infrastructure absent in standard flow deployments. Federated IDS [19] address cross-organizational data-sharing constraints, achieving near-centralized accuracy while preserving local privacy, which is an increasingly important practical requirement. Despite these architectural advances, the literature consistently relies on the same families of aggregate flow statistics (packet counts, byte volumes, flow duration) without principled informationtheoretic enrichment [2], [20]. Tama and Rhee [21] and Faker and Dogdu [22] establish strong baselines on NSL-KDD with XGBoost and deep learning respectively, but both operate on entirely conventional feature sets. The recurring limitation is therefore one of representation rather than architecture. Effort has concentrated on the classifier, while the input feature space has remained aggregate and blind to the distributional structure of traffic. Beyond representation, the reliability of IDS evaluation is itself contested. Engelen et al. [13] audited CICIDS-2017 and found labeling artifacts making some attack categories trivially separable by a single feature (e.g. Flow Duration= 0). Ring et al. [2] surveyed 34 IDS datasets and concluded that most published accuracy figures are inflated by dataset-specific artifacts. Sarhan et al. [23] showed that cross-dataset transfer typically collapses to near-chance performance. Recent work on standardized dataset evaluation frameworks [24] proposes MITRE ATT&CK-aligned metrics for assessing dataset relevance to real threat scenarios, reinforcing that single-dataset evaluations are insufficient for claiming generalization. Taken together, these findings indicate that single-dataset, within-distribution cross-validation (CV) evaluations reported through aggregate scores systematically overstate operational performance, yet such protocols remain the field norm.
2.2
Entropy-Based Feature Engineering
Information-theoretic measures have a foundational history in network anomaly detection. Wagner and Plattner [25] showed that Shannon entropy of packet-size and source-address distributions detects worm propagation and Denial-of-Service (DoS) floods, establishing that entropy captures the structural regularity of benign traffic more robustly than absolute counts. Nychis et al. [26] confirmed empirically that header-field entropy is an effective anomaly indicator across multiple operational networks. Xu et al. [27] showed that flooding attacks collapse destination-port entropy while benign traffic maintains it across all dimensions. Recent work continues to advance this direction: Kenyon [28] characterizes payload entropy profiles across flow types, confirming that entropy measurements reliably distinguish attack and benign flows even without payload decryption; Yu et al. [29] propose Rényi-entropy-driven anomaly detection with dynamic thresholding, achieving lower false-alarm rates than fixed-threshold Shannon-entropy detectors. Sequential complexity measures such as ApEn [4] and SampEn [5] require ordered packet sequences that are incompatible with
Page 2 of 20
Research Article
Bouke et al., 2026.
pre-aggregated CICFlowMeter output [7], [23]. A gap therefore persists between the well-documented discriminative value of entropy and the pre-aggregated flow formats that dominate practical IDS pipelines, for which no existing estimator derives entropy from the summary statistics the records already contain.
2.3
Explainability in IDS
Explainability has become an operational requirement for IDS in high-stakes and regulated environments. Mahbooba et al. [30] showed that rule-based explanations raise analyst alertvalidation rates from 61% to 94%. Patil et al. [31] demonstrated that SHAP attribution improves analyst trust in cloud-based IDS. A recent systematic review [32] covering explainable AI (XAI) methods applied to IDS (2020–2024) concludes that TreeSHAP is the dominant explanation approach for ensemble classifiers in industrial deployments, valued for its exact computation and consistency guarantees [12]. Bouke et al. [9], [10], [33] have applied SHAP to spam detection and IDS, confirming that treeensemble attributions are stable across folds and align with domain-expert intuition. These studies, however, attribute exclusively over conventional flow features; whether informationtheoretic features receive coherent attributions, and whether those attributions remain consistent across heterogeneous environments, has not been examined.
2.4
ous random variable X ∼ N (µ, σ 2 ), the differential entropy is h(X) = 21 ln(2πe σ 2 ), where e ≈ 2.718 is Euler’s number, µ is the mean, and σ 2 is the variance. h(X) is a monotone increasing function of σ , and is independent of µ. Definition 3 (Jensen-Shannon divergence [36]). For distributions P and Q with mixture M = 21 (P + Q), the Jensen-Shannon divergence is JSD(P∥Q) = 12 KL(P∥M) + 1 2 KL(Q∥M) ∈ [0, ln 2], where KL(P∥Q) is the Kullback-Leibler divergence. JSD is symmetric and bounded, making it a wellsuited measure of distributional asymmetry between forward and backward traffic.
3.2
SHAP [11] assigns each feature i a value φi satisfying fˆ(x) = p φ0 + ∑i=1 φi , where x ∈ R p is an input instance with p features, ˆf (x) is the model’s scalar output for that instance, φ0 = E[ fˆ] is the expected model output over the training set (the baseline prediction), and φi is the Shapley value from cooperative game theory, measuring the average marginal contribution of feature i across all possible feature subsets. TreeSHAP [12] computes exact Shapley values for tree ensembles in polynomial time, enabling efficient attribution across large feature sets.
4
Synthesis and Positioning
The threads above converge on a single under-addressed need. Architectural innovation in ML-based IDS has outpaced innovation in feature representation, which remains aggregate and structure-agnostic; entropy captures the distributional structure that aggregate statistics discard, but its established estimators are incompatible with the pre-aggregated datasets that dominate the field; explainability is now an operational requirement, yet entropy-derived features have never been subjected to attribution analysis; and prevailing single-dataset, aggregate-metric protocols overstate generalization.
SHAP Attribution
The MDE Framework
MDE is defined through four sub-sections: the design rationale (Section 4.1), theoretical propositions (Section 4.2), the three analytical entropy levels (L1–L3), and the composite score and dataset adaptation steps.
4.1
Behavioral Representation
MDE is a compact analytical feature construction method, not a feature selection method. Feature selection identifies which features within an existing set are most discriminative and discards the rest; the underlying feature pool is unchanged. MDE does not rank, filter, or reweight any existing flow feature. Instead, it constructs a small set of new features (entropy and divergence values) that are not present in any standard flow record and that did not exist before the transformation is applied. The inputs are the summary statistics already present in the record (means, standard deviations, packet counts, flag counts); the outputs are information-theoretic quantities grounded in a behavioral hypothesis about the structural difference between attack and benign traffic.
The present work is positioned precisely at this intersection. It derives entropy analytically from the summary statistics already present in flow records, removing the raw-packet barrier while preserving the distributional sensitivity that motivates entropy. Its closed-form features are natively interpretable, enabling a SHAP fold-stability analysis of entropy attributions. It is then evaluated under a leakage-free, multi-dataset protocol with full operational metrics that exposes the failure modes aggregate single-dataset scores conceal. The following sections formalize this framework (Section 4) and evaluate it (Section 6). MDE is also distinct from general-purpose feature transformation approaches such as principal component analysis or 3 Background kernel mappings. Those methods transform feature geometry without grounding the transformation in a domain theory. The MDE framework is grounded in three information-theoretic Each MDE feature corresponds to a specific, formalized behavconstructs and one attribution framework. This section defines ioral claim: L1 captures within-flow distributional complexity each in the precise form used throughout the paper. (Proposition 1); L2 captures directional asymmetry between source-to-destination and destination-to-source traffic (Propo3.1 Entropy Foundations sition 2); L3 captures protocol-flag diversity. These are closedDefinition 1 (Shannon entropy [34]). For a discrete random form information-theoretic expressions, not learned represenvariable X with probability mass function p, the Shannon en- tations, and their expected behavior under attack and benign tropy is H(X) = − ∑x p(x) log p(x), where log denotes the nat- conditions is analytically derivable from the underlying distriural logarithm throughout this paper (units: nats); H(X) ≥ 0 butional assumptions. with equality only when X is deterministic. Definition 2 (Gaussian differential entropy [35]). For a continuPage 3 of 20
Research Article
4.1.1
Bouke et al., 2026.
Feature Set Size and Behavioral Coverage
The feature count follows directly from the three-level hierarchy applied to whichever directional statistics are available in each dataset schema. For each available direction (forward, backward) and signal type (packet size, inter-arrival time), one ADE value is computed wherever the required mean and standard deviation are present. One JSD value is computed for each directional pair whose marginal distributions can be parameterised from available statistics. One flag-entropy value is computed where per-flow flag counts are recorded. This yields 7 features on schemas with minimal directional statistics (NSL-KDD) and up to 12 on richer CICFlowMeter-format datasets. The count is not tuned to optimize any classification metric. It reflects the three behavioral dimensions (distributional complexity, directional asymmetry, and protocol irregularity) with minimal redundancy. Adding further features within the same hierarchy would introduce correlated quantities computed from the same input statistics. 4.1.2
Analytical Properties of MDE Features
The compact analytical design provides four concrete properties. First, portability: the formulas apply to any flow record containing means, standard deviations, and packet counts, without retraining or dataset-specific engineering, enabling consistent application across NSL-KDD, CICIDS-2017/2018, and UNSWNB15 under a single analytical definition. Second, interpretability: each feature has a closed-form expression and a direct theoretical connection to traffic behavior, so SHAP attributions reflect genuinely domain-relevant quantities rather than opaque numerical artifacts. Third, analytical tractability: Propositions 1 and 2 provide closed-form bounds that predict when the features will and will not be discriminative, enabling a priori reasoning about expected behavior before any model is trained. Fourth, low inference overhead: computing 7–12 entropy values per flow adds negligible cost over conventional feature extraction. 4.1.3
Predictive Scope and Contribution Boundaries
MDE does not claim predictive uplift over conventional flow statistics. As Tables 2 and 3 confirm, combined and conventional conditions achieve statistically indistinguishable withindistribution F1 on all tested datasets. The contribution is one of representation, providing compact, theoretically grounded entropy features with schema-independent construction, native SHAP interpretability, and full operational metric reporting, as evaluated in Section 6.
4.2
Theoretical Foundations
Let D = {( fi , yi )}Ni=1 be a labeled IDS dataset where each flow fi ∈ Rd is a vector of pre-aggregated statistics and yi ∈ {0, 1} is a binary label (yi =1: attack, yi =0: benign). The MDE transformation is a deterministic map φ : Rd → Rk with k ≪ d that derives k entropy-valued features directly from the statistics already present in fi . The classifier operates on the full augmented space Rd+k (combined condition) or on Rk alone (entropy-only condition).
driven applications. Automated tools (scanners, Distributed Denial-of-Service (DDoS) agents, bots) operate under programmatic constraints. They generate packets of fixed or narrowly bounded size, fire probes at regular intervals, and produce nearunidirectional flows. Human-driven sessions (browsing, Voice over Internet Protocol (VoIP), streaming) exhibit natural variability in packet sizes, irregular inter-arrival timing, and bidirectional data exchange. This asymmetry motivates three entropybased discriminators: within-flow variance (L1), directional distributional asymmetry (L2), and protocol-flag diversity (L3). Proposition 1 (Regularity–Entropy Correspondence). Let Xa ∼ N (µa , σa2 ) and Xb ∼ N (µb , σb2 ) model the packet-size distributions of an attack flow a and a benign flow b respectively. If the attack tool imposes tighter size constraints than natural application traffic, then σa < σb , and h(Xa ) = 12 ln(2πe σa2 ) < 12 ln(2πe σb2 ) = h(Xb ).
(1)
Proof. increasing in σ 2 since 1 h(X) 2is strictly d 1 ln(2πe σ ) = 2σ 2 > 0. Therefore σa < σb ⇒ dσ 2 2 h(Xa ) < h(Xb ). Remark 1. Even when the true packet-size distribution is nonGaussian, the Gaussian differential entropy serves as a conservative lower bound. By the maximum-entropy principle, the Gaussian maximises entropy for a given variance. Hence any positive entropy gap between attack and benign flows under the Gaussian model implies a gap of at least equal magnitude under the true distribution. Remark 2 (Scope and limitations of the Gaussian approximation). The Gaussian ADE (Eq. 2) is an approximation and not a claim that network packet-size distributions are Gaussian. Real traffic exhibits heavy tails (skewness 1.5–9.7, excess kurtosis 0.9–110 across benchmarks) and may be multimodal (e.g. mixed HTTP/HTTPS flows). In such cases ADE underestimates the true differential entropy and the approximation may fail for flows whose classification relies on distributional shape rather than variance contrast. Three settings where the Gaussian assumption is expected to degrade: (1) multimodal traffic mixtures within a single flow, (2) datasets where attack and benign traffic share similar variance but differ in higher-order moments, and (3) encrypted traffic where packet payloads are padded to fixed sizes. Non-parametric alternatives (kernel-density entropy, Rényi/Tsallis entropy, histogram-based estimators) could address these cases but require design choices (bandwidth, order α) that introduce their own model assumptions; we regard this as important future work. Proposition 2 (Unidirectionality and Maximum JSD). Let P = N (µ f , σ 2f ) and Qε = N (ε µ f , ε 2 σ 2f ) for ε ∈ (0, 1], modeling backward traffic scaled to fraction ε of the forward flow. Then: (i) JSD(P∥Q1 ) = 0 (symmetric bidirectional flow), (ii) lim JSD(P∥Qε ) = ln 2 ≈ 0.693 (unidirectional ε→0+
flow).
Proof sketch. (i) is immediate since P = Q1 . For (ii), let Mε = 1 + The discriminative power of MDE rests on a fundamental gen- 2 (P + Qε ). As ε →0 , Qε concentrates near zero while P’s sup1 erative difference between automated attack tools and human- port is bounded away from zero, so Mε (x) → 2 P(x) P-almost
Page 4 of 20
Research Article
35
Empirical JSD: attack flows concentrate near ln2 0.693 nats
4.5
CICIDS-2017
A structural asymmetry between forward (source-to-destination) and backward (destination-to-source) traffic is a key characteristic of many attack types: DDoS floods are almost entirely unidirectional; port scans generate probes in one direction with negligible responses; legitimate sessions tend to be bidirectional and more balanced. MDE captures this via the JSD between the approximate forward and backward packet-length distributions: JSDpkt = JSD N (µ f , σ 2f ) ∥ N (µb , σb2 ) . (3)
UNSW-NB15 14 12 10 8 6 4 2 0
Benign Attack ln2
30 25
Density
Bouke et al., 2026.
20 15 10 5 0 0.0
0.1
0.2
0.3
0.4
0.5
jsd_pkt_len (nats)
0.6
0.7
Benign Attack ln2
0.0
0.1
0.2
0.3
0.4
0.5
jsd_pkt_sz (nats)
0.6
0.7
L2: Cross-Directional JSD
Figure 1:
Empirical JSD distributions for attack and benign flows (kernel density estimate, KDE). Attack flows in CICIDS-2017 (jsd_pkt_len) and UNSW-NB15 (jsd_pkt_sz) concentrate near the theoretical upper bound ln 2 ≈ 0.693 nats (dashed), while benign flows cluster near zero, validating Proposition 2 empirically.
Since the JSD between two Gaussians has no closed form, it is approximated by fitting a moment-matched Gaussian to the mixture M = 21 (P + Q): the mixture mean and variance 2 = 1 (σ 2 + σ 2 ) + 1 (µ − µ )2 , and are µM = 12 (µ f + µb ), σM f b f b 2 4 everywhere, giving KL(P∥Mε ) → KL(P∥ 12 P) = ln 2. Symmet- JSD is evaluated using these parameters in the KL divergence rically, near zero Mε (x) ≈ 12 Qε (x), giving KL(Qε ∥Mε ) → ln 2. formula from Definition 3. A directional balance entropy Hdir = −r log r − (1 − r) log(1 − r), where r ∈ [0, 1] is the fracSince JSD = 12 KL(P∥M) + 12 KL(Q∥M), the limit is 12 ln 2 + tion of packets (or bytes) in the forward direction, provides a 1 2 ln 2 = ln 2. complementary measure of traffic symmetry independent of the Gaussian approximation. The analytical upper bound ln 2 corresponds closely to the maximum SHAP-attributed JSD value (0.693 nats) observed for 4.6 L3: Flag-Pattern Entropy DDoS and port-scan flows in Section 6, suggesting that the TCP control flags (FIN, SYN, RST, PSH, ACK, URG) carry theoretical maximum is approached for strongly unidirectional protocol-level intent. Benign connections exhibit diverse flag attack flows. sequences over their lifetime; attacks often manipulate a narrow subset of flags (e.g. SYN-only for SYN floods, RSTFig. 1 shows that attack flows in both CICIDS-2017 and UNSWheavy for reset injections). Given the per-flow flag counts ck NB15 concentrate near ln 2, while benign flows cluster near for k ∈ {FIN,SYN,RST,PSH,ACK,URG} and total flag count zero, as expected for unidirectional attack bursts approaching C = ∑k ck , the flag-pattern entropy is: maximum directional asymmetry (Proposition 2). ck ck 4.3 Overview Hflags = − ∑ log . (4) C C k MDE computes entropy analytically from the statistical summaries already present in pre-aggregated flow records, without Here ck /C is the empirical probability of flag type k within the requiring raw packet sequences. Fig. 2 shows the end-to-end flow. Hflags = 0 when a single flag dominates (ck = C for one k); pipeline. Raw flow statistics feed three independent entropy Hflags = ln 6 at maximum when all six flag types occur equally. levels (L1–L3), whose outputs are concatenated into a feature Low flag entropy identifies flows dominated by a single flag matrix that is passed to a tree-based classifier, with SHAP protype, a strong indicator of specific attack patterns such as SYN viding post-hoc per-prediction explanations. The framework flooding or reset injection. is also dataset-aware. The specific feature columns used at each level adapt to the schema of each benchmark, while the 4.7 Composite MDE Score underlying analytical definitions stay uniform across all of them. A composite score aggregates across levels by normalizing each entropy feature to [0, 1] using training-set statistics and averag4.4 L1: Differential Entropy (ADE) ing: For a network flow with forward packet length mean µ f and h f − minT h f 1 sMDE = , (5) standard deviation σ f , we approximate the within-flow packet ∑ |F | f ∈F maxT h f − minT h f size distribution as Gaussian and compute: where F is the set of entropy features available for the given 2 1 (2) dataset schema, h f is the value of entropy feature f for the hADE fwd = 2 ln(2πe σ f ). current flow, and minT h f , maxT h f are the minimum and maxThe same is computed for the backward direction (hADE bwd ) and imum of feature f over the training set T . For temporal and for inter-arrival times. Where minimum and maximum packet hold-out experiments, normalization statistics are computed on lengths are available, a complementary uniform range entropy the historical training corpus and applied unchanged to held-out hrange = ln(max − min) is also computed. High differential flows. For cross-validation experiments, LightGBM and Ranentropy in packet sizes indicates variable, structurally complex dom Forest are invariant to monotonic feature rescaling, so the traffic; attack flows such as scanning or DDoS typically exhibit normalization choice does not affect any reported metric; s MDE low variance in packet size (all probes or all flood packets are serves as an interpretability summary in that context. This score identical), producing low ADE values. provides a single interpretable summary of the overall distributional complexity of a flow, ranging from 0 (maximally regular) to 1 (maximally entropic relative to the training distribution).
Page 5 of 20
Research Article
Bouke et al., 2026.
Figure 2: MDE four-stage pipeline. Stage 1 (Input): pre-aggregated flow statistics (means, standard deviations, packet counts, flag counts). Stage 2 (Entropy feature construction): three levels computed analytically without raw packet access: L1 Gaussian differential ADE per traffic direction, L2 cross-directional Jensen-Shannon divergence, L3 Shannon entropy over TCP flag counts. Stage 3 (Feature assembly): the 7–12 MDE entropy features are concatenated with 41–78 conventional flow features to form the combined input; entropy-only and conventional ablations use each branch independently. Stage 4 (Output): binary attack/benign prediction and per-instance SHAP attribution from a LightGBM or random forest classifier.
2018), (ii) attack taxonomy breadth (binary to 12 categories), and (iii) feature schema heterogeneity (NSL-KDD connection records to CICFlowMeter packet statistics). Class imbalance 5.1 Datasets ranges from near-balanced (NSL-KDD 43/57) to moderate Four publicly available benchmark datasets are selected to span (CICIDS-2018 72/28) and heavy (UNSW-NB15 93/7). This three structural dimensions: (i) traffic generation era (1999–
5
Experimental Setup
Page 6 of 20
Research Article
Bouke et al., 2026.