arXiv CSopen access

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection Lorenzo Guerra1,2 , Thomas Chapuis2 , Guillaume Duc1 , Pavlo Mozharovskyi1 , and Van-Tam Nguyen1 1

arXiv:2608.01454v1 [cs.CR] 2 Aug 2026

LTCI, Télécom Paris, Institut Polytechnique de Paris {lorenzo.guerra, guillaume.duc, pavlo.mozharovskyi, van-tam.nguyen}@telecom-paris.fr 2 Ampere Software Technology {lorenzo.guerra, thomas.chapuis}@ampere.cars

Abstract—Provenance-based intrusion detection systems (PIDS) frequently report strong performance, but the conclusions drawn from these results can be highly sensitive to benchmarking choices and evaluation protocols. We investigate this dependency by re-evaluating representative PIDS on public datasets that meet our audit, labeling, and calibration requirements. Focusing primarily on the audited DARPA TC E3 datasets, we apply a unified protocol with temporally separated test periods and validation-only checkpoint and threshold calibration, and ask which architectural claims are empirically supported. We find that alerting success and investigation utility can diverge sharply, as several systems surface attacks without providing enough process-level context to support forensic investigation. On three of the four primary datasets, a simple allowlist built from training executable names and paths matches or exceeds the selected learned baselines on key operating-point metrics, suggesting that much of their measured performance reflects lexical novelty rather than richer provenance modeling. To explain why only some datasets expose architectural differences, we measure semantic signal quality through feature completeness and field entropy. This analysis helps explain why several audited E3 datasets can expose alerting behavior without reliably separating model architectures, while Theia pairs the strongest semantic signal quality with the clearest improvements in ranking and node-level recovery by our reference model. These results show that architectural claims in PIDS should be interpreted together with the benchmark properties and evaluation protocol that produced them.

I. I NTRODUCTION Provenance-based intrusion detection systems (PIDS) are widely studied for host-based security [1], [2]. They represent system execution as a directed graph of causal relations between entities such as processes, files, and network connections, allowing multi-step attacks to be analyzed beyond individual log events [1], [3], [4]. Graph neural networks (GNNs) have become a common modeling choice for this setting because they can encode the structural relationships present in provenance graphs [5]–[7]. Reported detection results, however, are difficult to compare across papers. Prior studies use different benchmarks, labeling strategies, data splits, calibration rules, and reported metrics [8]–[10]. Bilot et al. [9] improve comparability by evaluating multiple PIDS baselines in a shared framework

and exposing several evaluation distortions. Such breadth is valuable, but architectural conclusions also depend on whether the benchmark preserves enough usable signal to distinguish richer models from simpler alternatives. In provenance benchmarks, graph construction and collection artifacts can change the available signal, so measured performance may reflect shortcut cues or dataset artifacts rather than architectural differences [11], [12]. Evaluation protocols introduce another source of ambiguity. Alerting metrics such as Attack Detection Precision (ADP) [9] evaluate whether a system can surface at least one useful signal from an attack, which is important for triage. However, this does not address the forensic task of ranking and isolating attack-relevant entities from large volumes of benign activity [13]–[16]. We report both alerting and investigation metrics, covering ranking quality and node-level recovery. We also enforce validation-only calibration, with both checkpoints and thresholds selected from validation data, since selecting either on the test period gives the detector information that would not be available before deployment [8], [9], [17]. The resulting protocol keeps labels, artifact handling, and calibration fixed across systems, with performance reported over multiple random seeds. We apply this methodology to public provenance datasets and use the DARPA TC E3 family [18] as the primary setting. E3 is the most suitable public setting we found for this controlled comparison because it provides reproducible processlevel labels, publicly documented artifact analysis, and temporal structure compatible with validation-only calibration. We refer to the E3 subset evaluated using the REAPr labels [19] and Liu et al.’s artifact analysis and scoring exclusions [10] as audited E3. We report ATLASv2 [20] separately because its provenance graph, reconstructed from EDR telemetry, conservatively overapproximates possible dependencies [21], and its attacks are concentrated in a single period with similar payload structure, leaving no separate attack data for validation. Within this scope, we examine which architectural conclusions survive a common protocol for calibration and scoring. The audited E3 datasets retain detectable attack signal, but most provide limited evidence for comparing learned archi-

tectures or evaluating process-level investigation. On Cadets, FiveDirections, and Trace, an allowlist over executable names and paths observed during training reproduces much of the fixed-threshold performance, suggesting that lexical novelty explains a substantial share of the reported detection gains. We therefore introduce Theseus, a reference model that combines E-GraphSAGE with a Transformer autoencoder over windowed provenance graphs and produces process-level anomaly scores. We use it to test whether greater model capacity helps when richer structure is available. Theseus shows no clear advantage over simpler baselines on Cadets, FiveDirections, or Trace, reinforcing this explanation. Theia is the exception, with more complete and varied semantic fields, and there Theseus improves both anomaly ranking and node-level recovery. These results suggest that the value of additional model capacity depends less on the architecture alone than on the quality of the semantic signal available in the dataset. The paper makes the following contributions: • We present an auditable PIDS evaluation protocol that combines shared label scope, artifact handling, temporal splitting, validation-only checkpoint and threshold calibration, and multi-seed reporting with metrics for both alerting and investigation utility. • We re-evaluate representative baselines under this shared methodology and show that high precision and ADP do not necessarily imply strong node-level recovery. • We analyze whether provenance datasets support architectural comparison by measuring semantic signal quality through feature completeness and field entropy. This analysis helps explain why some datasets reward lexical novelty or structural heuristics more than richer provenance modeling. • We introduce Theseus as a reference model for testing when additional model capacity improves provenancebased detection. In our audited E3 evaluation, its clearest gains occur on Theia, which has the highest semantic signal quality.

B. Learning on Provenance Graphs Recent work on intrusion detection has used learned, semantic, and graph-derived representations to reduce manual feature engineering over large audit streams [5], [29]. Sequential log models such as DeepLog [30] and LogBERT [31] apply neural anomaly detection to system logs, but they do not preserve the causal structure needed to analyze multi-step attacks. Provenance-based detectors therefore moved toward representations derived from causal graphs. Unicorn [32] summarizes long-running provenance graphs with streaming sketches, while ProvDetector [33] selects suspicious provenance paths and embeds them for outlier detection. More recent systems move toward representation learning directly on graphs. Kairos [6] uses a GNN encoder–decoder to model temporal changes in provenance graphs and assign anomaly scores to events, Magic [34] uses masked graph representation learning, while ThreaTrace [7] and Flash [35] compute node or neighborhood representations to identify suspicious graph elements. R-CAID [36] takes a different direction by embedding root-cause information into learned provenancebased detection. These systems often report strong detection results, but their empirical claims are difficult to compare directly because they use different label scopes, temporal splits, thresholding policies, and reporting conventions [8]–[10]. C. PIDS Evaluation and Benchmark Validity Recent work has examined PIDS evaluation more directly. Abrar et al. [8] audited the reproducibility of deep-learningbased PIDS and found that incomplete code, missing documentation, unavailable data or preprocessing steps, and unclear experimental procedures limit independent verification. Bilot et al. [9] improved comparability by implementing multiple PIDS baselines in a unified framework, introducing ADP as an alerting metric, and showing that a lightweight detector, Velox, can match or exceed more complex systems across many DARPA datasets. Liu et al. [10] shifted attention to dataset quality, showing that benchmark contents, background activity, collection artifacts, and attack conspicuousness can substantially affect endpoint threat-detection results. Our work connects these views. Broad baseline comparisons show that simple PIDS models can match or exceed richer architectures under many published benchmark settings, raising the question of whether those results reflect model design or the benchmark and protocol used to measure it. Datasetquality studies show that artifacts, background structure, and label ambiguity can shape measured performance, but they do not by themselves explain when a benchmark can separate learned architectures under a shared protocol. We combine these perspectives in a controlled evaluation that asks which claims about model design remain visible once labels, artifact handling, calibration, and scoring are fixed. Compared with prior broad evaluations, our protocol builds on Liu et al.’s artifact audit [10], extends validation-based selection to checkpoints as well as thresholds, and reports both alerting behavior and investigation utility.

II. R ELATED W ORK A. Provenance Collection and Reduction PIDS represent system activity as causal graphs. Provenance frameworks such as SPADE [22], CamFlow [23], and Linux Provenance Modules [24] showed how system-level provenance records can be represented as directed graphs of entities, including processes, files, and network connections, together with their causal interactions. Later systems used these graphs directly for security analysis. SLEUTH [3] reconstructs attack scenarios through tag propagation, while Holmes [4] correlates suspicious information flows to detect advanced persistent threats. A recurring challenge is dependence explosion, where long-running processes accumulate dense and noisy causal histories. This limits the scalability of raw provenance graphs for detection and investigation, motivating provenancereduction techniques [25]–[27] and priority-driven causality analysis [28].

2

This setup lets us ask whether the benchmark exposes enough usable signal for richer modeling. Dataset content dominated by sparsity, repetition, or artifacts may leave too little signal to distinguish richer architectures from simpler alternatives, while more complete and diverse semantic attributes can make those differences observable under the same protocol. We use semantic signal quality to characterize this signal and to explain why architectural differences are visible on some datasets but not others.

TABLE I C ORE VIABILITY CRITERIA FOR THE PRIMARY REGIME .

Benchmark Audited E3 ATLASv2 E5 OpTC

III. E VALUATION P ROTOCOL D ISTORTIONS

Compatible Public audit Attack Role labels support validation Yes Yes No No

Yes Yes No Yes

Yes No Yes No

Primary Secondary Excluded Excluded

be available before deployment, which is a form of data snooping [17]. Bilot et al. [9] identify the same issue in PIDS evaluations, including thresholding and clustering techniques that depend on the full score distribution or assume that malicious outliers are present. Random splits create a related problem by mixing future behavior into training or validation data. Similar concerns about temporal bias have been observed in evaluation for intrusion detection and malware detection, where unrealistic train/test splits can hide the distribution shifts faced after deployment [40], [41]. Selecting checkpoints on the test period tunes the model to that period, while random splitting masks the temporal structure that a deployed detector must handle. Our evaluation keeps checkpoint selection and threshold calibration independent of the held-out period. Both use only pre-test validation data, preventing these choices from adapting to the evaluation period and keeping false-positive behavior comparable across systems.

While prior PIDS evaluations vary widely, several protocol choices can inflate reported performance or mask critical failure modes [9], [17]. We focus on three controls needed for architectural comparison under our study: separating alerting from investigation utility, preventing test-period information from influencing calibration, and accounting for initialization variance. The first two affect what conclusions a benchmark can support, while the third prevents stochastic training effects from being mistaken for architectural differences. A. Alerting and Investigation Metrics No single metric captures both the alerting and forensic roles of a provenance-based detector. Bilot et al. [9] introduced Attack Detection Precision (ADP), which credits a system for surfacing at least one true positive from each distinct attack scenario without requiring full recovery of the attack footprint. This matches an important SOC use case, as one reliable alert can be enough to start triage. However, a single alert does not by itself show whether the detector supports investigation. Analysts must reconstruct the malicious footprint and distinguish it from large volumes of benign activity [14]–[16], [28]. As Jiang et al. [13] argue through the notion of “Quality of Attribution” (QoA), an alert is less useful if it leaves analysts to manually recover the relevant nodes from a massive graph. ADP captures whether a system surfaces at least one signal from an attack, but not how well it ranks or recovers the remaining attack-relevant entities. We therefore pair ADP with average precision (AP), area under the receiver operating characteristic curve (AUROC), and P MCC. We use standard average precision, computed as n (Rn − Rn−1 )Pn , where Pn and Rn denote precision and recall at successive score thresholds. Because labeled attack processes are rare, AP remains our primary ranking metric, as it reflects precision among retrieved processes, whereas AUROC can remain high under severe imbalance even when precision at useful operating points is poor [17], [37], [38]. MCC complements both ranking metrics by evaluating the selected operating point and penalizing missed attacks and false positives [39].

C. Initialization Variance PIDS models can vary substantially across random seeds [9]. Similar variance has been observed across machinelearning benchmarks more broadly, where stochastic training choices can make single-run comparisons unreliable [42], [43]. For this reason, we treat multi-seed reporting as part of the evaluation protocol and report mean performance together with variance, so that a favorable initialization is not mistaken for an architectural effect. These distortions motivate the methodology detailed below, which establishes a strict, shared evaluation protocol before comparing model architectures. IV. M ETHODOLOGY A. Benchmark Selection and Preparation 1) Primary E3 Regime: We evaluate models on an audited subset of the DARPA TC E3 provenance datasets [18]: Cadets, FiveDirections, Theia, and Trace. These datasets were collected during the same DARPA engagement and share the Common Data Model (CDM) representation, but they span different hosts, attack strategies, and operating systems. This gives us a controlled setting in which the main sources of evaluation variation can be fixed across datasets. Our primary regime requires three conditions. The dataset must provide reproducible process-level ground truth under a shared labeling procedure, public audit support for collection artifacts and ambiguous nodes, and enough temporal

B. Temporal Leakage and Calibration A second distortion comes from weak separation between model development and final testing [9], [17]. Using final evaluation data for model selection, threshold calibration, or post-processing gives the detector information that would not

3

TABLE II ATLAS V 2 GRAPH ADAPTATION AND UUID- BASED PROCESS - LABEL AUDIT. E VENT COUNTS ARE REPORTED SEPARATELY FOR TRAINING AND TESTING . ATTACK PREVALENCE IS THE PERCENTAGE OF SCORED TEST PROCESSES LABELED AS ATTACK . M ATCH RATE IS THE PERCENTAGE OF REAP R LABEL ROWS MAPPED UNIQUELY TO RECONSTRUCTED PROCESS NODES THROUGH C ARBON B LACK PROCESS UUID S .

Dataset ATLASv2-h1 ATLASv2-h2

Train Events (k)

Test Events (k)

Test Processes

Attack Scenarios

Attack Processes

Attack Prevalence (%)

Match Rate (%)

945 492

326 186

7,712 7,413

10 6

161 30

2.09 0.40

100.0 100.0

structure to isolate the test set and support validation-only calibration. Audited E3 satisfies these requirements through REAPr labels [19] and Liu et al.’s artifact analysis [10], which together define the scoring target and identify contaminated nodes to exclude. These constraints reduce cases where label construction, artifact handling, or threshold selection explain a performance difference. Table I summarizes the resulting benchmark roles, and Appendix B-A reports the selection details for each dataset. 2) Secondary ATLASv2 Evaluation: ATLASv2 [20], [44] provides a secondary evaluation setting with public Windows endpoint telemetry, but it does not match the audited E3 setting directly. We reconstruct a process-centric provenance graph from the Carbon Black EDR telemetry, which provides stable process GUIDs together with broad coverage of process, file, and network activity. Unlike the CDM audit streams used in E3, however, the EDR telemetry records the start of a data access rather than individual read and write operations and lacks stop events for data flows. The reconstructed graph therefore conservatively overapproximates possible dependencies [21]. We use the UUID-based REAPr ATLASv2 process labels, which expose the Carbon Black process UUID for every labeled attack process and define the attack scope using Windows EDR tracing heuristics [19], [21]. All h1 and h2 attack-label rows map uniquely to reconstructed process nodes. ATLASv2 nevertheless lacks an independent validation split containing attack data, as each host contains only one engagement window with attack activity and the scenarios share substantial payload structure [20]. We therefore calibrate thresholds from benign training scores and report ATLASv2 as corroborative evidence rather than as part of the primary comparison. Table II summarizes the resulting adaptation. 3) Excluded Benchmarks: E5 and OpTC fall outside the primary protocol because they would change the evaluation target. E5 has been used in prior broad evaluations, but we are not aware of public audit support that combines artifact handling, ambiguous-node treatment, and scoring exclusions with the label target used in those evaluations [9], [13]. OpTC has stronger public audit support after the correction and labeling work of Majorczyk et al. [45], but its labels reconstruct malicious host events and network flows from OpTC-specific evidence rather than the REAPr process-level target used here [10], [19]. It also lacks an independent validation split containing attack data. We therefore keep the main comparison on audited E3 and leave corrected OpTC to a separate evaluation with its own graph construction and label

reconciliation. Appendix B-A provides the dataset-specific details. 4) E3 Artifact Handling: Within E3, we follow the artifactanalysis protocol of Liu et al. [10]. Confirmed collection errors are removed from scoring and calibration, while ordinary system noise is retained. For Cadets, we keep the wwtawwtal_ bad_neighborhood artifact in the processed graphs and model inference, but exclude its labeled nodes from threshold calibration and all reported metric computation. For Theia, we use only the final stable time slice, T3, to avoid artifacts from workload-generator reconfiguration [10]. For Trace, we use a contiguous five-day window around the attack, which keeps computation tractable while preserving benign diversity. Table III reports aggregate statistics for the primary E3 temporal graph snapshots produced by our pipeline. B. Temporal Splits and Calibration The held-out test period is never used for training, checkpoint selection, or threshold calibration, which prevents lookahead bias from contaminating the reported results [8], [17]. Within the development period, a strictly monotonic train → validate → test schedule is not always practical after artifact treatment, because it would either leave too little attack data for reliable checkpoint selection or discard useful benign history. For this reason, we draw both training and validation data from the pre-test pool, while reserving the final period only for evaluation. Validation attacks are used only for checkpoint selection, using validation AP rather than loss alone, since validation loss can continue decreasing even after detection performance has peaked. This mismatch arises because the self-supervised training objective is not perfectly aligned with the downstream anomaly detection task. Threshold calibration uses only benign validation scores, with the deployment threshold set to the maximum benign validation anomaly score. We use the historical training split defined by each temporal partition without retroactively removing nodes that are later labeled as malicious. Some E3 labels attach to persistent entities rather than to the specific time windows in which those entities are known to be malicious, so a node can appear in earlier benign history and become attack-relevant only later. Removing such nodes from training would use future ground-truth knowledge and artificially clean the benign history available to the detector. It could also make those entities appear more anomalous at test time for reasons caused

4

TABLE III S TATISTICS FOR THE PRIMARY E3 TEMPORAL GRAPH SNAPSHOTS AFTER PREPROCESSING BY OUR GRAPH PIPELINE . C OUNTS COVER CACHED TRAINING , VALIDATION , AND TEST SNAPSHOTS USED BY T HESEUS . N ODE AND EDGE COUNTS ARE REPORTED AS INSTANCES , SO AN ENTITY APPEARING IN MULTIPLE SNAPSHOTS CONTRIBUTES MULTIPLE NODE INSTANCES BUT ONE UNIQUE ENTITY. P/F/N DENOTES THE PERCENTAGE DISTRIBUTION OF PROCESS , FILE , AND NETWORK - FLOW NODE INSTANCES .

Dataset

Total Snapshots

Train Snap.

Val. Snap.

Test Snap.

Unique Entities

Node Instances

Edge Instances

Median Nodes

Median Edges

Distribution (P/F/N)

Cadets FiveDir. Trace Theia (T3)

871 948 559 316

490 362 275 99

155 215 97 38

226 371 187 179

449,282 434,771 2,376,248 581,496

600,918 4,142,873 3,217,894 1,360,734

7,793,574 13,123,562 8,040,844 10,009,954

513 5430 5636 4380

7686 11,817 7456 15,100

33% / 54% / 13% 7% / 90% / 3% 15% / 36% / 49% 8% / 66% / 26%

address, source port, destination address, and destination port are all present. Feature completeness is P f ∈F filledf C= P , (1) f ∈F totalf

by the evaluation procedure rather than by the model. We therefore keep the training data intact. C. Scoring Scope and Ground Truth Our main evaluation uses the process-level labels released in REAPr [19], together with the artifact analysis and dataset audit of Liu et al. [10]. We use these labels for all main metrics and exclude contaminated nodes from scoring. The scored attack-chain processes are identified by intersecting a forward trace from the root cause(s) with a backward trace from the terminal impact(s). Processes reached only by the forward trace are labeled as “contaminated” and ignored during scoring. These nodes may include attacker-induced anomalies, but DARPA’s attack engagement reports describe the intrusions only at a coarse procedural level [18]. Excluding them prevents the evaluation from rewarding or penalizing detections on nodes whose role is ambiguous and cannot be confidently tied to the documented attack chain. Metrics are computed only over process nodes, although files and network flows remain available during training and message passing and are masked during scoring. Including unlabeled node types in metric computation would inflate performance because those nodes are overwhelmingly benign. Section VII-E discusses the limits of this process-level scope and how finer annotations can complement process labels when action-level localization is required.

where filledf is the number of non-empty records for field f and totalf is the number of records to which that field applies. For each field f , let hf be the Shannon entropy [46] of the corpus-level token frequency distribution over non-empty values. We aggregate field-level entropy as P f ∈F hf · filledf H= P , (2) f ∈F filledf which averages entropy values in proportion to the number of populated records. Missing values are excluded from this denominator because sparsity is captured by C. Semantic signal quality combines these quantities as Q = C · H.

(3)

The product penalizes attributes that are sparse or repetitive, since missing values provide no semantic evidence and lowentropy fields provide little variation for a model to learn. We use Q to interpret the audited benchmarks, not to rank provenance datasets in general. It complements the broader audit in Section VII, which examines entity linkage, raw feature coverage, temporal continuity, and label granularity. The values for each dataset are reported in Section V-B. E. Baselines and Diagnostic Allowlist

D. Semantic Signal Quality Diagnostic

We select representative baselines that combine reproducible artifacts, strong reported performance, and a practical path to adaptation under our shared protocol without requiring a reimplementation of their core model logic. This choice is informed by recent reproducibility work on deep-learningbased PIDS [8] and supports a controlled comparison rather than an exhaustive leaderboard. The selected systems are Velox [9], a lightweight MLPbased anomaly detector; Orthrus [13], which uses a Temporal Graph Network (TGN) encoder for evolving causal structure; and Magic [34], which uses self-supervised masked graph representation learning. We adapt each baseline to our shared calibration and label definitions while keeping their officially released model logic intact.

To assess whether a benchmark exposes semantic information that richer models can learn from, we compute a diagnostic from the processed node tables based on feature completeness and field entropy. Completeness measures how often predefined semantic attributes are populated, while entropy measures token-level diversity among populated values. We use this diagnostic only to interpret cross-dataset differences in the results; it does not enter training, calibration, threshold selection, or model selection. Let F denote the set of applicable semantic fields in the processed node tables, including process path, process command line, file path, and network-flow description. The network-flow field is treated as populated when its source

5

High

E-GraphSAGE

Transformer Encoder

evaluations [9], [13]. Attributes are tokenized by field type and embedded with field-specific Word2Vec models [51] trained only on the training split. Node features are formed as weighted means of these token embeddings. The weighting emphasizes executable and path fields for process and file nodes, and destination-service fields for network-flow nodes; Section V-D ablates this choice. Because our ground truth is process-level, Theseus scores only process nodes. File and network-flow nodes remain in the graph and participate in message passing, but they do not contribute directly to the reconstruction loss. For any snapshot with at least one process node, the loss is computed over Vproc ⊂ V : X 1 ∥sg(zv ) − ẑv ∥2 , (5) L= |Vproc |

Transformer Decoder

System Logs Windowed provenance graph

Low

Ranked process anomaly scores

Fig. 1. Theseus, the paper’s reference architecture for richer modeling. System logs are converted into windowed provenance graphs, encoded with E-GraphSAGE, and scored by a Transformer autoencoder. Sections IV-F and V-D give full architectural details and ablations.

We also include a zero-parameter diagnostic allowlist that flags any process whose executable name or path was unseen during training. This allowlist is not intended as a deployable defense, since it is easily bypassed by “Living off the Land” attacks that abuse known binaries [47], but it tests whether fixed-threshold performance can be explained by simple novelty in process names and paths rather than by learned behavior over provenance structure.

v∈Vproc

where sg(·) is the stop-gradient operator and ẑv is the reconstructed embedding. At evaluation time, the anomaly score for process node v is the corresponding reconstruction error, sv = ∥zv − ẑv ∥2 . We apply a symmetric binary attention mask inside the Transformer to reduce memorization of exact within-window co-occurrences, as evaluated in Section V-D1. The deployment threshold τ follows the shared policy and is set to the maximum score observed on benign validation nodes. Fig. 1 summarizes the pipeline.

F. Theseus: A Reference Model We introduce Theseus as a controlled reference model to test whether increased architectural capacity translates into empirical gains under a unified evaluation protocol. Rather than treating it only as a competitive baseline, we use Theseus to probe the interaction between model capacity and dataset signal in provenance-based detection. The architecture extends established graph-based detection frameworks [48] by operating on windowed heterogeneous provenance graphs and jointly encoding node and edge semantics. Theseus produces continuous node-level anomaly scores, enabling direct evaluation of ranking quality and recovery against the shared process-level ground truth. In our evaluation, strong gains are consistent with a benchmark exposing signal that richer modeling can exploit, while limited separation identifies settings in which the dataset, graph abstraction, or scoring scope may constrain architectural comparison. Theseus processes 15-minute provenance-graph snapshots in two stages. First, a single-hop E-GraphSAGE encoder [49] aggregates local causal context while retaining edge attributes. Then, a Transformer autoencoder [50] models co-occurrence structure within each snapshot and reconstructs the resulting node embeddings. The E-GraphSAGE update is: hN (v) = AGG ({Wmsg · [xu ||ruv ] | u ∈ N (v)}) , zv = Wcomb · [xv ||hN (v) ].

G. Implementation and Infrastructure All experiments ran on a compute cluster with NVIDIA L40S and A100 GPUs. Theseus was allocated up to 80 GB of RAM per experiment, while baseline allocations reached 128 GB for memory-intensive models such as Orthrus. Velox and Orthrus were run through PIDSMaker [52], the multimodel implementation framework released by their authors, while Magic used its official repository [53]. Each baseline was run in an isolated software environment. We keep the published model logic intact and adapt each codebase only as needed to run under the shared protocol. These adaptations align labels, scoring, and calibration across systems. For Orthrus, we replace clustering at evaluation time with the same validation-based thresholding used for the other learned models. This change follows Bilot et al.’s concern about data snooping [9], as clustering over predicted test scores requires access to the full evaluation distribution and assumes that malicious outliers are present. In deployment, attacks may be absent, and a detector must choose its operating point without seeing the full future score distribution. We therefore treat clustering over the evaluation scores as incompatible with the shared calibration policy and evaluate Orthrus using validation-only threshold selection. When released dataset configurations are available for a baseline, we use them as the primary settings. Otherwise, we start from the official implementation and select feasible settings within the shared resource budget. To guard against results driven by a configuration mismatch under our protocol,

(4)

Here, v is the target node, N (v) is its one-hop neighborhood, xv and xu are the target and neighbor feature vectors, and ruv is the attribute vector for edge (u, v). The mean-pooling aggregation operator AGG(·) is permutation-invariant, while Wmsg and Wcomb are learned linear projections. The resulting embedding zv is passed to the Transformer autoencoder. We represent each snapshot as a directed heterogeneous graph G = (V, E) over entities representing processes, files, and network flows, following prior provenance-based IDS

6

TABLE IV C OMPARATIVE PERFORMANCE ON THE PRIMARY E3 DATASETS . L EARNED MODELS SET THE THRESHOLD TO THE MAXIMUM BENIGN VALIDATION ANOMALY SCORE . T HE ALLOWLIST IS DETERMINISTIC . Q DENOTES SEMANTIC SIGNAL QUALITY. B OLD VALUES MARK THE BEST MEAN .

AP ↑

AUROC ↑

Precision ↑

F1 ↑

MCC ↑

ADP ↑

FPR ↓

Cadets (1.74)

Allowlist Orthrus Velox Magic Theseus

— 0.254 ± 0.009 0.266 ± 0.070 0.172 ± 0.007 0.115 ± 0.018

— 0.842 ± 0.006 0.840 ± 0.050 0.830 ± 0.009 0.647 ± 0.067

0.429 ± 0.000 0.600 ± 0.548 0.883 ± 0.162 0.000 ± 0.000 0.000 ± 0.000

0.021 ± 0.000 0.005 ± 0.004 0.007 ± 0.006 0.000 ± 0.000 0.000 ± 0.000

0.058 ± 0.000 0.036 ± 0.033 0.048 ± 0.022 0.000 ± 0.000 0.000 ± 0.000

— 0.801 ± 0.102 0.870 ± 0.031 0.291 ± 0.067 0.535 ± 0.368

0.0011 ± 0.0000 0.0000 ± 0.0000 0.0001 ± 0.0001 0.0000 ± 0.0000 0.0000 ± 0.0000

FiveDir. (3.97)

Allowlist Orthrus Velox Magic Theseus

— 0.007 ± 0.001 0.004 ± 0.001 0.003 ± 0.001 0.013 ± 0.002

— 0.842 ± 0.012 0.666 ± 0.023 0.947 ± 0.003 0.884 ± 0.023

0.003 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000

0.006 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000

0.011 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000

— 0.015 ± 0.002 0.008 ± 0.003 0.007 ± 0.004 0.069 ± 0.010

0.0014 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000

Trace (2.95)

Allowlist Orthrus Velox Magic Theseus

— 0.004 ± 0.002 0.062 ± 0.005 0.005 ± 0.001 0.038 ± 0.014

— 0.888 ± 0.002 0.868 ± 0.014 0.996 ± 0.000 0.969 ± 0.005

0.057 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.100 ± 0.224

0.107 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.008 ± 0.018

0.216 ± 0.000 0.000 ± 0.000 0.000 ± 0.001 0.000 ± 0.000 0.020 ± 0.045

— 0.020 ± 0.018 0.105 ± 0.005 0.008 ± 0.002 0.245 ± 0.163

0.0003 ± 0.0000 0.0000 ± 0.0000 0.0008 ± 0.0011 0.0000 ± 0.0000 0.0000 ± 0.0000

Theia (6.72)

Allowlist Orthrus Velox Magic Theseus

— 0.003 ± 0.001 0.028 ± 0.011 0.062 ± 0.016 0.671 ± 0.067

— 0.526 ± 0.095 0.491 ± 0.003 0.978 ± 0.003 0.993 ± 0.007

0.177 ± 0.000 0.000 ± 0.000 0.141 ± 0.005 0.000 ± 0.000 0.865 ± 0.112

0.273 ± 0.000 0.000 ± 0.000 0.097 ± 0.001 0.000 ± 0.000 0.445 ± 0.213

0.324 ± 0.000 -0.001 ± 0.000 0.102 ± 0.002 0.000 ± 0.000 0.507 ± 0.190

— 0.010 ± 0.005 0.512 ± 0.029 0.085 ± 0.038 0.527 ± 0.024

0.0023 ± 0.0000 0.0004 ± 0.0001 0.0004 ± 0.0000 0.0000 ± 0.0000 0.0000 ± 0.0000

Dataset (Q) System

process-level recovery. Each learned experiment is run with five random seeds, and we report mean and standard deviation.

we also ran bounded per-dataset sensitivity sweeps around both the released and adapted baseline settings. These sweeps varied the main training, regularization, and model capacity parameters while keeping model logic, scoring target, and calibration rule fixed; they did not change the qualitative conclusions. Theseus is implemented in PyTorch [54] and PyTorch Geometric [55]. It is trained with AdamW [56] for up to 400 epochs, with checkpoint selection performed on validation data. Since ATLASv2 lacks an independent validation stage, Theseus and Velox use configurations fixed across h1 and h2. Theseus is trained for 300 epochs and Velox for 12 epochs, with the final checkpoint retained in each case. Threshold calibration is performed afterward and separately using benign training scores; ATLASv2 attack labels are not used for stopping, checkpoint selection, or threshold calibration. Appendix C-A reports end-to-end runtime, graph construction cost, and memory usage for Theseus. We publicly release the code needed to reproduce the experiments.

V. R ESULTS AND B ENCHMARK A NALYSIS Table IV reports the main E3 results under the processlevel scope. Cadets, FiveDirections, and Trace provide limited evidence for architectural comparison, whereas Theia is the only primary dataset on which Theseus clearly improves both anomaly ranking and node-level recovery. We use Theseus as a controlled probe, and its performance on Theia shows that strong recovery remains possible under the shared protocol. Across datasets, Theseus performs best where the available semantic fields are more complete and varied. This pattern is consistent with semantic signal affecting attainable performance, but differences in graph structure, workloads, attacks, and background activity prevent us from fully isolating its contribution. A. Primary E3 Results 1) Limited separation on Cadets, FiveDirections, and Trace: On Cadets and Trace, some models surface attack scenarios, as reflected by precision at the selected threshold and by ADP across thresholds, but these alerts recover too little process-level context for investigation. FiveDirections provides little usable signal near the top of the ranking or at the selected operating point. This pattern limits architectural interpretation, because much of the measurable behavior is consistent with lexical novelty or structural heuristics rather than richer semantic modeling. On Cadets, alerting and node-level recovery diverge sharply. Velox reaches high precision (0.883) and ADP (0.870) with

H. Metrics The allowlist is excluded from AP and AUROC because it produces hard binary decisions rather than continuous anomaly scores. We also omit ADP because its two-level output does not provide a threshold sweep comparable to the continuous anomaly scores produced by the learned models. For learned models, AP and AUROC summarize score ranking across thresholds, while ADP summarizes attack coverage across thresholds. At the selected operating point, precision and FPR characterize alerting behavior, while F1 and MCC measure

7

Precision

Metric value

1.0

Recall

F1

Theia / Theseus

ADP=0.527±0.024

MCC

Attack Coverage

TABLE V S EMANTIC SIGNAL QUALITY BY DATASET, COMPUTED AS FEATURE COMPLETENESS MULTIPLIED BY FIELD ENTROPY.

Theia / Velox

ADP=0.512±0.029

0.8 0.6 0.4 0.2 0.0 0.25x

0.5x

1.0x

2.0x

4.0x 0.25x

0.5x

1.0x

2.0x

4.0x

Threshold / reported threshold (max benign calibration score) Fig. 2. Threshold sensitivity on Theia for frozen Theseus and Velox models. Curves and annotated ADP values are means over five seeds, and shaded bands show one standard deviation. The horizontal axis shows the threshold relative to the reported threshold, set to the maximum benign validation score.

Dataset

Nodes Feature Field entropy Semantic (M) compl. (%) (bits) signal quality

Cadets FiveDir. Trace Theia

2.7 1.4 44.5 1.5

Field Entropy (bits)

10

near-zero FPR, but its F1 (0.007) and MCC (0.048) remain negligible, and Orthrus follows a similar pattern. On FiveDirections, all learned systems have near-zero AP and fixedthreshold metrics despite AUROC values ranging from 0.666 to 0.947. On Trace, AUROC ranges from 0.868 to 0.996, yet AP remains at or below 0.062 and fixed-threshold recovery is negligible. This divergence indicates broad pairwise score separation without sufficient concentration of the rare attack processes near the top of the ranking. Cadets, FiveDirections, and Trace therefore provide limited evidence for distinguishing architectures in terms of investigation utility under the shared protocol. 2) Lexical novelty diagnostic: The allowlist indicates that much fixed-threshold behavior reflects lexical novelty rather than learned graph structure. It leads precision, F1, and MCC on FiveDirections and MCC on Trace; on Theia, it exceeds all learned baselines except Theseus in precision and MCC. Appendix B-E shows that exact command-line novelty raises recall but introduces many benign false positives. These results show that strong operating-point performance need not imply learned behavioral modeling. 3) Theia supports a clearer architectural comparison: Theseus achieves the top AP, AUROC, precision, F1, MCC, and ADP on Theia while keeping FPR at zero. Magic also reaches an AUROC of 0.978, but its AP is only 0.062 and its thresholded recovery is zero, illustrating why AUROC must be interpreted alongside AP and operating-point metrics. These results are consistent with Theia exposing semantic signal that Theseus can exploit under the shared protocol, although not all learned architectures make effective use of it. Velox reaches a similar ADP of 0.512 under the same threshold policy, but only Theseus combines strong alerting performance with meaningful recovery of the attack footprint. 4) Threshold sensitivity on Theia: To test whether Theia’s performance gap is caused by the chosen threshold, we evaluate frozen Theseus and Velox models across a range of decision thresholds, with the complete E3 curves reported in Appendix B-D. We choose Velox for this comparison because it is the strongest learned baseline on Theia under the reported threshold. Fig. 2 shows that lowering the Theseus threshold

24.0 42.4 53.0 87.0

7.25 9.35 5.56 7.72

1.74 3.97 2.95 6.72

FiveDirections

9

Theia

8

7

Cadets

6

Trace

5 20

30

40

50

60

70

80

90

Feature Completeness (%) Fig. 3. Feature completeness and field entropy in the evaluated E3 datasets. Theia has the strongest combination of populated semantic attributes and token diversity, while Cadets, FiveDirections, and Trace are constrained by different forms of missing or shallow signal.

to approximately 0.6× the reported value increases its mean F1/MCC from 0.445/0.507 to 0.627/0.633. Velox does not reach comparable node-level recovery at any threshold. On Cadets and FiveDirections, relaxing the threshold produces little improvement. On Trace, these changes shift Velox’s precision–recall trade-off but still do not yield reliable performance. These results support the main conclusion that threshold tuning can improve some alerting metrics, but it cannot turn weak top-ranked retrieval into reliable processlevel recovery. B. Semantic Signal Quality and Dataset Separation The results on Theia motivate examining whether architectural separation is related to the semantic evidence exposed by each dataset. We use the semantic signal quality diagnostic defined in Section IV-D, which combines feature completeness and field entropy in the processed node tables. Fig. 3 visualizes this dataset-level pattern, and Table V reports the corresponding values. Its node counts are the numbers of rows in the processed process, file, and network-flow tables used to compute Q; unlike Table III, they are not node instances summed across temporal graph snapshots. Because files and network flows contribute to graph representation and message passing even though metrics are computed only over processes, we compute Q across all three node types. Table VI reports the corresponding decomposition by node type.

8

(a) Theseus

(b) Magic

(c) Velox

(d) Orthrus

Fig. 4. Anomaly-score distributions by attack type for Cadets.

(a) Theseus

(b) Magic

(c) Velox

(d) Orthrus

Fig. 5. Anomaly-score distributions by attack type for FiveDirections.

semantics. Although its populated fields are moderately informative (7.25 bits), the dataset reaches only 24.0% completeness. Its process-command vocabulary is also highly restricted, with 130 unique strings across nearly 225,000 processes and 66 tokens under the Theseus tokenizer. Our analysis of the source data shows that the first 155,000 processes already cover 90% of this vocabulary. A few atomic commands, such as sleep and top, dominate the distribution, making novelty and topology more predictive than behavioral semantics. FiveDirections lacks coverage rather than semantic depth. Its populated attributes have the highest entropy in E3 (9.35 bits), but they appear in only 42.4% of nodes. The raw DARPA logs contain the largest command-token vocabulary in the suite, yet commands are populated for only 3.1% of processes. This explains the combination of high entropy and low completeness. With coverage this sparse, richer modeling cannot recover semantic signal that is absent from the data. These dataset-level patterns should also be read alongside Liu et al.’s audit [10]. FiveDirections’ high entropy may partly reflect Windows-specific vocabulary rather than uniformly rich process semantics, while Trace’s low entropy and completeness may partly reflect execution partitioning, because execution-unit nodes make up a large share of the graph used to compute the diagnostic. Therefore, semantic signal quality characterizes the representation available to the detector, including the effects of collection and graph construction choices.

TABLE VI F EATURE COMPLETENESS AND AVERAGE FIELD ENTROPY BY NODE TYPE .

Dataset

Node type

Feature compl. (%)

Avg. field entropy (bits)

Cadets

Process File Network

49.9 13.9 100.0

1.91 10.29 5.94

FiveDirections

Process File Network

1.5 95.6 70.7

4.06 9.65 7.19

Trace

Process File Network

50.0 98.1 100.0

2.75 9.94 4.96

Theia

Process File Network

99.6 77.7 100.0

5.06 9.67 7.39

Across the audited E3 datasets, the ordering of Q is consistent with the observed architectural separation: Theseus shows little separation from the learned baselines on Cadets and Trace, limited separation on FiveDirections, and clear separation on Theia. We therefore use Q to interpret the audited E3 results, rather than as a general predictor of performance across benchmarks. Among the primary E3 datasets, Theia is the only one pairing high completeness (87.0%) with high entropy (7.72 bits); Cadets, Trace, and FiveDirections are constrained by sparsity, repetition, or inconsistent coverage. Trace demonstrates the limitation of scale without diversity, as it contains 44.5 million nodes in the density corpus, yet its completeness is moderate (53.0%) and its entropy is the lowest in the suite (5.56 bits). High volume alone does not yield useful signal when the underlying data is repetitive and uninformative. Cadets is constrained by both sparsity and shallow process

C. Anomaly Score Distributions Figs. 4–7 show how the score distributions explain the results in Table IV. On Theia, Theseus produces a distinct malicious right tail, while the baseline score distributions overlap more heavily with benign activity. This matches the main results, where Theseus is the only model that combines strong alerting performance with meaningful node-level recovery.

9

(a) Theseus

(b) Magic

(c) Velox

(d) Orthrus

Fig. 6. Anomaly-score distributions by attack type for Trace.

(a) Theseus

(b) Magic

(c) Velox

(d) Orthrus

Fig. 7. Anomaly-score distributions by attack type for Theia.

On FiveDirections and Trace, malicious samples tend to receive higher anomaly scores, but remain heavily mixed with benign scores. This can yield high AUROC because many malicious–benign pairs are ordered correctly, while AP remains low because attack processes are rare and are not concentrated sufficiently near the top of the ranking. The same score distributions therefore fail to produce a stable fixedthreshold boundary under the conservative operating policy. Cadets exhibits a different failure mode, with attack samples leaning toward the anomalous side of the distribution while remaining broadly spread and overlapping with benign nodes that also receive elevated scores. This pattern is consistent with novelty-based scoring, where unfamiliar test-time attributes raise anomaly scores for both attack nodes and unusual benign nodes. As a result, a model can achieve high ADP while still producing poor node-level recovery, as reflected by low F1 and MCC.

TABLE VII M ASK RATIO ABLATION ON T HEIA . M ETRICS ARE REPORTED UNDER THE SAME CONSERVATIVE EVALUATION POLICY AS THE MAIN RESULTS . B OLD VALUES INDICATE THE BEST MEAN . AP ↑

MCC ↑

FPR ↓

0.634 ± 0.053 0.622 ± 0.054 0.593 ± 0.094 0.646 ± 0.061 0.671 ± 0.067 0.641 ± 0.066 0.635 ± 0.051

0.604 ± 0.062 0.578 ± 0.073 0.520 ± 0.115 0.490 ± 0.215 0.507 ± 0.190 0.541 ± 0.029 0.500 ± 0.175

0.00005 ± 0.00004 0.00006 ± 0.00003 0.00011 ± 0.00016 0.00003 ± 0.00003 0.00004 ± 0.00001 0.00004 ± 0.00001 0.00004 ± 0.00001

Mask Ratio 0.00 0.10 0.20 0.30 0.40 0.50 0.60

TABLE VIII T OKEN - WEIGHTING ABLATION ON T HEIA . MCC AND FPR USE THRESHOLDS SELECTED TO MAXIMIZE VALIDATION MCC. B OLD VALUES INDICATE THE BEST MEAN .

D. Theseus Ablations We ablate three design choices in Theseus: attention masking, token weighting, and structural features. Unless stated otherwise, ablations use the same validation-based threshold policy as the main evaluation. We report AP to measure thresholdindependent ranking performance, together with operatingpoint metrics to show how each change affects the selected decision boundary. 1) Attention masking: We use a symmetric binary attention mask to reduce memorization of exact within-window co-occurrences. Table VII reports the Theia sweep. At the reported mask ratio of 0.40, test AP is highest at 0.671. MCC is highest without masking and FPR remains near zero across all settings, but we use the 0.40 mask ratio because it gives the best threshold-independent ranking performance. This choice is consistent with our use of AP to evaluate ranking separately from the chosen operating point.

Configuration

AP ↑

MCC@τval ↑

FPR@τval ↓

Standard decay Reverse files Reverse network Reverse both

0.433 ± 0.169 0.335 ± 0.121 0.671 ± 0.067 0.545 ± 0.074

0.467 ± 0.143 0.483 ± 0.124 0.684 ± 0.045 0.591 ± 0.083

0.0008 ± 0.0007 0.0008 ± 0.0005 0.0003 ± 0.0002 0.0008 ± 0.0007

2) Token weighting: Table VIII reports the token-weighting ablation on Theia. Because token weighting changes both the learned representation and the scale of reconstruction errors, MCC and FPR use thresholds selected to maximize MCC on validation labels. The reported reverse network-flow weighting yields the highest test AP and MCC in this ablation, consistent with destination services being more stable than transient source ports. 3) Structural features: We evaluated edge counts and node degrees as explicit structural features. These features can help when semantic attributes are sparse, but they can also pull the model toward volume and degree patterns when semantic fields already provide stronger evidence.

10

TABLE IX ATLAS V 2 SECONDARY RESULTS USING UUID- BASED REAP R PROCESS LABELS ( MEAN ± STD OVER 5 SEEDS ). L EARNED METHODS USE BENIGN TRAINING SCORES FOR THRESHOLD CALIBRATION BECAUSE EACH HOST HAS A SINGLE ATTACK WINDOW AND NO INDEPENDENT VALIDATION SPLIT CONTAINING ATTACK DATA . AT THE REPORTED PRECISION , Q EQUALS FIELD ENTROPY BECAUSE FEATURE COMPLETENESS ROUNDS TO 100.0%. T HE ALLOWLIST IS DETERMINISTIC .

Host (Q)

AP ↑

System

AUROC ↑

Precision ↑

F1 Score ↑

Allowlist — — 0.641 ± 0.000 0.364 ± 0.000 ATLASv2-h1 Velox 0.193 ± 0.040 0.918 ± 0.004 0.000 ± 0.000 0.000 ± 0.000 (7.29) Theseus 0.216 ± 0.026 0.911 ± 0.004 0.530 ± 0.313 0.173 ± 0.098

MCC ↑

ADP ↑

0.397 ± 0.000 0.000 ± 0.000 0.229 ± 0.129

—

FPR ↓

0.0030 ± 0.0000 0.394 ± 0.184 0.0000 ± 0.0000 0.826 ± 0.122 0.0012 ± 0.0010

Allowlist — — 0.545 ± 0.000 0.462 ± 0.000 0.465 ± 0.000 — 0.0014 ± 0.0000 ATLASv2-h2 Velox 0.299 ± 0.015 0.987 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 −0.001 ± 0.000 0.410 ± 0.047 0.0002 ± 0.0001 (5.74) Theseus 0.267 ± 0.089 0.938 ± 0.007 0.411 ± 0.390 0.265 ± 0.258 0.283 ± 0.273 0.698 ± 0.211 0.0005 ± 0.0004

0.05

Dataset

Config

AP ↑

MCC ↑

ADP ↑

Cadets

W/O W/

0.094 ± 0.019 0.115 ± 0.018

0.010 ± 0.014 0.000 ± 0.000

0.810 ± 0.128 0.535 ± 0.368

FiveDir.

W/O W/

0.002 ± 0.003 0.013 ± 0.002

0.000 ± 0.000 0.000 ± 0.000

0.011 ± 0.021 0.069 ± 0.010

Trace

W/O W/

0.025 ± 0.020 0.038 ± 0.014

0.000 ± 0.000 0.020 ± 0.045

0.048 ± 0.040 0.245 ± 0.163

Theia

W/O W/

0.671 ± 0.067 0.620 ± 0.057

0.507 ± 0.190 0.211 ± 0.192

0.527 ± 0.024 0.583 ± 0.078

AP

TABLE X S TRUCTURAL - FEATURE ABLATION . M ETRICS FOLLOW THE MAIN CONSERVATIVE THRESHOLD POLICY. B OLD VALUES INDICATE THE BEST MEAN .

+0.021

+0.011

+0.013

0.00 0.05

-0.051

Cadets

FiveDir.

Trace

Theia

Fig. 8. Difference in AP after adding structural features (∆ AP). Structural features help on less informative datasets, but reduce performance on Theia, where semantic attributes provide a stronger signal.

E. Secondary ATLASv2 Results We evaluate ATLASv2 to check whether the main pattern extends beyond the DARPA CDM pipeline. For this evaluation, we use ATLASv2’s Carbon Black telemetry to reconstruct a process-centric graph, which differs from E3 in schema and graph topology. As described in Section IV-A2, the UUIDbased REAPr process labels map directly to the reconstructed Carbon Black process nodes. Because each host contains a single attack period with similar payload structure, thresholds are calibrated on benign training data only. Table IX shows mixed architectural separation in this secondary setting. Theseus achieves higher AP on the higherentropy h1 and higher ADP on both hosts, whereas Velox achieves higher AP on h2 and higher AUROC on both hosts. At the selected thresholds, Theseus retains nonzero F1 and MCC while Velox does not, although the allowlist achieves the strongest precision, F1, and MCC on both hosts. The fact that Theseus leads in AP only on the higher-entropy host is compatible with the semantic-signal interpretation observed on E3, but two hosts do not independently establish this relationship. Given the graph reconstruction and split limitations described above, we treat ATLASv2 as corroborative rather than primary evidence.

Table X and Fig. 8 show this trade-off. Adding structural features reduces AP on Theia from 0.671 to 0.620, suggesting that graph density cues interfere with the stronger semantic signal present in that dataset. On Trace, where semantic support is weaker, the same features improve AP, F1, MCC, and ADP, with ADP increasing from 0.048 to 0.245. This supports the broader interpretation that models rely more on structural heuristics when semantic fields are incomplete or repetitive. Cadets shows a similar effect because its process vocabulary is shallow and repetitive, so the model tends to treat novel text strings as anomalous. Without structural features, this produces high ADP (0.810) but poor node-level recovery (AP 0.094, MCC 0.010). Adding structural features improves AP by reducing the influence of text-only novelty, but ADP falls to 0.535. This sensitivity reinforces the need to interpret ADP alongside AP and MCC, since Cadets can support alerting behavior driven by textual novelty without supporting reliable attack reconstruction. Combined with the allowlist diagnostic, the structuralfeature ablation suggests that much of the measurable signal in these benchmarks comes from textual and attribute fields. Structural cues help on some datasets, especially where semantic fields are weak, but the results do not support treating strong baseline performance as evidence of richer graph modeling by itself.

VI. L IMITATIONS Our conclusions are tied to the benchmarks and evaluation protocol used in this study.

11

TABLE XI E3 B ENCHMARK Q UALITY AUDIT OF R AW C OLLECTION S TREAMS . E NTITY LINKAGE TRACKS EVENT RESOLUTION INTO GRAPH ENTITIES . F EATURE COVERAGE SHOWS THE FRACTION OF UNIQUE ENTITIES WITH A NON - EMPTY ATTRIBUTE . C ONTINUITY TRACKS WITHIN - STREAM TIMESTAMP REGRESSIONS AND GAPS .

Entity Linkage

Feature Coverage (%)

Timestamp Continuity

Dataset

Events (M)

Resolved (%)

Unlinkable (%)

Proc. cmd

Proc. path

File path

Network flow

Streams

Gaps >60s (%)

Out-of-order (%)

Max gap

Cadets FiveDir. Trace Theia

41.4 261.2 813.4 106.0

88.2 65.6 41.0 41.9

11.8 34.4 59.0 58.1

99.8 3.1 100.0 99.6

0.0 0.0 0.0 99.6

13.9 95.6 98.0 77.7

100.0 70.7 100.0 100.0

10 55 204 25

0.000 0.047 0.000 0.000

0.001 6.947 0.071 0.003

0s 8.5 days 3.1 days —

structure, documented benign workload variation, and granular labels that match the forensic task. Future benchmarks should treat these as release requirements and document collection conditions, intended uses, and known limitations [57]. These benchmark properties also motivate testable architectural hypotheses. Future work should evaluate whether combining semantic and structural information improves robustness, ablate semantic attributes and structural features separately to identify the source of performance gains, validate calibration under documented workload shifts, and measure both alerting and process-level recovery.

A. Scope and Generality Our comparative claims are limited to the audited DARPA TC E3 suite, with ATLASv2 used only as secondary evidence. This gives a consistent process-label target, public artifact analysis, and validation conditions compatible with our protocol, but limits dataset breadth. Adding E5 or OpTC would require different label, graph construction, artifact handling, and calibration choices, so we leave them to separate evaluations. This scope also limits how the semantic signal quality diagnostic should be interpreted. It helps explain the observed E3 pattern, but we do not treat it as a general predictor of benchmark difficulty. Testing whether feature completeness and field entropy predict architectural separation more broadly will require additional datasets with comparable labels and audit depth. Finally, our baseline results evaluate reproducible implementations under a shared protocol, not upper bounds for each architecture. We use released configurations when available and run bounded sensitivity checks, but additional datasetspecific engineering could improve individual systems. Our conclusions therefore concern what the audited benchmarks and calibration policy support, not model optimality or adversarial robustness.

A. Entity Linkage Graph construction loses information when the evaluation schema cannot represent the entity families referenced by raw events. We use the common PIDS abstraction over processes, files, and network flows [6], [9], [13] so that Theseus and the baselines operate on the same target. Table XI reports the fraction of raw events where both subject and object UUIDs resolve under this schema. In Cadets, 11.8% of raw events are unlinkable, including 10.89% that lack an object UUID entirely. In Theia, FiveDirections, and Trace, up to 59.0% of raw events contain an object UUID that does not resolve to a process, file, or network flow. These events may be valid in the raw CDM stream, but their relationships are dropped before detectors receive the data. Benchmark releases and PIDS evaluations should therefore preserve all security-relevant entity families or document and validate the information lost when reducing raw events to an evaluation graph.

B. Operating Policy Our fixed-threshold metrics reflect one conservative deployment policy, where the threshold is set to the maximum benign validation score. This matches the strict low-falsepositive emphasis of prior PIDS evaluations [9] and keeps calibration independent of the test period. Although different operating points change precision, recall, F1, and MCC, our threshold sweeps indicate that relaxing the threshold does not substantially improve recovery when ranking quality is weak. We interpret fixed-threshold metrics as performance under this conservative operating point, while AP and AUROC measure ranking quality across thresholds and ADP measures attack coverage across thresholds.

B. Semantic Support Even when events link correctly, missing attributes limit what a detector can learn from the graph. Some attribute loss is introduced by graph abstraction, but much of it is already present in the raw collection. Table XI also reports coverage over unique raw entities before downstream graph construction. Process paths are absent in Cadets, FiveDirections, and Trace. Cadets also has only 130 unique raw process command strings across the full collection. FiveDirections has a broader command vocabulary, but command lines are populated for only 3.1% of processes. Theia records basic execution metadata much more consistently. Benchmark preparation should

VII. T OWARD H IGHER -Q UALITY B ENCHMARKS Our benchmark analysis identifies five properties needed for reliable architectural comparison: faithful graph construction, complete and diverse semantic attributes, stable temporal

12

prioritize complete command lines, absolute process and file paths, and resolved network endpoints before applying additional graph abstractions.

libc.so

read rite

w

httpd execve execve (crash)

access.log

payload_4.9

sh

C. Collection Stability and Temporal Span execve

Recording instability changes the evaluation timeline before any model is trained. Table XI summarizes timestamp continuity within each stream for the released E3 logs. Cadets is internally consistent, with negligible timestamp regressions. FiveDirections shows substantial disorder, with 6.95% withinstream timestamp regressions and forward discontinuities lasting up to 8.5 days. Trace contains forward gaps up to 3.1 days. Theia has fewer within-stream regressions, but its documented workload reconfigurations create phase changes in the benign distribution [10]. These issues matter because current datasets are temporally narrow and concentrated around attack periods. Future benchmark releases should include longer capture windows with documented workload changes and explicit records of collection failures.

ail) open (f

/etc/shadow

python connec

connect

pg_hba.conf

read

sendto

postgres

t (fail)

10.0.0.5

read

write

dump.sql

198.51.100.1

Successful Attack Chain Failed Attempts (Excluded) Benign Execution (Conflated)

Fig. 9. Illustrative example of evaluation scopes. The current process-level scope typically captures only the successful execution path (solid red). Edgelevel annotations can distinguish failed attack-induced actions (dashed orange) from unrelated benign execution (dotted green) that are currently conflated under node-level labeling.

D. Support for Robustness Claims Robustness to distribution shift requires benchmarks that document multiple benign workload conditions. This allows training and validation to cover different benign behavior without using the final test period. The final evaluation can then include both in-distribution and out-of-distribution test periods, each with benign activity and attacks, to measure whether detection performance generalizes beyond the conditions used for model design and threshold selection. Adversarial robustness has a similar dependency on benchmark quality. If semantic attributes are sparse, benign workload variation is undocumented, or collection artifacts are hidden, adaptive attacks may expose benchmark limitations rather than reveal how the detector behaves under realistic adversarial changes. Robustness evaluations should therefore establish these conditions before introducing detector-specific adaptive attacks under precise threat models [17].

A compromised process may continue to perform benign actions unrelated to the attack, while only some of its operations are directly caused by the intrusion. Those malicious operations may include failed exploit attempts, persistence, lateral movement, or reconnaissance. A detector should receive credit for surfacing these actions even when they do not belong to the strict successful attack path, while unrelated activity from the same process should not automatically be treated as malicious. Our process-level scoring scope builds on the REAPr labels [19] and Liu et al.’s audit [10]. Scoring the documented attack path while excluding contaminated nodes avoids rewarding or penalizing detections whose role cannot be established confidently from the DARPA reports. This preserves a practical process-level target without imposing false precision on causally ambiguous activity, although the resulting evaluation does not measure failed attempts or localize malicious behavior to individual actions within a process. REAPr also provides causal seeds from which the time at which a process first enters the reconstructed attack subgraph can be derived, together with edge-level annotations for Cadets, FiveDirections, and Theia, but not Trace [19]. These annotations enable complementary timing and edgelevel analyses on the covered datasets, but they do not provide a common target across the four-dataset suite. Adopting them would require a separate protocol and system-specific adaptations to produce comparable edge scores; they cannot be substituted directly for the process labels. We therefore retain process-level scoring as the common and operationally relevant target for the controlled comparison. Where the available evidence supports them, future benchmark releases could complement process labels with edge- or event-level annotations across datasets. Such annotations can

E. Label Granularity Process-level labels provide the only common scoring target across the four audited E3 datasets and the evaluated system interfaces. They are a practical and reproducible target that aligns with common endpoint detection workflows, but they cannot localize when or how malicious behavior occurs within a process. A process is an execution context whose behavior can change over time, so a single process-level label can conflate malicious actions with unrelated benign activity. Fig. 9 illustrates two limitations of the current scoring target. A process-level label can conflate attack-induced actions with unrelated benign behavior performed by the same process across time, while the definition of the strict successful attack path can exclude behavior caused by failed attack attempts. Consistent action-level annotations are therefore needed to attribute malicious behavior to particular operations and time intervals rather than to the process as a whole.

13

support timing and action-level localization, but they should represent causal ambiguity explicitly and avoid implying perfectly precise attack boundaries.

The principal risks are misinterpretation and limited potential for misuse. The results could be overgeneralized as evidence that provenance-based detection is ineffective, although our conclusions concern specific benchmarks and protocol choices. The analysis may also reveal novelty cues or benchmark properties on which some measured results depend. We mitigate these risks by explicitly limiting the scope of our conclusions, distinguishing diagnostic evidence from causal conclusions or claims about adversarial robustness, and not providing an exploit or evasion procedure. Justice. Our findings may benefit defenders and future researchers while imposing reputational costs or additional validation and corrective work on authors of evaluated systems and benchmark maintainers. To reduce unfair attribution, we evaluate every compatible system under a common protocol for ground truth, data splits, calibration, and scoring, and report limitations and uncertainty consistently. We make no claims about researcher intent or competence, attribute observed differences to methodological and benchmark conditions, and release evidence that allows others to verify, contest, or extend our findings. Respect for law and the public interest. We use and redistribute released data and derived artifacts only as permitted by the applicable licenses and release conditions. We did not identify a vulnerability in a deployed system requiring coordinated disclosure. We report dataset ambiguities and reproducibility issues transparently and, subject to the original release conditions, provide code, processed benchmark files, and checkpoints to support verification. More reliable security evaluation serves the public interest by reducing the risk that deployment decisions rely on overstated or incomparable results.

VIII. C ONCLUSION PIDS benchmarks can support architectural comparison only when their data, graph abstraction, and evaluation protocols expose the differences that models are meant to capture. Under our auditable methodology and process-level scope, Cadets and Trace still help test whether models can surface attack scenarios, while FiveDirections provides little usable detection signal under this protocol. However, Cadets, FiveDirections, and Trace do not provide stable evidence for separating architectures or identifying attack-relevant context, because the observed detector behavior is often consistent with lexical novelty, structural cues, or threshold sensitivity rather than richer provenance modeling. The experiments also show that alerting and investigation can diverge, as systems can surface attacks while identifying only a small portion of the malicious footprint. Among the audited E3 datasets, only on Theia does our reference model measurably improve both ranking and node-level recovery. On datasets with limited semantic signal, apparent architectural gains may reflect benchmark-specific cues or calibration choices rather than improved investigation utility. Establishing stronger evidence for model design will require higher-fidelity provenance data with precise ground truth, alongside metrics that independently measure both alerting and investigation. Until then, reported architectural gains must be interpreted within the benchmark conditions and evaluation protocol that produced them. IX. E THICAL C ONSIDERATIONS

R EFERENCES

We structure this discussion around the principles of the Menlo Report [58]: respect for persons, beneficence, justice, and respect for law and the public interest. The principal stakeholders are defenders and analysts who may rely on provenance-based detectors, researchers and authors of the evaluated systems, and benchmark maintainers. Respect for persons and data. This study performs a secondary analysis of publicly released cybersecurity benchmarks, primarily DARPA Transparent Computing E3, with ATLASv2 as secondary evidence. We do not recruit participants, intervene in operational systems, or collect new user data. We limit our analysis to the released telemetry needed for the evaluation, do not link it with external identity information, and do not attempt to identify individuals. Beneficence and risk. The work aims to reduce the risk of placing excessive trust in detectors whose reported performance may depend on benchmark artifacts, lexical novelty, or evaluation look-ahead. Our results clarify the evidence that the evaluated systems provide for reliable alerting and investigation, while the shared protocol makes architectural claims easier to compare and identifies limitations relevant to future benchmark design.

[1] M. Zipperle, F. Gottwalt, E. Chang, and T. Dillon, “Provenance-based intrusion detection systems: A survey,” ACM Computing Surveys, vol. 55, no. 7, pp. 1–36, 2022. [2] F. Dong, S. Li, P. Jiang, D. Li, H. Wang, L. Huang, X. Xiao, J. Chen, X. Luo, Y. Guo et al., “Are we there yet? an industrial viewpoint on provenance-based endpoint detection and response tools,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, 2023, pp. 2396–2410. [3] M. N. Hossain, S. M. Milajerdi, J. Wang, B. Eshete, R. Gjomemo, R. Sekar, S. Stoller, and V. Venkatakrishnan, “SLEUTH: Real-time attack scenario reconstruction from COTS audit data,” in 26th USENIX Security Symposium (USENIX Security 17). USENIX Association, 2017, pp. 487–504. [4] S. M. Milajerdi, R. Gjomemo, B. Eshete, R. Sekar, and V. Venkatakrishnan, “HOLMES: Real-time APT detection through correlation of suspicious information flows,” in 2019 IEEE Symposium on Security and Privacy (SP). IEEE, 2019, pp. 1137–1152. [5] T. Bilot, N. El Madhoun, K. Al Agha, and A. Zouaoui, “Graph neural networks for intrusion detection: A survey,” IEEE Access, vol. 11, pp. 49 114–49 139, 2023. [6] Z. Cheng, Q. Lv, J. Liang, Y. Wang, D. Sun, T. Pasquier, and X. Han, “KAIROS: Practical intrusion detection and investigation using wholesystem provenance,” in 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024. [7] S. Wang, Z. Wang, T. Zhou, H. Sun, X. Yin, D. Han, H. Zhang, X. Shi, and J. Yang, “THREATRACE: Detecting and tracing host-based threats in node level through provenance graph learning,” IEEE Transactions on Information Forensics and Security, vol. 17, pp. 3972–3987, 2022.

14

[25] K. H. Lee, X. Zhang, and D. Xu, “Loggc: garbage collecting audit log,” in Proceedings of the 2013 ACM SIGSAC Conference on Computer and Communications Security, 2013, pp. 1005–1016. [26] ——, “High accuracy attack provenance via binary-based execution partition,” in 20th Annual Network and Distributed System Security Symposium (NDSS), vol. 16, 2013. [27] Z. Xu, Z. Wu, Z. Li, K. Jee, J. Rhee, X. Xiao, F. Xu, H. Wang, and G. Jiang, “High fidelity data reduction for big data security dependency analyses,” in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’16. New York, NY, USA: Association for Computing Machinery, 2016, pp. 504–516. [Online]. Available: https://doi.org/10.1145/2976749.2978378 [28] Y. Liu, M. Zhang, D. Li, K. Jee, Z. Li, Z. Wu, J. Rhee, and P. Mittal, “Towards a timely causality analysis for enterprise security,” in 25th Annual Network and Distributed System Security Symposium (NDSS), 2018. [29] J. Zeng, Z. L. Chua, Y. Chen, K. Ji, Z. Liang, and J. Mao, “WATSON: Abstracting behaviors from audit logs via aggregation of contextual semantics,” in 28th Annual Network and Distributed System Security Symposium (NDSS), 2021. [30] M. Du, F. Li, G. Zheng, and V. Srikumar, “DeepLog: Anomaly detection and diagnosis from system logs through deep learning,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2017, pp. 1285–1298. [31] H. Guo, S. Yuan, and X. Wu, “Logbert: Log anomaly detection via bert,” in 2021 international joint conference on neural networks (IJCNN). IEEE, 2021, pp. 1–8. [32] X. Han, T. Pasquier, A. Bates, J. Mickens, and M. Seltzer, “Unicorn: Runtime provenance-based detector for advanced persistent threats,” in 27th Network and Distributed System Security Symposium (NDSS), 2020. [33] Q. Wang, W. U. Hassan, D. Li, K. Jee, X. Yu, K. Zou, J. Rhee, Z. Chen, W. Cheng, C. A. Gunter, and H. Chen, “You are what you do: Hunting stealthy malware via data provenance analysis,” in 27th Annual Network and Distributed System Security Symposium (NDSS), 2020. [34] Z. Jia, Y. Xiong, Y. Nan, Y. Zhang, J. Zhao, and M. Wen, “MAGIC: Detecting advanced persistent threats via masked graph representation learning,” in 33rd USENIX Security Symposium, USENIX Security 2024, 2024. [35] M. U. Rehman, H. Ahmadi, and W. U. Hassan, “Flash: A comprehensive approach to intrusion detection via provenance graph representation learning,” in 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024, pp. 3552–3570. [36] A. Goyal, G. Wang, and A. Bates, “R-caid: Embedding root cause analysis within provenance-based intrusion detection,” in 2024 IEEE Symposium on Security and Privacy (SP), 2024, pp. 3515–3532. [37] J. Davis and M. Goadrich, “The relationship between precision-recall and roc curves,” in Proceedings of the 23rd International Conference on Machine Learning, 2006, pp. 233–240. [38] T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,” PloS one, vol. 10, no. 3, p. e0118432, 2015. [39] D. Chicco and G. Jurman, “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation,” BMC genomics, vol. 21, no. 1, p. 6, 2020. [40] Z. Kan, S. McFadden, D. Arp, F. Pendlebury, R. Jordaney, J. Kinder, F. Pierazzi, and L. Cavallaro, “TESSERACT: Eliminating experimental bias in malware classification across space and time (extended version),” 2025. [Online]. Available: https://arxiv.org/abs/2402.01359 [41] G. Apruzzese, P. Laskov, and J. Schneider, “SoK: Pragmatic assessment of machine learning for network intrusion detection,” in 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). IEEE, 2023, pp. 592–614. [42] N. Reimers and I. Gurevych, “Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 338–348. [43] X. Bouthillier, P. Delaunay, M. Bronzi, A. Trofimov, B. Nichyporuk, J. Szeto, N. Mohammadi Sepahvand, E. Raff, K. Madan, V. Voleti, S. Ebrahimi Kahou, V. Michalski, T. Arbel, C. Pal, G. Varoquaux, and P. Vincent, “Accounting for variance in machine learning benchmarks,” in Proceedings of Machine Learning and Systems, vol. 3, 2021, pp. 747–769. [Online]. Available: https://proceedings.mlsys.org/paper files/ paper/2021/file/0184b0cd3cfb185989f858a1d9f5c1eb-Paper.pdf

[8] T. Abrar, A. Shamail, M. J. Iqbal, A. Ahmed, M. Abdullah, M. Shayan, F. Zaffar, T. Pasquier, D. Eyers, and A. Gehani, “On the reproducibility of provenance-based intrusion detection that uses deep learning,” in Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ser. ACM REP ’25. New York, NY, USA: Association for Computing Machinery, 2025, pp. 14–28. [Online]. Available: https://doi.org/10.1145/3736731.3746140 [9] T. Bilot, B. Jiang, Z. Li, N. El Madhoun, K. Al Agha, A. Zouaoui, and T. Pasquier, “Sometimes simpler is better: A comprehensive analysis of state-of-the-art provenance-based intrusion detection systems,” in 34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 7193– 7212. [10] J. Liu, M. A. Inam, A. Goyal, A. Riddle, K. Westfall, and A. Bates, “What we talk about when we talk about logs: Understanding the effects of dataset quality on endpoint threat detection research,” in 2025 IEEE Symposium on Security and Privacy (SP), 2025, pp. 112–129. [11] N. Michael, J. Mink, J. Liu, S. Gaur, W. U. Hassan, and A. Bates, “On the forensic validity of approximated audit logs,” in Proceedings of the 36th Annual Computer Security Applications Conference, ser. ACSAC ’20. New York, NY, USA: Association for Computing Machinery, 2020, pp. 189–202. [Online]. Available: https://doi.org/10.1145/3427228.3427272 [12] R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, no. 11, pp. 665–673, 2020. [13] B. Jiang, T. Bilot, N. El Madhoun, K. Al Agha, A. Zouaoui, S. Iqbal, X. Han, and T. Pasquier, “ORTHRUS: Achieving high quality of attribution in provenance-based intrusion detection systems,” in 34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 7173– 7192. [14] W. U. Hassan, S. Guo, D. Li, Z. Chen, K. Jee, Z. Li, and A. Bates, “NoDoze: Combatting threat alert fatigue with automated provenance triage,” in 26th Annual Network and Distributed System Security Symposium (NDSS), 2019. [15] P. Fang, P. Gao, C. Liu, E. Ayday, K. Jee, T. Wang, Y. F. Ye, Z. Liu, and X. Xiao, “Back-Propagating system dependency impact for attack investigation,” in 31st USENIX Security Symposium (USENIX Security 22). Boston, MA: USENIX Association, Aug. 2022, pp. 2461–2478. [16] S. Li, F. Dong, X. Xiao, H. Wang, F. Shao, J. Chen, Y. Guo, X. Chen, and D. Li, “NODLINK: An online system for fine-grained apt attack detection and investigation,” in 31st Annual Network and Distributed System Security Symposium (NDSS), 2024. [17] D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wressnegger, L. Cavallaro, and K. Rieck, “Dos and don’ts of machine learning in computer security,” in 31st USENIX Security Symposium (USENIX Security 22), 2022, pp. 3971–3988. [18] DARPA, “Transparent computing engagement 3 data release,” https://github.com/darpa-i2o/Transparent-Computing/blob/master/ README-E3.md, 2018, commit: e94c9f2. Accessed: December 2025. [19] J. Liu, M. A. Inam, A. Goyal, K. Westfall, A. Riddle, and A. Bates, “REAPr: Recovery every attack process,” https://bitbucket.org/sts-lab/ reapr-ground-truth, 2023, commit: e726c01. Accessed: July 2026. [20] A. Riddle, K. Westfall, and A. Bates, “ATLASv2: Atlas attack engagements, version 2,” 2023. [Online]. Available: https://arxiv.org/ abs/2401.01341 [21] J. Liu, M. A. Inam, A. Goyal, D. Greenenwald, A. Bates, and S. Chittal, “How to effectively trace provenance on windows endpoint detection & response telemetry,” in Workshop on Attack Provenance, Reasoning, and Investigation for Security in the Monitored Environment (PRISM’26), 2026. [22] A. Gehani and D. Tariq, “SPADE: Support for provenance auditing in distributed environments,” in ACM/IFIP/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing. Springer, 2012, pp. 101–120. [23] T. Pasquier, X. Han, M. Goldstein, T. Moyer, D. Eyers, M. Seltzer, and J. Bacon, “Practical whole-system provenance capture,” in Proceedings of the ACM Symposium on Cloud Computing (SoCC ’17). ACM, 2017. [24] A. Bates, D. J. Tian, K. R. Butler, and T. Moyer, “Trustworthy WholeSystem provenance for the linux kernel,” in 24th USENIX Security Symposium (USENIX Security 15). Washington, D.C.: USENIX Association, Aug. 2015, pp. 319–334.

15

processed ATLASv2 files and labels used in the secondary evaluation. The repository also provides a uv.lock file and a container recipe to improve environment reproducibility across machines and clusters. The released code identifies the upstream versions on which our ground truth and baseline adaptations are based: REAPr commit e726c01, MAGIC commit aa0b647, and PIDSMaker commit 2a73886. Our repository contains the modifications required to integrate the baselines with the shared data and evaluation pipeline, while preserving their model logic. The released artifacts contain the exact groundtruth files used in our evaluation.

[44] A. Alsaheel, Y. Nan, S. Ma, L. Yu, G. Walkup, Z. B. Celik, X. Zhang, and D. Xu, “ATLAS: A sequence-based learning approach for attack investigation,” in 30th USENIX Security Symposium (USENIX Security 21). USENIX Association, Aug. 2021, pp. 3005–3022. [45] F. Majorczyk, B. Pilastre, and F. Dijoud, “A new hope for darpa optc,” in 2025 Annual Computer Security Applications Conference Workshops (ACSAC Workshops). IEEE, 2025, pp. 551–561. [46] C. E. Shannon, “A mathematical theory of communication,” The Bell system technical journal, vol. 27, no. 3, pp. 379–423, 1948. [47] F. Barr-Smith, X. Ugarte-Pedrero, M. Graziano, R. Spolaor, and I. Martinovic, “Survivalism: Systematic analysis of Windows malware livingoff-the-land,” in 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021, pp. 1557–1574. [48] L. Guerra, T. Chapuis, G. Duc, P. Mozharovskyi, and V.-T. Nguyen, “Self-supervised learning of graph representations for network intrusion detection,” in Advances in Neural Information Processing Systems, vol. 38. Curran Associates, Inc., 2025, pp. 109 471–109 501. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2025/file/9ddb13ae9150f99298065d889f951014-Paper-Conference.pdf [49] W. W. Lo, S. Layeghy, M. Sarhan, M. Gallagher, and M. Portmann, “EGraphSAGE: A graph neural network based intrusion detection system for IoT,” in NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium, 2022, pp. 1–9. [50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017, pp. 5998–6008. [51] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013. [52] T. Bilot et al., “PIDSMaker: An ml framework for building provenancebased intrusion detection systems,” https://github.com/ubc-provenance/ PIDSMaker, 2025, commit: 2a73886. Accessed: December 2025. [53] Z. Jia et al., “MAGIC: Official implementation,” https://github.com/ FDUDSDE/MAGIC, 2024, commit: aa0b647. Accessed: December 2025. [54] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “PyTorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019. [55] M. Fey and J. E. Lenssen, “Fast graph representation learning with PyTorch Geometric,” in ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019. [Online]. Available: https://arxiv.org/abs/1903.02428 [56] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations, 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7 [57] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford, “Datasheets for datasets,” Communications of the ACM, vol. 64, no. 12, pp. 86–92, 2021. [58] D. Dittrich and E. Kenneally, “The Menlo Report: Ethical principles guiding information and communication technology research,” U.S. Department of Homeland Security, Tech. Rep., 2012. [59] E. Manzoor, S. M. Milajerdi, and L. Akoglu, “Fast memory-efficient anomaly detection in streaming heterogeneous graphs,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 1035–1044. [Online]. Available: https://doi.org/10.1145/2939672.2939783

A PPENDIX B P ROTOCOL D ETAILS AND S ECONDARY D IAGNOSTICS A. Benchmark Viability Table XII details the inclusion flags that determine dataset eligibility under the primary protocol. E3 remains the central focus, with ATLASv2 providing secondary corroboration. a) E5: E5 has been used in prior broad PIDS evaluations, but we are not aware of public audit support that combines artifact handling, ambiguous-node treatment, and scoring exclusions with the label target used in those evaluations [9], [13]. Existing E5 evaluations rely on manually constructed ground truth from DARPA descriptions. Including E5 would make label construction and artifact handling part of the comparison, weakening the controlled architectural interpretation that the primary regime is designed to support. b) OpTC: OpTC is better documented after the correction and labeling work of Majorczyk et al. [45], but it defines a different target from the process-level scoring target used in our primary E3 experiments. Majorczyk et al.’s pipeline reconstructs malicious host events and network flows from OpTC-specific evidence, including Red Team descriptions, agent PIDs, C&C infrastructure, process-child propagation, and host/network correlation. This improves reproducibility relative to earlier uses of OpTC, but the resulting labels are not equivalent to the REAPr process labels combined with Liu et al.’s artifact analysis and scoring exclusions [10], [19]. A corrected OpTC comparison would require a separate graph construction and label reconciliation, followed by an audit of how those choices affect each baseline. OpTC also provides only one attack scenario per host, so it does not support the independent validation split containing attack data used in our primary regime without either omitting attack data from validation or splitting the same attack campaign. We leave corrected OpTC to future work rather than mixing it with the audited E3 comparison.

A PPENDIX A O PEN S CIENCE To support reproducibility and future work, we publicly release the implementation and adapted baseline evaluation code at https://github.com/lorenzo9uerra/theseus. The processed DARPA TC E3 node and event tables are archived at https://doi.org/10.5281/zenodo.18450778, while the evaluation artifacts are archived at https://doi.org/10.5281/zenodo. 19844784. The latter include graph caches, model checkpoints, retained baseline artifacts, sanitized evaluation logs, and the

B. Dataset Splits Table XIII outlines the fixed E3 and ATLASv2 partitions used throughout our study, with all development data strictly preceding the final test period. After accounting for artifact treatment and the availability of validation days containing attack activity, the E3 datasets do not always permit a perfectly chronological train → validate

16

TABLE XII B ENCHMARK VIABILITY UNDER THE AUDITABLE EVALUATION REGIME . P RIMARY EVIDENCE REQUIRES A COMPATIBLE PROCESS - LEVEL SCORING TARGET, PUBLIC SUPPORT FOR AUDITING ARTIFACTS AND AMBIGUOUS NODES , AND INDEPENDENT ATTACK DATA FOR VALIDATION UNDER THE SHARED CALIBRATION PROTOCOL . Dataset

Compatible Public audit Attack Role labels support validation

E3-Cadets E3-FiveDirections

Yes Yes

Yes Yes

Yes Yes

E3-Trace

Yes

Yes

Yes

E3-Theia E3-ClearScope

Yes No

Yes No

Yes No

ATLASv2

Yes

Yes

No

E5 (Cadets / Theia / ClearScope)

No

No

Yes

OpTC (H051 / H201 / H501)

No

Yes

No

StreamSpot [59] / Unicorn Wget [32]

No

No

No

Implication

Primary Primary

Satisfies the label, audit, and calibration requirements for the primary regime. Satisfies the primary regime and extends the audited comparison beyond datasets emphasized in prior broad baseline studies. Primary Satisfies the primary regime and extends the audited comparison beyond datasets emphasized in prior broad baseline studies. Primary Satisfies the label, audit, and calibration requirements for the primary regime. Excluded No released compatible process labels; the REAPr/Liu audit does not provide a comparable scoring target for ClearScope because its timestamps use arbitrary units [19]. Secondary UUID-based REAPr labels expose Carbon Black process UUIDs and map exactly to reconstructed process nodes. Each host contains a single attack window, so threshold calibration uses benign training scores rather than an independent validation split containing attack data. Excluded Prior evaluations use manually constructed ground truth and report benchmarklevel results, but we are not aware of public audit support that combines artifact handling, ambiguous-node treatment, and scoring exclusions for the label target used there. Excluded REAPr provides no OpTC labels. Liu et al. analyze OpTC, and Majorczyk et al. improve its labels and audit support, but the available labels are built from OpTC-specific host and network evidence rather than the E3 process-label target used here. Each selected host benchmark exposes only one attack scenario. Excluded Synthetic or narrowly scoped graph-level benchmarks without comparable process-level labels, artifact auditing, or independent attack data for separate validation and test sets.

→ test sequence. We therefore allow training and validation days to be interleaved within the pre-test development period while keeping the two sets disjoint. Validation days are not required to occur after all training days, provided that all development data still precede the final test period. Theia is a clear example of this constraint in practice. Following Liu et al. [10], we limit our experiments to the stable T3 period. Since Day 10 is the only day with attack activity available for validation during T3, we use it as our validation set and assign Day 11 for training, holding out Days 12–13 for the final evaluation. Trace follows a similar pattern within its five-day window: Day 10 provides attack data for validation, Days 9 and 11 provide benign training data, and Days 12–13 are reserved for testing. ATLASv2 uses benign training days only and has no validation split, since July 19–20 forms a single engagement window containing attack activity.

TABLE XIII T EMPORAL DATA SPLITS USED FOR EVALUATION . E3 DATES REFER TO A PRIL 2018 AND ATLAS V 2 DATES TO J ULY 2022. E VERY DEVELOPMENT DAY PRECEDES THE TEST PERIOD . ATTACK DAYS ARE SHOWN TO CLARIFY THE SPLIT DESIGN .

Dataset

Attack Days

Training Validation Days Days

Cadets 6, 11–13 2–5, 7–9 FiveDir. 9, 11–13 2–7 Trace 10, 12, 13 9, 11 Theia (T3) 10, 13 11 ATLASv2-h1 19–20 15–18 ATLASv2-h2 19–20 15–18

6, 10 8, 9 10 10 — —

Test Days 11–13 10–13 12, 13 12, 13 19–20 19–20

main tables. We report the released or adapted configurations as the primary baseline settings and treat these sweep results as a robustness check rather than as an exhaustive tuning study.

C. Baseline Configuration Checks

D. Threshold Sensitivity

For the learned baselines, we ran bounded sensitivity sweeps around the released or adapted configurations for each primary dataset. For Magic, we varied learning rate, weight decay, mask rate, and αl . For Velox, we varied learning rate, weight decay, and output dimension. For Orthrus, we varied learning rate, weight decay, dropout, hidden and output dimensions, and the temporal neighborhood size where feasible. On Trace and FiveDirections, the dense graph topology made larger temporal neighborhoods infeasible within the shared resource budget, with runs exceeding 128 GB of RAM. The sweeps produced local variation in individual metrics, but no setting changed the dataset-level interpretation or the comparisons reported in the

Section V summarizes the threshold sweeps reported here. Using frozen checkpoints, we vary only the threshold multiplier relative to the reported threshold, which is set by the maximum benign validation score. The full E3 curves for Theseus and Velox are shown in Fig. 10. E. Command-Line Novelty as an Allowlist Key To test whether richer textual fields strengthen the allowlist diagnostic, we replace executable/path novelty with novelty of the exact command line while keeping the same benign reference data and process labels. As Table XIV shows, the

17

Precision

F1

MCC

Attack Coverage

Cadets

1.0

Precision

FiveDirections

ADP=0.535±0.368

1.0

ADP=0.069±0.010

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

Theia

1.0

Trace

ADP=0.527±0.024

ADP=0.245±0.163

Metric value

Metric value

Recall

0.6

0.6

0.4

0.4

0.2

0.2 1.0x

2.0x

4.0x 0.25x

0.5x

1.0x

2.0x

0.0 0.25x

4.0x

MCC

Attack Coverage

Cadets

FiveDirections

Theia

Trace

ADP=0.870±0.031

1.0 0.8

0.5x

F1

ADP=0.008±0.003

0.0

0.8

0.0 0.25x

Recall

0.5x

ADP=0.512±0.029

1.0x

2.0x

4.0x 0.25x

0.5x

ADP=0.105±0.005

1.0x

Threshold / reported threshold (max benign calibration score)

Threshold / reported threshold (max benign calibration score)

(a) Theseus

(b) Velox

2.0x

4.0x

Fig. 10. Threshold sensitivity on DARPA TC E3 for frozen Theseus and Velox models. Curves and annotated ADP values are means over five seeds, and shaded bands show one standard deviation. The horizontal axis shows the threshold relative to the reported threshold, set to the maximum benign validation score, with the dotted line marking 1.0×.

TABLE XIV E XECUTABLE / PATH AND COMMAND - LINE NOVELTY ON AUDITED E3. B OLD VALUES MARK THE BETTER RESULT WITHIN EACH DATASET; LOWER FPR IS BETTER .

Dataset Novelty key

Flagged

Prec. Recall

TABLE XV RUNTIME OF T HESEUS ON REPRESENTATIVE 15- MINUTE TEMPORAL WINDOWS GENERATED FROM THE CACHED GRAPH - SIZE STATISTICS . L ATENCY IS END - TO - END , INCLUDING EMBEDDING , GRAPH CONSTRUCTION , AND INFERENCE ; CPU AND GPU MEMORY ARE PEAK VALUES .

FPR

Cadets

Executable/path Command line

36 0.4286 0.0105 0.0011 36 0.4286 0.0105 0.0011

FiveDir.

Executable/path Command line

322 0.0031 0.0385 0.0014 2,369 0.0013 0.1154 0.0102

Theia

Executable/path Command line

226 0.1770 0.5970 0.0023 7,998 0.0053 0.6269 0.0974

Trace

Executable/path 700 0.0573 0.8163 0.0003 Command line 690,673 0.0001 0.9796 0.2758

Dataset

Nodes Edges (avg.) (avg.)

Latency (ms)

Cadets FiveDir. Trace Theia

690 4370 5757 4306

15.3 ± 0.3 60.5 ± 0.4 82.1 ± 0.9 56.3 ± 1.8

8948 13 843 14 384 31 677

CPU mem. GPU mem. (MB) (MB) 1308 1335 1294 1283

306 709 361 698

and peak CPU/GPU memory usage on a single NVIDIA L40S. The model processes a single window in 15 to 82 ms across the evaluated datasets, achieving a throughput of 45,000 to 76,000 nodes per second. These results indicate that model inference is not the primary bottleneck at this window size. To manage resource consumption, the pipeline generates 15minute snapshots and partitions any graph exceeding 10,000 nodes into smaller subgraphs before the Transformer stage. This restricts the Transformer’s O(N 2 ) attention mechanism to under 1 GB of GPU memory throughout our experiments. Graph construction accounts for 40% of total latency on Theia and approximately 60% on Cadets, FiveDirections, and Trace. Because graph building takes two to three times longer than inference, future performance optimizations for this workload should prioritize stream processing and graph construction rather than further compressing the neural network.

richer key flags many more processes and so tends to raise recall, but it also introduces many benign false positives. Cadets is unchanged, since its command-line field adds no further distinction. On FiveDirections, command-line novelty flags more than seven times as many processes while lowering precision and F1. The effect is stronger on Theia and Trace, where recall rises only slightly but FPR climbs to 0.0974 and 0.2758, sharply reducing precision, F1, and MCC. Overall, using the exact command line as the allowlist key does not yield a stronger diagnostic than using executable names and paths. A PPENDIX C T HESEUS D ETAILS A. Runtime Analysis We profiled Theseus using representative 15-minute windows generated from the cached E3 graph-size statistics to assess the reference model’s computational cost under standard experimental conditions. Table XV reports end-to-end latency

B. Hyperparameters Table XVI lists the final Theseus hyperparameters established under our evaluation protocol. We developed these con-

18

TABLE XVI H YPERPARAMETERS FOR T HESEUS . Parameter

Cadets

FiveDir.

Theia

Trace

Training Epochs Patience SAGE LR Transformer LR SAGE Weight Decay Transformer Weight Decay SAGE Dropout Transformer Dropout Batch Size

300 20 8.7e-5 6.5e-3 0.3 7.5e-3 0.4 0.1 2

300 25 6.0e-5 7.0e-4 0.2 0.05 0.3 0.2 1

400 70 5.0e-5 7.5e-4 0.2 0.01 0.2 0.1 1

100 20 4.8e-5 7.5e-5 0.06 0.07 0.25 0.2 2

Architecture Transformer Embed. Dim Node Embed. Dim Attention Heads Transformer Layers Fused Edge Counts Node Degrees

96 192 2 1 Yes Yes

8 140 4 1 Yes Yes

128 128 8 2 No No

128 128 8 2 Yes Yes

figurations through a limited coarse random search followed by Bayesian optimization targeting AP. The selected configurations varied with the available dataset semantics. For Cadets, FiveDirections, and Trace, explicit structural features helped offset weaker node semantics. On Theia, the model performed better without these structural additions, relying instead on populated command-line and path attributes. For ATLASv2, we used a single configuration across both h1 and h2 hosts: 300 epochs, SAGE and Transformer learning rates of 10−5 and 10−4 respectively, 0.2 dropout in both modules, one hop, a node output dimension of 96, a Transformer embedding dimension of 32, four attention heads, one Transformer layer, 15-minute windows, a batch size of 2, and structural features enabled.

19