Integrating Perception, Comprehension, and Projection for Alert Validation: A Unified Cyber Situational Awareness Framework

arXiv:2610.39849v1 [cs.CR] 29 Oct 2026

Integrating Perception, Comprehension, and Projection for Alert Validation: A Unified Cyber Situational Awareness Framework Navodika Karunasingha✉∗

Mohan Baruwal Chhetri✉

Surya Nepal✉

n.karunasingha@unsw.edu.au School of Computer Science and Engineering, UNSW Sydney Australia

mohan.baruwalchhetri@csiro.au CSIRO Australia

surya.nepal@unsw.edu.au surya.nepal@csiro.au School of Computer Science and Engineering, UNSW Sydney Australia CSIRO Australia

Cecile Paris✉

Salil S. Kanhere✉

cecile.paris@csiro.au CSIRO Australia

salil.kanhere@unsw.edu.au School of Computer Science and Engineering, UNSW Sydney Australia

Abstract Security Operation Centres (SOCs) must validate large volumes of alerts with limited analyst resources, yet alerts often lack sufficient context to determine whether suspicious activity is attackrelated. While provenance-based intrusion detection research has progressed from perception-level signals to comprehension- and projection-level cyber situational awareness (CSA), it remains unclear whether richer CSA consistently improves alert validation. This paper presents the first systematic evaluation of perception-, comprehension-, and projection-level CSA within a unified alert validation framework. Using Captain and Kairos, we generate multilevel CSA and formulate alert validation as a positive–unlabelled multi-instance learning (PU-MIL) task over CSA-derived behavioural hypotheses. Experimental evaluation on the DARPA TC dataset shows that comprehension-level CSA substantially improves validation by incorporating causal attack context. Relative to the Queue and Max. CSG baselines, PU-MIL achieves up to 28.6 and 75 percentage-point gains in Recall@5, and 24.6% and 86.8% reductions in mean positive rank on Cadets and Theia, respectively. Projection-level CSA can yield additional benefits, reducing the mean positive rank on Theia from 22.0 to 8.5 at a 40-minute projection horizon, although its effectiveness varies across projection horizons. Overall, richer CSA does not necessarily lead to better alert validation. Rather, its value depends on when and how it is introduced, motivating adaptive CSA progression that selectively enriches alerts according to validation requirements.

CCS Concepts • Security and privacy → Human and societal aspects of security and privacy.

1

Introduction

Security Operation Centres (SOCs) rely heavily on intrusion detection systems (IDSs) to identify early signs of malicious activity in enterprise environments. Beyond raising alerts, IDSs underpin

cyber situational awareness (CSA), the analyst’s ability to determine whether an observed event is malicious or benign, and to reason about its type, source, and target [14]. Following Endsley’s model, CSA is commonly organised into three levels of increasing analytical depth: perception, comprehension, and projection [2, 7]. The growing sophistication of multi-stage attacks, coupled with the availability of fine-grained host telemetry, has driven intrusion detection toward system-level provenance analysis, where system execution is modelled as causally connected interactions among system entities such as processes, files, and network sockets [17, 28]. Building on this representation, provenance-based IDS (PIDSs) incorporate causal reasoning into alert generation by propagating suspiciousness across these interactions, enabling more contextaware detection and reducing false positives [9, 10, 24]. Despite these advances, modern alert generation pipelines continue to produce overwhelming volumes of alerts, many of which are false positives, requiring Tier-I analysts to manually validate each alert before escalation or response. Prior work shows that excessive alert volumes, coupled with limited contextual and causal attribution, significantly increase analyst workload and contribute to alert fatigue in SOCs [17, 28]. Because only a small fraction of generated alerts can realistically be investigated in operational SOCs, alert validation and prioritisation are closely intertwined. Under limited resources, analysts must determine which alerts warrant investigation and should therefore be prioritised. Validating a suspicious alert requires gathering related evidence and reconstructing its causal context. At the system level, this means correlating causally related system activities into connected provenance structures that capture an attack’s progression from its root cause through subsequent system interactions [17, 28]. Such provenance dependency graphs (PDGs) reveal the causal chains of activities associated with an alert, supporting causal awareness of how a security event emerged and evolved [14]. Recent provenance-analysis research has increasingly focused on automating PDG reconstruction from system activities and integrating them into alert generation [4, 8, 19, 20]. System activities are

Karunasingha et al.

causally linked to form PDGs, and alerts are generated based on suspiciousness propagated over these structures rather than isolated events. Consequently, alert generation moves beyond perceptionlevel signals and increasingly relies on comprehension-level CSA, where detection decisions are informed by reconstructed attack evolution and causal dependencies among system activities. More recently, provenance-guided analysis has shifted toward projectionlevel reasoning, aiming to anticipate future attacker actions and targets [14, 18, 27]. While these approaches primarily focus on estimating future attack states to support projection-level CSA generation, they remain largely decoupled from alert generation and validation workflows. Collectively, the literature reveals an evolutionary progression from perception-level CSA, with limited causal correlation support [9, 10, 24] to subsequent approaches that explicitly reconstruct PDGs for comprehension-level causal attack reconstruction [4, 8, 19, 20] and projection-oriented future attack-state reasoning [18, 27]. Yet, despite this progression, existing work focuses primarily on generating these distinct tiers of CSA rather than systematically evaluating their relative utility. Consequently, it remains unclear whether progressively richer CSA information consistently improves validation effectiveness, what marginal benefit each additional level provides, and whether the additional analytical complexity of generating richer CSA is operationally justified in a SOC. Richer CSA provides access to additional contextual and behavioural evidence, but it also introduces greater computational cost, analytical complexity, and potentially irrelevant information. So, we cannot assume that progressing to higher CSA levels will necessarily improve alert validation performance. To the best of our knowledge, no prior study has systematically compared perception-, comprehension-, and projection-level CSA within a unified framework to understand their relative contribution to alert validation effectiveness. This motivates a fundamental question: How do perception-, comprehension-, and projection-level CSA influence alert validation effectiveness, and what additional value does each CSA level contribute to the validation process? To answer this, we present a unified framework to incorporate and evaluate progressively richer CSA in alert validation. Alert validation provides a natural setting to directly assess the value of situational awareness information in distinguishing attack-related alerts from benign ones. Specifically, this paper makes the following contributions: • We formulate alert validation as a progressive CSA-aware decisionmaking task and present the first systematic evaluation of CSA levels for alert validation. • We propose a unified framework for progressive CSA generation and alert validation that integrates all three CSA levels through causal and predictive reasoning. • We instantiate the framework using state-of-the-art approaches, operationalising perception with CAPTAIN [24], comprehension and projection using Kairos [4] and alert validation using positive–unlabelled multi-instance learning (PU-MIL) [21]. • We empirically demonstrate the value of progressively richer CSA using Recall@K, Mean Positive Rank (MPR), and ProjectionInduced-Rank-Shift (PIRS).

2

Background

This section introduces the foundational concepts underpinning our approach and motivates alert validation as a progressive CSA-aware decision-making task. We discuss relevant literature throughout the paper, including the problem motivation (Section 1), framework components (Section 4), and alternative realisations (Appendix A).

2.1

Alert Analysis and Analyst Workflows

SOCs rely on automated detection systems to identify potential threats in large-scale enterprise environments. These systems generate alerts based on predefined rules, statistical anomalies, or learned patterns. However, in practice, analysts face high alert volumes and limited context, requiring manual investigation to determine whether an alert represents a true incident [22]. A key challenge in this workflow is the disconnect between alert generation and cyber situation awareness (CSA). Alerts are typically presented as discrete events, such as suspicious process executions or network connections, without sufficient context about their causal relationships or broader impacts. Consequently, analysts must reconstruct attack scenarios by correlating evidence from multiple data sources, including system logs, network traces, and audit records: a time-consuming and cognitively demanding task, particularly in high-alert environments. PIDSs aim to address this limitation by modelling system execution as a graph of causal dependencies, enabling analysts to reason about how events are connected [4, 12]. While these systems provide richer context, it remains unclear how progressively richer CSA information influences automated alert validation under weakly supervised settings.

2.2

Cyber Situational Awareness

CSA provides a conceptual framework for how human decisionmakers process information in complex environments [14], commonly structured into three levels: (i) perception, which identifies whether the system is in a malicious or benign state by identifying the type, source, and target of a malicious situation; (ii) comprehension, which explains the relationships and significance of these elements, such as interpreting causal dependencies between events; and (iii) projection, which anticipates future states and potential consequences, including how an attack may evolve. In SOCs, these levels correspond naturally to different forms of information; alerts generated by detection systems correspond to perception level CSA [9, 24], provenance-based summary graphs correspond to comprehension level CSA [4, 12], and predictions or future-oriented context correspond to projection level CSA [18, 27]. While CSA has been widely studied in domains such as aviation and military operations, its application in cybersecurity, particularly within alert validation workflows, remains underexplored. Existing systems focus primarily on improving detection performance rather than systematically structuring information to support CSA.

3

Motivating Example

Figure 1 presents a motivating example from the DARPA Transparent Computing (DARPA TC) Cadets dataset1 , illustrating a multistage attack. The attack begins with exploitation attempts against 1 https://github.com/darpa-i2o/Transparent-Computing, last accessed May 2026.

Unified CSA Framework for Alert Validation

Exploit attempt 1

Exploit attempt 2

1

(a)

Privilege escalation

2

Exploit attempt

against Nginx

failed.

Exploit succeeded. loaderDrakon connects to operator console.

11:21

11:22

4

Attacker downloads

elevate and runs it to obtain root privileges.

Privilege

Escalation

echnique

blic-Facing Application

(T1190);

Command & Scripting Interpreter

(T1059)

Privilege Escalation

Gain initial foothold on the

FreeBSD server;

blish interactive remote

access.

privileges for

Exploit Pu

Why this step?

Elevated process downloads and runs netrecon module.

Discovery

Exploitation for

Configuration

Discovery (T1016)

Obtain root

full control

ALERT A — Later-Stage Alert

12:04

12:08

Discovery

Defense Evasion /

Persistence

Application Layer Protocol

(T1071) y

blish C2

hannel for

k .

Type:

outbound

connection

to external host Outbound Connection Attempt

Subject:

netrecon

Action:

connect()

Object:

61.167.39.128:80

Rule:

Unauthorized outbound connection

to suspicious external host

process

(b)

61.167.39.129

Time:

81.49.200.

166

nginx

he

- j dule on the

host.

mo

57)

j

in ection

78.205.235.

65

/tmp/ vUgefal

vUgefal (elevated)

12:10

Hide within a trusted

s

(e.g., sshd) for

Impact

(Unintended)

(Consequence)

-

-

Proj. CSG A2

152.111.159.139

/var/log/devc

Process Enumeration

/var/log/devc

sshd

Outbound Connection Attempt

Decision for Alert A Decision for Alert A MALICIOUS

FreeBSD

Kernel Panic

(Benign Hypothesis)

Lost

connection

Decision for Alert A Decision for Alert A MALICIOUS

The future provenance behavior further reinforces the attack interpretation. Projection-level CSA further increases confidence

Confidence:

MEDIUM

Confidence:

HIGH

Projection-level CSA further increases confidence.

Projected CSGs for Alert B

CSGs for Alert B

Subject: Action: Object: Rule:

nginx connect() 78.205.235.65:80 Unusual outbound connection

from Internet-facing service 2018-04-06 ~11:22

Nginx is making an unusual outbound connection. Is this legitimate service activity or something suspicious?

81.49.200.166

nginx

CSG B2

(Benign Hypothesis)

81.49.200.166

(Benign Hypothesis)

Proj. CSG ANₒ

Proj. CSG B2

ojected CGS B1 (Attack Hypothesis)

Pr

(Benign Hypothesis)

78.205235.65

Type:

dly.

Projected CSGs for Alert A

61.167.39.128

This provenance chain clearly connects exploitation, payload staging, outbound communication, and injection behavior. Comprehension-level CSA is sufficient to validate the alert

d

terminate unexpecte

CSG AN꜀ (Benign Hypothesis)

154.145.113.18

(FAILED)

Attack campaign

persistent

Projected CGS A1 (Attack Hypothesis)

200.36.109. loaderDrakon 81.49.200.

214 166 (in-memory)

netrecon module

12:10+

~

Impact

Host destabilised

daemon to .

.

Lost connection

to loaderDrakon

on target.

~

I j (T1055)

target proces

victim

Host locks up

and experiences

kernel panic.

Process n ection

Identify suitable

Stage t

process in ection

CSG A2 (Benign Hypothesis)

CSG B1 (Ambiguous Hypothesis)

Time:

(T10

5)

sshd

ALERT B - Earliler-Stage Alert

(c)

(T110

y

CSGs for Alert A

This outbound connection attempt is suspicious.

It indicates possible C2 communication to an external host.

netrecon

(PID 4310)

Process Discover

Transfer

c

139.123.0.113

2018-04-06 11:42:00

Ingress Tool

attac er control

CSG A1 (Attack Hypothesis) netrecon

process netrecon

Attempts to inject

librdrakon into sshd

(PID 809) – failed.

Lists running

processes.

12:04

esta

.

10

Ingress Tool

Transfer

Successfull

bout the

C

9

11:42

Collect a

environment

Kernel panic

8

ommand and Control

information

.

Inject into sshd

7

Downloads

librdrakon module to /var/log/devc/.

C

System Network

(T1068)

Esta

Connect out to

61.167.39.128:80

succeeded.

onnection lost

List processes

6

11:38

Initial Access

Download librdrakon

5

11:33

Tactic

T

Netrecon connect

Run netrecon

3

nginx

78.205.235.65

CSG BN꜀

(Benign Hypothesis)

The connection follows unusual HTTP activity involving Nginx, but the available evidence is still insufficient to confirm malicious behaviour. Comprehension-level CSA leaves multiple plausible hypotheses

78.205.235 200.36.109. .65 214

61.167.39.

128

152.111.159.

139

loaderDrakon

(in-memory)

81.49.200.

166

139.123.0. 113

/tmp/ vUgefal

vUgefal (elevated)

/var/log/devc

Process Enumeration

sshd

/var/log/devc

sshd

netrecon 154.145.113.

18

module (FAILED)

Kernel

Panic

Subsequent activity reveals an attacker implant, external control, and privilege escalation. This confirms the earlier Nginx connection as part of the attack. Projection level CSA is required to validate the alert

Lost

connection

Proj. CSG BNₒ

(Benign Hypothesis)

Decisionfor forAlert AlertAA Decision A CO S Confidence: M

LI

I

U

HIGH

Projection-level CSA is REQUIRED to validate Alert B.

Figure 1: Motivating example of progressive CSA-based alert validation using the DARPA-TC E3 2 CADETS Day 6 attack campaign. (a) Attack scenario. (b) Validation of later-stage Alert A. (c) Validation of earlier-stage Alert B. an Nginx process on a FreeBSD host (Initial Access). Following a successful attempt, a connection is established between loaderDrakon and a remote attacker-controlled shell (Execution, Command and Control). The attacker downloads and executes malicious modules, including netrecon for outbound communication and attempts to inject libdrakon (Command and Control, Execution) into the privileged sshd daemon (Privilege Escalation, Defense Evasion). This causes system instability and loss of host connectivity (Impact). Alert A and Alert B are generated at different stages of the same attack, yet present substantially different behavioural evidence for downstream validation. At the perception level, both provide limited CSA (e.g., suspicious entities, triggering interactions, alert metadata), indicating anomalous behaviour without reliably distinguishing malicious from benign activity. To enrich this picture, we reconstruct the provenance activity around each alert into a PDG and decompose it into smaller candidate summary graphs (CSGs) that represent local behavioural hypotheses [4, 12]. Multiple competing CSGs may be generated for each alert (see Figure 1(b)); only some correspond to the attack progression, while others represent anomalous but benign behaviour. The two alerts differ in how much additional CSA information

contributes to alert validation. For Alert A, comprehension-level information is sufficient because the reconstructed CSGs already expose causally connected attack behaviour (e.g., malicious module execution, successful C2 communication, privileged daemon injection); projection-level CSA may further increase confidence by revealing the likely consequences (e.g., abnormal sshd behaviour, kernel instability, communication disruption). Hence, reliable validation is already possible at the comprehension level. In contrast, Alert B remains ambiguous even after comprehension-level reconstruction because its CSGs primarily capture suspicious but operationally plausible behaviour (e.g., transient process execution, failed outbound network communication); projection-level CSA reveals subsequent attack progression, including successful external communication, malicious payload deployment, daemon injection, and eventual host instability, making the attack path distinguishable from benign alternatives. This example highlights two challenges: (i) Alerts from the same attack campaign may require different levels of CSA for reliable validation. A fixed CSA level can be insufficient for some alerts while introducing unnecessary analysis overhead for others. Here, we focus on the availability of CSA information required for validation,

Karunasingha et al.

without explicitly modelling human cognitive limitations [7, 13, 22]. (ii) Provenance reconstruction produces multiple competing CSGs representing alternative behavioural hypotheses [4, 12]. While prior work has identified malicious entities and event labels [12, 24], no existing study, to our knowledge, determines which reconstructed CSGs correspond to the underlying attack path. Alert validation can be formulated as a weakly supervised learning problem in which progressively richer CSA helps identify the most plausible attackrelated CSGs among competing hypotheses [21]. This motivates a unified framework to systematically integrate and evaluate CSA across multiple levels.

4

Framework for Progressive CSA Generation and Alert Validation

The motivating example in Section 3 shows that alerts may require different levels of CSA for validation. Some can be confidently validated using preceding context (comprehension-level information); others require projected or observed attack context (projectionlevel information). Alert validation can therefore be viewed as progressively incorporating CSA into decision-making. Prior research has explored capabilities such as provenancebased intrusion detection [8–10, 24], attack reconstruction and provenance summarisation [4, 12], and attack forecasting [18, 27]. Although these capabilities correspond to different levels of CSA, researchers typically evaluate them only against their individual operational objectives. As a result, the systematic generation, integration, and use of progressively richer CSA for alert validation remain largely unexplored. To address this gap, we conceptualise progressive CSA generation as a modular workflow that incrementally enriches low-level system activity with perception-, comprehension-, and projectionlevel CSA information prior to alert validation. Each stage contributes distinct contextual evidence, resulting in heterogeneous and variable-sized CSA representations. The lack of fine-grained labels connecting provenance structures to attack progression makes alert validation a weakly supervised problem. Based on this conceptualisation, the proposed framework (see Figure 2) comprises four functional stages, discussed below.

4.1

Perception-Level CSA Generation

Perception-level CSA supports recognition, which determines whether an observed situation is malicious or benign, and identification, which determines the type, source, and target of that situation [14]. In provenance-based security analysis, provenance-based intrusion detection systems (PIDSs) typically generate perception-level CSAs. By analysing causal interactions among entities in provenance graphs, PIDSs identify anomalous or potentially malicious behaviour and generate alerts describing suspicious situations [4, 9, 10, 12, 24]. From a CSA perspective, these alerts establish awareness of suspicious activity while identifying the entities and interactions involved. The increasing adoption of PIDSs has led to diverse approaches for generating perception-level CSA. Although these approaches differ in how they infer suspiciousness, they share the common goal of transforming low-level provenance events into alerts that expose potentially malicious situations. These design

Perception-Level CSA 1

Initial detection and recognition of suspicious activity

4

Comprehension-Level CSA

Weakly Supervised Alert Validation

Input 2 System-Level Activity

Causal understanding of the attack

Projection-Level CSA 3

Decision making with positive–unlabelled supervision

Output ×

Malicious

✓

Benign

Anticipation of future evolution of the attack

Figure 2: Conceptual framework for progressive CSA generation and alert validation.

choices shape the form and granularity of perception-level CSA, as discussed further in Appendix A.1.

4.2

Comprehension-Level CSA Generation

Comprehension-level CSA supports causal awareness by enabling understanding of the causes, mechanisms, and significance of suspicious system behaviour through reconstruction of the causal relationships surrounding an alert [14]. Rather than treating an alert as an isolated event, comprehension-level CSA enriches it with provenance context that captures interactions among system entities and the causal progression of suspicious activities. This enables analysts to reason about how the situation evolved, its underlying causes, and its relationship to the broader attack progression. In provenance-based security analysis, comprehension-level CSA is typically generated by reconstructing provenance structures surrounding suspicious activity [4, 8, 15, 19, 20]. System execution is represented as causally connected interactions among system entities, from which attack-relevant provenance dependency graphs (PDGs) are constructed [8, 15, 20]. These structures capture behavioural relationships associated with an alert and provide causal explanations beyond those available at the perception level. The increasing adoption of PIDSs has led to diverse approaches for generating comprehension-level CSA. Although they differ in how they reconstruct attack-relevant provenance structures, they share the common objective of transforming low-level provenance activity into causally meaningful behavioural representations. The resulting representations vary in granularity and operational characteristics, and Appendix A.2 discusses different approaches to generating comprehension-level information.

4.3

Projection-Level CSA Generation

Projection-level CSA supports reasoning about how suspicious behaviour may evolve beyond the current system state by extending analyst awareness from understanding the present situation to anticipating plausible future developments [14]. Where perceptionlevel CSA identifies suspicious activity and comprehension-level CSA establishes causal awareness of how it emerged, projectionlevel CSA provides future-oriented context that supports anticipation of the likely progression of attacks, subsequent attacker actions, or system entities yet to be affected. In provenance-based security analysis, projection-level CSA is commonly generated by reasoning over provenance-derived behavioural representations, such as PDGs or attack graphs, to infer future attack evolution from observed system behaviour [18, 27].

Unified CSA Framework for Alert Validation

These mechanisms extend the reconstructed attack context beyond the currently observed provenance activity, predicting plausible future system states. From a CSA perspective, such predictions constitute projection-level CSA because they provide anticipatory awareness of how an ongoing attack may evolve. Existing approaches formulate projection-level CSA as a forecasting problem over attack or provenance graphs. Although they differ in how they model and predict future attack behaviour, they share the objective of estimating attack evolution from observed system behaviour. Resulting projections may include anticipated attacker actions, future paths to compromise, or system entities likely to be affected during subsequent attack stages. Appendix A.3 discusses alternative realisations of projection-level CSA generation.

4.4

Progressive CSA-Aware Alert Validation

Alert validation determines whether a generated alert constitutes a genuine security incident requiring further investigation. Alerts generated by PIDSs rarely provide sufficient CSA to support confident decision-making. Analysts, therefore, retrieve related system activity, reconstruct causal relationships, correlate supporting evidence, and examine subsequent system behaviour. This investigative effort constitutes a substantial component of analyst workload and contributes significantly to alert fatigue in modern SOCs [1, 26]. Recent research has recognised the value of contextual enrichment for alert validation. Existing approaches augment alerts with historical provenance dependencies, event relationships, and other forms of behavioural context to support triage [8, 23, 25]. Although these improve downstream decision-making, they generally treat contextual information as a single source of evidence and do not explicitly distinguish between CSA levels or investigate how progressively richer CSA contributes to validation. Within the proposed framework, alert validation is conceptualised as a progressive process of evidence accumulation. Rather than treating an alert as a static detection outcome, it is progressively enriched with CSA information generated by the upstream stages (Sections 4.1–4.3). Validation then evaluates the accumulated evidence, enabling increasingly informed decisions as additional CSA becomes available. This formulation reflects operational SOC workflows, where analysts progressively incorporate additional evidence throughout an investigation rather than relying solely on the initial alert. By explicitly organising validation around perception-, comprehension-, and projection-level CSA, the framework provides a structured foundation for automating alert validation while reducing manual investigation effort. The framework deliberately separates CSA generation from validation, making it independent of any specific algorithm and adaptable to different implementation strategies. Section 5 presents one concrete instantiation and demonstrates how the generated CSA is operationalised for automated alert validation.

5

Framework Instantiation

This section provides a concrete instantiation of the framework presented in Section 4 using selected state-of-the-art provenance-based security analysis techniques (see Appendix A for alternative instantiations). Rather than introducing new intrusion detection or attack reconstruction mechanisms, this instantiation operationalises the

framework through compatible components, enabling a systematic evaluation of how progressively richer CSA influences alert validation. Figure 3 presents the resulting end-to-end system.

5.1

System Instantiation Overview

Figure 3 illustrates the complete system instantiation and the information flow between its constituent components. The workflow begins with chronologically ordered audit records, which are transformed into whole-system provenance graphs that serve as the common input to subsequent stages. Captain analyses the provenance graphs to generate alerts together with their associated perception-level CSA. These alerts then anchor the adapted Kairos pipeline, which reconstructs the provenance context surrounding each alert and generates comprehension-level and projection-level CSGs across different observation horizons. The outputs of these stages are subsequently transformed into representations suitable for downstream validation. Perceptionlevel CSA contributes alert-centric information; comprehensionand projection-level CSA contribute variable-sized collections of behavioural hypotheses derived from the surrounding provenance context. Together, these heterogeneous representations form the evidence the PU-MIL model consumes. PU-MIL evaluates this evidence progressively across CSA levels. At the comprehension level, it learns from the behavioural structures available when the alert is generated. The learned comprehension-level representation is then transferred to the projection stage, where it helps interpret subsequent behavioural evolution. The resulting maliciousness estimates rank alerts, producing a prioritised alert queue. Sections 5.2–5.4 describe the generation and representation of each CSA level, while Figure 3 and Section 5.5 present their integration through the progressive PU-MIL validation process.

5.2

Perception-Level CSA Generation

At the perception level, low-level system activities are transformed into alert-centric CSA. System activities are first modelled as provenance graphs. Nodes represent system entities such as processes, files, sockets, and memory objects; edges capture causal interactions among these entities, including file reads and writes, process execution, process creation, and network communication. Following the PIDS adopted in this work [24], confidentiality and integrity tags are propagated through the graph according to predefined information-flow rules. Alerts are generated when the propagated tag values satisfy suspicious flow conditions. The resulting alert set A = {𝑎 1, 𝑎 2, . . . , 𝑎 𝑁 } forms the basis for perception-level CSA. For each alert, the attributes summarised in Table 1 are transformed into an alert-level feature representation 𝑥𝑖(𝑃 ) = 𝜙 𝑃 (𝑎𝑖 ), where 𝜙 𝑃 (𝑎𝑖 ) denotes the perception-level feature extraction function. The resulting vector 𝑥𝑖(𝑃 ) encodes the information directly available at the time of alert generation, including alert metadata, the participating entities and interactions, and tagrelated indicators. As such, it constitutes the perception-level CSA representation of alert 𝑎𝑖 and serves as an input for CSA-aware alert validation.

Karunasingha et al. Alert ai (detection time tᵢ)

Perception-level alert features

P

( )

xi

Features

Δt = 0 1

2

Candidate CSGs (bag)

Decompose retrospective PDG into candidate summary graphs

/ Latent

CSG instance features

Map each CSG to a fixeddimensional feature vector

Errors

x

Weights

Instance autoencoding

(reconstruction error)

4

Encode, decode and compute

reconstruction error

Calibrate RE to instance

maliciousness

3

RE (squared

L₂ error)

(l)

Gi1

Probabilities

(l) i1 (l)

fθ

Standardise

dz

z ij ∈ ℝ (l)

r i1

r i1

(l)

p i1

Learn contribution weight Contribution

weight (l)

w i1

Weighted aggregation

(Weighted Noisy-OR)

5

(l) i

(l)

xi k

(p

(l) ik

(l) ik

Π (l)

r i k (l)

(l)

r i k (l) i

i

Encoder

(l)

p i k (l) i

(l)

wi k (l) i

(l)

(l)

(l)

(l)

i

3

(autoencoder) to Step for projection level only

6

(C)

zi

~ (l)

Pi

l ∈ { C, O }

Pi

(C)

Descending alert ranking

π

Projection level (l = O) Fused projection

Residual

(from perception)

Feedforward

maliciousness

Pi

Decoder

Feedforward path from Step

+

Δi

Pi

(level l)

Fused comprehension maliciousness ~ (C)

(C)

Pi

(O)

Pooled latent (feedforward to projection–level fusion)

Final output

(alert ranking)

7

Fused maliciousness

comprehension

maliciousness

bag maliciousness

( p i k , wi k ) i

Level-specific final processing Fused

Residual (from perception) (l)

(l)

(l)

ₚ

d

ℝ

Comprehension level (l = C)

)

, w

gφ

i

6

Aggregate instance maliciousness to

bag maliciousness

(l)

Gi k

∈

Feedforward

Instance maliciousness

& contribution

Calibrate

(logistic)

(l)

(l)

/ Perception

context (O)

(O)

(C)

Δi

hi = φ O (z i )

+ Fused projection maliciousness

~ (O)

Pi

(comprehension embedding)

(l)

~ (l)

~ (l)

= argsort ( P 1 , …, P N )

Rank

Alert a i

~ (l)

Score P i

1

a 436

0.94

2

a 684

0.89

3

a1005

0.84

⋮

⋮

⋮

Prioritised

investigation

queue

Figure 3: Progressive PU-MIL instantiation for CSA-aware alert validation. CSA information is progressively integrated to estimate alert maliciousness and produce a prioritised investigation queue. Table 1: Feature categories used to instantiate perception-, comprehension-, and projection-level CSA from a security analyst’s perspective. They are transformed into fixed-dimensional feature vectors for CSA-aware alert validation. Representation

Feature Category

Encoded Information

Alert type

Type of suspicious activity inferred from system interactions (e.g., process execution, memory access, file access, and network communication), where the entity’s security attributes (e.g., confidentiality and integrity) satisfy the predefined alert-generation policies Subject and object of the suspicious interaction, where the subject is the acting entity initiating the interaction and the object is the entity affected. Entities include processes, files, sockets, network flows, and other provenance-tracked system objects. Integrity and confidentiality tags maintained for the participating system entities during provenance analysis. Indicators of whether the security attributes of the system entities satisfy the alert-generation conditions of the triggered event.

(𝑃 )

𝑥𝑖

System entities

Entity security attributes Alert condition satisfaction

(𝐶 )

(𝑂 )

𝑥𝑖 𝑗 , 𝑥𝑖 𝑗

5.3

Graph size Temporal coverage Alert-Entity correspondence Process entities File entities Network entities Interaction distribution Graph anomaly statistics

Number of entities and interactions represented in the observed or projected CSG. Temporal window spanned by the observed or projected CSG and its relation to the alert detection time. Indicators of whether alert-associated entities appear in the observed or projected CSG. Number of process entities in the observed or projected CSG. Number of file entities in the observed or projected CSG. Number of network communication entities in the observed or projected CSG. Frequencies of process, file, memory, and network interactions represented in the observed or projected CSG. Mean, maximum, and cumulative anomaly scores over the interactions in the observed or projected CSG.

Comprehension-Level CSA Instantiation

At the comprehension level, each alert is enriched with causally related provenance context by reconstructing attack-relevant provenance structures (e.g., PDGs) surrounding the alert. Specifically, the alert serves as an anchor for retrieving provenance activity from the underlying system execution at the time of the alert generation (Δ𝑡 = 0), enabling retrospective analysis of events contributing to the alert. To improve interpretability and support analyst investigation, we decompose the reconstructed PDG into smaller CSGs using a community detection algorithm (e.g., Louvain), where each CSG represents a locally coherent behavioural hypothesis associated with the alert [4]. The resulting comprehension-level CSA for alert 𝑎𝑖 is therefore represented as a collection/bag of CSGs, n o𝑘 (𝐶 ) ) 𝑖 𝐵𝑖(𝐶 ) = 𝐺𝑖(𝐶 𝑗

(1)

To incorporate these graph-based representations into downstream progressive CSA-aware alert validation training, we map each CSG to a fixed-dimensional feature vector. Applying 𝜙𝐶 (·), ) the comprehension-level feature extraction function, to each 𝐺𝑖(𝐶 𝑗   ) (𝐶 ) yields 𝑥𝑖(𝐶 , forming the feature set, 𝑗 = 𝜙𝐶 𝐺 𝑖 𝑗 n o𝑘 (𝐶 ) ) 𝑖 𝑋𝑖(𝐶 ) = 𝑥𝑖(𝐶 𝑗

(2)

𝑗=1

The features in 𝑋𝑖(𝐶 ) characterise the structural, temporal, entitylevel, interaction-level, and anomaly properties of each CSG, as summarised in Table 1.

5.4

Projection-Level CSA Instantiation

𝑗=1 ) (𝐶 ) 𝑡ℎ where 𝐺𝑖(𝐶 is the 𝑗 denotes the 𝑗 CSG associated with alert 𝑎𝑖 ; 𝑘𝑖 number of CSGs reconstructed for 𝑎𝑖 . Since different alerts occur within different provenance contexts, 𝑘𝑖(𝐶 ) may vary across alerts.

At the projection level, the provenance observation horizon is extended beyond the alert-detection time to incorporate subsequent system behaviour. For an alert 𝑎𝑖 detected at 𝑡𝑖 , the available provenance is expanded by a projection interval Δ𝑡. This produces an

Unified CSA Framework for Alert Validation

extended provenance trace that spans from the initial system interaction at 𝑡 0 to 𝑡𝑖 + Δ𝑡. The additional activities observed within (𝑡𝑖 , 𝑡𝑖 + Δ𝑡] are incorporated into the same reconstruction and summarisation model used at the comprehension-level. As a result, the CSGs associated with the alert may change, yielding the projectionlevel CSG collection: n o𝑘 (𝑂 ) ) 𝑖 𝐵𝑖(𝑂 ) = 𝐺𝑖(𝑂 𝑗

(3)

𝑗=1 ) 𝑡ℎ CSG associated with alert 𝑎 after exwhere 𝐺𝑖(𝑂 𝑖 𝑗 denotes the 𝑗

tending the provenance observation horizon, and 𝑘𝑖(𝑂 ) is the number of resulting CSGs. Because newly observed activities may alter the reconstructed provenance context, both the composition and the number of CSGs can differ from those obtained at the comprehension level. As in the comprehension stage, we map each projection-level CSG to a fixed-dimensional feature vector. Applying the graph fea) (𝑂 ) )
ture extraction function 𝜙𝑂 (·) to each 𝐺𝑖(𝑂 = 𝜙𝑂 𝐺𝑖(𝑂 , 𝑗 yields 𝑥𝑖 𝑗 𝑗 giving the projection-level feature set for alert 𝑎𝑖 , which is defined as n o𝑘 (𝑂 ) ) 𝑖 𝑋𝑖(𝑂 ) = 𝑥𝑖(𝑂 (4) 𝑗 𝑗=1

Accordingly, 𝑋𝑖(𝑂 ) characterises the same feature categories defined for comprehension-level CSGs in Table 1. The distinction between comprehension- and projection-level representations therefore arises solely from the provenance horizon over which their CSGs are reconstructed. While comprehension-level CSA captures the behavioural hypotheses available at the time of alert generation, projection-level CSA incorporates the subsequent evolution of those behaviours, providing information with which to anticipate how malicious activity may unfold after an alert is raised.

5.5

Progressive CSA-Aware Alert Validation

Problem Formulation. Progressive CSA-aware alert validation determines whether an alert corresponds to genuine malicious activity by leveraging evidence accumulated at different CSA levels: perception, comprehension, and projection. Unlike conventional alert classification, the learning problem exhibits two distinctive characteristics: the evidence per alert is a variable-sized collection of behavioural hypotheses rather than a single fixed observation, and reliable negative labels are unavailable. The first characteristic follows the structure of progressive CSA generation. Perception-level CSA provides an alert-level representation 𝑥𝑖(𝑃 ) , whereas comprehension and projection stages associate the same alert with collections of CSGs, denoted by 𝐵𝑖(𝐶 ) and 𝐵𝑖(𝑂 ) , respectively. Consequently, alert validation at successive CSA (𝑃 ) levels operates over increasingly h i richer evidence spaces: 𝑥𝑖 at the perception level, 𝑥𝑖(𝑃 ) , 𝐵𝑖(𝐶 ) at the comprehension level, and h i 𝑥𝑖(𝑃 ) , 𝐵𝑖(𝐶 ) , 𝐵𝑖(𝑂 ) at the projection level. Since 𝐵𝑖(𝐶 ) and 𝐵𝑖(𝑂 ) may contain multiple CSGs whose cardinalities vary across alerts, an alert cannot be treated as a single independently labelled instance. Moreover, ground truth is available only at the alert level, not for

individual CSGs. This motivates a multi-instance learning formulation, in which each alert is represented as a bag of CSA-derived evidence, and labels are observed at the bag level [3, 5, 11]. The second characteristic follows from the nature of the available ground truth. As described in section 6.1, we use the DARPA-TC dataset together with the interaction-level annotations released by Captain, which provide positive attack-event attribution derived from documented attack kill chains but do not exhaustively verify benign interactions [24]. A confirmed attack alert thus provides a positive label, whereas an alert without ground-truth attribution cannot be treated as confirmed benign. Thus, the observed label 𝑦𝑖 ∈ {0, 1} indicates whether alert 𝑎𝑖 is positively identified as malicious, not its true class. Progressive CSA-aware alert validation is consequently formulated as a positive-unlabeled multi-instance learning (PU-MIL) problem [6, 16, 21],aiming to learn an alert-level maliciousness score and latent alert representations from partially labelled, variable-cardinality CSA evidence. Instance-Level Maliciousness Modelling. Under the PU-MIL formulation, supervision is available only at the alert level, preventing direct estimation of maliciousness for individual CSA instances. PUMIL therefore parameterises instance-level maliciousness through reconstruction-based anomaly modelling, leveraging the intuition that behavioural patterns frequently observed in the training data should be reconstructed accurately, whereas rare or weakly represented patterns incur larger reconstruction discrepancies [21]. Given a CSA-derived feature representation 𝑥𝑖(𝑙𝑗 ) ∈ 𝑋𝑖(𝑙 ) where 𝑙 ∈ {𝐶, 𝑂 }, an encoder 𝑓𝜃 (𝑙 ) (·) produces a latent representation  𝑧𝑖(𝑙𝑗 ) = 𝑓𝜃 (𝑙 ) 𝑥𝑖(𝑙𝑗 ) , subsequently reconstructed by a decoder 𝑔𝜙 (𝑙 ) (·),   where 𝑥ˆ𝑖(𝑙𝑗 ) = 𝑔𝜙 (𝑙 ) 𝑧𝑖(𝑙𝑗 ) . Behavioural deviation is quantified using the squared reconstruction error (RE), 𝑟𝑖(𝑙𝑗 ) = 𝑥𝑖(𝑙𝑗 ) − 𝑥ˆ𝑖(𝑙𝑗 )

2 2

(5)

The REs from Eq. 5 quantify relative behavioural deviation but admit no direct probabilistic interpretation. To obtain a scale-invariant anomaly measure suitable for probability calibration [21], the REs are standardised, 𝑟˜𝑖(𝑙𝑗 ) =

𝑟𝑖(𝑙𝑗 ) − 𝜇𝑟

(6) 𝜎𝑟 + 𝜖 where 𝜇𝑟 and 𝜎𝑟 denote the mean and standard deviation of REs observed during training; 𝜖 is a small constant for numerical stability. The standardised anomaly scores 𝑟˜𝑖(𝑙𝑗 ) are transformed into instance-level maliciousness probabilities through a learnable logistic calibration function 𝑝𝑖(𝑙𝑗 ) = 𝜎 (𝛼 𝑟˜𝑖(𝑙𝑗 ) + 𝛽)

(7)

where 𝜎 (·) is the sigmoid function and 𝛼, 𝛽 are learnable calibration parameters [21]. This maps reconstruction-based anomaly evidence onto the unit interval while preserving the relative ordering of anomalous instances,enabling probabilistic reasoning within the subsequent PU-MIL modelling. 𝑖 The resulting probabilities {𝑝𝑖 𝑗 (𝑙 ) }𝑘𝑗=1 obtained from Eq. 7 constitute the instance-level maliciousness scores for the CSGs associated with alert 𝑎𝑖 , providing a differentiable parameterisation of

Karunasingha et al.

instance-level maliciousness that can subsequently be aggregated into alert-level estimates under bag-level supervision. Alert-Level Probability Estimation. Progressive CSA generation may associate multiple CSGs with the same alert, so we aggregate instance-level maliciousness estimates at the alert level. Under PU-MIL, an alert is malicious whenever at least one associated CSG exhibits malicious behaviour [11], so we estimate alert-level maliciousness using a weighted Noisy-OR formulation [21]. The probability that alert 𝑎𝑖 is malicious at CSA level 𝑙 ∈ {𝐶, 𝑂 } is: (𝑙 ) 𝑘𝑖

𝑃𝑖(𝑙 ) = 1 −

Ö

1 − 𝑝𝑖(𝑙𝑗 )

 𝑤 (𝑙 ) 𝑖𝑗

(8)

𝑗=1

where 𝑝𝑖(𝑙𝑗 ) is the maliciousness probability of the 𝑗 𝑡ℎ CSG representation in 𝑋𝑖(𝑙 ) , and 𝑤𝑖(𝑙𝑗 ) is its relative contribution to alert-level maliciousness estimation. The weighted product is the probability that all CSGs associated with alert 𝑎𝑖 are benign, while its complement is the probability that at least one CSG exhibits malicious behaviour [21]. Consequently, 𝑃𝑖(𝑙 ) provides an alert-level maliciousness estimate consistent with the underlying PU-MIL assumption. The comprehension-level instance latents are also aggregated into a fixed-dimensional bag representation using the same contribution weights, (𝐶 )

𝑘𝑖

𝑧𝑖(𝐶 ) =

∑︁

) (𝐶 ) 𝑤𝑖(𝐶 𝑗 𝑧𝑖 𝑗

Since projection-level CSA typically represents anticipated rather than directly observed behaviour, its reliability may vary across alerts. We account for this through a learned gating function,   𝑔𝑖(𝑂 ) = 𝑔𝑂 𝑥𝑖(𝑃 ) , 𝑧𝑖(𝐶 ) , 0 ≤ 𝑔𝑖(𝑂 ) ≤ 1 The resulting projection-level maliciousness estimate is,  
𝑃e𝑖(𝑂 ) = 𝜎 logit 𝑃𝑖(𝑂 ) + 𝑔𝑖(𝑂 ) Δ𝑖(𝑂 )

Conditioning projection-level fusion on 𝑧𝑖(𝐶 ) enables incorporation of behavioural understanding from comprehension-level analysis for projection-level validation. The gating function regulates the influence of projected behavioural evolution according to its estimated reliability. Positive–Unlabeled Optimisation. While reconstruction-based anomaly modelling identifies behavioural deviations, RE alone cannot distinguish malicious activity from rare but benign behaviour. Consequently, optimisation must leverage attack-confirmed alerts to associate anomalous behavioural patterns with malicious activity, without assuming that alerts lacking attack attribution are benign. We distinguish the attack-confirmed alerts A + = {𝑎𝑖 | 𝑦𝑖 = 1} from the remaining unlabelled alerts A𝑈 = {𝑎𝑖 | 𝑦𝑖 = 0}. Optimisation combines unlabelled reconstruction with supervision from attack-confirmed alerts and dynamically selected reliable negatives [21]. The unlabelled component is defined as,

(9)

𝑗=1

This contribution-weighted pooling preserves each CSG’s relative importance when constructing the alert-level representation, yielding 𝑧𝑖(𝐶 ) as the learned comprehension-level bag representation. Progressive CSA Fusion. Alert validation should preserve the awareness accumulated across preceding CSA levels. PU-MIL estimates an alert-level maliciousness probability 𝑃𝑖(𝑙 ) (obtained from Eq. 8) from the CSG representations associated with alert 𝑎𝑖 . Although these capture increasingly rich behavioural evidence, they may not fully retain the alert-level information from lower CSA levels. Consequently, perception-level alert features 𝑥𝑖(𝑃 ) are incorporated as a residual refinement of the CSG-derived maliciousness estimate. At the comprehension level, perception-level alert information refines this estimate through a bounded residual correction,    Δ𝑖(𝐶 ) = 𝜆𝐶 tanh ℎ𝐶 𝑥𝑖(𝑃 ) where ℎ𝑐 (.) is a learnable perception-level fusion function and 𝜆𝐶 controls the maximum correction magnitude. Applying this correction in logit space yields the fused comprehension-level estimate,     𝑃e𝑖(𝐶 ) = 𝜎 logit 𝑃𝑖(𝐶 ) + Δ𝑖(𝐶 ) (10) Projection-level validation must additionally remain consistent with the behavioural understanding established during comprehensionlevel analysis. This is achieved by conditioning fusion on both 𝑥𝑖(𝑃 ) and 𝑧𝑖(𝐶 ) , giving the projection-level refinement:,  h i  Δ𝑖(𝑂 ) = 𝜆𝑂 tanh ℎ𝑂 𝑥𝑖(𝑃 ) , 𝑧𝑖(𝐶 )

(11)

(𝑙 )

1 L𝑈(𝑙 ) = |A𝑈 |

𝑖 ∑︁ © 1 𝑘∑︁ ª ­ 𝑟𝑖(𝑙𝑗 ) ®® ­ (𝑙 ) 𝑗=1 𝑎𝑖 ∈ A 𝑈 𝑘 𝑖 « ¬

(12)

where 𝑟𝑖(𝑙𝑗 ) is the RE of the 𝑗 𝑡ℎ CSG instance for alert 𝑎𝑖 . Thus, all unlabelled alerts contribute to learning the behavioural distribution through reconstruction, without a negative class label. To provide negative supervision without treating all unlabelled alerts as benign, a reliable-negative set R (𝑙 ) ⊆ A𝑈 is selected at each training epoch. Specifically, the |A + | unlabelled alerts with the lowest current maliciousness estimates 𝑃e𝑖(𝑙 ) are selected, such that |R (𝑙 ) | = |A + |. As the maliciousness estimates evolve during optimisation, the reliable-negative set is re-selected at each epoch. The corresponding bag-level supervised objective is,   ∑︁ L𝑃(𝑙 ) = − log 𝑃e𝑖(𝑙 ) 𝑎𝑖 ∈ A +

−

∑︁ 𝑎𝑖 ∈ R (𝑙 ) 2

  log 1 − 𝑃e𝑖(𝑙 )

+ 𝜆 𝛼 + 𝛽2

(13)

The positive component of Eq. 13 promotes high maliciousness probabilities for attack-confirmed alerts, linking reconstructionbased anomaly evidence with known attack behaviour. The reliablenegative component constrains only the currently lowest-scoring unlabelled alerts towards low maliciousness probabilities, avoiding the assumption that all unlabelled alerts are benign. Regularisation of the calibration parameters 𝛼 and 𝛽 further reduces overfitting during optimisation. The overall objective at CSA level 𝑙 is therefore, L (𝑙 ) = L𝑈(𝑙 ) + L𝑃(𝑙 )

(14)

Unified CSA Framework for Alert Validation

Although supervision is available only at the alert level, 𝑃e𝑖(𝑙 ) (obtained from Eq. 10 or Eq. 11) is derived from the underlying instancelevel probabilities through the aggregation and fusion stages. Minimising L (𝑙 ) therefore propagates bag-level supervision through these stages, encouraging higher maliciousness probabilities for behavioural patterns associated with attack-confirmed alerts [21]. Consequently, the model learns to distinguish attack-associated anomalous behaviour from the behavioural variation in unlabelled alerts, rather than relying solely on RE magnitude. Alert Ranking. Following optimisation, each alert 𝑎𝑖 is associated with a fused maliciousness estimate 𝑃e𝑖(𝑙 ) (obtained in Eq. 10 or Eq. 11), which reflects the likelihood that the alert corresponds to malicious activity given the available CSA information. Because analyst resources are limited, alert validation ultimately becomes a prioritisation task: investigate alerts most likely to be malicious first. Alerts are ranked by their fused maliciousness estimates:   𝜋 (𝑙 ) = argsort 𝑃e1(𝑙 ) , 𝑃e2(𝑙 ) , . . . , 𝑃e𝑁(𝑙 )

(15)

where 𝜋 (𝑙 ) denotes the ranking of alerts in descending order of maliciousness probability at CSA level 𝑙. Alerts with larger 𝑃e𝑖(𝑙 ) receive higher investigation priority.

6

Evaluation and Results

We evaluate the progressive CSA information generation framework for alert validation, examining whether higher levels of CSA information, from perception to comprehension to projection, improve alert-validation effectiveness and how projection-level CSA effectiveness varies with the projection horizon Δ𝑡. The evaluation addresses the following research questions: RQ1: How does progressively richer CSA information influence alert validation effectiveness? RQ2: How does the projection horizon Δ𝑡 influence the usefulness of projection-level CSA information for alert validation?

6.1

Experimental Setup

Datasets. Dataset selection is constrained by the systems used to instantiate the framework: Captain generates perception-level CSA, while Kairos provides the provenance analysis used to construct comprehension- and projection-level CSA. Consequently, evaluation requires datasets supported by both systems. We use the DARPA-TC Engagement 3 (E3)2 and Engagement 5 (E5)3 datasets, widely used benchmarks for provenance-based security analysis. Table 2 summarises the overlap in Captain and Kairos support across engagements and provenance sources. Although CadetsE3, Cadets-E5, and Theia-E3 are compatible with the end-to-end framework, the pretrained Captain model for Cadets-E5 is not

2 DARPA-TC Engagement Data Release, https://github.com/darpa-i2o/Transparent-

Computing/blob/master/README-E3.md, last accessed May 2026. 3 DARPA-TC Engagement Data Release, https://github.com/darpa-i2o/transparentcomputing, last accessed May 2026.

Table 2: DARPA-TC dataset coverage and the datasets selected for end-to-end evaluation. Captain Kairos Data Source Cadets ClearScope FiveDirections Theia Trace

Selected Train Evaluation E3

E5

E3 E5

✓ – – ✓ ✓

✓ – – – ✓

✓ ✓ – ✓ –

✓ ✓ – ✓ –

E3 – – E3 –

7–12 – – 10–11 –

6 – – 12 –

publicly available. We therefore trained a Captain model on highmemory computing resources.4 However, the trained model produced only two attack-confirmed alerts, which were too few positive instances to support meaningful PU-MIL training and evaluation. Consequently, we excluded Cadets-E5 from subsequent experiments. All experiments are therefore conducted on Cadets-E3 and Theia-E3. These datasets provide complementary provenance settings with substantial variation in attack campaigns, benign workloads, scale, entity composition, and event structure, enabling evaluation across heterogeneous settings while remaining compatible with the CSA pipeline. Training and Evaluation Protocol. We adopt a cross-day protocol, training and evaluating on disjoint provenance structures. Compared with random alert-wise splitting, this tests generalisation to unseen attack executions while preventing structure leakage. Training and evaluation recordings are summarised in Table 2. For each dataset, we train a PU-MIL model using CSA information from the training days and evaluate it on the held-out day. Although DARPA-TC attack campaigns span multiple days, each day is an independent execution of the attack scenario rather than part of a continuous multi-day attack trace. The training days thus provide multiple independent executions for learning behavioural patterns; the held-out day assesses generalisation to an unseen execution. Comprehension- and projection-level evaluations differ only in the temporal scope of provenance available for CSA generation. Comprehension-level evaluation uses only provenance preceding the alert detection time (Δ𝑡 = 0); projection-level evaluation extends the observation window beyond detection by Δ𝑡 > 0. Varying Δ𝑡 lets us quantify how increasing future system behaviour affects alert validation, with all other factors held constant. We modified the protocol for Theia E3: a preliminary analysis showed Captain generating many false-positive alerts outside the documented attack period, substantially increasing provenance reconstruction cost without adding attack-relevant evidence. Therefore, we restricted comprehension- and projection-level evaluation to alerts within the documented attack window. This preprocessing improves computational efficiency only; it does not alter the ground truth or evaluated attack alerts, nor does it introduce information unavailable during alert validation. Since the original DARPA-TC datasets lack interaction-level ground-truth annotations, we use the interaction-level annotations 4 The trained Captain-E5 model will be made publicly available upon submission to a

peer-reviewed conference.

Karunasingha et al.

released with the Captain implementation only for PU-MIL optimisation and final evaluation, not during CSA generation (perceptionlevel alert generation or provenance reconstruction). Consistent with positive–unlabelled learning, we treat only attack-confirmed alerts as positive during optimisation; all others are unlabelled. Baselines. As described in Section 5.2, alert generation produces an unordered set with no mechanism for estimating the relative likelihood that individual alerts are malicious [24]. Perception-level CSA alone yields no ranking or validation score comparable to the proposed framework, and therefore cannot serve as an alert validation baseline. By contrast, the provenance-based attack reconstruction used for comprehension- and projection-level CSA produces anomaly scores when detecting anomalous activity [4]. These are the primary quantitative suspiciousness indicators available before CSA-aware learning, making them appropriate baselines. These anomaly scores originate from attack-centric provenance reconstruction rather than alert-centric validation. Since provenance reconstruction is anchored on the alert timestamp for both comprehension- and projection-level CSAs, multiple alerts tied to the same attack may share the same anomalous queue and reconstructed CSGs. The scores therefore reflect the suspiciousness of reconstructed attack behaviour rather than the relative maliciousness of individual alerts. PU-MIL addresses this by learning alert-specific validation scores from the combined CSA representations. Evaluation Metrics. We treat alert validation as a ranking problem, reflecting the operational setting where analysts investigate a limited number of alerts in descending order of estimated maliciousness. The primary metric is Recall@𝐾 (R@𝐾), the proportion of attack-confirmed alerts within the top-𝐾 positions: R@K =

|A + ∩ TopK(𝜋)| |A + |

(16)

where A + is the set of attack-confirmed alerts and TopK(𝜋) the first 𝐾 alerts in ranking 𝜋. Higher R@𝐾 means more attack-confirmed alerts fall within a fixed investigation budget. We report R@𝐾 at multiple investigation depths to capture both early-ranking effectiveness and broader alert coverage: smaller 𝐾 reflects performance under tightly constrained analyst capacity, while larger 𝐾 shows how quickly the ranking recovers all attack-confirmed alerts. R@𝐾 evaluates ranking only at discrete cut-offs. Mean Positive Rank (MPR) is a complementary measure of overall placement: MPR =

∑︁ 1 rank𝜋 (𝑎𝑖 ) + |A | +

(17)

𝑎𝑖 ∈ A

where rank𝜋 (𝑎𝑖 ) is the position of attack-confirmed alert 𝑎𝑖 in the ranked list. Lower MPR means attack-confirmed alerts surface earlier, on average. We use R@𝐾 as the primary operational metric for malicious-alert coverage under different budgets, and MPR as a complementary measure of overall ranking quality. To understand how projection-level CSA affects individual attackconfirmed alerts, we also analyse alert-level rank transitions. For alert 𝑎𝑖 , the Projection-Induced Rank Shift (PIRS) at projection horizon Δ𝑡 is defined as 𝐺𝑖 (Δ𝑡) = 𝑟𝑖 (0) − 𝑟𝑖 (Δ𝑡)

(18)

Table 3: Alert validation across Comprehension (C) and Projection (O) levels on Cadets E3 and Theia E3 using Δ𝑡 = 60 and 40 min. Higher Recall@𝐾 and lower MPR are better. Recall@𝐾 ↑

Dataset CSA level Evidence Use 5

10

MPR ↓

20

C

Queue Max. CSG PU-MIL

0.286 0.286 0.571 0.143 0.286 0.571 0.429 0.571 0.714

17.57 18.00 13.57

O60

Queue Max. CSG PU-MIL

0.000 0.000 0.286 0.143 0.286 0.571 0.429 0.714 0.714

36.14 20.00 12.57

C

Queue Max. CSG PU-MIL

0.000 0.000 0.000 0.000 0.250 0.250 0.750 0.750 0.750

166.75 95.00 22.00

O40

Queue Max. CSG PU-MIL

0.000 0.000 0.500 0.000 0.000 0.250 0.500 0.500 1.000

163.00 176.75 8.50

Cadets

Theia

where 𝑟𝑖 (0) is the rank under comprehension-level CSA (Δ𝑡 = 0) and 𝑟𝑖 (Δ𝑡) the rank after incorporating projection-level CSA. A positive value moves the alert closer to the front of the investigation queue; a negative value moves it in the opposite direction. Unlike R@𝐾 and MPR, which measure aggregate prioritisation, rank gain shows how individual alerts respond to different Δ.

6.2

Impact of CSA Richness on Alert Validation

RQ1 examines how progressively richer CSA influences alert validation effectiveness. Since perception-level CSA establishes awareness of suspicious situations but provides no alert-specific prioritisation score, our quantitative analysis focuses on the additional evidence introduced at the comprehension and projection levels. We examine these contributions from two complementary perspectives: the anomaly-score-based rankings characterise how attack-relevant evidence manifests in the generated provenance context; PU-MIL assesses how effectively the same CSA information can be exploited to improve alert validation. This separates the contribution of progressively richer CSA from the downstream validation mechanism. For the projection-level comparison, we use an attack-timescaleinformed reference horizon for each environment, selected from the documented temporal extent of the attack campaign2 : Δ𝑡 = 60 min for Cadets E3 and Δ𝑡 = 40 min for Theia E3, reflecting the different timescales over which post-alert attack behaviour unfolds in each environment. These horizons are representative configurations for comparing comprehension- and projection-level CSA, not choices optimised for validation performance; Section 6.3 systematically examines the effect of varying Δ𝑡. Comprehension provides alert-discriminative causal context. Across both evaluation environments, comprehension-level CSA transforms an isolated alert into behavioural hypotheses describing its causal provenance context. Anomaly-derived reference rankings show that this reconstructed context captures suspicious behaviour; however, anomalousness alone does not indicate whether an alert is attack-related. On Cadets E3, queue- and maximum-CSG anomaly rankings recover 28.6% and 14.3% of the attack-confirmed alerts

(b) Cadets: MPR

0

20

40

60

80 100

Δ𝑡 (minutes)

Recall@5

35 30 25 20 15 10 0

20

40

60

(c) Theia: 𝐾 = 5

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

MPR

(a) Cadets: 𝐾 = 5

0.6 0.5 0.4 0.3 0.2 0.1 0

MPR

Recall@5

Unified CSA Framework for Alert Validation

0

80 100

20

40

60

80 100

250 200 150 100 50 0

(d) Theia: MPR

0

20

Δ𝑡 (minutes)

Δ𝑡 (minutes)

40

60

80 100

Δ𝑡 (minutes)

Figure 4: Effect of projection horizon on alert prioritisation across Cadets E3 and Theia E3. Panels (a) and (b) report Recall@5 and mean positive rank (MPR) for Cadets E3; Panels (c) and (d) report the corresponding metrics for Theia E3. PU-MIL ( ), Queue ( ), and Max-CSG ( ) denote the three ranking methods. Δ𝑡 = 0 corresponds to comprehension-level CSA, while Δ𝑡 > 0 incorporates post-alert provenance activity. Higher Recall@5 and lower MPR indicate better prioritisation. within the top five, respectively. On Theia E3, neither ranking recovers an attack-confirmed alert within the top five, despite maximumCSG ranking recovering 25% within the top ten. Thus, reconstructed causal context can expose suspicious attack behaviour without providing a reliable alert-level ordering. PU-MIL provides a complementary perspective by explicitly learning how this comprehension-level evidence relates to alert validation. Using the same information, it places 42.9% of the Cadets attack-confirmed alerts and 75% of the Theia attack-confirmed alerts within the top five, with MPRs of 13.57 and 22.00, respectively. These results should not be interpreted primarily as outperforming the anomaly-derived rankings, since the two mechanisms optimise different objectives. Rather, they show that comprehension-level causal evidence can be explicitly associated with attack-confirmed alerts once a model is trained for alert validation. In other words, comprehension-level CSA contributes more than a suspiciousness estimate; by reconstructing the causal provenance context associated with an alert, it provides discriminative behavioural evidence for determining whether it is actually attack-related. A learning-based alert-validation model can exploit this to prioritise attack-confirmed alerts more effectively than anomaly severity alone.

performance across investigation depths. The effect is more pronounced on Theia E3. At the 40-minute horizon, PU-MIL changes from R@5/R@10/R@20 of 0.750/0.750/0.750 at comprehension to 0.500/0.500/1.000, while MPR decreases substantially from 22.00 to 8.50. Although fewer attack-confirmed alerts remain within the stricter top-5 and top-10 cutoffs, all are recovered within the top 20, with a substantial improvement in overall ranking positions. This suggests that projection-level CSA redistributes evidential support across alerts rather than uniformly strengthening already prioritised ones. Projection-level CSA is most valuable when comprehensionlevel context is insufficient to establish attack relevance, providing future behavioural evidence that helps distinguish difficult attack alerts from benign alternatives.

6.3

Impact of Projection Horizon

RQ2 examines how the temporal extent of projection-level CSA influences its usefulness for alert validation. We vary the projection horizon Δ𝑡 from 10 to 100 minutes, with Δ𝑡 = 0 corresponding to comprehension-level CSA, and evaluate its effect on both Cadets E3 and Theia E3 using Recall@5 and MPR, with a common Recall@𝐾 cutoff across both datasets. Figure 4 reports the horizon-dependent behaviour. Panels (a) and (b) show Recall@5 and MPR, respectively, Projection-level CSA improves prioritisation of difficult alerts. Projectionfor Cadets E3, while Panels (c) and (d) report the corresponding level CSA extends causal reconstruction beyond alert detection metrics for Theia E3. time, exposing how the associated behavioural context evolves. Across both environments, the anomaly-derived reference rankProjection utility is inherently horizon-dependent. Figure 4(a–d) ings indicate that incorporating projected provenance does not reveals this first through the anomaly-derived reference rankings. translate directly into stronger anomalousness-based prioritisation. Across both datasets, Queue and Max. CSG rankings vary nonOn Cadets E3, extending the observation horizon to 60 minutes monotonically as Δ𝑡 increases: extending the observation horizon reduces queue-anomaly R@20 from 0.571 to 0.286 and increases can improve, preserve, or degrade prioritisation of attack-confirmed MPR from 17.57 to 36.14, while maximum-CSG ranking preserves alerts. Since these rankings directly reflect the provenance reconits Recall@𝐾 values but increases MPR from 18.00 to 20.00. On structed at each horizon, this behaviour shows that progressively Theia E3, the 40-minute horizon leaves queue-level early recall low extending the temporal scope does not produce a correspondingly despite increasing R@20 to 0.500, while maximum-CSG ranking monotonic increase in validation-relevant anomalous evidence. Prodecreases from R@10=0.250 (comprehension) to 0.000 (projection), jection therefore changes the behavioural evidence available for with MPR rising from 95.00 to 176.75. Projection-level CSA does not analysis rather than simply accumulating uniformly useful context. inherently make attack-confirmed alerts appear more anomalous. PU-MIL exhibits the same horizon dependence when this inforInstead, it contributes subsequent behavioural evidence. mation is explicitly learned for alert validation. Its performance PU-MIL shows that this additional evidence can still inform alert does not improve monotonically with Δ𝑡 on either dataset, despite validation. On Cadets E3, projection preserves R@5 and R@20 at achieving substantial gains at particular intermediate horizons. On 0.429 and 0.714, respectively, while increasing R@10 from 0.571 to Cadets E3, projection can improve both Recall@5 and MPR rela0.714 and reducing MPR from 13.57 to 12.57. Projection therefore tive to comprehension-level information, whereas on Theia E3 the selectively refines alert ordering rather than uniformly improving benefit is more pronounced in the overall ranking: at Δ𝑡 = 40 min,

Karunasingha et al.

𝐴1 (2)

−2 +1

+1

𝐴2 (3)

+0

+0

−1 +0

−1 −1 +0

𝐴3 (4)

+2

+0

+2

+2

+1

−1 +2

+1

+0

𝐴4 (7)

−5 −5 −4 −3 +0

+1

+3

+2

+1

+0

𝐴5 (14)

−1 +9

+6

+6

+6

+3

+4

𝐴6 (32)

+7 +11 −14 −21 −12 +10 −2 −16 −15 −12

𝐴7 (33)

+9 +10 −15 −19 −12 −11 +12 −13 −13 +10 10

20

+1

30

−2 +1

+2

−1 +5

40

50

+1

60

+1

70

−1 +1

+1

−1 −1 +1

80

90

20 10 0 −10 −20

100

Figure 5: Projection-induced rank shift on Cadets E3. Alerts 𝐴1 –𝐴7 are shown in comprehension-rank order. Positive values improve prioritisation; negative values degrade it. MPR decreases from 22.00 to 8.50 even though Recall@5 does not exceed its comprehension-level value. Thus, task-specific learning can exploit projection-level information to improve alert validation, but it does not eliminate the underlying temporal sensitivity. Both anomaly-based rankings and PU-MIL show that projection-level CSA derives its value not from additional future activity, but from exposing validation-relevant evidence at specific projection horizons. Projection affects individual alerts heterogeneously. Figure 5 examines the alert rank transitions underlying the aggregate Cadets E3 results. The effect of varying Δ𝑡 is not uniform across attackconfirmed alerts. Alerts highly prioritised using comprehensionlevel CSA remain relatively stable across projection horizons, exhibiting only modest rank adjustments. In contrast, lower-ranked alerts exhibit substantially larger non-monotonic rank changes: the same alert may improve at some horizons and worsen at others. Consequently, horizon sensitivity arises primarily from greater ranking volatility in initially lower-ranked alerts rather than from reordering already highly prioritised alerts. Projection-level CSA has the greatest impact when attack relevance remains uncertain after comprehension-level analysis, whereas alerts already strongly supported by that evidence exhibit little sensitivity to projection horizon.

7

Discussion

This paper investigated whether progressively richer CSA information improves alert validation and how its utility varies with temporal scope. The results show that richer CSA levels can provide validation-relevant evidence beyond anomalousness alone, while the benefits of projection-level CSA depend strongly on the behavioural context captured within the observation horizon. This section discusses the implications for understanding progressive CSA, supporting SOC alert validation, and guiding future research.

7.1

Understanding Progressive CSA Behaviour

The results in Sections 6.2-6.3 reveal two complementary properties of progressive CSA. First, enriching an alert with causal and subsequent behavioural context provides additional evidence for validation, but behavioural anomalousness alone does not uniformly reflect this contribution. Instead, PU-MIL demonstrates that the generated CSA information captures validation-relevant signals beyond those reflected in anomalousness scores. Second, the

usefulness of projection-level information depends strongly on the horizon. Across both Cadets E3 and Theia E3, extending Δ𝑡 does not produce a monotonic improvement in alert prioritisation, because increasing Δ𝑡 does not simply provide more of the same evidence. Instead, it changes which portion of the post-alert system evolution becomes available for reasoning. Different horizons can therefore expose behaviour with different relevance to the alert under validation. The corresponding non-monotonic behaviour of the anomaly-derived reference rankings supports this interpretation: horizon sensitivity is already present in the reconstructed provenance evidence and is subsequently inherited by task-specific learning. The value of projection thus depends on the relevance of the behavioural context exposed within the observation horizon rather than on the amount of additional context alone.

7.2

Implications for SOC Alert Validation

The findings suggest that progressive CSA can support evidencedirected allocation of SOC investigation effort, while raising an important operational question: when should an alert be escalated to a richer CSA level? Our results indicate that comprehensionlevel evidence may suffice for some alerts, whereas others benefit from projection-level CSA. However, the current framework does not make this determination. This motivates adaptive progression policies in which the strength or uncertainty of validation evidence determines whether further CSA enrichment is warranted, rather than requiring every alert to undergo the full analysis pipeline. A related challenge concerns how far the projection should extend, if at all. The horizon sensitivity observed in Section 6.3 rules out a simple “longer is better” strategy. A fixed observation horizon may include irrelevant future context or fail to capture information important for alert validation, whether inferred through prediction or revealed through subsequent system behaviour. In operational settings, the projection horizon should be an adaptive decision that balances the expected validation value of additional evidence against the cost of acquiring, processing, or reasoning over it. Learning when to invoke projection, how far it should extend, and when sufficient evidence has been acquired remains a key challenge for operationalising progressive CSA in practice.

7.3

Limitations and Future Directions

Our findings should be interpreted in light of three limitations. First, end-to-end evaluation is constrained to datasets supported by all components of the progressive CSA pipeline. As a result, experiments are limited to Cadets E3 and Theia E3. Although these datasets represent distinct provenance environments and attack campaigns, they are insufficient to establish that the observed behaviour generalises across diverse SOC environments. Second, projection is based on subsequently observed provenance activity rather than forecasts of future attacker behaviour. While this enables a controlled evaluation of the value of post-alert behavioural evidence for validation, it does not establish that such evidence can be predicted accurately at alert time. Future work should investigate attacker-behaviour forecasting to determine whether predicted future context yields similar benefits. Finally, the framework relies on predefined projection horizons and fixed CSA progression levels.

Unified CSA Framework for Alert Validation

The performance variation observed across CSA levels and projection horizons suggests that these choices influence both validation effectiveness and operational efficiency. Future work should investigate adaptive progression and stopping policies that determine when projection is needed and how much future context to acquire based on available evidence and uncertainty. Collectively, these limitations motivate future research on adaptive and predictive progressive CSA frameworks for alert validation.

8

Conclusion

This work investigated how progressively richer CSA supports alert validation in SOCs. We introduced a unified framework that integrates perception-, comprehension-, and projection-level awareness, operationalised through provenance-based analysis and weakly supervised alert validation. Experimental evaluation demonstrates that comprehension-level CSA consistently improves validation by providing causal attack context beyond the initial alert. In contrast, the benefits of projection-level CSA are conditional rather than cumulative. Because projection-level awareness is inherently predictive, its effectiveness depends on the temporal horizon over which subsequent behaviour is anticipated. These findings challenge the assumption that progressively richer CSA necessarily leads to better validation outcomes. Instead, they reframe alert validation as an awareness-allocation problem, where additional evidence should be acquired and presented only when it actively improves decision-making. Future SOC assistance systems should move beyond providing ever-richer context towards adaptively delivering the specific level and type of awareness that an alert demands. Future work should focus on learning when higher levels of CSA are required, which evidence is most informative for different alert types, and when awareness is sufficient to support timely and accurate analyst decisions.

References [1] Bushra A Alahmadi, Louise Axon, and Ivan Martinovic. 2022. 99% false positives: A qualitative study of { SOC } analysts’ perspectives on security alarms. In 31st USENIX Security Symposium (USENIX Security 22). 2783–2800. [2] Paul Barford, Marc Dacier, Thomas G Dietterich, Matt Fredrikson, Jon Giffin, Sushil Jajodia, Somesh Jha, Jason Li, Peng Liu, Peng Ning, et al. 2009. Cyber SA: Situational awareness for cyber defense. In Cyber Situational Awareness: Issues and Research. Springer, 3–13. [3] Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. 2018. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition 77 (2018), 329–353. [4] Zijun Cheng, Qiujian Lv, Jinyuan Liang, Yan Wang, Degang Sun, Thomas Pasquier, and Xueyuan Han. 2024. Kairos: Practical intrusion detection and investigation using whole-system provenance. In 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 3533–3551. [5] Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. 1997. Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence 89, 1-2 (1997), 31–71. [6] Charles Elkan and Keith Noto. 2008. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 213–220. [7] Mica R Endsley. 2017. Toward a theory of situation awareness in dynamic systems. In Situational awareness. Routledge, 9–42. [8] Wajih Ul Hassan, Shengjian Guo, Ding Li, Zhengzhang Chen, Kangkook Jee, Zhichun Li, and Adam Bates. 2019. Nodoze: Combatting threat alert fatigue with automated provenance triage. In Network and Distributed System Security Symposium (NDSS). [9] Md Nahid Hossain, Sadegh M Milajerdi, Junao Wang, Birhanu Eshete, Rigel Gjomemo, R Sekar, Scott Stoller, and VN Venkatakrishnan. 2017. { SLEUTH } : Real-time attack scenario reconstruction from { COTS } audit data. In 26th USENIX Security Symposium (USENIX Security 17). 487–504.

[10] Md Nahid Hossain, Sanaz Sheikhi, and R Sekar. 2020. Combating dependence explosion in forensic analysis using alternative tag propagation semantics. In 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 1139–1155. [11] Maximilian Ilse, Jakub Tomczak, and Max Welling. 2018. Attention-based deep multiple instance learning. In International Conference on Machine Learning. PMLR, 2127–2136. [12] Baoxiang Jiang, Tristan Bilot, Nour El Madhoun, Khaldoun Al Agha, Anis Zouaoui, Shahrear Iqbal, Xueyuan Han, and Thomas Pasquier. 2025. { ORTHRUS } : Achieving High Quality of Attribution in Provenance-based Intrusion Detection Systems. In 34th USENIX Security Symposium (USENIX Security 25). 7173–7192. [13] Liuyue Jiang, Asangi Jayatilaka, Mehwish Nasim, Marthie Grobler, Mansooreh Zahedi, and M Ali Babar. 2022. Systematic literature review on cyber situational awareness visualizations. IEEE Access 10 (2022), 57525–57554. [14] Navodika Karunasingha, Mohan Baruwal Chhetri, Surya Nepal, Cecile Paris, and Salil S Kanhere. 2025. SoK: AI Support for Analyst Situation Awareness in Security Operation Centres. In 2025 European Symposium on Usable Security (EuroUSEC). IEEE, 151–163. [15] Samuel T King and Peter M Chen. 2005. Backtracking intrusions. ACM Transactions on Computer Systems (TOCS) 23, 1 (2005), 51–76. [16] Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. 2017. Positive-unlabeled learning with non-negative risk estimator. Advances in Neural Information Processing Systems 30 (2017). [17] Zhenyuan Li, Qi Alfred Chen, Runqing Yang, Yan Chen, and Wei Ruan. 2021. Threat detection and investigation with system-level provenance graphs: A survey. Computers & Security 106 (2021), 102282. [18] Haitian Liu and Rong Jiang. 2023. A causal graph-based approach for APT predictive analytics. Electronics 12, 8 (2023), 1849. [19] Sadegh M Milajerdi, Birhanu Eshete, Rigel Gjomemo, and VN Venkatakrishnan. 2019. Poirot: Aligning attack behavior with kernel audit records for cyber threat hunting. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 1795–1812. [20] Sadegh M Milajerdi, Rigel Gjomemo, Birhanu Eshete, Ramachandran Sekar, and VN Venkatakrishnan. 2019. Holmes: real-time apt detection through correlation of suspicious information flows. In 2019 IEEE Symposium on Security and Privacy (SP). IEEE, 1137–1152. [21] Lorenzo Perini, Vincent Vercruyssen, and Jesse Davis. 2023. Learning from positive and unlabeled multi-instance bags in anomaly detection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1897–1906. [22] Shahroz Tariq, Mohan Baruwal Chhetri, Surya Nepal, and Cecile Paris. 2025. Alert fatigue in security operations centres: Research challenges and opportunities. Comput. Surveys 57, 9 (2025), 1–38. [23] Thijs Van Ede, Hojjat Aghakhani, Noah Spahn, Riccardo Bortolameotti, Marco Cova, Andrea Continella, Maarten Van Steen, Andreas Peter, Christopher Kruegel, and Giovanni Vigna. 2022. Deepcase: Semi-supervised contextual analysis of security events. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 522–539. [24] Lingzhi Wang, Xiangmin Shen, Weijian Li, Zhenyuan Li, R. Sekar, Han Liu, and Yan Chen. 2025. Incorporating Gradients to Rules: Towards Lightweight, Adaptive Provenance-based Intrusion Detection. In Network and Distributed System Security Symposium (NDSS). [25] Xiaoyu Wang, Xiaobo Yang, Xueping Liang, Xiu Zhang, Wei Zhang, and Xiaorui Gong. 2024. Combating alert fatigue with AlertPro: Context-aware alert prioritization using reinforcement learning for multi-step attack detection. Computers & Security 137 (2024), 103583. [26] Limin Yang, Zhi Chen, Chenkai Wang, Zhenning Zhang, Sushruth Booma, Phuong Cao, Constantin Adam, Alexander Withers, Zbigniew Kalbarczyk, Ravishankar K Iyer, et al. 2024. True attacks, attack attempts, or benign triggers? An empirical measurement of network alerts in a security operations center. In 33rd USENIX Security Symposium (USENIX Security 24). 1525–1542. [27] Tiantian Zhu, Jie Ying, Tieming Chen, Chunlin Xiong, Wenrui Cheng, Qixuan Yuan, Aohan Zheng, Mingqi Lv, and Yan Chen. 2024. Nip in the bud: Forecasting and interpreting post-exploitation attacks in real-time through cyber threat intelligence reports. IEEE Transactions on Dependable and Secure Computing 22, 2 (2024), 1431–1447. [28] Michael Zipperle, Florian Gottwalt, Elizabeth Chang, and Tharam Dillon. 2022. Provenance-based intrusion detection systems: A survey. Comput. Surveys 55, 7 (2022), 1–36.

Open Science To support reproducibility and facilitate future research, we plan to publicly release the implementation of our progressive CSA-aware PU-MIL framework and the trained Captain model for Cadets-E5 upon submission to a peer-reviewed conference.

Karunasingha et al.

A

Alternative Realisations of Progressive CSA Generation A.1 Perception-Level CSA Generation Perception-level CSA generation can be realised through both knowledge-driven and data-driven provenance analysis paradigms. As shown in Table 4, the primary distinction lies in the source of knowledge used to infer suspiciousness from provenance data. Knowledge-driven approaches encode security semantics directly into provenance reasoning, for example, by propagating integrity or trust labels through provenance graphs to identify suspicious interactions [9, 10, 24]. In contrast, data-driven approaches infer suspiciousness from patterns learned directly from provenance data, such as identifying anomalous interactions using learned provenance representations [4, 12]. Both paradigms ultimately generate perception-level awareness of suspicious entities and interactions, but they expose different trade-offs for downstream CSA construction. Knowledge-driven approaches provide strong causal interpretability because suspiciousness remains explicitly associated with provenance entities and interactions throughout the detection process. In contrast, data-driven approaches can capture complex dependencies and previously unseen attack behaviours but often rely on learned representations whose security semantics may be less transparent. Consequently, knowledge-driven approaches provide a particularly suitable foundation for progressive CSA construction, where higher CSA levels depend on preserving and reasoning over the provenance context surrounding suspicious activity.

A.2

Comprehension-Level CSA Generation

Comprehension-level CSA generation can be realised through several provenance analysis paradigms that differ in how they reconstruct attack-relevant behavioural context from suspicious system activity. As shown in Table 5, the primary distinction lies in the mechanism used to construct causally meaningful provenance structures surrounding an alert. Seed-driven approaches reconstruct attack context by expanding causal dependencies outward from a suspicious event, alert, or detection point, recovering the provenance activity directly connected to the initial suspicious behaviour. Knowledge-guided approaches reconstruct attack context by constraining provenance reconstruction using predefined security knowledge, such as attack templates, threat intelligence, or MITRE ATT&CK tactics, techniques, and procedures (TTPs), thereby retaining only security-relevant provenance activity. Learning-based approaches reconstruct attack context by learning anomalous behavioural patterns directly from provenance data, identifying and summarising suspicious provenance activity into compact behavioural representations without relying on manually engineered security knowledge. Although these approaches differ in their reconstruction mechanisms, they all seek to generate causally meaningful behavioural representations that extend perception-level alerts with comprehension-level causal awareness. Seed-driven and knowledge-guided approaches preserve strong causal relationships by explicitly reconstructing provenance dependencies surrounding suspicious activity. Seed-driven reconstruction maximises causal completeness but often introduces large amounts

of benign provenance context, whereas knowledge-guided reconstruction improves security relevance by constraining reconstruction using expert knowledge. However, both approaches ultimately depend on predefined expansion criteria or manually engineered security semantics, which may limit scalability and generalisability across diverse attack scenarios. In contrast, learning-based provenance summarisation reconstructs comprehension-level CSA by identifying anomalous behavioural patterns directly from provenance activity rather than relying on predefined dependency expansion or handcrafted attack knowledge. Consequently, the resulting provenance summaries are typically more compact while still preserving the causal relationships necessary for causal awareness. This provides a particularly suitable foundation for progressive CSA construction, where higher CSA levels require compact behavioural representations that can be efficiently extended, integrated, and subsequently exploited during downstream alert validation.

A.3

Projection-Level CSA Generation

Projection-level CSA generation extends analysts’ awareness beyond the current attack state to anticipate how suspicious behaviour may subsequently evolve. Compared with perception- and comprehension-level CSA generation, relatively few approaches have explicitly addressed projection-level CSA in provenance-based security analysis. Existing work predominantly generates projectionlevel CSA by forecasting the evolution of future attacks from the currently observed attack context. As summarised in Table 6, these approaches fall broadly into probabilistic forecasting and temporal sequence learning. Probabilistic forecasting estimates future attack evolution using probabilistic reasoning over the currently observed attack context, whereas temporal sequence-learning approaches learn future attack behaviour directly from historical provenance sequences. Although both approaches enable the generation of projection-level CSA information, explicit attack forecasting remains inherently challenging because future system behaviour cannot be directly observed at prediction time. Consequently, forecasting performance is affected by uncertainty in future attack evolution, semantic gaps between high-level attack objectives and the low-level provenance activity, scarcity of representative attack data, and cumulative prediction errors over extended forecasting horizons [27]. Retrospective future-context construction provides an alternative realisation by extending the provenance observation horizon beyond the alert-detection point to incorporate subsequently observed system activity. Rather than forecasting future attack behaviour, this approach progressively reveals future provenance context while preserving causal continuity, as in comprehensionlevel provenance analysis. Although retrospective future-context construction does not provide real-time attack prediction, it offers a causally grounded mechanism for constructing projection-level CSA without the uncertainty associated with speculative futurestate forecasting. Consequently, it provides a suitable foundation for progressively integrating projection-level CSA into downstream CSA-aware alert validation.

Unified CSA Framework for Alert Validation

Table 4: Alternative realisations of perception-level CSA generation in provenance-based intrusion detection systems. Approach

Knowledge Source

Inference Mechanism

Representative Systems

Implications for Progressive CSA Construction

KnowledgeDriven

Explicit security knowledge encoded into provenance reasoning

Reasoning over security semantics associated with provenance activity

SLEUTH [9], MORSE [10], CAPTAIN [24]

Data-Driven

Patterns learned directly from provenance activity

Identifying deviations from learned provenance behaviour

KAIROS [4], THRUS [12]

Preserves causal interpretability and provenance semantics, facilitating downstream CSA expansion, summarisation, and reasoning Captures complex attack behaviours and previously unseen activity, but the resulting evidence may be less directly interpretable

OR-

Table 5: Alternative realisations of comprehension-level CSA generation in provenance-based security analysis.

Approach

Core Principle

Representative Systems

Strengths

Implications for Progressive CSA Construction

Seed-Driven Dependency Expansion

Expand provenance dependencies from suspicious seed events

BackTracker NoDoze [8]

[15],

Preserve causal relationships and provide comprehensive attack context around suspicious activity.

May reconstruct large provenance graphs that contain substantial benign but causally reachable activity, necessitating additional pruning or ranking mechanisms.

KnowledgeGuided Provenance Reconstruction

Apply expert knowledge, attack templates, threat intelligence, TTPs

POIROT HOLMES [20]

[19],

Produces security-grounded attack context with high semantic interpretability.

Depends on manually engineered attack knowledge, curated threat intelligence, and expert-defined rules, limiting adaptability to previously unseen attacks.

Learning-Based Provenance Summarisation

Learn anomalous provenance behaviour and summarise into compact graphs

KAIROS [4], THRUS [12]

OR-

Generates compact, scalable behavioural summaries while reducing reliance on handcrafted attack knowledge.

Effectiveness depends on the quality and representativeness of training data and learned behavioural models.

Table 6: Alternative realisations of projection-level CSA generation.

Approach

Core Principle

Representative Work

Strengths

Limitations

Probabilistic Forecasting

Predicts future attack evolution using probabilistic models that estimate future attack states from observed attack behaviour.

Liu et al. [18]

Provides explicit reasoning about how future attacks progress under uncertainty.

Forecast accuracy depends on the underlying probabilistic model, and uncertainty increases over longer prediction horizons.

Temporal Sequence Learning

Learns temporal attack evolution directly from historical attack or provenance sequences to forecast future attack behaviour.

EFI [27]

Captures the complex temporal evolution of attacks while preserving provenance semantics.

Requires representative attack data and remains susceptible to longhorizon prediction errors.

Retrospective Future-Context Construction

Constructs projection-level CSA by extending provenance analysis beyond the alert-detection point using subsequently observed future system activity rather than explicitly forecasting future attack behaviour.

This work

Preserves causal consistency while avoiding uncertainty associated with speculative future-state prediction.

Provides retrospective rather than real-time projection.

Karunasingha et al.

B

Extended Projection-Horizon Results

In Section 6.2 evaluation focuses on selected operating points to characterise how the usefulness of projection-level CSA varies with the projection horizon. For completeness, this appendix extends the analysis to broader analyst investigation budgets. Figure 6 reports Recall@𝐾 for Cadets E3 at 𝐾 ∈ {10, 20, 30}, while Figure 7 provides a broader analysis for Theia E3 at 𝐾 ∈ {10, 20, 30, 40, 50, 100}. The extended results are consistent with the principal observation from the main RQ2 analysis rather than indicating a different behaviour at larger investigation budgets. In particular, increasing the projection horizon does not produce a uniformly monotonic improvement in Recall@𝐾. Improvements, degradations, and relatively stable regions remain dependent on both the projection horizon and the investigation budget. Thus, the horizon sensitivity

observed at the operating points reported in the main evaluation is not confined to the highest-priority portion of the ranking; it remains visible when progressively larger portions of the prioritised alert queue are considered. At the same time, increasing 𝐾 progressively absorbs rank redistributions that are consequential under stricter investigation budgets. As a result, differences between horizons and ranking methods can diminish at larger 𝐾 as more attack-confirmed alerts fall within the investigated set. This complements the main RQ2 finding: projection-level CSA does not simply become more useful as additional future activity is observed. Rather, its contribution depends on which validation-relevant behavioural evidence is exposed at a given horizon and where the affected alerts subsequently fall within the prioritised investigation queue.

Unified CSA Framework for Alert Validation

Recall@𝐾

(a) 𝐾 = 10

(b) 𝐾 = 20

(c) 𝐾 = 30

1

1

1

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0

0 0

20

40

60

80

100

0 0

20

Δ𝑡 (min)

40

60

80

100

0

20

Δ𝑡 (min)

40

60

80

100

Δ𝑡 (min)

Figure 6: Extended Recall@𝐾 results for Cadets E3 across projection horizons at investigation budgets 𝐾 ∈ {10, 20, 30}. Δ𝑡 = 0 corresponds to comprehension-level CSA, while Δ𝑡 > 0 incorporates projection-level CSA. PU-MIL ( ), Queue ( ), and Max. CSG ( ) denote the three ranking methods.

Recall@𝐾

(a) 𝐾 = 10

(b) 𝐾 = 20 1

1

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0

0 0

20

40

60

80

100

0 0

20

(d) 𝐾 = 40

Recall@𝐾

(c) 𝐾 = 30

1

40

60

80

100

0

(e) 𝐾 = 50 1

1

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0

0 20

40

60

Δ𝑡 (min)

80

100

40

60

80

100

80

100

(f) 𝐾 = 100

1

0

20

0 0

20

40

60

Δ𝑡 (min)

80

100

0

20

40

60

Δ𝑡 (min)

Figure 7: Extended Recall@𝐾 results for Theia E3 across projection horizons at investigation budgets 𝐾 ∈ {10, 20, 30, 40, 50, 100}. PU-MIL ( ), Queue ( ), and Max. CSG ( ) denote the three ranking methods. Higher Recall@𝐾 indicates better recovery of attack-confirmed alerts within the corresponding investigation budget.

Related documents

• Integrating Perception, Comprehension, and Projection for Alert Validation: A Unified Cyber Situational Awareness Framework#1491244
• Security Logs to ATT&CK Insights: Leveraging LLMs for High-Level Threat Understanding and Cognitive Trait Inference#1488255
• 2026 SANS SOC Survey Insights: A Decade of Evolution in Cyber DefenseWhitepaperSecurity Awareness15 Jun 2026Christopher Crowley#281882
• MACHINE LEARNING IN CYBERSECURITY#1178739
• Agency Information Collection Activities: “Generic Clearance for the Collection of Qualitative Feedback on National Security and Emergency Preparedness Communications (NSEPC)”#512249