Date of publication xxxx 00, 2026, date of current version xxxx 00, 2026. Digital Object Identifier 10.1109/ACCESS.2026.DOI
Federated Stream-Processing and Latency-Gated Response for Cross-Sector Threat Detection and Collaborative Containment NAMIT MOHALE
arXiv:2605.17325v1 [cs.CR] 17 May 2026
Independent Researcher, San Francisco, CA, USA
Corresponding author: Namit Mohale (e-mail: [email protected]).
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
ABSTRACT Critical infrastructure defense is fundamentally bottlenecked by the operational reality that preventive controls are frequently bypassed by sophisticated supply-chain compromises and stolen administrative credentials. When prevention fails, defense relies entirely on rapid, post-ingress threat detection and automated response across sovereign sectors. We present a novel, federated, high-throughput streamprocessing and correlation framework designed to detect coordinated cross-sector threat campaigns and orchestrate containment at machine speed. By utilizing a stateless Pre-Filtering Dispatcher Subsystem (PFDS), in-memory lock-sharded state workers, and a 95% statistical watermark heuristic, our system maintains detection momentum during network partitions to evaluate speculative alerts. Delayed telemetry is subsequently reconciled directly within a version-keyed columnar storage engine via deterministic time-bucket hashing, eliminating state-retraction overhead. We evaluate a prototype of our framework—implemented in Go with an instantiated production-grade columnar analytical store—against a 500, 000 events per second workload. The results demonstrate an internal framework processing overhead of under 7 seconds, while achieving total end-to-end operational convergence-accounting for multi-sector detection, correlation, widearea network (WAN) propagation, windowing stability, VLAN-level response, and hardware level mitigation commitment-within a realistic 12-20 seconds window. INDEX TERMS Intrusion Detection, Incident Response, Stream Processing, Critical Infrastructure, Collaborative Containment, Speculative Execution.
I. INTRODUCTION
C
RITICAL infrastructure sectors—such as healthcare systems, energy grids, water treatment facilities, and transportation networks—have become the primary targets of highly sophisticated, state-sponsored Advanced Persistent Threat (APT) actors. Unlike conventional enterprise IT environments, intrusions into operational technology (OT) and industrial control systems (ICS) carry physical-world consequences, ranging from power grid blackouts to the compromise of life-support systems in regional hospitals. Historically, defense paradigms have focused heavily on perimeter-based prevention, employing firewalls, strict network segregation, and identity access controls. However, modern operational realities have exposed a fatal flaw in this preventive-first strategy: **prevention is inherently bypassed when adversaries leverage legitimate access vectors.** By compromising shared supply-chain elements, external mainVOLUME 11, 2023
tenance service providers, or common third-party identity credentials, an adversary can slip past perimeters undetected. Because these independent critical sectors are physically and logically isolated from one another, and because privacy regulations (such as HIPAA in healthcare or NERC CIP in utilities) strictly prohibit them from sharing raw telemetry, an adversary can execute a coordinated, cross-sector campaign completely in parallel. To each isolated security operations center (SOC), the attack appears as a minor, localized, and uncorrelated anomaly, leaving the broader campaign entirely invisible until catastrophic physical damage occurs. Consequently, modern critical infrastructure defense must undergo a paradigm shift: **assuming breach and prioritizing rapid, post-ingress detection and automated response (containment).** Rather than attempting the impossible task of preventing every credential abuse or supply-chain compromise, defenders must focus on minimizing the ‘‘blast radius’’ once 1
an intrusion has occurred. To outpace active adversaries and limit lateral damage across isolated sectors, a collaborative threat detection and response framework must operate under three uncompromising constraints: • Machine-Speed Detection and Containment: Postingress detection and subsequent containment actions must execute within a strict 20-second latency budget from the initial edge compromise to prevent physical damage. • Late-Data Resilience: The detection engine must remain highly resilient to out-of-order, delayed telemetry streams (‘‘stragglers’’) caused by degraded physical links, satellite bottlenecks, or edge reboots, without stalling detection in healthy sectors. • Sovereign Privacy Preservation: The system must achieve deep cross-sector threat correlation and coordinate response actions without ever transmitting raw event logs across sector boundaries. A. LIMITATIONS OF THE STATE-OF-THE-ART
Existing security toolsets are structurally incapable of satisfying these detection and response constraints. Traditional Security Information and Event Management (SIEM) systems and centralized security data lakes are fundamentally too slow; they rely on polling-based query models that introduce minutes of delay, rendering real-time containment impossible. To bypass this latency bottleneck, general-purpose Distributed Stream Processing Engines (DSPEs) like Apache Flink have been explored. However, these engines enforce a strict, binary Event-Time consistency model. If a single edge network partition stalls, the global detection watermark freezes. This ‘‘straggler problem’’ stalls the entire cross-sector detection pipeline, blinding defenders to active threats in healthy sectors. Furthermore, when these engines attempt to reconcile delayed logs for forensic response, they rely on heavy state backends that cause severe memory thrashing and state starvation under heavy security workloads. B. OUR PROPOSED SOLUTION: DETECTION AND REACTIVE RESPONSE
In this paper, we present a novel, federated, high-throughput stream-processing and correlation framework designed specifically for real-time, cross-sector critical infrastructure threat detection and automated containment. Our framework abandons the slow, centralized indexing paradigm of traditional SIEMs and the fragile, strictly consistent watermarking models of traditional DSPEs. Instead, we propose a decentralized, Rules-as-Code (RaC) execution model where security engineers write highly expressive, procedurally defined detection graphs via a unified Fluent API. The system operates through a highly optimized, multitiered pipeline: 1) The Pre-Filtering Dispatcher Subsystem (PFDS): This stateless, ultra-low-latency layer sits directly at 2
the ingestion edge. By compiling the stateless, initial bootstrap predicates of our rules into highly optimized Common Expression Language (CEL) ASTs or WebAssembly (Wasm) bytecode, the PFDS filters and discards up to 99.8% of background noise in a single pass to accelerate detection times. 2) Lock-Sharded Stateful Workers: The filtered, lowvolume streams are consumed by concurrent Go-based stream workers. By utilizing a shared-nothing, inmemory partition mapping with key-specific read-write locks, our workers eliminate global mutex contention, allowing parallel, sub-millisecond evaluation of complex sliding temporal windows. 3) 95% Statistical Watermarking & Speculative Detection: To handle the inevitable ‘‘straggler problem,’’ our watermark coordinator calculates log completeness based on the 95th percentile (P95 ) of historical latency. If data from a lagging sector falls into the remaining 5%, the engine does not stall; it evaluates the active window speculatively and publishes a speculative alert to initiate early-stage containment. 4) Idempotent Late-Data Reconciliation: When delayed logs eventually arrive, the engine leverages deterministic time-bucket hashing to generate an identical, unique Fact ID (Fid ). It writes the revised state back to a highly scalable, columnar analytical database engine designed to natively execute background, version-keyed row replacements. The storage layer automatically overwrites the speculative record with zero manual database administration overhead, and a lightweight revision event is published downstream to update active response measures. 5) Decoupled, Zero-Leak Automated Response: We implement a secure Claim-Check Pattern over a central CISA-managed Pub/Sub broker. Cooperating sectors never transmit raw logs; instead, they write heavy, sensitive forensic payloads locally and broadcast only lightweight, anonymized Cluster IDs. Downstream, decoupled reactive agents—including an Automated Containment Agent (which evaluates risk gates and signs Ed25519 commands), an advanced Gen-AI Triage Agent, and an Escalation Agent—subscribe to this broker, dynamically reclaiming authorized facts on-demand from the columnar storage layer to execute containment actions (such as local VLAN micro-segmentation or token revocation) without exposing raw logs. C. SUMMARY OF CONTRIBUTIONS
The principal contributions of this work are summarized as follows: • The Rules-as-Code Fluent API & PFDS Compiler: We present a functional, testable execution language for multi-sector security telemetry, with a compiler capable of separating and pushing down stateless predicates into a high-performance Wasm/CEL Pre-Filtering Dispatcher. VOLUME 11, 2023
Statistical Watermarking & Memory-Efficient Reconciliation: We formulate a 95% statistical watermark heuristic that prevents pipeline stalls during network outages, coupled with a deterministic hashing model that offloads late-data state reconciliation directly to the database storage layer. • Privacy-Preserving Claim-Check Response Plane: We design a decoupled, fan-out mitigation architecture that protects sector sovereignty, executing cryptographically signed, risk-gated network and system containment without exposing raw logs outside of local boundaries. • Empirical Validation on a National-Scale Testbed: We implement and benchmark our architecture using Go and ClickHouse-based prototype, subjecting it to a simulated 500, 000 eps workload mimicking a coordinated, multi-vector campaign targeting state, healthcare and utility networks. We demonstrate that the framework introduces only sub-7 seconds of internal computational latency, successfully decoupling software overhead from the unavoidable network and hardware convergence delays. This enables cross-sector correlation and microsegmentation in under 15 seconds in general scenarios and 20 seconds for more complex scenarios, meeting the critical operational threshold for post-ingress threat containment.
•
II. BACKGROUND AND PROBLEM FORMULATION
In this section, we construct the formal mathematical models for multi-sector event streams, define the mechanics of the latedata ‘‘straggler’’ bottleneck in stream-processing architectures, and formulate the core security and privacy constraints of collaborative, cross-sector threat detection. A. MULTI-SECTOR STREAM AND TELEMETRY MODEL
Let S = {S1 , S2 , . . . , SN } be a finite set of N heterogeneous, sovereign critical infrastructure sectors. Each sector Si ∈ S operates an independent network domain and generates an infinite, continuous stream of security, system, and operational telemetry: Ei = {e1 , e2 , . . . } (1) Because these sectors are completely network-isolated and air-gapped from one another, their raw telemetry streams are strictly disjoint: Ei ∩ Ej = ∅
∀i ̸= j
(2)
To allow unified computation over these disparate streams without violating sectoral schema integrity, we model each log record as a standardized tuple within our Unified Telemetry Schema (UTS): e = ⟨te , ta , k, x⟩ (3) where: + • te ∈ R is the Event-Time, representing the physical timestamp when the state transition or security event occurred at the edge asset. VOLUME 11, 2023
ta ∈ R+ is the Arrival-Time, representing the timestamp when the ingestion engine successfully received and normalized the event. • k ∈ K is a deterministic, cryptographically hashed partition key (e.g., a one-way SHA-256 hash of a username, source IP, or device MAC address) used to map identity across sectors without exposing raw, sensitive identifiers. • x ∈ X is a polymorphic payload vector containing domain-specific metrics, system calls, or SCADA register parameters. •
B. THE LATE-DATA STRAGGLER PROBLEM
In any distributed critical infrastructure network, physical propagation delays, WAN congestion, edge node reboots, and satellite link drops are inevitable. We define the propagation delay of an event e as: ∆p (e) = ta − te
(4)
In a perfectly synchronous system, ∆p (e) ≈ 0. However, in real-world critical infrastructure deployments, propagation delays are highly variable and unpredictable. Let W (t) represent the global system watermark at wallclock time t. Under a Strictly Consistent Event-Time Model, a stream-processing engine cannot safely close a temporal processing window of width τ until it is certain that all data within that window has arrived. Thus, the system’s watermark is strictly bounded by the slowest streaming sector: W (t) = min max (te ) (5) Si ∈S
e∈Ei ,ta ≤t
If a single sector Sstraggler ∈ S experiences a localized network partition or satellite drop at wall-clock time toutage , its ingestion stream halts. Consequently, the maximum eventtime observed from Sstraggler stalls at some historical timestamp tstall . Because the global watermark W (t) is constrained by a strict minimum function, it remains frozen at tstall for the duration of the outage: W (t) = tstall
∀t ≥ toutage
(6)
This freeze creates a linear, unbounded growth in the Systemic Detection Lag (Lsys ) across the entire multi-sector pipeline: Lsys (t) = t − W (t) (7) As illustrated in Figure 2, this structural vulnerability is highly exploitable. Under standard strictly consistent paradigms, a single localized network drop at toutage = 2 minutes causes Lsys to escalate aggressively over time. By freezing the global watermark, the entire cross-sector detection pipeline becomes blinded, preventing active security monitoring in completely healthy, unaffected sectors. In contrast, our proposed 95% statistical watermarking model—formulated mathematically in Section III and visualized as a flat baseline in Figure 2—maintains detection continuity. By bypassing the lagging 5% tail of late-arriving telemetry, the watermark monotonically advances, keeping 3
FIGURE 1: Proposed Federated System Architecture: End-to-end post-ingress threat detection, stateful stream evaluation, and late-data speculative response across independent, sovereign critical sectors.
systemic detection lag flat and bounded below 2 seconds regardless of ongoing edge network partitions.
1) Problem 1 (Watermark Boundary Optimization)
Our objective is to design a watermark coordinator that dynamically advances W (t) to minimize systemic detection lag Lsys (t) while maintaining a detection confidence bound of 1 − α (where α = 0.05, representing a 95% statistical data-completeness guarantee): min
E[Lsys (t)]
s.t.
Pr (ta ≤ W (t)) ≥ 0.95
∀e ∈ Ei
(8)
C. CROSS-SECTOR COLLABORATIVE CORRELATION AND RESPONSE CONSTRAINTS
FIGURE 2: Systemic detection lag (Lsys ) over time under strict consistency (dashed red line) showing linear growth post-partition, contrasted against the stable, flat baseline of our 95% speculative model (solid blue line). Let F(Sk (t)) be a stateful threat detection function operating over an in-memory sliding window Sk (t) of width τ . 4
To detect a sophisticated APT executing a synchronized, multivector campaign, our framework must correlate events across independent sectors. Let a Coordinated Cross-Sector Campaign C be defined as a set of localized threat anomalies occurring across M distinct sectors (M ≥ 2) within a maximum temporal correlation window ∆C : C = {ai1 , ai2 , . . . , aiM }
where
aij ∈ Sij
max(taij ) − min(taij ) ≤ ∆C
(9) (10)
VOLUME 11, 2023
Each localized anomaly a is characterized by a shared, cryptographically hashed identity attribute kactor ∈ K. 1) Sovereignty and Privacy Constraint
To comply with zero-trust directives, sector sovereignty, and privacy regulations, we enforce a zero raw-log leakage boundary. Let L(e) represent the information leakage function of raw telemetry. Our primary privacy constraint dictates that no raw event payload x may ever be transmitted outside of its originating sector’s sovereign network boundary: Lshared (e) = 0
∀e ∈ Ei
∀Si ∈ S
(11)
2) Problem 2 (Zero-Leak Detection and Response Correlation)
We must define a global correlation and response function C that operates entirely over a decentralized messaging infrastructure such that: C(E1 , E2 , . . . , EN ) → {C1 , C2 , . . . }
(12)
Lshared (e) = 0
(13)
subject to
∀e ∈ Ei
Our system must achieve this by executing a Claim-Check Pattern—writing heavy, localized forensic payloads locally to a secure, columnar database and broadcasting only completely anonymized, lightweight trigger hashes to coordinate automated containment. III. PROPOSED SYSTEM ARCHITECTURE
To address the latency, resilience, and privacy constraints formulated in Section II, we present the design of our federated threat detection and reactive containment framework. As illustrated in Figure 1, the architecture is divided into five distinct operational planes: the Ingestion Plane, the Pre-Filtering Dispatcher Subsystem (PFDS), the Stateful Execution Plane, the Storage & Broker Plane, and the Reactive Response Plane. a: Architectural Scope and Boundaries
While our functional prototype is fully realized using the technologies detailed in Section IV, the granular, code-level execution of individual threat-detection rules, local statemachine data transformations, and low-level parsing routines is considered out of scope for this paper. This work focuses strictly on the structural stream-routing mechanics, cryptographic data isolation (via the Claim-Check pattern), and the temporal synchronization guarantees of the architecture. The internal, operational implementation of specific telemetry parsing logic is treated as a modular runtime detail, fully decoupled from the core systemic security and scalability contributions of this framework. A. PRE-FILTERING DISPATCHER SUBSYSTEM (PFDS)
The first line of post-ingress defense is the Pre-Filtering Dispatcher Subsystem (PFDS), represented as the second operational tier in our architecture (Plane 2 of Figure 1). Because evaluating complex stateful rules over sliding windows is VOLUME 11, 2023
computationally expensive, we deploy stateless, single-pass filtering agents directly at each sector’s ingestion edge. The PFDS evaluates incoming telemetry Ei before serialization or transmission. To achieve sub-millisecond execution, security engineers write stateless bootstrap predicates in a unified, declarative grammar. The PFDS compiler parses these rules and generates Abstract Syntax Trees (ASTs) executed via the Common Expression Language (CEL) or compiles them directly into WebAssembly (Wasm) bytecode. For example, if a stateful detection rule seeks to identify brute-force SCADA write attempts, the PFDS compiles a stateless predicate to discard all non-write operations (e.g., standard read requests or system heartbeats) in a single pass. By dropping up to 99.8% of benign background noise at the edge in best-case scenarios, the PFDS ensures that only highly curated, threat-relevant streams are forwarded to the stateful workers, protecting downstream memory and processing queues from exhaustion attacks. The declarative, developer-friendly syntax of our Rulesas-Code (RaC) engine is illustrated in Listing 1. This fluent specification defines a multi-sector correlation rule that targets brute-force authentication anomalies followed by lateral SCADA register modifications within a sliding 15-minute window. 1
2
3 4 5
6 7 8
9 10
11 12 13
14 15 16 17
18
// Define a cross-sector multi-vector threat rule var BruteForceToSCADA = NewRule("Correlation-BruteForce-SCADA"). WithWindow(15 * time.Minute). WithKeys("user_hash", "source_ip_hash"). // First Event: Brute force detection on AD/Identity Sector OnStream("identity_events"). Where(func(e Event) bool { return e.GetString("action") == "AUTH_FAIL" && e.GetInt("attempt_count") > 5 }). // Second Event: Modbus SCADA write operation OnStream("utility_scada"). Where(func(e Event) bool { return e.GetString("protocol") == "modbus" && e.GetBool("is_write") == true }). // Correlation logic executed upon match Correlate(func(s1 State, s2 State) bool { return s1.GetTimestamp("last_fail") < s2.GetTimestamp("write_time") })
Listing 1: Fluent API Rules-as-Code Specification for CrossSector Anomalies To minimize CPU overhead and guarantee deterministic execution bounds at the resource-constrained ingress edge, the PFDS compiles declarative filtering rules into stateless Common Expression Language (CEL) expressions. These expressions are evaluated within a highly optimized, sandboxed WebAssembly (Wasm) runtime embedded directly within the ingestion daemon, completely bypassing heavy language runtime reflection or JSON parsing overhead. By offloading 5
this high-throughput, first-pass filter to a lightweight, compiled execution layer, the subsystem effectively drops the bulk of background telemetry noise with sub-millisecond local processing latency before any WAN transmission occurs.
can propagate laterally, effectively decoupling the speed of incident response from WAN latency or edge network drops. To formalize the runtime behavior of our stream workers when encountering network partitions, we outline the speculative evaluation heuristic in Algorithm 1.
B. LOCK-SHARDED STATEFUL STREAM WORKERS
The curated stream emitted by the PFDS is consumed by concurrent, stateful stream workers operating within the Stateful Execution Plane (Plane 3 of Figure 1). To prevent global mutex contention under high-throughput workloads (such as 500, 000 events per second), we implement a sharednothing, lock-sharded memory mapping model. Instead of maintaining a single, globally locked state store, the active memory space is divided into discrete, independent partitions. When an event e = ⟨te , ta , k, x⟩ arrives at a worker, the system hashes the partition key k to allocate a specific worker partition. A localized read-write lock is acquired for that key hash: ShardID = MurmurHash3(k)
(mod Nshards )
(14)
By sharding locks at the key level rather than the global level, multiple threads can concurrently read and write to independent state slices Sk,b without blocking each other. This keeps our lock contention overhead flat and near-zero, even under extreme telemetry surges. To accelerate contextual lookups during evaluation, each stream worker maintains a highly optimized local cache using an embedded key-value store (instantiated via RocksDB in our prototype). This cache stores localized enrichment context—such as active network topologies, operational system states, and asset-to-owner maps—allowing the engine to enrich incoming telemetry with local context in sub-millisecond intervals without querying external databases.
To maintain continuous detection capability during network partitions, the framework employs a speculative execution paradigm controlled by a dynamic three-tiered watermarking coordinator. Instead of tracking watermark progress strictly based on a binary 100% completeness model, the control plane calculates the 95th percentile (P95 ) of historical transit latency across all active ingestion streams. The watermark W (t) is allowed to monotonically advance even if a lagging sector (Sstraggler ) fails to emit telemetry. When W (t) advances past a sliding window’s end boundary, the stateful worker does not stall. It evaluates the compiled Rule Logic (F) speculatively based on the 95% complete data currently available in memory. If a threshold is crossed, the engine generates a speculative alert, marked with a unique, deterministically generated Fact ID (Fid ): (15)
By publishing this speculative alert downstream, the system initiates early-stage containment actions before the attacker 6
Require: State Sk,b , threshold θ = 0.95, watermark W (t) Ensure: Speculative alert or state reconciliation 1: tend ← GetWindowEnd(b) 2: C ← CalcCompleteness(Sk,b , W (t)) ▷ Ratio of active reporting sectors 3: if W (t) ≥ tend then 4: if C ≥ θ and ¬IsFinalized(Sk,b ) then 5: Alertspec ← EvaluateRule(F , Sk,b ) 6: if Alertspec ̸= ∅ then 7: Fid ← GenFactID(RuleID, k, b) 8: EmitAlert(Alertspec , Fid , "Speculative") 9: SetSpeculative(Sk,b , true) 10: end if 11: end if 12: end if 13: procedure ReconcileLateData(Event elate ) 14: b ← GetTimeBucket(elate .te ) 15: UpdateWindowState(Sk,b , elate ) 16: if IsSpeculative(Sk,b ) then 17: Alertfinal ← EvaluateRule(F , Sk,b ) 18: Fid ← GenFactID(RuleID, k, b) 19: EmitAlert(Alertfinal , Fid , "Finalized") 20: SetFinalized(Sk,b , true) 21: end if 22: end procedure
Complementary to window evaluation, the control plane must dynamically adjust the system watermark based on real-time ingestion latency. This process is formalized in Algorithm 2. Algorithm 2 Dynamic Watermark Calibration
C. SPECULATIVE DETECTION AND 95% STATISTICAL WATERMARKING
Fid = Hash(RuleID ∥ k ∥ b)
Algorithm 1 Speculative Window Evaluation
Require: Sectors S, buffer H (size M ), percentile P95 Ensure: System watermark W (t) 1: for each ingested event e = ⟨te , ta , k, x⟩ from Si do 2: ∆p (e) ← ta − te ▷ Event propagation delay 3: PushToBuffer(Hi , ∆p (e)) 4: if BufferSize(Hi ) > M then 5: PopOldest(Hi ) 6: end if 7: end for 8: procedure OnWatermarkIntervalTick 9: L←∅ 10: for each sector Si ∈ S do 11: Li ← CalcPercentile(Hi , P95 ) 12: L ← L ∪ {Li } 13: end for 14: Lsys_bound ← Mean(L) 15: tcurrent_wall ← GetWallClockTime() 16: W (t) ← tcurrent_wall − Lsys_bound 17: BroadcastWatermark(W (t)) 18: end procedure
D. IDEMPOTENT LATE-DATA RECONCILIATION
When the lagging sector recovers and its delayed telemetry eventually arrives, the system must reconcile the historical VOLUME 11, 2023
state without triggering redundant alerts or creating database inconsistencies. We offload this state reconciliation directly to the underlying columnar storage layer. When late-arriving events are processed, the stream worker hashes the events into their corresponding historical timebuckets. Because the Fact ID Fid is generated deterministically based on the Rule ID, partition key k, and temporal bucket b, the newly evaluated state generates an identical Fid but with an incremented version key. The worker writes this revised record to the Columnar Fact Store. The storage engine—configured as a version-keyed, background-merging columnar database—automatically overwrites the older speculative record matching the same Fid key. This background merge requires zero manual database administration and operates with O(1) write complexity. Simultaneously, a lightweight revision event is published downstream to notify the response agents that a speculative alert has been finalized or amended.
this desk acts as the human-in-the-loop bridge, routing telemetry to active pagers and security dashboards for manual containment confirmation. F. DATABASE SCHEMAS AND STORAGE LAYOUT
To support high-velocity telemetry writes while allowing instant cross-sector alert correlation, our storage architecture splits data into a localized active state cache and a centralized analytical column-store. The schema definitions for both storage layers are detailed in Table 1. The localized stream workers maintain an in-memory key-value map for immediate window evaluations, while the Columnar Fact Store leverages ClickHouse’s ReplacingMergeTree engine to handle raw forensic payloads and background late-data merging, although any other column-based DBMS can also be used instead. TABLE 1: Storage Layer Schemas and Field Specifications Field / Column Name
Data Type & Description
E. DECOUPLED REACTIVE RESPONSE PLANE (CLAIM-CHECK PATTERN)
Local State Store (Active Window Cache)
To preserve sector sovereignty and comply with strict zero raw-log leakage boundaries (Lshared (e) = 0), we implement a secure Claim-Check Pattern across our storage and response planes (Planes 4 and 5 of Figure 1). When a stateful worker fires a speculative or finalized alert, it does not broadcast the heavy, sensitive forensic log payload x to the central broker. Instead, it writes the forensic payload locally to its sovereign, secure database and publishes only a lightweight, cryptographically signed notification to a central, decentralized message broker managed by a federal coordinator (e.g., CISA) along with some other metadata; the raw logs data never travel across the WAN. This notification contains only:
Columnar Fact Store (Analytical DB)
Notification = ⟨Fid , ClusterID, SignEd25519 (Fid )⟩
String – Unique compound key representing RuleID ∥ k ∥ b. completeness_ratio Float – Live tracking metric of reporting sectors. aggregated_state BLOB – Serialized JSON state payload for active rule evaluation. last_updated Timestamp – Epoch timestamp used for automatic TTL cache eviction.
F_id (Primary Key) timestamp sector_id rule_id forensic_payload
(16)
Three independent, parallel reactive agents subscribe to this broker to orchestrate coordinated defense: • Automated Containment Agent: This agent evaluates a mathematical safety risk-gate formula to determine if automated isolation is authorized. If the risk is acceptable, it signs a localized containment command (using Ed25519 keypairs) and dispatches a gRPC call to trigger microsegmentation, such as shutting down a compromised switch port or revoking a user session, in under 15 seconds. • Gen-AI Triage Agent: This agent acts as a virtual tier1 analyst. It intercepts the lightweight notification and uses the authorized Fact ID to pull the localized forensic payload from the originating sector’s Columnar Fact Store on-demand (the "Claim" retrieval). It synthesizes this data into a highly structured markdown report for human operators, and dynamically handles updates if a speculative alert is later revised. • Escalation Desk: For high-impact speculative alerts that do not meet automated containment safety thresholds, VOLUME 11, 2023
key (Primary)
is_reconciled
UUID – Deterministic cryptographic transaction lookup hash. DateTime64 – Logged wall-clock occurrence of the security event. – FastLowCardinality(Str) indexing identifier of the originating sector. LowCardinality(Str) – Identifier of the triggered correlation rule. String – Full JSON representation of raw telemetry event x. UInt8 – Binary flag indicating background deduplication status.
The ClickHouse table for our analytical store is instantiated using the DDL specification shown in Listing 2. The use of the ReplacingMergeTree(timestamp) engine ensures that if late-arriving telemetry is written to the database after a speculative window has closed, ClickHouse automatically merges and deduplicates the records in the background using the deterministic F_id primary key. 1
2 3 4 5 6 7 8 9 10
CREATE TABLE IF NOT EXISTS security_facts.facts ( F_id UUID, timestamp DateTime64(3, ’UTC’), sector_id LowCardinality(String), rule_id LowCardinality(String), forensic_payload String, is_reconciled UInt8 DEFAULT 0 ) ENGINE = ReplacingMergeTree(timestamp) ORDER BY (rule_id, sector_id, F_id) SETTINGS index_granularity = 8192;
Listing 2: ClickHouse DDL for the Columnar Fact Store 7
IV. EXPERIMENTAL EVALUATION
In this section, we present an empirical evaluation of our federated threat detection and reactive containment framework. Our evaluation is designed to answer three key research questions: • RQ1 (Throughput & Scaling): Can our lock-sharded Go stream workers ingest and process extreme telemetry rates (500, 000 eps) without suffering from mutexlocking starvation? • RQ2 (Watermark Resilience): How effectively does our 95% statistical watermark minimize systemic detection lag compared to a strictly consistent Event-Time baseline during an active network partition? • RQ3 (End-to-End Containment Latency): What is the total latency budget consumed by our decentralized response plane from the initial edge compromise to the final cryptographically signed VLAN micro-segmentation command? A. EXPERIMENTAL SETUP AND TESTBED
To evaluate the framework under production-grade conditions, we deployed a multi-sector simulated testbed across a hybrid cloud environment. We instantiated N = 3 independent, sovereign sectors representing a regional healthcare network, a power utility SCADA network, and an air transportation identity database. Each sector’s edge infrastructure was simulated using dedicated nodes. The stream processing and database storage layers were built using Go v1.21.0 and Python v3.11. As established in our conceptual model, the abstract Columnar Fact Store was instantiated using ClickHouse (v23.8 LTS) as our high-performance columnar analytical engine. TABLE 2: Experimental System Configuration and Testbed Specifications Component / Layer
Hardware and Software Specifications
Pre-Filtering Agents (PFDS)
3× AWS EC2 t3.medium (2 vCPUs, 4GB RAM)
Stateful Stream Workers
3× AWS EC2 c6i.2xlarge (8 vCPUs, 16GB RAM)
Local Context Cache
Embedded wrapped)
Columnar Fact Store
ClickHouse v23.8 LTS on AWS i3en.xlarge
Central Message Broker
Apache Kafka Cluster (3 Brokers, NVMe Storage)
Reactive Response Agents
3× AWS EC2 c6i.large (2 vCPUs, 4GB RAM)
RocksDB
v8.1.1
(Go-
The exact hardware distribution and software configurations are summarized in Table 2. Telemetry generation was orchestrated using an open-source security log replayer, feeding realworld Windows Event logs, Linux Syslog, and Modbus/TCP register transactions at controlled rates up to 500, 000 events per second (eps). 8
B. RQ1: INGESTION THROUGHPUT AND LOCK CONTENTION
We evaluated the scalability of our Go stream workers by executing a high-velocity stress test. We bypassed the PreFiltering Dispatcher Subsystem (PFDS) to subject the stateful engine to raw, un-filtered telemetry streams scaling from 100, 000 to 500, 000 eps. We compared our lock-sharded partition mapping (Nshards = 1024) against a traditional, globally-locked state manager (representing a common threadsafe map implementation). The lock contention overhead (ωlock ) was measured as the average wait time (in microseconds) incurred by a processing thread waiting to acquire write privileges on a partition state Sk,b : M 1 X (j) (j) ωlock = tacquire − trequest (17) M j=1 As telemetry rates escalated to 500, 000 eps, the globallylocked baseline suffered an exponential increase in lock contention, with ωlock shooting up from 4.2 µs to 1, 842 µs. This severe thread starvation forced the global pipeline CPU utilization to cap at 100%, triggering packet drops and memory leaks. In contrast, our lock-sharded state architecture maintained a near-flat lock overhead, peaking at a mere 0.85 µs at peak capacity (500, 000 eps). This represents a 2, 100× improvement in lock efficiency, confirming that sharding write-locks based on key hashes effectively eliminates thread competition under extreme critical infrastructure workloads. C. RQ2: LATE-DATA RESILIENCE AND WATERMARK BEHAVIOR
To validate our statistical watermarking model, we simulated a coordinated multi-sector cyber campaign targeting the SCADA network and the healthcare enterprise. At wall-clock minute 2 (toutage ), we injected a complete network partition on the SCADA telemetry stream, delaying 100% of its event logs for exactly 8 minutes. Under the Strictly Consistent Event-Time baseline, the global watermark froze immediately at tstall = 2 minutes. This caused the systemic detection lag (Lsys ) to climb linearly at a 1.0 second-per-second rate, as previously modeled in Figure 2. Because the watermark could not advance, the crosssector correlation windows remained open and unresolved, preventing the system from identifying active lateral attacker movements occurring within the healthy healthcare network. Our proposed model calculated the 95th percentile of historical transit latency (P95 = 1.42 seconds). When the SCADA partition occurred, the watermark coordinator recognized the sector as a statistical outlier (> P95 ), bypassed its missing stream, and allowed the watermark W (t) to monotonically advance. As a result, systemic detection lag for the active, healthy sectors remained entirely flat, maintaining a baseline of 1.52 seconds throughout the duration of the SCADA outage. VOLUME 11, 2023
The stream workers successfully processed speculative windows and published early speculative alerts to the response plane, demonstrating that our architecture effectively immunizes the wider collective defense from localized physical disruptions. D. RQ3: END-TO-END CONTAINMENT LATENCY
Our final experiment evaluated the end-to-end performance of the reactive containment plane (Plane 5 of Figure 1) during the simulated multi-sector campaign. We measured the latency breakdown across five key operational phases, starting from the physical event occurrence at the edge to the final execution of the response payload, contrasting the synthetic "softwareonly" benchmarks against a production baseline that accounts for real-world network and hardware constraints. TABLE 3: End-to-End Reactive Response Latency Breakdown #
Phase
1 2
Ingestion Pre-Filtering Stream Processing Event Generation Forensic Ingestion DB Writing Pubsub Dispatch (Claim-Check) Containment Response Action
3 4 5
Total E2E Latency
Synthetic Framework Overhead
Operational System Convergence
120 ms 150 ms
1.0 s 3.0 s
840 ms
4.0 s
90 ms
1.0 s
800 ms
6.0 s
2s
15 s
Ingestion & WAN Propagation: While local telemetry ingestion at the edge occurs in 120 ms, production transit over encrypted VPN tunnels across geographic regions introduces a median delay of 1 s. • Stream Processing & Windowing Stability: To minimize false-positive triggers in critical infrastructure, the system utilizes a speculative "observation window". While the software evaluates execution logic in 150 ms/window-records, a 3-second total window is required to ensure signal stability and account for out-oforder event arrival from disparate sectors. • Network & Hardware-Level Containment: This phase represents the most significant divergence. Issuing a gRPC command from the Automated Containment Agent is near-instant (800 ms). However, the convergence time for the actuation of the command in the local networks to update Access Control Lists (ACLs), blocking systems, or re-tagging VLANs typically requires 4 to 8 seconds to commit and verify the change in the data plane. As shown in Table 3, the entire post-ingress detection and containment loop executed in 2 seconds in the synthetic plane and 15 seconds in the operational plane. Importantly, our decoupled Claim-Check pattern successfully separated the heavy, slow analytical storage operations •
VOLUME 11, 2023
from the critical alerting path. While the database engine took 840 ms to parse, write, and index the heavy raw forensic payload x, the lightweight alert notification containing only the Fid and ‘ClusterID‘ was dispatched over Kafka in just 90 ms. This architectural decoupling allowed our Automated Containment Agent to evaluate its risk-gate safety formula, cryptographically sign the micro-segmentation command using its private Ed25519 key, and dispatch a gRPC containment command to the target SCADA switches in 917.03 ms. When the delayed SCADA logs were eventually released at minute 10, the background ClickHouse ReplacingMergeTree merged the late forensic data using deterministic time-bucket hashing in the background, achieving database reconciliation with zero state backpressure, zero manual intervention, and zero system downtime. V. RELATED WORK
The design of real-time, collaborative critical infrastructure defense intersects three primary areas of research: centralized security analytics, distributed stream processing, and privacypreserving collaborative threat sharing. A. CENTRALIZED SIEM AND SOAR ARCHITECTURES
Traditional security operations rely heavily on Security Information and Event Management (SIEM) systems coupled with Security Orchestration, Automation, and Response (SOAR) platforms. While modern cloud-native SIEMs offer highly expressive correlation engines, they are fundamentally built on an ingestion-then-index paradigm. Telemetry must be completely written and indexed in centralized databases before search queries can execute. This database indexing pipeline introduces several minutes of inherent latency, making it mathematically impossible to orchestrate active containment within a strict 15-second operational window. Furthermore, SIEM and SOAR platforms are designed for single-tenant enterprise networks. They are structurally incapable of executing cross-sector correlation without centralizing raw event logs. In critical infrastructure settings, this centralization is blocked by legal, regulatory, and competitive boundaries. Our framework completely avoids this bottleneck by implementing the Claim-Check pattern, allowing independent sectors to maintain full data sovereignty while correlating alerts externally at machine speed. B. DISTRIBUTED STREAM PROCESSING ENGINES (DSPES)
To achieve sub-second execution speeds, researchers have explored general-purpose Distributed Stream Processing Engines (DSPEs) such as Apache Flink, Apache Storm, and Spark Streaming for intrusion detection. While these platforms process telemetry streams with low latency, they enforce a binary, strict Event-Time consistency model. In a multi-sector grid, network drops, physical link degradation, or satellite latency on a single channel will inevitably create "straggler" streams. In a standard DSPE, a single straggler freezes the global system watermark. This watermark freeze 9
stalls all downstream temporal sliding windows, effectively blinding the security pipeline to threats in healthy sectors. Our framework addresses this "straggler problem" by introducing a 95% statistical watermark heuristic. By speculatively evaluating sliding windows and offloading the late-data reconciliation directly to a version-keyed columnar storage layer, our architecture maintains continuous detection throughput during WAN partitions without stalling or suffering from memoryheavy state-retraction overheads. C. COLLABORATIVE DEFENSE AND THREAT INTELLIGENCE SHARING
Collaborative security networks like MISP (Malware Information Sharing Platform) and structured standards such as STIX/TAXII represent the state-of-the-art in multi-organization defense. These platforms are highly effective for sharing passive Indicators of Compromise (IoCs), such as known malicious IP addresses, file hashes, or domain names. However, these sharing models are fundamentally passive and retrospective. They are designed for post-incident forensic dissemination rather than real-time, reactive containment. They lack the streaming execution pipelines needed to detect coordinated, multi-vector campaigns in progress, and they cannot safely automate reactive response commands. In contrast, our framework acts as an active, real-time cooperative plane. It does not merely share static intelligence; it dynamically correlates stateless edge telemetry, executes complex stateful temporal rules across sectors, and orchestrates cryptographically secured, automated VLAN microsegmentation in under two seconds.
overhead, we effectively buy back time for the slower physical components of critical infrastructure-such as network and hardware-to respond. This architecture ensures that the defense system itself is never the bottleneck, allowing for a total response time that remains comfortably below the 20 second threshold required to disrupt lateral movement in APT campaigns. Future work will focus on three key directions: • Adversarial Watermark Manipulation: Investigating robust statistical guards to prevent sophisticated attackers from artificially inflating propagation delays to trick the watermark controller. • Homomorphic Policy Evaluation: Exploring the feasibility of executing certain stateful correlation rules directly over encrypted edge streams to further enhance privacy. • Autonomous Playbook Verification: Implementing formal verification methods within the Automated Containment Agent to mathematically guarantee that automated micro-segmentation commands do not accidentally disrupt critical, life-saving operational equipment.
VI. CONCLUSION
In this paper, we presented a novel, federated threat detection and reactive containment framework designed specifically for the strict operational constraints of multi-sector critical infrastructure networks. Our architecture successfully resolves the historical tension between real-time security coordination, WAN network resilience, and strict data sovereignty boundaries. By employing a stateless Pre-Filtering Dispatcher Subsystem (PFDS) at the ingestion edge, the framework discards up to 99.8% of background telemetry noise in a single pass. To prevent locking starvation under extreme workloads, we introduced a lock-sharded stream worker mapping model that keeps lock contention near-zero up to 500, 000 events per second. Furthermore, our 95% statistical watermarking heuristic prevents system watermarks from stalling during localized network partitions, allowing the system to speculatively evaluate rules and dispatch automated containment commands. The decoupled Claim-Check pattern guarantees zero raw-log leakage across administrative sectors while achieving an endto-end post-ingress containment response latency of just 2 seconds in the synthetic plane. A key finding of this research is the Convergence Gap. By optimizing the detection framework to a sub-2-second 10
VOLUME 11, 2023
REFERENCES [1] T. Akidau, R. Bradshaw, C. Chambers, S. Chernyak, R.J. FernándezMoctezuma, R. Lax et al. ‘‘The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing’’, Proceedings of the VLDB Endowment, Google Research, 8 (2015). [2] O. Babayomi and D.-S. Kim. ‘‘Federated Anomaly Detection and Mitigation for EV Charging Forecasting Under Cyberattacks’’, 2025. /emphInternational Conference on Information and Communication Technology Convergence, Jeju, Korea, Republic of, 2025, pp.996-1001, doi: 10.1109/ICTC66702.2025.11388140. [3] A. Vyas, P.-C. Lin, R.-H. Hwang and M. Tripathi, ‘‘Privacy-Preserving Federated Learning for Intrusion Detection in IoT Environments: A Survey’’, IEEE Access, vol. 12, pp. 127018-127050, 2024, doi: 10.1109/ACCESS.2024.3454211. [4] K. Thirasak, T. Chuaphanngam, D. Chainarong and S. Fugkeaw, ‘‘TF2ML: Threat Filtering With Two-Stage Machine Learning for Efficient Provenance-Aware Threat Detection and Response’’, IEEE Open Journal of the Computer Society, vol. 6, no. 01, pp. 1751-1762, 2025, doi: 10.1109/OJCS.2025.3618157. [5] M. Barni and F. Bartolini, Watermarking Systems Engineering: Enabling Digital Assets Security and Other Applications, CRC Press, 2024. [6] F. Dai, Md.A. Hossain and Y. Wang, ‘‘State of the Art in Parallel and Distributed Systems: Emerging Trends and Chellenges’’, MDPI Electronics, 2024, 14(4), 667, doi: 10.3390/electronics14040677. [7] E.M. Timofte, M. Dimian, A. Graur, A.D. Potorac, D. Balan, I. Croitoru et al. ‘‘Federated Learning for Cybersecurity: A Privacy-Preserving Approach’’, MDPI Applied Sciences, 2025, 15, no. 12, 6878, doi: 10.3390/app15126878. [8] M. Tawfik, A.A. Abu-Ein, H.M. Noaman et al. ‘‘FedMedSecure: Federated Few-Shot Learning with Cross-Attention Mechanisms and Explainable AI for Collaborative Healthcare Cybersecurity’’, Sci Rep 15, 40500, Nov. 2025. https://doi.org/10.1038/s41598-025-25107-z [9] K. Huang, Z. Yang and L. Zhou, ‘‘Agent Guide: A Simple Agent Behavioral Watermarking Framework’’, Apr. 2025. [Online]. Available: https://arxiv.org/html/2504.05871v1. [10] B. Wei, Y.S. Tay, H. Liu, J. Pan, K. Luo, Z. Zhu et al. ‘‘CORTEX: Collaborative LLM Agents for High-Stakes Alert Triage’’, NeurIPS, 2025. [11] V. Harsh, S. Sinha, H. Milner, B.A. Prakash, V. Sekar and H. Zhang. ‘‘MoCE: A Mixure-of-Context Aware Experts Framework for Troubleshooting Internet-scale Services’’, USENIX NSDI, 2026. [Online]. Available: https://www.usenix.org/conference/nsdi26/presentation/harsh [12] A. Shelupanov, O. Evsutin, A. Konev, E. Kostyuchenko, D. Kruchinin and D. Nikiforov. ‘‘Information Security Methods-Modern Research Directions’’, 2019. Symmetry. 11. 150. 10.3390/sym11020150
NAMIT MOHALE is an independent cybersecurity researcher and software engineer specializing in real-time data processing systems, high-throughput stream processing, and critical infrastructure defense architectures. Mr. Mohale received an M.S. degree in Computer Science from New York University, USA in 2020 and a Cybersecurity & Information Assurance specialization M.S. degree from Sofia University, USA in 2026. Since 2021, he has been a SOFTWARE ENGINEER at Google in San Francisco, USA, working with the Threat Detection & Response Platform developing, optimizing, and maintaining a system to analyze petabytes of data to detect, investigate, and respond to threats in Google’s environment quickly, reliably, and efficiently. He also became a member of ISC2 (International Information System Security Certification Consortium) after getting the CC certification in 2026.
VOLUME 11, 2023
11