ANTMAN: An Efficient and Interpretable RTL-Level Run-Time Detection Framework for Stealthy Branch Predictor Attacks on BOOM

ANTMAN: An Efficient and Interpretable RTL-Level Run-Time Detection Framework for Stealthy Branch Predictor Attacks on BOOM Muhammad Hassan∗ , Maria Mushtaq† , Jaan Raik∗ , Tara Ghasempouri∗ ∗ Department of Computer Systems, Tallinn University of Technology, Tallinn, Estonia ‡ Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France

arXiv:2608.09498v1 [cs.CR] 10 Aug 2026

{muhammad.hassan, jaan.raik, tara.ghasempouri}@taltech.ee maria.mushtaq@telecom-paris.fr, Abstract—Runtime detection of microarchitectural sidechannel attacks remains significantly underexplored in RISCV compared with x86 and ARM ISAs, posing a serious threat to critical applications. State-of-the-art branch predictor attacks bypass traditional data and instruction caches by directly exploiting the state of internal history tables, making them inherently stealthy. Recent research has explored offline detection of microarchitectural attacks on RISC-V; however, efficient runtime detection of microarchitectural attacks on RISC-V hardware remains significantly unaddressed. State-of-the-art hardware-based runtime detection solutions leverage hardware performance counters (HPCs) but suffer from a restricted set of counter registers and tradeoffs between detection accuracy, detection speed, and sampling granularity, making them impractical for stealthy attacks. Moreover, sampling HPCs after distinct intervals leaves intermediate relationships between different microarchitectural blocks unobserved. Additionally, proprietary x86 and ARM ISAs constrain researchers from modifying processor microarchitectural designs. To address these limitations, we propose the first secure-by-design, highly interpretable, non-intrusive, RTL-level runtime detection solution for stealthy branch predictor attacks on BOOM RISC-V, evaluated under both simplified Next-Line Predictor (NLP) and complex TAGE predictor configurations. The attack detection relies on association rules extracted offline and embedded in hardware as a non-intrusive rule monitor that enables runtime detection. The proposed approach achieves excellent detection speed, terminates execution before secret disclosure, and produces zero false positives while remaining flexible for detecting previously unseen variants within the same family of branch predictor attacks. Index Terms—Microarchitectural security, side-channel attacks, runtime detection, RISC-V, BOOM, RTL, branch predictor attacks, Next-Line Predictor (NLP), TAGE predictor, nonintrusive monitor.

I. I NTRODUCTION AND R ELATED W ORK Microarchitectural side-channel attacks have gained considerable research attention following the disclosure of the Spectre and Meltdown vulnerabilities. These vulnerabilities exposed hardware design flaws in speculative execution and out-of-order execution that can be exploited through software, posing serious security concerns for security-critical systems [1], [2]. In the literature, various detection and mitigation solutions have been proposed across different abstraction layers, ranging

from operating system (OS)-level approaches [3] to hardwarebased solutions [4], [5]. The state-of-the-art hardware-based runtime detection solutions leverage built-in Hardware Performance Counters (HPCs) [4]–[9]. Although these initial research efforts to detect Spectre and Meltdown by leveraging HPCs were a pragmatic solution at the time, subsequent research has revealed various limitations, indicating that HPC-based detection still has a long way to go before being considered a practical hardware-based solution [10]. HPC-based solutions fundamentally suffer because access is limited to a restricted set of hardware registers, while multiplexing issues further restrict the number of hardware registers that can be sampled simultaneously. Moreover, the coarse sampling granularity compromises detection speed, resulting in a speed-performance trade-off [10], [11]. Moreover, when attacks are hidden within benign applications (camouflaged setup), HPC-based detection solutions cannot detect them [12]. These limitations stems from the proprietary and confidential nature of commercial x86 processor microarchitectures, such as Intel and AMD. Consequently, access is limited to a restricted set of HPCs, which provide only aggregated event counts and cannot capture instantaneous processor behavior or internal RTL-level pipeline events. In recent years, RISC-V processors have gained widespread adoption due to their modularity, scalability, and open-source design. Nevertheless, their susceptibility to microarchitectural attacks due to the lack of hardware countermeasures remains a key security concern [13]. Recent research efforts have presented offline detection solutions using the gem5 computer architecture simulator to detect microarchitectural attacks [14]–[17]. However, runtime detection on real RISCV hardware remains unexplored. An initial effort to detect microarchitectural attacks on a RISC-V BOOM core leveraged Hardware Performance Counters (HPCs) [18]; however, it inherits the limitations of HPC-based detection. Recently, researchers demonstrated branch predictor attacks on the open-source RISC-V BOOM processor, targeting resourceconstrained devices and validating them on an FPGA [19]. Interestingly, different variants of these branch predictor attacks

exhibit timing differences of less than 20 CPU cycles. From a defensive perspective, timing differences of only 5–20 CPU cycles render these attacks inherently stealthy, making them completely indistinguishable from benign workloads using HPC-based detection. To address these limitations, we propose the first secure-bydesign, non-intrusive, RTL-level runtime detection solution for stealthy attacks on the RISC-V BOOM processor without speed-performance trade-offs, multiplexing issues, or performance overhead. This work addresses the inherent limitations of HPC-based detection by introducing a new secure-by-design research direction. Although the detection of microarchitectural attacks hidden inside benign workloads (camouflaged setup) remains beyond the scope of this paper, the proposed approach provides a strong foundation for future research in this direction. A. Contributions The key contributions of the paper are as follows. 1) To the best of our knowledge, this paper presents the first secure-by-design, non-intrusive, RTL-level runtime detection method for stealthy microarchitectural attacks on the RISC-V BOOM processor. 2) The proposed detection method is highly interpretable and automatically discovers frequent patterns using association rule mining, identifying the most relevant RTL signals for branch predictor attack detection. 3) An RTL-level runtime detection method that enables the finest possible access to every internal signal of the entire processor pipeline. In addition, the proposed non-intrusive rule-based monitor eliminates the inherent limitations of HPC-based detection, including event multiplexing, limited hardware counter availability, and coarse-grained event visibility, with negligible performance overhead. 4) Finally, we evaluate the proposed detection method across a sequence of computationally intensive, memoryintensive, and branch-heavy workloads, achieving zero false positives while demonstrating its adaptability to a previously unseen branch predictor attack variant. The remainder of this paper is organized as follows. Section II introduces the key terminology, Section III describes the threat model, Section IV presents the attack model, Section V outlines the proposed methodology, Section VI describes the experimental results, and Section VII concludes the paper and discusses future work. II. P RELIMINARIES This section introduces the key terminologies used throughout the paper and reviews the related work. Definition 1: Association Rule Mining (ARM) discovers association rules from frequent itemsets. An association rule is represented as X → Y , where X, Y ⊆ W and X ∩ Y = ∅. Here, W denotes the universal set of monitored RTL signals from RISC-V BOOM [20], [21].

Definition 2: Frequent Pattern Growth (FP-Growth) is one of the most efficient data mining algorithms for finding frequent itemsets from very large transactional datasets. It first compresses the transaction database into an FP-tree (Frequent Pattern Tree) and then extracts frequent patterns directly from the FP-tree. Definition 3: Support measures the frequency of occurrence of a frequent itemset and is defined as Supp(X) = P (X), where P (X) denotes the probability of occurrence of itemset X. In this paper, the minimum support threshold used as the input parameter to FP-Growth is set to θs = 5% [22]. Definition 4: Confidence measures the reliability of an association rule and is defined as Conf(X → Y ) = P (Y | X). In this paper, the minimum confidence threshold for association rule generation is set to θc = 90% [21]. Definition 5: Chipyard is an open-source RISC-V SoC generator framework integrating configurable RISC-V processor cores, memory subsystems, and peripheral devices. Definition 6: Verilator is an open-source Verilog/SystemVerilog simulator that compiles RTL designs into optimized C++ models for cycle-accurate simulation [23]. Definition 7: BOOM is an open-source out-of-order (O3) RISC-V processor core integrated within the Chipyard SoC generator framework [24]. Definition 8: Bimodal Predictor is a simple branch predictor that predicts the direction of conditional branches based on their previous outcomes [24]. Definition 9: Tagged Geometric History Length (TAGE) Predictor is a branch predictor that predicts branch directions using multiple prediction tables with different branch history lengths [24]. III. T HREAT M ODEL Our threat model considers an unprivileged attacker executing malicious code co-located with the victim on the same physical core of an out-of-order (O3) BOOM processor supporting speculative execution. The attacker can manipulate the shared branch predictor to launch branch predictor attacks and leak secret information through side channels but has no direct access to the internal branch predictor state and relies only on observable microarchitectural side effects. We assume a controlled and deterministic bare-metal execution environment and trusted hardware free from physical tampering and hardware Trojans. IV. ATTACK M ODEL The proposed run-time detection framework is designed for the branch predictor attack model adopted from [19], which consists of the following phases: 1. Training: The attacker trains the shared branch predictor through repeated branch executions. 2. Victim Execution: The victim executes secret-dependent branches, updating the shared branch predictor state. 3. Probe: The attacker probes the predictor state by measuring timing differences to infer secret information.

Step 1 : Configuring BOOM in Chipyard

Step 2 : RTL Signal Identification and Association Rule MiningBased Feature Selection

Physical Host System (x86 ISA) Chipyard Framework (Integrated SoC Design Framework) Hardware Generation and simulation Flow

customBoomV3Config 📜 (Scala)

Step 3 : Offline and Run-time detection

i- FST to VCD ii- VCD to CSV

Mined Association Rules

RISC-V CrossCompilation Workflow Attack/Benign 👻 Workloads (C/C++)🛡️ Cross-Compiler (GCC)

Chisel Hardware Generator ⚙️

Offline and Run-time Detection(RTD)

Embedding Rules in Hardware (AND LOGIC)

RISC-V executable file

SystemVerilog RTL

Verilator Cycle Accurate C++

⎍ BOOM Simulator

Cycle Accurate C++ BOOM Simulator (Verilated BOOM Simulator)

Re-Synthesize BOOM with Non-Intrusive Rule Monitor

Collect Traces (Debug mode) [FST Trace File]

RTD

Fig. 1. ANTMAN (Secure by design RTL level run-time detection framework)

V. M ETHODOLOGY In this paper, we propose a comprehensive branch predictor attack detection framework that automatically identifies the most relevant RTL signals using association rule mining and enables offline and run-time detection of stealthy branch predictor attacks on the RISC-V BOOM processor. In Step 1 of the proposed framework, the BOOM processor is configured within the Chipyard environment. The customized BOOM configuration is translated into synthesizable SystemVerilog and compiled by Verilator into a cycleaccurate C++ simulation model. The attack and benign workloads are cross-compiled into RISC-V executables and executed on the generated C++ model. Finally, the processor execution traces generated during each workload execution are captured as FST trace files in debug mode. In Step 2, the complete processor behavior during attack execution is available as FST traces. Initially, a set of RTL signals that may be affected by branch predictor attacks is selected based on domain knowledge. The selected FST traces are then converted into CSV format, where association rule mining generates association rules, enabling the automatic identification of the most relevant RTL signals affected during the attack execution. Finally, in Step 3, the association rules generated in Step 2 are used for offline attack detection across computational, memory-intensive, and branch-heavy workloads. The generated rules are then implemented in hardware as a non-intrusive rule monitor, where the detector triggers an assertion upon rule matching to detect stealthy branch predictor attacks at run time. The detailed implementation of each step illustrated in Fig. 1 is as follows. Step 1: Configuring BOOM in Chipyard The first step of the proposed ANTMAN framework is to configure the BOOM processor within the Chipyard

environment. Chipyard is a RISC-V-based system-on-chip (SoC) generation framework that provides an integrated environment for hardware generation, simulation, and software cross-compilation using the RISC-V GNU toolchain. In Chipyard, BOOM configuration and workload crosscompilation are performed as two independent phases. During the BOOM configuration phase, the same BOOM parameters used in [25] are adopted, as listed in row 2 of Table I. These BOOM configurations are first translated into a synthesizable SystemVerilog model of BOOM. Subsequently, Verilator translates this model into a cycle-accurate C++ model of BOOM. In the workload preparation phase, the inputs to this C++ model are generated by cross-compiling the source codes of the branch predictor attack workloads and a set of benign RISC-V benchmarks. The resulting cross-compiled executable files are provided as inputs to the BOOM model. Execution of these workload files in debug mode captures the complete end-to-end behavior of the CPU pipeline for each workload in FST waveform format. Table I summarizes the complete experimental setup and BOOM configurations. Step 2: RTL Signal Identification and Association Rule Mining-Based Feature Selection The output of Step 1 captures the complete CPU pipeline execution behavior under the attack workload in the form of an FST trace file. This step consists of two key phases: (a) RTL signal identification and (b) association rule mining-based feature selection. The following discussion briefly summarizes each phase. A. RTL Signal Identification In the first phase of Step 1, a set of RTL signals relevant to branch predictor attacks are selected from different

TABLE I E XPERIMENTAL S ETUP AND C ONFIGURATION Item Chipyard-Based Experimental Platform

BOOM Configuration [25]

Benign Workloads Attack Workloads Timing Configuration

Description Chipyard SoC generator framework BOOM v3 (Scaled Configuration) RV64GC ISA Verilator (Compiles Verilog/SystemVerilog RTL into cycle-accurate C++ models) RISC-V GNU Cross-Compilation Toolchain (riscv64-unknown-elf-gcc) Reorder Buffer: 48 entries Integer IQ: 16 entries Memory IQ: 10 entries FP IQ: 12 entries Integer PRF: 64 registers FP PRF: 48 registers Load Queue: 12 entries Store Queue: 12 entries Fetch Target Queue: 32 entries L1D Cache: 16 KB, 4-way, 64-B line L1I Cache: 16 KB, 4-way, 64-B line dhrystone, mm, median, qsort, spmv, towers, vvadd CBPA, IBPA 500 attack rounds using rdcycle-based latency measurements.

microarchitectural blocks of the BOOM processor based on the behavior of branch predictor attacks. As listed in Table II, these signals belong to microarchitectural blocks involved in branch prediction and recovery. The Frontend Branch Prediction Block, which combines the Next Line Predictor (NLP) and Branch Target Buffer (BTB), predicts the next instruction fetch address and provides branch target information during speculative instruction fetch. The BPD (Backing Predictor Direction) block predicts the branch direction (Taken/Not Taken) using predictors such as Bimodal or TAGE. Similarly, the Global History Register (GHR) block maintains and tracks previous branch outcomes and provides branch history information to improve future direction predictions. The Branch Resolution/Execution block determines the actual branch outcome during execution and compares it with the earlier speculative prediction to detect mispredictions. When a branch misprediction occurs, this information is propagated across microarchitectural blocks to initiate control-flow recovery. The Branch Update Broadcast block distributes branch resolution outcomes and misprediction information to the frontend and other microarchitectural blocks to coordinate recovery actions. The Frontend Redirect Control block redirects instruction fetch to the correct Program Counter (PC), while the Pipeline Flush/Recovery Control block flushes incorrectpath instructions and their dependent operations from queues and pipeline stages, ultimately restoring the processor state for correct execution. The FST trace file containing the selected RTL signals is first converted into VCD format and subsequently exported to CSV format to facilitate further analysis and extract meaningful insights. Table II summarizes the selected RTL signals.

TABLE II C ATEGORIZATION OF S ELECTED RTL S IGNALS Microarchitectural Block Global History Register (GHR) NLP / BTB (Next Line Predictor / Branch Target Buffer) BPD (Backing Predictor Direction) Branch Resolution / Execute Frontend Redirect Control Branch Update Broadcast Network

Pipeline Flush / Recovery Control

Associated RTL Signals ghist_s1_branch_not_taken, ghist_s2_branch_not_taken, ifu_redirect_ghist_saw_nt f3_btb_misp0, f3_btb_misp1, f3_btb_misp2, f3_btb_misp3 bpd_is_mispredict_update cpu_brupdate_b2_misp, exu_brinfo_misp, brinfo0_misp, b2_misp redirect_val, redirect_flush ifu_brupdate_b2_misp, lsu_brupdate_b2_misp, fp_rename_brupdate_b2_misp, rob_brupdate_b2_misp rob_flush_valid, rob_flush_frontend, dec_flush_pipeline, mem_issue_flush_pipeline, int_issue_flush_pipeline, fp_pipe_flush_pipeline, alu_queue_flush

B. Association Rule Mining-Based Feature Selection In Phase 2 of Step 2, after identifying the relevant RTL signals and capturing their complete execution behavior in a CSV file, the next important step is to apply association rule mining to discover relationships and patterns among the selected signals. This stage plays a crucial role, as humans cannot directly identify complex patterns among multiple signals across millions of clock cycles, considering signal variations and their interdependencies. Therefore, we opted for Frequent Pattern Growth (FP-Growth), defined in Definition 2, to efficiently identify hidden patterns in the execution of branch-predictor attacks. From the data mining output, we obtained a set of 16 interpretable association rules based on predefined minimum support 5% and confidence 90% thresholds, as defined in Definition 3 and Definition 4, respectively. From these association rules, we observe a highly structured and repeated coupling behavior among misprediction, redirect (control-flow correction), and full-pipeline flush signals. Three dominant behavioral clusters are identified: Global Branch Misprediction, Global Redirect (Control-Flow Correction), and Global Pipeline Flush. The following discussion presents a detailed analysis of each cluster. 1) Global Branch Misprediction: Table III presents the first cluster, which reveals strong coupling among branch misprediction signals across multiple microarchitectural blocks. Multiple units, including the Reorder Buffer (ROB), Instruction Fetch Unit (IFU), Load-Store Unit (LSU), and FloatingPoint Rename Unit (FP rename), simultaneously detect misprediction events along with control-flow correction signals (redirect_valid and redirect_flush), indicating coordinated pipeline-wide misprediction detection and recovery.

TABLE III G LOBAL B RANCH M ISPREDICTION A SSOCIATION RULES Rule

Antecedent

Consequent

Rule 1

cpu brupdate b2 misp AND ifu brupdate b2 misp AND b2 misp AND fp rename brupdate b2 misp AND lsu brupdate b2 misp AND redirect val AND redirect flush cpu brupdate b2 misp AND rob brupdate b2 misp AND b2 misp AND fp rename brupdate b2 misp AND lsu brupdate b2 misp AND redirect val AND redirect flush rob brupdate b2 misp AND ifu brupdate b2 misp AND b2 misp AND fp rename brupdate b2 misp AND lsu brupdate b2 misp AND redirect val AND redirect flush cpu brupdate b2 misp AND rob brupdate b2 misp AND ifu brupdate b2 misp AND b2 misp AND fp rename brupdate b2 misp AND redirect val AND redirect flush cpu brupdate b2 misp AND rob brupdate b2 misp AND ifu brupdate b2 misp AND fp rename brupdate b2 misp AND lsu brupdate b2 misp AND redirect val AND redirect flush cpu brupdate b2 misp AND rob brupdate b2 misp AND ifu brupdate b2 misp AND b2 misp AND lsu brupdate b2 misp AND redirect val AND redirect flush

rob brupdate b2 misp

Rule 2 Rule 3 Rule 4 Rule 5 Rule 6

ifu brupdate b2 misp cpu brupdate b2 misp lsu brupdate b2 misp b2 misp fp rename brupdate b2 misp

TABLE IV G LOBAL C ONTROL F LOW C ORRECTION A SSOCIATION RULES Rule

Antecedent

Consequent

Rule 1

dec flush pipeline AND int issue flush pipeline AND alu queue flush AND mem issue flush pipeline AND fp pipe flush pipeline AND ifu redirect ghist saw nt AND redirect flush cpu brupdate b2 misp AND rob brupdate b2 misp AND ifu brupdate b2 misp AND b2 misp AND fp rename brupdate b2 misp AND lsu brupdate b2 misp AND redirect flush cpu brupdate b2 misp AND rob brupdate b2 misp AND ifu brupdate b2 misp AND b2 misp AND fp rename brupdate b2 misp AND lsu brupdate b2 misp AND redirect val dec flush pipeline AND int issue flush pipeline AND alu queue flush AND mem issue flush pipeline AND fp pipe flush pipeline AND ifu redirect ghist saw nt AND redirect val

redirect val

Rule 2 Rule 3 Rule 4

redirect val redirect flush redirect flush

TABLE V G LOBAL R ECOVERY A SSOCIATION RULES Rule

Antecedent

Consequent

Rule 1

cpu brupdate b2 misp AND rob brupdate b2 misp AND ifu brupdate b2 misp AND b2 misp AND lsu brupdate b2 misp AND redirect val AND redirect flush int issue flush pipeline AND alu queue flush AND mem issue flush pipeline AND fp pipe flush pipeline AND ifu redirect ghist saw nt AND redirect val AND redirect flush dec flush pipeline AND int issue flush pipeline AND mem issue flush pipeline AND fp pipe flush pipeline AND ifu redirect ghist saw nt AND redirect val AND redirect flush dec flush pipeline AND alu queue flush AND mem issue flush pipeline AND fp pipe flush pipeline AND ifu redirect ghist saw nt AND redirect val AND redirect flush dec flush pipeline AND int issue flush pipeline AND alu queue flush AND fp pipe flush pipeline AND ifu redirect ghist saw nt AND redirect val AND redirect flush dec flush pipeline AND int issue flush pipeline AND alu queue flush AND mem issue flush pipeline AND ifu redirect ghist saw nt AND redirect val AND redirect flush dec flush pipeline AND int issue flush pipeline AND alu queue flush AND mem issue flush pipeline AND fp pipe flush pipeline AND redirect val AND redirect flush

fp rename brupdate b2 misp

Rule 2 Rule 3 Rule 4 Rule 5 Rule 6 Rule 7

2) Redirect (Control-flow correction): The second cluster, shown in Table IV, reveals strong coupling between redirect signals and multiple microarchitectural blocks. The

dec flush pipeline alu queue flush int issue flush pipeline mem issue flush pipeline fp pipe flush pipeline ifu redirect ghist saw nt

mined association rules show that redirect_valid and redirect_flush are strongly associated with pipeline flush and branch misprediction events across different microar-

RTL Signals Source cpu_brupdate_b2_misp

Non-Intrusive rule-monitor (RuleMonitor.scala) Rule 1

if (Rule X= 1) ; CounterX = CounterX + 1

ifu_brupdate_b2_misp

b2_misp

Counter 1 (16 bits)

When (All Rule’s Trigger)

Counter 2 (16 bits)

Trigger ASSERTION

Rule 2

fp_rename_brupdate_b2_misp

lsu_brupdate_b2_misp

Rule 3 redirect_valid

Counter 3 (16 bits)

redirect_flush rob_brupdate_b2_misp

Action in Core.Scala

Rule 4

Counter 4 (16 bits)

dec_flush_pipeline

Threshold comparison of each counter with predefined Support Threshold

ALARM: Any Branch Predictor attack detected

mem_issue_flush_pipeline

int_issue_flush_pipeline fp_pipe_flush_pipeline

Rule 16 Counter 16 (16 bits)

alu_queue_flush

Terminate the Simulation

ifu_redirect_ghist_saw_nt

Fig. 2. Non-Intrusive Rule Monitor

chitectural blocks, enabling coordinated control-flow recovery. When speculation fails, redirect decisions redirect instruction fetch to the correct path and initiate recovery by propagating control-flow correction across the processor pipeline. 3) Global Pipeline Flush (Recovery): The association rules in the third cluster (Table V) highlight strong coordination among pipeline flush signals across multiple microarchitectural blocks. The association rules show that flush events in the decode, issue, execution, memory, and floating-point units are strongly associated with redirect_valid and redirect_flush indicating coordinated pipeline recovery activity following control-flow redirection. Step 3: Offline and Run-time detection In the final step (Step 3), the extracted rules were first evaluated offline using benign and attack workloads under minimum support and confidence thresholds of 5% and 90%, respectively. After validation, the rules were implemented and synthesized as a non-intrusive hardware monitor that tracks rule activations and stops execution when the activation count exceeds the prescribed support threshold, enabling hardwarelevel detection of stealthy branch predictor attacks. From these association rules, we obtain a subset of pre-identified signals across different microarchitectural blocks that are most relevant to branch predictor attack detection. Consequently, the association-rule mining indirectly performs feature selection

by identifying the most relevant signals affected during attack execution, thereby enabling efficient run-time detection capability. Figure 2 shows the integration of the rule-based monitor into the BOOM RISC-V microarchitecture. It can be observed that only 14 automatically discovered RTL signals were tapped from different microarchitectural blocks, and the association rules were converted into hardware using AND logic without causing any interference with ongoing processor operations. The rule-monitoring hardware was implemented in a separate ‘RuleMonitor.scala’ module, which was instantiated in ‘Core.scala’, and the design was subsequently re-synthesized. These embedded rule monitors count every time an association rule is triggered and store the corresponding count for each rule in a dedicated 16-bit counter. If the trigger count of all association rules exceeds the predefined support threshold, an assertion is raised at run time, and program execution is immediately terminated. VI. E XPERIMENTAL R ESULTS In this section, we present a systematic evaluation of the proposed detection framework to detect stealthy branch predictor attacks across a sequence of computational, memoryintensive, and branch-heavy workloads. The systematic evaluation first validates the detection framework in an offline

environment using only the simplified Next-Line Predictor (NLP) configuration. Once the detection framework is validated using Association Rule Mining (ARM), we carry out run-time detection using both the simplified NLP and the more complex TAGE predictor configuration to evaluate the effectiveness of the proposed detector at run time. The evaluation also considers the effectiveness and practical aspects of the proposed non-intrusive run-time detection framework against stealthy attacks in RISC-V, including detection speed, interpretability, detection accuracy, and performance overhead, and compares the proposed framework with state-of-the-art detection solutions.

----------------------------------------------ANALYZING WORKLOAD TRACES: qsort.csv Summary for qsort.csv Rules Triggered : 0/16 (0%) Prediction: Very low probability of an attack. ----------------------------------------------ANALYZING WORKLOAD TRACES: median.csv Summary for median.csv Rules Triggered : 0/16 (0%)

A. Offline Detection This section aims to validate the effectiveness of Association Rule Mining with the simplified NLP configuration in an offline environment. The association rules are generated using one variant of branch predictor attack (CBPA) and evaluated against both attack variants (CBPA and IBPA) alongside all benign applications. It can be seen from Listing 1 that, with the simplified NLP configuration, both variants of branch predictor attacks can be detected with no false positives.

Prediction: Very low probability of an attack. ----------------------------------------------ANALYZING WORKLOAD TRACES: spmv.csv Summary for spmv.csv Rules Triggered : 0/16 (0%) Prediction: Very low probability of an attack.

B. Run-time detection After validating the effectiveness of association rule mining in detecting stealthy branch predictor attacks, we embed the extracted rules into the RISC-V BOOM hardware as a nonintrusive monitor to detect these attacks at run-time. In this section, we consider both variants of branch predictor attacks (CBPA and IBPA) against simplified NLP and TAGE predictor configurations. The benign workloads remain the same for each configuration, consisting of computational, memoryintensive, and branch-heavy workloads used for both predictor configurations. ANALYZING WORKLOAD TRACES: CBPA.csv Summary for CBPA.csv Rules Triggered : 16/16 (100%) Prediction: High probability that the branch predictor is under attack. ----------------------------------------------ANALYZING WORKLOAD TRACES: dhrystone.csv Summary for dhrystone.csv Rules Triggered : 0/16 (0%) Prediction: Very low probability of an attack. ----------------------------------------------ANALYZING WORKLOAD TRACES: mm.csv Summary for mm.csv Rules Triggered : 0/16 (0%) Prediction: Very low probability of an attack.

----------------------------------------------ANALYZING WORKLOAD TRACES: towers.csv Summary for towers.csv Rules Triggered : 0/16 (0%) Prediction: Very low probability of an attack. ----------------------------------------------ANALYZING WORKLOAD TRACES: vvadd.csv Summary for vvadd.csv Rules Triggered : 0/16 (0%) Prediction: Very low probability of an attack. ----------------------------------------------ANALYZING WORKLOAD TRACES: IBPA.csv Summary for IBPA.csv Rules Triggered : 16/16 (100%) Prediction: High probability that the branch predictor is under attack. Listing 1. Offline branch predictor attack detection using Association Rule Mining (ARM) with a simplified next-line predictor configuration.

[UART] UART0 is here (stdin/stdout). BRAD-v1 timing test starting... SECRET_DATA_LEN = 16 ATTACK_ROUNDS = 500 Index 0 | true=1 | avg(spy0)=97 cycles | avg(spy1)=78 cycles | faster=spy1 -> guess=1

----------------------------------------------Index 1 | true=0 | avg(spy0)=76 cycles | avg(spy1)=99 cycles | faster=spy0 -> guess=0 Index 2 | true=0 | avg(spy0)=76 cycles | avg(spy1)=99 cycles | faster=spy0 -> guess=0

Index 3 | true=1 | avg(spy0)=97 cycles | avg(spy1)=78 cycles | faster=spy1 -> guess=1 Assertion failed: Branch prediction related attack detected. Simulation terminated. -----------------------------------------------

[UART] UART0 is here (stdin/stdout). IBPA with indirect jump helper (rdcycle timing) Secret: 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 ATTACK_ROUNDS = 500 Index 0 : mean_probe = 97 cycles Index 1 : mean_probe = 105 cycles Index 2 : mean_probe = 105 cycles Index 3 : mean_probe = 97 cycles Index 4 : mean_probe = 105 cycles Index 5 : mean_probe = 97 cycles Index 6 : mean_probe = 105 cycles Index 7 : mean_probe = 97 cycles Index 8 : mean_probe = 97 cycles Index 9 : mean_probe = 97 cycles Assertion failed: Any branch prediction related attack detected. Simulation aborted.

[UART] UART0 is here (stdin/stdout). IBPA with indirect jump helper (rdcycle timing)

Listing 3. Run-time detection of CBPA and IBPA attacks under the complex TAGE configuration

Secret: 1 0 0 0 1 0 1 0 1 1 1 0 0 0 1 0 ATTACK_ROUNDS = 500 Index 0 : mean_probe = 97 cycles Index 1 : mean_probe = 105 cycles Index 2 : mean_probe = 105 cycles Index 3 : mean_probe = 97 cycles Assertion failed: Branch prediction related attack detected. Simulation aborted. Listing 2. Run-time detection of CBPA and IBPA attacks with the simplified NLP configuration.

[UART] UART0 is here (stdin/stdout). BRAD-v1 timing test starting... SECRET_DATA_LEN = 16 ATTACK_ROUNDS = 500 Index 0 | true=1 | avg(spy0)=76 cycles | avg(spy1)=100 cycles | faster=spy0 -> guess=0 Index 1 | true=0 | avg(spy0)=76 cycles | avg(spy1)=100 cycles | faster=spy0 -> guess=0 Index 2 | true=0 | avg(spy0)=98 cycles | avg(spy1)=78 cycles | faster=spy1 -> guess=1 Index 3 | true=1 | avg(spy0)=97 cycles | avg(spy1)=78 cycles | faster=spy1 -> guess=1 Assertion failed: Any branch prediction related attack detected. Simulation terminated.

From Listings 2 and 3, it can be observed that the ARM-based detector successfully detects and terminates attacks at runtime. Under the simplified NLP configuration, both CBPA and IBPA attacks are detected after 25% of the attack execution. As reported in [25], the TAGE predictor configuration only reduces the probability of successfully recovering the secret. Our simulation results validate this observation, showing an approximately 50% success rate for secret recovery under the TAGE configuration for CBPA. However, existing cryptanalytic techniques have demonstrated that recovering 50% of the cryptographic keys through sidechannel leakage can enable extraction of the remaining key information. Therefore, the proposed ARM-based detector demonstrates that stealthy attacks under complex TAGE configurations remain detectable at the RTL level. Under the TAGE configuration, CBPA is detected at 25% of attack completion with a 50% secret recovery success rate, whereas the unseen IBPA variant is detected at 56.25% of completion. Hence, the proposed RTL-level non-intrusive monitor provides an additional hardware security layer, enabling a secure-bydesign solution for run-time detection of stealthy attacks across simple and complex branch predictor configurations. Similarly, for all benign workloads, under both simplified NLP and complex TAGE configurations, no benign workload is terminated during execution, resulting in zero false positives. This detection capability is enabled by association rules that capture structured attack patterns comprising global branch mispredictions across multiple microarchitectural blocks, control-flow correction, and systematic recovery through flushing dependent instructions across the processor pipeline. Table VII summarizes the attack detection points, expressed as the percentage of attack completion, under different branch predictor configurations.

TABLE VI C OMPARISON OF THE P ROPOSED F RAMEWORK (ANTMAN) WITH S TATE - OF - THE -A RT A PPROACHES Literature Mushtaq et al. [4] Mushtaq et al. [5] Carnà et al. [6] Alam et al. [7] Polychronou et al. [8] Choudhari et al. [9] Awais et al. [26] Khan et al. [27] Palumbo [16] Hassan et al. [17] Le et al. [28] Proposed Framework (ANTMAN)

ISA

Sampling Source

Secure by Design

Offline / Runtime

Stealthy Attack

Detection Speed

Accuracy

x86 x86 x86 x86 x86 x86 x86 RISC-V RISC-V RISC-V RISC-V RISC-V

HPCs HPCs HPCs HPCs HPCs HPCs gem5 gem5 gem5 gem5 HPCs RTL traces

No No No No No No No No No No No Yes

Runtime Runtime Runtime Runtime Runtime Runtime Offline Offline Offline Offline N/A Runtime

No No No No No No No No No No No Yes

Yes Yes Yes Yes Yes Yes No No No No No Yes

Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

TABLE VII ATTACK D ETECTION POINTS U NDER D IFFERENT B RANCH P REDICTOR C ONFIGURATIONS Attack CBPA IBPA

Simplified NLP 25% 25%

Complex TAGE 25% 56.25%

C. Comparison with State-of-the-art The following table VI provides a systematic comparison of the proposed overall framework with state-of-the-art. From Table VI, it is evident that the proposed detection framework is the first RTL-level, secure-by-design, run-time attack detection framework enabled by the open-source nature of the RISC-V architecture. The framework is capable of detecting stealthy branch predictor attacks before attack completion, while also providing high interpretability and the flexibility to identify new variants within the same attack family. In contrast, stateof-the-art run-time detection approaches that rely on hardware performance counters (HPCs) suffer from inherent limitations, including restricted visibility into microarchitectural events and dependence on proprietary processor implementations, where only a limited set of events can be monitored through HPC sampling. VII. C ONCLUSION AND F UTURE WORK As open-source RISC-V processors become ubiquitous across diverse applications, including critical systems, their security remains a key concern. However, the open-source nature of RISC-V provides a unique opportunity for the research community to architect efficient, secure-by-design solutions. This work presents the first secure-by-design, interpretable, association rule-based, non-intrusive RTL-level runtime detection method for stealthy branch predictor attacks, validated on BOOM RISC-V. The proposed detection method enables efficient runtime detection without the inherent limitations of HPC-based approaches. Evaluation under simplified Next-Line Predictor (NLP) and complex TAGE predictor configurations

Flexibility Interpretability No No No No No No No No No Yes No Yes

Low Low Low Low Low Low Low Low Low High Low High

demonstrates fast detection, termination before secret disclosure, and flexibility in detecting previously unseen variants within the same branch predictor attack family. The robustness of the approach is further validated across computational, memory-intensive, and branch-heavy workloads, achieving zero false positives while ensuring runtime detection. From a future research perspective, the proposed framework establishes a foundation for extending this end-to-end proofof-concept to a broader range of microarchitectural attacks on RISC-V and enabling efficient mitigation solutions. A windowed approach can further be introduced to track associationrule activations during stochastic workload execution. This approach offers two key advantages: tracking the number of rule activations and passively monitoring execution duration across windows to identify patterns that exceed predefined support thresholds, thereby enabling non-intrusive detection of attacks hidden within benign workloads. Finally, a detailed power, performance, and area (PPA) analysis can quantify resource utilization and evaluate performance and area overheads prior to fabrication. ACKNOWLEDGMENT This work was supported by the Estonian Research Council grants PSG837. • The icons and graphical elements used in this work were generated with assistance from OpenAI’s ChatGPT image generation tool. The generated images were subsequently reviewed, modified, and adapted by the authors for inclusion in this manuscript. •

R EFERENCES [1] P. Kocher, J. Horn, A. Fogh, D. Genkin, D. Gruss, W. Haas, M. Hamburg, M. Lipp, S. Mangard, T. Prescher, M. Schwarz, and Y. Yarom, “Spectre attacks: exploiting speculative execution,” Commun. ACM, vol. 63, no. 7, p. 93–101, Jun. 2020. [Online]. Available: https://doi.org/10.1145/3399742 [2] M. Lipp, M. Schwarz, D. Gruss, T. Prescher, W. Haas, A. Fogh, J. Horn, S. Mangard, P. Kocher, D. Genkin, Y. Yarom, and M. Hamburg, “Meltdown: reading kernel memory from user space,” in Proceedings of the 27th USENIX Conference on Security Symposium, ser. SEC’18. USA: USENIX Association, 2018, p. 973–990. [3] M. Mushtaq, M. M. Yousaf, M. K. Bhatti, V. Lapotre, and G. Gogniat, “The kingsguard os-level mitigation against cache side-channel attacks using runtime detection,” Annals of Telecommunications, vol. 77, no. 11–12, pp. 731–747, 2022. [Online]. Available: https://doi.org/10.1007/s12243-021-00906-3 [4] M. Mushtaq, A. Akram, M. K. Bhatti, M. Chaudhry, V. Lapotre, and G. Gogniat, “Nights-watch: a cache-based side-channel intrusion detector using hardware performance counters,” in Proceedings of the 7th International Workshop on Hardware and Architectural Support for Security and Privacy, ser. HASP ’18. New York, NY, USA: Association for Computing Machinery, 2018. [Online]. Available: https://doi.org/10.1145/3214292.3214293 [5] M. Mushtaq, J. Bricq, M. K. Bhatti, A. Akram, V. Lapotre, G. Gogniat, and P. Benoit, “Whisper: A tool for run-time detection of side-channel attacks,” IEEE Access, vol. 8, pp. 83 871–83 900, 2020. [6] S. Carnà, S. Ferracci, F. Quaglia, and A. Pellegrini, “Fight hardware with hardware: Systemwide detection and mitigation of side-channel attacks using performance counters,” Digital Threats, vol. 4, no. 1, Mar. 2023. [Online]. Available: https://doi.org/10.1145/3519601 [7] M. Alam, S. Bhattacharya, and D. Mukhopadhyay, “Victims can be saviors: A machine learning–based detection for micro-architectural side-channel attacks,” J. Emerg. Technol. Comput. Syst., vol. 17, no. 2, Jan. 2021. [Online]. Available: https://doi.org/10.1145/3439189 [8] N. F. Polychronou, P.-H. Thevenon, M. Puys, and V. Beroulle, “Madman: Detection of software attacks targeting hardware vulnerabilities,” in 2021 24th Euromicro Conference on Digital System Design (DSD), 2021, pp. 355–362. [9] A. Choudhari, S. Guilley, and K. Karray, “Specdefender: Transient execution attack defender using performance counters,” in Proceedings of the 2022 Workshop on Attacks and Solutions in Hardware Security, ser. ASHES’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 15–24. [Online]. Available: https://doi.org/10.1145/3560834.3563830 [10] S. Das, J. Werner, M. Antonakakis, M. Polychronakis, and F. Monrose, “Sok: The challenges, pitfalls, and perils of using hardware performance counters for security,” in 2019 IEEE Symposium on Security and Privacy (SP), 2019, pp. 20–38. [11] M. Mushtaq, P. Benoit, and U. Farooq, “Challenges of Using Performance Counters in Security Against Side-Channel Leakage,” in CYBER 2020 - 5th International Conference on Cyber-Technologies and Cyber-Systems, Nice, France, Oct. 2020. [Online]. Available: https://hal.science/hal-02979362 [12] W. Kosasih, Y. Feng, C. Chuengsatiansup, Y. Yarom, and Z. Zhu, “Sok: Can we really detect cache side-channel attacks by monitoring performance counters?” in Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, ser. ASIA CCS ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 172–185. [Online]. Available: https://doi.org/10.1145/3634737. 3637649 [13] L. Gerlach, D. Weber, R. Zhang, and M. Schwarz, “A security risc: Microarchitectural attacks on hardware risc-v cpus,” in 2023 IEEE Symposium on Security and Privacy (SP), 2023, pp. 2321–2338. [14] M. Khan, M. Mushtaq, R. Pacalet, and L. Apvrille, “Side-channel attack detection using gem5 and machine learning: A case study on fault-based attacks in risc-v,” in 2025 IEEE 31st International Symposium on OnLine Testing and Robust System Design (IOLTS), 2025, pp. 1–5. [15] ——, “Assessing security risc: Analyzing flush+ fault attack on riscv using gem5 simulator,” in International Conference on Security and Cryptography (SECRYPT), 2025. [16] A. Palumbo, “Machine learning-based detection of microarchitectural attacks on risc-v via gem5,” in Proceedings of TechDefense 2025 -

IEEE International Workshop on Technologies for Defense and Security, Rome, Italy, November 2025, pp. 1–6, https://hal.sciencehal-05288651. [17] M. Hassan, M. Mushtaq, J. Raik, and T. Ghasempouri, “Drsam: Detection of fault-based microarchitectural side-channel attacks in risc-v using statistical preprocessing and association rule mining,” 2025. [Online]. Available: https://arxiv.org/abs/2510.18612 [18] A.-T. Le, T.-T. Hoang, B.-A. Dao, A. Tsukamoto, K. Suzaki, and C.-K. Pham, “Spectre attack detection with neutral network on riscv processor,” in 2022 IEEE International Symposium on Circuits and Systems (ISCAS), 2022, pp. 2467–2471. [19] M. Khan, M. E. Bin Mohd Shahfie, M. Mushtaq, R. Pacalet, and L. Apvrille, “Microarchitectural espionage: Fpga-based security analysis of branch prediction in risc-v out-of-order cores,” in 2026 14th International Symposium on Digital Forensics and Security (ISDFS), 2026, pp. 1–7. [20] T. Ghasempouri, S. Payandeh Azad, B. Niazmand, and J. Raik, “An automatic approach to evaluate assertions’ quality based on data-mining metrics,” in 2018 IEEE International Test Conference in Asia (ITC-Asia), 2018, pp. 61–66. [21] M. R. H. Iman, P. Chikul, G. Jervan, H. Bahsi, and T. Ghasempouri, “Anomalous file system activity detection through temporal association rule mining.” in ICISSP, 2023, pp. 733–740. [22] T. Ghasempouri, S. Payandeh Azad, B. Niazmand, and J. Raik, “An automatic approach to evaluate assertions’ quality based on data-mining metrics,” in 2018 IEEE International Test Conference in Asia (ITC-Asia), 2018, pp. 61–66. [23] W. Snyder, “Verilator: The Fastest Verilog/SystemVerilog Simulator,” https://www.veripool.org/verilator, 2023, accessed: 2024. [24] J. Zhao, B. Korpan, A. Gonzalez, and K. Asanovic, “SonicBOOM: The 3rd generation Berkeley out-of-order machine,” in Fourth Workshop on Computer Architecture Research with RISC-V (CARRV), May 2020. [25] M. Khan, M. E. Bin Mohd Shahfie, M. Mushtaq, R. Pacalet, and L. Apvrille, “Microarchitectural espionage: Fpga-based security analysis of branch prediction in risc-v out-of-order cores,” in 2026 14th International Symposium on Digital Forensics and Security (ISDFS), 2026, pp. 1–7. [26] M. Awais, M. Mushtaq, L. Naviner, F. Bruguier, P. Benoit, and J. Haj-Yahya, “Leveraging gem5 and machine learning for endto-end detection of cache-based side-channel attack patterns,” in PROOFS 2025 - 13th International Workshop on Security Proofs for Embedded Systems, Kuala Lumpur, Malaysia, September 2025, hAL ID: hal-05372979. [Online]. Available: https://hal.science/hal-05372979 [27] M. Khan, M. Mushtaq, R. Pacalet, and L. Apvrille, “Side-channel attack detection using gem5 and machine learning: A case study on fault-based attacks in risc-v,” in 2025 IEEE 31st International Symposium on OnLine Testing and Robust System Design (IOLTS), 2025, pp. 1–5. [28] T.-T. Hoang, B.-A. Dao, A. P. Tsukamoto, K. Suzaki, and C.-K. Pham, “Spectre attack detection with neutral network on risc-v processor,” in Proceedings of the 2022 IEEE International Symposium on Circuits and Systems (ISCAS), Austin, TX, USA, 2022, pp. 2467–2471.

Related documents

• SENTINEL: A Multi-Pathway Architecture for Detecting Living-Off-the-Land APT Attacks on Windows Command Lines#1084536
• CVE-2026-94382 — LOW 2.3#1040055
• DAIRE: A lightweight AI model for real-time detection of Controller Area Network attacks in the Internet of Vehicles#123949
• ISIA-AF: Orchestrating Reproducible Attacks and Multi-Source Data Collection for OT Systems#1011890
• ISIA-AF: Orchestrating Reproducible Attacks and Multi-Source Data Collection for OT Systems#965350