AttackonCTF: Defending Hardware Security Competition Benchmarks in the Age of LLMs Mohamadreza Rostami† , Nikhilesh Singh† , Stephen Muttathil§ , Lichao Wu‡ , Chen Chen§ , Huimin Li⋄, Jeyavijayan Rajendran§ and Ahmad-Reza Sadeghi† † Technical University of Darmstadt, Germany § Texas A&M University, USA ‡ University of Bristol, United Kingdom ⋄ Shenzhen University of Advanced Technology, China
arXiv:2606.15809v1 [cs.CR] 14 Jun 2026
{mohamadreza.rostami,nikhilesh.singh,ahmad.sadeghi}@trust.tu-darmstadt.de {chenc,stephen7929,jv.rajendran}@tamu.edu [email protected] [email protected]
Abstract Hardware security competitions such as HackTheSilicon serve as benchmarking platforms for evaluating vulnerability detection methods and for training humans and AI. However, our study reveals that LLMs threaten their validity. Instead of genuine security reasoning, detectors exploit a diff-style syntactic comparison, achieving an 83% detection rate, undermining fair evaluation. To mitigate this, we propose the first LLM-oriented, semanticspreserving obfuscation framework for these benchmarks. Unlike IP-protection approaches, it applies human-readable transformations and controlled diff-noise while preserving functionality. On HackTheSilicon, the framework reduces LLM-based detection accuracy by 50% with only 10% obfuscation and by 78.6% under complete obfuscation, restoring benchmark reliability.
CCS Concepts • Security and privacy → Security in hardware.
Keywords Hardware Security, HackTheSilicon, Hardware CTF, LLM ACM Reference Format: Mohamadreza Rostami† , Nikhilesh Singh† , Stephen Muttathil§ , Lichao Wu‡ , Chen Chen§ , Huimin Li⋄ , Jeyavijayan Rajendran§ and Ahmad-Reza Sadeghi† . 2026. AttackonCTF: Defending Hardware Security Competition Benchmarks in the Age of LLMs. In 63rd ACM/IEEE Design Automation Conference (DAC ’26), July 26–29, 2026, Long Beach, CA, USA. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3770743.3804202
1
Introduction
The exponential growth in hardware complexity has fundamentally transformed hardware security [14, 33, 2, 19]. However, the predominantly black-box nature of hardware creates a critical gap between industry needs and academic research capabilities [40, 14]. Indeed, commercial hardware designs are typically proprietary, limiting researchers’ access to realistic security challenges, thus creating a development and validation gap for vulnerability detection techniques [22, 40, 34, 14]. To address this challenge, we organize
This work is licensed under a Creative Commons Attribution 4.0 International License. DAC ’26, Long Beach, CA, USA © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2254-7/2026/07 https://doi.org/10.1145/3770743.3804202
the HackTheSilicon event in partnership with Intel and Synopsys [40]. Initiated in 2018, we have hosted 15 competitions with over 1,600 participants, using open-source SoC designs embedded with industry-informed vulnerabilities [35, 14, 9, 13, 40]. HackTheSilicon serves two primary purposes: (1) providing industry-validated open-source vulnerability benchmarks for evaluating existing or newly developed vulnerability detection tools, and (2) training hardware security researchers through realistic challenges. Furthermore, these benchmarks serve broader purposes: evaluating LLM-based vulnerability detection approaches [12], training AI models for hardware security analysis [12], and serving as demonstrative examples in the Common Weakness Enumeration (CWE) [24] database. “Shortcut” with LLM. HackTheSilicon is built on open-source SoC designs with injected vulnerabilities [40, 14, 18]. Due to the availability of the register-transfer level (RTL) code, participants or detection tools could, in principle, simply diff the original and modified SoCs to locate inserted bugs without performing any real security analysis. To prevent this, the bug submission process has always required a comprehensive technical report for each identified vulnerability, including its security impact, CWE/CVSS classifications, and proof-of-concept exploits. This requirement ensures that participants cannot rely solely on diffing and must demonstrate genuine hardware security understanding. However, recent advances in LLMs [23] undermine this protection mechanism [12, 1]. Specifically, LLM-based vulnerability detection tools can bypass the intended need for deep vulnerability analysis by combining simple diffing with LLM augmentation techniques, such as retrieval-augmented generation (RAG) that leverage prior open-source designs, history branches, and commits [32, 7]. Importantly, while RAG may appear to perform genuine analysis, it actually retrieves the original design context (based on similarity). This process of fetching similar code is functionally equivalent to an automated diff against the buggy code in the prompt. As a result, these methods generate high-quality, seemingly expert vulnerability reports without performing genuine security reasoning on the vulnerable design. While they may perform well in CTF settings, we argue that they threaten both the educational integrity of such competitions and the validity of benchmarks meant to evaluate LLM-based methods that are expected to reason directly on the malicious design. Indeed, high performance may simply reflect automated diffing either explicitly or implicitly via RAG and text generation rather than real analysis. Our Objective & Solution. Given the effectiveness of LLM-based detection methods, a natural question arises: How can we prevent existing LLM-based “shortcuts” while maintaining the educational
DAC ’26, July 26–29, 2026, Long Beach, CA, USA
Rostami et al.
Table 1: Traditional Obfuscation vs. AttackonCTF Characteristic Primary Goal Target Adversary Human Readability Educational Value AI Training Utility CWE Usability
Traditional IP Protection Human Reverse Engineers ✗ ✗ ✗ ✗
AttackonCTF Benchmark Integrity LLM-based Detectors ✓ ✓ ✓ ✓
and research integrity of the CTF? While potential solutions exist, such as patch randomization, structural refactoring, and multibranch reshuffling, an ideal solution must be lightweight enough for practical use, aligned with the objectives of HackTheSilicon, and, crucially, must preserve the ability of participants and researchers to learn from and reason about the malicious design. This leads to a natural alternative: hardware obfuscation. Traditional obfuscation tools [3, 37, 10] were developed for IP protection and rely on aggressive techniques, e.g., identifier encryption, hashing, and layout anonymization, that render RTL extremely challenging to understand. While ideal for preventing reverse engineering, as shown in Table 1, this level of obfuscation directly conflicts with the goals of a security competition or open benchmark: it prevents human learning, hinders the development of next-generation LLM-based detectors, and eliminates the usefulness of examples for CWE-style vulnerability documentation. In this work, we introduce the first LLM-oriented obfuscation framework tailored specifically for hardware security competitions and benchmarks. Our approach preserves readability while strategically disrupting the syntactic patterns and structural cues exploited by LLMs. The framework applies semantic-preserving transformations to break direct correlation with the original code and injects benign modifications to create controlled diff noise that masks the true security-relevant changes. Unlike traditional obfuscation, the resulting designs remain interpretable for human analysts and suitable for developing and evaluating (LLM-based) reasoning-based detection tools. Overall, our main contributions are: • We provide the first empirical demonstration that LLM-based vulnerability detection benefits heavily from diffing and syntactic comparison, achieving 75–83% detection rates without genuine hardware security reasoning. • We develop the first readability-preserving, LLM-specific obfuscation framework with diff noise, context dilution, and semantic misdirection, effectively defending diff-based LLM exploitations. • On HackTheSilicon benchmarks, our method reduces the detection accuracy of the existing approach by 50% at 10% obfuscation and by 78.6% at full obfuscation, without compromising human readability or functional correctness. • Our format-agnostic framework incorporates automated verification pipelines to ensure that all obfuscated designs remain semantically equivalent and fully functional.
2
Background
Unlike the software community, where numerous open-source applications exist, including widely-used and commercialized systems
such as the Linux kernel, which serve as benchmarks for security analysis and evaluation of vulnerability detection tools, the hardware community faces significant limitations. Due to the proprietary nature of hardware intellectual property (IP), open-source hardware designs are scarce, with only a limited number of projects available [7, 32]. This scarcity creates a substantial gap between the quality of security analysis requirements and standards maintained in industry versus those accessible to academia. To address this gap, initiatives combining academia and industry expertise have emerged. HackTheSilicon (also known as Hack@Event), founded in 2017, has become the world’s largest hardware security competition [17]. Beyond their educational value in training the next generation of hardware security researchers, these competitions serve as critical benchmarking platforms for the hardware security research community, providing standardized vulnerability datasets that enable researchers to evaluate and compare automated vulnerability detection tools and methodologies against industry-relevant security flaws. These competitions follow a structured format. First, organizers select complex open-source SoC designs such as OpenPiton [7] or OpenTitan [32] and inject vulnerabilities inspired by real-world vulnerabilities, guided by insights from industry partners and CWE classifications [25]. Vulnerabilities span various categories, including cryptographic weaknesses, access control flaws, information leakage channels, and life cycle security issues, providing academia with realistic benchmarks for evaluating vulnerability detection approaches. Next, teams receive the modified SoC design and industrystandard EDA tools from Synopsys [38], simulating real-world hardware security verification workflows. Participants are free to use any tools to detect vulnerabilities and submit comprehensive reports, including vulnerability localization, technical security analysis, CWE/CVSS classifications, proof-of-concept exploits, and methodology descriptions. Finally, a team of industry and academic experts evaluates these submissions.
3
Design and Implementation
To scientifically validate our hypothesis that LLMs exploit syntactic difference patterns rather than performing genuine hardware security analysis, in Section 3.1, we design four progressively sophisticated LLM-based vulnerability detection methods, inspired by recent LLM-assisted code analysis research [16, 12, 36, 39] and attack strategies observed from HackTheSilicon participants [40]. Guided by these insights, in Section 3.2, we develop an LLM-driven obfuscation framework to break these detection mechanisms while preserving readability and functional correctness.
3.1
LLM-Based Vulnerability Detection Methods
We consider four vulnerability detectors (D1-D4) with progressively richer contextual awareness to systematically characterize the LLM-assisted hardware vulnerability detection methods. These detectors represent realistic attack scenarios that competition participants or automated tools could employ, ranging from naive diff-based approaches to sophisticated systems incorporating repository history and domain knowledge. We standardize the prompting interface, decoding parameters, and output schema to isolate the effect of context available to the model.
AttackonCTF
DAC ’26, July 26–29, 2026, Long Beach, CA, USA
Prompt Design
HackTheSilicon Benchmark
Compare (DIFF) Buggy and Clean
RAG DB (RTL CODE)
Common Weakness Enumeration
RAG DB (CWE)
API CALL
Detector 1 (D1)
Prompt Design
Prompt: You are a hardware security… you will get buggy code and related Opentitian SoC code from GitHub as the context. Opentitian: …. BuggyCode: ….
API CALL
Detector 2 (D2)
Opentitan Benchmark
RAG DB (PR + COMMIT)
Prompt: You are a hardware security researcher… You should analyze code diffs for security vulnerabilities. Code diff (-,+): …
Prompt Design
Prompt: You are a hardware security … you will get buggy code and related Opentitian SoC code and PR and Commits from GitHub as the context. Opentitian: …. PRs: …. BuggyCode: ….
API CALL
Detector 3 (D3)
Prompt Design
Prompt: You are a hardware security… you will get buggy code and related Opentitian SoC from GitHub and CWE knowledge as the context. Opentitian: …. CWE: …. BuggyCode: ….
API CALL
Detector 4 (D4)
Figure 1: Overview of AttackonCTF detectors.
Figure 1 provides an overview of the experimental setup for different detectors. For each candidate program instance, regardless of detector type, the output requirements are standardized. Each detector must: (1) determine the presence of a vulnerability in the given source, (2) identify the lines responsible for the buggy behavior, and (3) produce a structured report explaining the defect mechanism, triggering conditions, and expected impact on program behavior. A critical distinction exists in the input modalities: D1 receives only diffs between buggy and original code, while D2-D4 receive complete buggy source code and leverage RAG systems to retrieve relevant context (e.g., RTL code and commits), representing typical LLM deployment patterns for code analysis. Diff-Aware LLM (D1). This detector represents the straightforward detection approach. The detector receives only a unified diff output between the candidate buggy code and the reference baseline from the repository’s latest commit. The LLM is directly prompted to analyze whether the given diff, i.e., Code diff(-,+), introduces vulnerabilities and produces a structured report describing the defect security mechanism, triggering conditions, and expected impact. This detector directly tests whether simple syntactic diff analysis suffices for vulnerability detection without deeper code comprehension or security reasoning. Repository-Aware RAG (D2). The detector receives the complete buggy source code (not a diff), representing a realistic deployment where users of the LLM detection method have access to the entire vulnerable program. Before inference, the repository is indexed at the function-level granularity using code-aware chunking, which preserves boundaries and symbol information. At query time, a retrieval query is constructed from the buggy source code, and the RAG system retrieves relevant code fragments, function definitions, and module documentation from the indexed repository. This is the typical deployment pattern for LLM-powered code analysis tools [16, 12, 36, 39]. The model analyzes the buggy code in light of the retrieved context to assess the presence of vulnerabilities, identify the responsible lines, and ground its explanation in a broader understanding of the codebase. This detector assesses whether LLMs can identify vulnerabilities through genuine security analysis with repository context, without requiring diff hints.
Figure 2: Example output from our obfuscation framework applied to the ibex_cs_registers module in OpenTitan [32]. Additions are marked with green (+) and original replaced code is marked with red (–).
History-Aware RAG (D3). Building on D2, this detector incorporates development history to test temporal reasoning. The RAG corpus is extended to include not only the SoC design but also relevant pull requests (PRs) and commit metadata, enabling the retrieval of discussions that motivated past changes to related components. Given the buggy code as input, the RAG system constructs queries spanning both code and history, retrieving relevant PR/commit alongside code fragments. The model was again prompted to assess the existence of any vulnerability in the given buggy code, considering the context. This detector assesses whether the historical development context enhances vulnerability reasoning beyond static code analysis. Weakness-Aware RAG (D4). This detector represents the most sophisticated threat scenario with domain-specific security knowledge. The RAG system augments D2’s repository context with a hardware security domain knowledge derived from the CWE database. At retrieval time, the system pairs repository context with concise CWE entries and implementation-oriented examples from the security domain, enabling the model to frame its diagnosis in terms of specific weakness classes. The model analyzes the buggy code against retrieved CWE patterns, determines the presence of bugs, highlights the responsible lines, and produces a report that labels the issue with CWE identifiers where applicable. This detector assesses whether domain-specific security knowledge facilitates genuine vulnerability reasoning, or whether LLMs continue to rely on pattern matching despite rich contextual support.
3.2
Obfuscation Framework
To enable a fair, research-grade benchmark that supports AI training, competition, and education while remaining suitable for evaluating LLM-based vulnerability detection methods, we introduce the first LLM-driven obfuscation framework for hardware security benchmarks. Given an existing industry-standard benchmark (e.g., HackTheSilicon [40]), our framework applies semantics-preserving transformations to security-critical hardware designs. Unlike conventional hardware obfuscation techniques intended for IP protection [3, 37, 10], our design explicitly targets LLM-based vulnerability analysis methods by preventing direct syntactic comparison (e.g.,
DAC ’26, July 26–29, 2026, Long Beach, CA, USA
Rostami et al.
Constraint Extraction
Hardware Security Benchmark
Obfuscated Benchmark
Obfuscation Adjustment
Span Selection
System Prompt: You are an expert RTL developer. Your task is to rewrite the RTL CODE file, considering the following constraints. Location/Interface/Equivalency Constraints: …. Rewrite Features: …. User Prompt: RTL CODE: ….
Prompt Design
API CALL
CODE Rewriting (Obfuscation)
Syntax Checking
Semantic Checking
Hardware Code Patch
Figure 3: Overview of the proposed obfuscation framework. diff-based detection), thereby enabling the assessment of their true reasoning-based security analysis capabilities. Figure 3 illustrates our introduced obfuscation pipeline. The obfuscation process begins with Constraint Extraction 1 , where the framework instantiates formal invariants that the LLM must preserve during code rewriting. These constraints (1) prohibit modifications outside specified line ranges, (2) changes to module interfaces or port declarations, (3) alterations to signal widths or data types, (4) modifications to reset behavior, and (5) changes to finitestate machine semantics. These invariants are enforced through the rewriter’s system prompt. The framework supports tunable obfuscation levels between 0% to 100% (Obfuscation Adjustment 1 ), enabling controlled evaluation of robustness versus overhead (e.g., token usage). Based on the configured obfuscation level, the Span Selection 2 identifies vulnerable code regions to transform. The system verifies the validity of each span, ensuring no overlap with constraints and protected areas, and computes target boundaries with variations in line count. This context enables the LLM to reason locally about equivalent rewrites while confining edits to the target range. Next, for Code Rewriting 3 , the framework invokes a stateof-the-art LLM to perform functionally equivalent transformations using five strategies: (1) identifier renaming with semantically neutral alternatives, (2) control-flow restructuring through equivalent expressions and state machine reorganization, (3) diff-noise injection which disrupts syntactic similarity with canonical reference designs, (4) context dilution which obscures or disperses securityrelevant cues and (5) logic rewriting through Boolean algebra and temporal transformations. Figure 2 illustrates our obfuscation mechanisms through a representative OpenTitan code transformation. The LLM returns the rewritten segment as a Hardware Code Patch 4 with metadata (technique applied, confidence score, and potential side effects). A triviality filter strips comments and whitespaces to ensure that only meaningful edits are performed. Syntax checking 5 compiles the modified design using Verilator [41] for type checking, and SystemVerilog rule verification. If compilation fails, the framework captures error messages and provides
them to the LLM for a retry attempt. This iterative refinement ensures all obfuscated code remains compilable. Beyond syntactic correctness, it is also crucial to ensure that the obfuscated code is functionally equivalent to the original source code. The Semantic Checking 6 stage performs this verification using the Yosys [44] tool, which uses SAT-based proofs for equivalence checking. If equivalence fails, the framework analyzes counterexample traces and restarts RTL rewriting with additional constraints derived from the failed proof to maintain benchmark integrity.
4
Evaluation
Our evaluation investigates four fundamental questions. First, we assess how effective are current LLM-based vulnerability detectors in identifying hardware vulnerabilities when diff-based or referencebased shortcuts remain available? (RQ1). Second, we examine whether LLM-based vulnerability detection methods perform principled security reasoning about vulnerabilities? (RQ2). Third, we measure how effectively our obfuscation framework preserves human readability while preventing LLM-based vulnerability detection from reporting vulnerabilities based on diff analysis rather than genuine reasoning about vulnerability and security feature malfunctions (RQ3). Finally, we analyze how different levels of obfuscation impact the accuracy of LLM-based vulnerability detection (RQ4). In the rest of the section, first we describe our evaluation setup in Section 4.1 and then answer each of our research questions based on our experiments, presented in Section 4.2, 4.3, and 4.4.
4.1
Evaluation Setup
Benchmark Characteristics. We evaluate our framework on three recent HackTheSilicon competition benchmarks: Hack@DATE’25, Hack@DAC’25, and Hack@CHES’25. All benchmarks are based on Google’s OpenTitan [32], an open-source silicon root-of-trust platform incorporating cryptographic accelerators, secure boot, and key management. The benchmarks contain 50 unique vulnerabilities across 27 CWE categories, including top Hardware CWEs [24]. Vulnerabilities span multiple security domains including sensitive information in resource not removed before reuse
AttackonCTF
4.2
RQ1: Baseline LLM Vulnerability Detection Effectiveness
We evaluate four detectors (D1–D4) on unobfuscated HackTheSilicon benchmarks [40]. Across both LLM models, we observe high vulnerability detection rates: D1 = 83.3%, D2 = 75.0%, D3 = 77.1%, and D4 = 77.1% (Figure 4). These results demonstrate that when syntactic or semantic reference information is accessible, either explicitly as in D1, or implicitly through contextual retrieval as in D2–D4, LLM-based detectors can identify approximately three out of four injected vulnerabilities, answering RQ1. However, the closely aligned performance between the direct diffbased approach (D1) and the context-retrieval approaches (D2–D4) suggests that all detectors employ similar internal strategies to identify vulnerabilities and raise the critical question: are LLMs performing genuine security analysis, or simply exploiting the availability of syntactically similar original code? In real-world industry scenarios, verification tools have access to only a single codebase, there is no "original clean version" to compare against. If these
Detection Rate (%)
100 80
83.3%
75.0%
77.1%
77.1%
D2
D3
D4
60 40 20 0
D1
Detector
Figure 4: Effectiveness of LLM-based Vulnerability Detectors on HackTheSilicon [40] Benchmarks. 100
Detection Rate (%)
(CWE-226 [30]), improper isolation of shared resources on SoC (CWE-1189 [26]), on-chip debug and test interface with improper access control (CWE-1191 [27]), improper handling of overlap between protected memory ranges (CWE-1260 [28]), and improper access control for register interface (CWE-1262 [29]). The vulnerabilities are distributed across 26 hardware modules, including AES/HMAC engines, flash/OTP controllers, entropy sources, and the Ibex RISC-V core [21]. LLM Models. We evaluate Claude Sonnet 4.5 [4] and GPT-5 [31], representing state-of-the-art LLMs for code analysis and reasoning [4, 31]. Both models accessed via API with 128, 000 token limits and temperature 0.0 to maximum reproducibility. Verification Tools. We employ Verilator [41] for syntax validation and Yosys [44] for formal semantic equivalence checking. Yosys performs RTL-to-RTL equivalence verification using equiv_make and equiv_simple commands with SAT-based proof generation. LLM-based Hardware Vulnerability Detector. We implemented all four vulnerability detection methods described in Section 3.1 and evaluated their performance against our obfuscation framework (Section 3.2) using four obfuscation levels (0%, 10%, 50%, 100%). Each experiment configuration is evaluated with three independent runs to account for LLM nondeterminism, using standardized prompts with temperature 0.0 and a 5,000-token output limit. For RQ2–RQ4, we focus on D1 and D4 as representative extremes: D1 isolates explicit syntactic comparison, while D4 represents the strongest RAG-enhanced detectors. Since preliminary runs showed D2/D3 exhibit similar trends to D4, evaluating D1 and D4 captures meaningful detector variation while managing experimental cost. Evaluation Metric. We measure two primary performance metrics: vulnerability detection rate and accuracy. Detection Rate quantifies the percentage of ground-truth injected vulnerabilities correctly identified by the detector, reflecting its recall capability. Detection Accuracy measures the percentage of detector-reported vulnerabilities that correspond to actual vulnerabilities, capturing its precision and resistance to false positives. Together, these metrics characterize both the ability of a detector to fully find real vulnerabilities and its reliability in avoiding erroneous reports.
DAC ’26, July 26–29, 2026, Long Beach, CA, USA
80
83.3%
77.1%
Detector 1 Detector 4 69.6% 56.5%
60
63.0%
55.8%
54.3%
48.9%
40 20 0
0
10
50
Obfuscation Level (%)
100
Figure 5: Effectiveness of Different Levels of Our Obfuscation on LLM-based Vulnerability Detection Rate.
LLM-based detectors are performing genuine security reasoning, they should maintain similar detection rates and accuracy even when the original code is not accessible. The following research questions (RQ2–RQ4) investigate this hypothesis by evaluating detector performance when our obfuscation framework simulates real-world conditions.
4.3
RQ2 & RQ3: Obfuscation Against Diff-Based Detection
Figure 5 and 6 present both detection rates and accuracy metrics across obfuscation levels for D1 and D4. While D1’s detection rate decreases from 83.3% to 54.3% (35% reduction) with 100% obfuscation, its accuracy falls far more severely from 67.2% to 15.5% (77% reduction). Similarly, D4 shows a decline in detection rate from 77.1% to 48.9% (37% reduction), while accuracy falls from 69.2% to 15.3% (78% reduction). The substantial gap between detection and accuracy degradation reveals a critical insight: as syntactic changes increase, LLMs report more vulnerabilities, resulting in more false positives (FPs) and consequently decreasing accuracy. If LLMs performed genuine security analysis, semantic-preserving obfuscation should have
DAC ’26, July 26–29, 2026, Long Beach, CA, USA
Rostami et al.
Average Accuracy (%)
100 80
Detector 1 Detector 4 67.2% 69.2%
60 35.8%
40
30.0%
25.2%
27.5% 15.5% 15.3%
20 0
0
10
50
Obfuscation Level (%)
100
Figure 6: Effectiveness of Different Levels of Our Obfuscation on LLM-based Vulnerability Detection Accuracy. minimal impact on the quality of detection. Instead, accuracy reductions far exceed detection rate reductions, empirically proving that LLM-based detectors focus on syntactic comparisons (diff-based detection) rather than actual vulnerability analysis. Notably, even D4, which receives RAG context rather than explicit diffs, exhibits this behavior, indicating that it ultimately performs diff-based analysis internally. This explains why increased syntactic differences between original and obfuscated benchmarks directly correlate with increased false positives, a pattern inconsistent with genuine security reasoning but consistent with diff-based exploitation.
4.4
RQ4: Impact of Obfuscation Levels on LLM Detection
Analyzing progressive degradation across obfuscation levels, Figure 5 and 6, reveals that detection rates decrease monotonically (D1: 83.3% → 54.3%; D4: 77.1% → 48.9%), while accuracy shows dramatic non-linear degradation with 48-55% drops at just 10% obfuscation. The RAG-enhanced detector (D4) demonstrates greater initial sensitivity (21% detection drop vs. 16% for D1 at 10%), suggesting RAG retrieval mechanisms are particularly vulnerable to syntactic transformation. Our results illustrated, even minimal 10% obfuscation using our method provides substantial accuracy reduction (48-55%), while 100% achieves near-complete mitigation (77-78% reduction).
5
Related Work
Software obfuscation has extensive literature [8, 11, 43]. Techniques include control flow obfuscation [6], data obfuscation [5], and opaque predicates [42]. Hardware obfuscation primarily focuses on IP protection. Traditional Hardware IP Protection Tools. The hardware security community has developed numerous obfuscation approaches to protect intellectual property from unauthorized disclosure and reverse engineering. Commercial solutions such as Aldec’s HDL Code Obfuscator [3] and Semantic Designs’ Verilog Obfuscator [37] employ systematic identifier renaming using cryptographic hashes, structural information removal, and formatting elimination to maximize protection against IP theft. The Verible toolchain [10] provides similar capabilities through automated symbol mangling and semantic cue elimination. While these tools effectively protect proprietary
designs, their aggressive obfuscation strategies—including complete identifier encryption and structural anonymization, render code unreadable to human and AI analysts, fundamentally incompatible with educational and AI learning objectives that require comprehensible security analysis. LLM-Based Gate-Level Obfuscation. Recent research has explored leveraging LLMs for automated hardware logic obfuscation [20, 15]. These approaches operate at the gate-level netlist abstraction, instructing LLMs to insert key-programmable logic gates (XOR/XNOR functions), multiplexers, and dummy circuit elements while preserving input-output interface specifications. The resulting obfuscated netlists maintain functional correctness but require sophisticated reverse engineering to extract the original design intent. However, gate-level obfuscation fundamentally transforms the analysis domain from RTL code examination to netlist reconstruction. This eliminates the core human and AI learning objective of hardware security competitions and benchmarks. Despite their effectiveness for IP protection, existing hardware obfuscation approaches are fundamentally unsuitable for creating a hardware security benchmark that can be used for training machine learning models, participating in capture-the-flag competitions, and evaluating LLM and AI-based vulnerability detection approaches. Traditional IP protection tools prioritize complete code obscuration over human and AI comprehensibility, employing encryption, aggressive symbol mangling, and structural elimination that prevent meaningful security analysis by both humans and AI. These limitations motivate our development of the first LLM-assisted obfuscation, designed explicitly to create benchmarks with unique considerations for human and AI learning settings, and to evaluate AI-based hardware vulnerability detection methods while preventing automated diff-based exploitation of LLM-based approaches.
6
Conclusion
This paper presented the first systematic analysis revealing that LLM-based hardware vulnerability detection approaches primarily rely on diff-based pattern matching rather than genuine security reasoning when evaluated on open-source benchmarks like HackTheSilicon, achieving 75-83% detection rates through automated exploitation of diff operations against known original designs. In response, we introduced the first LLM-based obfuscation framework specifically designed for hardware security competitions and benchmarks, employing semantic-preserving transformations that prevent diff-based exploitation while maintaining human readability. Experimental results demonstrate substantial effectiveness: even minimal 10% obfuscation reduces LLM accuracy by 48-55%, while comprehensive 100% obfuscation achieves maximum detection rate and accuracy reductions of 55% and 78% respectively, reducing final accuracy to approximately 15%, successfully disadvantaging diff-based approaches while maintaining functionality.
Acknowledgments Our research was partially funded by Intel’s Scalable Assurance Program, DFG-SFB 1119-236615297, NSF-DFG-Grant 538883423, and the ERC Programme-Grant 101055025-HYDRANOS. The views expressed in this work are those of the authors and do not represent endorsements or official positions of the funding organizations.
AttackonCTF
References [1]
[2] [3] [4] [5]
[6]
[7] [8] [9]
[10] [11] [12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21] [22]
[23]
[24]
[25]
Baleegh Ahmad, Hammond Pearce, Ramesh Karri, and Benjamin Tan. 2025. Lashed: llms and static hardware analysis for early detection of rtl bugs. arXiv preprint arXiv:2504.21770. Sonia Akter, Kasem Khalil, and Magdy Bayoumi. 2023. A survey on hardware security: current trends and challenges. IEEE Access, 11, 77543–77565. Aldec, Inc. [n. d.] HDL Code Obfuscation. https://www.aldec.com/en/support /resources/documentation/articles/1586. Accessed: 2025-11-10. (). Anthropic. 2025. Introducing Claude Sonnet 4.5. https://www.anthropic.com/n ews/claude-sonnet-4-5. Online; accessed 2025-11-10. (2025). David E Bakken, R Rarameswaran, Douglas M Blough, Andy A Franz, and Ty J Palmer. 2004. Data obfuscation: anonymity and desensitization of usable data sets. IEEE Security & Privacy, 2, 6, 34–41. Vivek Balachandran, Ng Wee Keong, and Sabu Emmanuel. 2014. Function level control flow obfuscation for software security. In 2014 Eighth International Conference on Complex, Intelligent and Software Intensive Systems. IEEE, 133– 140. Jonathan Balkind et al. 2016. Openpiton: an open source manycore research framework. ACM SIGPLAN Notices, 51, 4, 217–232. Boaz Barak. 2016. Hopes, fears, and software obfuscation. Communications of the ACM, 59, 3, 88–96. Chen Chen, Rahul Kande, Pouya Mahmoody, Ahmad-Reza Sadeghi, and JV Rajendran. 2022. Trusting the trust anchor: towards detecting cross-layer vulnerabilities with hardware fuzzing. In Proceedings of the 59th ACM/IEEE Design Automation Conference, 1379–1383. Chipsalliance. 2025. Verible: a suite of systemverilog developer tools. https://gi thub.com/chipsalliance/verible. Accessed: 2025-11-10. (2025). Christian Collberg, Clark Thomborson, and Douglas Low. 1997. A taxonomy of obfuscating transformations. (1997). Luca Collini, Baleegh Ahmad, Joey Ah-kiow, and Ramesh Karri. 2025. Marvel: multi-agent rtl vulnerability extraction using large language models. arXiv preprint arXiv:2505.11963. DAC ’19: Proceedings of the 56th Annual Design Automation Conference 2019. Las Vegas, NV, USA, (2019). Association for Computing Machinery. isbn: 9781450367257. Ghada Dessouky, David Gens, Patrick Haney, Garrett Persyn, Arun Kanuparthi, Hareesh Khattri, Jason M Fung, Ahmad-Reza Sadeghi, and Jeyavijayan Rajendran. 2019. HardFails: Insights into Software-Exploitable Hardware Bugs. USENIX Security Symposium. Jugal Gandhi, Diksha Shekhawat, M Santosh, Jaya Dofe, and Jai Gopal Pandey. 2024. Large language model driven logic locking: a generative approach to secure ic design. In 2024 IEEE 33rd Asian Test Symposium (ATS). IEEE, 1–4. Rui Guo, Avinash Ayalasomayajula, Henian Li, Jingbo Zhou, Sujan Kumar Saha, and Farimah Farahmandi. 2025. Svagent: ai agent for hardware security verification assertion. arXiv preprint arXiv:2507.16203. Intel. 2025. Raising awareness of hardware security weaknesses: intel research and hack@dac. https://www.intel.com/content/www/us/en/security/securi ty-practices/blogs/raising-awareness-hardware-security-weaknesses.html. Accessed: 2025-11-05. (2025). Arun Kanuparthi, Hareesh Khattri, Jason Fung, J. V. Rajendran, Ahmad-Reza Sadeghi, Rahul Kande, Chen Chen, and Mohamadreza Rostami. 2024. The hack@dac story: learnings from organizing the world’s largest hardware hacking competition. Presented at Black Hat USA. Briefings Presentation. (2024). https://i.blackhat.com/BH-US-24/Presentations/US-24-Kanuparthi-HackAt DACStory-Wednesday.pdf. Troya Çağıl Köylü, Cezar Rodolfo Wedig Reinbrecht, Anteneh Gebregiorgis, Said Hamdioui, and Mottaqiallah Taouil. 2023. A survey on machine learning in hardware security. ACM Journal on Emerging Technologies in Computing Systems, 19, 2, 1–37. Banafsheh Saber Latibari, Sujan Ghimire, Muhtasim Alam Chowdhury, Najmeh Nazari, Kevin Immanuel Gubbi, Houman Homayoun, Avesta Sasan, and Soheil Salehi. 2024. Automated hardware logic obfuscation framework using gpt. In 2024 IEEE 17th Dallas Circuits and Systems Conference (DCAS). IEEE, 1–5. lowRISC. [n. d.] lowRISC ibex RISC-V Core. https://github.com/lowRISC/ibex. Online; accessed 2025-11-10. (). Paul Marks. 2025. Security research gaps leave critical infrastructure open to cyberattack. Communications of the ACM (CACM) News. https://cacm.acm.org /news/security-research-gaps-leave-critical-infrastructure-open-to-cyberat tack/. Andrea Matarazzo and Riccardo Torlone. 2025. A survey on large language models with some insights on their capabilities and limitations. arXiv preprint arXiv:2501.04040. MITRE. 2025. 2025 most important hardware weaknesses. Online; accessed 2025-11-10. (2025). https://cwe.mitre.org/topHW/archive/2025/2025_CWE_MI HW.html. MITRE. [n. d.] Common weakness enumeration. Online; accessed 2025-11-10. (). https://cwe.mitre.org.
DAC ’26, July 26–29, 2026, Long Beach, CA, USA
[26]
[27]
[28]
[29]
[30]
[31] [32] [33]
[34]
[35]
[36] [37]
[38] [39]
[40] [41] [42]
[43]
[44]
MITRE. [n. d.] Cwe-1189: improper isolation of shared resources on systemon-a-chip (soc). Online; accessed 2025-11-10. (). https://cwe.mitre.org/data/def initions/1189.html. MITRE. [n. d.] Cwe-1191: on-chip debug and test interface with improper access control. Online; accessed 2025-11-10. (). https://cwe.mitre.org/data/defi nitions/1191.html. MITRE. [n. d.] CWE-1260: Improper Handling of Overlap Between Protected Memory Ranges. Online; accessed 2025-11-10. (). https://cwe.mitre.org/data/de finitions/1260.html. MITRE. [n. d.] CWE-1262: Improper Access Control for Register Interface. Online; accessed 2025-11-10. (). https://cwe.mitre.org/data/definitions/1262.ht ml. MITRE. [n. d.] CWE-226: Sensitive Information in Resource Not Removed Before Reuse. Online; accessed 2025-11-10. (). https://cwe.mitre.org/data/defin itions/226.html. OpenAI. [n. d.] GPT-5 is here. Online; accessed 2025-11-10. (). https://openai.c om/gpt-5/. OpenTitan Project. [n. d.] Opentitan: open source silicon root of trust. Accessed: 2025-11-12. (). https://opentitan.org. Mohamadreza Rostami, Chen Chen, Rahul Kande, Huimin Li, Jeyavijayan Rajendran, and Ahmad-Reza Sadeghi. 2024. Fuzzerfly Effect: Hardware Fuzzing for Memory Safety. IEEE Security & Privacy. Mohamadreza Rostami, Shaza Zeitouni, Rahul Kande, Chen Chen, Pouya Mahmoody, Jeyavijayan Rajendran, and Ahmad-Reza Sadeghi. 2024. Lost and Found in Speculation: Hybrid Speculative Vulnerability Detection. ACM/IEEE Design Automation Conference. Ahmad-Reza Sadeghi, Jeyavijayan Rajendran, and Rahul Kande. 2021. Organizing the world’s largest hardware security competition: challenges, opportunities, and lessons learned. In Proceedings of the 2021 Great Lakes Symposium on VLSI, 95–100. Dipayan Saha et al. 2025. Sv-llm: an agentic approach for soc security verification using large language models. arXiv preprint arXiv:2506.20415. Semantic Designs. [n. d.] Verilog Source Code Obfuscator. Accessed: 2025-1110. (). https://semanticdesigns.com/Products/Obfuscators/VerilogObfuscator.h tml. Synopsys. [n. d.] VCS: Functional Verification Solution. Online; accessed 202511-10. (). https://www.synopsys.com/verification/simulation/vcs.html. Shams Tarek, Dipayan Saha, Sujan Kumar Saha, and Farimah Farahmandi. 2025. Bugwhisperer: fine-tuning llms for soc hardware vulnerability detection. In 2025 IEEE 43rd VLSI Test Symposium (VTS). IEEE, 1–5. HackTheSilicon Team. [n. d.] Hack the silicon. Online; accessed 2025-11-10. (). https://hackthesilicon.com/. Veripool. [n. d.] Verilator, the fastest Verilog/SystemVerilog simulator. Online; accessed 2025-11-10. (). https://www.veripool.org/verilator/. Dongpeng Xu, Jiang Ming, and Dinghao Wu. 2016. Generalized dynamic opaque predicates: a new control flow obfuscation method. In International Conference on Information Security. Springer, 323–342. Hui Xu, Yangfan Zhou, Jiang Ming, and Michael Lyu. 2020. Layered obfuscation: a taxonomy of software obfuscation techniques for layered security. Cybersecurity, 3, 1, 9. YosysHQ. 2025. Yosys: a suite of open-source rtl synthesis tools. Online; accessed 2025-11-10. (2025). https://github.com/YosysHQ/yosys.