This paper will be presented at the 2026 IEEE International Conference on Omni-layer Intelligent Systems (COINS 2026), https://coinsconf.com/.
(Special Session)
VeriChat: An Agentic Conversational AI Assistant for Hardware Security Verification Dipayan Saha, Khan Thamid Hasan, Shams Tarek, Sujan Kumar Saha, Mark Tehranipoor, and Farimah Farahmandi
arXiv:2607.01668v1 [cs.CR] 2 Jul 2026
Department of Electrical and Computer Engineering, University of Florida Gainesville, FL, USA {dsaha, khanthamidhasan, shams.tarek, sujansaha}@ufl.edu, {tehranipoor, farimah}@ece.ufl.edu
Abstract—Hardware security verification is a multi-stage process in which engineers must navigate complex design analyses, threat considerations, and verification strategies. They often need security-focused guidance, yet current verification environments provide little structured support for such assistance. Although conversational AI could offer such on-demand assistance, directly using general-purpose chatbots like ChatGPT or Gemini is risky due to their tendency to hallucinate and their reliance on static, outdated knowledge. We present VeriChat, a domain-specialized conversational assistant designed to support, rather than replace, existing verification workflows by providing context-aware security guidance. VeriChat employs a retrieval-augmented, multiagent workflow in which three specialized agents collaboratively minimize hallucinations while improving the transparency and reliability of the response. Beyond question answering, VeriChat integrates open-source EDA tools, including Icarus Verilog, Yosys, and SymbiYosys, to perform syntax checking, synthesis analysis, simulation, and formal verification directly on user-provided RTL designs. Evaluated using a comprehensive methodology, VeriChat achieves a Faithfulness score of 87.73%, significantly outperforming the leading proprietary models. We demonstrate the framework through a hardware Trojan detection case study on an AES S-Box IP, where VeriChat autonomously identifies, simulates, and formally proves a covert key-leakage vulnerability through a multi-turn conversational workflow. Keywords—Hardware Security Verification, Large Language Model, Retrieval-Augmented Generation, EDA Tool Integration
I. I NTRODUCTION Hardware security verification is an incredibly demanding and tedious process that requires immense time and effort. Security verification engineers frequently encounter decision points throughout the verification lifecycle where they must exercise complex security-specific judgments. At many of these points, they would benefit from a helping hand or security-focused guidance. At any stage, they may need to seek ❶ conceptual clarification and reasoning-based assessments, for example, to understand security-relevant design behaviors, interpret threat relevance, or evaluate the soundness of their security properties and assumptions. As they move into planning and execution, they may seek ❷ suggestions on appropriate verification methods, aiming to identify which approaches are 979-8-3195-0489-0/26/$31.00 ©2026 IEEE We thank the U.S. National Science Foundation (NSF) for support through CAREER Award No. 2339971.
the most suitable for addressing specific security concerns. They may also participate in the ❸ development of ideas oriented to security, devising new strategies, rules, or metrics to fill the gaps left by existing approaches. During active verification, they often perform ❹ diagnostic and debugging reasoning to interpret ambiguous or unexpected outcomes and resolve failed checks. Finally, when results are obtained, they must perform ❺ results validation to determine whether the security coverage achieved is sufficient and aligned with accepted practices. Throughout these stages, engineers are repeatedly required to pause, reflect, and seek security-focused guidance, but current verification environments provide little structured support to help them do so. Throughout these stages, engineers frequently need security-focused guidance, yet current verification environments provide little structured support, forcing them to rely on manual information gathering that can lead to inefficiency, inconsistency, and overlooked vulnerabilities. This gap gives rise to our first research question: RQ1: How can security verification engineers be provided with on-demand, security-focused guidance that reduces their reliance on manual information gathering? A natural candidate to provide such guidance is a generalpurpose large language model (LLM) such as ChatGPT or Gemini. However, applying these models directly to hardware security poses serious risks. Despite their impressive language capabilities, these models have two fundamental limitations that make them unsuitable for high-assurance domains. The first is (1) hallucination, the tendency to produce factually incorrect or fabricated content. The second is their (2) static knowledge base: although modern LLMs increasingly support web-based retrieval to access recent information, these capabilities remain shallow and unstructured for specialized technical domains. General-purpose LLMs still cannot reliably incorporate newly discovered attacks, emerging threats, or evolving verification practices with the depth and precision required in hardware security. Relying on a tool that may confidently invent “facts” and lacks awareness of current security practices is unacceptable in a domain where correctness is the primary objective, as illustrated in Figure 1. These limitations raise our second research question: RQ2: How can a conversational assistant for hardware security verification be designed to deliver trustworthy, up-
Query I have a Register-Transfer Level (RTL) implementation of an AES-128 core written in Verilog. I need to perform a comprehensive pre-silicon security assessment to identify vulnerabilities to power side-channel attacks. Specifically, I am looking for a detailed overview of established frameworks that can be used for power sidechannel leakage assessment.
Response Established Frameworks for Power Side-Channel Leakage Assessment (Pre-Silicon) 1. SILVER – Key Idea: Estimates power side-channel leakage from RTL via switching activity and statistical metrics (KL divergence, TVD). Paper: “SILVER: Simulation Leakage Evaluation for Hardware Security Validation” – HOST 2019 2. RTL-PSC – Type: Simulation-based + Information-theoretic Key Idea: Characterizes RTL designs using MI and KL Divergence over Hamming weight or toggle traces. Features: Metrics: MI, KL, TVD; Trace models: HW, HD, toggle count Paper: “RTL-PSC: Automated Power Side-channel Leakage Assessment at RTL” – DATE 2020 3. SCARF – Key Idea: Uses symbolic execution on RTL to identify power side-channel vulnerabilities before synthesis. Reference: “SCARF: RTL-based Side Channel Analysis using Formal Methods” – DAC 2018
Fig. 1: An example of hallucination by ChatGPT-4o in response to a hardware security query. While SILVER [1] exists in the literature, its described methodology and goals here are entirely incorrect. RTL-PSC [2] is a real framework but is partially misrepresented, and SCARF is entirely fictitious. The red-highlighted lines indicate the incorrect information. to-date, and verifiable guidance while avoiding the risks of hallucination and knowledge staleness? Despite the limitations of general-purpose LLMs, a conversational AI assistant remains a promising direction for addressing RQ1, provided it is designed with safeguards to address RQ2. Unlike static documentation, a conversational system can dynamically adapt to the evolving context of the engineer’s tasks, allowing them to ask specific questions, clarify uncertainties, and explore alternative strategies as their verification work progresses. By shifting the effort from manual information gathering to focused question–answer exchanges, such an assistant can reduce the cognitive burden and allow engineers to devote their expertise to high-level security reasoning rather than low-level information search. The effectiveness of this interaction paradigm has already been demonstrated in related hardware design and verification contexts: for example, ChatIoT [3], ORAssistant [4], EDA-Copilot [5], and ChatEDA [6] have shown how conversational agents can enhance engineers’ productivity, but none have been designed to address the unique challenges of hardware security verification. Notably, ChatEDA focuses on autonomous EDA script generation without domain-specific security knowledge, and ORAssistant provides RAG-based assistance for the OpenROAD flow without verification tool execution. In contrast, VeriChat uniquely combines security-specialized RAG with direct EDA tool integration, enabling both domain-grounded guidance and concrete verification actions within a single conversational workflow. More recently, LLM- and agentbased approaches have been explored for a variety of hardware security tasks [7], [8], [9], [10], including vulnerability analysis [11], [12], [13], security asset identification [14], formal security verification [15], [16], [17], and test plan generation [18]. However, these efforts are designed for specific verifica-
tion tasks rather than to provide comprehensive conversational support throughout the hardware security verification lifecycle. To address these challenges, we propose VeriChat 1 , a domain-specialized conversational assistant designed to support, rather than replace, existing hardware security verification practices. Through interactive dialogue, it provides security-oriented guidance across all stages of the verification flow, from clarifying security concepts and reasoning about threat relevance to exploring verification methodologies, troubleshooting issues, and devising countermeasures. Crucially, it addresses the shortcomings of general-purpose LLMs by grounding its responses in curated, up-to-date security knowledge and applying strict validation strategies to prevent hallucinations. The key contributions of VeriChat are as follows: 1) Multi-Agent Conversational Framework: We present VeriChat, the first multi-agent chatbot designed to provide security-focused guidance throughout the hardware security verification flow via interactive conversation. 2) Comprehensive Domain Knowledge Base: We construct a large, curated, topic-partitioned database of 28K+ hardware security verification research papers and resources, enabling precise and context-rich information retrieval. 3) EDA Tool Integration: We integrate open-source EDA tools (Icarus Verilog, Yosys, SymbiYosys) into the conversational workflow, enabling VeriChat to perform syntax checking, synthesis analysis, simulation, and formal verification on user-provided RTL designs. 4) Comprehensive Evaluation Framework: We design a multi-faceted evaluation framework that rigorously assesses VeriChat at both the component and system levels to ensure its reliability and trustworthiness. II. P ROPOSED M ETHODOLOGY VeriChat, shown in Figure 2, uses a sequential multi-agent pipeline to ensure accurate answers. The Query Understanding and Optimization Agent (QUOA) first interprets and refines the user’s query, clarifying its intent and tagging its topic. The Hybrid Retrieval Agent (HRA) then searches curated knowledge bases and the web, merging and reranking results into a unified evidence set. Finally, the Generation Agent (GA) produces a response strictly based on this verified context, reducing hallucinations and ensuring factual reliability. When the user’s query involves a design file and requires verification actions, VeriChat routes the request through an integrated tool pipeline (Section II-C) that performs syntax checking, synthesis analysis, simulation, and formal verification on the provided RTL. A. Query Understanding and Optimization Agent (QUOA) The primary objective of QUOA is to act as a sophisticated gatekeeper and interpreter, transforming raw, often ambiguous user input into a validated, optimized, and actionable data packet. As shown in Figure 3, its operation is divided into three sequential core functions. 1 The benchmark, case studies, and evaluation results used in this work are released at https://bit.ly/3QPBiWr.
Raw Query
Query Understanding & Optimization Optimized Agent (QUOA) Query
Hybrid Retrieval Agent (HRA)
Top Relevant Contexts
Generation Agent (GA)
Final Result
Topic Tags
Fig. 2: Overview of VeriChat Framework. 1) Intent Recognition: QUOA first performs a triage step that classifies the user’s input by analyzing both the raw query and the previous conversation turn. Using a few-shot prompted LLM as a multi-class classifier, it assigns the input to one of four classes: Invalid Question (to block harmful or outof-scope inputs), Feedback (purely conversational remarks), Valid Question With Follow-Up (new query referencing prior context), and Valid Question Without Follow-Up (standalone information-seeking query). A secondary intent classifier determines whether the query requires verification tool execution (e.g., “Run a security analysis on this design”), routing such requests to the pipeline described in Section II-C. This classification ensures system safety and tailors the subsequent workflow, allowing only non-Invalid Question inputs to proceed to the core logic.
1st turn: “original query”: “I have a Register-Transfer Level (RTL) implementation of an AES-128 core written in Verilog. I need to perform a comprehensive presilicon security assessment to identify vulnerabilities to power side-channel attacks. Specifically, I am looking for a detailed overview of established frameworks that can be used for power side-channel leakage assessment.” “dense retriever optimized query”: “What are the established frameworks for conducting a comprehensive presilicon security assessment of power side-channel leakage vulnerabilities in a RegisterTransfer Level (RTL) implementation of an AES-128 core written in Verilog?” “google search optimized query”: “Frameworks for power side-channel leakage assessment AES-128 RTL Verilog.” 2nd turn: “original query”: “Can you break down 2nd framework more?” “dense retriever optimized query”: “Provide a detailed breakdown of the SCAR framework, which utilizes Graph Neural Networks for pre-silicon power side-channel analysis in RTL designs, focusing on its methodology and application in detecting vulnerabilities in cryptographic hardware.” “google search optimized query”: “SCAR framework Graph Neural Networks RTL power side-channel analysis”
Fig. 4: Two-turn query optimization examples. semantically similar but topically irrelevant content (e.g., general cryptography or chip architecture). This distracts the LLM and degrades generation quality. To address this, VeriChat uses a multi-retriever architecture with multiple specialized, topically coherent vector databases (e.g., pre-silicon sidechannel leakage assessment, RTL-level security). As the final QUOA step, the optimized query is classified by an LLM into one or more relevant topic labels from a predefined set aligned with these knowledge bases. This multilabel classification produces topic tags that are passed with the optimized queries to the next agent.
Fig. 3: Overview of QUOA. 2) Conversational Query Optimization: In multi-turn dialogues, user queries often become progressively more ambiguous and context-dependent due to the use of conversational shortcuts such as coreference (referencing previously mentioned entities by pronouns) [19] and ellipsis (omitting information recoverable from context) [20]. While these strategies are natural in conversation, they produce queries that are underspecified or incomprehensible when taken out of context, making them unsuitable for retrieval systems that require explicit, standalone formulations. Even single-turn queries can suffer from vagueness, using informal or overly broad phrasing that fails to capture the user’s precise technical intent. To address this, we use an LLM guided by a domainaware prompt to produce two optimized outputs: one for dense retrieval and another for web search. The prompt handles both single- and multi-turn inputs and provides domainspecific rewriting rules and examples to ensure semantic clarity. For dense retrieval, the LLM generates a self-contained, terminology-aligned query suitable for semantic embedding (Figure 4); for web search, it produces a keyword-focused version aligned with technical phrasing. If the query is classified as Valid Question With Follow-Up, short-term memory is incorporated during the rewriting. 3) Semantic Domain Categorization: Naive RAG systems [21] often rely on a single large, heterogeneous knowledge base. When queries target narrow subdomains (e.g., “RTL-level side-channel leakage assessment for cryptographic cores”), searching such broad databases frequently retrieves
Fig. 5: Overview of Hybrid Retrieval Agent. B. Hybrid Retrieval Agent (HRA) The HRA compiles a comprehensive and reliable evidence set for the QUOA’s optimized queries using a hybrid strategy that combines high-precision retrieval from curated domainspecific knowledge bases with broad coverage from real-time web sources, ensuring the evidence is both accurate and upto-date (Figure 5). 1) Construction of Internal Knowledge Base: The HRA’s internal retrieval capability is built on a comprehensive, domain-specific knowledge base. We first defined a hierarchical taxonomy of 225 topics covering the hardware security verification domain. These topics serve a dual role: they structure the knowledge base and form the same set of classes used by the QUOA for Semantic Domain Categorization, ensuring a direct mapping between query classification and
targeted retrieval. Using these topics as search queries, a custom Python-based web scraper collected high-quality, opensource academic papers and articles. To maintain relevance, each document was manually validated against its assigned topic before inclusion. The raw corpus then underwent a rigorous curation pipeline involving de-duplication, cleaning, and preprocessing, yielding 28,221 curated articles totaling about 61 GB. These documents were grouped by topic, segmented into overlapping text chunks, and transformed into highdimensional vector embeddings using state-of-the-art models. The resulting embeddings were indexed into separate topicspecific vector databases, enabling the HRA’s efficient multiretriever search strategy. 2) Parallel Retrieval Mechanisms: Upon receiving the optimized query and its associated topic metadata from the QUOA, the HRA initiates multiple retrievers that operate in parallel: • Targeted Internal Retrievers: For each topic tag identified by the QUOA, a dedicated retriever is activated. This retriever targets only the specific vector database corresponding to its assigned topic (e.g., RISC-V, Spectre Vulnerability). Within this partitioned database, it employs vector search to identify the top-k document chunks whose vector embeddings have the highest cosine similarity to the embedding of the optimized query. This multi-retriever approach on partitioned data is the primary reason for the system’s speed and its ability to retrieve highly precise, domain-specific information. • Real-time External Retriever: To account for information not present in the static knowledge bases, a separate retriever queries the open web. The optimized query is submitted to a Google Search API, which returns results ranked by Google’s proprietary relevance algorithm incorporating querydocument similarity, page authority, and freshness. A Web Scraper processes the content of the top-ranked search results. This transiently collected web content is then chunked and converted on-the-fly into a temporary vector database. A vector search is then performed on this “in-memory” database to retrieve the most relevant chunks, ensuring that the system’s knowledge is not stale. In both cases, a chunk size of 1,000 characters with an overlap of 200 characters is used to preserve context across segments. The embedding model used during vectorization is text-embedding-ada-002 from OpenAI, and Facebook AI Similarity Search (FAISS) is employed with cosine similarity as the distance metric to rank all chunks in the target vector store and select the top k=20 document chunks per retriever, where selection is based strictly on descending similarity score. 3) Fusion and Re-ranking: Since relevance scores from different retrievers are not directly comparable, the HRA fuses them using a modified reciprocal rank fusion (RRF) [22] with source-specific weighting. The weighted RRF score for each document chunk c is computed as: WeightedRRFscore (c) =
N X i=1
wi ·
1 k + ranki (c)
(1)
where wi is the weight assigned to the i-th source retriever, ranki (c) is the chunk’s position in the i-th ranked list, and k is a damping constant (set to 60). For instance, the internal retriever corresponding to the primary topic tag and the external Google search retriever are assigned equal priority (wi = 0.4), while retrievers for additional internal topics receive proportionally smaller weights. The resulting comprehensive and re-ranked list of all retrieved chunks is stored in a session-specific memory. From this list, the top M chunks (in our case M = 100 at most) are selected and passed to the Generation Agent as the final context for generating the answer. C. Generation Agent (GA) The GA is the final stage of the framework, responsible for both verifying the retrieved evidence and composing the user-facing response. Its objective is to first act as a quality assurance gate and then to synthesize the verified information into a fluent, well-articulated, and faithfully grounded answer. a) Sufficiency Check: Before attempting to generate a response, the GA must first ensure that the provided context is sufficient. The agent first analyzes the top-k chunks of context from the HRA to determine if they are sufficient to answer the query. If the initial context is deemed insufficient, the agent accesses the session-specific short-term memory, which contains the complete, unfiltered list of all documents retrieved by the HRA, and attempts to find more relevant information from this larger pool. If the context remains insufficient, the agent is designed to decline elaboration, thereby avoiding a speculative answer gracefully. b) Prompt-Guided Faithful Generation: Once the contexts seem sufficient, the GA proceeds to synthesize the information by creating an internal “answer plan” that identifies key themes and establishes a logical flow. Using this plan, the agent generates the final response. A detailed prompt meticulously controls this generation process, enforcing exclusive grounding (using only verified context), dynamic formatting (choosing the best format for clarity), rigorous attribution (including complete, IEEE-style citations), and a final selfscrutiny step to ensure the output is fully aligned with all constraints. The GA acts as a self-verifying synthesizer, transforming a raw knowledge corpus into a trustworthy and effective final response. c) EDA Tool Integration: Prior conversational assistants for hardware design cannot interact with actual verification toolchains [6], [4]. To address this, VeriChat integrates opensource EDA tools into its workflow. When the QUOA classifies a query as an eda tool execution intent, the system triggers a four-stage pipeline: (1) Syntax Check using Icarus Verilog; (2) Synthesis Analysis using Yosys to extract structural metrics such as flip-flop counts that may reveal suspicious overhead; (3) Simulation using Icarus Verilog and VVP with an LLMgenerated targeted testbench; and (4) Formal Verification using SymbiYosys with bounded model checking via the Z3 SMT solver. At each stage, the LLM interprets raw tool output in the context of RAG-retrieved security knowledge, mapping
Fig. 6: Frontend interface of VeriChat displaying a response to the same query shown in Figure 1. VeriChat suggested RTL-PSC [2], SCAR [23], and other relevant frameworks with correct description. findings to CWE categories and known attack patterns. If any stage fails, the system automatically refines its generated artifacts and retries up to three times using the failure logs to guide correction. III. E XPERIMENTS AND R ESULTS To validate VeriChat, we conducted a two-stage evaluation: a component-level analysis of each agent and an end-to-end assessment against baselines. All datasets, results, and scripts are available at https://bit.ly/3QPBiWr. A. System Implementation VeriChat (Figure 6) is implemented as a distributed system with a React.js frontend and a Flask backend connected via a RESTful API. The frontend offers an intuitive chat interface with components tailored for hardware security workflows, including file upload for RTL designs and structured display of verification results. The backend orchestrates the multi-agent framework and supports integration of any API-accessible model. In this work, GPT-4o and GPT-5 serve as the primary generators. For EDA tool execution, the backend interfaces with Icarus Verilog (native) and Yosys/SymbiYosys (via WSL) through secure subprocess calls with timeout enforcement and output truncation. Average end-to-end latency for Q&A queries is roughly 15–25 seconds (dominated by LLM inference and multi-retriever search); tool execution adds 30–90 seconds depending on design complexity. Although the current implementation uses proprietary API-accessible models, the modular architecture lets any API-compatible LLM serve as a drop-in replacement, enabling future use of open-source alternatives. B. Case Studies To demonstrate VeriChat’s effectiveness across stages of hardware security verification, we conducted representative case studies comparing its responses with those of ChatGPT-5 (Table I). The queries span key activities, including security requirement specification, asset identification, threat modeling, verification planning, debugging, and result validation. In each
case, VeriChat provided more structured, domain-specific, and evidence-backed guidance, while ChatGPT-5 produced generic or unsourced responses. 1) EDA-Integrated Case Study: Hardware Trojan Detection in a Third-Party AES S-Box IP: We demonstrate VeriChat’s end-to-end workflow through a realistic scenario in which a user with no prior knowledge of a third-party AES S-Box IP progressively uncovers a hidden hardware Trojan through natural conversation. The design under test contains a raretrigger sequential Trojan (Listing 1): a three-state FSM activated by the byte sequence 0xDE, 0xAD, 0xBE, which arms a payload that leaks the secret key through status_led over eight clock cycles. Per the Trust-HUB taxonomy, this is a Type-I sequential rare-event trigger with a covert sidechannel payload, mapping to CWE-1245, with an activation probability of 2124 ≈ 6 × 10−8 per valid cycle. Listing 1: Trojan trigger: rare 3-byte sequence detector FSM. reg [1:0] trig_cnt; always @(posedge clk or negedge rst_n) begin if (!rst_n) trig_cnt <= 2’d0; else if (trig_cnt == 2’d3) trig_cnt <= 2’d0; // single-cycle pulse else if (valid_in) case (trig_cnt) 2’d0: trig_cnt <= (data_in==8’hDE) ? 2’d1 : 2’d0; 2’d1: trig_cnt <= (data_in==8’hAD) ? 2’d2 : 2’d0; 2’d2: trig_cnt <= (data_in==8’hBE) ? 2’d3 : 2’d0; default: trig_cnt <= 2’d0; endcase end
The user begins by asking: “I received an AES S-Box module from a third-party IP vendor. What kind of security threats should I be worried about?” The QUOA classifies this as a security q a intent, optimizes the query, and routes it to three topic-specific vector stores (hardware trojan, supply chain attacks, side channel attacks). The HRA fuses retrieved chunks via weighted RRF alongside real-time web results, and the GA synthesizes a response covering seven threat categories with actionable mitigations grounded in retrieved academic references: VeriChat Response (excerpt) Key security threats for a third-party AES S-Box IP: 1. Malicious modifications (hardware Trojans, backdoors): Hidden triggers in FSMs that enable bypass or degrade security; can evade functional verification. Mitigation: fuzzing/pen-testing of RTL, evolutionary search for rare trigger sequences. 2. Side-channel leakage (power, EM, timing): Data-dependent switching may leak key information. Mitigation: TVLA-style fixed-vs-random tests, constant-time enforcement. 3. Fault-injection susceptibility: Glitches enable differential fault analysis for key recovery. 4. DFT/scan-chain exposure. 5. Undocumented debug interfaces. 6. Non-compliance with FIPS-197. 7. Logic locking/obfuscation affecting availability. [13 academic references cited]
The user then follows up: “How would I detect if there is a Trojan in it? Can you actually check my design?” The QUOA detects this as a Valid Question With Follow-Up and incorporates short-term memory from the previous turn into the query optimization, enriching the rewritten query with the AES S-Box and Trojan context already established. The HRA retrieves additional content on detection methods: formal verification, coverage-guided fuzzing, structural analysis, side-
TABLE I: Qualitative comparison between VeriChat and ChatGPT-5 responses on representative security verification tasks (full transcripts in the results repository under CaseStudies/). Query
Related Task
Reasoning behind VeriChat’s superiority
$Security Requirement Specification$ I prepared this Security Requirement Document for the I2C protocol. Now I want to know if there is any room for improvement to include more comprehensive security requirements. Also, what are the current practices that I missed in my requirement document?
Security Requirements Specification
VeriChat delivered a superior analysis by structuring its feedback around your SRD’s specific design components, making it an immediately actionable engineering review. It critically fortified its recommendations with verifiable sources like the Accellera IPSA standard and academic papers, providing an expert authority that ChatGPT-5’s generic, unsourced checklist lacked. This blend of tailored feedback and sourced credibility makes its guidance more professional and trustworthy.
I want to identify security assets from the RTL designs. I am trying to develop a flow for this. How to do this?
Security Asset Identification
VeriChat provides a detailed, expert-level methodology that is specific and actionable. It incorporates advanced concepts like threat modeling, concolic testing, and information flow tracking, and cites relevant academic references, lending its answer authority and verifiability that GPT-5 lacks.
I have developed an asset list for an RTL design. Now, I want to connect these assets with relevant threat models. Unfortunately, the threat models are scattered around the literature, and they are slightly different in description from one another from one literature to another. Hence, I am finding it difficult to unify this with my asset list. Can you show me a structured way to connect assets with relevant threat models?
Threat Modeling
VeriChat’s response is superior as it details an actionable security verification methodology aligned with engineering workflows, while ChatGPT-5 offers a more conceptual framework. VeriChat’s prescriptive process directly addresses the core requirement by normalizing scattered threat data into a scalable meta-model based on RTL trust boundaries and lifecycle phases. This defines an end-to-end workflow that culminates in concrete engineering deliverables, including a unified threat register and verifiable SVA property packs. Ultimately, this provides a clear, auditable path to security sign-off, a critical advantage over the more generalized guidance.
Can you show me a suitable cost function required for gray box fuzzing on my RISC-V processor design so that it can identify all access control-related vulnerabilities?
Verification Planning
VeriChat offers precise, mathematically-defined cost functions for concrete RISC-V vulnerabilities (like PMP bypasses and illegal CSR access), based on a verifiable academic paper (SoCFuzzer). ChatGPT-5 gives a generic template using abstract heuristics, which is less reliable for real-world fuzzing.
My formal tool is not converging on a security assertion that I have written. What should I do now?
Debugging
VeriChat provides specialized, expert-level guidance with correct academic references. It explains that many security properties are hyperproperties, needing specific techniques. ChatGPT-5 offers generic advice that fails to address the unique challenges of security assertions.
I am performing a power side-channel leakage assessment for an AES-128 design without countermeasures. The gate count is around 20k. For collecting 1M power traces at the RTL level, the tool required 30 minutes at my end with 8 CPU cores. Is my runtime compatible with the current practices in RT-level power side-channel leakage assessment?
Result Validation
VeriChat is fact-based and strategic, citing real academic sources. The ChatGPT-5 response, on the other hand, is fundamentally unreliable as it fabricates its citations to create a false veneer of authority. Beyond just reliability, VeriChat provides a more expert consultation by questioning the user’s underlying methodology–correctly identifying that one million traces is excessive for noiseless RTL analysis. This elevates its guidance from a simple runtime validation to a strategic recommendation that aligns the user’s work with established, efficient practices.
channel monitoring, and golden model comparison. The GA synthesizes these strategies and, recognizing the user’s request for active analysis, informs the user that VeriChat can execute its integrated EDA tool pipeline on an uploaded design file. The user uploads the AES S-Box Verilog file and requests: “Run a full security analysis on this design.” This time, the QUOA classifies the query as an eda tool execution intent, triggering the four-stage pipeline described in Section II-C. In Stage 1, the syntax check reports zero errors and zero warnings; the interpretation notes that syntactic correctness alone does not preclude semantic vulnerabilities. In Stage 2, synthesis creates registers for data_out, valid_out, and internal pipeline signals, but also reveals 7 surplus flip-flops absent from a pure S-Box: trig_cnt[1:0] (2), trojan_armed (1), leak_bit_cnt[2:0] (3), and status_led (1), consistent with Trust-HUB Trojan benchmarks. Augmented by retrieved CWE knowledge, VeriChat flags this as consistent with CWE-1245 and identifies status_led as structurally decoupled from the core S-Box datapath. In Stage 3, the system generates a targeted testbench (Listing 2) that applies random functional vectors, a near-miss sequence, and then the exact trigger sequence with a known key. Simulation confirms Trojan activation at 535 ns with keydependent information leakage through status_led over 8 cycles, while normal S-Box functionality remains correct throughout Phases 1–2. In Stage 4, VeriChat generates SVA properties encoding security invariants and invokes bounded
Listing 2: LLM-generated targeted testbench (excerpt). // Normal operation: random vectors avoiding // accidental trigger bytes for (i = 0; i < 12; i = i + 1) begin r = $random; if (r==8’hDE||r==8’hAD||r==8’hBE) r = r ˆ 8’h5A; send_valid_byte(r); end // Near-miss: should NOT trigger Trojan send_valid_byte(8’hDE); send_valid_byte(8’hAD); send_valid_byte(8’hBF); // wrong third byte // Targeted Trojan trigger key_in = 8’hA6; // 0b1010_0110 send_valid_byte(8’hDE); send_valid_byte(8’hAD); send_valid_byte(8’hBE); // Monitor key leakage on status_led leaked = 8’h00; for (k = 0; k < 8; k = k + 1) begin @(posedge clk); leaked[k] = status_led; $display("[%0t] LEAK: bit%0d=%0b", $time, k, status_led); end
model checking with the Z3 SMT solver [24]. The confidentiality property (Listing 3) asserts that status_led must remain inactive at all times in a purely functional S-Box. The formal engine reports BMC failed! at step 1, producing a counterexample trace that mathematically proves the confidentiality violation. The auto-retry mechanism was exercised during this stage: the first attempt included overly broad assertions on internal Trojan signals (e.g., trojan_armed, trig_cnt) that trivially failed, and the second attempt retained an integrity check on data_out that also violated. The system automatically fed each failure log back to the LLM, which progressively refined the property set until the
third attempt isolated the targeted confidentiality assertion. This progressive revelation across the conversation (clean syntax, suspicious synthesis overhead, confirmed simulation, and formally proven violation) mirrors established multi-layered hardware security assessment methodologies. Listing 3: LLM-generated SVA confidentiality property. // Confidentiality: after the DE->AD->BE // sequence, status_led must stay inactive // for 8 cycles (no key leakage permitted) reg [3:0] fv_leak_window; always @(posedge clk) begin if (!rst_n) fv_leak_window <= 4’d0; else if (fv_mon_pulse) fv_leak_window <= 4’d8; else if (fv_leak_window != 4’d0) fv_leak_window <= fv_leak_window - 4’d1; end always @(posedge clk) begin if (rst_n && fv_past_valid) if (fv_leak_window != 4’d0) assert(status_led == 1’b0); end
C. Evaluations 1) Benchmark: To facilitate a comprehensive evaluation, we curated a new benchmark by engaging 25 experienced researchers in the domain of hardware security verification. Each expert contributed a set of six distinct queries on a topic of their choice, covering fact-checking, reasoning, scenariobased analysis, real-time queries, tool usage queries, and multiple-choice questions. This process yielded a diverse and challenging benchmark comprising 150 queries, which we subsequently used to assess the performance of VeriChat and other baseline models. 2) Component-Level Evaluation: a) Test 1 & 2: Context Recall and Precision Evaluation: We assessed VeriChat’s Hybrid Retrieval Agent (HRA) using DeepEval [25] on 150 question–answer samples, measuring both Context Recall (coverage of key facts) and Context Precision (ratio of relevant to retrieved content). An LLMas-a-judge (GPT-4.1) checked whether each gold-standard statement was supported by the retrieved context (recall) and whether each retrieved node was relevant to the query (precision). The metrics are defined as: # Attributable Statements Context Recall = (2) # Statements inn Expected Output 1 X Rk Context Precision = · rk (3) R k k=1
where R is the total number of relevant nodes, Rk is the number of relevant nodes up to position k, and rk is a binary indicator of relevance at position k. VeriChat achieved a Context Recall of 76.40% and a Context Precision of 89.85%, far surpassing the Naive RAG baseline (23.00% and 20.31% respectively) As shown in Figure 7a. This demonstrates that the QUOA-guided hybrid retrieval effectively gathers all essential information while minimizing irrelevant noise, unlike the baseline, which often misses key facts and includes off-topic content. b) Test 3 & 4: Answer Relevancy and Prompt Alignment Evaluation: We further evaluated VeriChat’s Generation Agent on 150 prompts using DeepEval [25], measuring Answer Relevancy (topicality of the response) and Prompt
Alignment (adherence to specified constraints). An LLM-as-ajudge (GPT-4.1) scored each output for how well it addressed the user’s query and followed the given instructions, including tone, persona, safety protocols, coherence, and formatting. The metrics are defined as: # of Relevant Statements × 100% (4) Answer Relevancy = Total # of Statements # of Instructions Followed × 100% Prompt Alignment = Total # of Instructions (5) VeriChat achieved 91.78% in Answer Relevancy and 88.3% in Prompt Alignment, indicating that its responses are not only factually grounded and on-topic but also reliably follow complex formatting, citation, and behavioral constraints. 3) End-to-End System Evaluation: a) Test 5: Faithfulness Test: This experiment asks: to what extent are a model’s responses grounded in verifiable facts? We posed 40 hardware security questions to all nine models, anonymized the responses, and had a human domain expert conduct a blind review, decomposing each into discrete factual claims verified against ground-truth sources as “correct” or “incorrect.” The Faithfulness score is the ratio of correct to total claims: # correct claims × 100% (6) Faithfulness Score = # total claims The results shown in Figure 7b reveal that VeriChat is substantially more faithful to facts than its competitors, achieving a score of 87.73%, which is over 23 percentage points higher than the next best model. This significant gap indicates the effectiveness of VeriChat’s multi-agent, retrieval-first architecture in mitigating hallucination. b) Test 6: Counterfactual Robustness Test: A trustworthy AI system must not only answer correctly but also reject user-introduced misinformation. To evaluate this, we designed an adversarial attack testing the Counterfactual Robustness of VeriChat against other leading proprietary chatbots. This involved 25 carefully crafted “hallucinated prompts” that confidently embed fabricated concepts or factually incorrect premises and ask the model to elaborate. A sample prompt is shown below: Sample Hallucinated Prompt: A major innovation in anti-tamper technology is Metamaterial Resonance Shielding (MRS). This involves creating a smart mesh for ICs built from programmable metamaterials. Any physical intrusion attempt (like micro-probing) changes the material’s resonant frequency, triggering a response to zeroize critical data. Can you explain?
The Counterfactual Robustness Rate can be expressed as: # Successful Refusals Robustness Rate = × 100% (7) # Test Cases VeriChat achieved a Robustness Rate of 92%, while monolithic proprietary models proved highly vulnerable (Figure 7c). This robustness arises from VeriChat’s strict reliance on retrieved evidence: when the HRA finds no support for a fabricated concept, the Generation Agent reports insufficient information instead of hallucinating. c) Test 7: User Preference Test: We conducted a user preference evaluation using the Elo rating system, which is
(a) Context Recall and Precision Test
(b) Faithfulness Test
(c) Counterfactual Robustness Test
(d) User Preference Test
Fig. 7: Performance of VeriChat in different evaluations. well-suited to evaluating chatbots through pairwise compar[8] D. Saha, S. Tarek, H. A. Shaikh, K. T. Hasan, P. S. Nalluri, M. A. Hasan, N. Alam, J. Zhou, S. K. Saha, M. Tehranipoor et al., “Sv-llm: An agentic isons [26], [27]. Following [26], we designed a “chatbot arena” approach for soc security verification using large language models,” arXiv preprint arXiv:2506.20415, 2025. of 1,000 matches in which two randomly selected models [9] D. Saha, S. Tarek, K. Yahyaei, S. K. Saha, J. Zhou, M. Tehranipoor, and from a pool of nine (VeriChat and eight leading proprietary F. Farahmandi, “Llm for soc security: A paradigm shift,” IEEe Access, vol. 12, 155 498–155 521, 2024. chatbots) responded to the same prompt. GPT-4o served as [10] pp. D. Saha, K. Yahyaei, S. K. Saha, M. Tehranipoor, and F. Farahmandi, “Empowering hardware security with llm: The development of a vulnerable hardware an impartial judge, selecting the best of the two anonymous database,” in 2024 IEEE International Symposium on Hardware Oriented responses based on factual precision and supporting evidence. Security and Trust (HOST). IEEE, 2024, pp. 233–243. [11] S. Tarek et al., “Socurellm: An llm-driven approach for large-scale system-onEach match outcome updated the systems’ Elo ratings using ′ RA = RA + K(SA − EA ),
(8)
where SA is the match score (1 win, 0.5 tie, 0 loss), K a constant, and EA = (1+10(RB −RA )/400 )−1 the expected score [26]. As shown in Figure 7d, VeriChat achieved the highest Elo score (2350), over 100 points above its nearest competitor (Grok-3) and nearly 200 above ChatGPT-4.1, ranking as the most preferred chatbot for reliable, evidence-based responses. IV. C ONCLUSION We presented VeriChat, a multi-agent conversational framework for hardware security verification that combines retrievalaugmented generation with integrated open-source EDA tools to deliver both domain-grounded guidance and concrete verification actions. In the hardware Trojan detection case study, it progresses from knowledge retrieval to tool-driven assessment (synthesis analysis, simulation, and formal verification) within a single multi-turn conversation. VeriChat achieves 87.73% faithfulness and 92% counterfactual robustness, outperforming all eight baselines. Future work includes semi-automated knowledge base expansion, provenance tracking, and richer explainability such as citation-level attribution and confidence calibration. R EFERENCES [1] D. Knichel, P. Sasdrich, and A. Moradi, “Silver–statistical independence and leakage verification,” in International Conference on the Theory and Application of Cryptology and Information Security. Springer, 2020, pp. 787–816. [2] M. He, J. Park, A. Nahiyan, A. Vassilev, Y. Jin, and M. Tehranipoor, “Rtl-psc: Automated power side-channel leakage assessment at register-transfer level,” in 2019 IEEE 37th VLSI Test Symposium (VTS), 2019, pp. 1–6. [3] Y. Dong, Y. L. Aung, S. Chattopadhyay, and J. Zhou, “Chatiot: Large language model-based security assistant for internet of things with retrieval-augmented generation,” arXiv preprint arXiv:2502.09896, 2025. [4] A. Kaintura, S. S. Luar, I. I. Almeida et al., “Orassistant: A custom rag-based conversational assistant for openroad,” arXiv preprint arXiv:2410.03845, 2024. [5] Z. Xiao, X. He, H. Wu, B. Yu, and Y. Guo, “Eda-copilot: A rag-powered intelligent assistant for eda tools,” ACM Transactions on Design Automation of Electronic Systems, 2025. [6] H. Wu, Z. He, X. Zhang, X. Yao, S. Zheng, H. Zheng, and B. Yu, “Chateda: A large language model powered autonomous agent for eda,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2024. [7] K. T. Hasan, M. A. Hasan, N. Alam, M. T. Islam, U. Das, and F. Farahmandi, “Ai-assisted hardware security verification: A survey and ai accelerator case study,” in 2026 IEEE 44th VLSI Test Symposium (VTS). IEEE, 2026, pp. 1–7.
chip security verification and policy generation,” in 2025 IEEE International Symposium on Hardware Oriented Security and Trust (HOST). IEEE, 2025, pp. 335–345. [12] J. Talukdar, A. Seth, S. Banerjee, F. Firouzi, and K. Chakrabarty, “Malls: Multiagent llms for synthetic hardware vulnerability generation and detection,” in 2025 IEEE 43rd International Conference on Computer Design (ICCD), 2025, pp. 782–789. [13] S. Ghimire, P. Mirfasihi, M. A. Chowdhury, V. Pugazhenthi, H. K. Dharavath, F. Firouzi, R. Yasaei, P. Satam, and S. Salehi, “Can agents secure hardware? evaluating agentic llm-driven obfuscation for ip protection,” in 2026 IEEE 44th VLSI Test Symposium (VTS), 2026, pp. 1–5. [14] M. A. Hasan, D. Saha, K. T. Hasan, N. Alam, A. Uddin, S. K. Saha, M. Tehranipoor, and F. Farahmandi, “Lasset: An llm-assisted security asset identification framework for system-on-chip (soc) verification,” in 2026 Design, Automation & Test in Europe Conference (DATE), 2026, pp. 1–7. [15] S. Tarek, D. Saha, K. T. Hasan, S. K. Saha, M. Tehranipoor, and F. Farahmandi, “Assertain: Automated security assertion generation using large language models,” arXiv preprint arXiv:2604.01583, 2026. [16] D. R. Ankireddy, S. Paria, A. Dasgupta, S. Ray, and S. Bhunia, “Lasso: Llmaided security property generation for assertion-based soc verification,” in 2025 ACM/IEEE 7th Symposium on Machine Learning for CAD (MLCAD). IEEE, 2025, pp. 1–10. [17] A. Menon, S. Miftah, S. Kundu, S. Kundu, A. Srivastava, A. Raha, G. Sonnenschien, S. Banerjee, D. Mathaikutty, and K. Basu, “Enhancing large language models for hardware verification: A novel systemverilog assertion dataset,” ACM Trans. Des. Autom. Electron. Syst., vol. 31, no. 6, Jun. 2026. [Online]. Available: https://doi.org/10.1145/3764934 [18] D. Saha, H. Al Shaikh, S. Tarek, and F. Farahmandi, “Threatlens: Llm-guided threat modeling and test plan generation for hardware security verification,” Cryptology ePrint Archive, 2025. [19] I. Kostric and K. Balog, “A surprisingly simple yet effective multi-query rewriting method for conversational passage retrieval,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, pp. 2271–2275. [20] H. Su, X. Shen, R. Zhang, F. Sun, P. Hu, C. Niu, and J. Zhou, “Improving multi-turn dialogue modelling with utterance rewriter,” arXiv preprint arXiv:1906.07004, 2019. [21] X. Ma, Y. Gong, P. He, H. Zhao, and N. Duan, “Query rewriting in retrievalaugmented large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 5303–5315. [22] G. V. Cormack, C. L. Clarke, and S. Buettcher, “Reciprocal rank fusion outperforms condorcet and individual rank learning methods,” in Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, 2009, pp. 758–759. [23] A. Srivastava, S. Das, N. Choudhury, R. Psiakis, P. H. Silva, D. Pal, and K. Basu, “Scar: Power side-channel analysis at rtl level,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2024. [24] L. de Moura and N. Bjørner, “Z3: An Efficient SMT Solver,” in Proceedings of the 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). Springer, 2008, pp. 337–340. [25] DeepEval, “Deepeval: The open-source llm evaluation framework,” https://de epeval.com, 2024, accessed: 2025-07-12. [26] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023. [27] W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez et al., “Chatbot arena: An open platform for evaluating llms by human preference,” in Forty-first International Conference on Machine Learning, 2024.