VeriBugBench: An Empirically Grounded Framework for Constructing Verilog RTL Debugging Benchmarks

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

1

VeriBugBench: An Empirically Grounded Framework for Constructing Verilog RTL Debugging Benchmarks arXiv:2609.18022v1 [cs.AR] 16 Sep 2026

Xiankai Meng, Kejian Feng, Xinlin Zhao, Zhuo Zhang, Yan Lei, Xiaoguang Mao, and Jiang Wu*

This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

Abstract—RTL source-level debugging research requires benchmark artifacts that provide faulty designs together with precise change locations, executable test stimuli, and reproducible configurations. Available Verilog resources usually provide only a subset of these elements. We present VeriBugBench, a framework for constructing Verilog RTL debugging benchmarks through empirically grounded fault construction, LLM-based testbench enhancement, and execution-based retention. The mutation library maps recurring, multi-granularity repair patterns observed in RTL bug-fix histories to 19 executable inverse operators. For each project, an LLM generates a design-specific stimulus phase from the clean DUT and original testbench; the phase is composed with the original testbench for candidate execution. Applying the framework to 45 open-source projects yields VeriBugBench-v1.0, with 2,608 executable single-fault instances whose effects are observable at design outputs. Across the 45 projects, the assembled testbenches increase mean project-level fault observability from 36.01% to 39.54% and improve line coverage and execution-trace diversity on average. VeriBugBench Xiankai Meng and Kejian Feng are with the School of Computer and Information Engineering, Institute for Artificial Intelligence, Shanghai Polytechnic University, Shanghai, China. Xinlin Zhao is an Independent Researcher, Sunnyvale, CA, USA. Zhuo Zhang is with the School of Computer Science and Engineering, Xi’an University of Technology, Xi’an, China. Yan Lei is with Chongqing University, Chongqing, China. Xiaoguang Mao is with the National University of Defense Technology, Changsha, China. Jiang Wu is with the Academy of Military Sciences, Beijing, China (corresponding author, e-mail: wujadeon@outlook.com). This work was supported by the National Natural Science Foundation of China under Grant 62504255 and the Natural Science Foundation of Shanghai under Grant 25ZR1402173.

provides versioned RTL variants, source-level ground truth, testbenches, and execution artifacts for evaluating RTL debugging methods. Index Terms—RTL debugging, Verilog, benchmark construction, mutation testing, testbench augmentation

I. I NTRODUCTION As hardware design complexity grows, verification effort grows even faster [1], and debugging remains a major verification bottleneck [2]. Verilog RTL executes under concurrent and temporal semantics [3]. Because fault evidence evolves during execution, source-level debugging must connect an observed failure to the responsible RTL construct and its time-aware execution context [4]. Automated support for this process includes dynamic and mutation-based fault localization [5], [6], neural and LLM-assisted RTL debugging [7], [8], [9], and automated RTL repair [10]. Although these methods address different stages of debugging, their development and evaluation depend on reproducible faulty designs, precise source-level ground truth, test stimuli, and executable project environments. A debugging benchmark must therefore provide more than isolated faulty files. Existing resources cover important parts of this need. CIRFix supplies executable RTL repair cases [10], Hardware BugBase curates reproducible FPGA bugs and debugging infrastructure [11], and RTLLM provides clean designs and test environments for RTL-generation evaluation [12]. Curated real-bug artifacts offer direct provenance but are

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

costly to expand, while mutation can construct faults at scale only if its operators and execution conditions are credible. Empirical RTL bug-fix histories can ground those operators [13], [14], but the resulting candidates must still be activated and exposed by a project testbench before they become useful debugging instances. We present VeriBugBench, a framework for constructing reproducible Verilog RTL source-level debugging benchmarks. It combines empirically grounded fault injection, project-level testbench enhancement, and execution-based instance retention. We use the framework to construct VeriBugBenchv1.0, whose instances package observable singlefault RTL variants with source-level ground truth and replayable execution artifacts. The present implementation supports the Verilog accepted by the Pyverilog front end [15] and the adopted simulation flow. The main contributions of this work are: Construction Methodology. The framework maps multi-granularity repair patterns from RTL bug-fix histories to executable mutation operators, applies them at structurally valid source locations, enhances each project’s testbench, and retains candidates whose behavioral effects are observable during execution. • Benchmark Artifact. We construct VeriBugBench-v1.0 from 45 open-source projects, obtaining 2,608 executable singlefault RTL instances whose effects are observable at design outputs. The artifact includes clean and mutated RTL, original and assembled testbenches, operator and source-location ground truth, and reproduction configurations and scripts. • Empirical Characterization. We derive 19 executable operators from repair-pattern evidence obtained from 1,298 retained bug-fixrelated commit records across 300 Verilog projects and evaluate the constructed benchmark in terms of distributional consistency, fault observability, and execution-artifact richness. •

2

II. BACKGROUND AND R ELATED W ORK Verilog RTL combines concurrency, clocked state, and event-driven scheduling [3]. Consequently, a defect in a sequential design may require a temporal activation sequence and several cycles of propagation before becoming visible at a monitored output [16], [17]. A useful benchmark must therefore pair faulty source and location ground truth with the activating testbench and execution environment. Existing methods consume different parts of this evidence: LLM-based source analyses operate directly on RTL code context [9], [8], whereas semiformal, dynamic, and spectrum-based methods use counterexample or simulation traces, coverage, passing/failing outcomes, or learned hit-statement classifications [18], [5], [4], [19], [17], [7]. LLM-assisted hardware verification has explored design-specific test-stimulus generation [20], [21]. In VeriBugBench, LLM-generated stimuli are incorporated at the project level to enhance the shared test environment used for candidate execution. Wit-HW generates additional passing witness tests from a bug-triggering case to refine spectrum-based hardware bug localization [22]. Existing executable resources serve complementary purposes. CIRFix provides 32 experttransplanted repair scenarios with faulty RTL, instrumented testbenches, correctness information, and reproducibility configurations [10]; Hardware BugBase provides a testbed of reproducible bugs and associated debugging tools for open-source FPGA designs [11]. BugGen uses a self-correcting multi-agent LLM pipeline to generate, insert, and validate functional RTL faults [23]. RTLLM, VerilogEval, and RTL-Repo benchmark RTL generation at task and repository scales, whereas MetRex targets post-synthesis metric reasoning [12], [24], [25], [26]. VeriBugBench differs by deriving executable mutation operators from empirical repairpattern evidence and constructing a multi-project population of output-observable single faults with explicit operator and location provenance, projectlevel testbenches, and replayable execution artifacts. This article substantially extends our DAC 2023 conference paper, MANTRA [13], which used

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

3

Empirical fault-model grounding RTL bug–fix histories

Hierarchical, multi-granularity repair patterns

Mapping and inversion into 19 guarded mutation operators

Structure-driven fault construction Clean target RTL project

Structural-context matching

Legal operator– location opportunities

Single-fault RTL candidates

LLM-generated stimulus phase

Sanitization and phase-based assembly with original testbench

Common execution Clean RTL and each candidate Same assembled testbench

Project-level testbench enhancement Original testbench

Candidateindependent project-level prompt

Executionbased filtering Successful run Output divergence

VeriBugBench-v1.0 Single-fault RTL variants Source-level ground truth Original/assembled testbenches Manifests and configurations

Fig. 1. Overall architecture of VeriBugBench. Empirical repair patterns ground guarded inverse mutation operators, while targetproject analysis discovers legal injection opportunities and clean-DUT-based enhancement produces a project-level assembled testbench. Candidate faults are retained only when they execute successfully and produce output divergence from the clean design under the common testbench.

manually specified, real-bug-grounded operators for mutation testing and mutant generation. VeriBugBench instead maps empirical multi-granularity repair patterns [14] to executable inverse operators and combines them with structure-driven candidate discovery, project-level testbench enhancement, and execution-based filtering to construct a reproducible multi-project RTL debugging benchmark. III. V ERI B UG B ENCH F RAMEWORK OVERVIEW We use the VeriBugBench framework to construct VeriBugBench-v1.0 through the three coordinated streams shown in Fig. 1. Empirical repair patterns define executable inverse mutation operators; structural analysis of each target project identifies valid operator–location pairs; and the clean DUT with its original testbench provides the context for project-level testbench enhancement. Candidate RTL variants are compiled and simulated under the assembled testbench, after which executable faults with observable output divergence are retained and packaged with source-level ground truth, testbenches, manifests, and execution configurations. IV. E MPIRICALLY G ROUNDED M UTATION F RAMEWORK VIA R EPAIR -PATTERN M APPING Historical patches may contain coordinated edits, whereas benchmark construction requires controlled

single-location injections with explicit ground truth. We therefore use multi-granularity repair patterns and their structural contexts to derive executable inverse operator schemas, which are instantiated only at context-valid locations in clean target designs and subsequently filtered by execution. A. Multi-Granularity Empirical Repair Patterns The empirical study of Meng et al. [14] retained 1,298 bug-fix-related commit records from 300 Verilog projects. Each modified RTL fragment is parsed into an AST, and its buggy and fixed revisions are compared to obtain a hierarchical edit script. The analysis projects these scripts onto two complementary views. The expression-level view records the nearest affected RTL construct, while the statement/root-level view records its enclosing statement, declaration, or structural context. The published taxonomy contains 8,756 expression-level and 12,786 statement/root-level AST edit-action records. We report the two projections separately because the same repair may contribute information at both levels. Let D = {(Pbug , Pf ix )} denote the paired buggy and fixed RTL fragments. For each pair, ASTDiff(Pbug , Pf ix ) produces insert, delete, update, and move actions organized by the AST hierarchy. Statement/root actions identify repairbearing statements, declarations, and structural con-

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

structs; expression-level actions identify the affected expression, access, assignment, event-control, or direction construct within that context. We normalize a repair pattern as r = ⟨a, n, c⟩,

(1)

where a ∈ {update, insert, delete, move}, n is the affected RTL construct, and c is its structurally relevant enclosing context. Depending on the repair, n may denote an expression operator, indexed or ranged access, assignment, event control, or declaration, while c records the surrounding assignment, conditional, case, process, or module context. This representation preserves the distinction between the same edit action applied to different RTL constructs or procedural contexts. B. Repair-Pattern-to-Operator Mapping We aggregate the multi-granularity repair records by their mutation-relevant RTL targets and contexts. Let Nm denote the empirical categories represented by the operator library and F the target-construct families; the mapping ϕ : Nm → F combines parser-level node labels with the same RTL role. Each operator draws on the AST granularity that defines its target, while cross-level associations supply enclosing contexts where required. Operator formation then proceeds in four steps. We examine the edit classes within each targetconstruct family, associate the affected construct with its statement/root context to form ⟨a, n, c⟩, and split a family when different targets or edit directions require distinct inverse transformations. Contexts that realize the same mutation concept are consolidated. An operator is retained when its inverse transformation and applicability conditions can be implemented as a controlled, single-location change in the Pyverilog AST. Empirical frequency establishes which construct families recur in the repair data. Operator selection further requires a stable RTL-level interpretation, an explicit inverse transformation, and implementable applicability conditions. A mutation operator µ is specified by an executable operator schema Sµ : Sµ = ⟨m, g, t⟩,

(2)

4

where m is an AST matcher, g is a structural guard, and t is a transformation rule. The matcher identifies a target-node family, the guard constrains the enclosing context, and the transformation defines the injected change. Thus, EdgeFlip applies only to an event-control edge in an appropriate always context, whereas NSubDelete applies to a nonblocking assignment in a statement position. Repair patterns and context-specific implementations may share one paper-level mutation-operator ID when they realize the same mutation concept; the executable schemas retain the guards needed by individual implementations. For example, ExprUpdate represents contextvalid updates of expression operators rather than a single literal token replacement, while NSubInsert aggregates block-, if-, and case-specific implementations. The direction of this mapping is important. The r corpus records a repair Pbug − → Pf ix , whereas µ=r −1

injection applies the inverse Pclean −−−−→ Pmut . For updates and moves, the observed transformation is reversed; insertion and deletion exchange roles.

C. Empirical Evidence and Operator Support Table I organizes the 19 operators into five target-construct families and reports the associated empirical AST categories. Expression, access, and assignment constructs account for 90.31% of the expression-level taxonomy. The five represented families together account for 8,307 of 8,756 records (94.87%); the remaining 449 records concern FunctionCall and PARAMETRIZATION, for which the current library defines no operator. These percentages describe the prevalence of the selected constructs rather than the distribution of generated mutants. Statement/root records may directly support statement-level targets or supply the enclosing assignment, conditional, case, process, declaration, and module contexts required by operator guards. Together, the five families yield the 19 operators listed in Table I. The same IDs are used in the implementation and dataset-level reporting; matcher, guard, and transformation details are encoded by their executable schemas.

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

5

TABLE I E MPIRICAL CONSTRUCT- FAMILY ORGANIZATION OF THE 19 MUTATION OPERATORS . O PERATOR NAMES DENOTE THE INVERSE INJECTION DIRECTION . Family

Empirical AST categories and records

Mutation operators in family

Expression

InfixExpression, PrefixExpression, TERNARY: 4,710 (53.79%)

ExprUpdate, ExprInsert, ExprDelete

Bit-vector & access

MemberAccessing, DOWNTO: 1,873 (21.39%)

PartselectUpdate, PartselectInsert, PartselectDelete; PointerUpdate, PointerInsert, PointerDelete

Assignment

Assignment: 1,325 (15.13%)

NSubUpdate, NSubInsert, NSubDelete, NSubMove; AssignN2B, AssignB2N

Timing

RISING, FALLING: 287 (3.28%)

EdgeFlip, EdgeInsert, EdgeDelete

Port

HdlDirection: 112 (1.28%)

IOFlip

a) Illustrative directional mapping trace.: Consider a repair that inserts a missing nonblocking assignment into an if branch. AST differencing records the inserted assignment together with its enclosing conditional context, forming an insertion pattern ⟨a, n, c⟩. Because benchmark construction proceeds from fixed to faulty code, this pattern supports the inverse operator NSubDelete. Its executable definition matches a nonblocking assignment in the corresponding statement context and removes it from the clean design. The empirical insertion and the injected deletion thus describe opposite directions of the same structural change. D. Structure-Driven Candidate Discovery and Instance Construction Given a target RTL project P and operator library M, the framework collects the context-valid opportunities O(P ) and applies a configurable candidate policy π under construction budget B: O(P ) = {(µ, f, v) | µ ∈ M, v ∈ f, mµ (v) ∧ gµ (v)},
C(P ) = π O(P ); B .

is configurable. Each instantiated candidate contains one injected transformation and retains its operator name, source file, AST-node identifier, and sourceline metadata. E. Execution-Based Filtering and Observed Distribution We retain Pmut only if it compiles and differs behaviorally from Pclean under the associated testbench T B. Compilation removes transformations that are invalid in the concrete design context, while behavioral filtering removes equivalent or unobserved mutants. The retained population therefore contains executable candidates whose effects are observable under the associated testbench. The resulting operator distribution emerges from the structure-driven opportunity space, operator applicability, the selected instantiation policy, and execution-based filtering. Section VII compares this retained distribution with the empirical bug-fix corpus.

(3)

Here, f is a source file and v is an AST node. Candidate locations follow from the target design’s structure rather than a predefined fault-location profile. The policy selects a subset of the discovered opportunities while preserving single-location injection and source-level ground truth. The choice of π

V. LLM-BASED T ESTBENCH E NHANCEMENT The utility of an executable RTL debugging benchmark depends on whether the associated test environment can expose behavioral differences between the clean and faulty designs. The projectprovided testbenches in our source repositories were written for different purposes and vary considerably in stimulus scope. Some exercise only nominal

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

transactions or repeatedly apply a narrow input pattern. Such tests may execute the design successfully while leaving many fault effects unobserved. VeriBugBench therefore augments, rather than replaces, each original testbench. The LLM receives the clean DUT and its existing testbench and generates a design-specific stimulus block. After extraction and sanitization, the generated phase is composed with the retained original phase to form one project-level testbench, which is subsequently applied uniformly to all candidate mutants of that project. A. Construction Objective and Assessment Metrics Let T B 0 denote the original project testbench and SLLM the generated stimulus phase. The assembled testbench is T B + = Compose(SLLM , T B 0 ),

(4)

which we call the assembled testbench. The desired outcome is not merely more stimulus code, but more observable behavioral variation under the same DUT interface and logging mechanism. We assess this outcome using candidate-fault observability and three execution-spectrum metrics. Candidate-fault observability records whether a mutant produces an output trace distinguishable from that of the clean design and is defined in Section VII-E. For the spectrum metrics, a simulation is represented by M ∈ {0, 1}L×T ,

(5)

where Ml,t = 1 if executable RTL line l is active at time step t. Line coverage is P |{l : t Ml,t > 0}| × 100%. (6) Cov = L Let rl be row l of M and A = {l : ∥rl ∥1 > 0} the active RTL lines. If m(r) is the number of active lines sharing trace r, the implemented uniqueness score is 1 X I[m(rl ) = 1] . (7) Sunique = |A| l∈A

It is therefore the fraction of covered lines whose temporal activation trace is not shared by another

6

covered line. Finally, let ct be column t of M and let P (s) be the empirical frequency of column state s. Temporal state entropy is X H=− P (s) log2 P (s). (8) s

Together, Cov, Sunique , and H describe structural reach, line-level trace separation, and temporal diversity. RQ4 evaluates these metrics on the assembled execution artifacts. B. Clean-DUT-Based Prompt Construction The evaluated configuration instantiates one prompt template from each project’s complete clean DUT RTL and original testbench, including its drivers, clock/reset conventions, and logging mechanism. The same project-level prompt combines this context with fixed, candidate-independent operator-family guidance covering expressions, bit selections, control-flow outcomes, event-control ordering, and assignment timing. It requests explicit boundary values, all-zero, all-one, alternatingbit and walking-one patterns, branch-distinguishing transitions, and back-to-back transactions. The generated phase must contain valid Verilog stimulus, must not introduce randomization or duplicate logging, must not call $finish, and must set llm_go_signal=0 on completion. Once sanitized and assembled, the resulting versioned testbench is directly replayable, and the complete prompt is retained with the framework implementation. C. Stimulus Generation, Sanitization, and Integration The LLM returns a Verilog stimulus block rather than a complete replacement testbench. The framework extracts Verilog from the response and applies a lightweight sanitizer that moves declarations to the beginning of each initial block. This step handles a common Verilog compatibility problem in generated code without changing the stimulus intent. The sanitized block is inserted as the first stimulus phase. A global llm_go_signal, initialized

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

Clean project context

Clean DUT

Original testbench

Candidateindependent prompt Boundary and explicit-vector values Control/state transitions Corner-case guidance No randomization or $finish

7

One generation per project

LLM generation Projectspecific Verilog stimulus phase

Extraction and sanitization Verilog code extraction Declaration placement

Phase-based assembly Generated phase first llm_go_signal handoff Watchdog termination

Project-level assembled testbench Reused across all project candidate faults

retained original phase

Fig. 2. Project-level workflow for LLM-based testbench enhancement. A candidate-independent prompt constructed from the clean DUT and original testbench generates a stimulus phase, which is sanitized and composed with the retained original phase. The assembled testbench is reused across the project’s candidate faults.

to one, pauses the main original stimulus while time-zero initialization and background processes, such as clock generation and trace logging, remain active. The generated phase sets the signal to zero on completion, releasing the retained original stimulus. The existing logger is reused so that both phases contribute to the same execution artifact. A watchdog is injected to terminate non-ending simulations and close an identified file handle where applicable. The handoff does not reinitialize the DUT; the original phase continues from the state left by the generated phase. Figure 2 summarizes the workflow from prompt construction to assembled-testbench execution; the resulting artifacts are assessed in RQ3 and RQ4.

and project test environments serve as clean construction inputs. The selected groups range from compact functional modules to larger open-source designs. Table II reports their project sizes and the resulting benchmark scale.

B. LLM-Based Testbench Preparation For each project, the workflow in Section V produces a versioned assembled testbench used during candidate execution and filtering. Section VII evaluates its behavioral effect through fault observability and execution-artifact metrics.

C. Mutation Opportunity Analysis VI. DATASET C ONSTRUCTION AND C OMPOSITION A. Project Sources and Dataset Scope To instantiate VeriBugBench-v1.0, we selected 45 open-source Verilog projects from three sources that differ in origin and source-code scale: • Native projects: 11 open-source projects collected for this study. They form the largerdesign portion of the selected corpus and include hierarchical control and datapath logic. • CirFix projects: 11 designs drawn from the executable CirFix repair artifact and previously exercised in an RTL repair workflow [10]. • RTLLM designs: 23 compact designs originating from an RTL-generation benchmark [12]. Their reference RTL

Applying the operator matchers and structural guards defined in Section IV to the 45 project ASTs yields 167,750 legal ⟨operator, file, node⟩ opportunities (Table III). Bit-vector and access operators account for 93,957 opportunities, followed by assignment (44,276), expression (24,693), port (2,640), and timing operators (2,184). These opportunities define the structural input population from which benchmark candidates are instantiated.

D. Candidate Construction and Execution-Based Filtering VeriBugBench-v1.0 samples b = 20 proposals uniformly with replacement from each nonempty

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

8

TABLE II S CALE AND C OMPOSITION OF V ERI B UG B ENCH - V 1.0, WITH THE O RIGINAL C IR F IX A RTIFACT AS A D ESCRIPTIVE R EFERENCE

Source

SLOC (NCSL)

# Proj Total

Mean Med.

Original CirFix Artifact (Descriptive Reference) CirFix (origin) 11 6,979 634.5 106.0

Faulty RTL Instances Min–Max Total Mean Med. Min–Max 14–3,305

61

5.5

6.0

2–9

VeriBugBench-v1.0 by Project Source CirFix (VeriBugBench) 11 6,979 634.5 106.0 14–3,305 RTLLM (VeriBugBench) 23 803 34.9 32.0 11–94 Native (VeriBugBench) 11 22,052 2004.7 846.0 162–12,033

810 893 905

73.6 38.8 82.3

60.0 33.0 60.0

14–186 7–87 21–160

11–12,033 2,608

58.0

47.0

7–186

Total (VeriBugBench-v1.0)

45

29,834

663.0

45.0

Note: The VeriBugBench-v1.0 total aggregates only the three constructed project-source groups; the original CirFix row is shown as a descriptive reference and is not included in the total. TABLE III D ISTRIBUTION OF LEGAL OPERATOR – LOCATION OPPORTUNITIES ACROSS PROJECT CATEGORIES . # Proj.

Assign.

Timing

Expr.

Port

Bit-Vec.

Total

11 11 23 45

6,355 37,141 780 44,276

212 1,824 148 2,184

3,645 20,580 468 24,693

419 2,113 108 2,640

13,543 78,852 1,562 93,957

24,174 140,510 3,066 167,750

Category CirFix Native RTLLM Total

project–operator opportunity set and collapses repeated ⟨operator, file, node⟩ identifiers. This produces 7,358 instantiated and deduplicated candidates. The saved candidate manifests define the reported snapshot and support exact replay. Each candidate is compiled and simulated with its project-level assembled testbench. Filtering retains candidates that complete execution and differ from the clean design in the project-provided CSV observation trace. For each clean–faulty pair, the comparator loads the traces with their recorded time field as the index, compares their common row prefix in recorded row order, and checks every logged observation field as a string. Thus, X/Z values are treated as observed values, and an instance is output-observable when at least one compared row differs in any logged field; trace portions beyond the common prefix do not establish observability. These checks yield 2,608 executable and outputobservable single-fault instances. The construction funnel is therefore 167,750 legal opportunities, 7,358 instantiated candidates,

and 2,608 retained instances. The final population reflects project structure, proposal sampling, deduplication, executability, and testbench-conditioned observability. E. Final Observed Mutation Distribution Table IV reports the retained counts of the 19 mutation operators defined in Table I. Table I organizes the operator library by construct family, whereas Table IV shows the population obtained after project-specific opportunity discovery, proposal sampling, deduplication, and execution-based filtering. Section VII compares this retained distribution with the empirical reference corpus. Dataset availability. VeriBugBench-v1.0 and its construction framework are publicly released at https://github.com/wndif/VeriBugBench under the license specified in the repository. The versioned artifact includes the framework source, selected project versions and configurations, clean and injected RTL with operator and source-location

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

9

TABLE IV O BSERVED DISTRIBUTION OF RETAINED MUTATION OPERATORS IN V ERI B UG B ENCH - V 1.0. C OUNTS AND PERCENTAGES ARE MEASURED AFTER CANDIDATE CONSTRUCTION AND EXECUTION - BASED FILTERING . BAR COLORS ENCODE NORMALIZED AST TRANSFORMATION CLASSES : UPDATE , INSERT , DELETE , AND MOVE . Operator family

Mutation operator

Expression

ExprUpdate ExprInsert ExprDelete

8.13% (212) 8.40% (219) 10.93% (285)

Bit-vector & access

PartselectUpdate PartselectInsert PartselectDelete PointerUpdate PointerInsert PointerDelete

2.30% (60) 0.31% (8) 2.34% (61) 5.21% (136) 9.16% (239) 3.41% (89)

Assignment

NSubUpdate NSubInsert NSubDelete NSubMove AssignN2B AssignB2N

8.28% (216) 10.74% (280) 10.01% (261) 8.13% (212) 2.22% (58) 0.27% (7)

Timing

EdgeFlip EdgeDelete EdgeInsert

4.83% (126) 1.73% (45) 1.61% (42)

Port

IOFlip

1.99% (52)

ground truth, original and assembled testbenches, candidate manifests, simulation and coverage metadata, and reproduction scripts. VII. E VALUATION OF V ERI B UG B ENCH A. Research Questions The evaluation aims to answer the following research questions: • RQ1 (Benchmark Scale and Composition). What are the project scale, source composition, and retained-instance scale of VeriBugBenchv1.0? • RQ2 (Empirical Distributional Consistency). To what extent does the retained benchmark preserve the broad edit-location and action tendencies observed in empirical RTL bug-fix data? • RQ3 (Fault Observability). Does LLMgenerated testbench augmentation increase the

Observed distribution (% & count)

fraction of candidate mutants whose effects are observable at the design outputs? • RQ4 (Diagnostic Artifact Richness). Does the assembled testbench enrich the recorded execution artifacts in terms of structural reach, singleton-line trace uniqueness, and temporal state diversity?

B. Experimental Setup All experiments were conducted on a Linux server. RTL simulation, mutation execution, and spectrum collection were performed using the unified automation pipeline described in Section VI. For each project, we evaluated the projectprovided Original testbench T B 0 and the Assembled testbench T B + , formed by composing the generated stimulus phase with T B 0 . All mutation operators, injected faults, and evaluation scripts

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

were held identical across configurations to ensure fair comparison. LLM generation configuration. We used GPT4 with temperature 0.2 and a fixed candidateindependent prompt strategy for each project. Generation failures were not automatically retried. Because model-service behavior and LLM generation are stochastic, the released prompts, assembled testbenches, manifests, and execution configurations define the evaluated benchmark snapshot; we do not claim that a fresh model call reproduces the same generated stimulus verbatim. C. Benchmark Scale and Composition (RQ1) Table II summarizes VeriBugBench-v1.0: 45 projects, 29,834 SLOC, and 2,608 executable, output-observable single-fault instances. Project sizes range from 11 to 12,033 SLOC, and retained instances range from 7 to 186 per project, with a median of 47. The original CirFix artifact is included only as a descriptive reference because it differs in project scope and construction procedure. Answer to RQ1: VeriBugBench-v1.0 combines 45 projects from three source groups and retains 2,608 executable, output-observable instances, providing a multi-source and multi-scale RTL debugging benchmark.

10

Distribution comparison. Figures 3 and 4 compare the expression- and statement-level distributions with those of the empirical RTL bug-fix corpus and CirFix. At the expression level, InfixExpression and MemberAccessing are the two leading categories in both the empirical corpus (43.43% and 17.78%) and VeriBugBench (43.64% and 26.46%). CirFix shows a different concentration, with TERNARY as its largest category (27.87%). At the statement level, VeriBugBench emphasizes HdlStmAssign (44.69%) and HdlStmIf (40.40%), consistent with their prominence in the empirical corpus (29.20% and 24.83%), but underrepresents HdlIdDef (1.43% versus 26.08%). It retains Case and Process contexts at 6.72% and 6.08%, compared with 7.84% and 9.78% empirically. Update remains the largest action class in both VeriBugBench (47.20% expression; 82.89% statement) and the empirical reference (51.90%; 64.23%). The agreement is therefore directional rather than exact, reflecting the benchmark’s focus on executable single-location transformations. Answer to RQ2: VeriBugBench preserves broad expression, statement-context, and edit-action tendencies in empirical RTL bug-fix data, with differences due to its executable single-location scope. This supports aggregate empirical grounding, not exact distributional reproduction.

D. Empirical Distributional Consistency (RQ2) We analyze the original–mutant RTL pairs of all 2,608 retained instances using the same AST differencing procedure and taxonomy as the empirical reference [14]. A pair may produce multiple AST edit actions; after projection and within-level deduplication, these actions form analysis records rather than additional benchmark instances. The expression-level view assigns each record to the nearest affected AST category, whereas the statement/root-level view assigns it to the enclosing statement or structural context. The analysis yields 4,074 expression-level and 6,446 statement/rootlevel records. Percentages are normalized independently within each level, and these record counts are distinct from the 19-operator instance counts in Table IV.

E. Mutation Analysis: Fault Observability (RQ3) Metric. For project p, let Mp denote the common pool of generated candidate mutants evaluated under both configurations and Kp,c the number whose behavior is distinguishable from the clean design under testbench configuration c ∈ {Original, Assembled}. The project-level mutation score is Kp,c × 100%. (9) MSp,c = Mp A mutant is killed if at least one compared trace row produces an observable output mismatch under the execution-based criterion in Section VI-D. Thus, mutation score operationalizes candidate-mutant observability in this construction study. Because the projects contain substantially different numbers of

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

Empirical RTL Bug-Fix Corpus (N = 8,756)

VeriBugBench-v1.0 (N = 4,074)

43.43%

Infix expr. 17.78%

Member access

CirFix (N = 61) 43.64%

Infix expr.

5.20%

Assignment

5.50%

Member access

TERNARY

5.16%

DOWNTO

4.91%

Prefix expr.

13.29%

0

20

10.75%

Others

40

0

18.03%

Assignment

Prefix expr.

Others

21.31%

Infix expr.

8.74%

Prefix expr.

27.87%

TERNARY

26.46%

Member access

15.13%

Assignment

11

14.75%

9.84%

8.20%

Others

20

40

0

20

Share of Within-Level AST Edit-Action Records (%) Delete Insert Move Update

Fig. 3. Expression-level distribution of AST edit-action records in the empirical RTL bug-fix corpus, VeriBugBench-v1.0, and CirFix. Percentages are normalized independently within each source (N = 8,756, 4,074, and 61 records, respectively); colors denote AST edit-action classes. Empirical RTL Bug-Fix Corpus (N = 12,786)

VeriBugBench-v1.0 (N = 6,446)

29.20%

Assign

CirFix (N = 125) 44.69%

Assign

IdDef

26.08%

If

If

24.83%

Case

6.72%

Process

12.80%

Process

6.08%

Case

11.20%

9.78%

Process

7.84%

Case

2.28%

Others

0

20

40.40%

51.20%

Assign 18.40%

If

IdDef

1.43%

For

4.00%

Others

0.68%

Others

2.40%

0

20

40

0

25

50

Share of Within-Level AST Edit-Action Records (%) Delete Insert Move Update

Fig. 4. Statement-level distribution of AST edit-action records in the empirical RTL bug-fix corpus, VeriBugBench-v1.0, and CirFix. Percentages are normalized independently within each source (N = 12,786, 6,446, and 125 records, respectively); colors denote AST edit-action classes.

candidate mutants, we use the unweighted macroaverage MSc =

1 X MSp,c |P| p∈P

(10)

as the primary cross-project measure, so that each project contributes equally. The killed-mutant columns in Table V are sums over projects, whereas the mutation-score columns report MSc . For com-

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

12

TABLE V C ANDIDATE - MUTANT OBSERVABILITY UNDER THE ORIGINAL AND ASSEMBLED TESTBENCHES . O BSERVABLE - MUTANT COUNTS ARE SUMMED WITHIN EACH SOURCE , WHEREAS MS VALUES ARE UNWEIGHTED MEANS OF PROJECT- LEVEL MUTATION SCORES . Source

#Proj

CirFix Native RTLLM Total

11 11 23 45

Total Mutants 1,911 3,466 1,981 7,358

Observable Orig. Assem. 708 810 707 905 841 893 2,256 2,608

pleteness,P we also P report the mutant-level microaverage, K / p,c p p Mp , in the accompanying text. Candidates observable under the assembled testbenches constitute the retained VeriBugBench-v1.0 population. Results. Table V summarizes the results. The assembled testbenches make 352 more candidates observable in aggregate. The mean project-level mutation score increases from 36.01% to 39.54% (+3.53 percentage points), while the corresponding mutant-level micro-average increases from 30.66% to 35.44% (+4.78 percentage points). The largest macro-average gain occurs in the Native subset (+6.15 percentage points), whereas RTLLM shows a smaller average gain (+1.77 percentage points). These subset results describe aggregate tendencies; the augmentation is not guaranteed to improve every project. Illustrative project. The original RAM testbench repeats the same address-zero write/read sequence 100 times, whereas the assembled testbench adds reset handling, address sweeps, walking-one data, simultaneous read/write operations, and disabledread behavior. The number of output-observable candidates increases from 22 of 86 (25.58%) to 31 of 86 (36.05%). This project illustrates how designspecific stimulus broadens the exercised behavior behind the aggregate improvement. Answer to RQ3: From the common pool of 7,358 candidates, the assembled testbenches yield 2,608 output-observable instances, compared with 2,256 under the original testbenches. LLM-generated augmentation therefore increases candidate-mutant observability on average, although the gain varies across projects and source subsets.

Mean Project-Level MS (%) Orig. Assem. 38.00 42.58 21.34 27.49 42.08 43.84 36.01 39.54

Improv. (pp) 4.58 6.15 1.77 3.53

F. Diagnostic Artifact Richness (RQ4) Evaluation protocol. RQ4 computes line coverage (Cov), singleton-line trace uniqueness (Sunique ), and temporal state entropy (H), defined in Section V-A, from the original and assembled execution matrices. Table VI reports unweighted means of project-level values and paired differences; the Total row averages all 45 projects. Results and answer to RQ4. Table VI shows that the assembled testbenches increase mean line coverage from 73.57% to 76.77% (+3.20 percentage points), Sunique from 0.1886 to 0.2445 (+0.0559), and temporal state entropy from 1.57 to 1.94 (+0.37). The latter two metrics can improve even when reachability changes little: CirFix gains only 0.99 coverage points but increases Sunique by 0.0294 and H by 0.32, indicating richer execution spectra beyond aggregate line coverage. Gains vary across projects and sources; Native has the largest aggregate coverage and entropy increases, while RTLLM has the largest uniqueness increase. VIII. D ISCUSSION AND T HREATS TO VALIDITY A. Discussion VeriBugBench treats benchmark construction as an inspectable pipeline linking empirical repair patterns, executable mutation operators, structural opportunities, and execution-based retention. Mapping records, candidate manifests, and filtering metadata make this provenance auditable. The benchmark packages paired clean and faulty RTL, operator/location ground truth, original and assembled testbenches, and execution configurations for RTL source-level debugging research.

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

13

TABLE VI E XECUTION - ARTIFACT METRICS UNDER THE ORIGINAL AND ASSEMBLED TESTBENCHES . VALUES AND DELTAS ARE UNWEIGHTED MEANS OF PROJECT- LEVEL MEASUREMENTS . Source

#Proj

CirFix Native RTLLM Total

11 11 23 45

Line Cov. (%) Orig. Assem. ∆ 81.48 82.47 0.99 69.88 77.38 7.50 71.56 73.76 2.20 73.57 76.77 3.20

Orig. 0.3430 0.0198 0.1954 0.1886

B. Threats to Validity 1) Internal Validity: Internal validity depends on how empirical repair patterns are mapped to executable operators and on the sampling, deduplication, tooling, and filtering configurations. We expose the mappings, operator definitions, candidate manifests, and execution settings for inspection. The empirical corpus and benchmark projects overlap slightly. The reported distributional consistency therefore characterizes this end-to-end construction process. Evaluation against an independent RTL bug-fix corpus remains future work. 2) Construct Validity: Construct validity is bounded by the adopted measures: aggregate distributional consistency, candidate-fault observability, and the structural and temporal properties of execution artifacts. These measures characterize benchmark construction and artifact quality rather than the performance of a particular debugging technique. 3) External Validity: VeriBugBench-v1.0 is constructed from 45 open-source Verilog projects spanning multiple sizes and design categories. Its external validity is therefore bounded by this project population, the use of single injected source-level faults, and the Verilog subset accepted by the Pyverilog-based toolchain. Extending the framework to broader industrial designs, multi-edit or implementation-dependent faults, and SystemVerilog remains future work. 4) Reproducibility: VeriBugBench-v1.0 supports reproducibility through versioned RTL variants, original and assembled testbenches, manifests, metadata, execution configurations and scripts, and documented tool versions. Candidate sampling and LLM generation are stochastic, so fresh runs may

Sunique Assem. 0.3725 0.0372 0.2824 0.2445

∆ 0.0294 0.0175 0.0870 0.0559

Orig. 1.88 1.86 1.28 1.57

Entropy H Assem. 2.20 2.49 1.56 1.94

∆ 0.32 0.62 0.28 0.37

produce different candidates or stimuli. The released manifests and assembled testbenches therefore define the benchmark instance evaluated in this paper. IX. C ONCLUSION This paper presented VeriBugBench, a framework for constructing reproducible Verilog RTL source-level debugging benchmarks, and VeriBugBench-v1.0, a benchmark constructed from 45 open-source projects. The framework maps empirical repair patterns to 19 executable mutation operators, discovers structurally valid mutation opportunities, enhances project-level testbenches with an LLM, and retains faults through compilation, execution, and outputdivergence checks. VeriBugBench-v1.0 contains 2,608 executable, output-observable single-fault instances. The evaluation shows aggregate consistency with empirical edit-location and repair-action distributions. On average, assembled testbenches improve candidate-fault observability and all three execution-artifact metrics. Together, the released benchmark and execution artifacts support reproducible RTL source-level debugging research. ACKNOWLEDGMENT Generative-AI Disclosure: OpenAI Codex was used solely for language editing and iterative refinement of Figs. 1 and 2. R EFERENCES [1] N. Wu, Y. Li, H. Yang, H. Chen, S. Dai, C. Hao, C. Yu, and Y. Xie, “Survey of machine learning for software-assisted hardware design verification: Past, present, and prospect,” ACM Trans. Design Autom. Electr. Syst., vol. 29, no. 4, pp. 1–42, 2024.

VERIBUGBENCH: VERILOG RTL DEBUGGING BENCHMARK CONSTRUCTION

[2] H. Foster, “2024 Wilson research group IC/ASIC functional verification trend report,” Siemens Digital Industries Software, White Paper, Feb. 2025. [3] Q. Chen, N. Zhang, J. Wang, T. Tan, C. Xu, X. Ma, and Y. Li, “The essence of Verilog: A tractable and tested operational semantics for Verilog,” Proc. ACM Program. Lang., vol. 7, no. OOPSLA2, pp. 234–263, 2023. [4] J. Wu, Z. Zhang, D. Yang, X. Meng, J. He, X. Mao, and Y. Lei, “Fault localization for hardware design code with time-aware program spectrum,” in Proc. 40th IEEE Int. Conf. Computer Design (ICCD). IEEE, 2022, pp. 537– 544. [5] J. Wu, Z. Zhang, J. Xu, J. He, X. Mao, X. Meng, and P. Li, “Detraque: Dynamic execution tracing techniques for automatic fault localization of hardware design code,” PLOS ONE, vol. 17, no. 9, p. e0274515, 2022. [6] J. Wu, Z. Zhang, D. Yang, J. Xu, J. He, and X. Mao, “Knowledge-augmented mutation-based bug localization for hardware design code,” ACM Trans. Archit. Code Optim., vol. 21, no. 3, pp. 45:1–45:26, 2024. [7] M. Heidari and B. Alizadeh, “Localizing multiple bugs in RTL designs by classifying hit-statements using neural networks,” IEEE Trans. Computers, vol. 74, no. 5, pp. 1786–1799, 2025. [8] K. Xu, J. Sun, Y. Hu, X. Fang, W. Shan, X. Wang, and Z. Jiang, “MEIC: re-thinking RTL debug automation using LLMs,” in Proc. 43rd IEEE/ACM Int. Conf. ComputerAided Design (ICCAD). ACM, 2024, pp. 100:1–100:9. [9] B. Yao, N. Wang, J. Zhou, X. Wang, H. Gao, Z. Jiang, and N. Guan, “Location is key: Leveraging LLM for functional bug localization in Verilog design,” in Proc. 62nd ACM/IEEE Design Automation Conf. (DAC). IEEE, 2025, pp. 1–7. [10] H. Ahmad, Y. Huang, and W. Weimer, “CirFix: automatically repairing defects in hardware design code,” in Proc. 27th ACM Int. Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM, 2022, pp. 990–1003. [11] J. Ma, G. Zuo, K. Loughlin, H. Zhang, A. Quinn, and B. Kasikci, “Debugging in the brave new world of reconfigurable hardware,” in Proc. 27th ACM Int. Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM, 2022, pp. 946–962. [12] Y. Lu, S. Liu, Q. Zhang, and Z. Xie, “RTLLM: an opensource benchmark for design RTL generation with large language model,” in Proc. 29th Asia and South Pacific Design Automation Conf. (ASP-DAC). IEEE, 2024, pp. 722–727. [13] J. Wu, Y. Lei, Z. Zhang, X. Meng, D. Yang, P. Li, J. He, and X. Mao, “Mantra: Mutation testing of hardware design code based on real bugs,” in Proc. 60th ACM/IEEE Design Automation Conf. (DAC). IEEE, 2023, pp. 1–6. [14] X. Meng, X. Ji, G. Zhang, J. He, D. Yang, F. Chen, J. Wang, C. Yu, X. Zhao, and J. Wu, “RTL design flaws revisited: a data-driven study of systematic bug patterns in Verilog code,” J. Supercomput., vol. 81, no. 14, p. 1321, 2025. [15] S. Takamaeda-Yamazaki, “Pyverilog: A python-based hardware design processing toolkit for Verilog HDL,” in Applied Reconfigurable Computing, ser. Lecture Notes in

14

Computer Science, vol. 9040. Springer International Publishing, 2015, pp. 451–460. [16] J. Tang, J. Mu, T. Ma, Z. Chao, J. Ye, and H. Li, “Pecker: Bug localization framework for sequential designs via causal chain reconstruction,” 2026. [Online]. Available: https://arxiv.org/abs/2603.02583 [17] J. Wu, Z. Zhang, D. Yang, J. Xu, J. He, and X. Mao, “Time-aware spectrum-based bug localization for hardware design code with data purification,” ACM Trans. Archit. Code Optim., vol. 21, no. 3, pp. 64:1–64:25, 2024. [18] B. Kumar, V. S. Vineesh, P. Nemade, and M. Fujita, “Aries: A semiformal technique for fine-grained bug localization in hardware designs,” IEEE Trans. Comput. Aided Des. Integr. Circuits Syst., vol. 41, no. 12, pp. 5709–5721, 2022. [19] J. A. Jones and M. J. Harrold, “Empirical evaluation of the tarantula automatic fault-localization technique,” in Proc. 20th IEEE/ACM Int. Conf. Automated Software Engineering (ASE). ACM, 2005, pp. 273–282. [20] R. Ma, Y. Yang, Z. Liu, J. Zhang, M. Li, J. Huang, and G. Luo, “VerilogReader: LLM-aided hardware test generation,” in 2024 IEEE LLM Aided Design Workshop (LAD). IEEE, 2024, pp. 1–5. [21] Z. Zhang, B. Szekely, P. Gimenes, G. Chadwick, H. McNally, J. Cheng, R. Mullins, and Y. Zhao, “LLM4DV: using large language models for hardware test stimuli generation,” in Proc. 33rd IEEE Int. Symp. Field-Programmable Custom Computing Machines (FCCM). IEEE, 2025, pp. 133–137. [22] R. Ma, D. Kuang, Z. Liu, J. Zhang, P. Fan, and G. Luo, “Wit-HW: Bug localization in hardware design code via witness test case generation,” in Proc. IEEE/ACM Int. Conf. Computer-Aided Design (ICCAD). IEEE, 2025, pp. 1–9. [23] S. Jasper, M. Luu, E. Pan, A. Tyagi, M. Quinn, J. Hu, and D. K. Houngninou, “BugGen: A self-correcting multiagent LLM pipeline for realistic RTL bug synthesis,” in Proc. 7th ACM/IEEE Symp. Machine Learning for CAD (MLCAD). IEEE, 2025, pp. 1–9. [24] M. Liu, N. R. Pinckney, B. Khailany, and H. Ren, “Invited paper: VerilogEval: Evaluating large language models for Verilog code generation,” in Proc. IEEE/ACM Int. Conf. Computer-Aided Design (ICCAD). IEEE, 2023, pp. 1–8. [25] A. Allam and M. Shalan, “RTL-Repo: A benchmark for evaluating LLMs on large-scale RTL design projects,” in 2024 IEEE LLM Aided Design Workshop, LAD. IEEE, 2024, pp. 1–5. [26] M. Abdelatty, J. Ma, and S. Reda, “MetRex: A benchmark for Verilog code metric reasoning using LLMs,” in Proc. 30th Asia and South Pacific Design Automation Conf. (ASP-DAC). ACM, 2025, pp. 995–1001.

Related documents

• VeriBugBench: An Empirically Grounded Framework for Constructing Verilog RTL Debugging Benchmarks#1035618
• ParityFuzz: Finding Inconsistencies across Solidity Compilers via Fine-Grained Mutation and Differential Analysis#175346
• AI-Powered Debugging: Exploring Machine Learning Techniques for Identifying and Resolving Software Errors#70785
• Demystifying the Silence of Correctness Bugs in PyTorch Compiler#6036
• CCL-Bench 1.0: A Trace-Based Benchmark for LLM Infrastructure#168235