IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
1
From Mirage to Grounding: Towards Reliable Multimodal Circuit-to-Verilog Code Generation
arXiv:2604.27969v1 [cs.SE] 30 Apr 2026
Guang Yang, Xing Hu∗ , Xiang Chen, and Xin Xia
Abstract—Multimodal large language models (MLLMs) are increasingly used to translate visual artifacts into code, from UI mockups into HTML to scientific plots into Python scripts. A circuit diagram can be viewed as a visual domain-specific language for hardware: it encodes timing, topology, and bitlevel semantics that are invisible to casual inspection yet safetycritical once fabricated in silicon. Translating such diagrams into register-transfer-level (RTL) code therefore represents an extreme reliability test for vision-to-code generation. We reveal a phenomenon we call Mirage: replacing a circuit diagram with a blank image leaves Pass@k unchanged or even higher, because models bypass the visual input and instead exploit identifier semantics in the module_header to retrieve canonical RTL templates. This constitutes a new, highly covert class of defect in AI-assisted code generation that directly undermines MLLMs’ trustworthiness. To quantify the effect, we construct C2VE VAL and evaluate eight MLLMs under a paired Normal/Anony protocol in which Anony mode anonymizes all identifiers in both the diagram and the module header; Anony-mode scores drop sharply across all models, confirming that high Normal-mode accuracy is largely a Mirage. We then propose VeriGround (4B), trained with identifier anonymization, refusal augmentation, and D-ORPO (Decision-Focused ORPO) preference alignment that up-weights pivotal generate-or-refuse tokens. VeriGround achieves Functional Pass@1 of 46.11%/42.51% (Normal/Anony) with a False Refusal Rate of only 1.20%/0.00%, while maintaining ≥92% Refusal Rate on blank images. With only 4B parameters, VeriGround performs on par with GPT-5.4 under Normal and significantly outperforms all baselines under Anony, confirming genuine visual grounding. The evaluation methodology and training recipe generalize beyond hardware: C2VE VAL’s paired protocol can benchmark any vision-to-code pipeline, and VeriGround’s anonymization-refusal-alignment triad offers a transferable solution for balancing hallucination and over-refusal in AI code generation. Index Terms—multimodal large language models, vision-tocode generation, visual grounding, software trustworthiness, Verilog.
I. I NTRODUCTION Image-to-code generation has become a central paradigm in AI-assisted coding. From UI mockups rewritten as HTML [1]– [6], to scientific plots reverse-engineered into reproducible Python scripts [7]–[9], multimodal large language models (MLLMs) are shifting the front-end of software creation from Corresponding author: Xing Hu. Guang Yang is with the State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China, and also with the Hangzhou HighTech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China. Xing Hu and Xin Xia are with the State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China. Xiang Chen is with the School of Artificial Intelligence and Computer Science, Nantong University, Nantong, China. E-mail: [email protected], [email protected], [email protected], [email protected]. Manuscript received April 19, 2020; revised August xx, xxxx.
Fig. 1. Motivating example 1 of the Mirage phenomenon. The model generates correct code regardless of whether the input contains the real circuit diagram or a blank image.
purely textual specifications toward richer visual artifacts. In each of these tasks, the visual input can be viewed as a visual domain-specific language (visual DSL): a UI mockup specifies layout and interaction semantics, a chart encodes data and rendering logic, and each must be faithfully translated into executable code. Recently, this paradigm has been extended to hardware: MLLMs are used to translate circuit diagrams into synthesizable register-transfer-level (RTL) code, a task we call circuit-to-Verilog code generation [10], [11]. A circuit diagram is arguably the demanding visual DSL, because it encodes timing, topology, and bit-level semantics that are invisible to casual inspection yet safety-critical once fabricated in silicon. While unreliable vision-to-code translation is a general concern across all visual DSLs, the consequences in the hardware domain are uniquely severe [12], [13]. RTL sits at the very beginning of the silicon design flow and feeds downstream into synthesis, place-and-route, and fabrication; a single misread schematic can propagate silently through the entire toolchain and materialize as a silicon-level defect whose cost is orders of magnitude higher than a rendering or plotting error. More broadly, if an AI code generator appears to “understand” a visual specification while actually bypassing it, the resulting code carries a covert correctness risk that conventional testing may not catch. This directly threatens MLLMs’ trustworthiness, a core concern of the software engineering community. Circuit-to-Verilog generation therefore serves as a rigorous proving ground: any reliability flaw exposed here is likely
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
2
Fig. 2. Motivating example 2 of the Mirage phenomenon. When the real circuit diagram is provided, the model generates incorrect Verilog that fails the testbench. When the diagram is replaced by a blank image while the module_header is retained, the model instead produces correct code.
to manifest, in milder but equally insidious forms, in other vision-to-code pipelines. Although recent studies have demonstrated the feasibility of using MLLMs to read circuit diagrams and generate Verilog code, a fundamental question remains largely unexplored: do existing MLLMs truly read circuit diagrams, or merely exploit textual shortcuts? Our motivating examples make this concern concrete. In both Fig. 1 and Fig. 2 the model under test is Opus 4.6, one of the current frontier code-generation models. Following standard practice, the module_header (module name, ports, and parameters) is provided as part of the prompt so that the generated interface matches the testbench and Pass@k can be computed without name-mismatch failures. Yet this seemingly innocuous input turns out to be a powerful textual shortcut. In Fig. 1, the model produces identical correct code for a half-adder whether the input contains the real circuit or a blank image: the module name TopModule is semantically vacuous, but the output ports sum and cout already reveal the target function, making the diagram redundant. Fig. 2 is even more striking: for fsm_3state, the model fails with the real diagram yet succeeds with a blank image, suggesting that the visual input can actively interfere with generation. We term this failure mode the Mirage phenomenon [14]: high benchmark scores mask the fact that models rely on textual priors in the module_header rather than genuinely grounding in the visual circuit topology. Mirage constitutes a new, highly covert class of AI code-generation defect: the output may compile and even pass certain tests, yet it was
never derived from the visual specification, leaving latent errors that surface only in untested scenarios. A natural way to test this hypothesis is to anonymize the identifiers in both the module_header and the circuit diagram, thereby stripping the semantic cues that enable such shortcuts. Following this idea, we construct C2VE VAL (Circuit-toVerilog Evaluation), a benchmark that samples problems from established Verilog code-generation benchmarks and renders each reference solution into a circuit diagram via NETLISTSVG 1 , ensuring exact image-code correspondence. C2VE VAL has two variants: Normal, which retains the original identifiers in both the diagram and the module header, and Anony (anonymized), which replaces all semantically loaded identifiers with positional placeholders and re-renders the diagram accordingly, preserving circuit topology while removing semantic cues. We evaluate eight MLLMs, from 4B opensource models to frontier systems, under each variant in two modes: Original (real diagram provided) and Mirage (diagram replaced by a blank image, header retained). Three findings emerge: (i) under Normal, Mirage mode matches or exceeds Original on every model, suggesting that models largely bypass the circuit diagram and sometimes perform worse when it is present; (ii) under Anony, the Mirage advantage reverses for seven of eight models, isolating identifier semantics in the header as the primary driver of Normal-mode performance; and (iii) genuine visual grounding accounts for only ∼8–9% of samples, with the vast majority of tasks remaining unsolved 1 https://github.com/nturley/netlistsvg
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
once identifier shortcuts are removed. Based on these findings, we first curate a large-scale circuitdiagram-to-Verilog dataset by mining high-quality Verilog projects from GitHub and converting each design into a circuit diagram. We then introduce VeriGround (4B), a lightweight MLLM trained with three targeted interventions that together address the hallucination-vs-refusal trade-off common to all vision-to-code generators: (i) mixed supervised fine-tuning on Original and Anony data to reduce reliance on semantically loaded identifiers; (ii) refusal augmentation with blankdiagram and image-header-mismatch negatives, teaching the model to abstain when visual evidence is absent or inconsistent; (iii) D-ORPO (Decision-Focused ORPO) alignment, which up-weights the first K response tokens with a decision weight α, concentrating the preference signal on the generateor-refuse boundary and mitigating the over-refusal problem of standard ORPO [15]. Extensive experiments demonstrate that VeriGround effectively reverses the Mirage phenomenon. With only 4B parameters, VeriGround achieves Functional Pass@1 of 46.11% under Normal, approaching GPT-5.4 (45.51%) and surpassing GPT-4o (33.52%) and MiMo-v2-omni (37.72%); under Anony, VeriGround reaches 42.51%, significantly outperforming all baselines (p < 0.001, McNemar’s test). Meanwhile, the False Refusal Rate on valid inputs is reduced to 1.20%/0.00% (Normal/Anony) while the Refusal Rate on blank inputs remains above 92%. In summary, this paper makes four contributions: 1) Phenomenon. We identify and document the Mirage phenomenon: all eight evaluated MLLMs produce equal or higher scores when the circuit diagram is removed, and genuine visual grounding accounts for only ∼8–9% of samples. 2) Benchmark & Evaluation. We construct C2VE VAL with exact image-code correspondence and a paired Normal/Anony×Original/Mirage protocol that isolates identifier semantics as the single variable. 3) Method. We propose VeriGround (4B), trained with identifier anonymization, refusal augmentation, and DORPO (Decision-Focused ORPO) alignment. 4) Evaluation. VeriGround achieves Functional Pass@1 of 46.11%/42.51% (Normal/Anony) with False Refusal Rates of only 1.20%/0.00%, while maintaining ≥92% Refusal Rate on blank inputs. To facilitate the replication of VeriGround, we make our source code, trained models, and benchmark publicly available on GitHub.2 The remainder of this paper is organized as follows. Section II introduces the background and problem definition. Section III presents the empirical study that motivates the approach. Section IV details the proposed method. Section V reports the experimental results. Section VI discusses hyperparameter sensitivity, mismatch refusal, and threats to validity. Section VII reviews related work. Section VIII concludes the paper. 2 https://github.com/NTDXYG/VeriGround
3
II. BACKGROUND AND P RELIMINARIES A. Multimodal Large Language Models A multimodal large language model (MLLM) [16], [17] extends a text-only LLM to process both visual and textual inputs. Given a visual input I and a textual prompt T , an MLLM M auto-regressively generates an output sequence Ŷ = (y1 , . . . , yL ): P (Ŷ | I, T ; θ) =
L Y
P (yt | y<t , I, T ; θ),
(1)
t=1
where θ denotes the model parameters. Existing MLLMs broadly fall into two architectural paradigms. Connector-based MLLMs adopt a three-component architecture. A visual encoder (e.g., ViT [18]–[20]) extracts a feature sequence zv = Enc(I); a connector (typically an MLP) projects it into the LLM embedding space, yielding visual tokens hv = Proj(zv ); an LLM backbone then generates conditioned on [hv ; ht ], where ht = Embed(T ) is the text token embedding sequence. Representative models include LLaVA [21]–[23] and InternVL [24], [25]. Native multimodal MLLMs forgo the connector and jointly train vision and language components within a unified transformer. A visual encoder produces hv = Enc(I) and a token embedding layer produces ht = Embed(T ); both sequences are fed directly into a shared backbone without an intermediate projection, with decoupled parallel strategies for the heterogeneous modalities to maintain training efficiency. Representative models include Gemini [26], [27] and GPT4o [28]. B. Circuit-to-Verilog Code Generation We now specialize the MLLM formulation in Eq. (1) to the circuit-to-Verilog setting. Definition 1 (Circuit-to-Verilog Code Generation). Given a circuit diagram image I and a module header H, the task is to generate a Verilog module body V̂ such that the complete module H ⊕ V̂ is both syntactically valid and functionally equivalent to the reference implementation H ⊕V ∗ , where ⊕ denotes concatenation and V ∗ is the ground-truth module body. The textual prompt takes the form T = (instruction, H), where the instruction is a fixed task description. Since the instruction is constant across all samples, we abbreviate the generation as: V̂ = M(I, H). (2) We next detail the two variable inputs to M. Definition 2 (Module Header). A module header H specifies the external interface of a Verilog module [29]: H = module name [#(params)] (ports);
(3)
where name is the module identifier, params is an optional parameter list, and ports specifies input/output ports with their directions and bit-widths. In standard practice, H is always provided so that the generated interface matches the testbench and evaluation can proceed without name-mismatch failures.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
Definition 3 (Circuit Diagram). A circuit diagram I is a visual representation of the target circuit’s topology, depicting gates, flip-flops, multiplexers, and their interconnections [30]. In our setting, each diagram is rendered from V ∗ via NETLISTSVG, so the labels in I correspond exactly to the identifiers in V ∗ . III. E MPIRICAL S TUDY A. Benchmark Construction To investigate the Mirage phenomenon, we construct C2VE VAL, a circuit-to-Verilog benchmark with exact imagecode correspondence. The construction follows a three-stage pipeline. Stage 1: Seed dataset collection. We source problems from four established Verilog code-generation benchmarks: VerilogEval-v2 [31], [32], RTLLM-v2 [33], [34], ResBench [35], and ArchXBench [36], which collectively span diverse circuit categories, each accompanied by a complete testbench. Stage 2: Reference code curation. For each seed problem, we generate candidate Verilog solutions with two frontier MLLMs (GPT-5.4 and Opus-4.6) and simulate each candidate against the corresponding testbench; only solutions passing all assertions are retained as verified reference implementations V ∗. Stage 3: Diagram rendering and filtering. Each V ∗ is rendered into a circuit diagram via NETLISTSVG and rasterized to JPEG at 96 DPI, producing image I whose labels correspond exactly to the identifiers in V ∗ (Definition 3). Samples that fail to render or whose I exceeds 2,048 visual tokens under the Qwen-3.5 tokenizer are discarded. 1) Normal and Anony variants.: The above pipeline yields the Normal variant of C2VE VAL. To test whether high Normal-mode accuracy is a Mirage driven by identifier semantics rather than genuine visual understanding, we further construct an Anony (anonymized) variant: for each sample, the module name is replaced with a generic module_name, and all port and parameter identifiers are replaced with positional placeholders val_0, val_1, . . . ; the anonymized code is then re-rendered through the same Stage 3 pipeline to produce a new diagram Ianon paired with an anonymized header Hanon . The circuit topology is preserved; only the semantic cues are removed. For example, the following Normal-mode header: module sync_fifo #(DEPTH=32, WIDTH=8) (clk, rst_n, wr_en, rd_en);
is anonymized to: module module_name #(val_0=32, val_1=8) (val_2, val_3, val_4, val_5);
The corresponding circuit diagram is re-rendered with the same positional placeholders, ensuring that the only way to produce correct code is to read the visual topology. 2) Benchmark statistics.: The resulting C2VE VAL comprises 169 samples. Each sample is a tuple (I, H, V ∗ , T , D) together with its anonymized counterpart (Ianon , Hanon ), where I is the circuit diagram, H is the module header (Definition 2), V ∗ is the reference module body, T is the testbench, and D is the natural-language functional description.
4
TABLE I C ATEGORY DISTRIBUTION OF C2VE VAL .
Category
Count
Ratio
Basic Combinational Logic Sequential Building Blocks Finite State Machines Mathematical Operations & Algo.
81 43 32 11
48.5% 25.7% 19.2% 6.6%
Total
169
100%
Fig. 3. Token-count distributions (left: density histograms; right: CDFs) for circuit-diagram images (top, blue) and module headers (bottom, orange) in C2VE VAL.
Table I reports the category distribution: Basic Combinational Logic constitutes the largest share (48.5%), followed by Sequential Building Blocks (25.7%) and Finite State Machines (19.2%), with Mathematical Operations & Algorithms as the smallest category (6.6%). This distribution reflects the natural composition of introductory-to-intermediate RTL design tasks in the seed benchmarks. Fig. 3 visualizes the token-count distributions of the two inputs. Circuit-diagram images exhibit a heavy right-skewed distribution (µ = 340.6, median = 133.0): approximately half of the diagrams encode within 133 tokens, yet a few complex designs exceed 1,500 tokens. Module headers, by contrast, are remarkably compact (µ = 33.5, median = 30.0), with over 90% falling below 60 tokens. This order-of-magnitude asymmetry is noteworthy: the header occupies less than onetenth of the visual token budget, yet it encodes the module name, port names, and parameter values, which collectively carry rich semantic information. B. Empirical Setup 1) Evaluated Models: We evaluate eight MLLMs spanning 4B to frontier-scale parameters. The proprietary group includes GPT-4o [37], GPT-5.4 [38], Opus-4.6 [39], and MiMo-v2-omni [40]. The open-source group includes EGM4B, EGM-8B [41], Qwen3.5-4B, and Qwen3.5-9B [42]. Each
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
5
TABLE II PASS @k (%) ON C2VE VAL UNDER THE N ORMAL AND A NONY BENCHMARK VARIANTS . M IRAGE MODE REPLACES THE CIRCUIT DIAGRAM WITH A BLANK IMAGE WHILE THE MODULE HEADER IS RETAINED . B OLD DENOTES THE HIGHER SCORE BETWEEN O RIGINAL AND M IRAGE IN EACH PAIR .
Model GPT-5.4 GPT-4o MiMo-v2-omni Opus-4.6 EGM 4B EGM 8B Qwen3.5 4B Qwen3.5 9B
Normal
Mode
Anony
Syn. @1
Syn. @5
Func.@1
Func.@5
Syn. @1
Syn. @5
Func.@1
Func.@5
Original Mirage Original Mirage Original Mirage Original Mirage
97.60 100.00 92.22 94.61 85.03 86.23 98.20 100.00
– – – – – – – –
45.51 47.90 39.52 39.52 37.72 41.32 52.69 63.47
– – – – – – – –
93.41 100.00 79.64 94.01 79.64 82.63 88.62 99.40
– – – – – – – –
24.55 6.59 16.17 8.38 19.16 5.99 11.38 14.97
– – – – – – – –
Original Mirage Original Mirage Original Mirage Original Mirage
79.04 83.83 37.72 83.83 38.32 56.89 70.66 76.05
98.20 99.40 64.67 95.81 76.50 94.10 97.60 99.40
18.56 20.36 10.18 22.75 7.78 10.78 14.97 19.76
29.94 31.14 23.35 36.53 19.16 25.75 32.34 38.92
50.90 33.53 22.16 23.95 35.33 61.68 62.28 79.04
93.41 86.83 71.26 73.65 78.44 98.80 92.22 100.00
4.19 0.60 0.60 0.60 5.99 2.40 8.38 3.59
10.78 7.78 8.38 5.99 12.57 5.39 14.97 7.78
model is evaluated under both the Normal and Anony variants of C2VE VAL. 2) Evaluation Metrics: We adopt two Pass@k metrics computed with the unbiased estimator [43], [44]. Syntax Pass@k measures the probability that at least one of k sampled completions yields a syntactically compilable Verilog module. Functional Pass@k measures the probability that at least one of k completions passes all testbench assertions under simulation. Functional correctness implies syntactic validity, so Functional Pass@k ≤ Syntax Pass@k. 3) Implementation Details: Due to API cost constraints, proprietary models are reported as Pass@1 only. Open-source models are served locally; we sample n = 5 completions per problem at temperature = 0.7 and report both Pass@1 and Pass@5. All compilation and simulation are performed with Icarus Verilog 3 . C. Empirical Findings Finding 1: The Mirage phenomenon is universal: removing the circuit diagram does not degrade, and often improves, code generation. Under the Normal benchmark, Mirage mode matches or exceeds Original mode on every model across all metrics (Table II). The most extreme case is EGM 8B, whose Syntax Pass@1 jumps by 46 points upon removing the circuit diagram, while Opus-4.6 sees a Functional gain of over 10 points. The same trend holds at Pass@5: all four open-source models see Mirage surpass Original on every metric, ruling out singlesample variance. A sample-level decomposition (Table III) corroborates this pattern: on average, 13.1% of samples are solved by Mirage only versus 8.2% by Original only, meaning the diagram actively impairs generation on 60% more samples than it aids. 3 https://github.com/steveicarus/iverilog
TABLE III S AMPLE - LEVEL F UNCTIONAL PASS @1 BREAKDOWN (%) ON C2VE VAL (167 SAMPLES ). E ACH SAMPLE IS CATEGORIZED BY WHETHER O RIGINAL AND M IRAGE EACH PRODUCE CORRECT CODE .
Model
Both
Original
Mirage
Neither
GPT-5.4 GPT-4o MiMo-v2 Opus-4.6
35.3 30.5 26.3 43.1
10.2 9.0 11.4 9.6
12.6 9.0 15.0 20.4
41.9 51.5 47.3 26.9
EGM 4B EGM 8B Qwen3.5 4B Qwen3.5 9B
10.2 7.8 2.4 5.4
8.4 2.4 5.4 9.6
10.2 15.0 8.4 14.4
71.3 74.9 83.8 70.7
Avg.
20.1
8.2
13.1
58.5
GPT-5.4 GPT-4o MiMo-v2 Opus-4.6
6.0 4.2 2.4 4.8
18.6 12.0 16.8 6.6
0.6 4.2 3.6 10.2
74.9 79.6 77.2 78.4
EGM 4B EGM 8B Qwen3.5 4B Qwen3.5 9B
0.0 0.0 0.6 1.8
4.2 0.6 5.4 6.6
0.6 0.6 1.8 1.8
95.2 98.8 92.2 89.8
Avg.
2.5
8.8
2.9
85.8
Normal
Anony
Finding 2: Identifier semantics in the module header, not visual understanding, are the primary driver of Normal-mode performance. Anonymization strips semantic identifiers while preserving circuit topology. Under the Anony benchmark, the Mirage advantage vanishes: Original surpasses Mirage on Functional Pass@1 for seven of eight models (Table II), and this reversal extends to Pass@5 for all four open-source models. At the
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
6
Fig. 4. Training pipeline of VeriGround.
sample level (Table III), the Both category, i.e., samples solved by both modes, collapses from 20.1% to 2.5%, confirming that the vast majority of Normal-mode joint successes were driven by header-based shortcuts rather than diagram comprehension. Correspondingly, aggregate Functional Pass@1 drops drastically upon anonymization (e.g., GPT-5.4: 45.51 → 24.55; Opus-4.6: 52.69 → 11.38), further isolating identifier semantics as the dominant factor.
A. Dataset Collection
a) Step 1: Source collection.: Approximately 150K synthesisable Verilog snippets are harvested from public GitHub repositories, spanning a broad spectrum of hardware designs. b) Step 2: Synthesisability verification.: Each snippet is compiled with YOSYS [46] and discarded if synthesis fails, ensuring that every retained sample corresponds to a valid hardware design. c) Step 3: Benchmark decontamination.: To prevent data leakage, we remove any sample whose Rouge-L [47] similarity with the test sets exceeds 0.5. d) Step 4: Difficulty filtering.: Samples for which Qwen2.5-Coder-7B/32B-Instruct can produce a functionally correct solution within five attempts (verified via formal equivalence checking [45]) are discarded, retaining approximately 87K non-trivial instances. e) Step 5: Circuit-diagram rendering.: Each retained reference implementation V ∗ is synthesised into a gate-level netlist and rendered as an SVG schematic via NETLISTSVG. Samples that fail to render are excluded, leaving approximately 67K Circuit–Verilog pairs. f) Step 6: Visual-token budget filtering.: The SVG schematics are rasterised to JPEG at 96 DPI. Samples whose image exceeds 2,048 visual tokens under the Qwen-3.5 tokenizer are discarded, yielding the Normal training set of 27,260 samples. g) Anonymised variant.: Applying the same anonymisation procedure used for C2VE VAL (Section III-A), we replace all semantically loaded identifiers with positional placeholders and re-render the corresponding circuit diagrams, producing the Anony training set of 26,756 samples. The Normal and Anony sets are merged into a unified corpus Dall of 54,016 samples.
We construct a large-scale circuit-diagram-to-Verilog training corpus through a six-step pipeline. Steps 1–4 follow the data-curation workflow of CodeV-R1 [45]; Steps 5–6 extend it with diagram rendering and visual-token budgeting.
B. Mixed Supervised Fine-Tuning We partition Dall into two disjoint subsets: a seed pool Dalign of 5,000 samples reserved for the subsequent alignment
Finding 3: Genuine visual grounding is extremely limited, accounting for only ∼8–9% of samples. The Original-only rate in Table III, which captures samples where the diagram is necessary and sufficient for correct generation, remains nearly identical across Normal (8.2%) and Anony (8.8%). This stability suggests that approximately 8– 9% of samples contain visual cues that models can genuinely extract irrespective of whether identifiers carry semantic meaning. Beyond this small fraction, current MLLMs largely fail to leverage circuit diagrams: under Anony, 85.8% of samples are solved by neither mode, indicating that once identifier shortcuts are removed, the overwhelming majority of circuitto-code tasks remain out of reach. IV. M ETHOD Figure 4 illustrates the overall training pipeline of VeriGround, which consists of three stages: (i) training data collection and anonymisation, (ii) supervised fine-tuning (SFT) on the mixed Normal + Anony corpus, and (iii) preference alignment with D-ORPO.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
7
Prompt Template
TABLE IV P REFERENCE - PAIR COMPOSITION FOR THE ALIGNMENT STAGE .
Please write a Verilog module based on the provided circuit diagram image. Return only the Verilog code, without any explanation.
Cat.
Image condition
Chosen yw
Rejected yl
Ratio
M ATCH B LANK M ISMATCH
Correct diagram Blank image Unrelated diagram
Verilog Refusal Refusal
Refusal Verilog Verilog
40% 30% 30%
For example: ‘‘‘verilog your Verilog code here ‘‘‘ Module header (must not be changed): {module_header}
Refusal Response Template Based on the provided circuit diagram, I cannot accurately determine the Verilog implementation. The module header provided is: {module_header} However, the provided image does not match the given module header, so I cannot generate the correct Verilog code with confidence. Fig. 5. Prompt and refusal templates used for preference-pair construction. {module_header} is instantiated with the sample-specific header H.
stage (Section IV-D), and Dsft = Dall \Dalign (49,016 samples) used for supervised fine-tuning. Each SFT instance is a triple (I, H, V ∗ ), where I denotes the circuit diagram, H the module header, and V ∗ the reference Verilog module body. The model Mθ is optimised with the standard autoregressive objective:
rejected response yl follows the refusal template in Figure 5. This category reinforces the model’s generation capability, ensuring that valid visual evidence is not spuriously refused. b) Category B LANK (30%).: The circuit diagram is replaced with a blank white image I∅ , while the module header H is retained. The chosen response follows the refusal template; the rejected response is the header-matching code V ∗ . By explicitly penalising code generation in the absence of visual input, this category directly targets the Mirage shortcut. c) Category M ISMATCH (30%).: The circuit diagram I ′ is sampled from a different instance in Dalign such that I ′ depicts a circuit unrelated to H. Refusal is again chosen over the header-matching code. This category trains the model to detect semantic inconsistency between the visual and textual modalities, rather than defaulting to header-driven generation. Ratio design. Each source sample in Dalign naturally yields exactly one M ATCH pair, one B LANK pair, and one M IS MATCH pair, giving a raw ratio of 1 : 1 : 1. We observe that this equal split over-represents refusal-preferred pairs (two out of three categories choose refusal), biasing the model toward over-refusal. To counterbalance, we up-sample M ATCH to 40% and assign 30% each to B LANK and M ISMATCH, yielding a 4 : 3 : 3 split that keeps the three categories close in size while giving the generation-preferred category a slight majority.
|V ∗ |
LSFT = −
X
∗ log pθ (vt∗ | I, H, v<t ),
(4)
t=1
where vt∗ denotes the t-th token of V ∗ . Because Dsft interleaves Normal and Anony samples in roughly equal proportion, the same module header H appears with both semantically loaded and anonymised identifiers across different instances. This compels the model to ground its predictions in the visual topology of I rather than memorising identifier-to-template mappings, as a direct countermeasure to the Mirage phenomenon identified in Section III-C. C. Preference Pair Construction The alignment stage operates on preference pairs (yw , yl ) (chosen vs. rejected), each conditioned on an input (I, H). A well-calibrated circuit-to-Verilog model should produce correct code when the diagram faithfully depicts the target circuit, and refuse to answer when the visual evidence is absent or inconsistent with the header. All three categories share the unified prompt and refusal templates shown in Figure 5. We derive three complementary categories of preference pairs from Dalign , summarised in Table IV. a) Category M ATCH (40%).: The input pairs a matching circuit diagram I with its corresponding module header H. The chosen response yw is the reference module body V ∗ ; the
D. D-ORPO: Decision-Focused ORPO 1) ORPO Background: Preference alignment methods such as DPO [48] and GRPO [49] impose significant overhead on multimodal models: DPO doubles memory by maintaining a frozen reference model, while GRPO requires costly online rollouts with high-resolution image encoding. ORPO [15] avoids both costs by unifying supervised learning and preference optimisation in a single, reference-free objective. Given a preference pair (yw , yl ) conditioned on input x, ORPO combines the negative log-likelihood (NLL) of the chosen response with an odds-ratio (OR) penalty: 1 X log pθ (yw,t | x, yw,<t ) + β · LOR , (5) LORPO = − |yw | t | {z } LNLL
where β controls the preference strength and the OR term is oddsθ (yw | x) LOR = − log σ log , (6) oddsθ (yl | x) with oddsθ (y | x) ≜ pθ (y | x) / (1 − pθ (y | x)). In practice, the sequence-level log-probability is replaced by its token-level average: T
ℓ̄θ (y | x) =
1X log pθ (yt | x, y<t ), T t=1
(7)
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
where T = |y|. Note that every token contributes equally to ℓ̄θ ; we revisit this assumption below. 2) Motivation: In our generate-or-refuse setting, the binary decision is fully determined by the first few response tokens (cf. Figure 5): once the model emits ‘‘‘verilog it is committed to code generation, whereas a refusal opens with Based on the ..., I cannot ... . This creates a length asymmetry: code responses span hundreds of tokens while refusals contain only tens, so the uniform weighting of Eq. 7 dilutes the odds-ratio gradient on the pivotal opening tokens. We observe that this dilution leads to over-refusal: the model favours the shorter refusal path even for valid inputs, because the uniform odds ratio provides insufficient gradient on the initial decision to generate. 3) Formulation: To address this, we propose D-ORPO (Decision-Focused ORPO), which assigns a higher weight α > 1 to the first K response tokens in both the NLL and OR objectives. Let r = min{t : yt is a response token} denote the index of the first response token. We define a per-token weight function: ( α, if r ≤ t < r + K, wt = (8) 1, otherwise, where K is the decision window size and α is the decision weight. The weighted average log-probability replaces Eq. 7: PT wt · log pθ (yt | x, y<t ) D . (9) ℓ̄θ (y | x) = t=1 PT t=1 wt
8
Algorithm 1: D-ORPO Training Input: SFT-initialised model Mθ ; preference dataset Dpref ; decision window K; decision weight α; OR coefficient β Output: Aligned model Mθ (i) (i) B 1 for each mini-batch {(xi , yw , yl )}i=1 from Dpref do 2 Forward pass: compute per-token log-probs log pθ (yt | x, y<t ) for both yw and yl ; 3 Compute per-token weights wt via Eq. 8; 4 Compute ℓ̄θD (yw | x) and ℓ̄θD (yl | x) via Eq. 9; 5 Compute LD-ORPO via Eq. 10; 6 Update θ via back-propagation on LD-ORPO ;
Proof. (1) follows by direct substitution. For (2), dΓ/dα = K(Tr − Tc )/[αK + (Tr − K)]2 < 0 since Tr < Tc . For (3), dividing numerator and denominator by αK gives Γ → 1. Property (1) quantifies the imbalance under standard ORPO: the decision gradient fraction for the short refusal is Tc /Tr times that for the long code response, causing the OR objective to steer the decision boundary toward refusal far more strongly than toward generation. Properties (2) and (3) guarantee that D-ORPO monotonically reduces this imbalance toward unity as α increases. Algorithm 1 summarises the D-ORPO training procedure.
E. Training Details
measures the fractional contribution of the K decision tokens to the gradient of ℓ̄θD (Eq. 9). When α = 1, this reduces to K/T .
We instantiate VeriGround, initialised from Qwen3.5-4B. VeriGround is fine-tuned with LoRA [50] (r=16, αLoRA =16, dropout = 0) applied to all attention and MLP projection matrices in both the vision encoder and the language backbone. Training proceeds in two sequential stages. a) Stage 1: Supervised fine-tuning.: The model is trained on Dsft (49,016 samples) for 5 epochs using AdamW (8-bit) with a learning rate of 2 × 10−4 , and a maximum sequence length of 4,096 tokens. We hold out the last 5,000 samples as a validation set, evaluate every 1,000 steps, and retain the checkpoint with the lowest validation loss. b) Stage 2: D-ORPO alignment.: Starting from the best SFT checkpoint, we perform one epoch of D-ORPO training on the preference pairs derived from Dalign (Section IV-C). The learning rate is reduced to 5×10−6 . The D-ORPO-specific hyper-parameters are set to decision window K=8, decision weight α=2.0, and OR coefficient β=0.1.
Proposition 1 (Gradient Rebalancing). Let Tc > Tr > K denote the lengths of a code response and a refusal response, respectively. The decision gradient imbalance ratio
V. E XPERIMENTAL R ESULTS
Substituting ℓ̄θD for ℓ̄θ in Eqs. 5 and 6 yields the D-ORPO objective: D D LD-ORPO = LNLL + β · LOR . (10) When α = 1, D-ORPO reduces to standard ORPO, confirming it as a strict generalisation. 4) Theoretical Analysis: Definition 4 (Decision Gradient Fraction). For a response of length T with decision window K ≤ T and weight α ≥ 1, the decision gradient fraction ϕ(T, K, α) ≜
Γ(α) ≜
αK αK + (T − K)
ϕ(Tr , K, α) αK + (Tc − K) = ϕ(Tc , K, α) αK + (Tr − K)
satisfies: 1) Γ(1) = Tc /Tr ; 2) Γ(α) is strictly decreasing in α for all α ≥ 1; 3) limα→∞ Γ(α) = 1.
(11)
(12)
This section evaluates VeriGround along two research questions: RQ1: Does VeriGround improve grounded code generation? • RQ2: Does VeriGround refuse unreliable visual inputs without over-refusing valid diagrams? •
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
9
TABLE V PASS @k (%) OF V ERI G ROUND ABLATION VARIANTS ON C2VE VAL UNDER THE PAIRED N ORMAL /A NONY × O RIGINAL /M IRAGE PROTOCOL . F OR O RIGINAL ROWS , BOLD = HIGHEST ( BEST ); FOR M IRAGE ROWS , BOLD = LOWEST ( BEST ).
Variant Base (Qwen3.5-4B)
Anony-mixed SFT +ORPO
6 \ Q W D [ 3 D V V #
+D-ORPO
Syn. @5
Func.@1
Func.@5
Syn. @1
Syn. @5
Func.@1
Func.@5
38.32 56.89 87.43 81.44 96.41 88.02 95.81 8.38 96.41 3.59
76.50 94.10 97.01 98.80 98.80 100.00 98.80 34.73 98.80 17.37
7.78 10.78 28.74 12.57 43.71 14.37 41.92 0.60 46.11 0.00
19.16 25.75 46.11 28.74 61.08 28.14 60.48 4.79 61.08 2.99
35.33 61.68 91.02 86.83 97.01 88.02 95.81 3.59 96.41 1.80
78.44 98.80 100.00 100.00 99.40 100.00 99.40 11.98 99.40 10.78
5.99 2.40 11.98 2.99 40.72 4.19 40.72 0.00 42.51 0.00
12.57 5.39 29.34 4.19 54.49 11.38 51.50 0.00 56.29 0.00
D 6 \ Q W D [ 1 R U P D O
2 X U V
* 3 7 2 S X V * 3 7 R 0 L 0 R Y R P Q L
( * 0 % 4 Z H Q %
4 Z H Q % ( * 0 % % % %
%
7
F ) X Q F 1 R U P D O
) X Q F 3 D V V #
%
0 R G H O 6 L ] H
* 3 7 2 S X V 0 L 0 R Y R P Q L * 3 7 R
4 Z H Q % ( * 0 % 4 Z H Q % ( * 0 % % % %
9 H U L * U R X Q G % 2 X U V 2 S H Q V R X U F H 3 U R S U L H W D U \ V L ] H H V W
%
7
2 X U V
0 L 0 R Y R P Q L * 3 7 R ( * 0 % 4 Z H Q % ( * 0 % 4 Z H Q % % % %
E 6 \ Q W D [ $ Q R Q \
2 X U V
G ) X Q F $ Q R Q \
2 S X V * 3 7
2 X U V
7
Anony
Syn. @1 Original Mirage Original Mirage Original Mirage Original Mirage Original Mirage
SFT
Normal
Mode
* 3 7 0 L 0 R Y R P Q L * 3 7 R 2 S X V 4 Z H Q % 4 Z H Q % ( * 0 % ( * 0 % % % % % 7
0 R G H O 6 L ] H
Fig. 6. Original-mode Pass@1 comparison on C2VE VAL. (a–b) Syntax; (c– d) Functional. Open-source models (•) are at their parameter count; MiMov2-omni at 42B active (1T total, MoE). Proprietary models (■, size estimated) are in the shaded region. VeriGround (⋆) approaches GPT-5.4 under Normal and surpasses all models under Anony.
A. RQ1: Code Generation Accuracy To disentangle the contribution of each training component, we evaluate four cumulative ablation stages: (i) SFT, finetuned on Normal-only data; (ii) Anony-mixed SFT, fine-tuned on the merged Normal + Anony corpus; (iii) +ORPO, standard ORPO alignment after Anony-mixed SFT; (iv) +D-ORPO (i.e. VeriGround), decision-focused ORPO alignment after Anonymixed SFT. All variants are evaluated on C2VE VAL under the paired Normal/Anony × Original/Mirage protocol. Table V reports the ablation results; Figure 6 compares VeriGround against all baselines as a function of model size. We draw three observations. Obs. 1: Anony-mixed SFT is the key to visual grounding. Mixing anonymised training data into SFT yields substantial gains across all settings. Compared with Normalonly SFT, Anony-mixed SFT raises Functional Pass@1 from 28.74 → 43.71 (+14.97) under Normal-Original, and from 11.98 → 40.72 (+28.74) under Anony-Original, nearly clos-
ing the Normal–Anony gap (∆,=,2.99). This confirms that anonymisation training compels the model to ground predictions in visual topology rather than memorising identifierto-template mappings. However, SFT alone cannot teach the model to refuse when visual evidence is absent: Mirage Syntax Pass@1 remains as high as 88.02%, indicating that the model still produces syntactically valid but functionally incorrect code without any circuit diagram. Obs. 2: D-ORPO achieves the best generation–refusal trade-off. Standard ORPO effectively teaches refusal, reducing Mirage Syntax Pass@1 from 88.02% to 8.38%/3.59% (Normal/Anony), but at the cost of generation quality: Functional Pass@1 drops from 43.71 → 41.92 (Normal) and 54.49 → 51.50 (Func.@5, Anony). D-ORPO resolves this tension by further reducing Mirage Syntax Pass@1 to 3.59%/1.80% while simultaneously lifting Original Functional Pass@1 to 46.11% (Normal) and 42.51% (Anony), the highest among all ablation stages. The decision-focused weighting concentrates the preference signal on the generate-or-refuse boundary, preventing the over-refusal caused by length asymmetry in standard ORPO. Obs. 3: A 4B model rivals frontier-scale proprietary MLLMs. As shown in Fig. 6(c), VeriGround achieves Functional Pass@1 of 46.11% under Normal, approaching GPT-5.4 (45.51%) and surpassing both GPT-4o (33.52%) and MiMov2-omni (37.72%), despite having ∼50–250× fewer parameters. Under Anony (Fig. 6(d)), the advantage becomes decisive: VeriGround reaches 42.51%, exceeding GPT-5.4 (24.55%) by 17.96 points and Opus-4.6 (11.38%) by 31.13 points. This widening gap under anonymised evaluation confirms that VeriGround’s strength stems from genuine visual grounding rather than identifier-based shortcuts that benefit larger pretrained models. Statistical Significance. All models are evaluated on the same 167 C2VE VAL samples, yielding paired binary outcomes (pass/fail) per sample. We apply McNemar’s test to each VeriGround-vs-baseline pair: let b denote the number of samples that only VeriGround solves and c the number that only the baseline solves. For discordant counts b+c ≤ 25 we use the exact mid-p variant; otherwise the continuity-corrected χ2 form χ2 = (|b − c| − 1)2 /(b + c). All eight p-values
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
TABLE VI M C N EMAR ’ S TEST FOR F UNCTIONAL PASS @1 BETWEEN V ERI G ROUND AND BASELINE MODELS ON O RIGINAL INPUTS . b / c: SAMPLES SOLVED ONLY BY V ERI G ROUND / ONLY BY THE BASELINE . ∗∗ p < 0.01 AFTER H OLM –B ONFERRONI CORRECTION ; N . S .: NOT SIGNIFICANT.
Normal b
Baseline MiMo-v2-omni GPT-4o GPT-5.4 Opus-4.6
32 23 22 22
c 18 12 21 33
n.s. n.s. n.s. n.s.
b 48 50 42 56
TABLE VII R EFUSAL RELIABILITY OF +ORPO AND +D-ORPO ON C2VE VAL . FRR IS MEASURED ON O RIGINAL INPUTS (↓); RR ON M IRAGE INPUTS (↑); PASS @1 ON O RIGINAL INPUTS (↑). B OLD MARKS THE BETTER VALUE PER COLUMN .
Metric
Anony p
10
c
p
9 6 12 4
∗∗
< 0.001 < 0.001∗∗ < 0.001∗∗ < 0.001∗∗
(4 baselines × 2 conditions) are corrected jointly with the Holm–Bonferroni procedure at α=0.05. Table VI reveals a clear dichotomy. Under Normal evaluation, none of the four comparisons reaches significance: for instance, VeriGround vs. GPT-5.4 yields b=22, c=21 (p=1.0), indicating virtually identical per-sample outcomes. This confirms that VeriGround, with only 4B parameters, performs on par with frontier-scale proprietary models when standard signal names are available. Under Anony evaluation, all four comparisons are highly significant (p < 0.001), with b ≫ c in every case (e.g. 56 vs. 4 against Opus-4.6). While baseline models suffer severe performance degradation without meaningful identifiers (Opus-4.6: 52.69% → 11.38%; GPT5.4: 45.51% → 24.55%), VeriGround retains 42.51%—a drop of only 3.60. This statistically confirms that VeriGround’s accuracy stems from genuine visual grounding rather than identifier-based shortcuts. Summary of RQ1 Anony-mixed SFT builds strong visual grounding; DORPO further lifts Original Functional Pass@1 to 46.11%/42.51% (Normal/Anony) while driving Mirage generation to near zero. The resulting 4B model matches or exceeds frontier-scale proprietary MLLMs, with statistically significant gains under Anony evaluation.
B. RQ2: Refusal Reliability A reliable circuit-to-Verilog model should generate correct code when the diagram faithfully depicts the target circuit and refuse otherwise. We compare +ORPO and +D-ORPO on two complementary metrics, both computed on C2VE VAL: • False Refusal Rate (FRR ↓): fraction of valid Originaldiagram inputs on which the model erroneously refuses. • Refusal Rate (RR ↑): fraction of blank-image (Mirage) inputs on which the model correctly refuses. A well-calibrated model should minimise FRR while maximising RR; we additionally report Functional Pass@1 on Original inputs to confirm that refusal tuning does not degrade generation quality. Table VII shows that standard ORPO achieves near-perfect RR (≥98%) but suffers from severe over-refusal: FRR reaches 8.98% (Normal) and 7.19% (Anony), meaning the model
+ORPO
+D-ORPO
FRR (%) ↓ – Normal FRR (%) ↓ – Anony
8.98 7.19
1.20 0.00
RR (%) ↑ – Normal RR (%) ↑ – Anony
98.20 98.80
92.81 97.60
Func. Pass@1 (%) ↑ – Normal Func. Pass@1 (%) ↑ – Anony
44.31 38.32
46.11 42.51
TABLE VIII E FFECT OF DECISION WINDOW K (α=2.0 FIXED ) ON V ERI G ROUND -4B. B OLD = BEST PER COLUMN .
K 2 4 8 16
Pass@1 ↑
FRR ↓
RR ↑
Norm.
Anon.
Norm.
Anon.
Norm.
Anon.
45.51 46.11 46.11 41.32
40.12 42.51 42.51 42.51
1.80 1.20 1.20 4.19
0.60 1.80 0.00 0.00
88.62 89.22 92.81 84.43
96.41 97.01 97.60 95.81
spuriously rejects roughly one in twelve valid inputs. D-ORPO resolves this imbalance decisively: FRR drops to 1.20% under Normal and zero under Anony, a reduction of 7.78 and 7.19 respectively. RR decreases only moderately (98.20%,→,92.81% Normal; 98.80%,→,97.60% Anony), remaining above 92% across all settings and preserving reliable refusal capability. Crucially, the FRR reduction does not come at the expense of generation quality: D-ORPO improves Functional Pass@1 from 44.31%,→,46.11% (Normal) and 38.32%,→,42.51% (Anony), confirming that decision-focused weighting prevents over-refusal from eroding the code-generation pathway. Summary of RQ2 D-ORPO reduces False Refusal Rate by up to 7.78 compared with standard ORPO (to 1.20%/0.00% Normal/Anony) while retaining ≥92% Refusal Rate on blank images and simultaneously improving Functional Pass@1.
VI. D ISCUSSION A. Hyper-parameter Sensitivity D-ORPO introduces two hyper-parameters beyond standard ORPO: the decision window size K and the decision weight α. We conduct ablation studies on VeriGround-4B, evaluating Functional Pass@1 on Original inputs, False Refusal Rate (FRR) on Original inputs, and blank-image Refusal Rate (RR) under both Normal and Anony conditions. 1) Effect of Decision Window Size K: Table VIII fixes α=2.0 and varies K ∈ {2, 4, 8, 16}. K=8 achieves the best overall balance. When K is too small (K=2), the decision window covers too few tokens to
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
TABLE IX E FFECT OF DECISION WEIGHT α (K=8 FIXED ) ON V ERI G ROUND -4B. α=1 CORRESPONDS TO STANDARD ORPO. B OLD = BEST PER COLUMN .
α
Pass@1 ↑ Norm.
1 2 3 5 10 20
44.31 46.11 44.31 42.51 38.92 44.91
Anon. 38.32 42.51 41.92 41.92 43.11 41.92
FRR ↓ Norm. 8.98 1.20 1.80 1.80 3.59 1.20
Anon. 7.19 0.00 0.00 1.20 0.00 2.40
11
TABLE X F UNCTIONAL PASS @1 (F UNC ., %) AND M ISMATCH R EFUSAL R ATE (MRR, %) OF +ORPO AND +D-ORPO ACROSS FIVE ROUNDS OF RANDOM IMAGE MISMATCH .
RR ↑ Norm. 98.20 92.81 88.02 80.24 77.84 75.45
+ORPO
Anon. 98.80 97.60 97.01 97.01 92.22 95.81
capture the full decision preamble, yielding lower Pass@1 under Anony (40.12%) and reduced RR under Normal (88.62%). When K is too large (K=16), the window extends beyond the decision boundary into code-body tokens, causing Normal Pass@1 to drop to 41.32% and FRR to rise to 4.19%. K=8 simultaneously achieves the highest Pass@1, the lowest FRR, and the highest RR across both conditions. 2) Effect of Decision Weight α: Table IX fixes K=8 and varies α ∈ {1, 2, 3, 5, 10, 20}. Note that α=1 recovers standard ORPO. The results reveal a clear trade-off. Standard ORPO (α=1) achieves near-perfect RR (≥98%) but suffers from severe overrefusal (FRR: 8.98%/7.19%), confirming the gradient imbalance characterised by Proposition 1. Setting α=2 reduces FRR by 7.78 and 7.19 absolute points while maintaining RR above 92%, and simultaneously lifts Pass@1 to 46.11%/42.51%. Beyond α=2, Normal RR declines monotonically (88.02% → 80.24% → 77.84% → 75.45% for α=3, 5, 10, 20), while Pass@1 shows no further improvement. α=2 therefore provides the Pareto-optimal balance and is adopted as the default configuration. In summary, K=8 and α=2.0 yield the best trade-off among generation quality, false refusal, and blank-image refusal. DORPO is robust to moderate perturbations of either hyperparameter but degrades when K or α deviates substantially from these values. B. Mismatch Refusal Analysis The RR metric in Section V-B measures refusal on blank images. In practice, a model may also encounter mismatched diagrams, where the provided circuit image depicts a different module than the one specified in the header. To evaluate this scenario, we randomly construct five mismatched datasets: in each round, every test sample is paired with a circuit diagram drawn from a different instance in C2VE VAL. We compare +ORPO and +D-ORPO on Functional Pass@1 and Mismatch Refusal Rate (MRR). Table X shows that +ORPO refuses nearly all mismatched inputs (MRR 99.88%), consistent with its over-refusal tendency observed in Section V-B. +D-ORPO exhibits a lower MRR of 81.08%, meaning roughly one-fifth of mismatched samples are not explicitly refused. However, the Functional Pass@1 of +D-ORPO on these mismatched inputs is only 0.84%, virtually zero, indicating that the non-refused outputs still fail functional verification.
+D-ORPO
Round
Func.
MRR
Func.
MRR
1 2 3 4 5
0.00 0.00 0.00 0.00 0.00
100.00 100.00 100.00 100.00 99.40
0.60 1.20 0.60 0.60 1.20
80.84 82.04 80.84 83.23 78.44
Avg.
0.00
99.88
0.84
81.08
In other words, when +D-ORPO does not refuse a mismatched diagram, it does not hallucinate a “correct” implementation either; instead, the generated code is functionally incorrect and would be caught by standard testbench simulation. This result suggests that +D-ORPO’s lower refusal rate does not introduce a meaningful safety risk: the model may occasionally attempt generation on mismatched inputs, but the resulting code is unlikely to pass downstream verification, providing a practical safety net against silent misuse. A promising direction for future work is to further improve the mismatch refusal rate without increasing the false refusal rate on valid inputs, for example through contrastive training on hard negative pairs where the diagram and header share partial but inconsistent semantics. C. Threats to Validity Internal validity. Our ablation fixes one hyper-parameter while varying the other; joint interactions beyond the tested grid (K ∈ {2, 4, 8, 16}, α ∈ {1, 2, 3, 5, 10, 20}) may exist. All training runs use a single random seed; we mitigate this by reporting Pass@k with the unbiased estimator over n=5 samples per problem. External validity. C2VE VAL covers 169 Verilog problems drawn from four public benchmarks, spanning combinational logic, sequential blocks, FSMs, and arithmetic modules. Whether the findings generalise to other hardware description languages (e.g. VHDL, SystemVerilog) or larger circuit scales. We instantiate VeriGround at 4B parameters; scaling behaviour at larger or smaller model sizes is unexplored. Construct validity. Functional Pass@k relies on testbench simulation, which may not cover all corner cases. Refusal detection is based on template matching against the fixed refusal format (Figure 5); free-form refusals or hedged responses are not captured by this metric. VII. R ELATED W ORK A. Vision-to-Code Generation The broader vision-to-code paradigm treats visual artifacts as visual domain-specific languages (visual DSLs) that must be faithfully translated into executable code. Representative tasks include UI-to-HTML synthesis [1]–[6], where a mockup encodes layout and interaction semantics, and Plot-to-Python
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
generation [7]–[9], where a chart encodes data transformations and rendering logic. A common, yet underexplored, risk across these tasks is that models may exploit textual metadata (e.g., axis labels, component names) rather than genuinely parse the visual content, producing outputs that appear correct on standard benchmarks but fail on unseen inputs. Circuit-to-Verilog generation extends this paradigm to the hardware domain, where a circuit diagram serves as a visual DSL encoding timing, topology, and bit-level semantics. In the text-only setting, LLMs have been extensively applied to Verilog generation. Yang et al. [12] and Pan et al. [13] provide comprehensive surveys. On the benchmarking side, VerilogEval [31] and its updated version [32] evaluate functional correctness via testbench simulation; RTLLM [33] and OpenLLM-RTL [34] target design-level RTL; ResBench [35] adds resource-awareness for FPGA designs; and ArchXBench [36] covers complex digital subsystems. On the model side, QiMeng-CodeV-R1 [45] augments Verilog generation with chain-of-thought reasoning, while Yang et al. [44] address backdoor attacks through semantic consensus decoding. All of these works operate in a text-only setting. Only a handful of studies have explored visual inputs for Verilog generation. VGV [10] prompts MLLMs with circuit screenshots and reports encouraging Thinking Vision methods, but evaluates exclusively under semantically loaded identifiers, leaving the question of genuine visual grounding open. MGEMMV [11] proposes a specialised framework for GEMM-oriented circuits, yet likewise does not test whether models exploit header semantics rather than reading the diagram. Our work is the first to systematically expose and quantify this shortcut across multiple MLLMs, and to provide evidence that the reliability gap observed here reflects a general vulnerability of vision-to-code pipelines, with circuit diagrams representing the most extreme case due to the invisible and safety-critical nature of RTL semantics. B. Visual Grounding in Multimodal LLMs Whether MLLMs genuinely rely on visual inputs has attracted growing scrutiny. Asadi et al. [14] coin the term “Mirage” to describe a broad failure mode in which MLLMs appear to understand images but instead exploit spurious correlations or textual priors. Their study spans medical imaging, scientific figures, and general-purpose vision-language tasks, demonstrating that performance often persists after visual information is removed or corrupted. C. Preference Alignment for Language Models Aligning language models with human preferences has evolved from reinforcement learning from human feedback (RLHF) to more efficient offline methods. DPO [48] eliminates the reward model by directly optimising a preference objective, but requires a frozen reference model that doubles memory for large multimodal architectures. GRPO [49] employs grouprelative policy optimisation with online rollouts, achieving strong results in mathematical reasoning but incurring high sampling costs for vision-language models. ORPO [15] unifies
12
supervised learning and preference alignment in a single reference-free objective, making it well suited to resourceconstrained multimodal training. VIII. C ONCLUSION This paper reveals the Mirage phenomenon in circuit-toVerilog code generation: all eight evaluated MLLMs score equally or higher when the circuit diagram is replaced by a blank image, exploiting identifier semantics in the module header rather than the visual input. This exposes a covert defect type in AI code generation that threatens MLLMs’ trustworthiness. We construct C2VE VAL with paired Normal/Anony variants, showing that genuine visual grounding accounts for only about 8% of samples. To restore grounding, we propose VeriGround (4B), combining identifier anonymization, refusal augmentation, and D-ORPO alignment, a transferable recipe for balancing hallucination and over-refusal. VeriGround achieves 46.11%/42.51% Functional Pass@1 (Normal/Anony), approaching GPT-5.4 under Normal and outperforming all baselines under Anony, while keeping false refusal below 1.20% and blank-image refusal above 92%. In the future, we plan to scale VeriGround to both lighter and larger architectures and to expand C2VE VAL with industrial-scale circuits. We also intend to validate the transferability of the anonymization-refusal-alignment paradigm on other vision-to-code tasks to further establish its generality. ACKNOWLEDGEMENTS This work was supported by National Key R&D Program of China (No. 2024YFB4506400). The authors would like to thank the editors and the anonymous reviewers for their insightful comments and suggestions, which can substantially improve the quality of this work. R EFERENCES [1] S. Feng, S. Ma, J. Yu, C. Chen, T. Zhou, and Y. Zhen, “Auto-icon: An automated code generation tool for icon designs assisting in ui development,” in Proceedings of the 26th International Conference on Intelligent User Interfaces, 2021, pp. 59–69. [2] S. Xiao, Y. Chen, J. Li, L. Chen, L. Sun, and T. Zhou, “Prototype2code: End-to-end front-end code generation from ui design prototypes,” in International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, vol. 88353. American Society of Mechanical Engineers, 2024, p. V02BT02A038. [3] Y. Gui, Y. Wan, Z. Li, Z. Zhang, D. Chen, H. Zhang, Y. Su, B. Chen, X. Zhou, W. Jiang et al., “Uicopilot: Automating ui synthesis via hierarchical code generation from webpage designs,” in Proceedings of the ACM on Web Conference 2025, 2025, pp. 1846–1855. [4] T. Zhou, Y. Zhao, X. Hou, X. Sun, K. Chen, and H. Wang, “Declarui: Bridging design and development with automated declarative ui code generation,” Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 219–241, 2025. [5] J. Xiao, Y. Wan, Y. Huo, Z. Wang, X. Xu, W. Wang, Z. Xu, Y. Wang, and M. R. Lyu, “Interaction2code: Benchmarking mllm-based interactive webpage code generation from interactive prototyping,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 241–253. [6] Y. Wan, C. Wang, Y. Dong, W. Wang, S. Li, Y. Huo, and M. Lyu, “Divide-and-conquer: Generating ui code from screenshots,” Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 2099– 2122, 2025.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
[7] C. Wu, Z. Liang, Y. Ge, Q. Guo, Z. Lu, J. Wang, Y. Shan, and P. Luo, “Plot2code: A comprehensive benchmark for evaluating multimodal large language models in code generation from scientific plots,” in Findings of the Association for Computational Linguistics: NAACL 2025, 2025, pp. 3006–3028. [8] X. Zhao, X. Luo, Q. Shi, C. Chen, S. Wang, Z. Liu, and M. Sun, “Chartcoder: Advancing multimodal large language model for chartto-code generation,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 7333–7348. [9] S. Ouyang, D. Huang, J. Guo, Z. Sun, Q. Zhu, and J. M. Zhang, “Dscodebench: A realistic benchmark for data science code generation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 38, 2026, pp. 32 628–32 636. [10] S.-Z. Wong, G.-W. Wan, D. Liu, and X. Wang, “Vgv: Verilog generation using visual capabilities of multi-modal large language models,” in 2024 IEEE LLM Aided Design Workshop (LAD). IEEE, 2024, pp. 1–5. [11] G. Zhang, M. Wang, and Z. Wang, “Mgemmv: A multimodal llm framework for gemm verilog generation from circuit diagrams,” IEEE Transactions on Circuits and Systems I: Regular Papers, 2026. [12] G. Yang, W. Zheng, X. Chen, D. Liang, P. Hu, Y. Yang, S. Peng, Z. Li, J. Feng, X. Wei et al., “Large language model for verilog code generation: Literature review and the road ahead,” arXiv preprint arXiv:2512.00020, 2025. [13] J. Pan, G. Zhou, C.-C. Chang, I. Jacobson, J. Hu, and Y. Chen, “A survey of research in large language models for electronic design automation,” ACM Transactions on Design Automation of Electronic Systems, vol. 30, no. 3, pp. 1–21, 2025. [14] M. Asadi, J. W. O’Sullivan, F. Cao, T. Nedaee, K. Fardi, F.-F. Li, E. Adeli, and E. Ashley, “Mirage the illusion of visual understanding,” arXiv preprint arXiv:2603.21687, 2026. [15] J. Hong, N. Lee, and J. Thorne, “Orpo: Monolithic preference optimization without reference model,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 11 170– 11 189. [16] J. Wu, W. Gan, Z. Chen, S. Wan, and P. S. Yu, “Multimodal large language models: A survey,” in 2023 IEEE International Conference on Big Data (BigData). IEEE, 2023, pp. 2247–2256. [17] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,” National Science Review, vol. 11, no. 12, p. nwae403, 2024. [18] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu et al., “A survey on vision transformer,” IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 87–110, 2022. [19] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022. [20] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM computing surveys (CSUR), vol. 54, no. 10s, pp. 1–41, 2022. [21] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu et al., “Llava-onevision: Easy visual task transfer,” Transactions on Machine Learning Research. [22] C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,” Advances in Neural Information Processing Systems, vol. 36, pp. 28 541–28 564, 2023. [23] F. Cocchi, N. Moratelli, D. Caffagni, S. Sarto, L. Baraldi, M. Cornia, and R. Cucchiara, “Llava-more: A comparative study of llms and visual backbones for enhanced visual instruction tuning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 4278–4288. [24] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu et al., “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 24 185–24 198. [25] W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao et al., “Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency,” arXiv preprint arXiv:2508.18265, 2025. [26] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023.
13
[27] G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024. [28] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024. [29] Z. Navabi, Verilog digital system design. McGraw-Hill, 1999. [30] R. W. Mehler, Digital integrated circuit design using verilog and systemverilog. Elsevier, 2014. [31] M. Liu, N. Pinckney, B. Khailany, and H. Ren, “Verilogeval: Evaluating large language models for verilog code generation,” in 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 2023, pp. 1–8. [32] N. Pinckney, C. Batten, M. Liu, H. Ren, and B. Khailany, “Revisiting verilogeval: A year of improvements in large-language models for hardware code generation,” ACM Transactions on Design Automation of Electronic Systems, vol. 30, no. 6, pp. 1–20, 2025. [33] Y. Lu, S. Liu, Q. Zhang, and Z. Xie, “Rtllm: An open-source benchmark for design rtl generation with large language model,” in 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 2024, pp. 722–727. [34] S. Liu, Y. Lu, W. Fang, M. Li, and Z. Xie, “Openllm-rtl: Open dataset and benchmark for llm-aided design rtl generation(invited),” in Proceedings of 2024 IEEE/ACM International Conference on ComputerAided Design (ICCAD). ACM, 2024. [35] C. Guo and T. Zhao, “Resbench: A resource-aware benchmark for llmgenerated fpga designs,” in Proceedings of the 15th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies, 2025, pp. 25–34. [36] S. Purini, S. Garg, M. Gaur, S. Bhat, S. Mupparapu, and A. Ravindran, “Archxbench: A complex digital systems benchmark suite for llm driven rtl synthesis,” in 2025 ACM/IEEE 7th Symposium on Machine Learning for CAD (MLCAD). IEEE, 2025, pp. 1–10. [37] OpenAI, “Openai api docs: Gpt-4o,” 2024. [Online]. Available: https://developers.openai.com/api/docs/models/gpt-4o [38] ——, “Openai api docs: Gpt-5.4,” 2026. [Online]. Available: https://developers.openai.com/api/docs/models/gpt-5.4 [39] Anthropic, “Introducing claude opus 4.6,” 2026. [Online]. Available: https://www.anthropic.com/news/claude-opus-4-6 [40] Xiaomi MiMo Team, “Xiaomi mimo-v2-omni,” March 2026. [Online]. Available: https://mimo.xiaomi.com/mimo-v2-omni [41] G. Zhan, C. Li, Z. Liu, Y. Lu, Y. Wu, S. Han, and L. Zhu, “Egm: Efficient visual grounding language models,” 2026. [42] Qwen Team, “Qwen3.5: Towards native multimodal agents,” February 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3.5 [43] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107.03374 [44] G. Yang, X. Hu, X. Chen, and X. Xia, “Semantic consensus decoding: Backdoor defense for verilog code generation,” arXiv preprint arXiv:2602.04195, 2026. [45] Y. Zhu, D. Huang, H. Lyu, X. Zhang, C. Li, W. Shi, Y. Wu, J. Mu, J. Wang, P. Jin et al., “Qimeng-codev-r1: Reasoning-enhanced verilog generation,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems. [46] C. Wolf, J. Glaser, and J. Kepler, “Yosys-a free verilog synthesis suite,” in Proceedings of the 21st Austrian Workshop on Microelectronics (Austrochip), vol. 97, 2013, pp. 1–6. [47] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74–81. [48] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in neural information processing systems, vol. 36, pp. 53 728–53 741, 2023. [49] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li et al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,” arXiv preprint arXiv:2402.03300, 2024.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. XX, NO. XX, XX 2026
[50] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” Iclr, vol. 1, no. 2, p. 3, 2022.
14