Construction Contracts for Generated Embedded Domain-Specific Language Programs: A Measurement Protocol and a Conditional Effect

Construction Contracts for Generated Embedded Domain-Specific Language Programs: A Measurement Protocol and a Conditional Effect Haowen Guoa,∗, Yangzhao Huanga a School of Information and Software Engineering, University of Electronic Science and Technology of China, No. 2006 Xiyuan Avenue,

High-Tech Zone (West), Chengdu, 611731, Sichuan, China

arXiv:2611.06525v1 [cs.SE] 4 Nov 2026

Abstract

A generated domain-specific program can be accepted by its host and still miss the reference construction. We measure the two events separately and test a short construction contract. The vehicle is Open Quantum Assembly Language (OpenQASM) 3. One 3-billion-parameter generator is run under seven conditions on 100 quantum approximate optimization (QAOA) and 100 variational quantum eigensolver (VQE) instances. Eighty of 100 QAOA signatures appear in some training split. Exact McNemar tests and 95% paired bootstrap intervals are used. The contract, a card, lists structural decisions, enters the prompt, and is checked without a simulator. Parse rates are 0.55 to 0.97 and full structural match is 0.02 to 0.87. On QAOA, at fixed weights, the card moves full match by +32.0 percentage points on card-trained weights (interval +22 to +42) and by −13.0 (−23 to −3) without that training. The +45.0-point difference is not a deployment contrast. The same-step plan-free gain is +23.0 (+13 to +33); the primary distributional score does not separate it. Against the earlier checkpoint the gain is +27.0 (+17 to +37), and the checkpoint also changes. A rule card is not separated from the learned planner, nor is a predicted-card compiler (0.760 vs. 0.780). On 20 QAOA rows outside every training split, the interval includes zero. Under this card, VQE adherence is 0.020 and full match stays near 0.03. Report the two checks separately. Insert a card only into a generator trained to read it, and only where a pilot shows adherence. Keywords: code generation, domain-specific languages, software measurement, large language models, empirical software engineering

1. Introduction

Tool chains increasingly delegate a fragment of work to a large language model: a query, a constraint, a configuration unit, a circuit description. Each is an embedded domain-specific program. Its text means little on its own and much relative to an interface the host has already fixed, and the interface decides whether the fragment is accepted. Acceptance is weaker than correctness, because a fragment that parses can still violate the construction the host expects: the terms it covers, the symbols it shares, the order in which it commits to them. A consumer downstream can reject a fragment the parser accepted. Software engineering has met this gap before, under other names. Text-to-SQL reports exact match beside execution accuracy, because the two disagree on queries that are right in different ways (Yu et al., 2018), and the oracle problem says the same thing more generally: what decides correctness is the criterion the check applies, not a property ∗ Corresponding author.

Email addresses: 2025091203013@std.uestc.edu.cn (Haowen Guo), yangzhao_se2025@163.com (Yangzhao Huang)

of the program alone (Barr et al., 2015). Code-synthesis metrics add a syntax-level term to string similarity for the same reason (Ren et al., 2020). What these leave open is the operational question. Can a reference construction be recovered from the text by a walk over it, reported field by field, computed without a simulator, and then used as a design target? That is the middle object this paper measures, and it sits between the two numbers that evaluations of generated code usually report. The measurements come from one such interface. A finetuned generator writes parameterized quantum circuits in OpenQASM 3 (Cross et al., 2022), and a fixed host interface decides what it may declare and how many lines it may use. Across the 200 evaluation instances of the main comparison the parser accepts most of what it is given; far less of it matches the reference. In the most divergent cell 0.810 of the outputs parse and 0.020 match all six structural fields. The host can accept the artifact even though the circuit encoded in it differs structurally from the reference. We then place a specification between the instance and the artifact and ask whether it moves that second number. We call it a construction contract (a card): a finite product

of constant-size structural decisions. A walk extracts the card from a reference artifact, and a forward pass predicts it from the instance text. The word is narrower than design by contract (Meyer, 1992). There is no precondition, no postcondition, and no runtime assertion. The card records a reference construction. We score it as prompt content and as a text-walk predicate, and the prompt block is its serialized form. Seven conditions and one relevance control separate slot content from slot presence, from prompt format, from extra training, and from the weights that read the card. The two families differ in construction, so the boundary can be measured rather than assumed. Three findings organize what follows. Parse rate and full structural match move apart on the same texts. On QAOA the card raises full match only after card-conditioned training, and that estimate is conditional on the constructed set. On VQE, under this card, adherence stays near zero and full match does not rise. After a shared refit, the primary distributional measure does not separate the matched-step control (Sections 5.5 and 5.8). Circuits are the vehicle. Section 6.3 states the transfer test, and Section 7 records the limits.

is not separated from other generation-side failures. The 7B probe raises adherence to 0.210 on the reference card and to 0.172 on the predicted one, while full match on the predicted card stays at 0.070; it has no base stage and no plan-free continuation, so it is reported as an observation (Section 5.4, Table 10). The extractor, checker, prompts, per-row generations, and evaluation scripts are deposited as University of Electronic Science and Technology of China et al. (2026) (Section Data availability). Every table can be rebuilt from the per-row logs in that deposit. The raw circuit corpus has no public identifier and is not redistributed. Third parties can recompute the published tables from the deposited logs and cannot regenerate the reference circuits from a public problem set. The scope is deliberately narrow. No new generator, intermediate language, or claim about the best way to synthesize circuits is offered. The generator is held fixed across conditions, because the measurements concern the artifact and the practice. Structural conformance is referencerelative, and Section 5.8 measures how far it stands from semantic equivalence. The field list is local to the corpus that produced it, and Section 6.3 states what would have to hold for another embedded language.

1.1. Contributions The paper makes three contributions.

2. Background and related work

1. A two-predicate measurement protocol for generated domain-specific language (DSL) artifacts. Interface acceptance (parse) and reference structural conformance (full match) are separate columns. Skeleton match is the name-quotiented companion, and semantic quality is reported beside them rather than as a third predicate. The structural predicate is a walk over the text. It calls no simulator, and it names the field that failed (Sections 3.3, 5.1 and 5.7). 2. A checked, corruptible intermediate scored as prompt content and as a structural predicate, with controls for content, format, extra training, and a plannerfree card. On this QAOA template, card content and card-conditioned training are jointly necessary. At fixed weights the predicted card is worth +32.0 pt of full match on card-trained weights and costs 13.0 pt on weights that never saw a card. The +45.0 pt gap summarizes that two-by-two and is not a deployment contrast. A rule card and a model-free compiler of the predicted card are not separated from the learned pipeline at this n. The pipeline contrast against onestage also changes the checkpoint and is reported as a deployment figure (Sections 5.2, 5.3 and 6.1). 3. A boundary, read on a second family of the same data rather than as a pre-specified moderator. Under the card as defined here, VQE adherence is 0.020 and full match stays between 0.02 and 0.04; the reference card itself leaves full match at 0.030 while breaking the interface (parse 0.550). That pattern is consistent with an incomplete field list (no parameter budget) but

Pass@k summaries fit general-purpose languages with executable tests (Austin et al., 2021; Chen et al., 2021). An embedded DSL fragment is weaker for that protocol: the language is low-resource, “running it” may mean checking a relation, and the host interface that consumes the fragment is not the test harness. Quantum code-generation benchmarks make the gap concrete—accuracy below 0.40 with failures sorted into API, construction, and logic errors (Guo et al., 2025)—but still score the finished program without separating interface acceptance from construction match (Section 5.1). Software engineering already knows related separations. Insufficient tests can inflate pass rates by tens of points (Liu et al., 2023); DSL tooling is itself an engineering artifact (Ratiu et al., 2018); text-to-SQL and the oracle problem separate exact match from execution accuracy (Barr et al., 2015; Yu et al., 2018); and constraint-language generation in this journal separates well-formedness from correctness (Delgado et al., 2026). What those traditions still leave open for a host-embedded fragment is a predicate computable from the text alone—no suite, no simulator—that recovers the construction field by field. End-to-end execution feedback is expensive when the event is rare (Runeson and Höst, 2009; Wohlin et al., 2012; Yu et al., 2025), and unaided self-repair rarely helps without a localizing external signal (Chen et al., 2024; Huang et al., 2024; Kamoi et al., 2024; Madaan et al., 2023; Olausson et al., 2024). That is why the contract sits before generation, not as a critique of a finished circuit. 2

Sketches (Solar-Lezama, 2008), constrained decoding (Scholak et al., 2021), free-text self-plans (Jiang et al., 2024), human or protocol-guided plans (Ho and Li, 2026; Wei et al., 2026), and Spider-style component match (Yu et al., 2018) each stop short of one intermediate that is predicted from the instance, withheld or corrupted, and scored both as prompt content and as a structural predicate (Section 4). A free-text plan is not a finite field vector. Constrained decoding rejects illegal tokens during generation. The card is checked afterward and can be legal yet wrong for the instance (Section 5.9). A sketch is a partial program. The card does not emit gates until an executor or a generator realizes it. The word contract will be heard as design by contract (Meyer, 1992). Those traditions constrain behavior at a boundary. The object studied here records a reference construction and is neither a precondition nor a runtime check (Section 4.1). Host-language embedding and languageworkbench engineering are the natural neighbors of a fragment the parser accepts while the construction is still wrong. This study runs neither, and multi-file repair is out of scope. Full match certifies agreement with a fixed construction, which Table 14 shows is not a near miss on fitted energy (Ralph et al., 2021; Runeson and Höst, 2009; Wohlin et al., 2012). A single scalar often mixes the three events (Evtikhiev et al., 2023). Sections 3.3 and 5.1 give each event its own column, and Section 5.6 measures training–evaluation overlap instead of assuming a clean split.

4. Where the contract is realized. Does the generator carry the reference contract out on both families, and does structural match rise where it does? 3.2. Instances and artifacts An artifact is a parameterized circuit in OpenQASM 3, written against a fixed host interface: one declaration input float[64] x, no ancilla qubits, no mid-circuit measurement, and a line budget set by the family. This is the object a tool chain accepts or rejects, and it is small enough for a structural checker to read without a human judge and without a simulator. Two families appear because their constructions differ. That difference is what makes question 4 decidable. The variational family is not a second copy of the sample: it is where adherence collapses under the card as defined in this study, so the design can show a boundary rather than assume one. Quantum approximate optimization (QAOA) alternates cost and mixer layers on a problem graph (Farhi et al., 2014). The variational quantum eigensolver (VQE) repeats a motif drawn from an ansatz family (Peruzzo et al., 2014). The evaluation set holds 200 instances, 100 per family, and it is not a uniform sample of the test split. A fixed rule first keeps every row whose signature is absent from the QAOA and VQE training splits, with no further sampling. That family-clean cell (Table 11) has 46 QAOA rows and 82 VQE rows, 128 in all. Each family is then filled to 100 from rows whose signature was seen, stratified by qubit count (≤ 8, 9–12, > 12) at a fixed seed, so these absent rows are over-represented. Of the 128 family-clean rows, 102 are absent from every split, against the 17 such rows the parent split would contribute. Of the 46 familyclean QAOA rows, 20 are absent from every split; all 82 family-clean VQE rows are. Across the 200 rows the qubit bands are 75, 21 and 4 for QAOA and 17, 65 and 18 for VQE, and depth runs from p = 1 to 4 (62, 12, 15 and 11 QAOA instances; 64, 10, 14 and 12 for VQE). The qubit counts are uneven by construction. Each instance carries the problem type, the graph or the ansatz, the cost operator, the number of qubits, and, for QAOA, the depth; it carries no energy, no optimum and no reference parameter vector. Recovering the construction from that text is the generator’s task. A separate corpus of 10,207 reference circuits serves only to derive and audit the contract (Section 4.3); inference never sees it.

3. Study design We compare generation conditions on a fixed set of instances, and we hold the generator, the decoder, and the checker fixed. Only two things move: the checkpoint, when a control requires it, and the text placed in the contract slot at inference. 3.1. Research questions Four questions organize the measurements, and for each one Table 1 fixes the comparison that answers it and the measure that comparison may use. 1. Interface and construction. Do generated artifacts fail because they do not parse, or because the construction inside a parsed artifact is wrong? Are the parse rate and the structural match separable within a single condition? Semantic scores are taken up in Section 5.5. 2. Effect of the contract. When the generator is given a contract predicted from the instance, does structural match rise relative to a one-stage generator, and relative to a generator continued for the same number of steps without a contract? 3. What the gain depends on. Is a rise in structural match accounted for by extra training, by the checkpoint, by the presence of a formatted block, or by the content and the names inside the contract?

3.3. Measures Table 2 fixes the measures. Two of them are easy to conflate, and the rest of the paper treats them as different predicates. Reference-based structural conformance, abbreviated full match in the tables, is true when every structural field of the reference construction agrees with the emitted artifact. QAOA has five fields: the covered terms, the coefficient slot of each symbol, the allocation of symbols, 3

Table 1. Research design. Each question, the comparison that answers it, and the measure that comparison uses. Supporting measurements of solution quality and of benchmark overlap are reported with the question they qualify. Question

Comparison

Measure

1 2 3

All seven conditions, both families Conditioned against One-stage; Conditioned against Matched-step Equal-weight, Matched-step, Renamed, Shuffled, the rule card, and A1 QAOA, where the generator carries the reference contract out, against VQE, where the 3B generator does not

Parse rate against full match full match full match and skeleton match

4

full match and adherence

divergence falls between 0.02 and 0.6 at floors of 10−6 , 10−9 and 10−12 moves the largest value by 6 × 10−3 , and no verdict changes. By Pinsker’s inequality, a divergence of 0.1 keeps total variation at or below about 0.22 on the regularized quantity. The contrast is stable in the cut (+8.9 points at 0.1, +8.9 at 0.05, and +11.1 at 0.3; Section 5.5). The one-sided hit at tolerance τ , written q-hit@τ , is true when the fitted energy is at most the reference energy plus τ . The two-sided score is true when the absolute deviation is at most 0.2. VQE is not given a semantic conclusion, because adherence there is too low for the contract to have been executed. The arms emit a median of 93 parameters against 42 in the reference circuits. Splitting the variational rows at that median does not move the outcome (small half SREV 0.108 and HQCR 0.027; large half 0.094 and 0.031).

the order of lines, and inversion. Order and cover coincide on this corpus, because the reference order is the cover order, so the conjunction has four independent fields on QAOA and order is carried as a check. VQE has six: the motif, the repetition count, the sharing pattern, the tail, the gate set, and the arity. Skeleton match quotients symbol names and compares only the cover and the slot sequence. It is defined for QAOA. On VQE it is undefined, and the tables leave the cell blank. A blank cell is not a failure. The difference between the two predicates mixes field strictness with naming, so a naming effect is read from skeleton match and a full-field effect from full match. The measure certifies agreement with a construction a tool chain has fixed, not correctness in the abstract. Three layers stay apart: interface validity (the text parses and declares exactly the parameters it uses), reference structural conformance, and semantic quality. Only the middle layer is the primary measure. Conformance is a walk over the text, so it is sufficient for mechanical audit. It is not necessary for semantic equivalence. An artifact can implement the same state with a different construction (Section 5.8). It does not replace semantic validation, which is why those measures are reported beside it. A tool that reads the construction would make the predicate an engineering property. This study runs no such tool (Section 7.1). Adherence is true when the artifact implements the contract it was given, whether or not that contract matches the reference. It is computed for the conditioned condition only. Full match remains the primary structural measure, because a generator can adhere to a wrong contract. Scores on the parameters as emitted bind every symbol parameter to the constant 0.1, and the reference circuit the same way; the 0.1 here is not the KL threshold below. That protocol is degenerate (Section 5.1). The semantic measures are therefore computed from a refit of the reference circuit and of every emitted circuit under one budget: COBYLA, five restarts, 250 iterations, seed 0 on each call. HQCR is true when the Kullback–Leibler divergence of the reference output distribution from the emitted one is at most 0.1. It is the primary semantic measure because it does not saturate: the one-sided energy hit reaches 1.000 on both arms of the reference-contract condition and cannot separate them. Divergence is measured over computational-basis outcomes. Both circuits of a pair have the same number of qubits, so the supports coincide. Each probability is floored at 10−12 and the vectors are renormalized, with the reference on the left of DKL (p ∥ q). The reported quantity is this floor-regularized divergence. Recomputing the units whose

3.4. Conditions Seven conditions separate the contract from the checkpoint, and from the surface form of the prompt; Table 3 lists them. Two comparisons are primary, Conditioned against One-stage and Conditioned against Matched-step. Equal-weight shares the weights of Conditioned so that a difference can be attributed to the contract, and it is no performance baseline: with a placeholder in the contract slot it scores below the one-stage condition. Oracle-contract hands the generator the reference contract itself, which the generator may still fail to execute. Renamed and Shuffled corrupt that contract; they are controls, not methods. An eighth condition, A1, controls the relevance of the contract rather than its presence and runs after the seven: the conditioned generator receives a shape-compatible but instance-irrelevant reference contract from another instance, matched on family, qubit count and depth. Section 5.9 reports it with its result; it is not one of the seven conditions the main contrasts use. Both generators are LoRA continuations of public Qwen2.5 checkpoints (Qwen Team, 2024a,b). The main comparison starts from Qwen2.5-3B-Instruct (snapshot aa8e725) and the scale probe from Qwen2.5-7B-Instruct. Two stages precede the continuations: a base stage on 4,457 QAOA rows gives A0, and A0+ adds the variational family for 10,207 rows. A0+ is the one-stage condition and the frozen ancestor of both continuations, and it never sees a contract: its prompts carry no plan block at all. The planner is a separate LoRA on the same 3B base, trained for two epochs on 9,604 instance-contract pairs at a learning rate of 2 × 10−4 and a maximum 4

Table 2. Measures. Interface and structural measures are functions of the emitted text. Semantic measures are computed after a shared parameter refit and are interpreted on QAOA, where that refit is closed. Full match and skeleton match are different predicates. Measure

Definition

What a positive value asserts

SCR PCS Full match Skeleton match Adherence HQCR q-hit@τ

Fraction accepted by qasm3.loads Declared parameters equal referenced parameters All structural fields agree (5 on QAOA, 6 on VQE) Names quotiented; cover and slots only; QAOA The artifact implements the contract it received KL(Pref ∥Parm ) ≤ 0.1, floor-regularized Fitted energy ≤ Eref + τ

The text parses The interface is closed The construction matches the reference The name-free skeleton matches The plan was carried out The output distribution is close The solution is within tolerance

Table 3. Generation conditions: checkpoint, contract at inference and role. The primary comparisons are Conditioned against One-stage and Conditioned against Matched-step. Condition

Checkpoint

Contract at inference

Role

One-stage Matched-step Equal-weight Conditioned Oracle-contract Renamed Shuffled

One-stage One-stage, 638 further steps Shared with Conditioned Mixture, below Shared with Conditioned Shared with Conditioned Shared with Conditioned

None Placeholder Placeholder Predicted from the instance Reference contract Reference, names bijection Reference, grouping broken

Baseline Matched step count Equal weights Contract Oracle reference Naming control Content control

Predicted on plan-free weights Reference on plan-free weights

Without Without

Predicted contract Reference contract

Crossed cell Crossed cell

A1 Rule card

Shared with Conditioned Shared with Conditioned

Reference of another instance Deterministic rule on the term list

Relevance control Planner-free card

Equal-weight shares the Conditioned checkpoint. Oracle-contract receives the reference contract. Renamed and Shuffled corrupt it. The crossed cells are the same prompts on weights continued without a contract (Table 8). A1 and the rule card are reported in Sections 5.9 and 5.3.

resamples, is of that form. Every cell reports its n. Each test compares paired outcomes on this fixed set. It is not an inference to a superpopulation, and the set is the constructed sample of Section 3.2. The exact McNemar test is the primary inference and the bootstrap interval is descriptive: where the two disagree, as on the narrow conjunction of cover and slot against the reference contract, the exact test governs and the bootstrap interval is reported beside it. Three analyses are primary and the performance claim rests on those alone, all on QAOA full match: Conditioned against One-stage, Conditioned against Matched-step, and the interaction of Table 8, which is the estimate the twoby-two was built for. A Holm correction over the three leaves all three significant (adjusted p ≤ 7 × 10−5 ). The other rows of Table 7 compare attribution and control conditions; they carry exact tests and intervals, they are left uncorrected, and none is read on its own. The same convention applies to the semantic contrasts, which carry their discordant counts in Section 5.5, and to the overlap views, which are the same contrast measured three ways rather than three independent tests. Each primary contrast is also reported on three overlapping views of training– evaluation overlap, and the strictest view is treated as underpowered rather than as a negative result (Section 5.6). Semantic comparisons use the rows on which the shared refit converged, so their n is stated with the contrast.

sequence length of 3,072. Training data for the conditioned continuation mixes reference contracts, contracts produced by the planner, and placeholder contracts, with measured shares 0.497, 0.302, and 0.201; the plan-free continuation keeps the same block in every row and fills it with the placeholder. Both continuations that enter the primary contrast start from the same frozen ancestor. They run 638 optimizer steps at an effective batch of 16 (one sequence per device step, gradient accumulation 16), at learning rate 10−4 with a cosine schedule and 19 warmup steps, AdamW, maximum sequence length 4096, in bfloat16 on one RTX 4090 D (24 GB). The adapters are LoRA of rank 16 on the seven attention and MLP projections. Both corpora hold the same 9,604 rows, so the control fixes the schedule and the instance count and varies only the corpus. The seed is 42 for the frozen pair and 1337 for the replicate. Planning is a separate forward pass over the instance text, into whose prompt no energy, distribution, or reference parameter enters. Decoding is greedy throughout. Renaming applies a group-preserving bijection to the symbol names of the reference contract, drawn with a fixed seed; shuffling breaks the grouping of terms across coefficient classes. The plan text of both controls is a deterministic function of the reference contract and that seed, and, though not stored as a separate log, the evaluation scripts regenerate it. 3.5. Statistical analysis Comparisons are paired: an exact McNemar test on the discordant pairs, and a 95% percentile paired bootstrap interval at 4,000 resamples, which no bound moves by more than 0.3 points when recomputed at 20,000. Every bootstrap interval in the paper, including those at 20,000

4. The construction contract 4.1. What the contract is for A construction contract is an intermediate, machinecheckable specification of the structural decisions a generated artifact is expected to realize. The case for putting it 5

before generation is timing and price. A score of the finished artifact is computed after the last token, it is one number for the whole program, and on this benchmark it costs about 36 s per QAOA artifact and 3.4–4.2 min per VQE artifact once the variational refit is included. A dropped term stays invisible at the step where it was dropped. The contract moves a checkable description of the construction to the input side, where it costs one forward pass. There is no precondition, no postcondition, and no runtime check (Section 2). The object is a construction summary, written before the program exists and compared with the program afterward. Formally, a contract is a value in a product S = S1 × · · · × Sk in which each factor is constant or linear in the number of qubits. An extractor φ reads an artifact and returns the contract it realizes. A planner ψ reads the instance and returns a contract. The extractor is a deterministic walk over the text; the planner is a learned forward pass. The extractor does not call a simulator. Full match and adherence are then indicators of equality in S, full match(p) = 1[φ(p) = φ(p⋆ )],

(1)

adherence(p, s) = 1[φ(p) = s],

(2)

oracle, and 99.97% of the reference corpus falls under the extraction schema. Diagnosability: failures land in named fields, so a miss can be attributed; the coefficient slot is missed in 78% of the failed QAOA artifacts and the covered terms in 38%, and on VQE the repetition count in 93% and the motif in 73%. The third is a property of the pipeline. Realizability: the generator can carry a reference contract out at all. Adherence is 0.808 on QAOA and 0.020 on VQE at 3B, and the variational family is the case in which a realizable contract is not available (Section 5.4). Planner accuracy was considered and is not a requirement. The planner leads one-stage on the scored fields: 0.930 and 0.850 against 0.780 and 0.570 on QAOA, and 0.630 and 0.480 against 0.070 and 0.200 on VQE. A deterministic rule supplies a card of the same value, so the effect is not planner accuracy (Section 5.3). It is card-conditioned training, which that section measures and does not decompose. The three readings are a retrospective description of these data, not a forecast (Section 7.4). Table 4 is the reading used below. On the one-stage arm alone the QAOA slot is still missed in 87.8% of failures and the covered terms in 44.9%, so the concentration is not an artifact of arms built to damage a field. Handing VQE the reference card yields full match 0.030 while cutting parse from 0.810 to 0.550. A length-matched random card lands near that interface loss (parse 0.520), so much of the break tracks the volume of the block. Semantic contrasts on VQE would score a contract the model rarely executes.

where p⋆ is the reference artifact and s is the contract the generator was given. Two of the five QAOA fields are the contract’s own. Order coincides with cover on this benchmark, and inversion is carried as a check. Rename and shuffle are what separate full match from skeleton match (Table 13). Equation (2) would accept a generator that implements a wrong contract, which is why Equation (1) is primary. Skeleton match is the same indicator after quotienting the names. Let ρ be the relabelling that maps each symbol to its rank in the order of first appearance in the artifact, and let π keep only the cover and the sequence of slots, so that σ = π ◦ ρ sends a contract to its name-free skeleton. Then skeleton match(p) = 1[σ(φ(p)) = σ(φ(p⋆ ))],

4.3. The contract used in the study For QAOA each line of the contract names an operation, the set of terms it covers, and the coefficient slot of its symbol. Line order is recorded with the line. For VQE a line names the motif, the repetition count, the sharing pattern, and the tail, while the gate set and the arity are recorded in the header and checked once per artifact; all six are part of the predicate. The same header and the same line index are used for both families, so a comparison can be made line by line even though the fields differ. Fig. 1 shows the relation on one instance: a parsed artifact can still miss terms, mis-assign depth, or fail to share symbols, and the contract is the short object that records those choices. A predicted contract is 76–1,489 characters, with a median of 336 on QAOA and 101 on VQE. The artifact it describes is 438–9,272 characters, with a median of 1,502 and 3,707. Counts are of the emitted text, so they can be checked against the per-row logs. Fig. 2 shows the inference path. The planner reads the instance. The generator reads the instance and the contract. The checker compares contracts and does not call a simulator.

(3)

defined on QAOA only, because a variational contract carries no symbol that a relabelling could act on. Equation (3) is invariant under a group-preserving renaming of the reference by construction. That invariance is what makes the Renamed condition of Table 3 a control on the naming rather than on the construction. 4.2. Reading the two families after the fact Three descriptions are reported, because the measurements below do not mean what they say without them. They were formulated after looking at this benchmark and are evaluated on it retrospectively; none of them predicts which family pays. Each is measured on both families in Table 4. Two are properties of the object. Decidability: every field is settled by a walk over the text, with no semantic

4.4. How the fields were chosen The field list was fixed from the reference corpus before any condition was evaluated, by three filters that use the reference artifacts alone. Coverage removes the three circuits the schema cannot represent. Invariance drops a field 6

Table 4. Three retrospective readings (Section 4.2): what each one judges, how it is measured, and what the two families show.

Reading

Object

How it is measured

QAOA

Decidability

the contract

Coverage of the extraction schema

Diagnosability the error distribution Share of failed artifacts in which a decision field is missed Realizability

the generator

Adherence to the card received: predicted card, then reference card

VQE

99.97% of the corpus slot 78% repetition 93% cover 38% motif 73% 0.808 0.870

0.020 0.030 utility not estimable

Coverage uses 10,207 reference circuits. Adherence compares decisions the card carries. Allocation, order, and inversion are checked only by full match. Near-zero adherence leaves the utility of realized plans not estimable. reference construction

conditioned artifact

one-stage artifact

CARD qaoa n=5 p=1 1 cost [0,1,2] x0 2 cost [3,4,5,6,7,8] x1 3 cost [9] x2 4 lin [10,11,12] x3 5 lin [13,14] x4 6 mixer [0,1,2,3,4] x5

OPENQASM 3.0; include "stdgates.inc"; input float[64] x4; input float[64] x5; qubit[5] q; ··· h q[0]; rz(x5) q[0]; h q[0]; h q[1]; rz(x5) q[1]; h q[1]; ···

OPENQASM 3.0; include "stdgates.inc"; input float[64] x4; (no x5 declared) qubit[5] q; ··· h q[0]; rz(x4) q[0]; h q[0]; h q[1]; rz(x4) q[1]; h q[1]; ···

Fig. 1. One instance, three objects, reproduced from the per-row logs. Left: the contract extracted from the reference. Center and right: excerpts from the Conditioned and One-stage artifacts. Both parse and close the interface. Red marks the coefficient-slot difference; gray marks omitted lines. instance text

absent parameter budget is taken up in Section 5.4. What survives is a symbol convention read off the corpus rather than chosen: the median number of declarations is 7 against 13 terms, symbols are shared by coefficient class, and rx is never emitted. Two later observations are reported only as description. Failures concentrate in the coefficient slot and the covered terms (Section 4.2), and a placeholder block on the same checkpoint reaches full match 0.460 against 0.510 for one-stage. Neither observation entered the selection: the list was fixed first, and the concentration was measured afterward.

n, p, terms, coefficients planner, 1 pass

construction contract 76–1,489 characters generator, 1 pass

artifact: OpenQASM 3 438–9,272 characters

out of path: simulator / solver / retrieval

no fit, no run

structural checker, no simulator cover, slot, order, inversion

Fig. 2. Inference path of contract-conditioned generation. One forward pass predicts the contract from the instance text, a second generates the artifact from instance and contract, and the checker compares contracts by a text walk. The dashed box is deliberate: no simulator, classical solver, or retrieval step is used on this path (Section 3.3).

4.5. Alternatives we rejected Before settling on the contract we considered three other reductions of the task. A vector of fitted parameters fails decidability: comparing two vectors reintroduces the variational fit the reduction was meant to avoid. Per-block shares fail for the same reason. Solution support fails diagnosability, because two artifacts may cover the same terms and still differ in depth, order, and symbol sharing, so a support mismatch names no failing step. Only the contract met all three readings of Section 4.2, and it pays for them with a family-specific field list (Section 6.3).

that no reference circuit violates: allocation and inversion leave the decision list at this step, and ordering leaves separately, because the order of a reference circuit is always the order of its cover. Referential closure keeps the fields a walk over the text can settle, which is what excludes anything the variational evaluator would have to check. The test is a count over all 10,207 circuits rather than a threshold on a sample, so the resulting list does not depend on how the corpus is split. All three dropped fields remain in the predicate as checks: the checker certifies them, and Section 5.7 reports how often the generator misses them anyway. Table 5 records the role of each candidate. A decision is covered, unviolated by the references, and settled by a walk. A check is unviolated by the references. A field is excluded when the variational evaluator would have to verify it. The

5. Results All conditions use the same 200 instances, greedy decoding, and no post-hoc repair. They differ only in the checkpoint and in the contract-slot text (Section 3.4). Each cell is a paired McNemar test with a bootstrap interval (Section 3.5). Structural rates are read from the emitted text. Semantic measures use the shared refit and are interpreted on QAOA. The subsections follow Section 3.1. 7

Table 5. Why each candidate field is a decision, a check, or absent. The evidence column records what each field was judged on before any condition was evaluated. Field

Role

Evidence

Decision

cover (QAOA)

decision

kept

slot (QAOA)

decision

motif (VQE)

decision

repeat (VQE)

decision

allocation ordering

check check

inversion rx gate parameter budget

check excluded absent

covered by the schema; decidable by a walk; varies in the reference corpus covered by the schema; decidable by a walk; varies in the reference corpus covered by the schema; decidable by a walk; varies in the reference corpus covered by the schema; decidable by a walk; varies in the reference corpus never violated by a reference circuit never violated by a reference circuit; equal to cover in every condition never violated by a reference circuit never emitted (0% of artifacts) arms emit median 93 parameters against 42

kept kept kept kept as a check kept as a check kept as a check removed not carried; failure in Section 5.4

Filters use the reference corpus only, before any condition is evaluated. Failure location (Section 4.2) did not choose the list. Order equals cover, so the five QAOA fields are four verdicts plus one check.

+23.0 pt, interval [+13, +33], p = 3.4 × 10−5 . Ancestor, step count, and instance count are held fixed. The corpus is what differs. Both gains exceed the spread of the parse rate on this family, so neither is a parsing effect. The estimate is conditional on the set as constructed (Sections 5.6 and 7.4). Equal-weight holds the conditioned weights and empties the contract slot. Full match moves by +32.0 pt and skeleton match by +29.0 pt. The checkpoint cannot account for either difference, and +32.0 pt is larger than either trained contrast. Fig. 3 plots the intervals. The control holds the schedule fixed rather than the prompt-token count, because a contract occupies more of the prompt than a placeholder. The whole plan-free continuation is worth +4.0 pt against the uncontinued base, interval [−3, +11], p = 0.42. The 100 QAOA rows carry 98 distinct signatures. A cluster bootstrap (20,000 resamples) and a sign-flip test on cluster means leave the primary contrast at +23.0 pt, interval [+13.3, +33.7], p < 0.001. No other primary bound moves by more than 0.7 points. Recomputed on covered terms and coefficient slots, the two fields the contract decides, Conditioned reaches 0.840 against 0.570 for one-stage (+27.0 pt, [+18, +36]), +21.0 pt against matched-step ([+13, +30]), and +29.0 pt at fixed weights ([+19, +39]). Against matched-step the full-match difference is +23.0 pt, so the fields outside the contract account for those two points. The gap between the learned planner and the reference contract falls from 9.0 pt to 4.0 pt on that conjunction, and four discordant pairs do not establish it (Section 5.3). Of 22 conditioned mismatches, 16 fail on a covered term or a slot and 6 only on assignment or order. Matched-step splits 37 and 8 of 45 the same way. Re-training both continuations from the same ancestor with seed 1337 reaches 0.770 against 0.570 (+20.0 pt, [+10.0, +30.0], p = 3.2 × 10−4 ). Seed 2026 reaches 0.770 against 0.590 (+18.0 pt, [+9, +28], p = 9.1 × 10−4 ). The three seeds give +23.0, +20.0 and +18.0. The base checkpoint and the planner stay fixed, so the replicate covers the continuation only.

5.1. Parsing and structural match Question 1 asks where the artifacts fail, and whether a semantic score taken on the emitted parameters can see it. Table 6 reports the seven conditions on QAOA: parse lies between 0.790 and 0.970, full match between 0.400 and 0.870. Across both families the ranges widen to [0.55, 0.97] and [0.02, 0.87]. The extreme cell is on VQE (Section 5.4): one-stage parses 0.810 of those artifacts and matches all six fields on 0.020. A score taken on the parameters as emitted carries none of that spread. With every symbol bound to the constant 0.1, and the reference circuit bound the same way, the median absolute energy error and the median distributional divergence are zero for every condition, including those at full match 0.400 and 0.870. The protocol is degenerate by construction; the zero medians are reported once, here, as the reason the semantic measures are computed after the shared refit rather than read off the emitted text (Section 5.5). Answer to Question 1. On QAOA the failures are structural rather than syntactic. On VQE the carded cells lose parse as well (0.550 against 0.810), while full match stays between 0.02 and 0.04 either way. The emittedparameter score ranks nothing. 5.2. Effect of the contract Question 2 is answered on QAOA by two paired comparisons of full match, both with n = 100, on the constructed set of Section 3.2. Eighty of 100 QAOA signatures appear in some training split. The 20 rows absent from every split are in Table 12, and that interval includes zero. Conditioned reaches 0.780 and one-stage 0.510: +27.0 pt, interval [+17, +37], McNemar p = 1.4 × 10−6 . That contrast changes the checkpoint as well as the prompt, so +27.0 is the pipeline a practitioner deploys. Matched-step continues the same ancestor for the same 638 steps on the plan-free corpus and receives a placeholder. It reaches 0.550. Against Conditioned the difference is 8

Table 6. Structural and semantic results on QAOA over the seven conditions, 100 instances per condition. Full match is the primary structural measure. Condition

SCR

PCS

Full

Skel.

Adhere

HQCR

One-stage Matched-step Equal-weight Shuffled Renamed Conditioned Oracle-contract

0.940 0.910 0.870 0.790 0.930 0.970 0.970

0.790 0.770 0.720 0.670 0.680 0.900 0.940

0.510 0.550 0.460 0.400 0.530 0.780 0.870

0.570 0.630 0.550 0.660 0.710 0.840 0.880

— — — — — 0.808 —

0.670 0.780 0.724 0.683 0.753 0.856 0.938

SCR is the parse rate and PCS the parameter-interface rate. Skeleton match is defined on QAOA only. Adherence is computed for Conditioned (n = 99: one artifact could not be extracted). Cells marked — are undefined. HQCR uses converged rows only, n = 94, 91, 87, 79, 93, 97, 97 in row order. Order equals cover on this family (Table 13).

A length-matched random card keeps the predicted card’s length and character classes and replaces every alphanumeric character with a seeded random string. It parses 0.730, closes the interface on 0.430, and reaches full match 0.270 against 0.780: +51.0 pt, 52 discordant rows against 1, interval [+41, +61], p = 1.2 × 10−14 . It is longer than the placeholder on the same weights (0.460), so the length comparison is conservative. On VQE the same control reaches 0.020 against 0.030. The gain is not uniform. Splitting the 100 QAOA pairs by the length of the conditioned program, a split on a treatment outcome and therefore a mechanism check, gives +5.6 pt, +41.9 pt and +24.2 pt in the shortest, middle, and longest thirds against Matched-step. Matched artifacts are shorter (median 104 lines against 146). A fixed-weight probe continues Qwen2.5-7B-Instruct on the same 9,604-row plan-conditioned corpus for 638 steps from a fresh LoRA. It has no base stage, no one-stage arm, and no plan-free continuation, so it is a second scale, not a replication. On QAOA, reference card, predicted card, and placeholder give full match 0.920, 0.770 and 0.250. Adherence is 0.930 and 0.808, the latter equal to the 3B arm. On the 42 QAOA rows where both conditions close the interface, the predicted card reaches 0.952 against 0.548 for the placeholder. That check conditions on an outcome. The variational half does not move (Section 5.4). Pooled over 200 instances, Conditioned against Matched-step is +12.0 pt [+7.0, +17.5]. Table 7 collects the paired tests. Answer to Question 2. The same-step gain is +23.0 pt and the pipeline gain is +27.0 pt. Only the equal-weight contrast (+32.0 pt) holds the checkpoint fixed. The estimate is conditional on the constructed QAOA set.

Oracle-contract vs Matched-step

+32.0

Conditioned vs Equal-weight

+32.0

Conditioned vs Matched-step Conditioned vs One-stage

+23.0 +27.0

0 20 40 difference in full match (points) Fig. 3. Performance and attribution contrasts, plotted from the numbers of Table 7. Points are paired differences on QAOA full match, bars are paired bootstrap intervals, and the vertical line is no difference.

Table 8 is the two-by-two the design was built for. On weights trained with a card the contract is worth +32.0 pt (35/3, [+22, +42], p = 6.7 × 10−8 ). On weights trained without one the same card costs 13.0 pt against their own placeholder (8/21, [−23, −3], p = 0.024), and the reference contract costs 10.0 pt (8/18, [−20, +0], p = 0.076). The interaction, the difference of those simple effects, is +45.0 pt, interval [+36, +62], sign-flip p < 5 × 10−5 . The referencecontract cell points the same way, but its interval ends at zero, so the predicted-contract cell carries that half of the inference. The contrast establishes that the effect requires training on card-conditioned data. It does not separate which field the generator exploits. The conditioned corpus mixes reference, predicted, and placeholder cards in the shares 0.497, 0.302, and 0.201. The plan-free corpus contains none. Every row of both corpora carries the plan block, so the format is held fixed. A placeholder on conditioned weights (0.460) and a contract on plan-free weights (0.420–0.450) both fall below the one-stage baseline of 0.510. Neither training nor a formatted block substitutes for the pair. Names are load-bearing. A group-preserving renaming of the reference contract costs 17.0 pt on skeleton match (p = 7.6 × 10−5 ) and 34.0 pt on full match (p = 5.4 × 10−9 ), both against Oracle-contract. Breaking the cross-class grouping yields full match 0.400, below the placeholder at 0.460, but the paired comparison is −6.0 pt ([−19, +8], p = 0.47). The skeleton comparison on the same rows is +11.0 pt (p = 0.11). The sharper content evidence is the

5.3. What the gain depends on Question 3 asks which part of the gain survives once the checkpoint, the step budget, the formatted block, and the card content are separated. The shared checkpoint is not sufficient. Emptying the slot on the conditioned weights drops full match from 0.780 to 0.460 (+32.0 pt, [+22, +42], p = 6.7 × 10−8 ). The retrained pair gives +31.0 pt ([+20, +42], p = 3.4 × 10−7 ). The placeholder lies below one-stage (0.460 against 0.510) and is not a performance baseline. Extra steps are not sufficient either: Matched-step reaches 0.550. 9

Table 7. Paired comparisons on QAOA. Unless a row says otherwise, n = 100. Full match is the primary measure. Disc. counts the discordant pairs of the exact test. Comparison

Full match [interval]

Conditioned vs One-stage Conditioned vs Matched-step Conditioned vs Equal-weight Oracle-contract vs Matched-step Conditioned vs Oracle-contract Renamed vs Oracle-contract Shuffled vs Equal-weight Rule card vs Conditioned

+27.0 [+17, +37] +23.0 [+13, +33] +32.0 [+22, +42] +32.0 [+23, +41] −9.0 [−15, −4] −34.0 [−44, −24] −6.0 [−19, +8] −6.0 [−14, +2] −2.0 [−7, +3] +6.0 [+1, +12]

Executor vs Conditioned (predicted) Executor vs rule arm Replicate (seed 1337) Replicate (seed 2026)

+20.0 [+10, +30] +18.0 [+9, +28]

p

Skeleton [interval], p

Disc.

−7

30/3 27/4 35/3 32/0 0/9 2/36 21/27 6/12

0.69 0.070

— —

2/4 7/1

3.2 × 10−4 9.1 × 10−4

— —

25/5 23/5

−6

1.4 × 10 3.4 × 10−5 6.7 × 10−8 4.7 × 10−10 0.0039 5.4 × 10−9 0.47 0.24

+27.0 [+18, +37], 1.1 × 10 +21.0 [+13, +29], 5.7 × 10−6 +29.0 [+20, +39], 1.3 × 10−7 +25.0 [+17, +34], 6 × 10−8 −4.0 [−8, −1], 0.12 −17.0 [−25, −9], 7.6 × 10−5 +11.0 [−1, +23], 0.11 —

The two −6.0 rows are different contrasts: Shuffled versus Equal-weight is a content control on a corrupted reference card; Rule card versus Conditioned is the planner-free card against the learned planner’s card on the same conditioned weights (Section 5.3). Skeleton match is not scored for the rule or executor rows.

as executor failures. The executor is a QAOA baseline. A variational contract does not determine a gate sequence in the same way, and none is run there. Oracle-contract remains above Conditioned. Full match falls by 9.0 pt from 0.870 to 0.780 ([−15, −4], p = 0.0039). On the narrower conjunction the same comparison is −4.0 pt from four discordant pairs. The exact test gives p = 0.125, while the bootstrap interval [−8, −1] excludes zero. The residual is visible on the stricter predicate and is not established on the narrower one.

Table 8. Two-by-two on QAOA full match, n = 100. Rows are the second-stage training. Columns are the contract slot at inference. placeholder predicted reference with a card without a card

0.460 0.550

0.780 0.420

0.870 0.450

The lower-right cells hand a plan to weights that never saw a contract.

relevance control: a card of another instance is worth 0.130 against the placeholder’s 0.460 (Section 5.9). A deterministic rule can write a card of comparable value. It groups consecutive terms of equal coefficient into one block and names each block by its coefficient class. It reproduces the reference cover in 94 of 100 QAOA rows and the whole decision vector in 73. On the 34 rows where the rule and the reference disagree, the generator follows the card in 4, the reference in 17, and neither in 13. Handed to the conditioned generator, the rule card reaches full match 0.720 and is implemented in 0.700 of rows. The predicted contract reaches 0.780 (difference −6.0, [−14, +2], p = 0.24, discordant 6/12). The placeholder reaches 0.460 (+26.0, [+16, +36], p = 6.2×10−6 , 30/4) and the reference contract 0.870 (−15.0, [−22, −8], p = 6.1 × 10−5 ). The weights still have to know how to read a card. The rule was not scored on plan-free weights. Answer to Question 3. The contract has to be in the prompt, and the generator has to have been trained with one. Either alone leaves full match at or below the onestage baseline. A legal but unrelated plan is worse than none. On these 100 rows a rule that reads the term list is not distinguished from the planner. The executor reads the card and emits the fixed gate template of this corpus, one block per step, with no model in the loop. Fed the predicted card it reaches full match 0.760 and reproduces the card on 89 of the 92 it can emit. The generator reaches 0.780, a difference of −2.0 (2/4 discordant, [−7.0, +3.0], p = 0.69). Non-inferiority is not established: the interval admits a disadvantage of about 7 points. Fed the rule card the executor reaches 0.780 against 0.720 (+6.0, 7/1, [+1.0, +12.0], p = 0.070). The eight predicted cards that cannot be emitted are counted

5.4. The family on which the defined card fails Question 4 compares the two families on realizability, the reading that separates them in Table 4. On QAOA the conditioned generator carries the reference contract out (adherence 0.808) and full match rises by 27.0 points. On VQE adherence is 0.020, so the generator does not carry this card out. Table 9 shows the consequence. Full match sits between 0.020 and 0.040 for every 3B condition. Oraclecontract, which is given the reference contract, reaches full match 0.030 and parses only 0.550 of its outputs, the lowest parse rate in the table. The card as defined here is rarely realized (adherence 0.020 under Conditioned). Supplying the reference card does not raise full match, so planner field accuracy alone is not a sufficient repair; whether a different field list—in particular one that carries a parameter budget—would change the outcome is untested and not separated by this design. The two semantic columns on this family are descriptive of the emitted artifacts, not a semantic return of the contract: they are computed from the artifacts that were produced and refitted under the shared budget, and with adherence at 0.020 no contract-side benefit is defined here. Only one energy comparison remains estimable on this family: whether VQE became worse. On the paired subset of n = 45, Oracle-contract against Matched-step on q-hit@0.2 is −8.9 pt, with interval [−22.2, +4.4] and p = 0.34. The interval excludes a gain of about 15 points and includes a degradation of about 22 points. The study draws no conclusion of benefit or of harm on VQE energy. What it does locate is the failed step. Two measurements are consistent 10

Table 9. The VQE family over the same seven conditions, 100 instances per condition. Skeleton match is undefined here and is omitted. Condition

SCR

PCS

SCR∧PCS

Full match

Adherence

q-hit@0.2

HQCR

One-stage Matched-step Equal-weight Shuffled Renamed Conditioned Oracle-contract

0.810 0.700 0.790 0.650 0.810 0.690 0.550

0.810 0.670 0.720 0.590 0.730 0.640 0.470

0.770 0.650 0.720 0.590 0.730 0.630 0.470

0.020 0.020 0.030 0.040 0.030 0.030 0.030

— — — — — 0.020 —

0.296 0.286 0.184 0.185 0.198 0.290 0.182

0.037 0.043 0.026 0.031 0.037 0.029 0.054

PCS is the parameter-interface rate. SCR∧PCS is the joint event. Adherence is computed for Conditioned. The reference card leaves full match at 0.030, so the miss is not only planner error.

0.550. The reading and the data come from the same two families, so this is not prospective validation. On the 7B probe the reference card reaches 0.210 on both predicates, while the predicted card reaches adherence 0.172 and full match 0.070 (Table 10): the model often carries out a wrong construction. A two-stage, plan-free continuation at 7B is untested. The evidence fits a generation-side failure under this field list, including the absent parameter budget, and the design does not separate those reasons. The contract defined here is insufficient for VQE on these conditions.

Table 10. The variational half of the 7B probe by card and predicate: the two rows answer different questions and are not interchangeable. Card given predicted reference

Full match

nfull

Adherence

nadh

0.070 0.210

100 100

0.172 0.210

99 100

Adherence is decision-level (motif, repetition, sharing and tail); full match adds the gate set and the arity and is scored against the reference construction. nfull = 100 in both rows. nadh = 99 on the predicted row because one artifact could not be extracted; the reference row is out of 100 on both predicates.

with an execution failure. Conditioned and Oracle-contract emit a median of 93 parameters against 42 in the reference circuits, and the contract carries no field for that budget. The corpus spans 17 ansatz families, of which the evaluation set represents two, and the motif field on the Oracle-contract condition is only 0.410. Whether that failure remains at the 7B scale is left open. On the 7B probe the reference contract lifts variational full match from 0.050 to 0.210 on 16 discordant pairs, while the predicted contract does not lift it (0.070 against 0.050, 5/3); the four variational cells are in Table 10. Two relations there are worth stating. First, on the reference card adherence and full match coincide in all 100 rows, because the card and the reference construction are one object. On the QAOA reference cell they coincide in 99 of 100 rows; the one row that differs reproduces the card’s construction and allocates its lines correctly, but references a parameter it never declares, so it breaks two interface invariants that full match checks and the card does not carry. A decision-level adherence measure is therefore not a weaker copy of full match: it is the same predicate minus the fields the contract does not specify, and this is the one place in the study where the difference shows. Second, on the predicted card the two predicates separate widely — 0.070 full match against 0.172 adherence — and every row in that gap is an artifact that executed the predicted card faithfully, which was itself the wrong construction. The probe was run at fixed weights and has no stage in which the planner is retrained, so it cannot say whether better planning would move the 0.070 toward the 0.210 the reference contract reaches. Answer to Question 4. On the 3B conditions the card is carried out on QAOA (adherence 0.808) and not on VQE (0.020). Full match moves only where it is carried out, and the reference card takes the VQE parse rate from 0.810 to

5.5. Semantic measures after a shared refit On QAOA the structural gain moves some semantic summaries and not others. On the primary measure, HQCR, the same-budget control is not separated. Every difference is paired on rows where both conditions converged, so it need not equal the gap between the rates in Table 6. The shared refit converges on 94, 91, 87, 79, 93, 97 and 97 of 100 rows, in the order of Table 6. The paired intersection is 90 on the two primary contrasts. Nonconverged rows are dropped, not imputed, so the HQCR claim is a complete-case claim. Scoring non-convergence as failure could move the interval. The deposited logs support that recompute. The structural contrasts use all 100 rows and do not depend on it. Against the same-budget control the distributional score moves by +8.9 pt, with interval [0.0, +17.8], p = 0.096, n = 90: the interval touches zero and the comparison is not established at this n. Against the one-stage baseline it is +17.6 pt, interval [+7.7, +27.5], p = 0.0015, n = 91, but that baseline differs in the checkpoint as well as in the prompt; that is why the same-budget row governs. The two-sided energy score moves the same way and separates both: +17.6 pt against one-stage on 16/0 discordant rows ([+9.9, +25.3], p = 3.1 × 10−5 ) and +10.0 pt against matched-step on 9/0 ([+4.4, +16.7], p = 0.0039). The onesided hit is thinner, +6.7 pt on 6/0 (p = 0.031). Against matched-step the distributional score moves least of the three. What these numbers do not show is a movement of the typical artifact. On the converged intersection the paired median difference in distance to the reference circuit’s fitted energy is 0.0000 between the reference and predicted contracts (n = 95: 7 rows better, 8 worse) and 11

0.0000 between the predicted contract and the matchedstep control (n = 85: 23 better, 17 worse). Nine of those 85 rows crossing the 0.2 threshold is a change in the tail, not in the median. On the measure this study designated primary the control is therefore not separated, while two secondary scores move in the tail. The structural result is kept structural: the gain is not credited with a semantic return in the abstract, the contributions or the conclusion. The one-sided hit is not the primary semantic measure, because against Oracle-contract it equals 1.000 for both conditions and cannot separate them; HQCR remains that measure. A second refit with two restarts and 100 iterations, instead of five and 250, depresses every arm. On QAOA the distributional difference between Conditioned and Matchedstep is +16.7 points there against +8.9 under the main protocol, and the energy difference is +12.2 against +10.0 (n = 90 in all four cells). Direction is stable and significance is not. At the main budget the primary measure does not separate the control (p = 0.096). At the smaller budget it does (+16.7, 18/3 discordant, [+7.8, +26.7], p = 0.0015). That budget sits below what COBYLA needs for the largest circuits, so the run is a stress test. The claim is the one the main refit budget supports. VQE stays in the same low band under both protocols, −5.8 against −1.9 points on the two-sided score.

Table 11. The overlap vocabulary. “Training row” means a row of the corpus split that was used for fine-tuning; the signature is the corpus identifier of an instance, so two rows with the same signature are two renderings of one instance. Term

Definition used here

signature

the corpus identifier of an instance; equal signatures mean one instance rendered at different depths or qubit counts verbatim duplicate a test row whose prompt text and reference program both occur, character for character, in a training row instance variant a test row whose signature occurs in training but whose (family, qubits, depth) triple differs from any training row with that signature family-clean the signature occurs in no row of the QAOA or VQE training split instance-disjoint the signature occurs in no training row of any split

all rows (n=100)

+23.0

seen in training (n=80)

+22.5 +23.9

unseen in family split (n=46)

+25.0

unseen in all splits (n=20)

0

5.6. Overlap between training and evaluation This benchmark is not instance-disjoint. In the 400-row test split from which the evaluation set is drawn, 8.5% of rows copy a training row verbatim and 74.5% are variants of a training instance. The 200 evaluation rows are a selected subset of that split rather than a random draw (Section 3.2), which is why 102 of them are absent from every training split—far more than the parent split would contain. Table 12 repeats the primary matched-step comparison on QAOA under four overlap views and adds the pooled clean-strict cell over both families; Table 11 fixes the vocabulary, because a row can be an instance variant and still carry a signature unseen in training. The family-clean cell needs one qualifier before its pvalue is read. It holds the 46 QAOA rows whose signature is absent from the QAOA and VQE training splits used here, and the effect there is +23.9 pt with p = 0.0074. Those rows are not thereby unseen by every model that produced the corpus’s reference programs, since the corpus carries further splits that this study neither used nor names. On the 20 rows absent from every split the interval is [0, +50]; that interval, not the family-clean p-value, is the statement about instances outside training. Fig. 4 plots the same intervals. The views overlap: a signature absent from the family training split can still occur elsewhere, so the counts 46 and 80 are not complements of 100. What the four views show is narrower than the absence of memorization. The seen view gives +22.5 pt and the family-clean view +23.9 pt, so the estimated gain does not rise with instance exposure. That pattern rules out the

20 40 difference in full match (points)

Fig. 4. The primary contrast on the QAOA overlap views of Table 12, drawn to the same scale. The interval widens as the view becomes stricter because the number of rows falls; the point estimate does not rise with instance exposure, and the two widest views cannot exclude a difference of zero. The pooled clean-strict cell is tabulated but not drawn here.

simplest verbatim-recall account; a model that retained partial structure rather than whole instances could still produce it, and this design would not separate the two. Moderation is tested directly as the stratum effect minus the all-rows effect, with a paired bootstrap over rows (20,000 resamples). On the family-clean stratum the difference is +0.9 pt with interval [−10.0, +11.8]; on the instancedisjoint stratum it is +2.0 pt with [−22.8, +26.0]. Any inflation from overlap is therefore bounded near twelve points, and a deflation of about ten cannot be excluded. The second interval is wide enough to contain the estimate itself; that is a power limitation (Section 7.4), not evidence that the effect shrinks. Pooling the strictest definition over both families (n = 102) yields +5.9 pt [0.0, +11.8], p = 0.11: that cell is not a substitute for the QAOA primary. 5.7. Where the fields fail Aggregate rates say how often a condition is right; they do not say which field decides a miss. Table 13 breaks full match into the fields it conjoins. On QAOA, allocation is satisfied by at least 0.88 of artifacts in every condition and is rarely the deciding miss, while order equals cover in 12

Table 12. The primary contrast by benchmark overlap: Conditioned against Matched-step on full match. The first four rows are QAOA views from Table 11; the last row pools the strictest definition over both families. View All rows (QAOA) Seen in training Unseen in family split Unseen in all splits Pooled clean-strict (both families)

n

Matched-step

Conditioned

Difference [interval]

p

100 80 46 20 102

0.550 0.600 0.500 0.350 —

0.780 0.825 0.739 0.600 —

+23.0 [+13.0, +33.0] +22.5 [+12.5, +32.5] +23.9 [+8.7, +39.1] +25.0 [0.0, +50.0] +5.9 [0.0, +11.8]

3.4 × 10−5 1.2 × 10−4 0.0074 0.18 0.11

Terms are defined in Table 11. The strict QAOA interval is a power limitation (Section 7). Cell rates are omitted from the pooled row because the families are not pooled elsewhere. Difference of differences against the all-rows estimate: family-clean +0.9 pt [−10.0, +11.8]; instance-disjoint +2.0 pt [−22.8, +26.0] (20,000 resamples).

every arm because the reference order is the cover order; both remain checks in the sense of Table 5. The fields that move with the conditions are the coefficient slot (0.550 to 0.880) and inversion (0.670 to 0.940). Full match never exceeds the smaller of cover and slot. Renaming disturbs the slot without changing the grouping (0.660); shuffling breaks the grouping itself (0.610). The VQE block shows the opposite pathology. Arity stays at or above 0.99, the repetition count at or below 0.120, and the sharing pattern at or below 0.320. Motif and gate set remain low in every condition, including Oracle-contract, where the generator is handed the reference contract. At field resolution the failing fields sit in the emitted artifact even when the reference card is supplied, so planner field accuracy alone does not explain the miss; the design still does not separate that pattern from the absent parameter-budget field (Section 5.4). If the fields failed independently, full match would be the product of its marginals. The observed joint exceeds that product by a factor of 1.3 to 2.2 on QAOA and 10 to 40 on VQE. The gap rules out the reading in which per-field slips are independent and could be repaired one field at a time; it does not estimate which pairs move together.

the grouping of coefficients across symbols, which changes the parameterization rather than the spelling. Of the remaining 16%—allocation, order, or inversion alone—only 0.556 reach the reference distribution, so the group is not a set of equivalent artifacts that the measure wrongly rejects. A state-vector check on the same 188 rows (one uniform parameter draw per row, all-zero input, squared-overlap tolerance 10−6 ) puts a number on that decomposition: all 30 comparable cover-or-slot mismatches fall below the tolerance, three of the nine incidental mismatches do, and none of the 133 full matches is distinguished. An overlap below the tolerance refutes equality; an overlap of one does not establish it. No consumer that reads the construction is run (Section 7.1). 5.9. Is a legal contract enough? The controls so far either corrupt the reference contract or rename its symbols, and both still share its vocabulary. Condition A1 asks whether any concrete plan in the prompt would do: it hands the generator a shape-compatible but instance-irrelevant reference contract from another instance of the same family, qubit count, and depth. The donor is a deterministic function of the row key, so the condition does not depend on row order. An irrelevant but legal plan is worse than no plan at all: 0.130 against 0.460 for the placeholder on the same weights (−33.0 pt). The gain therefore sits in the relevance of the plan, not in its presence; giving these weights the right plan is worth +32.0 pt. A1 also breaks interface closure more often (0.400 against 0.720). Restricted to the rows on which both conditions close (n = 31), A1 still reaches only 0.323 against 0.613 (−29.0 pt), so the loss is not an artifact of the broken interface alone. Against the donor contract rather than the reference, cover and slot agreement are 0 of 100: the condition varies the applicability of a realized plan, not a uniform draw from the grammar of legal contracts (Section 7.5).

5.8. Does structural match imply a better artifact? A reference-based measure invites an obvious question: does it track anything the consumer of the artifact cares about? The per-row data answer part of it for the two arms on which structural and semantic outcomes both exist: Conditioned and Matched-step, on QAOA, on the 188 rows whose shared refit converged. The first row of the table that follows is definitional, and the reading below is built on the second. The first row of Table 14 is a consistency check on the shared refit, not a finding: an artifact whose construction already matches can reproduce the reference distribution once both circuits are fitted. The informative row is the second. Of the 55 artifacts that miss the construction, 0.891 still come within 0.2 of the reference energy on the one-sided criterion, while agreement with the reference distribution falls to 0.400. That separation holds inside each arm as well (Conditioned 0.316 against 0.987; Matched-step 0.444 against 1.000). Table 15 answers the objection that a reference construction is only one of several correct ones. Most mismatches are not of that kind: 84% change either the covered terms or

6. Discussion 6.1. Implications for reporting and practice The measurements support two reporting conventions. Interface success and structural match should be separate columns: parse rates between 0.55 and 0.97 do not recover full-match rates between 0.02 and 0.87. Where symbol 13

Table 13. Field-level pass rates: the fraction of artifacts whose field agrees with the reference construction, per condition and per family. QAOA: fields of the reference construction, 100 instances per condition Condition cover slot alloc. order inv. full match One-stage Matched-step Equal-weight Shuffled Renamed Conditioned Oracle-contract

0.780 0.850 0.740 0.860 0.830 0.920 0.930

0.570 0.630 0.550 0.610 0.660 0.840 0.880

0.950 0.970 0.950 0.880 0.900 0.950 0.950

0.780 0.850 0.740 0.860 0.830 0.920 0.930

0.790 0.770 0.720 0.670 0.680 0.900 0.940

0.510 0.550 0.460 0.400 0.530 0.780 0.870

VQE: fields of the reference construction, 100 instances per condition Condition motif repeat share tail gate set full match One-stage Matched-step Equal-weight Shuffled Renamed Conditioned Oracle-contract

0.200 0.250 0.270 0.430 0.270 0.200 0.410

0.070 0.120 0.110 0.090 0.110 0.100 0.090

0.260 0.270 0.300 0.300 0.300 0.320 0.280

0.660 0.570 0.610 0.540 0.590 0.560 0.560

0.230 0.280 0.320 0.500 0.320 0.210 0.500

0.020 0.020 0.030 0.040 0.030 0.030 0.030

Full match requires every field, so it lies below each column. Arity is omitted: it is 1.000 except 0.990 on the two one-stage-trained conditions. Order equals cover in every QAOA row. Allocation stays above 0.88 in every condition. Table 14. Row-level association between structural match and semantic outcome, for Conditioned and Matched-step on QAOA, over the rows on which the shared refit converged (n = 188). Group Full match No full match

n 133 55

HQCR 0.992 0.400

q-hit@0.2

Table 16. Relevance control on QAOA (n = 100): the conditioned and the reference condition against A1, which is given a shape-compatible but instance-irrelevant reference contract. Comparison

SREV (2s)

1.000 0.891

1.000 0.764

The first row is definitional and is a consistency check on the refit, not evidence: an artifact whose construction matches can reproduce the reference distribution by construction (Conditioned 0.987, Matched-step 1.000). The second row is the informative one, and it holds in each arm separately (Conditioned 0.316, Matched-step 0.444). Values are the fraction of rows on which the outcome holds. SREV is the two-sided score of Section 3.3: the fitted energy is within 0.2 of the reference energy in either direction. The one-sided hit rate is the least informative of the three, holding on 89% of the rows that miss the construction, whereas agreement with the reference distribution falls to 40%.

29% 55% 16%

Diff. [interval]

that consumer, the structural contrasts alone do not justify inserting a card. On this QAOA template the numbers support a shipping order. Fix the field list and the checker. Prefer a rule when one recovers the card. Compile the card when a fixed gate template exists, and call the generator when the card is not executable. On the 34 rows where the rule and the reference disagree, the generator follows the reference in 17 and the card in 4 (Section 5.3). Two stop rules follow. A card on weights that never saw one costs points. On VQE the reference card cuts parse from 0.810 to 0.550 without buying structural match (Table 9). An adherence pilot therefore precedes any deployment claim. A parameter-budget field is not shown to repair that family. Transfer still needs a host-specific walk (Section 6.3).

n share HQCR

covered terms (cover) 16 grouping of symbols (slot) 30 allocation, order or inversion only 9

B

A1 receives the reference contract of another instance of the same family, qubit count and depth, written in the surface form of Oracle-contract. A is the first condition of the pair, B the second, and values are full-match rates; differences are paired, in percentage points, with a row-level bootstrap interval. All three have exact McNemar p < 10−6 . Interface closure is reported beside the structural rates because a plan that does not fit the instance also disturbs the declaration list, which is a second route to failure.

Table 15. What differs in the artifacts that miss the reference construction. Conditioned and Matched-step, QAOA, rows on which the shared refit converged (n = 55 mismatches out of 188). The first two rows change the construction; the third changes only how the construction is annotated.

What differs

A

A1 vs placeholder (Equal-weight) 0.130 0.460 −33.0 [−44, −22] Conditioned vs A1 0.780 0.130 +65.0 [+55, +75] Oracle-contract vs A1 0.870 0.130 +74.0 [+65, +82]

0.312 0.400 0.556

names are arbitrary, skeleton match should sit beside the strict predicate. The renaming control costs 34.0 pt on full match and 17.0 pt on skeleton match. Contrasts should also carry the overlap strata of Section 5.6. The supported beneficiary is a host that reads the construction. For that host, Table 8 is the insertion rule: card content helps on weights trained with cards and costs points on plan-free weights, and seed 1337 replicates the direction. A host that thresholds fitted energy or a distributional score is a different consumer. Against matched-step the primary HQCR contrast is not separated, and the paired median energy gap is zero (Section 5.5; Table 14). For

6.2. Cost The checker is linear in the number of gates and calls no simulator. Inference is two forward passes. The shared refit costs about 36 s on QAOA and 3.4–4.2 min on VQE (Table 17). The dominant wall clock is the 638-step continuation (∼103 min), not the extra planner pass. Counted per matching QAOA artifact at 15 s of generation, wall clock 14

falls from 29 s (one-stage) to 19 s (conditioned) because more rows match, not because each generation is cheaper.

7.3. External validity Main generation uses one 3B checkpoint, greedy decoding, one planner, and one run per condition, with two further training seeds for the primary pair; those seeds retrain the continuation and hold the planner and the base checkpoint fixed. A 7B fixed-weight probe (Section 5.2) raises VQE adherence to 0.210 on the reference card and 0.172 on the predicted card, with predicted-card full match at 0.070 (Table 10). It does not replicate the two-stage comparison, and decoding, planner architecture, and pipeline are not varied. The contribution is a measurement framework and a design proposal for this pipeline, not a claim that contract-conditioned generation is universally effective. Two families are studied and are not pooled; evaluation covers two of seventeen VQE ansatz families; the overlap strata are part of the result rather than an afterthought. The QAOA evaluation set is dominated by the ≤ 8-qubit band (75 of 100 rows), with 21 rows at 9–12 qubits and 4 above 12; the family-level full-match contrast is therefore a small-circuit estimate unless a later reanalysis of the deposited logs shows otherwise.

6.3. What transfers The proposal is for embedded programs whose construction can be written as a short vector of decisions and checked by a walk. It has been examined on OpenQASM 3 and on two circuit families. Transfer to another DSL is a hypothesis, not a result. Another language would need its own corpus, coverage count, and failure rates before a contract there could be credited with a gain. VQE is the in-paper illustration: without realization, full match stays between 0.02 and 0.04. The field-list filters of Section 4.4 are a domaingeneral procedure, not a result tested outside this corpus. Reference-based structural match is an engineering property for a host that reads the construction, not a universal correctness criterion. The three readings in Section 4.2 were written after the fact. They separate these families by adherence. They do not predict gain elsewhere, and they do not promise a small field list on every corpus. Planner field accuracy is not a fourth reading (Section 5.3). A hierarchy of contracts is unanalyzed. The VQE result is consistent with a failing step below the contract.

7.4. Conclusion validity The instance-disjoint QAOA stratum has n = 20. The point estimate is +25.0 pt with interval [0, +50] and p = 0.18. The study reads that cell as underpowered, not as evidence of no effect, and does not replace it with either the QAOA all-rows estimate or the pooled clean-strict cell of Table 12 (n = 102, +5.9 pt [0.0, +11.8], p = 0.11). On VQE the interval [−22.2, +4.4] excludes a gain of about 15 points and admits a degradation of about 22; neither benefit nor harm is claimed. Overlap moderation is +0.9 pt [−10.0, +11.8] (Section 5.6). Semantic contrasts rest on the refit-converged subset (n = 91 and 90). Whether the effect holds on instances the generator has never seen is therefore not established here.

7. Threats to validity 7.1. Construct validity Full match records agreement with a reference construction. It does not record solution quality, and it is not acceptance by a downstream tool: no such tool is run here. Most mismatches differ as operators rather than as spelling (Section 5.8); of the nine that do not, four also miss the reference distribution (Table 15). Semantic measures bound the structural claim on QAOA only, where the shared refit is closed and adherence is high (Section 5.5). Emitted-parameter scores have median zero on every QAOA condition and are not used as outcomes.

7.5. The control that was not constructed One comparison is absent and is recorded so that the controls that were run are not expected to answer it: a contract drawn uniformly from the grammar of legal contracts, rather than from realized contracts in the evaluation corpus. Renaming, shuffle, and A1 all remain close to realized plans. The hand-written rule of Section 5.3 is likewise not a row of Table 6: it reconstructs the reference contract on QAOA, where consecutive equal-coefficient terms expose the grouping in the instance text, and the finding that the planner is not what carries the effect is established on that family only.

7.2. Internal validity Symbol names have no meaning in the circuit semantics, yet a group-preserving renaming still costs 17.0 pt on skeleton match. Both predicates are therefore reported. Refitting is a possible confounder because conditions emit different numbers of parameters; the budget is shared (five restarts, 250 iterations), and the VQE parameter-count gap is stated explicitly. Renaming and shuffle plans are deterministic functions of the reference contract and a fixed seed, recomputed by the evaluation scripts rather than stored as separate logs.

8. Conclusion A generated OpenQASM 3 fragment can pass its host and still miss the reference construction. Across seven conditions, parse rates lie between 0.55 and 0.97 and full match between 0.02 and 0.87. At the main refit budget 15

Table 17. Cost of the two signals. The evaluator column describes the refit-based score used for the semantic measures; the contract column describes the intermediate artifact used at inference. Both sets of figures were measured on one RTX 4090 D (24 GB) with bfloat16 weights and greedy decoding (Section 3.2). Quantity

Contract path

Evaluator path

Payload In the context before generation Check Check runs during decoding Inference Locates the failing step

76–1,489 chars yes

one scalar per artifact no

O(#gates) walk, no simulator no

36 s (QAOA), 3.4–4.2 min (VQE) no

two forward passes line level

one pass plus scoring no

The evaluator figures include the variational refit at the shared budget. A single forward pass is common to both paths. Neither path replaces the evaluator inside the decoding loop: the contract is written before generation and checked after it, and no contrast here is trained against a contract reward.

the primary distributional measure does not separate the same-step control, so the gain reported here is structural (Section 5.5). On the constructed QAOA set the card is worth +32.0 pt at fixed card-trained weights and costs 13.0 pt without that training. The training×slot interaction is +45.0 pt and is not a deployment contrast. The same-step gain is +23.0 pt. The one-stage gain is +27.0 pt and also changes the checkpoint. A rule card and a compiler of the predicted card are not separated from the learned pipeline. Eighty of 100 signatures appear in some training split, and the 20-row instance-disjoint interval includes zero. On VQE, adherence is 0.020 and full match stays between 0.02 and 0.04. The reference card leaves full match at 0.030 while cutting parse to 0.550, and the design does not separate a missing parameter budget from other generation failures. The 7B probe leaves that boundary open (Table 10). For a host that reads the construction, insert a card only into a generator trained to read it, prefer a rule or a compiler when the template is fixed, and pilot adherence before any deployment claim.

of the published article. The generative model evaluated in the study is named in Section 3.4; the assistance declared here was used for the manuscript, not for producing the artifacts. Appendix A. Contract grammar A contract is a header followed by one decision line per step. The grammar is small so that extraction and comparison are walks over the text. contract ::= header line+ header ::= "CARD" family "n=" int ["p=" int] line ::= index op set symbol op ::= "cost" | "lin" | "mixer" (* QAOA *) set ::= "[" int ("," int)* "]" symbol ::= "x" int

On VQE the same header is followed by exactly five lines, because a variational contract carries no symbol that a relabelling could act on. vqe

::= header motif repeat share tail gateset motif ::= "MOTIF rot=[" int ("," int)* "]" "ent=[" string ("," string)* "]" "full_pair=" ("0" | "1") repeat ::= "R " int share ::= "SHARE " mode mode ::= "per_gate" | "per_qubit" | "shared" tail ::= "TAIL " ("rot1q" | "none") gateset::= "GATESET " str ("," str)*

CRediT authorship contribution statement Haowen Guo: Conceptualization, Methodology, Software, Formal analysis, Writing – original draft. Yangzhao Huang: Data curation, Validation, Writing – review & editing. Declaration of competing interest

Both families use the same header and the same line index, which is what makes an index-wise comparison meaningful when the field predicates differ. The gateset and the arity are checked once per artifact rather than per line.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix B. Prompt templates

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

Two templates are used, and both are frozen across conditions. The generator receives the instance and a contract block. The planner receives the instance only. The only text that changes between conditions is the contract block: absent, a placeholder, a predicted contract, the reference contract, or a corruption of the reference contract.

During the preparation of this work the authors used Cursor to revise the wording of the manuscript. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the content 16

Funding

[system]

You are an expert at writing parameterized OpenQASM 3.0. [user] <instance text> [PLAN] <contract text> [planner] <instance text> Emit the construction contract for this instance.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Acknowledgments

The artifact repository contains the full strings, including the placeholder used by Equal-weight and Matched-step. A one-token change in that placeholder is a different condition, so the string is treated as data.

The authors thank colleagues who read earlier drafts for insisting that a structural predicate and a semantic score be reported separately. Section 3.3 is organized around that distinction.

Appendix C. Artifact layout

References

The deposit holds the extractor, the checker, both prompt templates, the per-row generations, the evaluation scripts, and the script that rebuilds every table from the logs. Table C.1 maps each display to those files. A reader checking one cell should start from the log of that condition: it carries the signature, the emitted text, the extracted contract, and the three structural predicates.

Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., Sutton, C., 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. doi:10.48550/arXiv.2108.07732. Barr, E.T., Harman, M., McMinn, P., Shahbaz, M., Yoo, S., 2015. The oracle problem in software testing: A survey. IEEE Trans. Softw. Eng. 41, 507–525. doi:10.1109/TSE.2014.2372785. Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde de Oliveira Pinto, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Petroski Such, F., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Hebgen Guss, W., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A.N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., Zaremba, W., 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. doi:10.48550/arXiv.2107.03374. Chen, X., Lin, M., Schärli, N., Zhou, D., 2024. Teaching large language models to self-debug, in: The Twelfth International Conference on Learning Representations, pp. 8746–8825. URL: https://proceedings. iclr.cc/paper_files/paper/2024/hash/2460396f2d0d421885997dd1612 ac56b-Abstract-Conference.html. (accessed 4 November 2026). Cross, A.W., Javadi-Abhari, A., Alexander, T., de Beaudrap, N., Bishop, L.S., Heidel, S., Ryan, C.A., Sivarajah, P., Smolin, J.A., Gambetta, J.M., Johnson, B.R., 2022. OpenQASM 3: A broader and deeper quantum assembly language. ACM Trans. Quantum Comput. 3, 12:1– 12:50. doi:10.1145/3505636. Delgado, D., Burgueño, L., Clarisó, R., 2026. A framework for assessing the capabilities of code generation of constraint domain-specific languages with large language models. J. Syst. Softw. 238, 112871. doi:10.1016/j.jss.2026.112871. Evtikhiev, M., Bogomolov, E., Sokolov, Y., Bryksin, T., 2023. Out of the BLEU: How should we assess quality of the code generation models? J. Syst. Softw. 203, 111741. doi:10.1016/j.jss.2023.111741. Farhi, E., Goldstone, J., Gutmann, S., 2014. A quantum approximate optimization algorithm. arXiv preprint arXiv:1411.4028. doi:10.48550 /arXiv.1411.4028. Guo, X., Wang, M., Zhao, J., 2025. QuanBench: Benchmarking quantum code generation with large language models, in: 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 2657–2669. doi:10.1109/ASE63991.2025.00218. Ho, M.K., Li, T.Y., 2026. A protocol-guided LLM agent for quantum program synthesis and execution. arXiv preprint arXiv:2609.20258. doi:10.48550/arXiv.2609.20258. Huang, J., Chen, X., Mishra, S., Zheng, H.S., Yu, A.W., Song, X., Zhou, D., 2024. Large language models cannot self-correct reasoning yet, in: The Twelfth International Conference on Learning Representations, pp. 32808–32824. URL: https://proceedings.iclr.cc/paper_files/paper /2024/hash/8b4add8b0aa8749d80a34ca5d941c355-Abstract-Confere nce.html. (accessed 4 November 2026). Jiang, X., Dong, Y., Wang, L., Fang, Z., Shang, Q., Li, G., Jin, Z., Jiao, W., 2024. Self-planning code generation with large language models. ACM Trans. Softw. Eng. Methodol. 33, 1–30. doi:10.1145/3672456. Article 182. Kamoi, R., Zhang, Y., Zhang, N., Han, J., Zhang, R., 2024. When can LLMs Actually correct their own mistakes? A critical survey of selfcorrection of LLMs. Trans. Assoc. Comput. Linguist. 12, 1417–1440. doi:10.1162/tacl_a_00713.

Table C.1. Reproduction map. Scripts are in code/part; logs and tables are in the artifact deposit. A row that lists two scripts reads the per-row log and then aggregates it. Output

Computed from

Script

Tables 6–9

arms/*.jsonl

part_arms_eval.py, _struct_fields.py seed_replication.py strat_homogeneity.py

Table 7 Table 12

arms/*_rows.jsonl keys200.jsonl, arms/*_rows.jsonl Table 13 STRUCT_FIELDS.json Table 14 arms/*_rows.jsonl, REFIT_V2_rows.jsonl Table 16 arms/A1_rows.jsonl Table 17 REFIT_V2.json, run logs Figs. 3 and 4 Tables 7 and 12 Fig. 1 arms/M1.jsonl, arms/P0.jsonl, gold_cards_test.jsonl

_struct_fields.py row_association.py seed_replication.py part_refit_v2.py make_figures.py none (excerpts printed from the logs)

Data availability The extractor, checker, prompt templates, per-row generations, evaluation scripts, and per-row logs are openly available as version v1 of University of Electronic Science and Technology of China et al. (2026) (https://doi.org/10 .5281/zenodo.23885644). From those logs a third party can rebuild every numbered table (Table C.1). The circuit corpus used for evaluation and for deriving the contract has no public identifier. Its license permits redistribution of derived artifacts but not of the raw circuits, which are not in the deposit. Reference constructions therefore cannot be regenerated from a public problem set. Derived indices and reference cards are included.

17

Liu, J., Xia, C.S., Wang, Y., Zhang, L., 2023. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation, in: Advances in Neural Information Processing Systems 36, pp. 21558–21572. doi:10.52202/075280-0943. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B.P., Hermann, K., Welleck, S., Yazdanbakhsh, A., Clark, P., 2023. Self-refine: Iterative refinement with self-feedback, in: Advances in Neural Information Processing Systems 36, pp. 46534–46594. doi:10.5 2202/075280-2019. Meyer, B., 1992. Applying “Design by Contract”. Computer 25, 40–51. doi:10.1109/2.161279. Olausson, T.X., Inala, J.P., Wang, C., Gao, J., Solar-Lezama, A., 2024. Is self-repair a silver bullet for code generation?, in: The Twelfth International Conference on Learning Representations, pp. 36545–36593. URL: https://proceedings.iclr.cc/paper_files/paper/2024/hash/9ddc1 41bdbf9d1db510cefff56c586ad-Abstract-Conference.html. (accessed 4 November 2026). Peruzzo, A., McClean, J., Shadbolt, P., Yung, M.H., Zhou, X.Q., Love, P.J., Aspuru-Guzik, A., O’Brien, J.L., 2014. A variational eigenvalue solver on a photonic quantum processor. Nat. Commun. 5, 4213. doi:10.1038/ncomms5213. Qwen Team, 2024a. Qwen2.5. Version 2.5 [software]. URL: https: //github.com/QwenLM/Qwen2.5. (accessed 4 November 2026). Qwen Team, 2024b. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. doi:10.48550/arXiv.2412.15115. Ralph, P., bin Ali, N., Baltes, S., Bianculli, D., Diaz, J., Dittrich, Y., Ernst, N., Felderer, M., Feldt, R., Filieri, A., de França, B.B.N., Furia, C.A., Gay, G., Gold, N., Graziotin, D., He, P., Hoda, R., Juristo, N., Kitchenham, B., Lenarduzzi, V., Martínez, J., Melegati, J., Mendez, D., Menzies, T., Molleri, J., Pfahl, D., Robbes, R., Russo, D., Saarimäki, N., Sarro, F., Taibi, D., Siegmund, J., Spinellis, D., Staron, M., Stol, K., Storey, M.A., Tamburri, D., Torchiano, M., Treude, C., Turhan, B., Wang, X., Vegas, S., 2021. Empirical standards for software engineering research. arXiv preprint arXiv:2010.03525. doi:10.48550/arXiv.2010.03525. Ratiu, D., Voelter, M., Pavletic, D., 2018. Automated testing of DSL implementations—experiences from building mbeddr. Softw. Qual. J. 26, 1483–1518. doi:10.1007/s11219-017-9390-6. Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., Ma, S., 2020. CodeBLEU: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297. doi:10 .48550/arXiv.2009.10297. Runeson, P., Höst, M., 2009. Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Eng. 14, 131–164. doi:10.1007/s10664-008-9102-8. Scholak, T., Schucher, N., Bahdanau, D., 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic. pp. 9895–9901. doi:10.18653/v1/2021.emnlp-main.779. Solar-Lezama, A., 2008. Program Synthesis by Sketching. Ph.D. thesis. University of California, Berkeley. URL: https://www2.eecs.berkeley .edu/Pubs/TechRpts/2008/EECS-2008-177.html. Technical Report UCB/EECS-2008-177 (accessed 4 November 2026). University of Electronic Science and Technology of China, Guo, H., Huang, Y., 2026. Artifact for: Construction Contracts for Generated Embedded-DSL Programs. Zenodo, version v1 [software]. URL: https: //doi.org/10.5281/zenodo.23885644, doi:10.5281/zenodo.23885644. published 2 November 2026. Wei, L., Wang, Y., Cao, C., Pang, L., Ling, H., 2026. AG-CoT: Verified algorithmic traces for LLM program synthesis on Clifford circuits. arXiv preprint arXiv:2609.33192. doi:10.48550/arXiv.2609.33192. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A., 2012. Experimentation in Software Engineering. Springer, Berlin. doi:10.1007/978-3-642-29044-2. Yu, C., Uotila, V., Deng, S., Wu, Q., Shi, T., Jiang, S., You, L., Zhao, B., 2025. QUASAR: Quantum assembly code generation using toolaugmented LLMs via agentic RL. arXiv preprint arXiv:2510.00967. doi:10.48550/arXiv.2510.00967. Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., Radev, D., 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium. pp. 3911– 3921. doi:10.18653/v1/D18-1425.

18

Related documents

• Large Language Models for Quantum Code Translation: An Exploratory Study#1457995
• Large Language Models for Quantum Code Translation: An Exploratory Study#1450864
• QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging#1062922
• Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models#1130381
• Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models#1108791