QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code

QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code Meriem Terki1 , Nouhaila Innan2,3 , Minghao Shao2 , Muhammad Kashif2,3 , Alberto Marchisio2,3,4 ∗ , Muhammad Shafique2,3 1 École Supérieure en Informatique de Sidi Bel Abbès, Algeria eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE 3 Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute, NYUAD, Abu Dhabi, UAE 4 SDU Microelectronics, Institute of Mechanical and Electrical Engineering, University of Southern Denmark, Odense, Denmark m.terki@esi-sba.dz, {nouhaila.innan, shao.minghao, muhammadkashif, muhammad.shafique}@nyu.edu, chisio@sdu.dk

arXiv:2610.13811v1 [cs.LG] 9 Oct 2026

2

Abstract—Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark’s verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker’s label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.

TABLE I: Two failure modes when generated tests are scored against the benchmark checker (Coffee Conundrum challenge). Mode

Test checks

Too weak

Entry-point checks only; no execution on official input Too strict Numerical GPT-5.4 [10] tolerance at semantic 10−4

DeepSeek V3.2 [9] syntactic

Probe scenario

Test

Checker

Oracle-wrong LLM output

Pass

Wrong

Human reference output

Fail

Correct

Legend: Red cells mark test verdicts that disagree with the checker; blue cells show oracle labels.

To be useful, a generated test must (i) execute without infrastructure failure, (ii) accept oracle-correct candidates, and (iii) reject oracle-wrong ones. QTestBench scores LLMgenerated pytest suites on all three counts against an independent benchmark oracle, never against the tests themselves, spanning PennyLane QHack [7] and Qiskit HumanEval [6]. A. Motivational Case Study

Consider the PennyLane Coffee Conundrum challenge and two ways an LLM-generated test can fail. The task asks for Index Terms—Quantum computing, large language models, software testing, code generation, benchmarking, Qiskit, PennyLane. a small quantum circuit whose execution function returns a numeric answer as a string; the benchmark provides two official input/output pairs and a built-in acceptance routine. In QTestBench, an LLM first writes a pytest module for this task. I. I NTRODUCTION We then inject candidate solutions, a trusted human reference Large language models (LLMs) now turn natural-language and LLM-generated alternatives, and ask two questions on the prompts into executable code, performing strongly on classical same input: (1) does the generated test pass? and (2) does the benchmarks such as HumanEval [1] and MBPP [2]. Developers benchmark checker label the candidate correct? The checker increasingly ask the same models to write the unit tests that is the independent ground truth; the generated test is what guard this code, yet a passing generated test is not necessarily we evaluate. Table I shows two real mismatches from our trustworthy: a suite that accepts incorrect code is worse than no evaluation, one per test generator on this same task; in each test at all [3]. The risk intensifies in quantum computing, where row the test module is valid Python, yet its verdict disagrees outputs are probabilistic, tolerances matter, and correctness is with the checker. In the first row, the syntactic suite never executes the often defined only through domain-specific checkers [4], [5]. Benchmarks such as Qiskit HumanEval [6], QHackBench [7], candidate on official inputs, so oracle-wrong LLM solutions and QuanBench [8] evaluate whether LLMs write correct pass every sub-test even though their circuits return incorrect quantum solutions; none measures whether they write tests that values. In the second row, the human reference passes the task separate correct from incorrect quantum code. We close this acceptance routine, yet the generated semantic suite rejects it: two numerically close outputs land just outside the test’s gap with QTestBench. tolerance. Fig. 1 lists the assertions behind each row; harness ∗ This work has been carried out while at NYUAD. boilerplate is omitted. Neither mismatch is visible when grading

Assertions behind Table I Task. Coffee Conundrum: candidates are loaded at runtime; official pairs are embedded in the generated suite; the benchmark checker is the task acceptance routine. (A) Too weak: DeepSeek V3.2 syntactic (row 1) Structure checks only; oracle-wrong output still yields Test=Pass, Checker=Wrong. 1 2 3 4

exec(sol, ns) assert "run" in ns and callable(ns["run"]) assert "check" in ns and callable(ns["check"]) # no run(’5’) -> Pass despite wrong circuit

(B) Too strict: GPT-5.4 semantic (row 2) Human reference passes the acceptance routine; strict allclose rejects numerically close outputs (Test=Fail, Checker=Correct). 1 2 3

out = ns["run"](’5’) # ’0.6725000000000004’ exp = ’0.6735’ assert np.allclose(float(out), float(exp), atol=1e-4)

Fig. 1: Illustrative generated-test assertions for the Coffee Conundrum challenge (Table I). Each snippet shows the discriminating checks; harness boilerplate is omitted.

Section II reviews related work; Section III describes the pipeline and evaluation protocol; Section IV reports QHack and Qiskit HumanEval results; Section V concludes. II. BACKGROUND AND R ELATED W ORK A. LLM Code Generation and Evaluation LLMs such as Codex [1] and StarCoder [14] turn naturallanguage prompts into executable code and are benchmarked on HumanEval [1] and MBPP [2]. Because judging code automatically is itself difficult, a parallel line of work studies how generated code should be scored. Text-similarity metrics such as CodeBLEU [15] compare program text, while execution-aware methods such as CodeScore [16] predict run-time success, and FormalEval [17] checks correctness without hand-written tests. Recent reviews note that these benchmarks are fragile and often disagree [18]. Separately, LLMs are increasingly used to author unit tests for the code they generate [3], but classical benchmarks do not score whether those tests reliably discriminate correct from incorrect implementations. B. LLM Quantum Code Generation

only the generated code; both surface only when grading the generated tests against the checker, and exposing exactly such mismatches is what QTestBench is built for.

Quantum programming introduces additional difficulty: outputs are probabilistic, domain-specific checkers define correctness, and tasks cover circuit construction, simulation, and hardware-oriented APIs. For IBM Qiskit, the Qiskit Code Assistant fine-tunes models for quantum programming [19], B. Research Challenges while Qiskit HumanEval provides curated tasks with reference Evaluating LLM-generated quantum tests raises four chal- solutions and simulator-based checks [6]. For PennyLane, lenges: (i) existing benchmarks grade solutions, not tests; QHackBench evaluates models on real QHack coding chal(ii) ground truth must come from an oracle independent of the lenges drawn from hackathon-style problem sets [7] and generated suite; (iii) naturally occurring wrong solutions are implemented using the PennyLane framework [20]. PennyLang too sparse for stable bug-detection estimates; and (iv) scores introduces a PennyLane-centric dataset and studies retrievalmust transfer across SDKs with different task formats and augmented quantum code generation [21], while PennyCoder execution models. develops lightweight domain-specific models fine-tuned for PennyLane code generation [22]. PennySynth further uses retrieval-driven data synthesis and code-aware retrieval to C. Our Contributions improve generation on QHack challenges [23]. Beyond SDKWe introduce QTestBench, a framework for measuring LLM- specific studies, QuanBench covers broader quantum programwritten quantum test quality. Our contributions are: ming tasks and reports low accuracy for current models [8]; • Oracle-grounded test evaluation. QTestBench generates QCoder incorporates simulator feedback [24], and multi-agent syntactic, semantic, and behavioral pytest modules per pipelines iteratively repair incorrect solutions [25]. These task and scores them against benchmark checkers that studies share a common evaluation target: whether a model label each candidate correct, wrong, or unknown. can generate correct quantum code. However, none evaluates • Discrimination metrics. We define executability, cor- whether generated test suites can reliably distinguish correct rectness acceptance, bug-detection rejection, test quality, from incorrect quantum programs across multiple SDKs. checker-input coverage, and composite quality for quantum test suites (Section III-C). C. Trusting LLM-Generated Artifacts • Three validation cohorts. Configurations A–C (Section III-D) isolate executability and acceptance on trusted Studies that inspect LLM-generated code find that high references, discrimination against oracle-labeled LLM benchmark accuracy can still hide defects, so accuracy alone solutions, and calibrated rejection via synthetic mutants. is not a reliable quality signal [26], [27]. Generated artifacts • Cross-SDK evaluation. We benchmark Claude should be checked against an independent source of truth. This Opus 4.6 [11], DeepSeek V3.2 [9], Gemini 3 Pro [12], caution applies equally to generated tests: a suite that passes GPT-5.4 [10], and Qwen3-Coder [13] on eight QHack broken code is actively misleading. QTestBench therefore never challenges and a fixed ten-task Qiskit HumanEval scope, uses generated tests as ground truth; benchmark checkers label using the same metrics on PennyLane and Qiskit. every candidate solution.

D. Test Oracles and Mutation Testing Two classical software-testing ideas underpin QTestBench. A test oracle decides whether an output is correct; oracle selection is a long-standing challenge [28], especially for quantum code with probabilistic outputs and numerical tolerances. Mutation testing inserts faults into correct programs and scores suites by mutant detection [29]. QTestBench uses each benchmark’s official checker as an independent oracle and oracle-verified synthetic mutants when natural wrong solutions are too sparse for stable bug-detection estimates. III. M ETHODOLOGY QTestBench evaluates LLM performance in quantum test generation through an automated benchmarking framework for PennyLane QHack challenges [7] and Qiskit HumanEval tasks [6]. Our methodology separates test generation (Stage 1) from oracle-grounded validation (Stage 2): an LLM first writes pytest modules; the framework then injects candidate solutions, runs those modules, and compares outcomes against independent benchmark checkers. Section III-C defines discrimination metrics; Section III-D specifies three validation cohorts; Stage 2 applies both within the pipeline in Fig. 2 and Algorithm 1. The same workflow applies to both SDKs; only task materialization and oracle execution differ. A. Benchmark Tasks and Inputs To ensure consistency across SDKs, every evaluable task is materialized into the same three inputs (Fig. 2): • Task description: problem statement and entry-point stub that define the callable interface a solution must implement. • Official test cases: published input/output pairs that ground semantic and behavioral assertions. Stage 1 embeds these pairs in generated suites; the reference solution is withheld from test-generation prompts. • Official checker (oracle): benchmark logic that labels any candidate correct, wrong, or unknown. Stage 2 uses the checker only to label the solution cohort; generated tests never serve as ground truth. QHack (PennyLane). We use eight challenges from the QHack framework bundle, each with a template notebook, human reference submissions, and a task-specific acceptance routine. The framework extracts prompts, stubs, and official I/O pairs into a uniform layout shared with Stage 1 prompts and Stage 2 oracle execution. Qiskit HumanEval. Curated task records are converted into executable benchmark units (prompt, stub, reference solution, and published checker). Oracle execution invokes the required entry-point function and applies the benchmark acceptance logic. All reported Qiskit results use a fixed ten-task evaluation scope chosen before any test generation, so every model and configuration is compared on the same tasks. B. Automated Test Generation (Stage 1) To ensure reproducibility, Stage 1 follows the same four-step sequence for each test type; only the type-specific instructions in step 2 change (Fig. 2):

TABLE II: Generated test types in Stage 1: scope and oracle interaction. Type

Question

What runs

Ground truth used

Syntactic

Valid structure?

Semantic

Correct outputs?

Parse/load candidate; no circuit execution Candidate execution on official cases run() then candidate check()

Required entry points only Embedded official I/O values Candidate’s embedded check()

Behavioral Passes harness rules?

Legend: Row shading groups the three Stage 1 test types (see Fig. 3). Behavioral suites call the candidate’s template-local check() helper (not the benchmark’s official Stage 2 oracle).

1) Prepare task. Assemble the problem statement, entrypoint stub, and official input/output pairs. The reference solution is excluded from the prompt so the model cannot copy correct answers into assertions. 2) Build prompt. Combine task context with harness rules (tests must load injected candidate code at runtime) and instructions for syntactic, semantic, or behavioral checking. 3) Query the LLM. Issue one request per test type at temperature 0.8 with no repair or re-prompt loop. 4) Normalize and ground. Strip markdown fences, validate Python syntax, prepend a file header, and embed official input/output pairs so semantic and behavioral suites compare against benchmark values. Three modules are saved per (task, test-generator model): Syntactic: valid Python, harness load, required callable entry points; no circuit execution. • Semantic: candidate execution on official cases with numerical comparison at atol = 10−4 . • Behavioral: on QHack, call the candidate’s template-local run()/check() helpers (harness acceptance, not the official checker); on Qiskit, invoke the injected entry point end-to-end. •

Table II and Fig. 3 summarize how each type interacts with the oracle on the Rainy Days forest-parity challenge.

C. Discrimination Metrics Let T be a generated test suite and let a validation cohort denote the set of candidate solutions evaluated under one configuration (Section III-D). For that cohort, Sc is the set of oracle-correct candidates (the trusted baseline plus any probe the oracle labels correct) and Sw is the set of oracle-wrong candidates. Candidates labeled unknown are excluded from both pools. A suite accepts a candidate when all test functions pass; it rejects when any function fails or errors. The executability score (ES) is a binary indicator of whether the suite runs without infrastructure failure on the trustedcorrect baseline: ( 1, if T executes on baseline without failure, ES(T ) = 0, otherwise. (1)

INPUTS

Task description Problem statement Function template Official benchmark specification

STAGE 1: Unit test generation Steps 1-4 run once per type: syntactic semantic

→

1

Prepare task

Build prompt

2

Task description Function template Official benchmark specification

→

Input output pairs that define ground truth

Query LLM

Runtime harness rules

One request per type

Type-specific instructions

Single pass (no repair loop)

4 Normalize and ground

Normalize output Inject benchmark context Validate syntax

Output: one file per type, same pipeline, only instructions change Syntactic Valid code with required functions?

Official test cases

3

→ behavioral

Semantic Correct outputs on test inputs?

✓ Parses as valid Python ✓ Loads via runtime harness ✓ Entry point is callable ✓ No quantum execution needed

✓ Runs on official inputs ✓ Outputs match expected values ✓ Correct shape, type, properties ✓ Tolerant numeric comparison

Behavioral Passes all task validation rules?

✓ Full run through task interface ✓ Candidate run() + local check() ✓ Prompt: edge cases + fixed seeds (stochastic)

STAGE 2: Validation and scoring Four steps applied for each configuration 1 Build solution cohort

Human solutions

Official checker (oracle) Labels solutions : correct, wrong, unknown

LLM generated solutions Synthetic wrong solutions

2 Label with oracle

Official checker runs independently to label all the solutions

3

Execute tests

4

Score and report

Map outcomes to metrics

Inject candidate Run pytest suite Record pass/fail/ error

Tag results by configuration

Three configurations: one per study run, not a sequence

Config B: Checker labeled LLM cohort Reference + LLM generated solutions with oracle labels

Config A: Trusted correct only Reference solution only

✓ Tests accept known-correct code ✓ Pipeline runs without errors

✓ Tests align with benchmark truth ✓ Accept correct and reject wrong

Config C: Synthetic incorrect calibration Reference + oracle-verified wrong solutions

✓ Stable rejection signal ✓ Used when wrong probes are scarce

Fig. 2: QTestBench pipeline. Inputs: task description, official cases, and benchmark checker (oracle). Stage 1 generates syntactic, semantic, and behavioral pytest modules (one LLM call per type, temperature 0.8, no repair loop). Stage 2 builds a solution cohort, labels probes with the oracle, executes generated tests via runtime solution injection, and reports the metrics in Section III-C under Configuration A (trusted correct only), Configuration B (checker-labeled LLM cohort), and Configuration C (synthetic incorrect calibration; Section III-D).

The correctness score (CS) is the fraction of oracle-correct candidates accepted by the generated tests: CS(T ) =

#{s ∈ Sc : T accepts s} . |Sc |

(2)

D. Validation Configurations

Stage 2 is repeated under three cohort compositions (Fig. 2). We refer to them as Configuration A, Configuration B, and The bug-detection score (BDS) is the fraction of oracle- Configuration C (Config A–C in tables). Each configuration wrong candidates rejected by the generated tests, defined only answers a distinct diagnostic question while applying the when |Sw | > 0; otherwise BDS is n/a: metrics in Section III-C: #{s ∈ Sw : T rejects s} BDS(T ) = when |Sw | > 0. (3) • Configuration A (trusted-reference cohort). Only the |Sw | canonical reference solution is probed. This isolates The test quality score (TQS) combines discrimination executability and acceptance of known-correct code; BDS components: is undefined and TQS = ES × CS.  • Configuration B (natural LLM solution cohort). The  ES × CS, Config. A,  reference plus oracle-labeled solutions from LLM code (4) TQS(T ) = ES × CS × BDS, |Sw | > 0,  generators run independently of test generation. This  n/a, |Sw | = 0 (B/C). measures discrimination against naturally occurring wrong Under Configuration A, BDS is n/a and TQS = ES × CS. implementations; TQS = ES × CS × BDS when wrong Under Configurations B and C, reporting BDS or TQS as zero probes exist. When |Sw | = 0, BDS, TQS, and CQ are when |Sw | = 0 would incorrectly penalize suites that accept n/a. all available correct probes but have no wrong probes to reject; • Configuration C (synthetic mutant cohort). The referBDS, TQS, and CQ are therefore n/a whenever the cohort ence plus oracle-verified mutants of the reference, retained contains no oracle-wrong candidate. We also report checkeronly when the oracle confirms they are wrong (synthetic input coverage (Cin ), the fraction of official checker literals incorrect calibration in Fig. 2). This supplies calibrated present√ in the generated test body, and composite quality (CQ), negatives when Configuration B lacks wrong probes per CQ = TQS × Cin when TQS is defined. task.

Stage 1 assertion patterns (Rainy Days challenge) Task. Claude Opus 4.6 [11] generator; parity of forest runs on eight-qubit inputs. Only discriminating assertion bodies are shown. Syntactic: valid structure and entry points 1 2 3 4

tree = ast.parse(sol) assert "run" in funcs and "check" in funcs exec(sol, ns) assert callable(ns["run"]) and callable(ns["check"])

Semantic: correct outputs on official inputs 1 2 3 4

exec(sol, ns) for inp, exp in TEST_CASES: out = json.loads(ns["run"](inp)) assert np.allclose(out, json.loads(exp), atol=1e-4)

Behavioral: candidate harness check() (not the oracle) 1 2

exec(sol, ns) for inp, exp in TEST_CASES: out = ns["run"](inp) ns["check"](out, exp) # solution-local helper, not Stage-2 oracle

Algorithm 1 Unit Test Execution Harness (score aggregation follows Section III-C). Require: Generated test file F , candidate pool P = Sc ∪ Sw ∪ Su , trusted baseline s0 ∈ Sc Ensure: Per-candidate run records R 1: R ← ∅ 2: Inject s0 into harness namespace; run pytest on F with timeout 3: if run fails with infrastructure error then 4: Mark F as non-executable; set ES(F ) ← 0 5: return R 6: end if 7: Set ES(F ) ← 1 8: for each candidate s ∈ P do 9: Write s to a temporary module; inject candidate source and entry point into the harness namespace 10: Execute pytest on F with timeout τ 11: Parse passed / failed / error counts from pytest output 12: Classify outcome: ACCEPT if all tests pass, R EJECT otherwise 13: R ← R ∪ {(F, s, outcome, trace)} 14: end for 15: Aggregate R by test type, task, and model 16: return R

0.8 (one call per type; no repair). Configuration B uses their separately generated solutions as probes. Prompts. A fixed system prompt and type-specific user prompts request self-contained pytest modules with at least Fig. 3: Core assertion patterns for the three generated test types (Table II), exemplified on the Rainy Days challenge. three unit tests, runtime loading of the candidate (never the reference solution), and atol = 10−4 . Qiskit prompts also name the required entry point. Syntactic prompts request structure E. Validation and Scoring (Stage 2) Stage 2 applies Section III-D under the metrics in Sec- checks; semantic prompts request execution on official or tion III-C (Algorithm 1). For each configuration the pipeline specification-implied cases; behavioral prompts request end-to(i) assembles a solution cohort, (ii) labels non-baseline probes end harness checks (QHack: run()/check(); Qiskit: entrywith the benchmark checker, (iii) executes generated tests with point call). Mentions of edge cases or fixed seeds in prompts are runtime candidate injection, and (iv) aggregates scores by test soft guidance for stochastic tasks, not hard scoring requirements, and are not enforced by the harness. type, task, and test-generator model. Mutants and sandbox. Configuration C applies deterEvery cohort includes a trusted baseline s0 . On QHack, ministic mutations to the trusted reference and keeps only we select s0 before oracle-based scoring: for each challenge, mutants the official checker labels wrong: return None; every human submission file is executed against the three flip == to !=; remove measure_all(); replace the first Stage 1 test modules (syntactic, semantic, behavioral); we CX or Hadamard with an X gate; perturb numeric literals sum the pytest sub-tests each file passes and assign s0 to (1, 000 → 10, 100 → 1); and, on QHack only, force a local the submission with the highest total (ties broken by file check() failure. Up to twelve candidates are tried; the first order). This chooses a human reference that is compatible three wrong probes are kept. Probes run in isolated temporary with the generated suites; it does not define ground truth. directories with runtime candidate injection. On QHack, pytest Stage 2 then applies the benchmark’s official checker to every limits are 180 s per file and 60 s per test, with 90 s oracle probe, including s0 , labeling each correct, wrong, or unknown labeling per candidate; on Qiskit HumanEval the corresponding (excluded from Sc and Sw ). For each probe, the framework limits are 240 s, 120 s, and 120 s. injects candidate source into the pytest harness and records Table aggregation. Cells average bottom-up over suite types, pass/fail outcomes. The trusted baseline is always executed first: then generators, then tasks (Tables IV–VII). When |Sw | = 0, infrastructure failure on the baseline (import error, timeout, BDS, TQS, and CQ are n/a (never zero); model-level BDS/CQ or pytest crash) sets ES = 0 and zeros ES/CS for that suite use only tasks with |Sw | > 0. Qiskit Configuration B omits (BDS/TQS remain n/a when |Sw | = 0). TQS because TQS requires BDS and three scoped tasks have |Sw | = 0 (Case C, Fig. 4). F. Experimental Design Table III walks through ledger-to-score aggregation for one Task selection and models. We evaluate eight QHack Configuration B run; Section IV then reports benchmark means. challenges and ten Qiskit HumanEval tasks (0000–0003, 0019, 0020, 0024, 0034, 0035, 0063), fixed before generation as IV. R ESULTS AND D ISCUSSION a stratified easy/intermediate/hard mix so that every model and configuration uses the same tasks. Five LLMs (Claude This section asks whether each suite runs and accepts correct Opus 4.6 [11], DeepSeek V3.2 [9], Gemini 3 Pro [12], GPT- code (A), catches bugs models actually make (B), and catches 5.4 [10], Qwen3-Coder [13]) generate tests at temperature injected faults (C). Metrics follow Sections III-C and III-D. 3 4

TABLE III: Scoring ledger: Chalet Random Gate, Configuration B, Claude Opus 4.6 (test generator). Probe counts then CS, BDS, and TQS (avg. row over test types).

Realistic bugs are caught only moderately, and sometimes cannot be measured. Configuration B enriches the pool with LLM solutions (199 probes per run: 119 correct, 80 wrong). Suite ES Sc CS Sw BDS TQS Bug detection is middling (BDS = 0.45–0.60): Qwen3-Coder Syntactic 1 6/6 1.00 2/3 0.67 0.67 rejects the most wrong code (0.60) while Claude Opus 4.6 Semantic 1 6/6 1.00 3/3 1.00 1.00 accepts the most correct code (CS = 0.84), again the Behavioral 1 6/6 1.00 3/3 1.00 1.00 acceptance–rejection trade-off. Three of the ten tasks attract Avg. – – 1.00 – 0.89 0.89 no wrong solutions and report n/a for BDS, TQS, and CQ, so the cohort, not the suite, is limiting (Case C). Because Legend: Sc /Sw : oracle-labeled probe pass/tot and fail/tot counts; TQS = ES × CS × BDS. Row colors match Table II; TQS shading matches result tables. TQS is defined as ES × CS × BDS only when |Sw | > 0, Table VI omits the TQS column for Configuration B rather Across both benchmarks, models write tests that run, but are than reporting zeros. better at accepting correct code than at rejecting incorrect code. Synthetic faults restore measurability but expose low ceilings. Configuration C injects verified mutants into every task, so A. QHack Results BDS is always defined: Gemini 3 Pro then leads (TQS = 0.34), Table IV reports per-challenge means averaged over the five with Claude Opus 4.6 at 0.29. These ceilings sit well below QHack because many Qiskit checkers contain few extractable test generators; Table V aggregates the same run by model. Tests run reliably, but acceptance already separates models. literals (Deutsch–Jozsa has Cin = 0.00), pushing CQ toward Under Configuration A, executability is at least 0.79 for every zero even when acceptance is moderate. model and reaches 1.00 for Claude Opus 4.6, Gemini 3 Pro, and Illustrative case studies: metrics in practice GPT-5.4, so runtime injection is reliable. Acceptance already spreads the field: Gemini 3 Pro accepts almost all correct code Each panel reports probe verdicts (Test vs. benchmark Checker) and the (CS = TQS = 0.96), ahead of Claude Opus 4.6 (0.88) and resulting ES/CS/BDS/TQS scores under the stated configuration. GPT-5.4 (0.83), whereas DeepSeek V3.2 (0.42) and Qwen3Case A: Too weak (missed bugs; Configuration B) Coder (0.37) write tests so strict that they reject much of the Challenge Mach–Zehnder Cabin | Generator GPT-5.4 syntactic trusted reference before any bug is considered. The Coffee Candidate probe Test Checker Conundrum row (CS = 0.47; Table I, Fig. 1) shows that high executability alone is insufficient when semantic checks Oracle-wrong LLM solution Pass Wrong Human baseline Pass Correct disagree with the benchmark checker. Metrics: ES=1 CS=1 BDS=0 TQS=0 Bug detection, not acceptance, decides overall quality. Note: ES × CS = 1 masks zero bug detection; BDS is required. Configurations B and C add oracle-wrong probes, so BDS Case B: Too strict (false reject; Configuration B) contributes to TQS and reorders the ranking. Claude Opus 4.6 attains the highest test quality (TQS = 0.69 and 0.71) by Challenge Contextuality Dunes | Generator DeepSeek V3.2 semantic pairing strong acceptance with the best bug detection (BDS Candidate probe Test Checker = 0.80 and 0.83). Gemini 3 Pro keeps the highest acceptance Human reference Fail Correct (CS = 0.96) but detects fewer bugs (BDS = 0.67), so Oracle-wrong LLM solutions Fail Wrong its Configuration A lead does not survive once rejection Metrics: ES=1 CS=0 BDS=1 TQS=0 Note: High BDS alone is insufficient when CS rejects trusted code. matters. Configurations B and C agree closely for every model, indicating that synthetic mutants reproduce the discrimination Case C: Cohort design (GHZ construction; Config. B vs. C) signal of naturally occurring wrong solutions on QHack. Generator Claude Opus 4.6 | same suites, different probe pools (Table VI). High coverage does not imply good tests. Coffee embeds Config. Wrong probes ES CS BDS TQS almost all checker inputs (Cin = 0.97) yet reaches only TQS = B (LLM cohort) 0 (|Sw | = 0) 0.93 0.80 n/a n/a 0.13, so coverage without discrimination is not enough. C (synthetic) oracle-wrong mutants 0.93 0.13 0.83 0.03 Table V aggregates the same run by test-generator model. Note: With |Sw | = 0, BDS/TQS are n/a; Configuration C restores measurable rejection, but low CS caps TQS at 0.03.

B. Qiskit HumanEval Results Tables VI–VII report the fixed ten-task scope, organized around the same three questions. The protocol transfers, but tasks vary widely. Under Configuration A, harness injection carries over unchanged (four models reach ES = 1.00; Qwen3-Coder averages 0.93), and Claude Opus 4.6 again leads on acceptance (CS = TQS = 0.77). What changes is the spread across tasks: the Bell-state simulator reaches TQS = 0.87 (Cin = 0.93) while GHZ-state construction collapses to TQS = 0.13 at comparable coverage (Cin = 0.78), because semantic suites sometimes echo checker literals without accepting the reference.

Fig. 4: Three case studies on challenges other than the motivational example, connecting QTestBench metrics to concrete oracle vs. test outcomes.

C. Metric Case Studies The aggregate tables show that discrimination is the bottleneck; Fig. 4 illustrates why on challenges other than Section I. Case A: a perfect-looking suite can be worthless. On Mach– Zehnder Cabin, a GPT-5.4 [10] syntactic suite accepts every correct probe (ES = CS = 1) but also passes oracle-wrong

TABLE IV: QHack benchmark: mean per-challenge scores (averaged over five test generators). Configs A–C appear as column groups; see legend below for TQS/CQ shading. Config A

Config B

Config C

Challenge

ES

CS

TQS Cin

CQ ES

CS

BDS TQS Cin

CQ ES

CS

BDS TQS Cin

CQ

QSP swamp Chalet gate Coffee Contextuality Mach–Zehnder Rainy days Eigentracks Triple-H

0.93 0.87 1.00 0.93 0.93 1.00 1.00 0.93

0.73 0.80 0.47 0.80 0.80 0.40 0.87 0.67

0.73 0.80 0.47 0.80 0.80 0.40 0.87 0.67

0.47 0.93 0.53 0.87 0.45 1.00 0.40 0.93 0.47 0.93 0.31 1.00 0.53 1.00 0.40 0.93

0.73 0.80 0.47 0.80 0.80 0.44 0.87 0.73

0.73 0.64 0.67 0.60 0.60 0.93 0.71 0.60

0.47 0.93 0.53 0.87 0.13 1.00 0.40 0.93 0.47 0.93 0.40 1.00 0.53 1.00 0.40 0.93

0.73 0.80 0.47 0.80 0.80 0.40 0.87 0.67

0.73 0.67 0.67 0.60 0.60 0.93 0.73 0.60

0.47 0.53 0.13 0.40 0.47 0.31 0.53 0.40

≥0.50;

≥0.25;

0.50 0.60 0.97 0.47 0.53 0.50 0.53 0.53

≥0.75;

Legend: TQS and CQ cell shading (higher is better):

0.53 0.58 0.13 0.47 0.47 0.37 0.57 0.40

0.50 0.60 0.97 0.47 0.53 0.50 0.53 0.53

0.53 0.60 0.13 0.47 0.47 0.33 0.60 0.40

0.50 0.60 0.97 0.47 0.53 0.50 0.53 0.53

n/a = BDS/TQS/CQ undefined (|Sw | = 0).

<0.25;

TABLE V: QHack benchmark: mean scores per test-generator model (Configs A–C as column groups; best TQS per configuration in bold). Layout matches Table IV. Config A

Config B

Config C

Model

ES

CS

TQS Cin

CQ ES

CS

BDS TQS Cin

CQ ES

CS

BDS TQS Cin

CQ

Claude Opus 4.6 DeepSeek V3.2 Gemini 3 Pro GPT-5.4 Qwen3-Coder

1.00 0.96 1.00 1.00 0.79

0.88 0.42 0.96 0.83 0.37

0.88 0.42 0.96 0.83 0.37

0.57 1.00 0.28 0.96 0.67 1.00 0.58 1.00 0.12 0.79

0.88 0.47 0.96 0.84 0.37

0.80 0.75 0.67 0.71 0.50

0.69 0.26 0.63 0.55 0.08

0.55 1.00 0.27 0.96 0.63 1.00 0.56 1.00 0.08 0.79

0.88 0.42 0.96 0.83 0.37

0.83 0.75 0.67 0.71 0.50

0.71 0.25 0.63 0.54 0.08

0.53 0.25 0.63 0.54 0.08

≥0.50;

≥0.25;

0.69 0.35 0.71 0.71 0.44

≥0.75;

Legend: TQS and CQ cell shading (higher is better):

0.69 0.35 0.71 0.71 0.44

<0.25;

0.69 0.35 0.71 0.71 0.44

n/a = BDS/TQS/CQ undefined (|Sw | = 0).

TABLE VI: Qiskit HumanEval: mean per-task scores (ten-task scope; averaged over five test generators). Configuration A: ES, CS, TQS (TQS = ES × CS), Cin , CQ. Configuration B: ES, CS, BDS (n/a if no oracle-wrong probes), Cin , CQ. Configuration C: full metric set. Rows list short HumanEval task identifiers. Config A Task

ES

Config B

TQS Cin

CS

create_quantum 1.00 0.73 circuit run_bell_state 1.00 0.87 simulator create_bell 1.00 0.73 statevector create_ghz 0.93 0.13 transpile 1.00 0.40 circuit_maxopt transpile_ghz 1.00 0.27 customlayout dj_algorithm 1.00 0.47 run_jobs_on 1.00 0.47 batch run_circuit 0.93 0.47 with_dd_trex bb84_circuit 1.00 0.53 generate_key

Config C

BDS Cin

CQ ES

CS

0.00

n/a 1.00

0.73 1.00

0.73 0.00 0.00

0.87 0.93 0.83 1.00 0.87 0.73 0.93 0.60 1.00 0.93 0.62

0.56 0.93 0.59

0.73 0.93 0.71 1.00 0.73 0.73 0.93 0.47 1.00 0.73 0.00

0.00 0.93 0.00

0.13 0.78 0.11 0.93 0.80 0.40 0.00 0.00 1.00 0.45

0.13 0.83 0.40 0.00

0.03 0.78 0.04 0.00 0.00 0.00

0.27 0.00 0.00 1.00 0.27 0.87 0.00 0.00 1.00 0.27 0.00

0.00 0.00 0.00

0.47 0.00 0.00 1.00 0.79 0.88 0.00 0.00 1.00 0.47 0.77 0.47 0.65 0.30 1.00 0.47 0.60 0.65 0.13 1.00 0.47 0.53

0.23 0.00 0.00 0.00 0.65 0.00

0.47 0.00 0.00 0.93 0.47 0.66 0.00 0.00 0.93 0.47 0.67

0.20 0.00 0.00

0.53 0.00 0.00 1.00 0.51 0.70 0.00 0.00 1.00 0.53 0.70

0.23 0.00 0.00

CQ ES

CS

0.73 0.00 0.00 1.00 0.70

Legend: TQS and CQ cell shading (higher is better): ≥0.75; because TQS requires BDS and three tasks have |Sw | = 0.

≥0.50;

n/a

n/a n/a

≥0.25;

0.78 0.00

<0.25;

n/a 0.93 n/a 1.00

BDS TQS Cin

CQ

n/a = metric undefined (|Sw | = 0); Config B has no TQS column

TABLE VII: Qiskit HumanEval: mean scores per test-generator model over the scoped ten-task runs. Same column layout as Table VI; best TQS per configuration in bold (Config A and Config C). Config A

Config B

Config C

Model

ES

CS

TQS Cin

CQ ES

CS

BDS Cin

CQ ES

CS

BDS TQS Cin

CQ

Claude Opus 4.6 DeepSeek V3.2 Gemini 3 Pro GPT-5.4 Qwen3-Coder

1.00 1.00 1.00 1.00 0.93

0.77 0.30 0.70 0.47 0.30

0.77 0.30 0.70 0.47 0.30

0.28 1.00 0.13 1.00 0.24 1.00 0.16 1.00 0.16 0.93

0.84 0.42 0.82 0.57 0.37

0.65 0.73 0.75 0.72 0.85

0.27 1.00 0.07 1.00 0.19 1.00 0.14 1.00 0.18 0.93

0.77 0.33 0.70 0.47 0.30

0.46 0.52 0.54 0.48 0.56

0.07 0.03 0.09 0.05 0.08

Legend: TQS and CQ cell shading (higher is better): ≥0.75; because TQS requires BDS and three tasks have |Sw | = 0.

0.36 0.37 0.33 0.33 0.26

≥0.50;

≥0.25;

solutions never run on official inputs (BDS = 0), so TQS collapses to zero. Case B: the opposite failure is just as damaging. On

0.36 0.37 0.33 0.33 0.26

<0.25;

0.29 0.06 0.34 0.14 0.16

0.36 0.37 0.33 0.33 0.26

n/a = metric undefined (|Sw | = 0); Config B has no TQS column

Contextuality Dunes, a DeepSeek V3.2 [9] suite rejects even the correct human reference (BDS = 1, CS = 0, TQS = 0). Cases A and B are mirror images, so acceptance and rejection

must be reported separately. Case C: sometimes the cohort, not the suite, sets the score. On create_ghz, the same Claude Opus 4.6 suites give BDS/TQS n/a under Configuration B (|Sw | = 0) but BDS = 0.83 under Configuration C; low CS (0.13) caps TQS at 0.03, which is why Qiskit Configuration B omits a pooled TQS column. D. Cross-Framework Synthesis The same metric definitions transfer across PennyLane and Qiskit without changing the protocol. Executability rarely limits (ES ≥ 0.93 for most pairs), while CS and BDS separate generators. Claude Opus 4.6 and Gemini 3 Pro lead on QHack (TQS = 0.71 and 0.63 under Configuration C); Qiskit peaks lower (0.34, Gemini), reflecting sparser wrong probes and fewer extractable checker literals. Absolute TQS depends on probe composition, so BDS should be read with probe counts and as n/a when |Sw | = 0. On QHack, Configurations B and C agree closely, so synthetic mutants reproduce the natural cohort signal. On Qiskit, Configuration B leaves three tasks with |Sw | = 0 (no pooled TQS), and Configuration C restores defined BDS with only moderate TQS. High Cin alone is insufficient: Coffee reaches Cin = 0.97 with TQS = 0.13, the same too-weak pattern illustrated by Case A on Mach–Zehnder Cabin (high acceptance without bug detection). Even under Configuration C, singlepass generation remains limited; refinement with execution feedback [7] is a natural next step. Limitations. The evaluation is limited to a fixed temperature of 0.8, a single generation pass without repair, ten selected Qiskit tasks, and benchmark-specific checker oracles. V. C ONCLUSION We introduce QTestBench, a framework for evaluating LLMgenerated quantum test suites against independent benchmark oracles. It unifies PennyLane QHack and Qiskit HumanEval through a two-stage pipeline (single-pass generation of syntactic, semantic, and behavioral pytest modules, followed by validation under three cohort configurations) and reports executability, correctness acceptance, bug-detection rejection, test quality, checker-input coverage, and composite quality with consistent definitions across SDKs. Experiments with Claude Opus 4.6, DeepSeek V3.2, Gemini 3 Pro, GPT-5.4, and Qwen3-Coder show that runnable quantum tests are achievable (ES ≈ 1.00 on QHack for four models), but discrimination remains the primary bottleneck: leading generators reach only moderate bug detection (Claude Opus 4.6, TQS = 0.71 on QHack), and Qiskit scores are lower and more task-heterogeneous. By scoring tests against benchmark oracles, QTestBench addresses a dimension of quantum LLM evaluation not covered by solution-centric benchmarks. ACKNOWLEDGMENT This work was supported in part by the NYUAD Center for Quantum and Topological Systems (CQTS), funded by Tamkeen under the NYUAD Research Institute grant CG008, and the Center for Cyber Security (CCS), funded by Tamkeen under the NYUAD Research Institute Award G1104.

C ODE AVAILABILITY The source code and experiment configurations are available at https://github.com/eBrain4Everyone/QTestBench.

R EFERENCES [1] M. Chen et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [2] J. Austin et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [3] J. Wang et al., “Software testing with large language models: Survey, landscape, and vision,” IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 911–936, 2024. [4] A. Miranskyy and L. Zhang, “On testing quantum programs,” in IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), 2019, pp. 57–60. [5] A. Garcı́a de la Barrera et al., “Quantum software testing: State of the art,” Journal of Software: Evolution and Process, vol. 35, no. 4, p. e2419, 2023. [6] S. Vishwakarma et al., “Qiskit humaneval: An evaluation benchmark for quantum code generative models,” arXiv preprint arXiv:2406.14712, 2024. [7] A. Basit et al., “Qhackbench: Benchmarking large language models for quantum code generation using pennylane hackathon challenges,” in IEEE International Conference on Quantum Artificial Intelligence (QAI), 2025. [8] X. Guo, M. Wang, and J. Zhao, “Quanbench: Benchmarking quantum code generation with large language models,” arXiv preprint arXiv:2510.16779, 2025. [9] DeepSeek-AI, “Deepseek-v3.2: A strong mixture-of-experts language model,” https://github.com/deepseek-ai/DeepSeek-V3, 2025, accessed June 2026. [10] OpenAI, “Gpt-5.4,” https://openai.com/, 2026, accessed June 2026. [11] Anthropic, “Claude opus 4.6,” https://www.anthropic.com/claude/opus, 2026, accessed June 2026. [12] Google DeepMind, “Gemini 3 pro,” https://deepmind.google/models/ gemini/pro/, 2026, accessed June 2026. [13] Alibaba Cloud, “Qwen3-coder: Agentic coding in the world,” https: //github.com/QwenLM/Qwen3-Coder, 2025, accessed June 2026. [14] R. Li et al., “Starcoder: May the source be with you!” arXiv preprint arXiv:2305.06161, 2023. [15] S. Ren et al., “Codebleu: A method for automatic evaluation of code synthesis,” arXiv preprint arXiv:2009.10297, 2020. [16] Y. Dong et al., “Codescore: Evaluating code generation by learning code execution,” ACM Transactions on Software Engineering and Methodology, 2025. [17] S. Yang and Y. Yang, “Formaleval: A method for automatic evaluation of code generation via large language models,” in 2024 International Symposium on Electronic Design Automation (ISEDA), 2024, pp. 1–6. [18] D. G. Paul, H. Zhu, and I. Bayley, “Benchmarks and metrics for evaluations of code generation: A critical review,” arXiv preprint arXiv:2406.12655, 2024. [19] N. Dupuis et al., “Qiskit code assistant: Training llms for generating quantum computing code,” arXiv preprint arXiv:2405.19495, 2024. [20] V. Bergholm et al., “Pennylane: Automatic differentiation of hybrid quantum-classical computations,” arXiv preprint arXiv:1811.04968, 2018. [21] A. Basit et al., “A pennylane-centric dataset to enhance llm-based quantum code generation using rag,” arXiv preprint arXiv:2503.02497, 2025. [22] ——, “Pennycoder: Efficient domain-specific llms for pennylane-based quantum code generation,” in 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), vol. 2. IEEE, 2025, pp. 229–234. [23] M. Shao et al., “Pennysynth: Rag-driven data synthesis for automated quantum code generation,” arXiv preprint arXiv:2605.25572, 2026. [24] T. Mikuriya et al., “Qcoder benchmark: Bridging language generation and quantum hardware through simulator-based feedback,” in Proceedings of the 18th International Natural Language Generation Conference (INLG), 2025, pp. 43–55, arXiv:2510.26101. [25] C. Campbell et al., “Enhancing llm-based quantum code generation with multi-agent optimization and quantum error correction,” arXiv preprint arXiv:2506.20008, 2025. [26] A. Sabra, O. Schmitt, and J. Tyler, “Assessing the quality and security of ai-generated code: A quantitative analysis,” arXiv preprint arXiv:2410.02069, 2024. [27] E. L. Melin et al., “Precision or peril: Evaluating code quality from quantized large language models,” arXiv preprint arXiv:2408.01703, 2024. [28] E. T. Barr et al., “The oracle problem in software testing: A survey,” IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015. [29] Y. Jia and M. Harman, “An analysis and survey of the development of mutation testing,” IEEE Transactions on Software Engineering, vol. 37, no. 5, pp. 649–678, 2011.

Related documents

• QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code#1298307
• QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging#1062922
• Randomized and Diverse Input State Generation for Quantum Program Testing#155357
• Observing the Quantum Compiler through Automatic Experiment Tracking for Qiskit#429418
• QUTest: A Native Testing Framework for Quantum Programs#204843