arXiv CSopen access

Auditing Empirical Comparisons in Quantum Software

Auditing Empirical Comparisons in Quantum Software Boshuai Ye∗ , Peng Liang† , Maryam Tavassoli Sabzevari∗ , and Arif Ali Khan∗ ∗ University of Oulu, Finland

arXiv:2607.00516v1 [cs.SE] 1 Jul 2026

Email: {boshuai.ye, maryam.tavassolisabzevari, arif.khan}@oulu.fi † Wuhan University, China Email: [email protected]

Abstract—Empirical quantum-software papers often report comparative claims: one compiler produces smaller circuits, one optimizer is more reliable, or one ansatz achieves better solution quality. These “A beats B” conclusions are not properties of a tool alone; they can change with benchmark scope, circuit construction, compilation, sampling, backend and noise assumptions, optimizer choices, and resource budgets. Existing testing, benchmarking, and reproducibility methods help assess programs, tools, executions, platforms, and experiments, but they do not directly audit whether the reported comparison itself is supported by the evidence exposed in the paper and artifact. We present C LAIM S TAB -QC, a source-bounded framework for auditing empirical comparisons in quantum software. Given a reported comparison, the framework records the compared baselines, metric, relation, and admissible evidence; locks the comparison design before outcomes are computed; and reports either a scoped relation outcome or an explicit evidence boundary. For strict scalar-directional comparisons, the reported direction is classified as Sustained, Unresolved, or Reversed within the locked audit scope. We evaluate the framework on 455 comparative claims from 119 quantum-software papers. The central finding is a materialization gap: 175 claims can be represented for audit planning, 79 become scalar-directional planning records, and 53 yield lockable audit or diagnostic designs, but only 8 expose enough matched evidence to audit the original comparison without proxy reconstruction. These 8 records yield 2 Sustained, 4 Unresolved, and 2 Reversed outcomes. Controlled diagnostics over 24 benchmark-relevant quantum-workload comparisons further show that simpler checks can preserve apparent directions whose support weakens under locked audit designs. The results suggest that empirical quantumsoftware sources should report performance orderings together with the matched evidence needed to audit the scope of those orderings.

I. I NTRODUCTION Empirical quantum-software papers often report comparative claims: one compiler produces smaller circuits, one optimizer converges faster, one ansatz gives better solution quality, or one backend configuration is more efficient. These claims usually have the form “A beats B” under a stated metric. In quantum software engineering (QSE), however, such orderings are not properties of a technique alone. They are produced by a stack of frameworks, circuit libraries, compilers, optimizers, simulators, cloud platforms, and hardware backends [1]–[9]. The ordering between two baselines can change with benchmark scope, circuit construction, compilation, sampling, backend/noise assumptions, optimizer stochasticity, and resource budget [10],

[11]. This makes comparative evidence especially important in QSE: empirical quantum-software studies need to show not only who wins, but also what evidence supports that ordering and where the support ends. Rerunning an experiment and auditing a comparison answer different questions. A rerun checks whether one reported result can be reproduced under one set of conditions; a comparison audit checks whether the stated “A beats B” relation is supported by the evidence exposed in the source paper and artifact. Existing methods provide important pieces of this evidence: testing checks programs and platforms [12], benchmarking measures tools and backends [13], reproducibility infrastructure reruns experiments [14], and hardware-stability studies characterize device variation [15]. However, these studies leave one relation-level object unchecked: the reported comparison itself. C LAIM S TAB -QC therefore asks a different question: whether the stated comparative relation is supported by matched evidence exposed in the source paper or artifact. Baselines, backend settings, compiler seeds, shot budgets, and outcome rules must come from the source evidence; unsupported choices become evidence boundaries. Challenges: Auditing comparative relations in QSE is difficult for three reasons. (1) The reported winner can depend on the stack: benchmark scope, circuit construction, compilation choices, backend/noise assumptions, and resource policies can change which baseline appears better [10], [11], [13], [15]. (2) The audit must stay within the source evidence: baselines, devices, seeds, budgets, and outcome rules must come from the source; unsupported additions create a new comparison. (3) Different claims need different evidence: gate-count, runtime, aggregate-quality, and crossover claims require different paired data and decision rules. These challenges are especially important in QSE because quantum-software comparisons are sensitive to compilation choices, noise, stochastic optimization, and backend variation [15]–[18]. Thus, a source can expose executable artifacts while still leaving its reported comparison unauditable. To address these challenges, C LAIM S TAB -QC treats the reported comparative relation as the audit unit. It locks sourcesupported evidence and reports either a scoped relation-level outcome or an evidence boundary. The central constraint is the materialization gap: empirical QSE practice may report clear comparative relations without exposing the matched evidence

needed for proxy-free auditing. We apply C LAIM S TAB -QC to 455 comparative claims from 119 quantum-software papers. The study exposes a materialization funnel from reported comparisons to auditable evidence: many comparisons can be identified, represented, and planned, but far fewer expose enough matched evidence for proxy-free relation-level auditing. This paper makes three contributions: • Framework. We present C LAIM S TAB -QC, a QSE framework that makes reported empirical comparisons auditable through source-bounded comparison records, locked audit designs, relation-specific outcomes, and explicit evidence boundaries. • Empirical study. We apply the framework to 455 claims from 119 papers and reveal a materialization funnel from reported comparisons to auditable evidence, showing where proxy-free auditing is possible and where locked audit designs weaken apparent comparative support. • Reusable artifact. We release an audit-evidence package containing claim representations, audit records, locked traces, missing-evidence annotations, and reporting guidance for inspecting empirical quantum-software comparisons [19]. II. BACKGROUND Quantum computing and the quantum-software stack: Quantum programs are expressed as circuits of gates acting on qubits, executed on simulators or hardware backends, and read out by measurement [1], [4]. In practice, quantum software is evaluated through a layered stack of programming frameworks, circuit libraries, compilers, optimizers, simulators, cloud platforms, and hardware backends [7]–[9]. Empirical QSE comparisons therefore evaluate techniques within interacting stack layers, not isolated circuits or algorithms alone. Empirical comparisons in quantum software: Quantumsoftware papers frequently report that one method performs better, worse, or differently than another on a chosen metric. Examples include compiler or layout claims about two-qubit gates, controlled-NOT (CNOT)/SWAP count, or circuit depth [10], [13]; aggregate benchmarking claims such as geometricmean resource reductions across benchmark rows [20]; and variational-algorithm claims, including variational quantum eigensolver (VQE) and quantum approximate optimization algorithm (QAOA) claims, about solution quality or resource cost under a chosen ansatz, depth, or configuration [5], [6]. Across these cases, the claim is not only that an experiment ran, but that one baseline outperformed another under stated software-stack conditions. Those conditions define what the claim covers and what requires additional evidence. Stack-dependent sources of variation: An “A beats B” result can change when the software stack changes. Prior work shows that qubit mapping, routing, transpilation settings, benchmark suites, backend/noise behavior, and resource policies can affect circuit metrics, tool comparisons, and reported outcomes [10], [11], [15], [20]. Table I summarizes the audit-axis groups used by C LAIM S TAB -QC to inspect this stack. SCOPE selects the

TABLE I AUDIT- AXIS GROUPS INSPECTED BEFORE OUTCOME COMPUTATION . Axis

Example inspected dimensions

Audit role

SCOPE

problem, circuit, qubit, or graph size; benchmark row; curve or device point; instance distribution

selects compared evidence points

L1

circuit or ansatz construction, decomposition, routing, layout seed, optimization level, basis gates

construction and compilation choices

L2

shots per estimate, measurement seed, observable grouping, estimator precision, exact-vs-sampled expectation

sampling and estimation choices

L3

simulator or hardware backend, noise model, calibration, drift, readout and two-qubit error regimes

backend and noise assumptions

ALGO

optimizer seed, initialization, restart policy, optimizer family

algorithmic stochasticity

BUDGET iteration count, wall time, function evaluations, resource-policy total shot budget, stopping rule, timeout, choices repetitions

evidence points being compared; L1, L2, L3, ALGO, and BUDGET describe how those points are produced or measured. Unsupported or relation-changing conditions are recorded as evidence boundaries. Motivating example: A benchmarking study may report that Tket produces fewer two-qubit gates than Qiskit on selected circuits [13]. To check that ordering, the reader needs paired rows: for the same circuit, compare the Qiskit and Tket two-qubit-gate counts under the same metric definition, compiler versions, basis gates, optimization level, and seed policy. A Qiskit count from one circuit, optimization level, or seed policy cannot support a Tket comparison reported under another setting. If the paired rows and settings are exposed, the comparison can be audited within that scope; if they are missing, the missing row or setting becomes an evidence boundary. III. T HE C LAIM S TAB -QC F RAMEWORK This section defines the C LAIM S TAB -QC audit procedure. A. Framework Overview Figure 1 summarizes the workflow. A reported comparison is represented as a claim card, mapped to source-supported audit records, locked before outcome computation, and reported as either a relation-level outcome or an evidence boundary. Scalardirectional records receive Sustained, Unresolved, or Reversed outcomes; non-scalar records use the source-supported relation rule when that rule can be locked. Four safeguards keep the audit faithful to the source comparison: • Audit the reported comparison. Outcomes are scoped to audit records, not source-level correctness. • Lock before outcome computation. Designs, source rows, aggregate rules, device curves, or outcome rules are fixed before labels or statistics are computed.

1. Claim card

2. Audit record

⟨P, BA , BB , M, R⟩

comparison object

3. Locked design

4. Records

5. Outcome

6. Report

SCOPE,×,ENV before outcomes

matched cells rows / curves

S/U/R or source rule

outcome + evidence boundary

Fig. 1. The C LAIM S TAB -QC workflow. A reported comparison is represented as a claim card, locked before outcome computation, evaluated through comparison records, and reported with an explicit evidence boundary.

TABLE II C ORE C LAIM S TAB -QC CONCEPTS . Concept

Meaning

Reported comparison

A source-paper statement comparing two baselines under a metric, scope, and outcome rule. Fixed representation of a reported comparison: problem, baselines, metric, and relation. One source-supported comparison unit checked by the framework. Evidence points and evaluation conditions fixed before outcomes are computed. Paired A/B evaluation under the same locked evidence point and setting. Whether the source exposes enough evidence to lock the design or outcome rule without proxy reconstruction. Missing or unsupported evidence that limits what can be audited. Scalar-directional outcome: Sustained, Unresolved, or Reversed within the locked design.

Claim card Audit record Locked audit design Matched cell Materialization Evidence boundary S/U/R verdict

when the source comparison requires distinct comparator pairs, metrics, relation rules, evidence scopes, or outcome rules. For the scalar-directional instantiation used in the evaluation (Section V-B), R is a scalar-directional ordering D ∈ {<, >} over M. The problem field P records the reported comparison context needed to interpret the claim, such as the problem family, instance distribution, benchmark scope, and fixed assumptions stated with the comparison. Audit records: An audit record a is the source-supported comparison object on which the relation R is checked. For a scalar-directional record, the locked audit design is the SCOPE × ENV product Aa = Sa × Ea ,

(2)

predeclared and locked before signed margins or outcomes are measured with Na = |Aa | matched cells. A matched • Vary only admissible conditions. Active axes must be comparison record pairs one BA and one BB evaluation under source-supported, metric-affecting, and relation-preserving. the same admissible SCOPE point and ENV setting, producing a signed margin and a binary direction-preservation indicator. • Summarize by the reported relation. Scalar-directional records are assigned Sustained/Unresolved/Reversed (S/U/R) A design with one comparison record yields only a local outcomes using the Wilson decision rule defined in Sec- preserved, tied, or contradicted outcome; a design with multiple tion III-E; non-scalar records use their locked reported predeclared comparison records supports an aggregate scoperelative outcome. relation rules. Not all audit records are scalar-directional. When the source B. Claim Cards and Audit Records comparison is aggregate, regional, runtime-based, diagnostic, Table II fixes the audit vocabulary used by C LAIM S TAB - table-valued, or multi-objective, C LAIM S TAB -QC locks the QC. Claim cards are needed because a reported ordering is not evidence object and outcome rule required by that relation. If yet an auditable unit: a single source statement may combine the required evidence, statistic, convention, or outcome rule multiple comparator pairs, metrics, scopes, or outcome rules. cannot be reconstructed from the available source or artifact The card fixes these fields before any verdict is assigned, evidence, the audit stops with an evidence boundary. preventing the audit from conflating the reported comparison with a particular rerun or proxy reconstruction. For example, a C. Audit Space and Materialization compiler claim such as “Qiskit produces fewer two-qubit gates Materialization asks whether the source exposes enough than Tket” becomes a claim card; once the source-supported evidence to lock the comparison object, evidence points, circuits, optimization levels, seeds, baselines, and metric are evaluation conditions, and outcome rule. Auditing then asks locked, each circuit–optimization-level–seed setting becomes a what the locked comparison supports; records that cannot be matched comparison cell. locked are reported as boundary records. Claim cards: A claim card structures a comparative statement Audit space: For a given audit record, the admissible audit from the source as space is defined as the product of the evidence scope and the C = ⟨P, BA , BB , M, R⟩, (1) evaluation conditions: where P specifies the problem and evidence universe, BA and BB identify the compared baselines, M specifies the metric or measurement object, and R is the reported comparison. The relation R may be scalar-directional, aggregate, categorical or equality-based, crossover/regional, runtime-ratio, table-valued, or multi-objective. A claim card yields one audit record when the source comparison is atomic; it yields multiple records

Saudit = Sscope × Eenv.

(3)

Sscope contains the evidence points being compared, while Eenv contains the conditions under which those points are produced or evaluated. Table I defines the six audit-axis groups used to inspect this space: SCOPE selects the evidence points, and ENV groups L1, L2, L3, ALGO, and BUDGET describe

construction, execution, estimation, optimization, and resourcepolicy choices. Inspection and activation: For each audit record involving circuit construction, simulation, or execution, C LAIM S TAB -QC inspects the six audit-axis groups in Table I before measuring outcomes. Inspection is broader than variation: unmentioned or unsupported axes are recorded but not varied. An axis is activated as a varying dimension only when the proposed variation is source-supported, metric-affecting, and relationpreserving. All other axes are kept fixed, confined to the source-exposed setting, marked inadmissible, or recorded as insufficiently supported/evidence-boundary cases, together with the source pointer or missing-evidence note that justifies the decision. Materialization test: For scalar-directional audit records, materialization requires pairable baseline records under the same cell setting. For relation-specific audit records, such as aggregate, crossover, runtime, or table-valued records, it requires the source rows, aggregate rule, curve definition, timing convention, table rule, or regional outcome rule. If these cannot be locked from the available source or artifact evidence, the auditor does not infer missing values, introduce proxy baselines, or replace the source comparison with a new benchmark choice. C LAIM S TAB -QC instead returns a boundary record that names the missing evidence, records the inspected sources, and carries no audit verdict. Records that pass this test proceed to the admissibility rules below; RQ1 applies the test at the corpus scale (Section V-A). Admissibility rules: To keep variations within the audited comparison, admissible variations must satisfy three conditions: 1) Source support: the variation is stated by the source, exposed in the artifact, implied by the reported experimental design, or corresponds to a standard execution degree of freedom needed to reproduce the comparison. 2) Metric relevance: the variation can affect the audited metric M or the measured evidence object. 3) Relation preservation: the variation does not change the compared baselines, metric, relation type, or intended interpretation of the source comparison. In practice, admissible variations are supported in three ways: the problem or benchmark scope implied by the claim, the evaluation conditions that the claim depends on, and standard-practice degrees of freedom, such as transpiler seed or optimization level, which are metric-affecting and relationpreserving. Excluded axes are still recorded with reasons; for example, a compiler/transpiler audit record may admit transpiler seed, optimization level, and basis-gate variation within the stated family, but excludes adding comparator tools or switching to a runtime metric for a gate-count claim. Runtime and timing repetitions are treated as measurement-protocol ENV settings. D. Dry-Run-First Audit Procedure C LAIM S TAB -QC enforces lock-before-observing through the dry-run-first procedure in Algorithm 1. The procedure first checks whether the audit record can be materialized, labels

Algorithm 1: C LAIM S TAB -QC dry-run-first audit input: audit record a from claim card C; source/artifact evidence; threshold τ ; audit budget Nmax output: scoped outcome O, explanation report X, and evidence boundary B 1 r ← R ELATION T YPE (a); (ok, B) ← C HECK M ATERIALIZATION(a, source, artifact); 2 if not ok then 3 return B OUNDARYO NLY(B); (axes, B) ← L ABEL A XES(a, source, artifact); G ← F ORM C ANDIDATE D ESIGN(a, axes); (V, B) ← A PPLY VALIDITY F ILTERS(G, B); E MIT D RY RUN(V, axes, B); 5 if V = ∅ then 6 return B OUNDARYO NLY(B); 4

A ← L OCK D ESIGN(V, Nmax ); if r = S CALAR D IRECTIONAL then 9 ∆ ← M EASURE M ARGINS(A, BA , BB , M); (k, N ) ← P RESERVATION C OUNT(∆, R); O ← W ILSON SUR(k, N, τ ); 10 else 11 O ← R ELATION S PECIFIC O UTCOME(A, r, R); 7

8

12

(X, B) ← E XPLAIN O UTCOME(O, A, axes, B); return (O, X, B);

admissible and inactive axes, forms and filters a candidate design, and emits a dry-run report containing schemas, counts, exclusions, and missing evidence. Only after this dry run is the design locked and the outcome computed. Audit-budget handling: The comparison-record count N is the size of the locked design: the full SCOPE × ENV product supported by source when it fits the audit budget Nmax , or an outcome-blind predeclared subset over the same axes when the product exceeds Nmax . The subset rule, random seed, valid counts, exclusions, and missing fields are reported in the dryrun summary before outcome computation. For a non-scalar record, the outcome rule is applied once its statistic, convention, and rule are locked; otherwise the record returns a boundary record. E. Scalar-Directional Outcome Assignment Scalar-directional records need a relation-level rule for deciding whether the reported direction is preserved across the locked design. C LAIM S TAB -QC assigns one of three scoperelative outcomes: Sustained, Unresolved, or Reversed. Other relation types use their locked source-supported rules. Signed margin and preservation: For scalar-directional records, C LAIM S TAB -QC treats direction preservation as the event being audited. In each locked comparison record, C LAIM S TAB -QC records the signed margin (c)

(c)

δc = MA − MB ,

(4)

whose sign determines whether the claimed direction D is preserved. For N comparison records with k preserving the direction, the empirical preservation rate is ŝ = k/N . Because the audited relation is strict, ties do not preserve the relation; they are reported separately from strict contradictions to

distinguish tie-driven loss of improvement from opposite- the report therefore names the explanation mode behind direction dominance. This tie policy applies to strict directional a verdict, and in particular records a Reversed verdict as relations such as “lower”, “fewer”, or “better”. Non-strict strict-contradiction-dominated when opposite-direction margins relations, such as equivalence or no-worse-than claims, are dominate or as tie-dominated when strict improvement is lost handled by their reported relation rules. mainly through ties. Wilson S/U/R rule: The Wilson rule [21], [22] is a transparent Diagnostic extensions: Two optional diagnostics refine or audit policy over locked comparison records: a relation is stress-test scalar-directional verdicts without changing the Sustained when the Wilson lower bound reaches τ , Reversed predeclared primary audit. First, when a defensible practicalwhen the Wilson upper bound falls below 1−τ , and Unresolved effect threshold ϵ is available, C LAIM S TAB -QC can replace otherwise. The interpretation unit is the locked design: the the basic preservation indicator with preservation count k/N and S/U/R label are read relative to Icϵ = ⊮[δc preserves R ∧ |δc | ≥ ϵ] , (5) the locked metric, preservation threshold, confidence level, and comparison records. The preservation threshold and so a comparison record supports the claim only when its confidence level are predeclared separately. We set τ = 0.95 signed margin both preserves the reported relation and exceeds as a conservative threshold for strict directional claims. We the practical threshold. Wilson classification can then be P report scalar-directional results using Wilson intervals at the recomputed using kϵ = c Icϵ and N = |A|. commonly used 95% confidence level [23], with bounds w Second, for an active audit axis x, C LAIM S TAB -QC reports and w̄. Because locked cells may share circuits, instances, marginal preservation summaries seeds, optimization levels, timing groups, or environments, X 1 RQ4 (Section V-D) checks sensitivity to alternative τ values Ic , (6) sx (v) = |Ax=v | and dependence-aware choices. Formally, C LAIM S TAB -QC c∈Ax=v assigns: where Ax=v is the subset of locked comparison records whose • Sustained: w ≥ τ . axis value is v, and Ic is the primary preservation indicator. • Reversed within audited scope: w̄ ≤ 1 − τ . These summaries identify axis values associated with changes in • Unresolved: neither boundary holds. preservation or signed margins. They are descriptive diagnostics Each verdict is therefore interpreted within the locked audit reported alongside the primary locked-design verdict. scope. IV. I MPLEMENTATION AND A RTIFACT F. Explanation and Diagnostic Reports C LAIM S TAB -QC is implemented in Python using Qiskit [24], After assigning the primary scalar-directional outcome, NumPy [25], and SciPy [26]. The artifact records the toolchain C LAIM S TAB -QC records robustness checks, explanation fields, used to produce the released audit evidence and includes and optional diagnostics. These reports are scope-relative, with lightweight checks for consistency of the committed evidence explicit evidence boundaries, and leave the primary outcome package [19]. The implementation separates shared framework unchanged. logic from record-specific adapters through three layers: (1) a Robustness and sensitivity checks: The primary scalar- claim registry and materialization checker store claim-card directional outcome uses the locked design and the Wilson specifications and determine whether the required source S/U/R rule defined in Section III-E. C LAIM S TAB -QC then evidence can be locked; (2) a locking layer freezes comparison reports secondary checks when applicable: dependence-aware records, axis labels, seeds, baseline measurements, exclusions, grouping checks, threshold sensitivity over predeclared τ values, and evidence boundaries before outcome computation; and (3) tie-handling sensitivity for strict scalar-directional records with relation checkers compute scalar-directional S/U/R verdicts or per-cell traces, budget or seed-count expansion while keeping relation-specific summaries, while an exporter writes claim the locked comparison design fixed, and practical-effect and cards, locked evidence manifests, per-cell or aggregate traces, ALGO-bookkeeping checks when the required evidence is outcome rules, explanation fields, and boundary records. available. Missing traces or unsupported checks are recorded Record-specific adapters translate source formats, source as evidence boundaries. These checks qualify the robustness of rows, recovered comparisons, or executable comparators into the primary outcome without changing the predeclared primary locked audit designs when the required evidence is available. verdict. They do not redefine baselines, metrics, admissible axes, or Explanation report: The scalar explanation report separates outcome rules; they connect source-supported evidence to the statistical and policy uncertainty from audit-axis and evidence- shared audit logic. boundary information. Statistical and policy uncertainty inV. E VALUATION cludes the Wilson interval, cluster-resampling check, sensitivity to threshold, audit budget, and practical-effect threshold. AuditWe evaluate C LAIM S TAB -QC as a source-bounded proaxis and evidence-boundary information records preservation, cedure for materializing and auditing empirical comparative ties, strict contradictions, active axis groups, inactive or source- relations. The evaluation is organized around four research confined axes, inadmissible axes, and evidence boundaries. questions (RQs): (1) whether published comparative claims Because the audited direction is strict, ties are non-preserving; can be materialized for relation-level auditing; (2) what

auditing materialized comparison records reveal; (3) whether C LAIM S TAB -QC explains where comparative support is lost; and (4) whether verdicts are robust to statistical and declared audit choices. The artifact provides the locked evidence package used for inspection, including claim cards, audit records, released per-cell or aggregate traces, manifest files, verification scripts, and missing-evidence annotations [19]. A. RQ1: Can published comparative claims be materialized for relation-level auditing? Corpus construction and screening: We constructed the corpus through a structured Google Scholar searchand-screen protocol. We used Google Scholar because it provides broad scholarly discovery across publication venues, preprint repositories, and other research sources. The search was run on January 28, 2026 using 13 predefined query strings that combined quantum-software terms (quantum compiler, quantum compilation, transpiler, QAOA, VQE, variational quantum, and quantum software) with comparison terms (benchmark, comparison, and ansatz comparison). Results were restricted to 2020–2024 using Google Scholar year filters and source metadata during screening, recorded as per-query result snapshots, deduplicated by title and available identifiers such as DOI or arXiv ID, and capped at 250 candidate papers to form a fixed five-year sampled corpus for manual audit. We included papers that reported at least one quantitative empirical comparison in quantum software, including compilation, transpilation, benchmarking, variational algorithms, simulators, backend evaluation, or software-stack configuration studies. We excluded papers that were outside quantum software; hardware-only, pulse-level-only, error-correctiononly, algorithmic-complexity-only, survey-only, tutorial-only, or application-only without a software-stack comparison; did not report a quantitative comparative claim; or did not expose enough text to verify the comparison. Screening retained 119 papers; after claim extraction and claim-level correction, RQ1 uses 455 accepted comparative claims. The artifact records the query strings, search date, result snapshots, perquery result counts, deduplication records, screening decisions, exclusion reasons, full-text extraction records, and per-paper worksheets [19]. Claim extraction and coding validity: From the retained corpus, we extracted comparative claims through a threestage worksheet protocol (candidate → verbatim-verified → accepted), summarized with the materialization funnel in Figure 2. Each accepted claim is then checked at three increasingly strong layers. Representation checks whether the source text fills the claim-card fields ⟨P, BA , BB , M, R⟩ (Section III-B). Planning checks whether the comparator pair, metric, relation type, and admissible evidence scope can be specified from the available support. Materialization checks whether the required evidence can be locked without proxy reconstruction (Section III-D), including matched-cell requirements for scalar-directional records and aggregation rule, source universe, comparator evidence, and zero/failure

Extraction: 119 papers → 630 candidates → 628 verbatim-verified (99.7%) 455 accepted comparative claims (142 compilation / 313 variational) 175 represented as claim cards 79 scalar-directional planning records (145 scalar ∩ 93 planning-feasible)

8 proxy-free scoped audit records

53 lockable designs (8 scoped + 2 sourcematerialized + 43 extension)

Fig. 2. Corpus extraction and materialization funnel. The 53 lockable designs are registry-level design instances; the 8 proxy-free scoped audit records without proxy reconstruction form the Tier-1 RQ2 subset.

convention for non-scalar records. A runnable artifact can still fail materialization if it exposes code but not these fields. Coding followed a shared worksheet protocol: one annotator performed the initial annotation, a second annotator independently recoded the items, and a third annotator conducted a stratified 120-item consistency audit over the main auditabilityfunnel boundaries. Disagreements were adjudicated against the source paper, coding worksheet, and auditability definitions. Agreement was high for the consequential decisions: 92.11% for planning feasibility (κ = 0.835, n = 114), 96.55% for auditable-design classification (κ = 0.782, n = 58), and 100.0% for proxy-free scoped vs. non-exact classification (κ = 1.000, n = 58). The proxy-free scoped count and all eight Tier-1 records remained unchanged; supporting agreement records are provided in the artifact [19]. Auditability surface: Applying this protocol to the accepted claims yields the funnel in Figure 2. Of the 455 accepted comparative claims, 175 can be represented as claim cards. Among these, 145 state strict scalar-directional relations and 93 are planning-feasible; their intersection yields 79 scalardirectional planning records. Of these 79 records, 53 yield a lockable audit or diagnostic design, and 8 of them are proxy-free scoped audit records of the reported comparison itself, without substituting unsupported evidence, settings, or baselines. The 79-to-8 drop is the materialization-gap result. Table III categorizes the 79 scalar-directional planning records by the strongest evidence that could be locked. The 53 lockable cases consist of 8 proxy-free scoped audits of the reported comparison, 43 source-supported extension or diagnostic designs, and 2 source-materialized cases without matched A/B source data. The remaining 26 records stop at a boundary because the available evidence supports only a source-stated rule or aggregate summary, a verification-only check, or an incomplete timing/comparator convention. The PauliOpt runtime claim illustrates the timing-boundary path; the OLSQ-versus-tket source-table evidence illustrates a source-materialized audit with bounded rerun-configuration evidence [27], [28].

TABLE III M ATERIALIZATION BREAKDOWN OF THE 79 SCALAR - DIRECTIONAL PLANNING RECORDS . C OUNTS ARE RECORD - LEVEL AND SUM TO 79;

τ

1−τ EX-C1

RECORDED REASONS SUMMARIZE THE STRONGEST LOCKABLE EVIDENCE OBJECT OR THE BLOCKING EVIDENCE BOUNDARY.

EX-C2 EX-C3 EX-C4

Category

Recorded reason

Lockable matched comparison evidence; eligible for Tier-1 S/U/R verdicts Lockable, not proxy- Source-supported extension or difree scoped agnostic design; not the reported comparison itself Lockable, not proxy- Source-materialized evidence withfree scoped out matched A/B source data for proxy-free scoped auditing Boundary Source-rule-only evidence; no matched scalar A/B design lockable for proxy-free scoped auditing Boundary Verification-only record, not an auditable design Boundary Missing timing protocol, comparator implementation, or other lockable source convention

Records

Proxy-free scoped

8

EX-C5 EX-C6 EX-C7 EX-C8

43

0

0.25

0.50

0.75

1.00

Preservation rate, k/N

2 20

Sustained

Unresolved

Reversed

Fig. 3. Wilson 95% confidence intervals for the eight Tier-1 proxy-free scoped audit records, colored by primary locked verdict (ŝ = k/N ; primary locked values in Table IV). opt 0

opt 1

opt 2

opt 3

lin-3 remote

5/0

4/1

4/1

4/1

lin-4 ladder

5/0

0/5

0/5

0/5

lin-4 bidir

5/0

5/0

5/0

5/0

lin-4 repeat

5/0

5/0

5/0

5/0

2 4

Takeaway. Proxy-free comparative auditing requires evidence that many papers leave implicit; empirical QSE papers should report the baselines, settings, instances, budgets, and outcome rules needed to lock the comparison.

contradicted

tie

Fig. 4. EX-C7 locked-cell grid. Each cell aggregates five transpiler seeds and reports contradicted/tied counts. No cell preserves the claimed direction.

B. RQ2: What does auditing materialized comparison records reveal? RQ2 reports audit outputs by evidence tier. Tier-1 proxy-free 1 audits. Seed and repetition axes are finite locked budgets, scoped scalar-directional records receive scoped S/U/R verdicts not enumerations of all possible executions. Figure 3 shows the primary locked Wilson 95% intervals over locked comparison records. Tier-2 records report sourcefor the eight Tier-1 records. EX-C1 and EX-C2 are Sustained, supported summaries or diagnostics, not Tier-1 verdicts on the EX-C3–EX-C6 are Unresolved, and EX-C7 and EX-C8 are original comparison. Tier-3 records become boundary records Reversed within their audited scopes. Both Reversed records when the design, evidence object, or outcome rule cannot preserve the reported direction in 0/80 primary locked cells. be locked. Table IV reports these records, and the 53-design The artifact provides source links, audit manifests, locked auditability inventory is provided in the artifact [19]. traces, replay traces, worksheet identifiers, and aggregate/cluster Tier-1 eligibility gate: The Tier-1 set contains records for traceability boundaries for EX-C2 and EX-C4 [19]. which scalar-directional auditing is possible without changing Example Tier-1 audit record, EX-C7: EX-C7 [13] illusthe reported comparison or introducing unsupported baselines, metrics, settings, or outcome rules. A record enters Tier 1 trates relation-level auditing beyond a measurement rerun when it (1) states a scalar-directional relation, (2) can be for a compiler comparison. C LAIM S TAB -QC reconstructs materialized without proxy reconstruction, (3) pairs the two the reported Qiskit-versus-Tket ordering as a claim card, reported comparators under matched comparison cells or a locks the connectivity-stress scope, and evaluates four circuit source-supported aggregate object, and (4) admits a locked families across four Qiskit optimization levels and five transpiler design before outcomes are observed. Eight audit records meet seeds. The resulting 80 matched cells contain 0 preserved, 62 these requirements and form the proxy-free scoped subset used contradicted, and 18 tied comparisons (Figure 4). Because the reported relation is strict, ties are non-preserving in the for S/U/R verdicts. Tier-1 proxy-free scalar-directional audit records: Each Tier- primary rule; RQ4 reports tie-handling sensitivity separately. 1 record uses a primary locked design over source-supported The outcome is a Reversed verdict within audited scope, driven axes. For records with per-cell evidence, N denotes the number by strict contradictions, and the locked seed-count expansion of locked comparison records in that design, not a post- yields 0/320 preserved cells. hoc sampling target. The primary verdicts cover the full Summary records: Tier-2 records report the comparison using source-supported primary design in each of the eight Tier- the evidence and outcome rule available in the source, such as

TABLE IV RQ2 AUDIT SUMMARY BY EVIDENCE TIER . ROWS REPORT AUDIT RECORDS , LOCKED DESIGNS , TIER - SPECIFIC OUTPUTS , AND BOUNDARY OR QUALIFICATION NOTES . Audit record Claim / comparison object

Audit design

Output / status

Tier 1 — Proxy-free scoped scalar-directional audit records (Wilson S/U/R verdicts) EX-C1 [20] Pytket < Qiskit, compile time 4 circuits × 4 opt levels × 5 timing 80/80 → Sustained reps EX-C2 [29] QAOA approx.-ratio improvement 16 instances × 5 copula depths 80/80 → Sustained

Boundary / qualification L2/L3 metric-irrelevant

Tket < Qiskit, CNOT count

EX-C4 [29]

primary: 4 circuits × 4 opt levels × 62/80 → Unresolved; seed exp. 5 seeds; seed exp.: same × 20 seeds 243/320 → U QAOA success-prob. improvement 5 sizes × 8 depths × 2 optimizers 45/80 → Unresolved

aggregate/cluster traceability boundary L2/L3 source-confined

EX-C5 [13]

Tket < Qiskit, 2Q depth

aggregate/cluster traceability boundary L2/L3 source-confined

EX-C6 [20]

Qiskit > Pytket, CX depth

EX-C7 [13]

Qiskit < Tket, 2Q gates

EX-C8 [13]

Cirq < Qiskit, build time

EX-C3 [27]

primary: 4 circuits × 4 opt levels × 44/80 → Unresolved; seed exp. 5 seeds; seed exp.: same × 20 seeds 180/320 → U primary: 4 opt levels × 20 seeds 34/80 → Unresolved; seed exp. (fixed graph); seed exp.: same × 80 144/320 → U seeds primary: 4 connectivity-stress 0/80 (62 contra, 18 tie) → circuits × 4 opt levels × 5 seeds; Reversed; seed exp. 0/320 → R seed exp.: same × 20 seeds 4 circuit sizes × 20 timing reps

0/80 → Reversed

seed tied to fixed graph scope source-supported connectivity-stress scope; strict-contradictiondominated no ties; strict reversal

Tier 2 — Source-supported summaries and diagnostic records SD1 [28] OLSQ < tket, CX count; source 22 OLSQ source rows; no ENV rows sweep SD2 [28] OLSQ < tket, depth & SWAP; 7 recovered comparisons recovered SD3 [30] Qrisp < Qiskit-lib, QAOA depth; 3 sizes × 5 graphs × 3 p-depths extension SD4 [31] CNOT-OPT < current baseline, recovered circuits × 2 architectures CNOT (240 cells) SD5 [27] PauliOpt < TKET, per-device 5 IBM devices × gadget-size sweep CNOT crossover SD6 [28]

OLSQ aggregate CX reduction; geometric-mean rule

SD7 [31]

SD4 baseline-policy sensitivity

Source-row summary: 16/22 tket version per source preserve, 6 tie Recovered summary: 7/7 preserve small N for depth and SWAP Diagnostic extension: 45/45 preserve extension, not reported comparison Diagnostic surface: 0/240 preserve baseline-version boundary Source-supported crossover summary: derived predicate; 5/5 devices match the reported limited tail evidence pattern 22 source rows + geometric-mean Source-rule summary: geometric row-level 16/22 rule mean 0.36 (< 1) supports the diagnostic only aggregate relation 2 architectures × 4 Qiskit opt levels Diagnostic sensitivity surface: opt. sensitivity to SD4 level 0: 5/50 & 5/70; opt. levels baseline policy ≥1: 0

Tier 3 — Boundary audit records BD1 [32] SQUANDER < Qiskit, depth cannot lock comparison design (intended) BD2 [27] PauliOpt < Qiskit/TKET, runtime cannot lock timing evidence (intended)

No verdict No verdict

OCR-unreliable; 467× vs 479.5× uncontrolled host timing; no source runtime

S/U/R are Wilson outcomes for Tier-1 proxy-free scoped scalar-directional audit records (τ = 0.95). Tier-2 rows report source-supported summaries or diagnostics, not Tier-1 verdicts. No verdict means that the design, evidence object, or outcome rule could not be locked. Seed exp. replays the locked generator with more seeds only; U/R abbreviate Unresolved/Reversed. 2Q denotes two-qubit; CX denotes CNOT. Per-record manifests and traces are in the artifact [19].

source rows, recovered comparisons, device curves, aggregate rules, or diagnostic surfaces (Table IV). SD6 illustrates why the relation rule matters: the geometric-mean aggregate stated in the source (0.36 < 1) supports the reported reduction, while the row-level preservation summary is reported only as diagnostic evidence. Boundary audit records: BD1 and BD2 cannot lock a comparison design (Table IV): one has inconsistent source table values, and the other is a runtime comparison in an uncontrolled timing environment. C LAIM S TAB -QC records the evidence boundary and assigns no verdict.

Takeaway. C LAIM S TAB -QC aligns audit outputs with the available evidence: proxy-free scoped records receive scoped verdicts, source-supported non-Tier-1 records receive source-rule summaries or diagnostic outputs, and unsupported records become explicit evidence boundaries. C. RQ3: Can C LAIM S TAB -QC explain where comparative support is lost? RQ3 evaluates whether C LAIM S TAB -QC explains where comparative support is lost once a comparison is audited over locked, source-aligned axes. We use three controlled

mechanism checks: (1) a C1–C24 canonical diagnostic contrast TABLE V V ERDICT- LEVEL ROBUSTNESS CHECKS . over benchmark-relevant quantum workloads and problem settings; (2) axis-controlled diagnostic surfaces that isolate Robustness check Scope Result and interpretation L1, L2, L3, ALGO, and combined stack effects; and (3) Threshold τ Tier-1 scoped + 24 Tier-1 labels are unchanged at τ ∈ decision-rule controls for the Wilson S/U/R policy under known canonical {0.90, 0.95}; at τ = 0.99, finite-N extreme labels soften to Unresolved. The preservation rates and reversed cases. canonical check shifts 10 borderline cases Canonical diagnostic contrast: The C1–C24 contrast conto Unresolved. tains 24 controlled comparisons from eight benchmark-relevant Dependence-aware 47-record union The 47-record grouping check agrees with the base setups. C1–C9 are structural compiler-output comparisons grouping + Tier-1 per-cell primary classifications. Tier-1 per-cell checks records identify EX-C3 as cluster-bootstrap-sensitive; over Arithmetic, GHZ, and QFT circuits, comparing pytket EX-C2 and EX-C4 are unavailable for this and Qiskit on two-qubit gate count. Such circuit families and check. structural metrics are common in quantum compiler and circuit- Tie handling Strict scalar- EX-C7 has no preserved cells and remains directional per-cell contradicted-dominated, but its categorical benchmarking practice [13], [20], [33], [34]. C10–C24 are records Reversed label is sensitive to alternative tie QAOA layer-count comparisons over Max-2-SAT and MaxCut handling; records without per-cell traces are unavailable. instances, comparing p=2 and p=1 at fixed angles γ = 0.8 and β = 0.4; MaxCut and 2-SAT are standard QAOA benchmarking Budget / seed-count Replayable Tier-1 All replayable compiler records keep their compiler records primary labels through N = 320; EX-C7 problems, and MaxCut depth comparisons beyond p=1 have expansion remains at 0/320 preserved cells. been studied directly [6], [16], [35]. The contrast compares Practical-effect ϵ and Applicable diagnos- No classification changes among analyzable three simpler audit protocols with the full locked audit design. ALGO bookkeeping tic records practical-effect checks; ALGO bookkeeping does not change matched records or Wilson A single-configuration run reports only a direction and matches labels. the full-audit direction in 19/24 records, while opposing it in 5/24. A seed-only audit calls all 24 records Unresolved and agrees with the full audit on 14/24. A single-axis L1 audit is available for 15 records and disagrees with the full audit on fixes a true preservation rate strue and locked-design size N , 10/15. The full locked audit design yields 10 Sustained, 14 samples binary direction-preservation indicators, and applies the Unresolved, and 0 Reversed within audited scope. Thus, on S/U/R rule to the resulting preservation count. The grid contains canonical quantum workloads, simpler protocols can preserve 18 profiles, crossing strue ∈ {0.50, 0.70, 0.85, 0.95, 0.99, 1.00} an apparent direction while missing the audit-axis combinations with N ∈ {10, 30, 100}; across 3000 trials, the Wilson 95% interval contains the true preservation rate in 94.97% of trials. that weaken comparative support. Axis-controlled diagnostic surfaces: The controlled diag- Second, a reversed-detection control detects 26 of 27 knownnostic suite operationalizes audit-axis groups through finite, reversed cases. The artifact provides the diagnostic grids, perpredeclared grids while keeping the comparator pair, metric, case audit-space construction, per-diagnostic summaries, and relation rule, and inactive axes fixed. It covers five surfaces: verification pointers for RQ3 [19]. L1 varies within-Qiskit compilation settings over structural Takeaway. On benchmark-relevant quantum workloads and circuits; L2 varies shot count and simulator seed for fixedcontrolled audit surfaces, C LAIM S TAB -QC exposes how angle QAOA on deterministic MaxCut graphs; L3 varies six apparent comparative directions can weaken or become simulator-noise regimes over the same QAOA comparison; Unresolved when source-aligned stack axes are locked and ALGO varies optimizer family and initialization seed under a inspected. fixed iteration budget; and a combined surface jointly varies L1, L2, and L3 on a smaller MaxCut pool. Across the 15 D. RQ4: Are verdicts robust to statistical and declared audit controlled-surface records, the suite yields 0 Sustained, 13 choices? Unresolved, and 2 Reversed within audited scope. The two RQ4 checks whether scalar-directional verdicts depend on Reversed outcomes are tie-driven L1 structural cases, while L2, L3, ALGO, and combined surfaces remain Unresolved under statistical choices or declared audit choices. We treat S/U/R labels as scoped classifications over locked audit designs, then the locked grids. Example controlled diagnostic: One controlled MaxCut- evaluate sensitivity to preservation threshold τ , dependenceQAOA diagnostic audits the direction p=2 > p=1, a QAOA aware grouping, tie handling, budget or seed-count expansion, layer-count comparison studied in MaxCut analyses [6], [35]. practical-effect threshold ϵ, and ALGO-axis bookkeeping when Under a combined L1×L2×L3 stress design with δ = 0.05, the required traces are available. Table V summarizes the the direction is preserved in 723 of 1,440 locked cells checks. (ŝ = 0.502, Wilson 95% interval [0.476, 0.528]), yielding The checks preserve the primary verdicts under the predean Unresolved outcome. The apparent layer-count advantage clared audit settings and identify the remaining boundary cases. therefore becomes unsupported once compilation, sampling, EX-C3 is cluster-bootstrap-sensitive; single-cluster deletion and noise axes are inspected together. keeps its primary Unresolved label unchanged. EX-C2 and EXDecision-rule controls: We evaluate the scalar-directional C4 retain their primary verdicts; per-cell dependence and tiedecision rule with two controls. First, a Wilson calibration grid handling checks are unavailable because the required traces are

not materialized. EX-C7 is contradicted-dominated: no locked or expanded cell preserves the reported direction, including 0/80 primary cells and 0/320 cells under locked seed-count expansion. Under stricter threshold or alternative tie-handling policies, the categorical Reversed label is marked as policysensitive, even though the observed preservation count remains zero.

records. Compiler and runtime verdicts are scoped to the locked toolchain version and recorded timing environment.

VII. R ELATED W ORK This work is closest to robustness analysis, quantumsoftware testing, benchmarking, reproducibility infrastructure, and platform-stability studies. These areas provide adjacent evidence; C LAIM S TAB -QC audits the reported comparison Takeaway. Stable outcomes are grounded in locked evitself. idence and declared audit choices, while sensitivity and Specification-space and robustness analyses: Multiverse missing traces are reported as explicit evidence boundaries. analysis [36] and specification-curve analysis [37] test whether an empirical finding survives alternative defensible analysis Evidence synthesis: Together, the four RQs show that choices. C LAIM S TAB -QC shares this robustness intuition, but reported comparisons in empirical QSE require auditing the its admissible space is source-bounded, not analyst-controlled: comparison itself alongside execution-level reproducibility. baselines, circuits, seeds, budgets, backend settings, and C LAIM S TAB -QC makes this audit inspectable by identifying outcome rules can vary only when they are source-supported, what can be materialized, what the locked evidence supports, metric-affecting, and relation-preserving. and where the evidence boundary lies. Quantum-software testing and benchmarking: MorphQ [12], QDiff [38], Muskit [39], Quito [40], MorphQ++ [41], and VI. T HREATS TO VALIDITY QITE [42] test programs, platforms, or implementations. This work is subject to four main threats to validity, Benchmarking suites such as Benchpress [13], Arline [20], concerning decision-rule dependence, human judgment, corpus QASMBench [33], MQT Bench [34], and QSimBench [43] produce measurements from which comparative claims are and diagnostic scope, and hardware/toolchain scope. Decision rule and dependence: S/U/R labels are scoped often drawn. C LAIM S TAB -QC instead checks whether the decisions over locked comparison records, not claims about all reported comparison is materializable and preserved under the possible executions. Audit cells may share circuits, instances, source-supported comparison design. Reproducibility, artifacts, and platform variation: Empirical seeds, optimization levels, timing groups, or environment software engineering and quantum-software reproducibility settings. We therefore interpret verdicts within the locked finite design and report sensitivity checks when traces permit; work asks whether artifacts can be accessed, executed, reused, or rerun [14], [44]–[47]. Hardware-stability and platform unsupported checks are recorded as evidence boundaries. Human judgment: Claim extraction, claim-card construction, studies explain why quantum-software evidence varies across admissibility decisions, and evidence-boundary annotations time, platforms, and toolchains [15], [17], [18], [48]–[50]. require judgment. We mitigate this threat through the coding C LAIM S TAB -QC uses this insight to separate admissible protocol and inter-auditor checks in Section V-A, including variation from unsupported variation, which becomes an blinded recoding, stratified consistency auditing, and adjudica- evidence boundary. tion against the source paper and artifact. The artifact records VIII. C ONCLUSIONS decisions, exclusions, axis labels, missing-evidence reasons, Empirical QSE comparisons are often reported as perforand adjudication records [19]. mance orderings, but our audit shows that the ordering and Corpus and diagnostic scope: The corpus is Google Scholar- its supporting evidence are separable. A comparison can be based and reflects a sampled corpus. Relevant comparisons identifiable, representable, and even partially reproducible while outside the indexed or screened results, or under different termi- still lacking the matched baselines, settings, instances, budnology, may be missed. The materialization funnel is therefore gets, conventions, or execution records needed for proxy-free evidence for the sampled corpus. The eight proxy-free scoped relation-level auditing. C LAIM S TAB -QC makes this distinction records support scoped S/U/R outcomes over materialized explicit by reporting either a scoped relation outcome or an records, while their small number is part of the materialization- evidence boundary. gap result. Controlled diagnostics use benchmark-relevant The main lesson is that many empirical quantum-software quantum-workload cases and axis-controlled surfaces, including comparisons are reported as outcomes without the evidence structural circuit-compilation and MaxCut/Max-2-SAT QAOA needed to audit the comparison itself. For empirical QSE, this comparisons, to expose mechanisms of comparative-support shifts the reporting target: sources should state the relation being loss under fixed audit axes; they are not prevalence estimates claimed, expose the evidence needed to lock that relation, and over all empirical QSE claims. clearly mark where the available evidence no longer supports Hardware and toolchain scope: The L3 evidence in this the comparison. study is limited to controlled simulator-noise regimes on IX. DATA AVAILABILITY Qiskit Aer and Qiskit fake backends. Live-hardware drift, cross-provider variation, queue effects, calibration updates, An artifact package will be released publicly after the review and future toolchain changes are outside the locked audit period.

R EFERENCES [1] M. A. Nielsen and I. L. Chuang, Quantum computation and quantum information. Cambridge University Press, 2010. [2] P. W. Shor, “Algorithms for quantum computation: discrete logarithms and factoring,” in Proceedings 35th Annual Symposium on Foundations of Computer Science (FOCS). IEEE, 1994, pp. 124–134. [3] L. K. Grover, “A fast quantum mechanical algorithm for database search,” in Proceedings of the 28th Annual ACM symposium on Theory of computing (STOC). ACM, 1996, pp. 212–219. [4] J. Preskill, “Quantum computing in the NISQ era and beyond,” Quantum, vol. 2, p. 79, 2018. [5] A. Peruzzo, J. McClean, P. Shadbolt, M.-H. Yung, X.-Q. Zhou, P. J. Love, A. Aspuru-Guzik, and J. L. O’brien, “A variational eigenvalue solver on a photonic quantum processor,” Nature Communications, vol. 5, no. 1, p. 4213, 2014. [6] E. Farhi, J. Goldstone, and S. Gutmann, “A quantum approximate optimization algorithm,” arXiv preprint arXiv:1411.4028, 2014. [7] Cirq Developers, “Cirq: A python framework for creating, editing, and invoking noisy intermediate-scale quantum (NISQ) circuits,” https:// github.com/quantumlib/Cirq, 2022, quantum AI Team, Google. [8] V. Bergholm, J. Izaac, M. Schuld, C. Gogolin, S. Ahmed, V. Ajith, M. S. Alam, G. Alonso-Linaje, B. AkashNarayanan, A. Asadi et al., “PennyLane: Automatic differentiation of hybrid quantum-classical computations,” arXiv preprint arXiv:1811.04968, 2018. [9] T. Rohe, F. H. Ruiloba, S. Egger, S. von Beck, J. Stein, and C. LinnhoffPopien, “Quantum computer benchmarking: An explorative systematic literature review,” arXiv preprint arXiv:2509.03078, 2025. [10] G. Li, Y. Ding, and Y. Xie, “Tackling the qubit mapping problem for nisq-era quantum devices,” in Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM, 2019, pp. 1001–1014. [11] M. D. Stefano, D. D. Nucci, F. Palomba, and A. D. Lucia, “An empirical study into the effects of transpilation on quantum circuit smells,” Empirical Software Engineering, vol. 29, no. 3, p. 61, 2024. [12] M. Paltenghi and M. Pradel, “MorphQ: Metamorphic testing of the qiskit quantum computing platform,” in Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2413–2424. [13] P. D. Nation, A. A. Saki, S. Brandhofer, L. Bello, S. Garion, M. Treinish, and A. Javadi-Abhari, “Benchmarking the performance of quantum computing software for quantum circuit creation, manipulation and compilation,” Nature Computational Science, vol. 5, pp. 427–435, 2025. [14] W. Mauerer and S. Scherzinger, “1-2-3 reproducibility for quantum software experiments,” in Proceedings of the 29th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2022, pp. 1247–1248. [15] S. Dasgupta, “Stability of quantum computers,” arXiv preprint arXiv:2404.19082, 2024. [16] M. Willsch, D. Willsch, F. Jin, H. De Raedt, and K. Michielsen, “Benchmarking the quantum approximate optimization algorithm,” Quantum Information Processing, vol. 19, no. 7, p. 197, 2020. [17] B. Baheri, Q. Guan, V. Chaudhary, and A. Li, “Quantum noise in the flow of time: A temporal study of the noise in quantum computers,” in Proceedings of the 28th IEEE International Symposium on On-Line Testing and Robust System Design (IOLTS). IEEE, 2022, pp. 1–5. [18] S. Dasgupta, T. S. Humble, and A. Danageozian, “Adaptive mitigation of time-varying quantum noise,” in Proceedings of the 4th IEEE International Conference on Quantum Computing and Engineering (QCE). IEEE, 2023, pp. 99–110. [19] B. Ye, P. Liang, M. T. Sabzevari, and A. A. Khan, “CLAIMSTAB-QC: Audit evidence package,” 2026, artifact package to be released publicly after the review period. [20] Y. Kharkov, A. Ivanova, E. Mikhantiev, and A. Kotelnikov, “Arline benchmarks: Automated benchmarking platform for quantum compilers,” arXiv preprint arXiv:2202.14025, 2022. [21] E. B. Wilson, “Probable inference, the law of succession, and statistical inference,” Journal of the American Statistical Association, vol. 22, no. 158, pp. 209–212, 1927. [22] L. D. Brown, T. T. Cai, and A. DasGupta, “Interval estimation for a binomial proportion,” Statistical Science, vol. 16, no. 2, pp. 101–133, 2001.

[23] D. G. Altman, D. Machin, T. N. Bryant, and M. J. Gardner, Eds., Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd ed. London: BMJ Books, 2000. [24] A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood, J. Lishman, J. Gacon, S. Martiel, P. D. Nation, L. S. Bishop, A. W. Cross et al., “Quantum computing with Qiskit,” arXiv preprint arXiv:2405.08810, 2024. [25] C. R. Harris, K. J. Millman, S. J. Van Der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith et al., “Array programming with numpy,” Nature, vol. 585, no. 7825, pp. 357–362, 2020. [26] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright et al., “SciPy 1.0: Fundamental algorithms for scientific computing in Python,” Nature Methods, vol. 17, no. 3, pp. 261–272, 2020. [27] A. Meijer-van de Griend, “A comparison of quantum compilers using a DAG-based or phase polynomial-based intermediate representation,” arXiv preprint arXiv:2304.08814, 2023. [28] B. Tan and J. Cong, “Optimal layout synthesis for quantum computing,” arXiv preprint arXiv:2007.15671, 2020. [29] P. Christiansen, L. Binkowski, D. Ramacciotti, and S. Wilkening, “Quantum tree generator improves QAOA state-of-the-art for the knapsack problem,” arXiv preprint arXiv:2411.00518, 2024. [30] E. Osaba, M. Petrič, I. Oregi, R. Seidel, A. Ruiz, S. Bock, and M.-A. Kourtis, “Eclipse Qrisp QAOA: description and preliminary comparison with Qiskit counterparts,” arXiv preprint arXiv:2405.20173, 2024. [31] V. Gheorghiu, J. Huang, S. M. Li, M. Mosca, and P. Mukhopadhyay, “Reducing the CNOT count for Clifford+T circuits on NISQ architectures,” arXiv preprint arXiv:2011.12191, 2020. [32] P. Rakyta, G. Morse, J. Nádori, Z. Majnay-Takács, O. Mencer, and Z. Zimborás, “Highly optimized quantum circuits synthesized via dataflow engines,” arXiv preprint arXiv:2211.07685, 2022. [33] A. Li, S. Stein, S. Krishnamoorthy, and J. Ang, “QASMBench: A lowlevel quantum benchmark suite for NISQ evaluation and simulation,” ACM Transactions on Quantum Computing, vol. 4, no. 2, pp. 1–26, 2023. [34] N. Quetschlich, L. Burgholzer, and R. Wille, “MQT Bench: Benchmarking software and design automation tools for quantum computing,” Quantum, vol. 7, p. 1062, 2023. [35] J. Wurtz and P. J. Love, “MaxCut quantum approximate optimization algorithm performance guarantees for p > 1,” Physical Review A, vol. 103, no. 4, p. 042612, 2021. [36] S. Steegen, F. Tuerlinckx, A. Gelman, and W. Vanpaemel, “Increasing transparency through a multiverse analysis,” Perspectives on Psychological Science, vol. 11, no. 5, pp. 702–712, 2016. [37] U. Simonsohn, J. P. Simmons, and L. D. Nelson, “Specification curve analysis,” Nature Human Behaviour, vol. 4, no. 11, pp. 1208–1214, 2020. [38] J. Wang, Q. Zhang, G. H. Xu, and M. Kim, “Qdiff: Differential testing of quantum software stacks,” in Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 692–704. [39] E. Mendiluze, S. Ali, P. Arcaini, and T. Yue, “Muskit: A mutation analysis tool for quantum software testing,” in Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 1266–1270. [40] X. Wang, P. Arcaini, T. Yue, and S. Ali, “Quito: a coverage-guided test generator for quantum programs,” in Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 1237–1241. [41] L. J. Kitt and M. B. Cohen, “MorphQ++: A reproducibility study of metamorphic testing on quantum compilers,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW). ACM, 2024, pp. 8–14. [42] M. Paltenghi and M. Pradel, “Qite: Assembly-level, cross-platform testing of quantum computing platforms,” arXiv preprint arXiv:2503.17322, 2025. [43] G. Bisicchia, A. Bocci, J. Garcı́a-Alonso, J. M. Murillo, and A. Brogi, “Qsimbench: An execution-level benchmark suite for quantum software engineering,” in Proceedings of the 6th IEEE International Conference on Quantum Computing and Engineering (QCE). IEEE, 2025, pp. 175–180. [44] A. Muttakin, S. Mondal, and C. K. Roy, “The state of open science in software engineering research: A case study of ICSE artifacts,” arXiv preprint arXiv:2601.02066, 2026.

[45] V. Gierisch and W. Mauerer, “Qef: Reproducible and exploratory quantum software experiments,” arXiv preprint arXiv:2511.04563, 2025. [46] E. Moguel, J. A. Parejo, A. Ruiz-Cortés, J. Garcia-Alonso, and J. M. Murillo, “Quantum software experiments: A reporting and laboratory package structure guidelines proposal,” in Proceedings of the 4th IEEE International Conference on Quantum Software (QSW). IEEE, 2025, pp. 185–194. [47] S. Dasgupta and T. S. Humble, “Reproducibility in quantum computing,” in Proceedings of the 20th IEEE Computer Society Annual Symposium on VLSI (ISVLSI). IEEE, 2021, pp. 458–461. [48] M. Paltenghi and M. Pradel, “Bugs in quantum computing platforms: an empirical study,” Proceedings of the ACM on Programming Languages, vol. 6, no. OOPSLA1, pp. 1–27, 2022. [49] M. De Stefano, F. Pecorelli, D. Di Nucci, F. Palomba, and A. De Lucia, “The quantum frontier of software engineering: A systematic mapping study,” Information and Software Technology, vol. 175, p. 107525, 2024. [50] A. Garcı́a de la Barrera, I. Garcı́a-Rodrı́guez de Guzmán, M. Polo, and M. Piattini, “Quantum software testing: State of the art,” Journal of Software: Evolution and Process, vol. 35, no. 4, p. e2419, 2023.