Benchmarking Quantum Software Testing with Scalable Quantum Programs Yuechen Li1,2,* , Minqi Shao2 , Xiyuan Li2 , Jianjun Zhao2 , and Kai-Yuan Cai1 1
arXiv:2607.02029v1 [cs.SE] 2 Jul 2026
2
Beihang University, {liyuechen, kycai}@buaa.edu.cn Kyushu University, {shao.minqi.229, li.xiyuan.868}@s.kyushu-u.ac.jp, [email protected] * Corresponding author
Abstract Quantum software testing (QST) checks whether quantum programs behave according to their intended specifications. A key requirement for QST research is a benchmark that supports rigorous empirical evaluation on programs that are testable and better reflect current software development practices. However, existing studies heavily rely on small hard-coded or circuit-level benchmarks, while available quantum programs are scattered across repositories without clear selection criteria, which limits fair comparison and systematic reproducibility. To this end, we present Qolumbina, a benchmark infrastructure for controlled QST experiments on scalable quantum programs. Qolumbina curates 40 programs from open-source repositories, turns them into test-ready subjects through systematic selection, refactoring, specifications, test case examples, unit tests, and standardized interfaces. We also propose QST-oriented criteria to characterize quantum programs along functionality, output behavior, development complexity, and quantumspecific execution complexity. Using these criteria, our empirical study shows that Qolumbina covers diverse testing-relevant properties and supports scalability analysis beyond fixed-size circuit benchmarks. Through controlled experiments with two recent QST approaches, we demonstrate the feasibility of using Qolumbina for execution-cost and fault-detection studies, and highlight backend-dependent effects that can influence QST result interpretation.
Keywords: quantum software testing, software infrastructure, empirical study
1
Introduction
With the advancement of quantum computing (QC), there is a growing need for high-quality quantum programs. Due to non-intuitive principles of quantum mechanics and increasing program scale, ensuring program quality and reliability remains challenging. As a solution in quantum software engineering (QSE) [43, 26], quantum software testing (QST) [24] is a critical activity in the quantum software development life cycle (QSDLC), which assesses runtime behaviors of quantum programs and checks whether they satisfy given specifications. Recently, empirical studies on QST have received growing attention. In such studies, benchmark quantum programs play a fundamental role in enabling fair and effective evaluation of testing approaches. Evaluating testing techniques requires benchmarks that go beyond small toy examples and cover programs with varying scales along with approximation to real-world scenarios. However, many existing QST studies still rely on fixedsize quantum circuits (i.e., low-level quantum programs in [19]) as programs under test (PUT). These programs are often written in hardware-oriented assembly languages such as OpenQASM or expressed in low-abstraction code, making them poorly scalable and rarely maintainable in a practical QSDLC. In contrast, scalable quantum programs (i.e., high-level quantum programs in [19]) employ richer abstractions and allow classical arguments to produce diverse quantum circuits. Their modular design more naturally follows software engineering (SE) practices, and such programs commonly appear in mainstream quantum software development kits (SDKs) such as Qiskit [30]. Therefore, the gap between current research benchmarks and practical programs may limit the applicability of existing QST techniques in practical program-level testing scenarios. A recent study [19] underscores that prior QST studies rely on programs collected from diverse and fragmented sources. Even the two most frequently used benchmarks are only partially aligned with QST. Bugs4Q [46, 47] includes many classical programs in quantum software stacks, while MQT Bench [34] was originally built for quantum hardware testing with quantum circuits as test inputs rather than PUTs. This purpose mismatch matters for software testing: when reused for QST experiments, such benchmarks provide limited support for functional specifications and standardized interfaces, making it difficult to design valid test cases and interpret
1
test outcomes consistently. Therefore, without a benchmark tailored to QST, it remains difficult to conduct comprehensive, fair, and reproducible evaluations across studies. Motivated by these limitations, we propose a benchmark-construction methodology for controlled QST evaluation on scalable quantum programs, and instantiate it as Qolumbina. It comprises 40 scalable Qiskit programs systematically filtered from 8 open-source repositories of programs with real-world provenance. To mitigate threats to testability and empirical validity arising from current quantum programming practices, including heterogeneous program interfaces and under-specified input constraints, the original programs are moderately refactored into standardized and executable benchmark subjects by improving interface compatibility, aligning language usage, and validating test inputs. To enhance the functional testability, we provide program specifications that document intended program behavior, together with unit tests as executable examples. We further conduct a systematic empirical study to characterize the involved quantum programs and examine testing-relevant properties of scalable quantum programs. From the perspective of QST, we establish a set of criteria for analyzing program functionalities in terms of application domains and output characteristics. Empirical evidence reveals the functional diversity of Qolumbina based on the covered categories and indicates that the design of testing approaches should take such differences into account, although this issue has received limited attention in prior QST research. Then, we characterize program scale along both development and execution dimensions, using standard SE metrics overlooked in QST (e.g., lines of code (LOC)) and quantumspecific metrics (e.g., circuit width). The results show that, unlike fixed-size subjects, our benchmark programs instantiate circuits whose scale depends on classical inputs. Finally, we conduct two controlled experiments in both noiseless and noisy scenarios to examine the feasibility of using Qolumbina in QST experiments. By applying two recent QST approaches [17, 18] and aligning with their scopes, our experiments consider 23 programs with sufficient test suites for execution-cost analysis and 26 programs with 7 buggy variants each for fault-detection analysis. Experimental results demonstrate that Qolumbina can reproduce findings from prior studies and also provide insights into how fake backends affect QST result interpretation. The main contributions of this paper are listed as follows: • We present Qolumbina, a publicly available benchmark infrastructure for controlled QST research on scalable quantum programs. It contains 40 scalable Qiskit programs that are closer to practical quantum software development than the low-abstraction and circuit-like programs commonly used in prior studies. • Instead of simply collecting programs, we propose a construction pipeline that turns open-source quantum programs into test-ready benchmark subjects through systematic selection, program refactoring, program specifications, unit tests, and standardized interfaces. • We establish systematic QST-oriented taxonomies and criteria for characterizing program functionalities and scales, and also use them to validate benchmark diversity and scalability support of Qolumbina. • We conduct two controlled experiments with recent QST approaches, demonstrating the feasibility of using Qolumbina and providing empirical findings for future QST research. The remainder of this paper is organized as below. Section 2 introduces preliminaries of quantum programs and reviews related benchmarks. Section 3 presents the methodology for infrastructure construction. Section 4 reports the empirical study, and Section 5 discusses threats to validity. Section 6 outlines lessons for future research. Finally, Section 7 concludes the paper.
2
Background and Related Work
2.1
Quantum Programs and Their Design
Quantum programs are designed to implement quantum algorithms or subroutines. Especially for the mainstream gate-based quantum computation, the computational procedure of a quantum algorithm is realized by a sequence of quantum gates acting on qubits. The pure state of a qubit can be denoted as a state vector |ψ⟩, and a quantum gate can map the qubit state to U |ψ⟩, where U is mathematically a unitary operator. Regarding running quantum programs on physical hardware or their corresponding classical simulation, program outputs are not accessible until they are transformed into probabilistic outcomes through quantum measurement. Today, quantum programming languages and frameworks, such as Qiskit, built on the host language Python, provide fruitful APIs for programmers to develop quantum applications. Following the practice of classical software engineering (CSE), these official APIs largely involve modularity and scalability, which requires appropriate code abstractions. To compare code at different abstraction levels, Figure 1 illustrates two Qiskit examples, both of which realize the Greenberger-Horne-Zeilinger (GHZ) state preparation. In the left code, each statement after circuit construction corresponds to a concrete quantum gate, but this hard-coded approach limits functionality to 2
┌─┐ q_0: ───────────────┤X├─ ┌─┐└┬┘ q_1: ────────────┤X├─■─ ┌─┐└┬┘ q_2: ─────────┤X├─■──── ┌─┐└┬┘ q_3: ──────┤X├─■─────── ┌─┐└┬┘ q_4: ───┤X├─■────────── ┌─┐└┬┘ q_5: ┤H├─■───────────── └─┘
Low-level quantum program
High-level quantum program (snippet)
# Prepare a 6-qubit GHZ state q = QuantumRegister(6, "q") qc = QuantumCircuit(q, name="ghz")
... # Prepare a {num_qubits}-qubit GHZ state # Included in class GHZState num_qubits = self._num_qubits q = QuantumRegister(num_qubits, "q") qc = QuantumCircuit(q, name="ghz") qc.h(q[-1]) for i in range(1, num_qubits): qc.cx(q[num_qubits - i], q[num_qubits - i - 1]) self.compose(qc, qubits=self.qubits, inplace=True)
qc.h(q[5]) qc.cx(q[5], q[4]) qc.cx(q[4], q[3]) Higher abstraction qc.cx(q[3], q[2]) Quantum qc.cx(q[2], q[1]) qc.cx(q[1], q[0])
circuit
Figure 1: Two different implementations of GHZ state preparation preparing a 6-qubit state only. By contrast, the right-hand example from Qolumbina incorporates higher abstraction for quantum gates and is scalable to prepare a general GHZ state, whose qubit number can be flexibly determined by the classical variable self._num_qubits. Abstraction hides low-level implementation details behind high-level interfaces, supporting the modularity, reusability, and scalability needed as quantum systems grow in both size and complexity [45]. Nevertheless, prior QST studies have paid limited attention to scalable quantum programs with such abstractions, considering only a few exceptions including [20, 21, 17, 18]. Hence, Qolumbina addresses this gap by enabling systematic evaluation of testing techniques on scalable quantum programs.
2.2
Benchmarks for Quantum Software Table 1: Comparison among open-source benchmarks for general research on quantum software Name
Used?
Scalable?
Framework
Target
Component
Size
Bugs4Q [46, 47]
✓
✗
Qiskit
Software
Bugs
36, 42
MQT Bench [34]
✓
✗*
OpenQASM
Hardware
Algorithms
28
QSimBench [7]
✗
✗
Qiskit
Software
Algorithms
14
Lubinski et al. [22]
✗
✓
Qiskit, Cirq, Braket, Q#
Hardware
Algorithms
13
SupermarQ [38]
✗
✓
Qiskit
Hardware, Software
Algorithms
8
VeriQBench [8]
✓
✗*
OpenQASM
Software
Algorithms
21
QASMBench [15]
✓
✗
OpenQASM
Hardware
Algorithms
35
RevLib [40]
✓
✗
SyReC
Hardware
Circuits
154
Qolumbina (ours)
N/A
✓
Qiskit
Software
Algorithms
40
“Used?” indicates whether the benchmark has been applied in existing QST studies. Compared to “✗” in the column of “Scalable?” denoting no scalable quantum programs provided, “✗*” means that several, but not most, of the source quantum programs in the benchmark backend are scalable. “Size” corresponds to the component number reported in the literature, combined with multiple levels, like algorithm types, circuit instances and bugs.
Recent studies have proposed benchmarks for quantum code, such as QuanBench [11] and QCircuitBench [42], which are intended to evaluate large language models in code generation. With a focus on underlying use for QST, Table 1 lists more relevant open-source benchmarks covering QC and SE perspectives. For the scope beyond assembly-level and fixed-size OpenQASM programs, Qiskit is the most popular framework, which motivates building Qolumbina on top of it. In detail, Bugs4Q collects real-world bug-fix pairs in quantum programming, where many bugs do not result from unexpected behavior of quantum programs, but from misuse of quantum software stacks. Meanwhile, RevLib is an early repository of general reversible circuits for computer-aided design, rather than being dedicated to QC. The latest benchmark, QSimBench, is oriented toward QSE research on circuit-level objects and provides precomputed outcomes rather than program code. Since Table 1 counts heterogeneous entities, benchmark sizes are not directly comparable, while Qolumbina still has the largest subject count among benchmarks whose primary components are quantum algorithms. 3
Several benchmarks [34, 22, 15, 40] aim to evaluate performance of quantum hardware and its simulation. OpenQASM programs are common as they serve as an assembly-level intermediate representation that bridges high-level quantum algorithms and low-level hardware execution layers. Therefore, QC research uses these circuit-like programs as test cases to evaluate hardware performance characteristics such as fidelity under noise and entanglement capability [15]. Nevertheless, developers rarely construct applications at such a low level. QST focuses on program functionality and bug detection within the QSDLC. This distinction highlights the difference between testing quantum hardware and testing quantum programs. Bugs4Q, MQT Bench, VeriQBench, QASMBench, and RevLib have been used for QST, but scalable quantum programs have been scarcely involved. Benchmarks [22, 38] have not been explored for QST but include highquality scalable programs, making them appropriate sources for Qolumbina to enhance program diversity. In addition, compared to many scalable programs included in existing benchmarks, which only consider input arguments directly mapped to qubit counts, Qolumbina also exposes the input ports of other classical arguments (e.g., the database to be searched in Grover Search), thereby enabling richer program configurations and input spaces for QST experiments.
3
Methodology of Benchmark Design
3.1
Pipeline of Benchmark Construction
Benchmark programs Test case examples
Open-source repositories
Testability refactoring
Standardized interfaces Qolumbina
Collection & filtering Available references
Unit testing Program specifications
Documentation improvement
Figure 2: Overview of Qolumbina’s construction Qolumbina is a benchmark infrastructure built on Qiskit 2.3.0 to support controlled and reproducible QST experiments. Figure 2 illustrates a pipeline for its construction and structure, with details provided in the following subsections. We first collect open-source quantum software repositories to identify scalable quantum programs. The selected programs are then moderately refactored to standardize interfaces and support valid test execution. Since many repositories lack clear descriptions of program functionality and usage, we consult accessible literature and tutorials to prepare QST-oriented documentation. Finally, we develop unit tests for each benchmark program and iteratively refine both the refactored programs and documentation based on test feedback.
3.2
Program Collection and Filtering
To promote program diversity while ensuring a reasonable selection scope, we consider three types of accessible sources for benchmark programs with real-world provenance: • C1: Program sources or benchmarks mentioned or considered by QST studies • C2: Artifacts of QST studies, which explicitly provide original PUTs • C3: Relevant benchmarks cited by the publications that propose the benchmarks marked as C1 Specifically, we use real-world provenance to refer to traceable sources rather than deployment evidence. A program is considered to exhibit real-world provenance if it is claimed in the corresponding research paper, carries copyright information from a recognized quantum software project, or is hosted in an official quantum SDK repository like Qiskit. An existing survey [19] summarizes 15 C1 sources and 33 C2 sources. For C3, we newly extracted 2 sources, i.e., Lubinski et al.’s benchmark [22] and SupermarQ [38], by revisiting the literature [34]. 4
Then, we filtered out the candidate quantum programs that violated at least one of the following three criteria: • F1: Programs scalable in both logic and structure, allowing classical arguments to flexibly determine quantum circuits • F2: Programs accompanied by functional descriptions in their corresponding sources, or whose underlying algorithmic principles are elaborated on in accessible literature • F3: Programs implementing a unitary operation of a quantum algorithm in a noise-free situation F1 helps exclude programs that inherently represent fixed-size quantum circuits (e.g., preparing a Bell state with only two qubits), or heavily hard-coded logic (e.g., the left program in Figure 1). F2 is motivated by the assumption of testability, as test oracles would be infeasible to properly design or systematically derive in the complete absence of knowledge about expected behavior of the program. F3 follows the current focus on testing-related tasks, in which the evolution of the qubit within a quantum circuit is mathematically modeled as a unitary operation U [16, 27, 25]. In fact, a general class of quantum algorithms involving mid-circuit measurements falls outside the scope described above. Nevertheless, the difficulty in accurately depicting the program specification, together with the scarcity of related QST studies [29], makes it reasonable for Qolumbina to consider these cases as future extensions. We completed program collection before February 15, 2026. Using F1–F3 as exclusion criteria, and after author discussion and consensus, we retained 40 deduplicated programs, treating them as distinct subjects when they exhibited observable structural or functional differences. To maintain source traceability, we preserved copyright information and repository links for each retained program. Especially, C1 provided 22 programs from sources previously used for testing, where three official Qiskit repositories [30, 32, 31] and MQT Bench [34] offered 17 and 5 programs, respectively. There were 12 directly included in artifacts of three prior studies [1, 17, 21] (C2), while Lubinski et al.’s benchmark [22] contributed the remaining 6 (C3). To present a compact overview of benchmark programs, we manually organized them into 24 descriptive families according to their documented primary functionality, where all these families and included programs are listed in our artifact [6]. For example, the family quantum_adder incorporates three types of adders executable in quantum circuits: Draper adder, full adder, and weighted adder.
3.3
Testability Refactoring
Following testability refactoring in SE practices [35], we moderately refactored the collected programs to unify heterogeneous interfaces and support unit testing. Overall, 95% of original programs underwent at least one of the following four testability-oriented adaptations with localized code changes intended to improve testability and maintainability while preserving output behavior for valid inputs. Modifications for refactoring are documented in code comments of the corresponding programs. In Section 3.5, we will further introduce the use of unit tests to practically mitigate potential functional differences after refactoring. • Structure reorganization: Program structures are reorganized to unify input-output ports across our infrastructure. • Dependency decoupling: External dependencies that may affect program functionality or evolve over time are replaced with local equivalent implementations. • Input validation: Assertions are inserted to prevent executing invalid test inputs that violate the program specification. • Cross-language translation: Programs written in other frameworks or languages are translated into Qiskit/Python and validated against their program specifications. Thirty-seven original programs were structurally reorganized, including unifying programs into classes that inherit from Qiskit’s QuantumCircuit and converting fixed classical variables into configurable input arguments. Dependency decoupling was performed in 15 programs, following the insights of [36]. For example, QFT subroutines that originally relied on the Qiskit package were replaced with equivalent implementations provided in Qolumbina. This improves the long-term usability and controllability of the benchmark, because the current QFT implementation is scheduled for deprecation in an upcoming version of Qiskit [33] (commit 3fe73d9). Input validation was applied to 13 programs to support sound functional testing by rejecting invalid test inputs that would otherwise trigger undefined behavior. Specifically, constraints were enforced for both physical interpretability and logical rationality, such as the non-negativity of the evolution time in Hamiltonian simulation and the requirement that integers provided to comparator circuits be representable with the given number of qubits. 5
Finally, 10 programs originally written in Q# were manually translated into Qiskit counterparts conforming to the program specifications given by their source [21]. Since not all implementations have one-to-one counterparts in Qiskit, we constructed equivalent quantum operations when needed; for example, Qiskit’s Hadamard and multi-controlled X gates were used to implement Q#’s multi-controlled Z gate.
3.4
Testing-oriented Program Specifications
The program specifications provided on a webpage [3] aim to facilitate controlled QST experiments by elaborating on the input arguments and expected behavior for each benchmark program. These documents are based on source descriptions and algorithm literature, such as the error-bound analysis of Quantum Monte Carlo [41], to support test oracle design and semantic checks after refactoring. To ensure specification readability, we consider the following five elements: Formula-based specification. According to [20, 19], we utilized mathematical formulas to formally depict the expected output or structure of each benchmark program. For example, a mapping between state vectors, |y⟩n |x⟩n 7→ |(x + y) mod 2n ⟩n |x⟩n , is introduced for draper_adder with two n-qubit input computational basis states |x⟩n and |y⟩n (x, y ∈ N). Meanwhile, for programs with multiple subroutines, such as grover_search, we specified subroutine-level semantics, including the phase oracle, diffusion operator, and iteration-wise state updates. To mitigate confusion in specification interpretation, we underscored the little-endian ordering and qubit counts in the formal representation—details that are sometimes overlooked or insufficiently clarified even in parts of Qiskit’s official documentation. The little-endian ordering allows us to rigorously specify register indices, such as the above initial state |y⟩n |x⟩n ≡ |yn−1 · · · y0 xn−1 · · · x0 ⟩2n with the bits yn−1 and x0 respectively occupying the most and least significant positions within the length-2n array. Our explicit emphasis on qubit counts is motivated by the two versions integer_comparator_greedy and integer_comparator_old sourced from Qiskit. Although both provide similar functionality, they employ inconsistent numbers of qubits (i.e., 1 and n, respectively) to encode the comparison result, which may otherwise lead to ambiguity in formal specifications. API documentation. The API documentation is generated via Sphinx [37] that automatically parses and formats the docstrings of program modules. The program variables are linked with the symbols denoted in the formula-based specifications. The potentially raised errors are listed to remind testers to avoid invalid inputs. In addition, we noticed that the official Qiskit documentation might omit functional descriptions of some arguments used in our benchmark programs, such as the basis variable in two programs in the pauli_rotations family. Thus, we carefully reviewed the source code and supplemented the specification for the corresponding variables. Code example. We provide code examples to facilitate the construction of relatively complex test inputs. For example, counting requires an oracle operation as input, which may be represented either by a state vector or by a quantum circuit. Accordingly, we include two code examples to demonstrate both representations. Documentation references. We provide references used to prepare our program specifications, which strengthen traceability and provide supporting materials for testers. Technical debt. Given the growing attention to technical debt in quantum software [28, 12], we explicitly marked two debts in the documentation to make known limitations and future maintenance costs visible. One is a development debt in multiplier_hrs, where some functional arguments remain unexposed due to redesign risks and validation costs. The other is a testing debt in monte_carlo, where simple and common QST oracles, such as those that compare individual outcomes or full output distributions [2], are theoretically unreliable for this approximate algorithm.
3.5
Unit Testing
Qolumbina includes unit tests both as executable examples for follow-up QST research and as specificationbased conformance checks. These tests exercise the standardized interfaces and check whether the refactored programs conform to the documented input constraints and expected outputs, thereby helping mitigate functional drift induced by refactoring. For a quasi-blind validation, two authors, independent of refactoring original programs and documenting program specifications, designed and executed 220 program-level unit tests across the 40 benchmark programs. They provided feedback based on the test results, further improving refactoring quality and program specifications. As a consequence, all provided test cases passed, achieving a statement coverage of 85%. This provides practical confidence that the refactored programs conform to the documented specifications on the exercised cases. Objectively, it is unrealistic to establish semantic equivalence to all original behaviors because the refactoring itself introduces reasonable changes to refine program testability. In detail, all tests were executed through standardized interfaces provided by Qolumbina, as illustrated in Figure 3. These interfaces expose a subroutine under test, classical arguments, quantum inputs and measurements, and execution configurations, like shots, backends, random seeds, and output qubits. This higher-abstraction interface design simplifies test construction by hiding low-level Qiskit API details. Both shot-based and shot-independent backends support test execution. The shot-based one, following the measurement-based execution commonly assumed in QST studies, supports one ideal simulator and 60 6
Preparation (①) from qolumbina.utils import int_to_gate_list, QuantumSystemSimulation, AnalyticIdealSimulation from qolumbina.programs.comparator import IntegerComparatorGreedy
Test inputs (②)
Pre-processing (③)
y, k = 5, 3 # Initial state = |5> and integer k = 3 geq = True # For comparison y >= k? n = 3 # 3 qubits for encoding y meas_basis = ["i"] * n + ["z"] * 1 # Pauli-Z for q3
init_state = int_to_gate_list(y, n) # ['x', 'i', 'x'] init_qubits = list(range(n)) # For q0, q1, q2 meas_qubit_mapping = {n: 0} # q3 -> c0
Configurations for measurements (④)
Configurations related to statevectors (④)
if_noisy = True # Use a noisy simulator random_seed = 42 # For reproducibility shots = 100 # 100 shots per test meas_backend = "fake_kyoto" # Simulate IBM hardware
stat_backend = "statevector" # Use state vector target_qubits = [n] # Partial state for q3
Shot-based simulation (⑤)
Shot-independent simulation (⑤)
ms_test = QuantumSystemSimulation( init_state=init_state, init_qubits=init_qubits, tested_subroutine= lambda: IntegerComparatorGreedy(n, k, geq), meas_basis=meas_basis, shots=shots, backend=backend_name, random_seed=random_seed, if_noisy=if_noisy, meas_mapping=meas_qubit_mapping )
sv_test = AnalyticIdealSimulation( init_state=init_state, init_qubits=init_qubits, tested_subroutine= lambda: IntegerComparatorGreedy(n, k, geq), backend=stat_backend, target_qubits=target_qubits )
{'0': 34, '1': 66}
Statevector([ 8.90187929e-18+0.00000000e+00j, -1.00000000e+00+3.06909191e-16j], dims=(2,))
Measurement outcomes (ms_test.test_output)
Partial statevector (sv_test.test_output)
Figure 3: A code example to run the benchmark quantum program integer_comparator_greedy through two backends
7
fake backends. The ideal simulator mainly targets theoretical analysis of computational procedure, while fake backends can better approximate characteristics of real quantum hardware. The shot-independent one, rather than simulating execution on physical quantum hardware, produces statevectors or unitary operators to enable direct checks of quantum states or transformations; this has recently gained attention in research [25] and is also used in Qiskit testing practice [30].
4
Empirical Study
4.1
Overview of Research Questions
To characterize testing-related properties of scalable quantum programs and outline implications for controlled experiments, we propose the following research questions (RQs): RQ1 What are benchmark programs’ functional properties? RQ1.1 To what extent do the programs with real-world provenance go beyond purely pedagogical and artificial instances? RQ1.2 What are the output characteristics of programs? RQ2 What are the scales of benchmark programs? RQ2.1 What is the development complexity of the programs? RQ2.2 What is the execution complexity of the programs? RQ3 What is the performance of adopting benchmark programs for controlled QST experiments? RQ3.1 What is the time cost of program execution? RQ3.2 What is the performance of employing benchmark programs for fault detection? Motivated by QST’s goal of checking the functional correctness of PUTs, RQ1 characterizes benchmark functionality through real-world provenance and output characteristics. Concerning that prior QST studies often used tutorial-style instances [19], mostly designed around artificial problems to demonstrate quantum advantages, RQ1.1 examines whether Qolumbina includes programs beyond pedagogical or artificial instances, thereby supporting empirical research on more practically relevant subjects. RQ1.2 investigates program outputs, since interpreting test outcomes requires knowing what expected output should be compared against observed behavior, which directly affects test oracle design. Test scalability is one crucial requirement that expects the test approaches to still work for large-scale and sophisticated programs. To this end, RQ2 evaluates whether Qolumbina provides suitable PUTs for studying test scalability. RQ2.1 assesses development complexity, as developers are generally more likely to introduce defects in programs with complex structures. RQ2.2 complements this view from the execution perspective by quantifying quantum circuit scale, a commonly considered factor in existing QST studies. RQ3 conducts two controlled experiments using Qolumbina on shot-based simulators to evaluate whether it supports methodology of existing QST empirical studies. In terms of test cost and effectiveness, we reuse MSTC [17] and DOSS [18] for feasibility analysis of Qolumbina because they are recent QST approaches with both available and functional artifacts, along with prior evaluations on scalable Qiskit programs. Across different backends, RQ3.1 follows MSTC by comparing execution time between two test-suite designs, while RQ3.2 uses DOSS as a comparably reliable test oracle to study reported failures on buggy benchmark variants.
4.2
RQ1: Functional Analysis
4.2.1
Data Annotation
Three authors with strong expertise in QSE performed the category annotation of each benchmark program in terms of their functionalities. Table 2 lists the four involved tasks, where T1, T2, and T3 are proposed to answer RQ1.1, while T4 is considered especially for RQ1.2. For T3 and T4 involving predefined categories, since no existing taxonomy was found to match such analysis for QST, we carefully reviewed program specifications and source codes to determine the candidate categories along with their definitions. Then, three authors independently annotated all the 40 programs, following a clear guideline provided in our artifact [5]. Fleiss’ κ [10] is used to measure the agreement among authors beyond two. The agreement scores for T1–T4 are respectively 0.77, 0.70, 0.73, and 0.81. For the multi-label tasks T3 and T4, we compute Fleiss’ κ on each binary label and report the support-weighted macro average. According to [14], all scores indicate substantial agreement. Furthermore, the authors organized an online discussion to resolve disagreements and then decide on the annotated metadata. 8
Table 2: Introduction of data annotation Task
Description
Return
T1
Determines whether the quantum algorithm of a program is primarily instructional. For example, an algorithm that often occurs in official tutorials or learning modules, and mainly illustrates theoretical principles and algorithmic advantages rather than practical usage is marked as “Yes”.
A Boolean tag (“Yes” or “No”)
T2
Identifies whether a quantum program is reusable as a subroutine of larger quantum algorithms or applications. Such a program does not implement a fully standalone function or complete application; instead, it is designed modularly, making it easy to integrate into a larger circuit.
A Boolean tag (“Yes” or “No”)
T3
Investigates what application domains the quantum program is designed for.
At least one candidate category
T4
Investigates what type of solution-related output is expected by solely executing a quantum program upon a simulated or physical quantum system.
At least one candidate category
4.2.2
RQ1.1: Beyond Pedagogical and Artificial Instances