QUTest: A Native Testing Framework for Quantum Programs José Campos [email protected] Faculdade de Engenharia da Universidade do Porto, Porto, Portugal LASIGE, Faculdade de Ciências, Universidade de Lisboa, Lisboa, Portugal
arXiv:2605.19736v1 [quant-ph] 19 May 2026
Abstract Quantum programs are often shared as OpenQASM 3 circuits, but tests are still written in host languages such as Python with Qiskit. We present QUTest, a native framework in which both programs and tests are standard .qasm files. Tests follow the Arrange / Act / Assert pattern, while configuration, runtime requirements, and assertions are encoded as pragma comments (//%), preserving compatibility with existing OpenQASM tools. QUTest provides 12 assertion types spanning deterministic, statistical, quantumstate, and structural checks, plus a linter and an environment-aware mode for running the same test across selected runtime versions in isolated environments. Its CLI supports automatic test discovery, runtime compatibility checks, and XML reports for continuous integration. We describe the pragma language, implementation, and a planned evaluation using coverage and mutation testing. QUTest is available at https://github.com/QBugs/qutest. Video demo: https://youtu.be/FvgvsiAXuW0.
1
Introduction
Software testing is a cornerstone of classical software engineering. Testing frameworks such as JUnit for Java and PyTest for Python, allow developers to write and execute tests in the same language as their production code. This co-location principle—that code and tests share a language, toolchain, and repository—is widely regarded as essential for maintainability and readability [1]. Quantum programs are often written in circuit languages, most notably OpenQASM 3 [4]. OpenQASM 3 provides a frameworkneutral textual format for describing quantum circuits, yet such programs still lack a native testing framework. Today, testing a quantum program written in OpenQASM typically requires a developer to write a separate test harness in, for example, a Python-based SDK such as Qiskit [5] and to use the PyTest framework. The Python test harness in Listing 2 for the Bell-state program in Listing 1 must (1) load the program with Qiskit [5], (2) configure the simulator backend, (3) transpile the program, (4) execute the program on the simulator, (5) extract the measurement counts, and (6) apply a statistical test, e.g., a chisquared goodness-of-fit test at the 95% confidence level, to those counts. The result is a test suite in which the code under test resides in one language (OpenQASM), whereas the test harness resides in another (Python), is coupled to a specific SDK version (Qiskit), and is fragile with respect to API changes that have nothing to do with the quantum algorithm itself (e.g., [2, 10]). This language gap introduces several problems. First, readability suffers: understanding a test requires familiarity with both OpenQASM semantics and the Qiskit API. Second, portability is reduced: tests written against Qiskit do not transfer to Cirq [3], Pennylane [6], or other SDKs without rewriting the host-language glue. Third, maintenance costs increase: SDK version upgrades can break
Listing 1: Bell-state program written in OpenQASM 3. 1 2 3 4 5 6 7
OPENQASM 3; include "stdgates.inc"; qubit[2] q; bit[2] m; h q[0]; cx q[0], q[1]; m = measure q;
Listing 2: Test harness for Listing 1 written in Python. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
from pathlib import Path from scipy.stats import chisquare from qiskit import qasm3, transpile from qiskit_aer import AerSimulator def test_distribution(): # Arrange circuit = qasm3.load(Path("bell.qasm")) # Act backend = AerSimulator(seed_simulator=42) transpiled = transpile(circuit, backend=backend) result = backend.run(transpiled, shots=10_000).result() counts = result.get_counts() # Assert EXPECTED = {"00": 0.5, "11": 0.5} unexpected = sum(v for k, v in counts.items() if k not in EXPECTED) assert unexpected == 0 observed = [counts.get(k, 0) for k in EXPECTED] expected = [10_000 * p for p in EXPECTED.values()] _, p_value = chisquare(observed, expected) assert p_value >= 0.05
the test harness even when the quantum algorithm itself is unchanged [13]. Fourth, the barrier to entry rises: researchers who write OpenQASM should not need Python expertise merely to verify that a circuit produces the correct output distribution. We present QUTest, a testing framework that eliminates this gap. In QUTest, both the software under test and the test cases live in standard .qasm files. Test are defined as OpenQASM 3 subroutines (def) and follow the familiar Arrange/Act/Assert pattern [1]: Arrange prepares qubits and classical registers, Act applies the quantum algorithm and measures, and Assert checks the measurement outcomes. Backend configuration (shots, seed, simulator type, and runtime SDK/version requirements) and assertions (expected output, distribution metrics, entanglement witnesses, and structural budgets) are expressed as pragma comments prefixed with //%. Because these pragmas are syntactically valid OpenQASM comments, every QUTest file remains valid OpenQASM 3 syntax. The contributions of this paper are as follows: ⋆ A pragma-based annotation language for expressing QASM-
native quantum tests (Section 2). ⋆ The QUTest tool, which implements this language through a
command-line workflow for test discovery, static linting, execution, assertion evaluation, environment-aware runtime selection, and CI-oriented reporting (Section 3).
Campos
Listing 3: Bell-state program and one test case, both written in OpenQASM 3 and using QUTest’s pragmas. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
OPENQASM 3; include "stdgates.inc"; def bell(qubit[2] q) { h q[0]; cx q[0], q[1]; } def test_distribution() { // Configuration pragma (see Section 2.1) //% shots 10000 //% seed 42 //% backend ideal // Arrange qubit[2] q; bit[2] m; // Act bell(q); m = measure q;
}
2
// Assert: Assertion Pragmas (see Section 2.2) //% expect distribution ref {"00": 0.5, "11": 0.5} //% expect distribution chi2 >= 0.05
Pragma Language Design
The QUTest pragma language extends OpenQASM 3 files with structured metadata embedded in comments. Each pragma line begins with the prefix //%, followed by a directive. Pragmas fall into two groups: configuration pragmas, which control how a circuit is executed (Section 2.1), and assertion pragmas, which define pass/fail criteria (Section 2.2). Annotation-based test metadata is common in classical frameworks. JUnit uses Java annotations (@Test, @BeforeEach), whereas PyTest uses decorators (@pytest.mark). Listing 3 shows the Bell-state program, previously presented in Listing 1, wrapped in a function named bell (lines 4–7). It also shows the same test (test_distribution in Listing 2), written directly in OpenQASM 3 using QUTest’s pragmas (lines 9–26). The test first declares the quantum and classical registers (lines 16 and 17), calls the program under test (lines 20 and 21), and then applies the same chi-squared goodness-of-fit test at the 95% confidence level as the Python baseline (lines 24 and 25).
(4) Runtime: qiskit. The runtime pragma specifies which quantum software runtime executes the test. In the current implementation, QUTest supports only Qiskit. //% runtime qiskit
(5) Runtime version: a quoted, comma-separated list of exact runtime versions. If this pragma is absent, the test runs once in the active Python environment. When it is present, the test is executed independently under each listed version. //% runtime_version "2.3.0,2.4.0"
2.2
Assertion Pragmas
Assertion pragmas are central to QUTest. The framework currently supports 12 assertion types organized into four categories (e.g., lines 24 and 25 in Listing 3). Although many additional assertion mechanisms have been proposed, e.g., [11], implementing all of them is beyond the scope of a prototype such as QUTest. 2.2.1 Deterministic Output. The output assertion checks whether every shot produces the same classical value: //% expect output m == 1
This assertion is the quantum analogue of a classical equality check and is appropriate for circuits that deterministically prepare a computational-basis state (e.g., a sequence of Pauli-X gates). 2.2.2 Statistical Distribution. For probabilistic circuits, QUTest supports assertions over the measured output distribution, e.g., line 24 in Listing 3. The following pragmas compare the empirical distribution against the provided reference using distribution distances or statistical goodness-of-fit tests. Each assertion accepts a comparison operator and a threshold. Í (1) Total variation distance: 𝛿 (𝑃, 𝑄) = 12 𝑥 |𝑃 (𝑥) − 𝑄 (𝑥)|. //% expect distribution total_variation_distance <= 0.03
(2) Hellinger distance: 𝐻 (𝑃, 𝑄) = √1
2
√︃ √︁ Í √︁ 2 𝑥 ( 𝑃 (𝑥) − 𝑄 (𝑥)) .
//% expect distribution hellinger <= 0.05
(3) Kullback–Leibler divergence: 𝐷 KL (𝑃 ∥𝑄) =
2.1
Configuration Pragmas
Configuration pragmas set the execution environment for a test function. QUTest supports five such directives (e.g., lines 11–13 in Listing 3). (1) Measurement shots: 1024 (default). //% shots ⟨𝑁 ⟩
(2) Simulator seed: random (default). //% seed ⟨𝑁 ⟩
(3) Backend: ideal (default), noisy, or hardware. The ideal backend uses a noiseless statevector simulator. The noisy backend applies a depolarising noise model with a single-qubit error rate of 𝑝 1 = 10−3 and a two-qubit error rate of 𝑝 2 = 10−2 , representative of current superconducting-qubit hardware. The hardware backend is reserved for future integration with cloudbased quantum processors. //% backend ⟨𝐵⟩
𝑃 (𝑥 ) 𝑥 𝑃 (𝑥) ln 𝑄 (𝑥 ) .
Í
//% expect distribution kl <= 0.10 2 Í 𝑥) (4) Chi-squared goodness-of-fit test: 𝜒 2 = 𝑥 (𝑂𝑥 𝐸−𝐸 , where 𝑥 𝑂 𝑥 is the observed count and 𝐸𝑥 = 𝑁𝑄 (𝑥) is the expected count under the reference distribution, and compares the resulting p-value.
//% expect distribution chi2 >= 0.05
2.2.3 Quantum-State Properties. This category contains eight assertion types that probe different aspects of the output. (1) Marginal probability: Assert the probability of a single qubit outcome. //% expect marginal m[0] p1 ~= 0.5 atol=0.05
(2) Pauli-𝑍 observable: Compute ⟨𝑍𝑖 𝑍 𝑗 · · · ⟩ from measurement counts, with each bitstring contributing (−1) parity at the specified qubit positions. //% expect observable "Z0 Z1" ~= 1.0 atol=0.05
QUTest: A Native Testing Framework for Quantum Programs
(3) Shannon entropy: Assert the entropy (in bits) of the output Í distribution: 𝐻 = − 𝑥 𝑃 (𝑥) log2 𝑃 (𝑥). //% expect entropy ~= 1.0 atol=0.05
(4) Pearson correlation: Assert the classical correlation between two output bits, which ranges from −1 to +1. //% expect correlation m[0] m[1] ~= 1.0 atol=0.05
(5) Bitstring probability: Assert the probability of a specific outcome without requiring a full reference distribution. //% expect probability "00" ~= 0.5 atol=0.05
(6) Most-frequent outcome: Assert that the most frequently observed bitstring equals an expected value—a simple argmax check with no statistical test. //% expect most_frequent "00"
(7) Classical fidelity: Compute the squared Bhattacharyya coeffi2 Í √︁ cient, 𝐹 = 𝑥 𝑃 (𝑥)𝑄 (𝑥) , between the measured distribution and an automatically obtained ideal simulation. //% expect fidelity >= 0.95
(8) Entanglement witness: Verify that two qubit partitions are entangled by computing the von Neumann entropy 𝑆 (𝜌𝐴 ) = −tr(𝜌𝐴 log2 𝜌𝐴 ) of the reduced density matrix obtained via partial trace. Entanglement is confirmed when 𝑆 (𝜌𝐴 ) > 0. //% expect entangled [0] [1]
2.2.4 Structural Assertions. Are evaluated on the transpiled circuit before execution, allowing developers to enforce hardware constraints without consuming simulator time. • Gate-set membership: Assert that all gates in the transpiled circuit belong to a specified set. //% expect gateset subset_of [h, cx, rz]
• Depth: Assert an upper bound on the transpiled circuit depth. //% expect depth <= 10
2.3
Comparison Operators
Assertions support two comparison forms. Approximate equality (~= value atol=tolerance) checks |𝑎𝑐𝑡𝑢𝑎𝑙 − 𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 | ≤ atol for marginals, observables, entropy, correlation, probability, and fidelity. Standard operators (<, <=, ==, >, >=, !=) express exact or ordered checks for outputs, distribution metrics, and depth bounds. Entropy supports both.
3
QUTest
This section describes the QUTest framework.
The linter scans .qasm files without executing circuits. It ensures that //% directives appear inside def test*() functions, parses all directives, and reports line-level diagnostics with repair hints. This mitigates a key drawback of comment-embedded pragmas: they preserve OpenQASM compatibility but lack type checking.
3.3
Each .qasm file is parsed to extract function definitions. Names beginning with test are treated as tests; all others are considered part of the software under test. The pragma parser extracts //% lines from each test body and converts them into objects. At the momement, Qiskit’s loader does not support OpenQASM 3 subroutines (def). Thus, to address this limitation, QUTest applies a function-inlining pass before handing the source to Qiskit. Each call in a test body to an existing subroutine is replaced by the callee body, substituting formal parameters with actual arguments. This process repeats transitively until all calls are expanded.
3.4
3.5
Execution
Each worker loads the inlined OpenQASM source and transpiles it for the configured backend with transpile(). Structural assertions (gate-set membership and circuit depth) are checked on the transpiled circuit before execution. It then runs the circuit on the selected AerSimulator with the configured shots and optional seed. Statevector-dependent tests (e.g., the entanglement witness) use a separate simulation of a measurement-free copy.
Assertion Evaluation
Discovery
QUTest recursively scans a directory (or a single file) for .qasm files and identifies all subroutines whose names begin with test.
3.2
Environment-Aware Runtime Execution
QUTest splits ordinary tests from environment-aware tests, and groups the latter by requested version. For each version, it creates or reuses a managed virtual environment under .qutest/runtimes/ <runtime>/<version>/, installs the local QUTest package and the exact runtime version. Before running any test, each managed environment must pass a runtime compatibility probe. The probe loads a minimal OpenQASM circuit, selects a simulator backend, transpiles and executes the circuit, and checks a deterministic oracle. If it fails, QUTest reports a framework/runtime compatibility error rather than a user-test failure. In effect, runtime_version means “run this test under that runtime version if QUTest supports it.” Each version group runs in a worker subprocess using that environment’s Python interpreter. If runtime_version is present, ordinary tests also also run in a subprocess under the active interpreter, while the parent process handles discovery, environment setup, worker invocation, and result aggregation. This isolation prevents import leakage across environments. Any setup, probe, runtime-test, or worker failure makes QUTest exit non-zero.
3.6 3.1
Parsing and Function Inlining
Static Linting
Because QUTest encodes tests configuration and oracles as comments, standard OpenQASM parsers cannot detect malformed pragmas. To expose such errors before execution, QUTest provides a static linter, qutest lint.
Each assertion pragma has a dedicated evaluator that receives measurement counts, a transpiled circuit, a statevector, or ideal counts, and returns an AssertionResult with a status (pass, fail, or error), a message, and the actual and expected values.
3.7
Reporting
QUTest supports two reporting modes. Console reporter: prints coloured checkmarks (✓) for passing tests and crosses (×) for failures. In verbose mode, all assertion details are shown; in the default
Campos
mode, only failing assertions are expanded. XML reporter: generates an XML file following the JUnit/xUnit schema, suitable for CI systems such as Jenkins, GitHub Actions, and GitLab CI.
3.8
CLI
QUTest is a Python 3 package built on Qiskit ≥1.0, Qiskit Aer ≥0.13, openqasm3, and qiskit-qasm3-import, with NumPy and SciPy for numerics. Its CLI exposes lint for the static checks in Section 3.2 and run for the execution workflow in Sections 3.3 to 3.7.
3.9
Design Trade-offs
of the test specification. The same QASM test can thus be run across Qiskit versions to expose behavioral drift. Examples include the QAOA migration regression reported by Cardinal et al. [2] and Qiskit issue #12124 [10], where qiskit.qasm2.loads failed to recognize gates from qelib1.inc.
6
Related Work
QuCAT [15] and QuSBT [14] generate Python-based test inputs, and QuraTest [16] and NovaQ [9] generate OpenQASM test circuits. These tools are complementary to QUTest: they generate inputs, whereas QUTest provides a QASM-native format for complete test cases, including execution configuration and assertions. In particular, generated inputs still need a harness to load, execute, and check expected behavior; QUTest provides that missing layer. Future tools could combine both by generating native QUTest tests.
Pragmas vs. language extension. Rather than adding native OpenQASM 3 assertions, QUTest uses comment pragmas: no grammar or parser changes and compatibility with existing tools, at the cost of weaker tooling (no type checking, limited IDE support). The linter and a language-server extension could help. Inlining vs. native support. Because the current Qiskit importer cannot load def subroutines, QUTest inlines them before execution on the runtime. Once parsers support def, this pass can disappear. The current inliner supports simple parameter substitution, but not nested-scope name collisions.
We have presented QUTest, the first framework that enables quantum developers to write test directly in OpenQASM 3. Future work includes parameterised tests, support for other quantum SDKs, e.g., Cirq [3] and Pennylane [6], and real hardware backends.
4
References
Planned Evaluation
We plan to evaluate QUTest along two dimensions: developer effort and test effectiveness. Developer Effort. The first study would assess whether developers can write QASM-native tests with QUTest more easily than Python-based harnesses. In a within-subject design, participants would test small OpenQASM programs from natural-language tasks, with task order counterbalanced. We would collect objective measures (e.g., completion time, number of files and lines of test code, edit-run-debug cycles, and syntax/oracle mistakes) and subjective measures from questionnaires (e.g., perceived difficulty, confidence, readability, and required host-SDK familiarity). Together, these data would show whether QASM-native pragmas reduce test-writing effort or introduce usability costs of their own. Test Effectiveness. The second study would evaluate the adequacy of Python-based and QUTest test suites at exercising quantum programs structure [8] and detecting faults [7, 12]. This will allow us to assess whether reduced test-writing effort weakens fault detection or whether QASM-native tests enable more effective oracles.
5
Discussion
Python-Based Testing vs. QUTest. Comparing Listings 1 and 2 with Listing 3 highlights the difference in development overhead. The former requires (a) two languages (OpenQASM and Python) and three Python libraries (Qiskit, SciPy, and PyTest), whereas the latter requires only OpenQASM and a single CLI tool (QUTest); and (b) uses 7 + 18 = 25 lines to implement the program under test and one single-assertion test case. The latter uses 21 lines for the same program and test. Beyond the listing, QUTest also allows a single test to target multiple runtime environments (see Section 2.1) and express multiple assertions as one-line pragmas (see Section 2.2). Who is QUTest for? Researchers and developers who write OpenQASM and do not want—or may not know how—to write test code in a quantum SDK just to verify expected behavior. SDK maintainers. With the runtime and runtime_version pragmas (see Section 3.4), QUTest treats the execution environment as part
7
Conclusion and Future Work
[1] Kent Beck. 2002. Test-Driven Development: By Example. Addison-Wesley Prof. [2] Julien Cardinal, Imen Benzarti, Ghizlane El boussaidi, and Christophe Pere. 2025. Migrating QAOA from Qiskit 1.x to 2.x: An experience report. arXiv:2512.08245 [cs.SE] https://arxiv.org/abs/2512.08245 [3] Cirq Developers. 2026. Cirq: An open source framework for programming quantum computers. https://quantumai.google/cirq Accessed: 2026-05. [4] Andrew Cross and et. al. 2022. OpenQASM 3: A Broader and Deeper Quantum Assembly Language. ACM Transactions on Quantum Computing 3, 3, Article 12 (Sept. 2022), 50 pages. doi:10.1145/3505636 [5] Gadi Aleksandrowicz et al. 2019. Qiskit: An Open-source Framework for Quantum Computing. (Jan. 2019). doi:10.5281/zenodo.2562111 [6] Ville Bergholm et al. 2022. PennyLane: Automatic differentiation of hybrid quantum-classical computations. https://arxiv.org/abs/1811.04968 [7] Daniel Fortunato, José Campos, and Rui Abreu. 2022. Mutation Testing of Quantum Programs: A Case Study With Qiskit. IEEE Transactions on Quantum Engineering 3 (2022), 1–17. doi:10.1109/TQE.2022.3195061 [8] Daniel Fortunato, José Campos, and Rui Abreu. 2026. Probabilistic Condition, Decision and Path Coverage of Circuit-based Quantum Programs. https://arxiv. org/abs/2604.26609 [9] Tiancheng Jin, Shangzhou Xia, and Jianjun Zhao. 2025. NovaQ: Improving Quantum Program Testing through Diversity-Guided Test Case Generation. In IEEE/ACM International Conference on Automated Software Engineering (ASE). 3881–3885. doi:10.1109/ASE63991.2025.00335 [10] Kanguk Lee. 2024. Standard Gates not recognized by qiskit.qasm2.loads. https: //github.com/Qiskit/qiskit/issues/12124 Accessed: 2026-05. [11] Yuechen Li, Kai-Yuan Cai, and Beibei Yin. 2026. A Dynamic Test Oracle for Quantum Programs With Separable Output States. IEEE Transactions on Software Engineering 52, 4 (2026), 1568–1591. doi:10.1109/TSE.2026.3670211 [12] Eñaut Mendiluze, Shaukat Ali, Paolo Arcaini, and Tao Yue. 2021. Muskit: A Mutation Analysis Tool for Quantum Software Testing. In IEEE/ACM International Conference on Automated Software Engineering (ASE). 1266–1270. doi:10.1109/ASE51524.2021.9678563 [13] Matteo Paltenghi and Michael Pradel. 2022. Bugs in Quantum computing platforms: an empirical study. Proc. ACM Program. Lang. 6, OOPSLA1, Article 86, 27 pages. doi:10.1145/3527330 [14] Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022. QuSBT: SearchBased Testing of Quantum Programs. In ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-C). 173–177. doi:10.1145/ 3510454.3516839 [15] Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2023. QuCAT: A Combinatorial Testing Tool for Quantum Software. In IEEE/ACM International Conference on Automated Software Engineering (ASE). 2066–2069. doi:10.1109/ASE56229. 2023.00062 [16] Jiaming Ye, Shangzhou Xia, Fuyuan Zhang, Paolo Arcaini, Lei Ma, Jianjun Zhao, and Fuyuki Ishikawa. 2023. QuraTest: Integrating Quantum Specific Features in Quantum Program Testing. In IEEE/ACM International Conference on Automated Software Engineering (ASE). 1149–1161. doi:10.1109/ASE56229.2023.00196