arXiv CSopen access

Robust Mutation Analysis of Quantum Programs Under Noise

Robust Mutation Analysis of Quantum Programs Under Noise SOPHIE FORTZ∗ , King’s College London, UK EÑAUT MENDILUZE USANDIZAGA∗ , Simula Research Laboratory and Oslo Metropolitan University,

arXiv:2605.13279v1 [cs.SE] 13 May 2026

Norway

SHAUKAT ALI, Simula Research Laboratory and Oslo Metropolitan University, Norway PAOLO ARCAINI, National Institute of Informatics, Japan MOHAMMAD REZA MOUSAVI, King’s College London, UK Mutation analysis has long been used in classical software testing and has recently been adopted for assessing the robustness of quantum software testing techniques. However, existing studies assume ideal, noiseless execution, overlooking the impact of quantum hardware noise. In this paper, we present an empirical study of noise-aware mutation analysis for quantum programs. We analyze how noise affects mutant detection using 41 quantum programs, executed on noiseless and noisy simulators emulating three IBM devices with different noise profiles. We compare several distance metrics and thresholding strategies to evaluate mutant detection under realistic noise. Our results show that noise significantly alters the behavioral distance between programs and mutants, making equivalent mutants harder to distinguish from real faults. Density-matrix metrics achieve the best discrimination, with misclassification rates up to 16.77%, but are not accessible on real hardware. Among practical alternatives, output-distribution metrics reach up to 73.03% accuracy and 74.89% F1-score. Noise-specific thresholds further improve detection compared to noiseless thresholds. We also find that noise effects correlate more with algorithm and circuit characteristics than with mutation types. Overall, our results highlight the need to adapt mutation analysis, and more generally quantum program comparison, to the noise profiles of target quantum devices. CCS Concepts: • Hardware → Quantum computation; • Theory of computation → Quantum computation theory; • Software and its engineering → Empirical software validation; Software verification and validation; Software testing and debugging. Additional Key Words and Phrases: Mutation Analysis, Quantum Computing, Quantum Noise ACM Reference Format: Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi. 2026. Robust Mutation Analysis of Quantum Programs Under Noise. ACM Trans. Softw. Eng. Methodol. X, X, Article X (February 2026), 49 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1

Introduction

Mutation analysis, a well-established technique in classical software testing [55], has only recently been explored for quantum software testing [23, 45]. The core idea is to introduce small changes ∗ Both authors contributed equally to this research.

Authors’ Contact Information: Sophie Fortz, [email protected], King’s College London, London, UK; Eñaut Mendiluze Usandizaga, [email protected], Simula Research Laboratory and Oslo Metropolitan University, Oslo, Norway; Shaukat Ali, [email protected], Simula Research Laboratory and Oslo Metropolitan University, Oslo, Norway; Paolo Arcaini, [email protected], National Institute of Informatics, Tokyo, Japan; Mohammad Reza Mousavi, [email protected], King’s College London, London, UK. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1557-7392/2026/2-ARTX https://doi.org/10.1145/nnnnnnn.nnnnnnn ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:2

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

(mutations) into a program to assess the quality of a test suite and its ability to detect defects. In the quantum domain, this typically involves modifying a quantum circuit (e.g., adding, removing, or replacing gates) and observing whether the mutation produces a detectable behavioral change. Most existing studies on quantum mutation analysis evaluate programs in idealized, noiseless simulators [23, 45]. While these analyses have laid important groundwork, they fail to capture the behavior of quantum software executing on real quantum computer, where noise and decoherence unavoidably affect execution. As a result, it remains unclear how well current mutation analysis techniques translate into real quantum computers, or how noise impacts the ability to distinguish correct from faulty program behavior. In this paper, we systematically study how different metrics perform in noisy settings, and how detection thresholds can be adapted to distinguish between noise-induced variations and genuine mutant-induced behavioral differences. Rather than attempting to mitigate noise, we investigate how testing methods can be made more resilient to it, acknowledging its presence as an unavoidable feature of current quantum hardware. Understanding how to compare quantum program executions is fundamental for evaluating correctness, robustness, and reliability. In mutation analysis, these comparisons determine whether a mutant behaves differently from its original program. However, in noisy environments, small stochastic fluctuations can blur this distinction, making the choice of comparison metric and decision threshold crucial for reliable detection. By analyzing executions across both noiseless and noisy simulators, our study provides an empirical foundation for understanding how noise influences mutation analysis and, more broadly, behavioral comparison in quantum software. We conduct a comprehensive empirical evaluation using six algorithms implemented in multiple circuit sizes ranging from 2 to 8 qubits, for a total of 41 circuits. From these, we generated 2,224 mutants, comprising 1,054 equivalent and 1,170 non-equivalent variants. Test inputs include both classical and quantum states, with 2 × (2# qubits) inputs for smaller circuits and 2# qubits for larger circuits. We systematically evaluate five distance metrics under four threshold strategies and examine the impact of nine circuit-, algorithm-, and mutation-related characteristics. Each configuration is tested across three distinct noise models derived from IBM quantum devices, as well as in a noiseless simulator, providing a broad empirical foundation for analyzing mutant detectability under realistic noise conditions. Our results show that quantum program mutation analysis is highly sensitive to noise, both in terms of metric performance and threshold calibration. Noise alters the behavioral distance between quantum programs and their mutants, making the distinction between equivalent and non-equivalent cases more challenging. Under realistic noise models, the distance distributions deviate markedly from noiseless baselines: equivalent mutants exhibit greater dispersion, while non-equivalent ones become more tightly clustered. These effects intensify with stronger noise models, demonstrating that the sensitivity of distance-based analyses depends on both the level and nature of noise—rendering traditional, noiseless assumptions unreliable. Among the evaluated metrics, density matrix–based measures such as Trace distance and Fidelity achieve the clearest separation between mutant types, but are computationally expensive and therefore only suited for experiments involving simulator executions. Output distribution–based and expectation value–based metrics, while applicable to real hardware, tend to blur behavioral distinctions under noise, revealing a trade-off between precision and practicality. Thresholds, in turn, define the level of behavioral deviation considered acceptable when deciding mutant equivalence. Conventional, noiseless thresholds prove ill-suited for noisy environments, as they fail to account for noise-induced fluctuations and often inflate false positives. To address this, we introduce noise-specific thresholds that calibrate detection sensitivity to each noise model. These adaptive thresholds significantly improve detection reliability across metrics and simulators, ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:3

yielding more consistent classification. Although their optimal values depend on the chosen metric and noise profile, they consistently outperform generic thresholds, underscoring the need for noise-aware calibration in quantum mutation analysis. Beyond metric and threshold effects, results reveal that noise interacts unevenly with circuit and mutation characteristics. Circuit size and gate depth moderately influence noise sensitivity, whereas algorithmic factors (particularly the algorithm type and output structure) exert the strongest impact, shaping each program’s unique noise profile. By contrast, mutation-level characteristics contribute minimally, suggesting that noise can overshadow the behavioral impact of injected faults. Together, our findings underscore the importance of contextual, noise-aware mutation analysis and motivate the development of adaptive frameworks that jointly calibrate metrics and thresholds to specific noise and algorithmic conditions. Beyond mutation analysis, these insights extend more broadly to comparison of executions of quantum programs and program analysis under realistic noise, offering practical guidance for researchers and tool developers striving to design more robust quantum software development and testing practices. Contributions. This paper provides the first empirical study on how noise impacts mutation analysis in quantum software testing. The key contributions of this paper include: • A curated dataset of 41 quantum programs implementing six algorithms, each accompanied by its dedicated test suite and corresponding set of mutants. • An extensive experimental study, executing all programs and mutants across four simulators: a noiseless simulator and three noisy simulators based on IBM devices (Brisbane, Kyiv, and Sherbrooke). • A systematic evaluation of 20 output assessment configurations, combining five distance metrics with four thresholding strategies, and empirical insights into selecting effective combinations for reliable mutation detection under noise. • A detailed analysis of how noise affects mutant detection and how this impact correlates with circuit-, algorithm-, and mutation-level characteristics. To support open science, all data, implementations, and results are available at [24]. Paper structure. Sect. 2 introduces the necessary background. Sect. 3 defines the research questions and describes the design of our empirical study. Sect. 4 presents the experimental findings, and Sect. 5 discusses their implications, outlines future research directions, and addresses potential threats to validity. Sect. 6 reviews related work on quantum testing and quantum noise. Finally, Sect. 7 summarizes the main contributions and key insights of the paper. 2

Background

This section provides necessary background concepts. Sect. 2.1 presents the basics of quantum computing; Sect. 2.2 defines key mutation analysis concepts and discusses open research challenges in the field; and Sect. 2.3 reviews recent adaptations of mutation analysis for quantum programs. 2.1 Quantum Computing As quantum computing continues to evolve, understanding the principles that differentiate it from classical computing becomes increasingly important. At the heart of quantum computation is the quantum bit or qubit, the fundamental unit of quantum information. A qubit’s state can be described as a state vector in a two-dimensional complex Hilbert space. Any pure qubit state can be written as |𝜓 ⟩ = 𝑎 |0⟩ + 𝑏 |1⟩, where 𝑎, 𝑏 ∈ C and |𝑎| 2 + |𝑏 | 2 = 1 [51]. Unlike a classical bit, which can exist only in a state of 0 or 1, a qubit can exist in a superposition of both |0⟩ and |1⟩ states simultaneously. Superposition is one of the most fundamental principles underlying quantum computation, enabling quantum systems to process information in ways that can transcend classical capabilities, forming ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:4

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

(a) Quantum circuit components.

(b) Circuit Under Test (CUT)

Fig. 1. Quantum circuit example consisting of four qubits (i.e., 𝑞[0] to 𝑞[3]) and a group of four classical bits, together denoted as 𝑐 4 .

one of the foundation for quantum speedup. Any measurement collapses a qubit’s state from superposition into a definite classical value (i.e., either 0 or 1) [77]. This collapse is crucial in quantum computing, as it is used to obtain measurement results from quantum operations [64]. Another key concept in quantum computing is entanglement. When qubits are entangled, they exhibit correlations such that the state of each qubit cannot be described independently of the others. For instance, measuring the state of one qubit immediately determines the state of its entangled partner, regardless of the distance separating them [78]. 2.1.1 Quantum programs as circuits. Quantum programs are usually represented as quantum circuits. Fig. 1a illustrates an entanglement circuit composed of four qubits. In quantum circuits, quantum gates are used as operations to modify the state of a qubit. Gate operations can involve either single-qubit or multi-qubit operations. A single-qubit gate, such as the Hadamard (𝐻 ) gate, operates on a single qubit. The 𝐻 gate transforms a qubit from a classical state into an equal superposition of |0⟩ and |1⟩. In contrast, multi-qubit gates act on two or more qubits. For example, the Controlled NOT (𝐶𝑁𝑂𝑇 ) gate uses one qubit as the control (represented by a small filled circle) and another one as the target (indicated by a plus sign in a circle). 𝐶𝑁𝑂𝑇 flips the state of the target qubit if the control qubit is in state |1⟩. Otherwise, if the control qubit is in state |0⟩, the target qubit remains unchanged. At the end of the quantum circuit, measurements are performed to transform the quantum states into classical states. 2.1.2 Quantum Measurements. Measurements are performed with respect to a chosen basis, which determines how the quantum state is observed. These are formalized through observables, which are Hermitian operators (i.e., matrices that are equal to their own conjugate
transpose) representing 0 measurable quantities [51]. For instance, the Pauli-𝑍 matrix 𝑍 = 10 −1 defines the so-called 𝑍 basis, which is the default basis to measure whether a qubit is in state |0⟩ or |1⟩ (e.g., in the Qiskit measure function1 ). Measurements project the quantum state onto one of the basis vectors, causing the qubits to collapse from their superposition into definite classical states. Observables are central to extracting information from quantum systems, since the internal quantum state itself—e.g., the amplitudes 𝑎 and 𝑏 in |𝜓 ⟩ = 𝑎 |0⟩ + 𝑏 |1⟩—cannot be directly observed. 2.1.3 Inner products and orthogonality. The inner product between quantum states is central to understanding quantum measurements and distinguishability. Given states |𝑣⟩ and |𝑤⟩, their inner product is ⟨𝑣 |𝑤⟩ = 𝑣 †𝑤, where † denotes the conjugate transpose. For a complex matrix or vector 𝐴, the operation 𝐴† involves transposing 𝐴 and taking the complex conjugate of each entry. Two states are orthogonal if ⟨𝑣 |𝑤⟩ = 0, meaning they can be perfectly distinguished by a measurement. 1 https://quantum.cloud.ibm.com/docs/en/guides/measure-qubits

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:5

Non-orthogonal states, by contrast, cannot be distinguished without uncertainty, reflecting another fundamental difference from classical computing. 2.1.4 Quantum Software Testing. Software testing consists in evaluating a program to ensure that it behaves as expected and meets its requirements. As mentioned before, quantum computing introduces unique principles that need adaptations of testing approaches. In classical software testing, we refer to the system being tested as the system under test (SUT), while, in quantum computing, this is known as the circuit under test (CUT). A quantum circuit can be divided into three main components, as shown in Fig. 1b. The first component is the initialization stage, which provides the input to the circuit. This input may consist of a classical state, or, alternatively, a more complex input can be prepared by employing another quantum circuit. The second component is the circuit itself, representing the computation. Finally, the measurement stage outputs the circuit’s result. As quantum computations are inherently non-deterministic, testing typically relies on multiple executions (shots) to obtain a distribution over the possible classical outputs. 2.1.5 Quantum Noise. Unlike classical bits, qubits are extremely sensitive to environmental and hardware-induced disturbances. Despite their potential, today’s quantum systems are noisy. Quantum computations are subject to various sources of noise that can disrupt or corrupt quantum states. A primary source is decoherence, which occurs when qubits interact with their surrounding environment, leading to information loss and state degradation [3, 12, 65]. This can be caused by magnetic fields or radiations for example. Another source is crosstalk noise, where unintended interactions between qubits during gate execution introduce computational errors [9, 62, 68]. Additionally, while calibration of quantum gates is necessary to fine-tune gate parameter’s optimization and error minimization, calibration might introduce subtle but compounding errors such as unintended phase shifts or amplitude changes [10]. These noise effects can manifest at any point during a quantum circuit’s execution and tend to accumulate, ultimately impacting the reliability of the final measurement outcomes [7]. A significant body of research has focused on addressing errors at the physical layer of quantum computing, including gate-level calibration [10, 37, 76, 81] and qubit stabilization [28, 34, 40, 66] for instance. These efforts are crucial for advancing quantum hardware. In contrast, our work operates at the software level, focusing on techniques that remain applicable within current hardware constraints. By emphasizing software-level assessment and analysis, we aim to support practical, near-term quantum applications where noise is unavoidable but must be accounted for through algorithmic and methodological design to noise mitigation rather than physical fault tolerance. To study and mitigate quantum noise effects, researchers use noise models as abstract representations of common physical errors. Popular examples include depolarizing noise, amplitude damping, and phase damping [46]. These models allow for controlled simulation of realistic noise behaviors and are essential for evaluating how noise impacts both original and mutated quantum programs. To formally capture the uncertainty and partial knowledge inherent in noisy quantum systems, quantum mechanics uses the density matrix formalism [51]. A density matrix 𝜌 represents a Í probabilistic mixture of pure states as 𝜌 = 𝑖 𝑝𝑖 |𝜓𝑖 ⟩ ⟨𝜓𝑖 |, where each |𝜓𝑖 ⟩ is a pure state and 𝑝𝑖 is its associated probability. This formulation supports reasoning about both pure states (described by a single state vector) and mixed states (statistical ensembles), the latter arising naturally from noise or incomplete information. 2.2

Mutation Analysis

Mutation analysis is used to evaluate and enhance the quality of software testing techniques [4, 25, 36, 55]. The primary goal of mutation analysis is to evaluate the effectiveness of a test suite by measuring its ability to detect artificially injected faults–called mutations–in the original code. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:6

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

These mutations generate different variants of the program, called mutants. To produce these mutants, various mutation operators can be applied. For instance, in classical software testing, conditional statements may be altered by modifying their logical operators, such as changing an “AND” to an “OR”, effectively introducing an artificial bug into the original program. Once a set of mutants is generated, the test suite is executed on each mutant to determine whether it is detected. For a mutant to be detected, a test must reach the mutated statement and then activate the mutation that will affect the program state by altering it. If these two conditions are met, the mutation is said to be weakly detected. If, in addition, the infected state propagates to the program’s final output and causes the test result to differ, the mutation is considered strongly detected. Detecting a mutant indicates that the test suite has successfully identified the introduced fault. The effectiveness of a test suite is commonly quantified by the mutation score, defined as the proportion of detected mutants relative to the total number of mutants. A higher mutation score reflects a greater ability to uncover potential bugs in the software [56]. In this paper, we focus on strong mutation analysis and, by a slight abuse of terminology, use the term behavior interchangeably with its manifestation in the program output. Thus, when we state that “a behavioral drift is observed between a mutant and its original program”, we specifically mean that a difference is observed in their produced outputs. One long-standing research challenge in the software engineering community is the detection of equivalent mutants [36]. Mutation operators introduce syntactic changes in the code without ensuring these changes affect the system’s behavior. As a result, some mutants may be semantically identical to the original program, making them indistinguishable by any test suite. These equivalent mutants neither simulate realistic faults nor contribute to fault detection [1, 29, 41, 53, 54]. In this work, we revisit this challenge for quantum mutation analysis. We focus on defining an output assessment criterion that distinguishes between true behavioral drift (non-equivalent mutants) and mere syntactic variation (equivalent mutants). Establishing such a criterion is essential for building reliable benchmarks and for evaluating the effectiveness of quantum mutant detection. 2.3

Quantum Mutation Analysis

Early studies in quantum mutation analysis relied on manually created mutants to evaluate the effectiveness of testing techniques [2, 32, 72, 73]. More recent work has aimed to automate the mutant generation process, leading to the development of two notable tools: Muskit [45, 71] and QMutPy [21–23]. These approaches share a common underlying concept: quantum mutation is performed at the circuit level, applying mutation operators that add, remove, or replace quantum gates to introduce faults. The mutation process in these tools generally follows three main steps. First, a position in the quantum circuit is selected, each circuit gate serving as a potential mutation point. Second, an operator is chosen to alter that position, either by inserting a new gate, replacing the existing one, or removing it. Third, the mutation is applied according to the selected operator. Transferring mutation analysis to quantum computing presents unique challenges. In classical software, mutants can often be strongly detected by comparing outputs for functional equivalence. In contrast, quantum programs produce inherently probabilistic outputs whose distributions vary across executions and are highly sensitive to noise. This randomness complicates the notion of mutant detection, which no longer depends on deterministic deviations but instead on comparing output distributions, expectation values, or quantum states. In practice, behavioral differences may manifest only subtly within probability distributions and can be further obscured or distorted by noise. This dual challenge calls for new methods to determine when two quantum programs differ significantly, that is, when a mutant should be considered detected. Furthermore, internal quantum states are only observable in simulation, motivating our focus on strong mutation analysis, where the central challenge lies in output comparison. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:7

Recent work in quantum mutation analysis has approached the problem in different ways. For example, QMutPy [21–23] assumes the existence of a predefined test suite equipped with its own oracle mechanism (i.e., providing both inputs and expected outputs). However, it does not address how such a suite should be built, thereby leaving the fundamental challenge of output comparison unresolved. In contrast, Muskit [45, 71] employs a single fixed metric—the Chi-Squared distance between two output distributions—without providing justification for this choice or considering alternative measures. In Muskit, a mutant is deemed detected when the statistical test yields a significance level below a given threshold (typically 𝑝-value < 0.05), indicating a statistically significant deviation from the expected distribution. In this study, we rely on Muskit exclusively for mutant generation, but we substantially extend its detection perspective. Specifically, we examine multiple distance metrics, spanning output distributions, density matrices, and expectation values, and combine them with different thresholding strategies. This allows us to investigate how metric selection and threshold calibration influence the robustness and reliability of mutant detection. Despite their contributions, none of these tools rigorously justify their assessment methodology or evaluate its robustness. Most importantly, none explicitly account for the presence of noise, despite its well-known and unavoidable influence in realistic quantum environments. Noise affects quantum gate operations, qubit stability, and overall circuit fidelity, and can therefore substantially distort both output comparison and mutant detection. To assess the detection power of a mutant detection approach, it is essential to observe its behavior on both equivalent and non-equivalent mutants. A sound approach should avoid falsely detecting equivalent mutants, while still reliably identifying mutants that exhibit genuine behavioral differences. The distinction between these two categories therefore provides a meaningful basis for evaluating the robustness and discriminative ability of a detection strategy. In classical mutation testing, a mutant is considered equivalent if no input can cause its behavior to differ from that of the original program, although determining such equivalence is generally undecidable. In our work, we exploit the restricted nature of pure quantum circuits (i.e., without classical control, loops, or hybrid constructs) to define a more structured notion of equivalence. Both the original circuit and its mutants implement deterministic quantum transformations followed by a final measurement. We define a quantum mutant as equivalent to the original circuit if, for every valid input, both lead to exactly the same quantum state (density matrix) when executed in an ideal, noise-free setting. In other words, no input can distinguish the mutant from the original based on their observable outcomes. This definition deliberately assumes an ideal environment: in the presence of noise, circuits that are functionally identical may appear different, while genuinely different circuits may appear similar, making equivalence harder to assess. Moreover, as in the classical case, internal circuit differences may remain unobservable under a given test suite, highlighting the gap between theoretical equivalence and practical detectability. 2.4

Quantifying Divergence in Quantum Program Outputs

Detecting behavioral differences between quantum programs (such as in mutation analysis) requires methods capable of accounting for the inherently probabilistic and noise-sensitive nature of quantum computation [51]. To enable meaningful comparisons between original and mutated programs, it is therefore essential to adopt output assessment methods that can represent and quantify these differences consistently across execution contexts (e.g., under varying noise conditions). In this work, we introduce a systematic methodology for comparing quantum programs under noisy conditions. Our approach builds upon established quantum output assessment principles and extends them to evaluate behavioral divergence in simulated environments. The methodology decomposes the problem into three interdependent components: ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:8

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

(1) Output Assessment Method: Specifies how quantum program results are represented to support consistent and interpretable comparisons across executions. (2) Comparison Metric: Defines a quantitative measure to evaluate behavioral divergence between the original and the mutated programs, based on the selected output representation. (3) Detection Threshold: Establishes the sensitivity level for mutation detection by distinguishing significant behavioral deviations from variations introduced by noise. We consider three representative output assessment methods: density-matrix-based, outputdistribution-based, and expectation-value-based approaches. For each method, we analyze associated metrics, discussing their computational requirements, interpretability, and applicability in practice. The remainder of this section provides background on these output assessment methods and their corresponding metrics. To the best of our knowledge, threshold selection for quantum mutation analysis has rarely been discussed in the quantum software engineering literature. For example, Muskit [45, 71] employs an output-distribution-based metric—the Chi-Squared distance—but relies on a fixed threshold of 0.05. In contrast, the next section (Sect. 3.3) introduces our approach for defining thresholds based on actual noise for robustness. 2.4.1 Density Matrix-based Metrics. Density matrices provide a comprehensive mathematical description of quantum states, encompassing both pure and mixed states, and enabling precise comparisons through well-defined distance metrics. They offer an exact representation of quantum state evolution and can, in principle, capture system behavior at any stage of execution. In this study, we focus on the final density matrix prior to measurement (i.e., the start of the “Output” box in Fig. 1b) to analyze the effects of strong mutations. While density matrices provide a rich and descriptive representation of quantum states, they are computationally demanding: their size scales exponentially with the number of qubits. This exponential growth in memory and resource requirements makes large-scale analysis infeasible on current simulators. Moreover, density matrices cannot be obtained on real quantum hardware, restricting their use to simulation environments. To quantify the similarity or divergence between two quantum states, we rely on two canonical metrics derived from density matrices [26]: the trace distance and the fidelity. Trace distance. The trace distance captures the degree of distinguishability between two quantum states. For two quantum states 𝜎 and 𝜏 (represented as density matrices), it is defined √︁  as: 𝐷 (𝜎, 𝜏) = 1 † (𝜎 − 𝜏) (𝜎 − 𝜏) . Values of the 2 ∥𝜎 − 𝜏 ∥ 1 , where the trace norm ∥ · ∥ 1 is given by: ∥𝜎 − 𝜏 ∥ 1 = Tr trace distance range from 0 to 1, where 0 indicates that the states are identical and 1 signifies that they are fully distinct. Fidelity. The fidelity quantifies the overlap—or closeness—between quantum states 𝜎 and 𝜏. It  √︁√ √   2 is defined as: 𝐹 (𝜎, 𝜏) = Tr 𝜎𝜏 𝜎 . For pure states |𝜓 ⟩ and |𝜙⟩, the expression simplifies to: 𝐹 (|𝜓 ⟩, |𝜙⟩) = |⟨𝜓 |𝜙⟩| 2 . Fidelity values range from 0 for orthogonal, fully distinct states, to 1 for identical states. Consistent with the criteria for a “good” quantum distance metric established by Gilchrist et al. [26], we adopt this formulation of fidelity rather than its square-root variant commonly found in other sources [51]. The trace distance and fidelity provide complementary perspectives on quantum state comparison at the state level: the former emphasizes distinguishability under optimal measurement, while the latter quantifies the intrinsic overlap between states. 2.4.2 Output Distribution-based Metrics. Output distributions reflect the probabilities of observing each possible measurement outcome and can be obtained from both quantum simulators and real ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:9

hardware. They provide a natural representation of quantum program behavior but are susceptible to stochastic fluctuations and noise. Accurate estimation requires a large number of repeated executions (shots), which increase exponentially with the number of qubits, posing a significant scalability challenge for large-scale quantum programs. Moreover, as output distributions depend on a carefully chosen measurement observable (as discussed in Sect. 2.1), their ability to distinguish quantum states varies with how well that observable captures the underlying differences. When the observable aligns well, even small behavioral variations may be detected; otherwise, certain deviations can remain hidden. While a difference in the density matrix guarantees a significant overall discrepancy between states, output distribution-based metrics capture only deviations along the specific observable used. Several metrics have been proposed to quantify differences between output distributions. Among these, the Hellinger and Jensen–Shannon distances are prevalent in quantum software engineering studies [19, 31, 38, 42, 47, 50, 52, 60, 75], and are the two metrics considered in this work. Hellinger distance. The Hellinger distance between two probability distributions 𝑃 and 𝑄 is defined as: √︄ 2 √︁ 1 ∑︁ √︁ 𝐻 (𝑃, 𝑄) = √ 𝑃 (𝑖) − 𝑄 (𝑖) . 2 𝑖 The Hellinger distance quantifies the degree of overlap between the probability mass of 𝑃 and 𝑄 (that is, the amount of probability assigned to each possible outcome), penalizing mismatches in how probability is distributed across outcomes. The distance ranges from 0 (identical distributions) to 1 (completely disjoint distributions), offering a clear geometric interpretation of divergence in distribution shape. Jensen–Shannon distance. The Jensen–Shannon distance provides a smoothed, symmetric alternative derived from the Kullback–Leibler divergence. It is given by: √︂ 𝐷 KL (𝑃 ∥𝑀) + 𝐷 KL (𝑄 ∥𝑀) JSDistance (𝑃, 𝑄) = , 2 where 𝑀 = 21 (𝑃 + 𝑄) and 𝐷 KL (𝑃 ∥𝑄) is the Kullback-Leibler divergence: ∑︁ 𝑃 (𝑖) 𝑃 (𝑖) log 𝐷 KL (𝑃 ∥𝑄) = . 𝑄 (𝑖) 𝑖 By measuring the average information loss when each distribution is approximated by their midpoint 𝑀, the Jensen–Shannon distance captures global divergence while mitigating sensitivity to local fluctuations. Like the Hellinger distance, it ranges from 0 to 1, where 0 denotes identical distributions and 1 indicates complete dissimilarity. In summary, both metrics measure behavioral divergence between quantum programs from probabilistic output distributions. The Hellinger distance tends to emphasize local shape differences, whereas the Jensen-Shannon distance captures the global divergence, smoothen the comparison through the average distribution. 2.4.3 Expectation Value-based Metric. Expectation values [17] quantify the average measurement outcome obtained over multiple executions of a quantum circuit. As statistical estimates, they are inherently sensitive to the number of measurement shots and affected by both noise and sampling variance, which may obscure subtle behavioral differences between mutants. Compared to density matrix analysis, they offer a computationally efficient alternative that can be directly estimated on hardware. However, their accuracy depends on the choice of measurement observable, which may limit their ability to capture all forms of behavioral divergence (similarly to how measurement basis choice affects output distributions) and thus leave certain mutations undetected. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:10

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Rather than reconstructing the full quantum state, one may compute expectation values to summarize the system’s behavior as the weighted average of measurement outcomes. This approach substantially reduces memory requirements and is well suited to both simulation and hardware execution. The expectation value of an observable 𝑂 in a pure quantum state |𝜓 ⟩ is defined as: ⟨𝑂⟩ = ⟨𝜓 |𝑂 |𝜓 ⟩, and for a mixed state 𝜌, it is expressed as: ⟨𝑂⟩ = Tr(𝜌𝑂). The expectation value corresponds to the mean outcome obtained by repeatedly measuring the observable 𝑂 on identically prepared copies of the system and can be interpreted as the probabilityweighted average of the eigenvalues of 𝑂. To compare two states, 𝜎 and 𝜏, we compute the absolute difference between their expectation values with respect to a chosen observable. Consistent with the measurement basis used for output distributions, our evaluation employs the Pauli-(Z) operator: |⟨𝑍 ⟩𝜎 − ⟨𝑍 ⟩𝜏 | = |Tr(𝜎𝑍 ) − Tr(𝜏𝑍 )|. This metric provides a compact and efficient estimate of behavioral divergence between two quantum states. 3

Experiment Design

To answer our research questions, we designed and conducted an empirical evaluation. Sect. 3.1 introduces the research questions and the experimental design underlying our study. Sect. 3.2 presents the subject systems, consisting of quantum circuits and their corresponding mutants, along with the input test cases. Sect. 3.3 discusses thresholds to separate between mutants and equivalent circuits. Sect. 3.4 introduces various characteristics of quantum programs whose impact on mutant detection we aim to evaluate. Finally, Sect. 3.5 details the experimental infrastructure, including server specifications and noise models. 3.1

Research Questions and Experiment Design Overview

Our study aims to investigate how noise impacts the ability to detect mutants, a central challenge in quantum mutation analysis. This leads us to the following research questions: 𝑅𝑄 1 How does noise impact output assessment? This question investigates how different output assessment methods perform in the presence of noise, and more broadly, when we can consider a mutant detected under noisy quantum conditions. We answer this question through the following sub-questions: 𝑅𝑄 1.1 Does the output difference between mutants and original programs vary under noise? 𝑅𝑄 1.2 Which metric best distinguishes non-equivalent mutants from equivalent programs? 𝑅𝑄 1.3 Can the same threshold be used to detect non-equivalent mutants in both noiseless and noisy scenarios? 𝑅𝑄 1.4 What is the most effective way to define a threshold adapted to each noise model? 𝑅𝑄 2 How do circuit characteristics impact noise resilience in mutation analysis? This question explores whether circuit features affect mutant detectability, potentially making some mutants more difficult to detect or more sensitive to noise. Understanding these influences can guide the design of more robust testing strategies for different circuit structures. 𝑅𝑄 3 How do the algorithm characteristics impact noise resilience in mutation analysis? This question investigates whether the nature of the tested algorithm impacts mutant detectability and noise sensitivity, helping to identify algorithms that may require specialized testing strategies. 𝑅𝑄 4 How do mutation characteristics impact noise resilience in mutation analysis? This question investigates whether specific mutation types affect detectability and robustness under noise, providing insight into which mutations may require more targeted testing or analysis approaches—and, more broadly, which kinds of bugs are inherently harder to detect. To address our research questions, we developed a configurable framework that supports systematic experimentation. Fig. 2 illustrates a high-level overview of the framework, outlining how ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:11

Fig. 2. Experimental Platform Overview

its main components interact and integrate into a coherent execution pipeline. The experimental workflow proceeds through five main steps (detailed in the remainder of this section), each associated with distinct computational and time requirements: A – Mutant Generation and Selection: We generate mutants from each CUT, and we select a subset of them to reduce the workload of the next steps while maintaining variability. This step is computationally lightweight and completes within a few minutes for all CUTs. B – Execution: We execute CUTs and mutants across all input configurations and noise profiles. This step constitutes the dominant cost of the workflow, requiring approximately two to three weeks of parallel execution on a dedicated HPC cluster. It also resulted in substantial data generation (about 293 GB per noise model). C – Distance Measurements: We collect execution data of all CUT and mutant executions and compute distance metrics. Due to the volume of data collected, this phase requires several days of execution. D – Threshold Application: We evaluate the computed distances against various threshold configurations to determine mutant detection outcomes. As this step operates on preprocessed CSV files, execution time remains moderate (typically within a few minutes), even when assessing multiple threshold strategies. Note that the thresholds must have been defined beforehand. E – Analysis: We aggregate and analyze detection outcomes to answer our research questions. Once all prior data is available, this step can be completed within a few minutes. These computational constraints shaped several of our experimental decisions (for example, limiting the number of qubits and sampling a reduced set of inputs for larger programs). 3.2

Evaluation Benchmark

This section introduces our evaluation benchmark, detailing the original subject systems, the process used to generate their mutants, and the methodology for constructing test suites. 3.2.1 Original subject systems. For this study, we selected six quantum programs from the MQTbench [63] benchmark suite that represent a diverse range of circuit characteristics. Tab. 1 highlights the main characteristics of these programs. Each circuit in the benchmark will be referred to as a Circuit Under Test (CUT) throughout this work. Following the classification proposed by Mendiluze Usandizaga et al. [71], we included three programs with dominant outputs (ae, qpe, and vqe), and three with diverse outputs (qft, qft-entangled, and w-state). Dominant output algorithms focus on identifying the state with the highest probability, commonly used in optimization tasks. In contrast, diverse output algorithms identify multiple states with similar probabilities. This is especially useful for applications that require varied outcome distributions, such as the Quantum Fourier Transform, where creating a distribution of several high-probability outcomes determines the hidden period of the function [30]. We used the MQTbench scalability option to obtain versions of each program with varying qubit counts, ranging from 2 to 8 qubits, resulting in a total of 41 ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:12

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Table 1. Characteristics of the original benchmarks (i.e., quantum algorithms). Algorithm

#qubits

#gates

depth

#single-qubit gates

#multi-qubit gates

ae qft qft-entangled qpe vqe w-state

2-8 2-8 2-8 2-8 3-8 2-8

8-65 5-41 7-49 5-47 14-39 6-30

6-42 5-17 7-19 4-23 8-13 5-17

5-29 2-8 3-9 3-15 9-24 3-15

3-36 3-33 4-40 2-32 5-15 3-15

*Amplitude Estimation (ae); Quantum Fourier Transform (qft); Quantun Fourier Transform Entangled (qft-entangled); Quantum Phase Estimation (qpe); Variational Quantum Eigensolver ( vqe); W-State (w-state).

CUTs.2 We limited the maximum number of qubits to 8 due to resource constraints, as storing the density matrices of circuits with more than 8 qubits required an infeasible storage capacity. 3.2.2 Mutant generation. After selecting the programs, we generated mutants for each of them using the Muskit [45] quantum mutation generation tool (Step A in Fig. 2). We selected Muskit over QMutPy because it keeps mutants and test cases decoupled. In contrast, QMutPy aggregates the circuit and test inputs into a single circuit file, which prevents reusing different inputs for the same circuit. By keeping mutants and tests separate, we can apply multiple test cases and metrics independently, aligning better with our workflow. To evaluate the metrics, we need to assess the ability of each of them in detecting mutants. This means that we need both non-equivalent mutants (to assess that they are correctly detected), but also equivalent mutants (to assess that we do not mark them as mutants, as they are not semantically different from the CUT). At first, we used Muskit to apply all possible mutation combinations. Given the total number of mutants generated (25,105) and the number of executions required to run all tests for each mutant, we decided to sample a subset based on specific criteria to ensure a balanced distribution. These selection criteria allowed us to evenly spread the mutants across different positions in the circuit, covering positions from the beginning to the end. Also, we ensured that there was at least one mutant for each type of gate and balanced the number of mutants for each mutation operator. Then, we compared the theoretical density matrix of each generated mutants with that of the CUT. To do this, we computed the distance between the two matrices, using both trace distance and fidelity. A mutant was considered equivalent if both measures were zero, allowing us to separate equivalent mutants from non-equivalent ones. This resulted in a total of 1,483 mutants: 1,170 non-equivalent and 313 equivalent. To obtain a more balanced dataset, we generated additional mutants, equivalent by construction. To do so, we extended Muskit to implement gate reversibility based on the quantum gate reversibility principle [51]. Specifically, we repeatedly inserted the same gate in such a way that their cumulative effect cancels out, thereby producing a mutant circuit that is functionally equivalent to the original one. This resulted in a total of 15,312 equivalent mutants, from which we sampled following the same criteria as before. With this, we added a total of 741 new equivalent mutants to our sample set, totaling 2,224 mutants: 1,170 non-equivalent and 1,054 equivalent. Tab. 2 presents the distribution of these mutants among the benchmarks. 3.2.3 Test suites. To construct test cases for each CUT, we generated input states using an inputpreparation circuit that matches the qubit dimension of the CUT. We first considered Classical input states, randomly sampling bitstrings and encoding them into computational basis states. While 2 Note that MQTbench does not support the VQE algorithm in a 2-qubit setting.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:13

Table 2. Mutants distribution. Algorithm

Non-equivalent Mutants

Equivalent Mutants

Total

ae qft qft-entangled qpe vqe w-state

252 182 208 193 172 163

238 158 183 175 156 144

490 340 391 368 328 307

Total

1170

1054

2224

classical inputs provide a baseline for behavioral comparison, they do not capture the richness of quantum computation. To address this limitation, we augmented the test suites with Quantum input states, including entangled and superposed states generated using QuraTest [79]. Among QuraTest generators, we selected the UCNOT generator, which constructs parametrized quantum states based on 𝑈 and CNOT gate patterns. This choice was based on its superior mutation score and strong performance in both input diversity and output coverage, as reported in [79]. Input selection also depended on program size. For CUT with up to four qubits, we generated all possible classical input states to ensure exhaustive state-space coverage. For larger CUT, we sampled half of the classical states to maintain tractable execution time. In both cases, we complemented classical states with the same number of quantum inputs. This resulted in 2#𝑞𝑢𝑏𝑖𝑡𝑠 inputs for small circuits, and 2 × (2#𝑞𝑢𝑏𝑖𝑡𝑠 ) inputs for larger ones. This strategy ensures meaningful diversity in initial states, enabling assessment of the CUT behavior under both classical and quantum input conditions. All generated inputs are exported as Quantum Assembly Language (QASM) [18] files, allowing reproducibility and independent reuse. These input were subsequently executed for each CUT and mutants during Step B of the experimental pipeline (Fig. 2). 3.3

Thresholds

As discussed in Sect. 2.2, identifying non-equivalent mutants is fundamental to mutation analysis. Under noisy conditions, this identification becomes even more challenging, as noise can cause non-equivalent mutants to appear equivalent—or conversely, make equivalent ones seem different—thereby distorting the resulting mutation score. The distance metrics introduced in Sect. 2.4 quantify behavioral differences between program executions. However, to determine whether a mutant should be considered detected, each metric requires a threshold that discriminates between meaningful divergences (caused by actual mutations) and acceptable variations (introduced by noise or statistical fluctuations inherent to quantum measurements). In other words, the threshold is the criterion for distinguishing equivalent from non-equivalent mutants. Importantly, for a mutant to be detectable, the distance between its output and that of the CUT must exceed the expected variability introduced by noise and statistical error. A central motivation for this paper is the hypothesis that the optimal threshold in a noisy environment differs from that in a noiseless setting. This hypothesis is supported by the observations in Fig. 3. In a noiseless environment, there are some non-negligible variations, visible in the first boxplot for each metric (in blue), but these are minimal compared to the effects introduced by noise. The green boxplots in the same figure illustrate the much larger deviations caused by three different noise models. To investigate this hypothesis, 𝑅𝑄 1.3 compares a threshold derived from noiseless executions, referred to as the noiseless threshold 𝑇 Noiseless , with custom, noise-specific thresholds 𝑇 Noise , where Noise denotes a specific hardware platform or noise model. The goal is to estimate ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:14

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Fig. 3. Distances between the theoretical output of 41 benchmark programs and the output obtained by executing the same programs on four simulators. Each boxplot reflects the variability across 30 runs for each simulator and distance metric.

Fig. 4. Visualization of the threshold distribution for the Hellinger distance metric.

each 𝑇 Noise solely based on the CUTs and the characteristics of the corresponding hardware or noise model. Crucially, we refrain from using mutants in the computation of 𝑇 Noise , for the following reasons: • Noise compensation only: The purpose of the threshold is to quantify the effects of noise and stochasticity, enabling us to distinguish between deviations caused by these factors and those resulting from the mutation itself. Therefore, CUTs should be sufficient for computing 𝑇 Noise and estimating the impact of noise. • Practicality: Mutant generation and execution are computationally expensive, as they require running a large number of program variants, with a large number of inputs. In contrast, CUTs are fewer in number and thus more practical to analyze at scale for estimating noise-specific thresholds. • Evaluation independence: To avoid overfitting, the threshold must be computed from data distinct from the evaluation set. Using mutants for both would violate this principle. Hence, we compute 𝑇 Noise on CUTs and assess it on mutants. • Representativeness: The subject systems (Sect. 3.2.1) are selected to be diverse and representative enough to generalize the results. To assess the robustness and potential for improvement of our custom thresholds 𝑇 Noise , 𝑅𝑄 1.4 explores two additional thresholds, 𝑇 Middle and 𝑇 Above , chosen to bracket the entire range of 𝑇 Noise values, as illustrated in Fig. 4. This allows us to evaluate how sensitive the detection performance is to threshold variations. In the following, we detail our methodology to define 𝑇 Noiseless , 𝑇 Noise , 𝑇 Middle , and 𝑇 Above . 3.3.1 Noiseless threshold (𝑇 Noiseless ). The noiseless threshold, denoted 𝑇 Noiseless , serves as a baseline for our evaluation. In a noiseless environment, the threshold only accounts for computational approximations and minor deviations in output probabilities. These deviations are visible in Fig. 3, shown in blue (the first boxplot for each metric). In the absence of noise, the density matrix is deterministic. However, preliminary experiments revealed that Trace distance and Fidelity still exhibit negligible floating-point differences between executions. Even when mutants are equivalent, these metrics may deviate by values as small as ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:15

10−13 for Trace distance and 10−14 for Fidelity. These deviations remain negligible compared to mutation-induced effects, which always exceed 10−3 . Moreover, Qiskit’s Estimator3 computes expectation values in a shot-dependent manner, meaning that the results vary slightly across runs. Similarly, simulations introduce small statistical fluctuations in output distributions. For instance, in a simulation with 10000 shots, a theoretical distribution with two equally likely outputs might produce slightly uneven counts, such as 5004/4996 instead of exactly 5000/5000. To obtain a reliable estimate of these deviations, we base the noiseless threshold on the mean deviation between shot-based distance metrics and a theoretically perfect reference derived from the exact (i.e., noiseless) density matrix. Because shot-based metrics are inherently stochastic, a single execution does not yield a reliable estimate; instead, averaging over multiple independent runs is required. We therefore execute each CUT 𝑟 = 30 times on a noiseless simulator. This value for 𝑟 represents a deliberate trade-off between statistical stability and computational feasibility. Prior empirical software engineering research identifies 30 as a commonly used lower bound for estimating means and standard deviations in the presence of randomness, while emphasizing that larger sample sizes should be used whenever computationally feasible [5]. In our setting, executing the full benchmark is computationally expensive: the 30 repetitions required several days of computation and approximately 103 GB of storage per noise model. Since threshold computation is intended to be repeated (e.g., when introducing or updating noise models), substantially increasing the number of runs would make the process impractical. We therefore adopt 𝑟 = 30 as a pragmatic compromise that aligns with established methodological guidance while keeping the overall cost manageable. For each program 𝑖, we measure the distance between each observed output and the corresponding theoretical reference derived from the exact density matrix. Let 𝑃𝑖𝑗 denote the observed output distribution of the 𝑗 th run of program 𝑖, and ⟨𝑍 ⟩𝑖𝑗 the corresponding observed expectation value. These per-run observations form the basis for estimating the deviation introduced by shot-based stochasticity and are subsequently used to define the noiseless threshold 𝑇 Noiseless . The noiseless threshold computation proceeds as follows, first quantifying per-program deviations and then aggregating them across the benchmark: Step 1 – Compute the theoretical reference outputs: • For output distributions, we extract the exact measurement probabilities 𝑃𝑖𝜃 from the program’s final density matrix. We then scale this theoretical distribution to match the total shot count. • For expectation values, we calculate the theoretical expectation ⟨𝑍 ⟩𝑖𝜃 of the Pauli-Z operator directly from the density matrix. Step 2 – Measure per-run distances to the theoretical output: For each run 𝑗 ∈ {1, . . . , 𝑟 } of program 𝑖, we compute the distance 𝑑𝑖 𝑗 between the theoretical reference and the observed output. Depending on the chosen metric, the distance is defined as follows:    𝜃, 𝑃 𝑗  𝐻 𝑃 (Hellinger distance)  𝑖 𝑖       𝑗 𝜃 𝑑𝑖 𝑗 = JSDistance 𝑃𝑖 , 𝑃𝑖 (1) (Jensen-Shannon distance)     (Expectation value difference)  ⟨𝑍 ⟩𝑖𝜃 − ⟨𝑍 ⟩𝑖𝑗  This step yields 𝑟 = 30 distance values per metric for each program 𝑖. Step 3 – Compute the mean and standard error of distances per program: For each program 𝑖, we calculate: 3 https://docs.quantum.ibm.com/api/qiskit-ibm-runtime/qiskit_ibm_runtime.EstimatorV2

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:16

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Table 3. Metrics and thresholds for each simulator. TNoise }|

Metric

TNoiseless

z TKyiv

TBrisbane

{ TSherbrooke

TMiddle

TAbove

Trace Distance Fidelity Expectation Value Hellinger Jensen-Shannon

10−13 1 − 10−14 0.01428 0.06561 0.06045

0.08959 0.80481 0.17837 0.32652 0.27611

0.07080 0.82417 0.27261 0.36616 0.30977

0.14328 0.66146 0.25560 0.42757 0.36368

0.03540 0.91208 0.09633 0.19607 0.16828

0.17868 0.57354 0.35465 0.55802 0.47151

𝜇𝑖 =

1 ∑︁ 𝑑𝑖 𝑗 𝑟 𝑗=1

• The mean distance: 𝑟

(2)

• The standard error of the distances: 𝑠𝑖 𝜎𝑖 = √ , 𝑟

v u t where 𝑠𝑖 =

𝑟 1 ∑︁ (𝑑𝑖 𝑗 − 𝜇𝑖 ) 2 𝑟 − 1 𝑗=1

(3)

Step 4 – Determine quantiles over means and standard errors: We treat the sets {𝜇1, . . . , 𝜇𝑛 } and {𝜎1, . . . , 𝜎𝑛 } as two separate distributions. Let 𝑄 87.5% (𝑋 ) denote the 87.5th percentile of a set 𝑋 . Then: 𝑄 𝜇 = 𝑄 87.5% ({𝜇1, 𝜇2, ..., 𝜇𝑛 })

𝑄 𝜎 = 𝑄 87.5% ({𝜎1, 𝜎2, ..., 𝜎𝑛 })

(4)

Although the choice of percentile reflects a trade-off between sensitivity and robustness, preliminary experimentation showed that the 87.5% percentile yielded the best empirical results in our setting. Step 5 – Compute the final threshold: The final noiseless threshold for the given metric is then defined as: 𝑇 Noiseless = 𝑄 𝜇 + 𝑄 𝜎 (5) The first column of Tab. 3 presents the results of this procedure applied on the benchmark presented in Sect. 3.2. 3.3.2 Noise-specific threshold (𝑇 Noise ). In contrast to the baseline threshold 𝑇 Noiseless , the noisespecific threshold 𝑇 Noise must account not only for computational and statistical fluctuations but also for perturbations inherent to quantum noise. In noisy settings, the density matrix is no longer deterministic, and even ideal outputs become altered by decoherence, depolarization, and other quantum noise effects. As a result, the threshold must absorb these effects while still remaining sensitive to behavioral shifts caused by mutations. The three green boxplots in Fig. 3 illustrate the typical variability observed for each metric and under different noise conditions. As in the noiseless setting, the estimation of 𝑇 Noise relies on averaging over multiple independent executions to obtain stable estimates in the presence of stochasticity. We therefore use the same number of repetitions, 𝑟 = 30, for each noise model. The statistical and computational rationale for this choice is identical to that discussed for 𝑇 Noiseless in Sect. 3.3.1. To compute each 𝑇 Noise , we reuse the general procedure introduced for 𝑇 Noiseless , adapted to incorporate noise-induced discrepancies. Let 𝑇 Noiseless denote the specific hardware or noise model. The procedure is detailed below: ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:17

Step 1 – Compute the theoretical reference outputs: As in the noiseless case, we begin with deriving the reference outputs: • For density matrices, we simulate each quantum program on the noiseless simulator to extract the exact density matrix, 𝜎𝑖𝜃 . • For output distributions, we extract the theoretical probabilities 𝑃𝑖𝜃 from 𝜎𝑖𝜃 and scale them to match the total shot count. • For expectation values, we compute the theoretical expectation ⟨𝑍 ⟩𝑖𝜃 of the Pauli-Z operator from 𝜎𝑖𝜃 . Step 2 – Simulate noisy executions and measure per-run distances: As in the noiseless case, each quantum program 𝑖 is executed 𝑟 = 30 times on a noisy simulator configured with a specific noise model to account for noise and probabilistic stochasticity. Let 𝜎𝑖𝑗,Noise , 𝑃𝑖𝑗,Noise and ⟨𝑍 ⟩𝑖𝑗,Noise respectively denote the density matrix, the output distribution and the expectation value obtained during the 𝑗 th execution on the noisy simulator Noise. For each run, we compute the distance 𝑑𝑖 𝑗,Noise between the theoretical output and the observed noisy result:    𝜃 , 𝜎 𝑗,Noise  𝐷 𝜎 (Trace distance)    𝑖 𝑖     𝑗,Noise 𝜃 𝐹 𝜎 , 𝜎 (Fidelity)  𝑖 𝑖       𝑑𝑖 𝑗,Noise = 𝐻 𝑃𝑖𝜃 , 𝑃𝑖𝑗,Noise (6) (Hellinger distance)      𝑗,Noise 𝜃   JSDistance 𝑃𝑖 , 𝑃𝑖 (Jensen-Shannon distance)       ⟨𝑍 ⟩𝑖𝜃 − ⟨𝑍 ⟩𝑖𝑗,Noise (Expectation value difference)  This results in 30 distance values per metric for each program 𝑖 and each noise model Noise. Steps 3–5 – Same as in the noiseless case: The remaining steps (i.e., computing per-program means and standard errors, extracting quantiles, and computing𝑇 Noise thresholds) are identical to those described for 𝑇 Noiseless in Sect. 3.3.1. Columns 2–4 of Tab. 3 report the values of 𝑇 Noise across the three different noise models used in our experiments (see Sect. 3.5 for further details on these models). 3.3.3 Middle threshold (𝑇 Middle ). We hypothesize that 𝑇 Noise approximates the optimal noise-aware threshold. To evaluate this, we define a midpoint 𝑇 Middle between 𝑇 Noiseless and the minimum of all noise-specific thresholds: 𝑇 Middle =

min(𝑇 Noise ) + 𝑇 Noiseless 2

(7)

where Noise represents the specific noise models. The values of 𝑇 Middle are reported in the fifth column of Tab. 3. 3.3.4 Above threshold (𝑇 Above ). Analogously, to test whether further increasing the threshold improves performance, we define 𝑇 Above as a value higher than all noise-specific thresholds. Specifically, 𝑇 Above is at the same distance to the maximum noise-specific threshold as 𝑇 Middle is to the minimum noise-specific threshold:   𝑇 Above = max(𝑇 Noise ) + min(𝑇 Noise ) − 𝑇 Middle (8) where Noise is the specific noise model. 𝑇 Above values are given in the final column of Tab. 3. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:18

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Table 4. Summary of the characteristics evaluated in 𝑅𝑄 2 . RQ

Characteristic

Type

Categories / Range

𝑅𝑄 2 – Circuit Characteristics

#qubits #gates depth

Quantitative Quantitative Quantitative

[2-8] [5-65] [4-42]

Categorical Categorical Categorical

{ae, qft, qft-entangled, qpe, vqe, w-state} {Dominant, Diverse} {Classical input, Quantum input}

Categorical Categorical Quantitative

{Add, Remove, Replace} {Single-qubit, Multi-qubit} {Beginning, Pre-middle, Middle, Post-middle, End}

algorithm 𝑅𝑄 3 – Algorithm Characteristics output type input type 𝑅𝑄 4 – Mutation Characteristics

operator gate type relative position

3.3.5 Threshold Application Process. During execution, we collect output assessment data (e.g., density matrices, output distributions, expectation values) and store it in Pickle files (Fig. 2, step C). We then compute the distances defined in Sect. 2.4, and record the resulting values in CSV files. In step D (Fig. 2), we apply our defined thresholds. Separating distance computation from threshold evaluation offers a modular design that allows new thresholds to be introduced without recomputing the underlying distances. While distance computation is time-consuming, requiring the parsing of all Pickle files, threshold evaluation is significantly faster, as it operates directly on the preprocessed CSV data, reducing runtime from hours to minutes. This modularity is particularly beneficial when exploring multiple threshold configurations. For each metric–threshold pair, we assign a detection flag as follows: • True (i.e., detected) if the distance exceeds the threshold, indicating that the mutant has been detected. • False (i.e., non-detected) if the distance remains below the threshold, suggesting the mutant and CUT behave equivalently. 3.4

Quantum Circuit, Algorithm and Mutation Characteristics

In this section, we present the quantum program characteristics analyzed in 𝑅𝑄 2 , organized into three main categories aligned with each sub-research question and as has been previously done in the literature by Mendiluze Usandizaga et al [71]: circuit characteristics, algorithm characteristics, and mutation characteristics. Tab. 4 provides an overview of the specific properties examined under each group. We distinguish between two types of data: categorical data, which consists of discrete, non-numeric labels, and quantitative data, which are numerical values defined over a continuous or ordinal scale, where the magnitude and ordering of values carry meaning. Circuit Characteristics. We study various structural properties of quantum circuits to understand how circuit design may influence mutation effects. These include the number of qubits (#qubits) and the total number of gates (#gates) present in the circuit. We also consider the circuit depth, which captures the longest gate-dependent path in the circuit. Algorithm Characteristics. In our study, we consider each algorithm as a distinct unit of analysis, capturing potential variations in behavior across different experimental conditions. Algorithms are grouped according to their output type, either dominant or diverse, as defined in Sect. 3.2.1. This output type abstracts the behavior of multiple algorithms, allowing us to classify and compare programs based on their overall output patterns. Our selection includes representative algorithms from both categories, following established classifications by Mendiluze Usandizaga et al. [71] and supported in the literature [6, 57]. In addition, we also distinguish the input type used to initialize the circuit, as described in Sect. 3.2.3. We define two categories: classical inputs, where ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:19

qubits are initialized in a computational basis, and quantum inputs, where qubits are prepared in superposition and entangled states. Input types are applied independently of the algorithm or its output type, ensuring a diverse set of initial conditions. Mutation Characteristics. We study mutation characteristics from three perspectives. First, we consider the mutation operator, following the classification commonly used in the literature [23, 45, 71], which comprises three types: add, remove, and replace. Second, we analyze the type of gate affected by the mutation (gate type), distinguishing between single-qubit and multi-qubit gates. Lastly, we consider the relative position of the mutation within the circuit. Because the absolute gate index is not meaningful across circuits of different depths, we instead divide each circuit uniformly into five segments, which serve as arbitrary but consistent reference regions: beginning (first 20% of gates), pre-middle (20–40%), middle (40–60%), post-middle (60–80%), and end (final 20%). This discretization allows us to capture only the relative location of a mutation within the circuit (i.e., what matters is whether it occurs earlier or later in the computation) while avoiding issues associated with absolute gate positions. We treat this relative position as an ordered numerical variable, reflecting the progression of operations within the computation. By doing so, we are able to investigate whether the location of a mutation within the circuit structure influences its resilience to noise, rather than restricting the analysis to categorical comparisons only. 3.5

Experimental Setup & Execution

All experiments were conducted under controlled computational conditions to ensure reproducibility. Circuit executions (Step B in Fig. 2), encompassing all CUTs and mutants across every test input and simulator configuration (Noiseless, Sherbrooke, Brisbane, and Kyiv), were performed on a national high-performance server cluster equipped with 2x AMD Epyc 7601 processors, 2 TB of RAM, an AMD Vega20 GPU, and a high-speed 4 TB NVMe drive. Auxiliary steps (A, C–E) were executed locally on standard workstations with modern processors: a Windows laptop with an Intel Core i9 processor and 64 GB of RAM, and a MacBook Pro with an Intel Core i7 processor and 32 GB of RAM. CUTs, mutants, and test inputs were provided as QASM files, and all intermediate results were persistently stored to facilitate reuse and further analyses. All programs were executed using the Aer simulator from the Qiskit-aer module version 0.14.1 and Qiskit version 1.1.0.4 For consistency and reproducibility, a fixed random seed, a critical parameter in the Aer simulator, was applied across all simulations to control the intrinsic variability of quantum circuit execution. Estimating an appropriate number of measurement shots is a nontrivial task. The literature reports a wide range of shot counts, for example from 12 to 3,200 [59], 10,000 [8, 11, 39, 74, 79], and up to 100,000 [71]. While these works generally do not provide a detailed justification for their choice of shot count, their usage patterns provide useful empirical guidance. In this work, we adopt 10,000 shots per circuit execution as a conservative compromise between statistical precision and experimental cost. Notably, the only study employing a substantially higher number of shots (100,000 [71]) considers circuits of up to 30 qubits, whereas our experiments are limited to circuits of at most 8 qubits. Moreover, the Qiskit documentation5 reports 4,000 shots as the default setting, indicating that our choice lies well above commonly used baseline configurations. To execute the quantum programs under noisy conditions, we first extracted noise models from actual IBM quantum backends and integrated them into Qiskit’s Aer simulator. These noise models were retrieved using the IBM Quantum API, in accordance with the official IBM documentation for accessing quantum processing unit (QPU) details.6 . Each backend’s noise model was serialized and 4 Qiskit’s Aer documentation: https://qiskit.github.io/qiskit-aer/stubs/qiskit_aer.AerSimulator.html#qiskit_aer.AerSimulator 5 https://quantum.cloud.ibm.com/docs/en/guides/specify-runtime-options, last accessed on May 14, 2026 6 Tutorial on how to access backend information: https://docs.quantum.ibm.com/guides/get-qpu-information

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:20

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Table 5. Summary of evaluation goals and methods RQ

Goal

Evaluation Method

𝑅𝑄 1

𝑅𝑄 1.1 : Noise effect on distances

• KDE distributions • Boxplots by metric & noise • Fligner–Killeen, with ratio of variances • Boxplots by noise model • Cross-environment comparison • Mann–Whitney, with Cliff’s Delta • Confusion matrix (TP/FP/TN/FN) • Accuracy & F1-scores • Boxplots by noise model, with threshold overlays

𝑅𝑄 1.2 : Noise impact across mutant types

𝑅𝑄 1.3 : Threshold robustness 𝑅𝑄 1.4 : Threshold validation 𝑅𝑄 2

Effect of circuit characteristics on noise variability

• Correlation analysis • Quantitative: Pearson

𝑅𝑄 3

Effect of algorithm characteristics on noise variability

• Correlation analysis • Binary: Mann–Whitney • Multi-valued: Kruskal–Wallis

𝑅𝑄 4

Effect of mutation characteristics on noise variability

• Correlation analysis • Quantitative: Pearson • Binary: Mann–Whitney • Multi-valued: Kruskal–Wallis

stored as a separate Pickle file, which was subsequently loaded before every simulation run. This approach ensures reproducibility and allows for consistent comparisons with mutant variants. Because IBM’s quantum backends undergo periodic recalibration, storing the noise models is essential to prevent inconsistencies that could arise from changes in backend characteristics over time.7 Our experiments were conducted using three distinct noise models—Sherbrooke, Brisbane, and Kyiv. While each model exhibits unique noise characteristics, they all share a common structure, including the same set of basis gates. 3.6

Evaluation method and statistical tests

This section presents our method to answer our research questions, including a description of the statistical tests we performed. Tab. 5 summaries the evaluation steps, the analyses performed, and the corresponding statistical tests. The process corresponds to step E in Fig. 2. 3.6.1 Answering 𝑅𝑄 1 . By comparing noiseless and noisy simulations, we evaluate noise robustness of mutation detection. The goal is to identify the most effective combinations of metric and threshold, taking into account the constraints imposed by the type of output assessment method used. To address 𝑅𝑄 1.1 , we examine whether the distance between mutants and CUTs varies under noise. We begin by visualizing the overall distribution of distances using a kernel density estimate plot. To strengthen our findings, we further analyze the impact of noise separately for equivalent and non-equivalent mutants. We present boxplots showing the distances between mutants and their original counterparts. The boxplots are grouped by metric, with different colors representing distinct noise models, enabling a clear comparison across environments. To statistically assess whether noise systematically affects the dispersion of distances, we complement the visual analysis with a nonparametric scale test from the Levene’s family, the Fligner– Killeen test [20]. Levene-type tests evaluate whether two or more groups differ in variability by 7 All noise model data used in this study were retrieved on 24 January 2025 at 16:30 CET.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:21

comparing the absolute deviations from a central location. The Fligner–Killeen test is a rankbased variant that is robust to skewed and heavy-tailed distributions, making it suitable for the heterogeneous distance distributions. Given the very large number of paired observations, we do not rely on p-values alone, which are 2 2 expected to be vanishingly small. We therefore report the ratio of variances (𝑉 𝑅 = 𝜎noisy /𝜎noiseless ) as an effect size measure, which captures the magnitude of the dispersion effect. Values above 1 correspond to variance inflation (more scattered distributions), while values below 1 correspond to variance attenuation (more concentrated distributions). To address 𝑅𝑄 1.2 , we again use boxplots to compare the distances between mutants and CUTs. This time, the plots are grouped by noise model, with color distinguishing between equivalent and non-equivalent mutants. This organization helps assess how noise influences execution in each setting and highlights the role played by the choice of distance metric. To statistically confirm the observed differences between equivalent and non-equivalent mutants under each noise model, we apply the Mann–Whitney U test [43, 70], a non-parametric test suitable for comparing independent samples without assuming normality of the underlying distributions. We also report Cliff’s Delta [14] (𝛿) as an effect size measure, quantifying the magnitude and direction of the differences in terms of probabilistic dominance. This combination allows us to assess not only whether the two groups differ, but also how substantial these differences are across noise models and distance metrics. Values of |𝛿 | < 0.15 indicate [44] a Negligible effect, 0.15 ≤ |𝛿 | < 0.33 a Weak effect, 0.33 ≤ |𝛿 | < 0.47 a Moderate effect, and |𝛿 | ≥ 0.47 a Strong effect. To address 𝑅𝑄 1.3 , we evaluate how the 𝑇 Noise thresholds compare with 𝑇 Noiseless across three different noise environments using confusion matrices. These matrices report the percentages of true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), defined as: True Positive (TP): Mutants correctly detected. False Positive (FP): Equivalent programs incorrectly identified as mutants. True Negative (TN): Equivalent programs correctly identified as non-mutants. False Negative (FN): Mutants incorrectly identified as equivalent. Excessive sensitivity to noise is reflected in a high false positive rate, meaning equivalent programs are mistakenly flagged as mutants. Conversely, a high false negative rate suggests leniency: non-equivalent mutants go undetected and appear indistinguishable from noisy executions of the CUT. Both situations indicate a lack of robustness in detecting mutants under noise. Due to space constraints, we focus our discussion on the Brisbane noise model. Results for the Kyiv and Sherbrooke noise models, along with visualizations, are available in our replication package [24]. While Kyiv and Brisbane exhibit comparable noise characteristics, we chose Brisbane over Sherbrooke because it introduces slightly less noise, thereby minimizing bias. Excessively noisy settings might make it harder to rely on mutation analysis tools such as Muskit and could exaggerate the benefits of our approach. In extreme cases, mutation analysis would become entirely irrelevant. By instead using a model with moderate noise, we show that even low noise levels already require carefully calibrated thresholds. To address 𝑅𝑄 1.4 , we revisit the boxplots from 𝑅𝑄 1.2 , this time adding horizontal lines to illustrate our different threshold values. We further validate our findings using standard classification metrics: Accuracy and F1-score. These are defined as:

Precision =

𝑇𝑃 , 𝑇𝑃 + 𝐹𝑃

Recall =

𝑇𝑃 , 𝑇𝑃 + 𝐹𝑁

Accuracy =

𝑇𝑃 +𝑇𝑁 , 𝑇𝑃 +𝑇 𝑁 + 𝐹𝑃 + 𝐹𝑁

F1-score = 2 ·

Precision · Recall . Precision + Recall

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:22

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Accuracy provides an overall measure of correctness by reflecting the proportion of correctly classified cases. The F1-score provides a more balanced measure by combining both Precision and Recall, making it particularly useful when false positives and false negatives are equally costly. 3.6.2 Answering 𝑅𝑄 2 , 𝑅𝑄 3 , and 𝑅𝑄 4 . In 𝑅𝑄 2 , 𝑅𝑄 3 , and 𝑅𝑄 4 , we investigate how various factors (including the characteristics of the CUT, algorithms, and mutations) interact with and are influenced by noise in the context of mutation analysis. Our goal is to analyze the trends in the noise associated with each characteristic, independent of the noise model used. To achieve this, we evaluate the impact of these characteristics across the three noise models simultaneously. To make our findings measurable, we examine the correlation between the noise observed during noisy executions and each program characteristic. For the numerical variables, we utilize Pearson’s correlation test [13]. As shown in Tab. 4, we have identified four quantitative variables: the number of qubits, the number of gates, the depth of the circuit and the relative position. Pearson’s correlation coefficient (𝑟 ) quantifies the linear relationship between two variables and can range from −1 to 1. A value near −1 represents a strong negative correlation, whereas a value close to 1 indicates a strong positive correlation. If the value is around 0, it implies that there is minimal or no linear correlation between the variables. The interpretation of the correlation strength follows the widely recognized guidelines [16]: (i) Negligible: |𝑟 | < 0.10 (ii) Weak: 0.10 ≤ |𝑟 | < 0.30 (iii) Moderate: 0.30 ≤ |𝑟 | < 0.50 (iv) Strong: |𝑟 | ≥ 0.50. For categorical variables, we used non-parametric tests depending on the number of categories. When the variable had only two categories (such as input type, output type, and mutated gate type), we compared the distributions using the Mann–Whitney U test [43, 70]. For variables with more than two categories (such as algorithm and operator), we used the Kruskal–Wallis test, a generalization of the Mann–Whitney U that allows comparison across more than two independent groups [70]. To account for multiple comparisons across variables and metrics, p-values were adjusted using the Holm correction to control the family-wise error rate [27]. In the case of the two tests used for categorical variables, we need to determine the effect size to make a statement about the strength of the correlation. For the Mann-Whitney U test, we require the test statistic 𝑍 and the number of pairs 𝑁 . We can then calculate the effect size using the formula provided in the literature [70]: √𝑍 . 𝑁 In the Kruskal-Wallis test, we calculate eta-squared using the H-statistic as follows [70]: 𝜂 2 [𝐻 ] =

𝐻 −𝑘 +1 𝑛 −𝑘

where 𝐻 represents the value obtained from the Kruskal-Wallis test, 𝑘 is the number of groups, and 𝑛 denotes the total number of observations. The eta-squared estimate ranges from 0 to 1, and when multiplied by 100, it indicates the percentage of variance in the dependent variable explained by the independent variable. To determine the eta-squared value, we need to establish different strength ranges. This will allow us to compare the Kruskal-Wallis test with the Mann-Whitney test and Pearson’s correlation by defining the strength of the correlation [70]: (i) Negligible: 𝜂 2 [𝐻 ] < 0.01 (ii) Weak: 0.01 ≤ 𝜂 2 [𝐻 ] < 0.06 (iii) Moderate: 0.06 ≤ 𝜂 2 [𝐻 ] < 0.14 (iv) Strong: 𝜂 2 [𝐻 ] ≥ 0.14. 4

Results and Analysis

This section presents our different results and answers our research questions. 4.1 Results for 𝑅𝑄 1 – Noise impact on output assessment 4.1.1 Results for 𝑅𝑄 1.1 . To evaluate whether the distance between mutants and CUTs increase or decrease under noise, we use a kernel density estimate (KDE) plot in Fig. 5. For each noise ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:23

Fig. 5. (𝑅𝑄 1.1 ) Relationship between the original and mutated programs under various noise conditions, measured in terms of distance metrics.

model (displayed as rows), we compare the distances obtained in the corresponding noisy simulator against those from the noiseless simulator. The x-axis represents the distance between the CUT and its mutants in the noiseless simulator, while the y-axis shows the same distance when mutants are executed under noise. The black dashed line marks the case where noise has no impact on behavior (i.e., equal distances in both settings). Divergence from this line indicates how strongly noise affects program behavior. The columns correspond to the different distance metrics. Fig. 5 shows that noise has a noticeable impact on program behavior in the majority of cases. Expectation values appear least affected by noise, as most points cluster along the black dashed line, though some dispersion is still observed. Overall, the expectation values are much more compressed; the points lie so closely together that distinguishing behavioral differences between executions becomes difficult. In other words, the metric provides little actionable discrimination, with both equivalent and non-equivalent mutants map to similar values, making it less sensitive to meaningful behavioral variations. For the other four metrics, a consistent trend emerges: mutants that were originally similar to the CUT tend to show increased distances under noise, while highly divergent ones display reduced distances. This effect also appears to be more pronounced for the Fidelity (for density-matrix) and Hellinger (for output distribution) metrics, suggesting that these metrics may be more sensitive to noise. The pattern remains consistent across all noise models, with a slightly stronger impact observed for the Sherbrooke model. To further confirm these findings, we separately analyze the impact of noise on non-equivalent (Fig. 6a) and equivalent mutants (Fig. 6b). Both figures present boxplots of the noisy distances between mutants and their original counterparts. In these plots, blue boxes represent noiseless executions, while the three shades of green correspond to the different noise models. As in all boxplots in this paper, values close to zero indicate similar behavior, while higher values indicate greater behavioral differences—except for Fidelity, which behaves oppositely. As discussed in ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:24

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

(a) Impact of different noise conditions on the distance between CUTs and non-equivalent mutants.

(b) Impact of different noise conditions on the distance between CUTs and equivalent mutants. Fig. 6. (𝑅𝑄 1.1 ) Distance between all original and mutated programs, evaluated under various noise conditions.

Sect. 2.4.1, Fidelity equals 1 for perfect similarity, so its boxplots must be interpreted with vertical symmetry (i.e., values closer to 1 indicate greater similarity). One clear observation is that under noise, the distribution of non-equivalent mutants (Fig. 6a) becomes more concentrated (i.e., exhibits smaller variance) than in the noiseless case, indicating that noise tends to mask mutation effects by reducing their distance from the CUTs. However, most values remain above zero, showing that some behavioral differences between mutants and CUTs can still be detected despite noise (with the notable exception of expectation values, discussed below). Overall, the distributions remain quite spread, with considerably long whiskers. In contrast, equivalent mutants (Fig. 6b) display the opposite behavior, showing increased variance under noise. In the noiseless setting, results align with expectations: Trace distance and Fidelity indicate exact equivalence (0 for Trace distance, 1 for Fidelity), while the other metrics yield values close to, but not exactly, zero. The Hellinger and Jensen-Shannon distances exhibit longer whiskers, reflecting the stochastic nature of output distributions. Once noise is introduced, distances increase (or Fidelity decreases), highlighting the distorting effect of noise. These visual observations are quantitatively confirmed by the variance ratio 𝑉 𝑅 reported in Tab. 6. All p-values are extremely small (<0.05), confirming that the observed variance changes are statistically significant. For non-equivalent mutants, VR is consistently below 1, revealing a systematic compression of distances under noise. In contrast, equivalent mutants often exhibit VR values far above 1, corresponding to more scattered distributions and showing that noise strongly inflates distances among mutants that should be equivalent. A few exceptions appear for the Hellinger and Jensen-Shannon distances with equivalent mutants. Visually, these boxplots show wider interquartile ranges under noise, yet the corresponding VR values are slightly below 1, suggesting a more concentrated distribution. This apparent discrepancy arises because the noiseless distributions contain outliers that inflate the whiskers. Since VR captures overall dispersion (including extreme values), noise can appear to reduce variance relative to this inflated baseline. This highlights the importance of interpreting VR and p-values alongside visual plots, especially when distributions contain outliers.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:25

Table 6. (𝑅𝑄 1.1 ) Statistical test results for the relation between noiseless and noisy executions. For each metric and noise model, we report the variance ratio (𝑉 𝑅), p-value, and how the distribution changes under noise (more concentrated or more scattered), separately for non-equivalent and equivalent mutants. Metric

Noise Model

Non-Equivalent Mutants VR p-value Change

Trace Distance

Kyiv Brisbane Sherbrooke

0.885 0.896 0.837

< 0.05 < 0.05 < 0.05

Concentrated Concentrated Concentrated

8.05e+28 4.48e+28 2.48e+29

< 0.05 < 0.05 < 0.05

Scattered Scattered Scattered

Fidelity

Kyiv Brisbane Sherbrooke

0.739 0.751 0.625

< 0.05 < 0.05 < 0.05

Concentrated Concentrated Concentrated

6.34e+24 4.20e+24 2.53e+25

< 0.05 < 0.05 < 0.05

Scattered Scattered Scattered

Hellinger

Kyiv Brisbane Sherbrooke

0.836 0.803 0.778

< 0.05 < 0.05 < 0.05

Concentrated Concentrated Concentrated

0.820 0.789 0.772

< 0.05 < 0.05 < 0.05

Concentrated Concentrated Concentrated

Jensen–Shannon

Kyiv Brisbane Sherbrooke

0.832 0.794 0.765

< 0.05 < 0.05 < 0.05

Concentrated Concentrated Concentrated

0.801 0.765 0.750

< 0.05 < 0.05 < 0.05

Concentrated Concentrated Concentrated

Expectation Value

Kyiv Brisbane Sherbrooke

0.860 0.851 0.914

< 0.05 < 0.05 < 0.05

Concentrated Concentrated Concentrated

72.43 127.14 190.47

< 0.05 < 0.05 < 0.05

Scattered Scattered Scattered

VR

Equivalent Mutants p-value Change

These trends are consistent with the KDE plots (Fig. 5): the Kyiv and Brisbane models behave similarly, while Sherbrooke introduces slightly more noise, causing a larger difference with respect to the noiseless simulation. In Fig. 6a, the boxes for non-equivalent mutants under Sherbrooke are more compressed, suggesting reduced distance variability, while in Fig. 6b, the boxes for equivalent mutants are more spread out and with higher values (and lower for Fidelity), falsely indicating behavioral differences caused by noise. Accordingly, Tab. 6 consistently shows the strongest effects for Sherbrooke, with the lowest (respectively highest) VR values when variance decreases (respectively increases). Finally, expectation values behave differently from all other metrics: although they show near-zero values for equivalent mutants, as expected, they also show near-zero values for non-equivalent mutants. This indicates that expectation values are not reliable for detecting mutations, particularly in noisy environments. Answer to 𝑹𝑸 1.1 (Distance variance under noise): Under noisy conditions, the distance between mutants and CUTs varies significantly: equivalent mutants tend to show more dispersed values, whereas non-equivalent mutants appear more compressed. We observe a clear divergence between the noiseless and noisy cases, as the distributions no longer align with the expected. Among the noise models, the Sherbrooke simulator introduces more noticeable noise, leading to the largest deviations in measured distances. 4.1.2 Results for 𝑅𝑄 1.2 . To answer RQ1.2, we analyze which metric best distinguishes mutants from equivalent programs by comparing boxplots of the distances between each mutant and its corresponding CUT (Fig. 7). Results are reported across all simulator configurations (including the noiseless simulator and the three noisy variants) with mutants grouped as either equivalent or non-equivalent. Each sub-figure corresponds to a different distance metric. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:26

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

(a) Distances computed using the Trace distance metric.

(b) Distances computed using the Fidelity metric.

(c) Distances computed using the Hellinger distance metric.

(d) Distances computed using the Jensen-Shannon distance metric.

(e) Distances computed using the expectation value. Fig. 7. (𝑅𝑄 1.2 ) Comparison of distances between original and mutated programs under different noise conditions. Each sub-figure uses a different distance metric, with mutants grouped as equivalent or nonequivalent.

Using Trace distance (Fig. 7a), the noiseless simulator yields the expected outcome: most equivalent mutants remain perfectly undetected (with a distance of 0), while non-equivalent mutants are ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:27

clearly distinguishable (with distances > 0). Although the whiskers indicate some extreme values (close to 0 or 1), the median distance for non-equivalent mutants is around 0.3. Under the Kyiv and Brisbane noise models, the effect of noise becomes visible: equivalent mutants no longer have a distance of exactly 0, showing a small variance and a median distance below 0.1. The Sherbrooke model further increases this variance—consistent with 𝑅𝑄 1.1 , which identified it as the noisiest environment—with some equivalent mutants at a distance of 0.4 from their original counterpart. These observations have two key implications: • On the positive side, it remains possible to distinguish the majority of equivalent and nonequivalent mutants (i.e., a threshold can still be defined that separates most equivalent from most non-equivalent cases). • On the negative side, the existence of indistinguishable mutants suggests that computing a perfectly accurate mutation score becomes infeasible in noisy settings. When using Fidelity instead (Fig. 7b), the resulting plot is nearly a vertical mirror image of the one for Trace distance. Since high Fidelity indicates greater similarity between programs, an analogous analysis applies: Fidelity remains effective at distinguishing equivalent from nonequivalent mutants, although the interpretation is inverted (higher values indicate closer states). Importantly, this distinction remains clear in both noiseless and noisy scenarios, confirming that Fidelity—like Trace distance—is a robust metric for differentiating between the two classes of mutants across diverse hardware conditions. As Fig. 7c and Fig. 7d show very similar results, we can discuss the Hellinger and Jensen-Shannon distances together. The boxplots are notably more spread out compared to those for Trace distance and Fidelity. In particular, equivalent mutants span almost the entire range of distance values, although the majority are still clustered. In the noiseless scenario, the distance between the original and equivalent mutant program executions is generally below 0.1, while most non-equivalent mutants are more distant, typically between 0.2 and 0.6. This clear separation allows us to distinguish equivalent from non-equivalent mutants effectively. However, under noise, the situation becomes more complex. Equivalent mutants tend to be further from the CUTs, with distances ranging from 0.1 to 0.3. Unlike with Fidelity and Trace distance, it is no longer possible to clearly separate equivalent from non-equivalent mutants. This suggests that a trade-off between false negatives (undetected non-equivalent mutants) and false positives (equivalent mutants mistakenly identified as behavioral divergences) will always be necessary. Finally, the expectation value yields very poor results. The boxplots (Fig. 7e) are extremely compact, indicating minimal divergence in program execution, even when mutations have a significant impact. Moreover, equivalent and non-equivalent mutants overlap considerably, which indicate that adjusting the scale to observe smaller variations will likely not resolve the issue. To complement the visual analysis, we quantitatively assessed the discriminative power of each metric using a Mann–Whitney U test and Cliff’s delta (𝛿) as a non-parametric effect size. The results are reported in Tab. 7. For all metrics and hardware backends, the difference between equivalent and non-equivalent mutants is statistically significant (𝑝 < 0.05). However, the magnitude of the effect varies substantially across metrics. Density matrix–based metrics exhibit the strongest and most consistent separation. Trace distance and Fidelity achieve large effect sizes across all backends, with |𝛿 | ranging from 0.74 to 0.89 for Trace distance and from 0.79 to 0.97 for Fidelity, confirming the clear separation observed in the boxplots. Fidelity consistently attains the highest |𝛿 | values, indicating that it is the most discriminative metric overall.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:28

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Table 7. (𝑅𝑄 1.2 ) Statistical test results using Cliff’s delta for the distinction between Equivalent and NonEquivalent mutants. Each column reports, respectively, the metric under analysis, the hardware backend, the absolute Cliff’s delta, the associated p-value, and the magnitude of the observed relationship. Metric

Hardware

Cliff’s |𝜹 |

p-value

Strength

Trace Distance

Noiseless Brisbane Kyiv Sherbrooke

0.8904 0.7940 0.7801 0.7406

< 0.05 < 0.05 < 0.05 < 0.05

Strong Strong Strong Strong

Fidelity

Noiseless Brisbane Kyiv Sherbrooke

0.9662 0.8651 0.8521 0.7940

< 0.05 < 0.05 < 0.05 < 0.05

Strong Strong Strong Strong

Hellinger

Noiseless Brisbane Kyiv Sherbrooke

0.6700 0.4697 0.5110 0.4236

< 0.05 < 0.05 < 0.05 < 0.05

Strong Moderate Strong Moderate

Jensen-Shannon

Noiseless Brisbane Kyiv Sherbrooke

0.6773 0.5026 0.5378 0.4558

< 0.05 < 0.05 < 0.05 < 0.05

Strong Strong Strong Moderate

Expectation Value

Noiseless Brisbane Kyiv Sherbrooke

0.3948 0.1470 0.1936 0.1253

< 0.05 < 0.05 < 0.05 < 0.05

Moderate Negligible Weak Negligible

Hellinger and Jensen–Shannon distances achieve moderate to strong effect sizes depending on the backend. In the noiseless setting, both metrics still reach large effects (|𝛿 | ≈ 0.67), but under noisy conditions their discriminative power degrades to moderate levels (|𝛿 | ≈ 0.42–0.51), which is consistent with the increased overlap observed in Figs. 7c and 7d. Expectation value–based distances perform markedly worse. Except for the noiseless simulator (|𝛿 | = 0.39), all noisy backends exhibit only weak or negligible effects (|𝛿 | ≤ 0.19), confirming that expectation values fail to reliably distinguish equivalent from non-equivalent mutants. Answer to 𝑹𝑸 1.2 (Metric choice): Density matrix–based metrics are the most effective in distinguishing between equivalent and non-equivalent mutants, providing clearer separation. In contrast, output distribution–based metrics exhibit longer whiskers, which may lead to misclassifications between the two categories. Metrics based on expectation values are the least specific and make it harder to distinguish between mutant types, regardless of the noise level. 4.1.3 Results for 𝑅𝑄 1.3 . We answer 𝑅𝑄 1.3 in two steps. First, we assess whether the noiseless threshold 𝑇 Noiseless effectively detects non-equivalent mutants in a noiseless environment, and how it performs when applied to noisy environments. Second, we show that introducing a noise-specific threshold 𝑇 Noise significantly improves detection accuracy under noise. As discussed in Sect. 3.6, improving mutant detection corresponds to reducing both false positives and false negatives. In ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:29

(a) Mutant detection on the noiseless simulator, with threshold 𝑇 Noiseless .

(b) Mutant detection under Brisbane noise model, with threshold 𝑇 Noiseless .

(c) Mutant detection under Brisbane noise model, with threshold 𝑇 Brisbane . Fig. 8. (𝑅𝑄 1.3 ) Confusion matrices comparing the detection of equivalent and non-equivalent mutants across all distance metrics. The Y-axis corresponds to the true nature of the mutant (i.e., the expected result), while the X-axis represents the results of our mutant detection. Each sub-figure evaluates the alignment between noisy and noiseless detections using a different noise model and/or threshold strategy.

terms of confusion matrices, this improvement is reflected by a shift from the main diagonal entries toward the off-diagonal (i.e., fewer misclassifications and more correct detections). Fig. 8a shows the results of mutant detection in the noiseless environment using 𝑇 Noiseless . For Trace distance and Fidelity, false negatives are very low (5.31% and 2.12%, respectively), and no false positives are observed. This indicates that 𝑇 Noiseless is reasonable but slightly conservative, as some non-equivalent mutants remain undetected. For output distribution distances, detection errors (also referred to as misclassifications and defined as cases where either a false positive or a false negative occurs) remain below 15%, with similar rates for both types of errors, suggesting limited room for improvement. Consistent with 𝑅𝑄 1.1 and 𝑅𝑄 1.2 observations, the expectation value metric performs poorly, with more than 35% misclassification. Overall, 𝑇 Noiseless can be considered adequate in a perfect noiseless environment. When noise is introduced (Fig. 8b), the same threshold becomes suboptimal. For density matrix–based metrics (i.e., Trace and Fidelity), all programs are detected as mutants, whether equivalent or not. Hellinger and Jensen–Shannon metrics show more than 44% false positives for only about 2% false negatives. Expectation values perform even worse than before, with over 45% misclassification and nearly double the number of false positives w.r.t. the noiseless simulation. These results indicate that 𝑇 Noiseless is too low under noise, misclassifying behavioral drift caused by noise as actual mutations. Fig. 8c confirms this by showing detection results with the custom 𝑇 Brisbane threshold. As shown in Tab. 3, 𝑇 Brisbane was designed to be higher than 𝑇 Noiseless to account for noise-induced differences. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:30

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

The confusion matrices reveal that this adjustment significantly reduces false positives for all metrics. While false negatives increase, the overall misclassification rate greatly decreases: Trace distance 48.72% → 16.77%, Fidelity 48.72% → 14.38%, Hellinger 47.12% → 33.98%, and JensenShannon 47.28% → 30.36%. For Trace distance and Fidelity in particular, the false positive and false negative rates are now balanced, suggesting that the threshold leaves little room for further improvement. Answer to 𝑹𝑸 1.3 (Noise-specific threshold evaluation): In noisy environments, adopting a noise-specific threshold (𝑇 Noise ) significantly improves mutant detection over the noiseless threshold (𝑇 Noiseless ). While 𝑇 Noiseless performs well in noiseless simulators, it becomes overly sensitive under noise, misclassifying noise-induced variations as behavioral differences (up to 48.72% false positives). In contrast, 𝑇 Noise , tailored to each noise setting, substantially reduces false positives while maintaining a balanced trade-off with false negatives. 4.1.4 Results for 𝑅𝑄 1.4 . In the previous section, we assessed that a noise-specific threshold was necessary to detect true mutants from equivalent programs in noisy environments. In this section, to evaluate the choice of this particular custom threshold, we are going to compare it to two other thresholds, 𝑇 Middle and 𝑇 Above surrounding it. To visualize the impact of thresholds, Figs. 9a to 9e reuse the boxplots from 𝑅𝑄 1.2 (Sect. 4.1.2), this time with horizontal lines marking each threshold. This representation makes it possible to see at a glance how well a threshold separates equivalent from non-equivalent mutants. Consistent with our earlier discussion in Sect. 4.1.3, all metrics show that 𝑇 Noiseless , while suitable in a noiseless environment, is systematically too strict under noise: it falls below the first quartile (resp. above the third for Fidelity) for both equivalent and non-equivalent mutants, leading to almost all programs being flagged as mutants, whether equivalent or not. At the opposite extreme, 𝑇 Above is overly permissive. While for all metrics it lies above the third quartile (resp. below the first quartile for Fidelity) of equivalent mutants, meaning that they are rarely detected as mutants, for Hellinger, Jensen–Shannon, and expectation values (Figs. 9c, 9d, and 9e), 𝑇 Above is so extreme that most non-equivalent mutants also remain undetected, making it an unsuitable choice. Trace distance (Fig. 9a) performs only slightly better: 𝑇 Above lies between the median and first quartile, yet still fails to detect more than 25% of mutants. Fidelity (Fig. 9b) shows a comparable situation, with 𝑇 Above between the median and third quartile, leaving a substantial fraction of mutants undetected. Intermediate thresholds offer more nuanced trade-offs. 𝑇 Middle is conservative: for all metrics except expectation values, it lies below the first quartile of non-equivalent mutants (resp. above the third for Fidelity), ensuring most non-equivalent mutants are detected. However, it produces a high number of false positives, meaning developers would need to manually filter many false alarms. By contrast, 𝑇 Noise adopts a less conservative stance. For Trace distance and Fidelity, it approaches an optimal separation by sitting between the third (resp. first) quartile of equivalent mutants and the first (resp. third) quartile of non-equivalent ones. For Hellinger and Jensen–Shannon, however, neither 𝑇 Middle nor 𝑇 Noise provides a satisfactory separation: 𝑇 Middle is too sensitive, while 𝑇 Noise allows too many non-equivalent mutants to slip through. Additionally, Tab. 8 reports the accuracy and F1-score of mutant detection under each noise model. Results are shown for all five metrics and four thresholds: 𝑇 Noiseless , 𝑇 Middle , 𝑇 Noise (i.e., 𝑇 Kyiv , 𝑇 Brisbane , or 𝑇 Sherbrooke ), and 𝑇 Above . The best values are highlighted in green and the worst in red. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:31

(a) Distances computed using the Trace distance metric.

(b) Distances computed using the Fidelity metric.

(c) Distances computed using the Hellinger distance metric.

(d) Distances computed using the Jensen-Shannon distance metric.

(e) Distances computed using the expectation value. Fig. 9. (𝑅𝑄 1.4 ) Comparison of distances between original and mutated programs under different noise conditions. Each sub-figure uses a different distance metric, with mutants grouped as equivalent or nonequivalent. Horizontal lines indicate the mutant detection thresholds.

Both accuracy and F1-score are consistent across the three noise models (Kyiv, Brisbane, and Sherbrooke). The only exception is for the expectation value on Sherbrooke, where 𝑇 Middle and 𝑇 Noiseless swap ranks, though the difference is negligible (< 1%). ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:32

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Table 8. (𝑅𝑄 1.4 ) Accuracy and F1-score for mutation detection across different noise models. For each metric and noise model, the best-performing threshold is highlighted in green , and the worst in red .

Accuracy (%)

F1-Score (%)

Metric

Threshold

Kyiv

Brisbane

Sherbrooke

Kyiv

Brisbane

Sherbrooke

Trace Distance

𝑇 Noiseless 𝑇 Middle 𝑇 Noise 𝑇 Above

51.28% 59.33% 82.94% 83.85%

51.28% 61.49% 83.22% 83.67%

51.28% 59.44% 80.92% 82.51%

67.80% 70.18% 83.28% 81.35%

67.80% 71.28% 83.86% 81.06%

67.80% 70.21% 80.87% 80.78%

Fidelity

𝑇 Noiseless 𝑇 Middle 𝑇 Noise 𝑇 Above

51.28% 58.39% 85.17% 83.81%

51.28% 59.74% 85.63% 83.57%

51.28% 59.53% 82.06% 81.56%

67.80% 70.67% 85.84% 81.26%

67.80% 71.35% 86.30% 80.92%

67.80% 71.24% 82.30% 80.46%

Hellinger

𝑇 Noiseless 𝑇 Middle 𝑇 Noise 𝑇 Above

53.98% 73.03% 69.31% 55.17%

52.88% 70.45% 66.02% 55.16%

53.69% 63.87% 61.53% 55.38%

67.85% 74.89% 63.62% 33.72%

67.46% 73.70% 57.99% 33.75%

67.77% 70.77% 49.64% 34.33%

Jensen Shannon

𝑇 Noiseless 𝑇 Middle 𝑇 Noise 𝑇 Above

53.75% 72.87% 73.70% 56.79%

52.72% 69.45% 69.64% 56.78%

53.52% 61.96% 64.79% 57.28%

67.80% 75.63% 70.36% 37.65%

67.43% 74.00% 64.11% 37.66%

67.75% 70.39% 55.82% 38.86%

Expectation Value

𝑇 Noiseless 𝑇 Middle 𝑇 Noise 𝑇 Above

55.58% 55.09% 52.95% 51.69%

54.22% 53.83% 51.45% 51.08%

53.53% 53.76% 51.33% 50.87%

62.06% 37.25% 27.74% 11.21%

61.52% 37.49% 22.80% 17.69%

61.27% 37.13% 23.68% 18.97%

For accuracy, 𝑇 Noise achieves the best scores for Fidelity and Jensen–Shannon. For Hellinger and Trace distance, it ranks second, with only a marginal gap: in Hellinger, 𝑇 Middle performs slightly better, while Trace distance is most accurate with a higher threshold such as 𝑇 Above . Across all four metrics, 𝑇 Noiseless consistently yields the worst accuracy. For F1-score, 𝑇 Noise is the best threshold for Trace distance and Fidelity. In contrast, output distribution metrics favor lower thresholds: here, 𝑇 Middle and sometimes 𝑇 Noiseless outperform 𝑇 Noise , while 𝑇 Above performs the worst. This highlights a clear distinction: output distribution metrics benefit from lower thresholds, while density matrix–based metrics require higher thresholds (with 𝑇 Noiseless again being the worst). Notably, for every metric except expectation value, at least one threshold achieves an accuracy of 73.03% on Kyiv, 69.64% on Brisbane, and 63.87% on Sherbrooke. Similarly, the best F1-scores reach 74.89%, 73.70%, and 70.39%, respectively. These results confirm that mutation analysis is feasible on quantum systems even under noise, provided that proper tools and threshold strategies are applied. By contrast, the expectation value performs poorly. Its accuracy ranges only between 50% and 56%, barely above random classification, while its F1-score peaks at 62% with 𝑇 Noiseless , still far below the other metrics. Answer to 𝑹𝑸 1.4 (Additional threshold comparison): While 𝑇 Noise generally outperforms the baseline 𝑇 Noiseless in noisy environments, it is not universally optimal. For density matrix–based metrics (Trace distance and Fidelity), 𝑇 Noise achieves a balanced trade-off between accuracy and error rates. In contrast, for output distribution metrics (Hellinger and Jensen–Shannon), 𝑇 Middle performs slightly better, though

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:33

Table 9. (𝑅𝑄 2 ) Statistical test results and effect sizes for the relation between noise and circuit characteristics for equivalent mutants. Each column reports, respectively, the metric under analysis, the specific characteristic assessed, the effect size derived from the statistical analysis, the associated p-value indicating the level of significance, and the magnitude of the observed relationship. Metric

Variable

Effect-size

p-value

Strength

Trace Distance

#qubits depth #gates

0.4698 0.1941 0.3816

< 0.05 < 0.05 < 0.05

Moderate Weak Moderate

Fidelity

#qubits depth #gates

-0.5267 -0.1879 -0.4484

< 0.05 < 0.05 < 0.05

Strong Weak Moderate

Hellinger

#qubits depth #gates

0.1332 -0.0668 -0.0119

< 0.05 < 0.05 < 0.05

Weak Negligible Negligible

Jensen-Shannon

#qubits depth #gates

0.1447 -0.0528 0.0053

< 0.05 < 0.05 < 0.05

Weak Negligible Negligible

Expectation Value

#qubits depth #gates

-0.0065 -0.2036 -0.3383

< 0.05 < 0.05 < 0.05

Negligible Weak Moderate

more conservative (i.e., more false positives). 𝑇 Above remains consistently ineffective, leaving many non-equivalent mutants undetected. Overall, 𝑇 Noise is a strong practical choice, but the best threshold ultimately depends on the metric and the desired trade-off. 4.2

Results for 𝑅𝑄 2 – Circuit characteristics impact on noise in mutation analysis

In this second research question, we investigate the relationship between various circuit-specific characteristics of quantum programs and the impact of noise in the context of mutation analysis. We define these characteristics as the structural properties of a quantum circuit, referring to the number of qubits, the number of gates, and the circuit depth. To analyze this relationship, we use the statistical tests described in Sect. 3.6. The evaluation is conducted across all noise models simultaneously, as our goal is to study the general effects of these characteristics under noise. First, Tab. 9 shows the correlation of the circuit characteristics with noise in the equivalent mutants. We can observe that for the output distribution-related metrics, Hellinger and JensenShannon, the variables exhibit lower correlation compared to the other metrics, with only the #qubits showing some weak correlation. In contrast, the remaining metrics demonstrate generally higher correlation. While the depth of the circuit consistently shows weak correlation, the #gates consistently achieves a moderate correlation. The #qubits varies the most, showing a negligible correlation with expectation values and demonstrating moderate to strong correlation with density matrix-related metrics, Trace distance and Fidelity. Tab. 10 presents the results of the correlation analysis for non-equivalent mutants on the relationship between circuit characteristics and noise in mutation. In general, we observe a much weaker correlation strength compared to equivalent mutants. There is only a weak correlation observed for depth using the Hellinger metric, and some higher correlations for all variables concerning expectation values. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:34

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Table 10. (𝑅𝑄 2 ) Statistical test results and effect sizes for the relation between noise and circuit characteristics for non-equivalent mutants. Each column reports, respectively, the metric under analysis, the specific characteristic assessed, the effect size derived from the statistical analysis, the associated p-value indicating the level of significance, and the magnitude of the observed relationship. Metric

Variable

Effect-size

p-value

Strength

Trace Distance

#qubits depth #gates

0.0323 -0.0561 -0.0480

< 0.05 < 0.05 < 0.05

Negligible Negligible Negligible

Fidelity

#qubits depth #gates

-0.0840 -0.0326 -0.0760

< 0.05 < 0.05 < 0.05

Negligible Negligible Negligible

Hellinger

#qubits depth #gates

0.0763 -0.1022 -0.0683

< 0.05 < 0.05 < 0.05

Negligible Weak Negligible

Jensen-Shannon

#qubits depth #gates

0.0764 -0.0867 -0.0564

< 0.05 < 0.05 < 0.05

Negligible Negligible Negligible

Expectation Value

#qubits depth #gates

-0.1989 -0.2261 -0.3651

< 0.05 < 0.05 < 0.05

Weak Weak Moderate

Overall, we see that the noise in equivalent mutants is more closely correlated with the circuit characteristics than in non-equivalent mutants. Among the metrics, the expectation value has the most significant influence of the circuit characteristics, followed by the density matrix-based metrics in equivalent mutants. The output distribution-based metrics seem to be less affected by the circuit characteristics, suggesting that, in this case, the structural characteristics of a circuit do not significantly impact the amount of noise the output distribution experiences. Among the circuit characteristics, #gates and #qubits appear to have the highest correlation with noise, indicating that, depending on the #gates and #qubits, the circuit may exhibit more or less noise. Answer to 𝑹𝑸 2 (Circuit characteristics): The structural characteristics of a circuit appear to have some minor influence on output distribution-based metrics, while having a greater impact on metrics based on the density matrix and expectation values. The noise in equivalent mutants is more influenced by circuit characteristics than in non-equivalent mutants. The variable that most influences the noise depends on the metric used; the #gates is the strongest for expectation values, while the #qubits is more significant for the other metrics. 4.3

Results for 𝑅𝑄 3 – Algorithm characteristics impact on noise in mutation analysis

In this research question we study the relationship between algorithm characteristics and noise in the context of mutation analysis. We define algorithm characteristics as the characteristics that differentiate quantum circuits based on their behavior, including input type, output type, and the algorithm itself. Tab. 11 illustrates the relationship between these characteristics and the noise observed in mutation for equivalent mutants. The results clearly indicate that the algorithm exerts a stronger ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:35

Table 11. (𝑅𝑄 3 ) Statistical test results and effect sizes for the relation between noise and algorithm characteristics for equivalent mutants. Each column reports, respectively, the metric under analysis, the specific characteristic assessed, the effect size derived from the statistical analysis, the associated p-value indicating the level of significance, and the magnitude of the observed relationship. Metric

Variable

Effect Size

p-value

Strength

Trace Distance

algorithm output type input type

0.2058 0.0834 0.0779

< 0.05 < 0.05 < 0.05

Strong Negligible Negligible

Fidelity

algorithm output type input type

0.3092 0.2456 0.1009

< 0.05 < 0.05 < 0.05

Strong Weak Weak

Hellinger

algorithm output type input type

0.5413 0.2961 0.1420

< 0.05 < 0.05 < 0.05

Strong Weak Weak

Jensen-Shannon

algorithm output type input type

0.5193 0.2932 0.1296

< 0.05 < 0.05 < 0.05

Strong Weak Weak

Expectation Value

algorithm output type input type

0.2938 0.1301 0.0409

< 0.05 < 0.05 < 0.05

Strong Weak Negligible

influence on noise than either the input type or output types. Across all metrics, we can observe that the algorithm achieves a strong correlation, showing a greater effect size for the output distribution-based metrics, Hellinger and Jensen-Shannon. Aside from that, the metrics appear to show similar behavior, with both the input type and output type having a comparably minimal influence on them. Tab. 12 presents the results of the correlation tests conducted for the non-equivalent mutants concerning the algorithm characteristics and their influence on noise. We can observe that the algorithm remains the characteristic that influences noise the most. Additionally, for non-equivalent mutants, the Hellinger and Jensen-Shannon metrics show more influence from the output type compared to equivalent mutants. The input type appears to have a negligible effect on all the metrics and is non-significant in the case of Hellinger. Overall, for both mutant groups, we can clearly state that the algorithm is the characteristic that most influences the noise, with the output distribution metrics being the most affected, as indicated by the effect size. This indicates that each independent algorithm may have its own specific noise, as they consist of quantum circuits with different gate sets and arrangements, making it more challenging to generalize across various quantum circuits. Additionally, the output type also has some influence, indicating that depending on the circuit’s output type, an algorithm might exhibit more or less noise. Finally, the input type does not have a noticeable impact on the noise effect, meaning that the initialization of the qubits does not directly affect the circuit’s noise. Answer to 𝑹𝑸 3 (Algorithm characteristics): The algorithm is the characteristic that shows the strongest correlation with noise in the quantum circuits. The input type and output type display similar behavior, although the output type has a slightly greater influence, particularly on the metrics based on output

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:36

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Table 12. (𝑅𝑄 3 ) Statistical test results and effect sizes for the relation between noise and algorithm characteristics for non-equivalent mutants. Each column reports, respectively, the metric under analysis, the specific characteristic assessed, the effect size derived from the statistical analysis, the associated p-value indicating the level of significance, and the magnitude of the observed relationship. Metric

Variable

Effect Size

p-value

Strength

Trace Distance

algorithm output type input type

0.1482 0.2271 0.0206

< 0.05 < 0.05 < 0.05

Strong Weak Negligible

Fidelity

algorithm output type input type

0.0150 0.0519 0.0206

< 0.05 < 0.05 < 0.05

Weak Negligible Negligible

Hellinger

algorithm output type input type

0.4128 0.3607 0.0637

< 0.05 < 0.05 0.8629

Strong Moderate Not-Significant

Jensen-Shannon

algorithm output type input type

0.3833 0.3587 0.0451

< 0.05 < 0.05 < 0.05

Strong Moderate Negligible

Expectation Value

algorithm output type input type

0.2761 0.0377 0.0338

< 0.05 < 0.05 < 0.05

Strong Negligible Negligible

distribution in the case of non-equivalent mutants. Overall, all metrics exhibit similar behavior, with output distribution-based metrics being more affected by the magnitude of the effect. 4.4

Results for 𝑅𝑄 4 – Mutation characteristics impact on noise in mutation analysis

In this last research question we address mutation characteristics and their relationship with noise. We define these characteristics as the specific characteristics that define the mutations applied: the operator, the type of mutated gate, and the relative position. Tab. 13 presents the results of the correlation test conducted to examine the relationship between mutation characteristics and noise in equivalent mutants. It is important to note that, as we generate equivalent mutants using the gate reversibility principle (see Sect. 3.2.2), we do not have different operators to analyze. Thus, the operator type was not included for these equivalent mutants analysis. From the relationship between the mutation characteristics and the noise in the equivalent mutant, we can observe that both characteristics have either a negligible or a non-significant effect on almost all the metrics. The only instances where one of the variables, gate type, shows a weak correlation are for the metrics based on density matrices, Trace distance and Fidelity. While this correlation is still weak, it suggests that the density matrices are more influenced by the mutation characteristics than the other metrics. Tab. 14 illustrates the relationship between mutation characteristics and the noise affecting nonequivalent mutants. Similarly to equivalent mutants, the relative position shows a negligible relationship with the noise across all metrics. Regarding gate type, we observe a slightly stronger effect on non-equivalent mutants compared to equivalent mutants, although the effect is still weak. The variable that exhibits the strongest correlation in this case is the operator, which shows a moderate correlation strength with Fidelity. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:37

Table 13. (𝑅𝑄 4 ) Statistical test results and effect sizes for the relation between noise and mutation characteristics for equivalent mutants. Each column reports, respectively, the metric under analysis, the specific characteristic assessed, the effect size derived from the statistical analysis, the associated p-value indicating the level of significance, and the magnitude of the observed relationship. Metric

Variable

Effect Size

p-value

Strength

Trace Distance

gate type relative position

0.1117 0.0235

< 0.05 < 0.05

Weak Negligible

Fidelity

gate type relative position

0.2300 -0.0237

< 0.05 < 0.05

Weak Negligible

Hellinger

gate type relative position

0.0475 0.0080

< 0.05 0.4778

Negligible Not-Significant

Jensen-Shannon

gate type relative position

0.0417 0.0088

< 0.05 0.6088

Negligible Not-Significant

Expectation Value

gate type relative position

0.0600 0.0097

< 0.05 0.5034

Negligible Not-Significant

Table 14. (𝑅𝑄 4 ) Statistical test results and effect sizes for the relation between noise and mutation characteristics for non-equivalent mutants. Each column reports, respectively, the metric under analysis, the specific characteristic assessed, the effect size derived from the statistical analysis, the associated p-value indicating the level of significance, and the magnitude of the observed relationship. Metric

Variable

Effect Size

p-value

Strength

Trace Distance

gate type operator relative position

0.1948 0.0440 -0.0210

< 0.05 < 0.05 < 0.05

Weak Weak Negligible

Fidelity

gate type operator relative position

0.1721 0.0747 -0.1120

< 0.05 < 0.05 < 0.05

Weak Moderate Weak

Hellinger

gate type operator relative position

0.1412 0.0168 0.0222

< 0.05 < 0.05 < 0.05

Weak Weak Negligible

Jensen-Shannon

gate type operator relative position

0.1424 0.0186 0.0212

< 0.05 < 0.05 < 0.05

Weak Weak Negligible

Expectation Value

gate type operator relative position

0.0773 0.0018 0.0394

< 0.05 < 0.05 < 0.05

Negligible Negligible Negligible

Generally, the mutation characteristics do not appear to have a strong influence on the noise in quantum circuits. The relative position of mutations seem to have a negligible effect, suggesting that the position of the mutation does not affect the noise effect. The gate type shows a slightly higher correlation with noise, but this relationship remains weak. When mutating a single gate operation, the variability in noise introduced by that single gate does not have a major impact. The characteristic that seems to have the greatest effect is the operator itself. Introducing a new gate may influence noise differently than replacing an existing gate or removing a gate. Regarding ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:38

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

the metrics, the expectation values seem to be the least influenced by mutation characteristics, while the other metrics behave in a similar manner. Answer to 𝑹𝑸 4 (Mutation characteristics): The mutation characteristics have a low influence on the noise in the circuit. Among these characteristics, the operator appears to have the most significant impact, achieving a moderate correlation for Fidelity in non-equivalent mutants. In terms of the metrics, all metrics behave in a similar manner, except for the expectation value, which consistently shows a negligible effect. Overall, the non-equivalent mutants are slightly more affected than the equivalent ones, although both still exhibit a weak correlation. 5

Discussion

This section discusses some recommendations and implications of our results on research and development practices, and discusses possible threats the validity of the study. 5.1

Recommendations and Implications for Researchers

The findings of this study highlight two key aspects that enhance the field of quantum mutation analysis: the assessment of output assessment methods in the presence of noise and the understanding of how characteristics of circuits and mutations influence the effects of noise on mutant detection. These contributions provide both insights and practical guidance on thresholding and fault detection for quantum software testing on noisy quantum computers, helping researchers address the challenges they present. 5.1.1 Output Assessment Methods under Noise. Our findings on output assessment methods offer a clear path for researchers working in noisy quantum environments. We found that noise fundamentally changes the behavior of mutants, making assumptions from noiseless conditions unreliable on noisy hardware. Although this study is framed through the lens of mutation analysis, its broader goal is to evaluate the robustness of output assessment metrics under noise. In this sense, mutation analysis serves as a systematic means to probe metric sensitivity and resilience. The experiments show that density matrix-based metrics consistently provide a clearer separation between the mutants, especially the trace distance metric. However, due to their high computational complexity and the inability to obtain them from real quantum computers, their practical application is limited. We recommend using them as a baseline for experiments on simulators to benchmark and validate results. For practical use cases, we recommend prioritizing output distribution-based metrics, with Hellinger distance being the most reliable. These metrics offer a pragmatic trade-off between accuracy and deployability on actual quantum hardware, making them a reliable choice for experiments on both simulators and real quantum computers. In contrast, expectation value-based metrics failed to reliably distinguish mutant categories, even in lower-noise environments, and should therefore be avoided for behavioral comparison tasks. The role of thresholds is equally important in our analysis. We demonstrate that using noisespecific thresholds (𝑇 Noise ) significantly reduces misclassification rates, especially when applied to density matrix-based metrics. In contrast, static thresholds based on ideal scenarios (𝑇 Noiseless ) tend to inflate false positive rates. For instance, adopting the noise-specific threshold (𝑇 Noise ) reduced misclassification rates for trace distance and for fidelity, confirming the practical value of noise-aware thresholds. Nevertheless, the issue of threshold is not completely resolved; for instance, metrics like Hellinger or Jensen–Shannon tend to perform better with more conservative thresholds (𝑇 Middle ), while overly permissive thresholds (𝑇 Above ) generally lead to under-performance. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:39

These findings highlight that reliable mutation detection or, more generally, reliable behavioral differentiation, cannot depend on a single strategy. Instead, it requires an adaptive, metric-sensitive, and noise-calibrated approach that accounts for the variability of quantum hardware. Finally, we emphasize that the thresholds derived in this study can be applied in different ways depending on the experimental context. They can be directly employed for experiments involving the same noise models (Brisbane, Kyiv, and Sherbrooke), thereby ensuring consistency and comparability across experiments. Alternatively, the proposed methodology provides a foundation for recalibrating thresholds when targeting new hardware platforms or updated noise profiles. In such cases, researchers should replicate the procedure to derive thresholds tailored to their specific experimental conditions, either by reusing our dataset or by retraining the thresholds on newly collected quantum programs. 5.1.2 Characteristics. In a noiseless environment, Mendiluze Usandizaga et al. [71], already analyzed the effect that various circuit, algorithm and mutation characteristics have in relation to mutant detection. In our study, we aimed to explore these relationships further. Specifically, we wanted to investigate how these relations translate to a noisy environment and whether the conclusions drawn by Mendiluze Usandizaga et al. still hold true in the presence of noise. As has been pointed out in the literature [50], our findings support that noise is not a uniform disturbance, its effects are significantly influenced by the characteristics of the quantum program. In our study, we highlighted that not only does noise not behave similarly across all conditions, but that some characteristics have a greater impact on the noise disturbance, thereby masking more or less the presence of faults in the circuit. The properties of the circuit and the algorithm have a much greater impact on noise than the properties of the mutations themselves. This indicates that noise manifestation is more dependent on the circuit in which the mutation is applied rather than on the mutation itself. In contrast to the conclusions drawn by Mendiluze Usandizaga et al., who state that circuit characteristics do not significantly affect the detection of mutants, we conclude that noise highly depends on the original circuit, which alters the detection of the mutants in noisy environments. Consequently, mutations cannot be evaluated in an abstract manner; rather, they must be considered within the context of the programs to which they are applied. Specifically, we found that circuit-level characteristics, such as the number of qubits and gate depth, more significantly impact density matrix and expectation value metrics. This finding supports our earlier recommendation for using distribution-based metrics for practical applications, as they appear more resilient to these circuit-level noise effects. Interestingly, equivalent mutants were also found to be more sensitive to noise, making them even more problematic for the purpose of quantum mutation analysis. Furthermore, algorithmic characteristics had the most substantial effect. The types of input and output, particularly the output type, were crucial in how noise affected mutant detection. This aligns with the findings of the study conducted by Mendiluze Usandizaga et al., which indicates that the detection of mutants is related to the output type of the algorithm. Since each algorithm is structurally different, it contains unique noise patterns. This emphasizes the importance of treating each algorithm as an individual case and suggests a future research direction focused on developing noise-specific and algorithm-agnostic metrics. Characteristics at the mutation level, contrary to the findings of Mendiluze Usandizaga et al., have a limited impact. Although there are some subtle correlations related to operator type in fidelity measures, their influence on other metrics remains weak. Practically speaking, this suggests that the disturbance caused by noise has a greater impact on fault detection than the faults themselves. This raises concerns about how noise can mask faults in real environments. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:40

5.2

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Threats to Validity

While our approach provides a structured framework for noise-aware mutation analysis, several limitations must be acknowledged to ensure a comprehensive understanding of its validity and applicability. We discuss these limitations in terms of construct, internal, and external validity threats, following established guidelines from the literature [58, 67]. Construct Validity. In our context, construct validity is the extent to which our experimental setup and metrics accurately measure mutant detectability and noise sensitivity. A key threat arises from our reliance on noisy simulators rather than real quantum hardware. While simulators offer precise control over noise models and ensure experimental reproducibility, they cannot fully reproduce hardware-specific phenomena such as calibration errors, crosstalk, or drift. Consequently, the observed behavior may not perfectly align with that of physical quantum devices. To mitigate this limitation, we employed some of the most realistic noise models publicly available from IBM, derived from empirical calibration data of real hardware. This limitation is shared across much of the current quantum software engineering literature (e.g., [33, 35, 61, 69]), where simulator-based studies remain the norm due to restricted access to quantum devices. Importantly, our contribution extends beyond the specific results obtained here: we propose a generalizable methodology for evaluating mutation sensitivity under noise, which can be directly replicated across different hardware backends and noise profiles, thereby supporting reproducibility and long-term applicability as technology evolves. Moreover, even when limited to simulation, our metrics provide actionable insights for assessing quantum testing techniques and for guiding the design and evaluation of noise mitigation strategies. The choice of the metrics constitutes another potential threat. As part of our evaluation relies on expectation-value-based and output-distribution-based metrics, the results are inherently dependent on the selected measurement basis. An inappropriate basis may reduce the sensitivity of certain metrics to behavioral differences between mutants. Furthermore, the stochastic nature of quantum measurement makes the number of shots a critical factor for obtaining statistically meaningful results. To mitigate these threats, each circuit was executed with 10,000 shots, ensuring statistical stability and reproducibility. A further construct-related limitation concerns the choice of output distribution distance. While Muskit [45] employs the chi-square test as a distance metric, we intentionally avoid using it due to its restrictive assumptions and limited interpretability in our context. The chi-square test presupposes distributions with identical support and sufficiently large expected frequencies [15], conditions that rarely hold in quantum program executions. Even when two executions produce distributions over the same number of shot count and nominal output range, their effective support may differ. For example, consider a two-qubit circuit which returns two outputs: |00⟩ and |11⟩. If a single shot out of 10, 000 yields 01 or 10 due to noise, the chi-square statistic will spike, effectively reporting a maximal distance. In contrast, other metrics return a negligible distance, reflecting that the two distributions are almost identical. In such cases, the output of the chi-square test becomes difficult to interpret: the reported distance exaggerates a minor and non-semantic deviation, obscuring the true similarity between executions. Our attempts to mitigate this, either by enforcing exact support alignment or by smoothing probabilities, proved unsatisfactory. The former tends to over-penalize minor stochastic effects, while the latter risks suppressing meaningful discrepancies between the two distributions. We therefore adopt alternative distance metrics that more faithfully reflect the probabilistic and noisy nature of quantum program execution. Our empirical threshold definition relies solely on executions of the CUT under a given noise model. While this provides a consistent and model-specific baseline for identifying deviations in realistic noisy environments, it does not incorporate additional information such as circuit-level or ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:41

algorithmic characteristics. At present, there is no established reference or prior threshold definition to compare against, so our choice should be seen as a first practical approximation rather than a definitive model for threshold calibration under noise. Internal Validity. A potential internal threat arises from the fidelity of the noise models used in simulation. We assume that these models faithfully reproduce realistic noise behavior, but any simplification or incomplete modeling may bias the observed relationships between noise and mutant behavior. We mitigated this by selecting IBM noise models derived from empirical calibration data of real devices, providing a reasonable proxy for physical noise. Nevertheless, discrepancies between simulated and actual hardware behavior may still affect causal interpretations. Another internal threat concerns the representativeness of generated mutants. Equivalent mutants, produced via the gate-reversibility principle (Sect. 3.2.2), may not correspond to practical developer errors. However, their inclusion is intentional: they provide controlled conditions for isolating the influence of noise on distinguishability, thereby validating the causal relationship between noise intensity and mutant behavior. Although these mutants are synthetic, they are instrumental in studying how noise obscures otherwise detectable behavioral differences and in calibrating threshold-based detection strategies. Finally, stochastic fluctuations inherent to quantum measurement and simulator nondeterminism could introduce variability in observed distances between executions. We mitigated this by repeating each experiment multiple times under identical conditions and aggregating the results statistically, ensuring that the observed effects are robust and not driven by randomness or execution order. External Validity. Our study focuses on widely used quantum programs from the literature, analyzing multiple versions that differ in qubit count to assess scalability. However, the largest circuits examined contain only eight qubits, which is modest compared to the capabilities of emerging quantum hardware, as current quantum computers can execute larger circuits. Unfortunately, this capability does not reflect the limitations of quantum simulations as they depend on the capabilities of the classical machine executing them. In our case, we faced resource constraints primarily related to execution time and storage. Moreover, given the current state of quantum providers, the cost of conducting such empirical evaluations on real quantum computers would be infeasible. Extending the analysis to larger circuits would likely require restricting evaluation to output distribution–based metrics, as density matrix–based ones become computationally infeasible. The mutation operators employed in this study were reused from the state-of-the-art mutation testing framework Muskit [45]. While this choice enhances reproducibility and alignment with prior work, it may limit representativeness. In particular, other mutation schemes (such as gate parameter perturbations) could further diversify the mutations space and provide a broader assessment of mutation sensitivity under noise. Regarding noise modeling, we focused on IBM’s publicly available backends, which are widely used and well-documented within the quantum community. At the time of this study, we incorporated all noise models available through IBM’s API, assuming they offered realistic representations of publicly accessible quantum devices. However, our methodology is easily transferable to other hardware providers or custom noise models, facilitating replication and cross-platform comparison. Finally, metrics based on density matrices are inherently restricted to simulation environments, limiting their direct applicability to current quantum hardware. Although computationally expensive, they remain valuable from a foundational perspective, enabling fine-grained observation of quantum state evolution and offering deeper insight into both mutation behavior and noise effects. Their exponential growth with system size, however, imposes practical constraints that must be acknowledged when extending our methodology to larger systems or hardware-based experiments.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:42

6

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Related Work

This section reviews related works. We begin with existing quantum mutation analysis frameworks that inform our approach. Then, we recognize relevant studies in quantum noise estimation and mitigation, which are crucial to help situate our contribution within the field. 6.1

Quantum Mutation Analysis

Developed by Mendiluze Usandizaga et al., Muskit [45] is a Python-based mutation tool for Qiskit programs. It supports the mutation of 19 Qiskit gates and offers several mutation operators, such as gate insertion, deletion, and replacement. To determine whether mutants are detected by tests, Muskit requires users to specify the quantum program’s specifications, which demands expertise in quantum computing. Although effective, the large number of mutants generated—many of which may be equivalent or irrelevant—indicates the need for further refinement of the tool. QMutPy [21–23] extends Muskit’s capabilities by supporting 40 Qiskit gates (21 more than Muskit) and it introduces two additional mutation operators for measurement calls: measurement insertion and measurement deletion. QMutPy also addresses bug patterns identified in [80] and reduces the number of equivalent mutants by mutating quantum gates with “syntactically-equivalent” gates of the same number and types of arguments. Unlike Muskit, QMutPy does not require the specification of the quantum program. Instead, it relies on test assertions from Qiskit-Aqua’s manually written tests to assess whether mutants are detected, whereas limited, it simplifies its usage for practitioners. Both QMutPy and Muskit can generate a large number of mutants; however, they are prone to producing equivalent or redundant mutants, as well as mutants that are computationally infeasible to execute due to the associated computational overhead. None of these tools has been tested in noisy environments, highlighting the urgent need for mutation analysis methods that are robust to noise. Our work explores the interplay between noise and quantum mutation analysis. Specifically, we compare results obtained under three different noise models and evaluate the impact of various mutation and circuit characteristics on mutant survivability through a largescale empirical evaluation. Our study aims to provide evidence that can guide researchers and practitioners in designing meaningful mutants while incorporating noise robustness into mutation analysis strategies. Mendiluze Usandizaga et al. [71] also conducted an empirical study on quantum circuit mutants. In this study, the authors analyze how various characteristics affect the detection of these mutants. They define a metric called the survival rate, which indicates the probability of survival for a mutant with specific characteristics. The study consists of more than 300 CUTs and their respective 700,000 mutants, making it the largest analysis to date of quantum circuit mutants. The authors examine different groups of characteristics, including those related to the quantum circuits, algorithmic behavior, and mutation characteristics. Their focus is on the survival of mutants under noiseless simulations, using a single detection criterion based on output distribution assessment with the chi-square detection metric and a threshold of 0.01 for test detections. In our study, we similarly analyze the relationship between these characteristics; however, we investigate the behavior of mutants under noisy conditions. In contrast to the previous research, we also explore the feasibility of different output assessment methods and metrics, while establishing new thresholds. 6.2

Quantum Noise

Quantum noise has become an important topic recently. Several noise estimation and mitigation techniques have emerged to assess the noise in quantum programs and mitigate its impact. 6.2.1 Noise Estimation. Hung et al. [35] and Tao et al. [69] both tackle the challenge of analyzing and reasoning about errors in noisy programs, adopting distinct yet complementary approaches. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:43

Hung et al. propose the Logic of Quantum Robustness (LQR), a formal framework that introduces the (𝑄, 𝜆)-diamond norm to quantify the effect of noise on quantum programs. This norm measures the distance between a noisy program and its ideal counterpart, restricted to those input states that meet the predicate 𝑄 with degree at least 𝜆. Here, 𝑄 constrains the admissible inputs, while 𝜆 specifies the extent to which these inputs must fulfill the predicate. By adopting this conditional perspective, LQR enables more precise robustness bounds, particularly when knowledge about specific devices or input states is available. The framework also accounts for entanglement between program components, allowing compositional reasoning about quantum errors. Building on Quantum Hoare Logic, LQR incorporates these robustness norms into logical judgments of the form (𝑄, 𝜆) ⊢ 𝑃˜ ≤ 𝜖, ˜ Additionally, the authors demonstrate how where 𝜖 denotes the error bound for a noisy program 𝑃. error correction schemes can influence robustness, with effective schemes mitigating noise and ineffective ones exacerbating it. Despite its theoretical strength, LQR’s applicability is limited by the challenge of deriving practical quantum predicates. In contrast, Tao et al. introduce Gleipnir, a practical methodology for computing verified error bounds in noisy programs. Gleipnir introduces ˆ 𝛿)-diamond norm, which quantifies the distance 𝛿 between an approximate state 𝜌ˆ and the ( 𝜌, the ideal state 𝜌. By employing tensor networks and Matrix Product States, Gleipnir adaptively computes quantum predicates, enabling scalable error analysis for programs with 10 to 100 qubits. Unlike LQR, Gleipnir provides a computationally efficient method for deriving quantum predicates, bridging theoretical robustness with practical implementation. Muqeet et al. [46] present a search-based approach to approximate the noise models of quantum computers using genetic programming. They develop an expression-based quantum noise model for quantum computers. The approach consists of several expressions that capture different noise aspects, such as depolarizing, amplitude damping, and phase damping errors. They define a fitness function based on the Hellinger distance to evaluate the optimization process. The approach was tested in five random noise models and a real IBM quantum computer using three quantum circuits. Results indicate that this method can accurately estimate the noise with only a 2% deviation for the simulators and a 15% deviation for the actual quantum computers, which is considerably less than the baseline that achieves a 40% deviation. Together, these works have mostly focused on describing how noise affects quantum programs or on building models that capture noise behavior. However, there has been less work on understanding how noise influences the testing process itself. Our study looks at this practical aspect by examining how different quantum testing metrics can detect faults when noise is present. By adjusting a detection threshold, we explore how sensitive each metric is to noise and how reliable it remains under noisy conditions. This helps show which testing approaches are more robust in realistic, noisy quantum environments. 6.2.2 Noise Mitigation. Muqeet et al. propose various approaches for noise mitigation. In their work on QOIN [50], they introduce classical machine learning techniques to learn noise patterns from the outputs of quantum circuits and mitigate their effects. To evaluate their approach, they assess the distributions of the mitigated outputs and employ the Hellinger distance metric to measure the reduction in noise. They also generate artificial faults (i.e., mutants) to determine whether these faults can be detected after noise mitigation. Results show that most of the mutants are influenced by the noise. The influence either amplifies or diminishes the effect of the mutation, bringing it closer to or further away from the original circuit, so distorting its behavior. In a later approach [60], the authors implement QOIN as a noise mitigation technique alongside the property-based testing technique developed by Pontolillo et al. [59, 61]. In this work, the authors demonstrate the applicability of a noise mitigation tool together with a testing technique. The results show that although noise mitigation does not guarantee property preservation, it ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:44

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

does improve alignment with ideal results and significantly reduces false positives when applied selectively. The authors conclude that while QOIN is capable of reducing median and mean noise at the circuit level, a noticeable gap remains in assertion failures compared to ideal execution. In QLEAR [49], Muqeet et al. propose a machine learning approach to extract novel features from noise, enabling the prediction and reduction of estimated noise. The method utilizes various circuit characteristics along with what they refer to as “depth cut” to obtain several relevant features of the circuit, which help in predicting the noise in the output of a quantum program. To estimate the output error, the approach employs the Hellinger distance. The results indicate that the extracted features demonstrate a significant correlation with the output error of the circuit. The authors compare their method with state-of-the-art techniques, showing that QLEAR is effective in estimating noise in both simulations and real quantum computers. Muqeet et al. [48] also implemented the research conducted in QLEAR into a practical tool called QUIET. QUIET consists of two separate modules: a training module and a mitigation module. The training module is responsible for developing a machine learning model using the new feature data from QLEAR. The mitigation module functions as a REST API that applies the trained model to mitigate the effect of noise from the output of quantum circuits. The tool was validated using five real-world quantum programs on both an IBM quantum simulator and a real quantum computer. The authors measured the distance between the original circuit output and the noisy output using the Hellinger distance, and then compared it to the distance after mitigation. The results demonstrated that QUIET significantly reduced the distance from the original circuit output after mitigating the noise for all the quantum circuits in both the simulator and the real quantum computer. All the aforementioned works focus on characterizing and mitigating noise in quantum computers. In contrast, our study does not seek to reduce noise but to understand how existing metrics can adapt to it. Specifically, we investigate how threshold adjustments can help distinguish noiseinduced variations from genuine faults. While prior studies often rely on a single metric (typically the Hellinger distance based on output distributions), we evaluate several metrics and output assessment methods, including Hellinger, to determine their effectiveness in estimating noise effects and detecting faults. Our study aims to empirically identify which approaches best capture the impact of noise and whether faults can be reliably distinguished from it. 7

Concluding Remarks and Research Outlook

This study presented an empirical investigation of noise-aware mutation analysis for quantum programs. By executing 41 quantum programs implementing six algorithms across both noiseless and noisy simulators emulating three IBM backends with distinct noise profiles, we examined how different output assessment metrics and thresholding strategies perform under realistic noise conditions. Our results show that noise substantially alters mutant detectability and that methods effective in noiseless environments cannot be reliably transferred to noisy settings. Density-matrix-based metrics, particularly the trace distance, offer the most precise discrimination between mutants, but their computational cost and limited availability on real hardware constrain practical applicability. Among experimentally feasible options, distribution-based metrics such as the Hellinger distance achieve the best compromise between detection capability and deployability on today’s hardware. We also showed that fixed thresholds derived from noiseless conditions lead to inflated misclassification rates, often resulting in all circuits, equivalent or not, to be detected as mutants; whereas noise-specific thresholds markedly improve detection reliability. However, no single strategy performs optimally for all metrics, showing the need for adaptive, noise-calibrated approaches. The thresholds and procedures derived in this work can be directly reused for comparable noise models or recalibrated for new hardware platforms. More broadly, this study establishes a reproducible methodology for studying how noise interacts with comparison metrics between ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:45

quantum programs. Overall, our findings offer both empirical evidence and methodological guidance toward more reliable testing and analysis of quantum software in noisy environments, contributing to the broader goal of developing robust and trustworthy quantum software. Looking ahead, several research directions emerge for mutant detection. First, future work could explore adaptive thresholding schemes that adjust to noise, backend drift, or circuit-specific traits. Second, investigating how mutations propagate throughout circuit execution might reveal different patterns of resilience or vulnerability. Integrating noise-mitigation techniques into the mutation-analysis workflow could also be an interesting avenue to explore, as it could reduce noise-induced false positives. Lastly, leveraging machine learning for direct classification of mutants from execution traces presents a promising alternative to threshold-based decisions. Together, these research avenues can enhance noise-aware software testing for quantum programs and bridge the gap between simulation and physical hardware deployment. Acknowledgments S. Fortz and M.R. Mousavi are supported by the EPSRC project on Verified Simulation for Large Quantum Systems (VSL-Q), grant reference EP/Y005244/1; by the EPSRC project on Robust and Reliable Quantum Computing (RoaRQ), Investigation 009 Model-based monitoring and calibration of quantum computations and (ModeMCQ) grant reference EP/W032635/1 and by InnovateUK QAssure. E. Mendiluze Usandizaga is supported by Simula’s internal strategic project on quantum software engineering. S. Ali is supported by the Qu-Test project (Project #299827) funded by the Research Council of Norway and Oslo Metropolitan University’s Quantum Hub. P. Arcaini is supported by the ASPIRE grant No. JPMJAP2301, JST. M.R. Mousavi is also supported by the ITEA/InnovateUK projects GENIUS and ITEA/InnovateUK GreenCode. The research presented in this paper has benefited from the Experimental Infrastructure for Exploration of Exascale Computing (eX3), which is financially supported by the Research Council of Norway under contract 270053. References [1] Konstantinos Adamopoulos, Mark Harman, and Robert M. Hierons. 2004. How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution. In Genetic and Evolutionary Computation GECCO 2004, Genetic and Evolutionary Computation Conference, Seattle, WA, USA, June 26-30, 2004, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 3103). Springer, Seattle, WA, USA, 1338–1349. doi:10.1007/978-3-540-24855-2_155 [2] Shaukat Ali, Paolo Arcaini, Xinyi Wang, and Tao Yue. 2021. Assessing the Effectiveness of Input and Output Coverage Criteria for Testing Quantum Programs. In 14th IEEE Conference on Software Testing, Verification and Validation, ICST 2021, Porto de Galinhas, Brazil, April 12-16, 2021. IEEE, Porto de Galinhas, Brazil, 13–23. doi:10.1109/ICST49551.2021. 00014 [3] R Alicki. 2004. Decoherence and the appearance of a classical world in quantum theory. Journal of Physics A: Mathematical and General 37, 5 (2004), 1948–1949. [4] James H. Andrews, Lionel C. Briand, Yvan Labiche, and Akbar Siami Namin. 2006. Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria. IEEE Trans. Software Eng. 32, 8 (2006), 608–624. doi:10.1109/TSE.2006.83 [5] Andrea Arcuri and Lionel C. Briand. 2014. A Hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test. Verification Reliab. 24, 3 (2014), 219–250. doi:10.1002/STVR.1486 [6] Pablo Arnault, Pablo Arrighi, Steven Herbert, Evi Kasnetsi, and Tianyi Li. 2024. A typology of quantum algorithms. CoRR abs/2407.05178 (2024). arXiv:2407.05178 doi:10.48550/ARXIV.2407.05178 [7] Jeff P. Barnes, Colin J. Trout, Dennis Lucarelli, and B. D. Clader. 2017. Quantum error-correction failure distributions: Comparison of coherent and stochastic error models. Phys. Rev. A 95 (Jun 2017), 062338. Issue 6. doi:10.1103/PhysRevA. 95.062338 [8] Khadeejah Bepari, Sarah Malik, Michael Spannowsky, and Simon Williams. 2021. Towards a quantum computing algorithm for helicity amplitudes and parton showers. Phys. Rev. D 103 (Apr 2021), 076020. Issue 7. doi:10.1103/ PhysRevD.103.076020 [9] Teresa Brecht, Wolfgang Pfaff, Chen Wang, Yiwen Chu, Luigi Frunzio, Michel H Devoret, and Robert J Schoelkopf. 2016. Multilayer microwave integrated quantum circuits for scalable quantum computing. npj Quantum Information 2, ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:46

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

1 (Feb. 2016), 16002. doi:10.1038/npjqi.2016.2 [10] Pascal Cerfontaine, René Otten, and Hendrik Bluhm. 2020. Self-Consistent Calibration of Quantum-Gate Sets. Phys. Rev. Appl. 13 (Apr 2020), 044071. Issue 4. doi:10.1103/PhysRevApplied.13.044071 [11] Kausthubh Chandramouli, Kelly Mae Allen, Christopher Mori, Dror Baron, and Mário A. T. Figueiredo. 2025. Statistical Signal Processing for Quantum Error Mitigation. CoRR abs/2506.00683 (2025). arXiv:2506.00683 doi:10.48550/ARXIV. 2506.00683 [12] Isaac L Chuang, Raymond Laflamme, Peter W Shor, and Wojciech H Zurek. 1995. Quantum computers, factoring, and decoherence. Science 270, 5242 (1995), 1633–1635. [13] Ton J. Cleophas and Aeilko H. Zwinderman. 2018. Bayesian Pearson Correlation Analysis. Springer International Publishing, Cham, 111–118. doi:10.1007/978-3-319-92747-3_11 [14] Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological bulletin 114, 3 (1993), 494. doi:10.1037/0033-2909.114.3.494 [15] William G Cochran. 1952. The 𝜒2 test of goodness of fit. The Annals of mathematical statistics (1952), 315–345. [16] Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. Routledge, New York, NY, USA. [17] James D. Cresser. 2011. Quantum Physics Notes. [18] Andrew W Cross, Lev S Bishop, John A Smolin, and Jay M Gambetta. 2017. Open quantum assembly language. arXiv preprint arXiv:1707.03429 (2017). doi:10.48550/arXiv.1707.03429 [19] Samudra Dasgupta and Travis S. Humble. 2022. Characterizing the Reproducibility of Noisy Quantum Circuits. Entropy 24, 2 (2022), 244. doi:10.3390/E24020244 [20] Michael A Fligner and Timothy J Killeen. 1976. Distribution-Free Two-Sample Tests for Scale. J. Amer. Statist. Assoc. 71, 353 (1976), 210–213. arXiv:https://www.tandfonline.com/doi/pdf/10.1080/01621459.1976.10481517 doi:10.1080/ 01621459.1976.10481517 [21] Daniel Fortunato, José Campos, and Rui Abreu. 2022. Mutation Testing of Quantum Programs: A Case Study With Qiskit. IEEE Transactions on Quantum Engineering 3 (2022), 1–17. doi:10.1109/TQE.2022.3195061 [22] Daniel Fortunato, José Campos, and Rui Abreu. 2022. Mutation Testing of Quantum Programs Written in QISKit. In 44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2022, Pittsburgh, PA, USA, May 22-24, 2022. ACM/IEEE, Pittsburgh, PA, USA, 358–359. doi:10.1145/3510454.3528649 [23] Daniel Fortunato, José Campos, and Rui Abreu. 2022. QMutPy: a mutation testing tool for Quantum algorithms and applications in Qiskit. In ISSTA ’22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, South Korea, July 18 - 22, 2022, Sukyoung Ryu and Yannis Smaragdakis (Eds.). ACM, Virtual Event, South Korea, 797–800. doi:10.1145/3533767.3543296 [24] Sophie Fortz and Eñaut Mendiluze. 2026. Supplementary material for the paper “Robust Mutation Analysis of Quantum Programs Under Noise”. https://github.com/sfortz/Noise-aware_Mutants [25] Gordon Fraser and Andreas Zeller. 2012. Mutation-Driven Generation of Unit Tests and Oracles. IEEE Trans. Software Eng. 38, 2 (2012), 278–292. doi:10.1109/TSE.2011.93 [26] Alexei Gilchrist, Nathan K. Langford, and Michael A. Nielsen. 2005. Distance measures to compare real and ideal quantum processes. Phys. Rev. A 71 (Jun 2005), 062310. Issue 6. doi:10.1103/PhysRevA.71.062310 [27] Stanton A. Glantz. 2012. Primer of Biostatistics (7th edition ed.). McGraw Hill, New York. https://www.accessscience. com/content/book/9780071781503 [28] Alexander Grimm, Nicholas E. Frattini, Shruti Puri, Shantanu O. Mundhada, Steven Touzard, Mazyar Mirrahimi, Steven M. Girvin, Shyam Shankar, and Michel H. Devoret. 2020. Stabilization and operation of a Kerr-cat qubit. Nature 584, 7820 (Aug. 2020), 205–209. doi:10.1038/s41586-020-2587-z [29] Bernhard J. M. Grün, David Schuler, and Andreas Zeller. 2009. The Impact of Equivalent Mutants. In Second International Conference on Software Testing Verification and Validation, ICST 2009, Denver, Colorado, USA, April 1-4, 2009, Workshops Proceedings. IEEE Computer Society, Denver, Colorado, USA, 192–199. doi:10.1109/ICSTW.2009.37 [30] Lisa Hales and Sean Hallgren. 2000. An Improved Quantum Fourier Transform Algorithm and Applications. In 41st Annual Symposium on Foundations of Computer Science, FOCS 2000, Redondo Beach, California, USA, November 12-14, 2000. IEEE Computer Society, Redondo Beach, California, USA, 515–525. doi:10.1109/SFCS.2000.892139 [31] Robin Harper, Steven T. Flammia, and Joel J. Wallman. 2020. Efficient learning of quantum noise. Nature Physics 16, 12 (Dec. 2020), 1184–1188. doi:10.1038/s41567-020-0992-8 [32] Shahin Honarvar, Mohammad Reza Mousavi, and Rajagopal Nagarajan. 2020. Property-based Testing of Quantum Programs in Q#. In ICSE ’20: 42nd International Conference on Software Engineering, Workshops, Seoul, Republic of Korea, 27 June - 19 July, 2020. ACM, Seoul, Republic of Korea, 430–435. doi:10.1145/3387940.3391459 [33] Yipeng Huang and Margaret Martonosi. 2019. Statistical assertions for validating patterns and finding bugs in quantum programs. In Proceedings of the 46th International Symposium on Computer Architecture, ISCA 2019, Phoenix, AZ, USA, June 22-26, 2019, Srilatha Bobbie Manne, Hillery C. Hunter, and Erik R. Altman (Eds.). ACM, Phoenix, AZ, USA, 541–553. doi:10.1145/3307650.3322213

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:47

[34] Ziwen Huang, Yao Lu, Eliot Kapit, David I. Schuster, and Jens Koch. 2018. Universal stabilization of single-qubit states using a tunable coupler. Phys. Rev. A 97 (Jun 2018), 062345. Issue 6. doi:10.1103/PhysRevA.97.062345 [35] Shih-Han Hung, Kesha Hietala, Shaopeng Zhu, Mingsheng Ying, Michael Hicks, and Xiaodi Wu. 2019. Quantitative robustness analysis of quantum programs. Proc. ACM Program. Lang. 3, POPL (2019), 31:1–31:29. doi:10.1145/3290344 [36] Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Software Eng. 37, 5 (2011), 649–678. doi:10.1109/TSE.2010.62 [37] Shelby Kimmel, Guang Hao Low, and Theodore J. Yoder. 2015. Robust calibration of a universal single-qubit gate set via robust phase estimation. Phys. Rev. A 92 (Dec 2015), 062315. Issue 6. doi:10.1103/PhysRevA.92.062315 [38] Jonas Klamroth, Max Scheerer, and Oliver Denninger. 2025. Detecting and Tolerating Faults in Hybrid Quantum Software Systems Using Architectural Redundancy. In IEEE International Conference on Quantum Software, QSW 2025, Helsinki, Finland, July 7-12, 2025, Rong N. Chang, Carl K. Chang, Jingwei Yang, Nimanthi Atukorala, Dan Chen, Sumi Helal, Sasu Tarkoma, Qiang He, Tevfik Kosar, Claudio A. Ardagna, Sebastian Feld, Elisabetta Di Nitto, and Manuel Wimmer (Eds.). IEEE, Helsinki, Finland, 162–172. doi:10.1109/QSW67625.2025.00028 [39] Junyu Liu, Frederik Wilde, Antonio Anna Mele, Liang Jiang, and Jens Eisert. 2022. Noise can be helpful for variational quantum algorithms. CoRR abs/2210.06723 (2022). arXiv:2210.06723 doi:10.48550/ARXIV.2210.06723 [40] Yao Lu, S. Chakram, N. Leung, N. Earnest, R. K. Naik, Ziwen Huang, Peter Groszkowski, Eliot Kapit, Jens Koch, and David I. Schuster. 2017. Universal Stabilization of a Parametrically Coupled Qubit. Phys. Rev. Lett. 119 (Oct 2017), 150502. Issue 15. doi:10.1103/PhysRevLett.119.150502 [41] Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Jozala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Trans. Software Eng. 40, 1 (2014), 23–42. doi:10.1109/TSE.2013.44 [42] A. P. Majtey, P. W. Lamberti, and D. P. Prato. 2005. Jensen-Shannon divergence as a measure of distinguishability between mixed quantum states. Phys. Rev. A 72 (Nov 2005), 052310. Issue 5. doi:10.1103/PhysRevA.72.052310 [43] Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50–60. [44] Kane Meissel and Esther S Yao. 2024. Using Cliff’s delta as a non-parametric effect size measure: an accessible web app and R tutorial. Practical Assessment, Research, and Evaluation 29, 1 (2024). doi:10.7275/pare.1977 [45] Eñaut Mendiluze, Shaukat Ali, Paolo Arcaini, and Tao Yue. 2021. Muskit: A Mutation Analysis Tool for Quantum Software Testing. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021. IEEE, Melbourne, Australia, 1266–1270. doi:10.1109/ASE51524.2021.9678563 [46] Asmar Muqeet, Shaukat Ali, and Paolo Arcaini. 2024. Approximating Stochastic Quantum Noise Through Genetic Programming. In Search-Based Software Engineering - 16th International Symposium, SSBSE 2024, Porto de Galinhas, Brazil, July 15, 2024, Proceedings (Lecture Notes in Computer Science, Vol. 14767), Gunel Jahangirova and Foutse Khomh (Eds.). Springer, Porto de Galinhas, Brazil, 56–62. doi:10.1007/978-3-031-64573-0_5 [47] Asmar Muqeet, Shaukat Ali, and Paolo Arcaini. 2024. Quantum Program Testing Through Commuting Pauli Strings on IBM’s Quantum Computers. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024, Vladimir Filkov, Baishakhi Ray, and Minghui Zhou (Eds.). ACM, Sacramento, CA, USA, 2130–2141. doi:10.1145/3691620.3695275 [48] Asmar Muqeet, Shaukat Ali, and Paolo Arcaini. 2025. Tool: QUIET: A Tool for Sampling-Based Quantum Noise Error Mitigation. IEEE Softw. 42, 6 (2025), 28–34. doi:10.1109/MS.2025.3532106 [49] Asmar Muqeet, Shaukat Ali, Tao Yue, and Paolo Arcaini. 2024. A Machine Learning-Based Error Mitigation Approach for Reliable Software Development on IBM’s Quantum Computers. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, Porto de Galinhas, Brazil, July 15-19, 2024, Marcelo d’Amorim (Ed.). ACM, Porto de Galinhas, Brazil, 80–91. doi:10.1145/3663529.3663830 [50] Asmar Muqeet, Tao Yue, Shaukat Ali, and Paolo Arcaini. 2024. Mitigating Noise in Quantum Software Testing Using Machine Learning. IEEE Trans. Software Eng. 50, 11 (2024), 2947–2961. doi:10.1109/TSE.2024.3462974 [51] Michael A. Nielsen and Isaac L. Chuang. 2016. Quantum Computation and Quantum Information (10th Anniversary edition). Cambridge University Press, Cambridge, UK. https://www.cambridge.org/de/academic/subjects/ physics/quantum-physics-quantum-information-and-quantum-computation/quantum-computation-and-quantuminformation-10th-anniversary-edition?format=HB [52] Matteo Paltenghi. 2025. Testing and analysis of quantum software. Ph. D. Dissertation. University of Stuttgart, Germany. https://nbn-resolving.org/urn:nbn:de:bsz:93-opus-ds-167500 [53] Mike Papadakis, Márcio Eduardo Delamaro, and Yves Le Traon. 2014. Mitigating the effects of equivalent mutants with mutant classification strategies. Sci. Comput. Program. 95 (2014), 298–319. doi:10.1016/J.SCICO.2014.05.012 [54] Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique. In 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy, May 16-24, 2015, Volume 1, Antonia Bertolino, Gerardo

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

X:48

Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi

Canfora, and Sebastian G. Elbaum (Eds.). IEEE Computer Society, Florence, Italy, 936–946. doi:10.1109/ICSE.2015.103 [55] Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Adv. Comput. 112 (2019), 275–378. doi:10.1016/BS.ADCOM.2018.03.015 [56] Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. Are mutation scores correlated with real fault detection?: a large scale empirical study on the relationship between mutants and real faults. In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (Eds.). ACM, Gothenburg, Sweden, 537–548. doi:10.1145/3180155.3180183 [57] Tirthak Patel and Devesh Tiwari. 2021. Qraft: reverse your Quantum circuit and know the correct program output. In ASPLOS ’21: 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Virtual Event, USA, April 19-23, 2021, Tim Sherwood, Emery D. Berger, and Christos Kozyrakis (Eds.). ACM, Virtual Event, USA, 443–455. doi:10.1145/3445814.3446743 [58] Kai Petersen and Çigdem Gencel. 2013. Worldviews, Research Methods, and their Relationship to Validity in Empirical Software Engineering Research. In 2013 Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement, Ankara, Turkey, October 23-26, 2013. IEEE Computer Society, Ankara, Turkey, 81–89. doi:10.1109/IWSM-MENSURA.2013.22 [59] Gabriel Pontolillo, Mohammad Reza Mousavi, and Marek Grzesiuk. 2025. QuCheck: A Property-based Testing Framework for Quantum Programs in Qiskit. CoRR abs/2503.22641 (2025). arXiv:2503.22641 doi:10.48550/ARXIV.2503. 22641 [60] Gabriel Pontolillo, Asmar Muqeet, Shaukat Ali, and Mohammad Reza Mousavi. 2025. From Ideal to Noisy: Adapting Property-Based Testing for Real-World Noisy Quantum Computers. In 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), Vol. 01. 405–416. doi:10.1109/QCE65121.2025.00053 [61] Gabriel Joseph Pontolillo and Mohammad Reza Mousavi. 2024. Delta Debugging for Property-Based Regression Testing of Quantum Programs. In Proceedings of the 5th ACM/IEEE International Workshop on Quantum Software Engineering, Q-SE 2024, Lisbon, Portugal, 16 April 2024. ACM, Lisbon, Portugal, 1–8. doi:10.1145/3643667.3648219 [62] J. S. Pratt and J. H. Eberly. 2001. Qubit cross talk and entanglement decay. Phys. Rev. B 64 (Oct 2001), 195314. Issue 19. doi:10.1103/PhysRevB.64.195314 [63] Nils Quetschlich, Lukas Burgholzer, and Robert Wille. 2023. MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing. Quantum 7 (2023), 1062. doi:10.22331/Q-2023-07-20-1062 [64] Timothy C. Ralph. 2012. Howard Wiseman and Gerard Milburn: Quantum measurement and control. Quantum Inf. Process. 11, 1 (2012), 313–315. doi:10.1007/S11128-011-0277-3 [65] Salonik Resch and Ulya R. Karpuzcu. 2022. Benchmarking Quantum Computers and the Impact of Quantum Noise. ACM Comput. Surv. 54, 7 (2022), 142:1–142:35. doi:10.1145/3464420 [66] Diego Riste, Stefano Poletto, M-Z Huang, Alessandro Bruno, Visa Vesterinen, O-P Saira, and Leonardo DiCarlo. 2015. Detecting bit-flip errors in a logical qubit using stabilizer measurements. Nature Communications 6, 1 (April 2015), 6983. doi:10.1038/ncomms7983 [67] Per Runeson and Martin Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Eng. 14, 2 (2009), 131–164. doi:10.1007/S10664-008-9102-8 [68] Mohan Sarovar, Timothy Proctor, Kenneth Rudinger, Kevin C. Young, Erik Nielsen, and Robin Blume-Kohout. 2020. Detecting crosstalk errors in quantum information processors. Quantum 4 (2020), 321. doi:10.22331/Q-2020-09-11-321 [69] Runzhou Tao, Yunong Shi, Jianan Yao, John Hui, Frederic T. Chong, and Ronghui Gu. 2021. Gleipnir: toward practical error analysis for Quantum programs. In PLDI ’21: 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, Virtual Event, Canada, June 20-25, 2021, Stephen N. Freund and Eran Yahav (Eds.). ACM, Virtual Event, Canada, 48–64. doi:10.1145/3453483.3454029 [70] Ewa Tomczak and Maciej Tomczak. 2014. The need to report effect size estimates revisited. An overview of some recommended measures of effect size. TRENDS in Sport Sciences 21, 1 (2014). [71] Eñaut Mendiluze Usandizaga, Shaukat Ali, Tao Yue, and Paolo Arcaini. 2025. Quantum circuit mutants: Empirical analysis and recommendations. Empir. Softw. Eng. 30, 3 (2025), 100. doi:10.1007/S10664-025-10643-Z [72] Xinyi Wang, Paolo Arcaini, and Tao Yue and< Shaukat Ali. 2021. Quito: a Coverage-Guided Test Generator for Quantum Programs. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021. IEEE, Melbourne, Australia, 1237–1241. doi:10.1109/ASE51524.2021.9678798 [73] Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2021. Generating Failing Test Suites for Quantum Programs With Search. In Search-Based Software Engineering - 13th International Symposium, SSBSE 2021, Bari, Italy, October 11-12, 2021, Proceedings (Lecture Notes in Computer Science, Vol. 12914), Una-May O’Reilly and Xavier Devroey (Eds.). Springer, Bari, Italy, 9–25. doi:10.1007/978-3-030-88106-1_2 [74] Dominic Widdows, Jyoti Rani, and Emmanuel M. Pothos. 2023. Quantum Circuit Components for Cognitive DecisionMaking. Entropy 25, 4 (2023), 548. doi:10.3390/E25040548

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.

Robust Mutation Analysis of Quantum Programs Under Noise

X:49

[75] Ellis Wilson, Frank Mueller, Lindsay Bassman, and Costin Iancu. 2021. Empirical evaluation of circuit approximations on noisy quantum devices. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, November 14-19, 2021, Bronis R. de Supinski, Mary W. Hall, and Todd Gamblin (Eds.). ACM, St. Louis, Missouri, USA, 96. doi:10.1145/3458817.3476189 [76] Nicolas Wittler, Federico Roy, Kevin Pack, Max Werninghaus, Anurag Saha Roy, Daniel J. Egger, Stefan Filipp, Frank K. Wilhelm, and Shai Machnes. 2021. Integrated Tool Set for Control, Calibration, and Characterization of Quantum Devices Applied to Superconducting Qubits. Phys. Rev. Appl. 15 (Mar 2021), 034080. Issue 3. doi:10.1103/PhysRevApplied. 15.034080 [77] William K. Wootters and Wojciech H. Zurek. 1982. A single quantum cannot be cloned. Nature 299, 5886 (01 10 1982), 802–803. doi:10.1038/299802a0 [78] Noson S. Yanofsky and Mirco Mannucci. 2008. Quantum Computing for Computer Scientists. Cambridge University Press, Cambridge, UK. [79] Jiaming Ye, Shangzhou Xia, Fuyuan Zhang, Paolo Arcaini, Lei Ma, Jianjun Zhao, and Fuyuki Ishikawa. 2023. QuraTest: Integrating Quantum Specific Features in Quantum Program Testing. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, Luxembourg, 1149–1161. doi:10.1109/ASE56229.2023.00196 [80] Pengzhan Zhao, Jianjun Zhao, and Lei Ma. 2021. Identifying Bug Patterns in Quantum Programs. In 2nd IEEE/ACM International Workshop on Quantum Software Engineering, Q-SE@ICSE 2021, Madrid, Spain, June 1-2, 2021. IEEE, Madrid, Spain, 16–21. doi:10.1109/Q-SE52541.2021.00011 [81] Yuchen Zhu, Jinglei Cheng, Boxi Li, Kecheng Liu, Yidong Zhou, Hanrui Wang, Yufei Ding, and Zhiding Liang. 2025. Hardware-aware Calibration Protocol for Quantum Computers. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, ISCA 2025, Tokyo, Japan, June 21-25, 2025. ACM, Tokyo, Japan, 241–256. doi:10. 1145/3695053.3731036

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: February 2026.